[
  {
    "path": ".clangd",
    "content": "Style:\n  AngledHeaders:\n    # public headers (under include/seastar) use angle brackets\n    - \"seastar/.*\"\n"
  },
  {
    "path": ".claude/CLAUDE.md",
    "content": "# Seastar Project Instructions\n\n## Building\n\nThe project uses CMake with ninja as the build system. Build directories are located in `build/<mode>`.\n\n### Build Commands\n\n**Build specific target:**\n```bash\nninja -C build/<mode> <target>\n```\n\nWhere `<mode>` is typically:\n- `dev` - development build\n- `debug` - debug build\n- `release` - release build\n- `sanitize` - sanitizer build (ASan/UBSan)\n\n**List available targets:**\n```bash\nninja -C build/<mode> -t targets\n```\n\n**List all targets (including test targets):**\n```bash\nninja -C build/<mode> -t targets all\n```\n\n### Common Build Examples\n\n```bash\n# Build ioinfo tool in dev mode\nninja -C build/dev apps/io_tester/ioinfo\n\n# Build io_tester in dev mode\nninja -C build/dev apps/io_tester/io_tester\n\n# Build all in release mode\nninja -C build/release\n\n# List all available targets\nninja -C build/dev -t targets all\n```\n\n### Important Notes\n\n- **Check for existing build directories first** - if `build/<mode>` exists, use it directly; otherwise run `./configure.py --mode=<mode>` to create it\n- **Always use `ninja -C build/<mode>`** from the repository root\n- **Do NOT use** plain `ninja` from the root directory - it will fail with \"Execute Ninja in a build directory\"\n- The working directory during builds should typically be the repository root\n- After code changes to specific components, rebuild only that target for faster iteration\n\n## Testing\n\nThe project uses CTest for running tests. Test executables are built in `build/<mode>/tests/unit/` and `build/<mode>/tests/perf/`.\n\n### Test Runners\n\n**`./test.py`** (preferred) - Python wrapper with useful options:\n```bash\n./test.py --mode dev --name <pattern>  # Run specific test in dev mode\n./test.py --mode dev --fast            # Run only fast tests\n./test.py --mode dev -v                # Verbose output\n```\n\nOptions:\n- `--mode <mode>` - Run for specific build mode (otherwise runs ALL modes)\n- `--name <pattern>` - Filter tests by name\n- `--fast` - Run only fast tests\n- `--timeout <secs>` - Set timeout (default 300s)\n- `--smp <n>` / `-c <n>` - Number of threads for multi-core tests (default 2)\n- `--verbose` / `-v` - Verbose output\n- `--offline` - Disable tests that access the internet\n\n**`ninja -C build/<mode> test`** - Basic ctest invocation for a single mode. Less flexible, no `--output-on-failure` by default.\n\n### Running Specific Tests\n\n```bash\n# Using test.py (preferred)\n./test.py --mode dev --name circular_buffer\n\n# Using ctest directly\nctest --test-dir build/dev -R circular_buffer --output-on-failure\n\n# Run test executable directly\n./build/dev/tests/unit/circular_buffer_test\n```\n\n**List all available tests:**\n```bash\nctest --test-dir build/dev -N\n```\n\n### Test Types\n\n- **Unit tests**: Located in `tests/unit/`, use Boost.Test framework\n- **Performance tests**: Located in `tests/perf/`, benchmark various components\n\n### Important Notes\n\n- **When iterating on code changes**, choose an appropriate test that covers the modified code and run only that test\n- **Running all tests is slow** - only run the full test suite when explicitly requested\n- Most unit tests use Boost.Test and accept standard Boost.Test command line options\n- Some tests (like allocator tests) may run for the default duration (5 seconds) unless interrupted\n\n## Style\n\nMinimum C++ version is C++20. Use C++20 API where possible.\n\nFollow existing style.\n\nComment all public methods carefully following the style of the same source file (e.g., doxygen if existing comments use that style).\n"
  },
  {
    "path": ".dockerignore",
    "content": ".git\nbuild\n"
  },
  {
    "path": ".gitattributes",
    "content": "*.cc diff=cpp\n*.hh diff=cpp\n"
  },
  {
    "path": ".github/workflows/alpinelinux.yaml",
    "content": "name: Alpine Linux\n\non:\n  pull_request:\n  workflow_dispatch:  # Allows manual triggering\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\n\njobs:\n  build-and-test:\n    runs-on: ubuntu-latest\n    container:\n      image: alpine:latest\n\n    steps:\n    - name: Checkout repository\n      uses: actions/checkout@v4\n\n    - name: Install build dependencies\n      run: |\n        apk update\n        apk add --no-cache   \\\n          boost-dev          \\\n          bsd-compat-headers \\\n          c-ares-dev         \\\n          cmake              \\\n          crypto++-dev       \\\n          gcc                \\\n          g++                \\\n          fmt-dev            \\\n          gnutls-dev         \\\n          hwloc-dev          \\\n          libpciaccess-dev   \\\n          libucontext-dev    \\\n          libunwind-dev      \\\n          liburing-dev       \\\n          lksctp-tools-dev   \\\n          lz4-dev            \\\n          numactl-dev        \\\n          openssl            \\\n          openssl-dev        \\\n          protobuf-dev       \\\n          py3-yaml           \\\n          ragel              \\\n          samurai            \\\n          util-linux-dev     \\\n          valgrind-dev       \\\n          xfsprogs-dev       \\\n          yaml-cpp-dev\n\n    - name: Configure build\n      run: |\n        cmake -B build -G Ninja            \\\n         -DCMAKE_BUILD_TYPE=RelWithDebInfo \\\n         -DSeastar_DOCS=OFF\n\n    - name: Build Seastar\n      run: |\n        cmake --build build\n\n    - name: Run unit tests\n      run: |\n        ctest --test-dir build --output-on-failure -j2\n"
  },
  {
    "path": ".github/workflows/docker.yaml",
    "content": "name: Verify Dockerfile Build\n\non:\n  pull_request:\n    paths:\n      - 'docker/dev/Dockerfile'\n      - 'install-dependencies.sh'\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\n\njobs:\n  build:\n    timeout-minutes: 20\n    runs-on: ubuntu-24.04\n    steps:\n      - name: Checkout repository\n        uses: actions/checkout@v4\n        with:\n          sparse-checkout: |\n            docker/dev/Dockerfile\n            install-dependencies.sh\n\n      - name: Set up Docker Buildx\n        uses: docker/setup-buildx-action@v3\n\n      - name: Build Docker image\n        uses: docker/build-push-action@v6\n        with:\n          context: .\n          file: docker/dev/Dockerfile\n          push: false\n          cache-from: type=gha\n          cache-to: type=gha,mode=max\n"
  },
  {
    "path": ".github/workflows/pre-commit.yaml",
    "content": "name: Pre-commit\n\non: [push, pull_request]\n\njobs:\n  pre-commit:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-python@v5\n        with:\n          python-version: '3.x'\n      - uses: pre-commit/action@v3.0.1\n      - name: Show diff of fixes ($ at EOL)\n        if: failure()\n        # trailing whitespace is hard to see, so add $\n        # to the end of diff lines\n        run: git diff | sed '/^[+-]/s/$/$/'\n"
  },
  {
    "path": ".github/workflows/python-lint.yaml",
    "content": "name: Python format\n\non: [push, pull_request]\n\njobs:\n  python-format:\n    name: Enforce python format\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: psf/black@24.8.0\n        with:\n            version: \"24.8.0\"\n            src: ./scripts\n            # override options so that we can specify only specific files for now\n            options: \"--check --diff --include=.*addr2line.*\"\n"
  },
  {
    "path": ".github/workflows/test.yaml",
    "content": "name: Test\n\npermissions:\n  contents: read\n\non:\n  workflow_call:\n    inputs:\n      compiler:\n        description: 'the C++ compiler to use'\n        type: string\n        required: true\n      standard:\n        description: 'the C++ standard to use'\n        type: number\n        required: true\n      mode:\n        description: 'build mode (debug, dev or release)'\n        type: string\n        required: true\n      enables:\n        description: 'the --enable-* option passed to configure.py'\n        type: string\n        default: ''\n        required: false\n      enable-ccache:\n        description: 'build with ccache enabled'\n        type: boolean\n        default: true\n        required: false\n      options:\n        description: 'additional options passed to configure.py'\n        type: string\n        default: ''\n        required: false\n      test-args:\n        description: 'additional arguments passed to test.py'\n        type: string\n        default: ''\n        required: false\n\njobs:\n  test:\n    timeout-minutes: 40\n    runs-on: ubuntu-24.04\n    steps:\n      - uses: actions/checkout@v4\n        with:\n          submodules: \"${{ contains(inputs.enables, 'dpdk') }}\"\n\n      - run: |\n          sudo apt-get update\n\n      - name: Install build dependencies\n        run: |\n          sudo ./install-dependencies.sh\n\n      - name: Install clang++\n        if: ${{ inputs.compiler == 'clang++' }}\n        run: |\n          sudo apt-get -y install clang-20\n\n      - name: Install clang-scan-deps\n        if: ${{ contains(inputs.enables, 'cxx-modules') }}\n        run: |\n          sudo apt-get -y install clang-tools-20\n\n      - name: Install g++\n        if: ${{ inputs.compiler == 'g++' }}\n        run: |\n          sudo apt-get -y install gcc-14 g++-14\n\n      - name: Install ccache\n        if: ${{ inputs.enable-ccache }}\n        run: |\n          sudo apt-get -y install ccache\n\n      - name: Setup ccache\n        if: ${{ inputs.enable-ccache }}\n        uses: hendrikmuhs/ccache-action@v1\n        with:\n          key: ${{ inputs.compiler }}-${{ inputs.standard }}-${{ inputs.mode }}-${{ inputs.enables }}\n\n      - name: Configure\n        run: |\n          if [ ${{ inputs.compiler }} = \"clang++\" ]; then\n            CC=clang-20\n            CPP=clang++-20\n          else\n            CC=gcc-14\n            CPP=g++-14\n          fi\n          if ${{ inputs.enable-ccache }}; then\n            MAYBE_CCACHE_OPT=\"--compiler-cache=ccache\"\n          fi\n          ./configure.py                          \\\n            --c++-standard ${{ inputs.standard }} \\\n            --compiler $CPP                       \\\n            --c-compiler $CC                      \\\n            --mode ${{ inputs.mode }}             \\\n            $MAYBE_CCACHE_OPT                     \\\n            ${{ inputs.options }}                 \\\n            ${{ inputs.enables }}\n\n      - name: Build\n        run: cmake --build build/${{inputs.mode}}\n\n      - name: Check Header\n        if: ${{ inputs.mode == 'dev' && inputs.compiler == 'clang++' }}\n        run: cmake --build build/${{ inputs.mode }} --target checkheaders\n\n      - name: Check Include Style\n        if: ${{ inputs.mode == 'dev' && inputs.compiler == 'clang++' }}\n        run: cmake --build build/${{ inputs.mode }} --target check-include-style\n\n      - name: Build with C++20 modules\n        if: ${{ contains(inputs.enables, 'cxx-modules') }}\n        run: cmake --build build/${{ inputs.mode }} --target hello_cxx_module\n\n      - name: Test\n        if: ${{ ! contains(inputs.enables, 'cxx-modules') }}\n        run: ./test.py --mode=${{ inputs.mode }} ${{ inputs.test-args }}\n"
  },
  {
    "path": ".github/workflows/tests.yaml",
    "content": "name: Test\n\npermissions:\n  contents: read\n\non:\n  push:\n  pull_request:\n  workflow_dispatch:  # Allows manual triggering\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\n\njobs:\n  regular_test:\n    name: \"Test (${{ matrix.compiler }}, C++${{ matrix.standard}}, ${{ matrix.mode }})\"\n    uses: ./.github/workflows/test.yaml\n    strategy:\n      fail-fast: false\n      matrix:\n        compiler: [clang++, g++]\n        standard: [20, 23]\n        mode: [dev, debug, release]\n    with:\n      compiler: ${{ matrix.compiler }}\n      standard: ${{ matrix.standard }}\n      mode: ${{ matrix.mode }}\n      enables: ${{ matrix.enables }}\n      options: ${{ matrix.options }}\n  build_with_dpdk:\n    name: \"Test with DPDK enabled\"\n    uses: ./.github/workflows/test.yaml\n    strategy:\n      fail-fast: false\n    with:\n      compiler: clang++\n      standard: 23\n      mode: release\n      enables: --enable-dpdk\n      options: --cook dpdk --dpdk-machine corei7-avx\n\n  build_with_cxx_modules:\n    name: \"Test with C++20 modules enabled\"\n    uses: ./.github/workflows/test.yaml\n    strategy:\n      fail-fast: false\n    with:\n      compiler: clang++\n      standard: 23\n      mode: debug\n      enables: --enable-cxx-modules\n      enable-ccache: false\n\n  fuzz_test:\n    name: \"Fuzz Tests\"\n    uses: ./.github/workflows/test.yaml\n    with:\n      compiler: clang++\n      standard: 23\n      mode: fuzz\n      test-args: \"-- -R 'Seastar.fuzz.'\"\n"
  },
  {
    "path": ".gitignore",
    "content": ".cooking_memory\n.cproject\n.project\n.settings\nbuild*\nbuild.ninja\ncscope.*\n__pycache__/\ncmake/Cooking.cmake\ntags\n.idea/\n.vscode/\ncompile_commands.json\n.cache\n"
  },
  {
    "path": ".gitmodules",
    "content": "[submodule \"dpdk\"]\n\tpath = dpdk\n\turl = ../dpdk\n"
  },
  {
    "path": ".gitorderfile",
    "content": "*.py\n*.hh\n*.rl\n*.cc\n*\n"
  },
  {
    "path": ".mailmap",
    "content": "Avi Kivity <avi@scylladb.com> Avi Kivity' via seastar-dev <seastar-dev@googlegroups.com>\nRaphael S. Carvalho <raphaelsc@scylladb.com> Raphael S. Carvalho' via seastar-dev <seastar-dev@googlegroups.com>\nPavel Emelyanov <xemul@scylladb.com> Pavel Emelyanov' via seastar-dev <seastar-dev@googlegroups.com>\n"
  },
  {
    "path": ".pre-commit-config.yaml",
    "content": "repos:\n  - repo: https://github.com/pre-commit/pre-commit-hooks\n    rev: v5.0.0\n    hooks:\n      - id: trailing-whitespace\n"
  },
  {
    "path": "CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\ncmake_minimum_required (VERSION 3.13)\n\nlist (APPEND CMAKE_MODULE_PATH\n  ${CMAKE_CURRENT_SOURCE_DIR}/cmake\n  ${CMAKE_CURRENT_BINARY_DIR})\n\ncmake_policy (SET CMP0090 NEW)\nforeach (policy CMP0127 CMP0135 CMP0167)\n  if (POLICY ${policy})\n    cmake_policy (SET ${policy} NEW)\n  endif ()\nendforeach ()\n\ninclude (Cooking OPTIONAL)\n\n# This variable impacts the way DPDK is configured by cmake-cooking (if DPDK is enabled), so its definition needs to\n# come before PROJECT.\nset (Seastar_DPDK_MACHINE\n  \"native\"\n  CACHE\n  STRING\n  \"Configure DPDK for this processor architecture (if `Seastar_DPDK` is enabled). It configures -march or -mcpu\")\n\nproject (Seastar\n  VERSION 1.0\n  LANGUAGES CXX)\n\nset (Seastar_ALLOC_FAILURE_INJECTION\n  \"DEFAULT\"\n  CACHE\n  STRING\n  \"Enable failure injection into the Seastar allocator. Can be ON, OFF or DEFAULT (which enables it for Dev mode)\")\n\noption (Seastar_TASK_BACKTRACE\n  \"Collect backtrace at deferring points.\"\n  OFF)\n\noption (Seastar_DEBUG_ALLOCATIONS\n  \"For now just writes 0xab to newly allocated memory\"\n  OFF)\n\noption (Seastar_SSTRING\n  \"Use seastar's own string implementation\"\n  ON)\n\noption (Seastar_DEPRECATED_OSTREAM_FORMATTERS\n  \"Enable operator<< for formatting standard library containers, which will be deprecated in future\"\n  ON)\n\nset (Seastar_API_LEVEL\n  \"9\"\n  CACHE\n  STRING\n  \"Seastar compatibility API level (7=unified CPU/IO scheduling groups, 8=noncopyable function in json_return_type, 9=new sink API\")\n\nset_property (CACHE Seastar_API_LEVEL\n  PROPERTY\n  STRINGS 7)\n\nset (Seastar_SCHEDULING_GROUPS_COUNT\n  \"16\"\n  CACHE\n  STRING\n  \"A positive number to set Seastar's reactor number of allowed different scheduling groups.\")\n\nif (NOT Seastar_SCHEDULING_GROUPS_COUNT MATCHES \"^[1-9][0-9]*\")\n  message(FATAL_ERROR \"Seastar_SCHEDULING_GROUPS_COUNT must be a positive number (${Seastar_SCHEDULING_GROUPS_COUNT})\")\nendif ()\n\n#\n# Add a dev build type.\n#\n# All pre-defined build modes include optimizations or debug info,\n# which make them slow to build. The dev build mode is intended for\n# fast build/test iteration.\n#\n\nif (CMAKE_CXX_COMPILER_ID MATCHES Clang)\n  set (CMAKE_CXX_FLAGS_DEV_OPT_LEVEL \"-O2\")\nelse ()\n  set (CMAKE_CXX_FLAGS_DEV_OPT_LEVEL \"-O1\")\nendif ()\n\nset (CMAKE_CXX_FLAGS_DEV\n  \"${CMAKE_CXX_FLAGS_DEV_OPT_LEVEL}\"\n  CACHE\n  STRING\n  \"Flags used by the C++ compiler during dev builds.\"\n  FORCE)\n\nset (CMAKE_C_FLAGS_DEV\n  \"-O1\"\n  CACHE\n  STRING\n  \"Flags used by the C compiler during dev builds.\"\n  FORCE)\n\nset (CMAKE_EXE_LINKER_FLAGS_DEV\n  \"\"\n  CACHE\n  STRING\n  \"Flags used for linking binaries during dev builds.\"\n  FORCE)\n\nset (CMAKE_SHARED_LINKER_FLAGS_DEV\n  \"\"\n  CACHE\n  STRING\n  \"Flags used by the shared libraries linker during builds.\"\n  FORCE)\n\nmark_as_advanced (\n    CMAKE_CXX_FLAGS_DEV\n    CMAKE_C_FLAGS_DEV\n    CMAKE_EXE_LINKER_FLAGS_DEV\n    CMAKE_SHARED_LINKER_FLAGS_DEV)\n\nset (CMAKE_CXX_FLAGS_SANITIZE\n  \"-Os -g\"\n  CACHE\n  STRING\n  \"Flags used by the C++ compiler during sanitize builds.\"\n  FORCE)\n\nset (CMAKE_CXX_STANDARD\n  \"23\"\n  CACHE\n  STRING\n  \"C++ standard to build with.\")\n\ninclude (CMakeDependentOption)\ncmake_dependent_option (Seastar_MODULE\n  \"Build a C++20 module instead of a traditional library\" OFF\n  \"CMAKE_VERSION VERSION_GREATER_EQUAL 3.26;CMAKE_CXX_STANDARD GREATER_EQUAL 20\" OFF)\n\nset (CMAKE_C_FLAGS_SANITIZE\n  \"-Os -g\"\n  CACHE\n  STRING\n  \"Flags used by the C compiler during sanitize builds.\"\n  FORCE)\n\nset (CMAKE_EXE_LINKER_FLAGS_SANITIZE\n  \"\"\n  CACHE\n  STRING\n  \"Flags used for linking binaries during sanitize builds.\"\n  FORCE)\n\nset (CMAKE_SHARED_LINKER_FLAGS_SANITIZE\n  \"\"\n  CACHE\n  STRING\n  \"Flags used by the shared libraries linker during sanitize builds.\"\n  FORCE)\n\nmark_as_advanced (\n    CMAKE_CXX_FLAGS_SANITIZE\n    CMAKE_C_FLAGS_SANITIZE\n    CMAKE_EXE_LINKER_FLAGS_SANITIZE\n    CMAKE_SHARED_LINKER_FLAGS_SANITIZE)\n\nset (CMAKE_CXX_FLAGS_FUZZ\n  \"-O1 -g\"\n  CACHE\n  STRING\n  \"Flags used by the C++ compiler during fuzz builds.\"\n  FORCE)\n\nset (CMAKE_C_FLAGS_FUZZ\n  \"-O1 -g\"\n  CACHE\n  STRING\n  \"Flags used by the C compiler during fuzz builds.\"\n  FORCE)\n\nset (CMAKE_EXE_LINKER_FLAGS_FUZZ\n  \"\"\n  CACHE\n  STRING\n  \"Flags used for linking binaries during fuzz builds.\"\n  FORCE)\n\nset (CMAKE_SHARED_LINKER_FLAGS_FUZZ\n  \"\"\n  CACHE\n  STRING\n  \"Flags used by the shared libraries linker during fuzz builds.\"\n  FORCE)\n\nmark_as_advanced (\n    CMAKE_CXX_FLAGS_FUZZ\n    CMAKE_C_FLAGS_FUZZ\n    CMAKE_EXE_LINKER_FLAGS_FUZZ\n    CMAKE_SHARED_LINKER_FLAGS_FUZZ)\n\nset (CMAKE_BUILD_TYPE\n  \"${CMAKE_BUILD_TYPE}\"\n  CACHE\n  STRING\n  \"Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel Dev Sanitize Fuzz.\"\n  FORCE)\n\nif (NOT CMAKE_BUILD_TYPE)\n  set (CMAKE_BUILD_TYPE \"Release\")\nendif ()\n\nset (Seastar_ALLOC_PAGE_SIZE\n  \"\"\n  CACHE\n  STRING\n  \"Override the Seastar allocator page size, in bytes.\")\n\nfunction (set_option_if_package_is_found option_name package_name)\n  # if the package is found, set the option on behalf of user unless it is\n  # explicitly specified,\n  if (DEFINED ${option_name})\n    return ()\n  endif ()\n  if (${package_name}_FOUND)\n    set (${option_name} ON CACHE BOOL \"\")\n  endif ()\nendfunction ()\n\n# When Seastar is a top-level project, enable the non-library targets by default.\n# If it is embedded with `add_subdirectory`, disable them.\nif (CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_SOURCE_DIR)\n  set (Seastar_MASTER_PROJECT ON)\nelse ()\n  set (Seastar_MASTER_PROJECT OFF)\nendif ()\n\noption (Seastar_APPS\n  \"Enable application targets.\"\n  ${Seastar_MASTER_PROJECT})\n\nset (Seastar_CXX_FLAGS\n  \"\"\n  CACHE\n  STRING\n  \"Semicolon-separated list of extra compilation flags for Seastar itself.\")\n\nset (Seastar_SOURCE_SUBDIR\n  \"\"\n  CACHE\n  STRING\n  \"When non-empty, add -ffile-prefix-map rules so that gdb can resolve Seastar sources. Set to the path of the Seastar source tree relative to the consumer project root (e.g. 'seastar/'). The rules map CMAKE_CURRENT_BINARY_DIR to '.' and paths under it to this subdir, and are appended after Seastar_CXX_FLAGS so they take precedence over any broader -ffile-prefix-map rule.\")\n\noption (Seastar_DEMOS\n  \"Enable demonstration targets.\"\n  ${Seastar_MASTER_PROJECT})\n\noption (Seastar_DOCS\n  \"Enable documentation targets.\"\n  ${Seastar_MASTER_PROJECT})\n\noption (Seastar_DPDK\n  \"Enable DPDK support.\"\n  OFF)\n\noption (Seastar_EXCLUDE_APPS_FROM_ALL\n  \"When enabled alongside Seastar_APPS, do not build applications by default.\"\n  OFF)\n\noption (Seastar_EXCLUDE_DEMOS_FROM_ALL\n  \"When enabled alongside Seastar_DEMOS, do not build demonstrations by default.\"\n  OFF)\n\noption (Seastar_EXCLUDE_TESTS_FROM_ALL\n  \"When enabled alongside Seastar_TESTING, do not build tests by default.\"\n  OFF)\n\noption (Seastar_EXECUTE_ONLY_FAST_TESTS\n  \"Only execute tests which run quickly.\"\n  OFF)\n\noption (Seastar_HWLOC\n  \"Enable hwloc support.\"\n  ON)\n\noption (Seastar_IO_URING\n  \"Enable io_uring support.\"\n  ON)\n\nset (Seastar_JENKINS\n  \"\"\n  CACHE\n  STRING\n  \"If non-empty, the prefix for XML files containing the results of running tests (for Jenkins).\")\n\nset (Seastar_LD_FLAGS\n  \"\"\n  CACHE\n  STRING\n  \"Semicolon-separated list of extra linking flags for Seastar itself.\")\n\noption (Seastar_INSTALL\n  \"Install targets.\"\n  ${Seastar_MASTER_PROJECT})\n\noption (Seastar_TESTING\n  \"Enable testing targets.\"\n  ${Seastar_MASTER_PROJECT})\n\ninclude (CMakeDependentOption)\ncmake_dependent_option (Seastar_ENABLE_TESTS_ACCESSING_INTERNET\n  \"Enable tests accessing internet.\" ON\n  \"Seastar_TESTING\" OFF)\n\noption (Seastar_COMPRESS_DEBUG\n  \"Compress debug info.\"\n  ON)\n\noption (Seastar_SPLIT_DWARF\n  \"Use split dwarf.\"\n  OFF)\n\noption (Seastar_HEAP_PROFILING\n    \"Enable heap profiling. No effect when Seastar is compiled with the default allocator.\"\n    OFF)\n\noption (Seastar_DEFERRED_ACTION_REQUIRE_NOEXCEPT\n    \"Enable noexcept requirement for deferred actions.\"\n    ON)\n\nset (Seastar_TEST_TIMEOUT\n  \"300\"\n  CACHE\n  STRING\n  \"Maximum allowed time for a test to run, in seconds.\")\n\noption (BUILD_SHARED_LIBS\n  \"Build seastar library as shared libraries instead of static\"\n  OFF)\n# We set the following environment variables\n# * ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1:verify_asan_link_order=0\n#   By default ASan disables core dumps because they used to be\n#   huge. This is no longer the case since the shadow memory is\n#   excluded, so it is safe to enable them.\n# * UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1\n#   Fail the test if any undefined behavior is found and use abort\n#   instead of exit. Using abort is what causes core dumps to be\n#   produced.\n# * BOOST_TEST_CATCH_SYSTEM_ERRORS=no\n#   Normally the boost test library handles SIGABRT and prevents core\n#   dumps from being produced.\n# This works great with clang and gcc 10.2, but unfortunately not any\n# previous gcc.\nset (Seastar_ASAN_OPTIONS \"disable_coredump=0:abort_on_error=1\")\nif ((NOT (CMAKE_CXX_COMPILER_ID STREQUAL \"GNU\")) OR\n    (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 10.2))\n  string (APPEND Seastar_ASAN_OPTIONS \":detect_stack_use_after_return=1\")\nendif ()\n\nset (Seastar_TEST_ENVIRONMENT\n  \"ASAN_OPTIONS=${Seastar_ASAN_OPTIONS};UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1;BOOST_TEST_CATCH_SYSTEM_ERRORS=no\"\n  CACHE\n  STRING\n  \"Environment variables for running tests\")\n\noption (Seastar_UNUSED_RESULT_ERROR\n  \"Make [[nodiscard]] violations an error (instead of a warning).\"\n  OFF)\n\nset (Seastar_STACK_GUARDS\n  \"DEFAULT\"\n  CACHE\n  STRING\n  \"Enable stack guards. Can be ON, OFF or DEFAULT (which enables it for non release builds)\")\n\nset (Seastar_SANITIZE\n  \"DEFAULT\"\n  CACHE\n  STRING\n  \"Enable ASAN and UBSAN. Can be ON, OFF or DEFAULT (which enables it for Debug and Sanitize)\")\n\nset (Seastar_DEBUG_SHARED_PTR\n  \"DEFAULT\"\n  CACHE\n  STRING\n  \"Enable shared_ptr debugging. Can be ON, OFF or DEFAULT (which enables it for Debug and Sanitize)\")\n\n#\n# Useful (non-cache) variables.\n#\n\nset (Seastar_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})\nset (Seastar_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})\nset (Seastar_GEN_BINARY_DIR ${Seastar_BINARY_DIR}/gen)\n\n#\n# Dependencies.\n#\n\ninclude (SeastarDependencies)\nseastar_find_dependencies ()\n\n# Private build dependencies not visible to consumers\nfind_package (ragel 6.10 REQUIRED)\nfind_package (Threads REQUIRED)\nfind_package (PthreadSetName REQUIRED)\nfind_package (Valgrind REQUIRED)\n\ncmake_dependent_option (Seastar_LOGGER_COMPILE_TIME_FMT\n  \"Enable the compile-time {fmt} check when formatting logging messages\" ON\n  \"fmt_VERSION VERSION_GREATER_EQUAL 9.0.0\" OFF)\n\n#\n# Code generation helpers.\n#\n\nfunction (seastar_generate_protobuf)\n  set (one_value_args TARGET VAR IN_FILE OUT_DIR)\n  cmake_parse_arguments (args \"\" \"${one_value_args}\" \"\" ${ARGN})\n  get_filename_component (in_file_name ${args_IN_FILE} NAME_WE)\n  get_filename_component (in_file_dir ${args_IN_FILE} DIRECTORY)\n  set (header_out ${args_OUT_DIR}/${in_file_name}.pb.h)\n  set (source_out ${args_OUT_DIR}/${in_file_name}.pb.cc)\n\n  add_custom_command (\n    DEPENDS ${args_IN_FILE} protobuf::protoc\n    OUTPUT ${header_out} ${source_out}\n    COMMAND ${CMAKE_COMMAND} -E make_directory ${args_OUT_DIR}\n    COMMAND protobuf::protoc\n    ARGS --cpp_out=${args_OUT_DIR} -I${in_file_dir} ${args_IN_FILE})\n\n  add_custom_target (${args_TARGET}\n    DEPENDS\n      ${header_out}\n      ${source_out})\n\n  set (${args_VAR} ${header_out} ${source_out} PARENT_SCOPE)\nendfunction ()\n\nfunction (seastar_generate_ragel)\n  set (one_value_args TARGET VAR IN_FILE OUT_FILE)\n  cmake_parse_arguments (args \"\" \"${one_value_args}\" \"\" ${ARGN})\n  get_filename_component (out_dir ${args_OUT_FILE} DIRECTORY)\n\n  add_custom_command (\n    DEPENDS ${args_IN_FILE}\n    OUTPUT ${args_OUT_FILE}\n    COMMAND ${CMAKE_COMMAND} -E make_directory ${out_dir}\n    COMMAND ${ragel_RAGEL_EXECUTABLE} -G2 -o ${args_OUT_FILE} ${args_IN_FILE}\n    COMMAND sed -i -e \"'1h;2,$$H;$$!d;g'\" -re \"'s/static const char _nfa[^;]*;//g'\" ${args_OUT_FILE}\n    COMMAND touch ${args_OUT_FILE} -r ${args_IN_FILE})\n\n  add_custom_target (${args_TARGET}\n    DEPENDS ${args_OUT_FILE})\n\n  set (${args_VAR} ${args_OUT_FILE} PARENT_SCOPE)\nendfunction ()\n\nfunction (seastar_generate_swagger)\n  set (one_value_args TARGET VAR IN_FILE OUT_DIR)\n  cmake_parse_arguments (args \"\" \"${one_value_args}\" \"\" ${ARGN})\n  get_filename_component (in_file_name ${args_IN_FILE} NAME)\n  set (generator ${Seastar_SOURCE_DIR}/scripts/seastar-json2code.py)\n  set (header_out ${args_OUT_DIR}/${in_file_name}.hh)\n  set (source_out ${args_OUT_DIR}/${in_file_name}.cc)\n\n  add_custom_command (\n    DEPENDS\n      ${args_IN_FILE}\n      ${generator}\n    OUTPUT ${header_out} ${source_out}\n    COMMAND ${CMAKE_COMMAND} -E make_directory ${args_OUT_DIR}\n    COMMAND ${generator} --create-cc -f ${args_IN_FILE} -o ${header_out})\n\n  add_custom_target (${args_TARGET}\n    DEPENDS\n      ${header_out}\n      ${source_out})\n\n  set (${args_VAR} ${header_out} ${source_out} PARENT_SCOPE)\nendfunction ()\n\n#\n# The `seastar` library.\n#\n\nseastar_generate_ragel (\n  TARGET seastar_http_chunk_parsers\n  VAR http_chunk_parsers_file\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/src/http/chunk_parsers.rl\n  OUT_FILE ${Seastar_GEN_BINARY_DIR}/include/seastar/http/chunk_parsers.hh)\n\nseastar_generate_ragel (\n  TARGET seastar_http_request_parser\n  VAR http_request_parser_file\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/src/http/request_parser.rl\n  OUT_FILE ${Seastar_GEN_BINARY_DIR}/include/seastar/http/request_parser.hh)\n\nseastar_generate_ragel (\n  TARGET seastar_http_response_parser\n  VAR http_response_parser_file\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/src/http/response_parser.rl\n  OUT_FILE ${Seastar_GEN_BINARY_DIR}/include/seastar/http/response_parser.hh)\n\nseastar_generate_protobuf (\n  TARGET seastar_proto_metrics2\n  VAR proto_metrics2_files\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/src/proto/metrics2.proto\n  OUT_DIR ${Seastar_GEN_BINARY_DIR}/src/proto)\n\nadd_library (seastar\n  ${http_chunk_parsers_file}\n  ${http_request_parser_file}\n  ${proto_metrics2_files}\n  ${seastar_dpdk_obj}\n  include/seastar/core/abort_source.hh\n  include/seastar/core/alien.hh\n  include/seastar/core/align.hh\n  include/seastar/core/aligned_buffer.hh\n  include/seastar/core/app-template.hh\n  include/seastar/core/bitops.hh\n  include/seastar/core/bitset-iter.hh\n  include/seastar/core/byteorder.hh\n  include/seastar/core/cacheline.hh\n  include/seastar/core/checked_ptr.hh\n  include/seastar/core/chunked_fifo.hh\n  include/seastar/core/circular_buffer.hh\n  include/seastar/core/circular_buffer_fixed_capacity.hh\n  include/seastar/core/condition-variable.hh\n  include/seastar/core/deleter.hh\n  include/seastar/core/disk_params.hh\n  include/seastar/core/distributed.hh\n  include/seastar/core/do_with.hh\n  include/seastar/core/dpdk_rte.hh\n  include/seastar/core/enum.hh\n  include/seastar/core/exception_hacks.hh\n  include/seastar/core/execution_stage.hh\n  include/seastar/core/expiring_fifo.hh\n  include/seastar/core/fair_queue.hh\n  include/seastar/core/file.hh\n  include/seastar/core/file-types.hh\n  include/seastar/core/fsqual.hh\n  include/seastar/core/fstream.hh\n  include/seastar/core/function_traits.hh\n  include/seastar/core/future-util.hh\n  include/seastar/core/future.hh\n  include/seastar/core/gate.hh\n  include/seastar/core/iostream-impl.hh\n  include/seastar/core/iostream.hh\n  include/seastar/util/later.hh\n  include/seastar/core/layered_file.hh\n  include/seastar/core/loop.hh\n  include/seastar/core/lowres_clock.hh\n  include/seastar/core/manual_clock.hh\n  include/seastar/core/map_reduce.hh\n  include/seastar/core/memory.hh\n  include/seastar/core/metrics.hh\n  include/seastar/core/metrics_api.hh\n  include/seastar/core/metrics_registration.hh\n  include/seastar/core/metrics_types.hh\n  include/seastar/core/pipe.hh\n  include/seastar/core/posix.hh\n  include/seastar/core/preempt.hh\n  include/seastar/core/prefetch.hh\n  include/seastar/core/print.hh\n  include/seastar/core/prometheus.hh\n  include/seastar/core/queue.hh\n  include/seastar/core/ragel.hh\n  include/seastar/core/reactor.hh\n  include/seastar/core/report_exception.hh\n  include/seastar/core/resource.hh\n  include/seastar/core/rwlock.hh\n  include/seastar/core/scattered_message.hh\n  include/seastar/core/scheduling.hh\n  include/seastar/core/scollectd.hh\n  include/seastar/core/scollectd_api.hh\n  include/seastar/core/seastar.hh\n  include/seastar/core/smp.hh\n  include/seastar/core/semaphore.hh\n  include/seastar/core/shard_id.hh\n  include/seastar/core/sharded.hh\n  include/seastar/core/shared_future.hh\n  include/seastar/core/shared_mutex.hh\n  include/seastar/core/shared_ptr.hh\n  include/seastar/core/shared_ptr_debug_helper.hh\n  include/seastar/core/shared_ptr_incomplete.hh\n  include/seastar/core/simple-stream.hh\n  include/seastar/core/signal.hh\n  include/seastar/core/slab.hh\n  include/seastar/core/sleep.hh\n  include/seastar/core/sstring.hh\n  include/seastar/core/stall_sampler.hh\n  include/seastar/core/stream.hh\n  include/seastar/core/task.hh\n  include/seastar/core/temporary_buffer.hh\n  include/seastar/core/thread.hh\n  include/seastar/core/thread_cputime_clock.hh\n  include/seastar/core/thread_impl.hh\n  include/seastar/core/timed_out_error.hh\n  include/seastar/core/timer-set.hh\n  include/seastar/core/timer.hh\n  include/seastar/core/transfer.hh\n  include/seastar/core/unaligned.hh\n  include/seastar/core/units.hh\n  include/seastar/core/vector-data-sink.hh\n  include/seastar/core/weak_ptr.hh\n  include/seastar/core/when_all.hh\n  include/seastar/core/with_scheduling_group.hh\n  include/seastar/core/with_timeout.hh\n  include/seastar/http/api_docs.hh\n  include/seastar/http/common.hh\n  include/seastar/http/exception.hh\n  include/seastar/http/file_handler.hh\n  include/seastar/http/function_handlers.hh\n  include/seastar/http/handlers.hh\n  include/seastar/http/httpd.hh\n  include/seastar/http/json_path.hh\n  include/seastar/http/matcher.hh\n  include/seastar/http/matchrules.hh\n  include/seastar/http/mime_types.hh\n  include/seastar/http/reply.hh\n  include/seastar/http/request.hh\n  include/seastar/http/routes.hh\n  include/seastar/http/short_streams.hh\n  include/seastar/http/transformers.hh\n  include/seastar/http/client.hh\n  include/seastar/json/formatter.hh\n  include/seastar/json/json_elements.hh\n  include/seastar/net/api.hh\n  include/seastar/net/arp.hh\n  include/seastar/net/byteorder.hh\n  include/seastar/net/config.hh\n  include/seastar/net/const.hh\n  include/seastar/net/dhcp.hh\n  include/seastar/net/dns.hh\n  include/seastar/net/dpdk.hh\n  include/seastar/net/ethernet.hh\n  include/seastar/net/inet_address.hh\n  include/seastar/net/ip.hh\n  include/seastar/net/ip_checksum.hh\n  include/seastar/net/native-stack.hh\n  include/seastar/net/net.hh\n  include/seastar/net/packet-data-source.hh\n  include/seastar/net/packet-util.hh\n  include/seastar/net/packet.hh\n  include/seastar/net/posix-stack.hh\n  include/seastar/net/proxy.hh\n  include/seastar/net/socket_defs.hh\n  include/seastar/net/stack.hh\n  include/seastar/net/tcp-stack.hh\n  include/seastar/net/tcp.hh\n  include/seastar/net/tls.hh\n  include/seastar/net/toeplitz.hh\n  include/seastar/net/udp.hh\n  include/seastar/net/unix_address.hh\n  include/seastar/net/virtio-interface.hh\n  include/seastar/net/virtio.hh\n  include/seastar/rpc/lz4_compressor.hh\n  include/seastar/rpc/lz4_fragmented_compressor.hh\n  include/seastar/rpc/multi_algo_compressor_factory.hh\n  include/seastar/rpc/rpc.hh\n  include/seastar/rpc/rpc_impl.hh\n  include/seastar/rpc/rpc_types.hh\n  include/seastar/util/alloc_failure_injector.hh\n  include/seastar/util/backtrace.hh\n  include/seastar/util/bool_class.hh\n  include/seastar/util/conversions.hh\n  include/seastar/util/defer.hh\n  include/seastar/util/eclipse.hh\n  include/seastar/util/function_input_iterator.hh\n  include/seastar/util/indirect.hh\n  include/seastar/util/is_smart_ptr.hh\n  include/seastar/util/lazy.hh\n  include/seastar/util/log-cli.hh\n  include/seastar/util/log-impl.hh\n  include/seastar/util/log.hh\n  include/seastar/util/noncopyable_function.hh\n  include/seastar/util/optimized_optional.hh\n  include/seastar/util/print_safe.hh\n  include/seastar/util/process.hh\n  include/seastar/util/program-options.hh\n  include/seastar/util/read_first_line.hh\n  include/seastar/util/reference_wrapper.hh\n  include/seastar/util/spinlock.hh\n  include/seastar/util/std-compat.hh\n  include/seastar/util/transform_iterator.hh\n  include/seastar/util/tuple_utils.hh\n  include/seastar/util/variant_utils.hh\n  include/seastar/util/closeable.hh\n  include/seastar/util/short_streams.hh\n  include/seastar/websocket/client.hh\n  include/seastar/websocket/common.hh\n  include/seastar/websocket/server.hh\n  src/core/alien.cc\n  src/core/file.cc\n  src/core/fair_queue.cc\n  src/core/reactor_backend.cc\n  src/core/thread_pool.cc\n  src/core/app-template.cc\n  src/core/disk_params.cc\n  src/core/dpdk_rte.cc\n  src/core/exception_hacks.cc\n  src/core/execution_stage.cc\n  src/core/file-impl.hh\n  src/core/fsnotify.cc\n  src/core/fsqual.cc\n  src/core/fstream.cc\n  src/core/future.cc\n  src/core/future-util.cc\n  src/core/linux-aio.cc\n  src/core/memory.cc\n  src/core/metrics.cc\n  src/core/on_internal_error.cc\n  src/core/posix.cc\n  src/core/prometheus.cc\n  src/core/program_options.cc\n  src/core/reactor.cc\n  src/core/resource.cc\n  src/core/sharded.cc\n  src/core/scollectd.cc\n  src/core/scollectd-impl.hh\n  src/core/signal.cc\n  src/core/systemwide_memory_barrier.cc\n  src/core/smp.cc\n  src/core/sstring.cc\n  src/core/thread.cc\n  src/core/uname.cc\n  src/core/vla.hh\n  src/core/io_queue.cc\n  src/core/semaphore.cc\n  src/core/condition-variable.cc\n  src/http/api_docs.cc\n  src/http/common.cc\n  src/http/file_handler.cc\n  src/http/httpd.cc\n  src/http/json_path.cc\n  src/http/matcher.cc\n  src/http/mime_types.cc\n  src/http/reply.cc\n  src/http/routes.cc\n  src/http/transformers.cc\n  src/http/url.cc\n  src/http/client.cc\n  src/http/request.cc\n  src/http/retry_strategy.cc\n  src/json/formatter.cc\n  src/json/json_elements.cc\n  src/net/arp.cc\n  src/net/config.cc\n  src/net/dhcp.cc\n  src/net/dns.cc\n  src/net/dpdk.cc\n  src/net/ethernet.cc\n  src/net/inet_address.cc\n  src/net/ip.cc\n  src/net/ip_checksum.cc\n  src/net/native-stack-impl.hh\n  src/net/native-stack.cc\n  src/net/net.cc\n  src/net/packet.cc\n  src/net/posix-stack.cc\n  src/net/proxy.cc\n  src/net/socket_address.cc\n  src/net/stack.cc\n  src/net/tcp.cc\n  src/net/tls.cc\n  src/net/udp.cc\n  src/net/unix_address.cc\n  src/net/virtio.cc\n  src/rpc/lz4_compressor.cc\n  src/rpc/lz4_fragmented_compressor.cc\n  src/rpc/rpc.cc\n  src/util/alloc_failure_injector.cc\n  src/util/backtrace.cc\n  src/util/conversions.cc\n  src/util/exceptions.cc\n  src/util/file.cc\n  src/util/log.cc\n  src/util/process.cc\n  src/util/program-options.cc\n  src/util/read_first_line.cc\n  src/util/tmp_file.cc\n  src/util/short_streams.cc\n  src/websocket/parser.cc\n  src/websocket/common.cc\n  src/websocket/client.cc\n  src/websocket/server.cc\n  )\n\nif (Seastar_MODULE)\n  target_sources(seastar\n    PUBLIC\n      FILE_SET CXX_MODULES\n      TYPE CXX_MODULES\n      FILES\n        src/seastar.cppm\n  )\nendif ()\n\nadd_library (Seastar::seastar ALIAS seastar)\n\nadd_dependencies (seastar\n  seastar_http_chunk_parsers\n  seastar_http_request_parser\n  seastar_http_response_parser\n  seastar_proto_metrics2)\n\ntarget_include_directories (seastar\n  PUBLIC\n    $<INSTALL_INTERFACE:include>\n    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>\n    $<BUILD_INTERFACE:${Seastar_GEN_BINARY_DIR}/include>\n    $<BUILD_INTERFACE:${Seastar_GEN_BINARY_DIR}/src>\n  PRIVATE\n    ${CMAKE_CURRENT_SOURCE_DIR}/src)\n\nset (Seastar_PRIVATE_CXX_FLAGS\n  -fno-semantic-interposition\n  -Wall\n  -Werror\n  -Wimplicit-fallthrough\n  -Wdeprecated\n  -Wno-error=deprecated)\n\nif (CMAKE_CXX_COMPILER_ID STREQUAL \"GNU\")\n  include (CheckGcc107852)\n  if (NOT Cxx_Compiler_BZ107852_Free)\n    list (APPEND Seastar_PRIVATE_CXX_FLAGS\n      -Wno-error=stringop-overflow\n      -Wno-error=array-bounds)\n  endif ()\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS\n    -Wdeprecated-declarations\n    -Wno-error=deprecated-declarations)\nendif ()\n\nif (CMAKE_CXX_STANDARD GREATER_EQUAL 23)\n  include (CheckP2582R1)\n  if (Cxx_Compiler_IMPLEMENTS_P2581R1)\n    target_compile_definitions (seastar\n      PUBLIC SEASTAR_P2581R1)\n  endif ()\nendif ()\n\nif (BUILD_SHARED_LIBS)\n  # use initial-exec TLS, as it puts the TLS variables in the static TLS space\n  # instead of allocating them using malloc. otherwise intercepting mallocs and\n  # friends could lead to recursive call of malloc functions when a dlopen'ed\n  # shared object references a TLS variable and it in turn uses malloc. the\n  # downside of this workaround is that the static TLS space is used, and it is\n  # a global resource.\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS\n    $<$<IN_LIST:$<CONFIG>,RelWithDebInfo;Dev>:-ftls-model=initial-exec>)\nelse ()\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS -fvisibility=hidden)\nendif ()\n\nif (Seastar_COMPRESS_DEBUG)\n  # -gz doesn't imply -g, so it is safe to add it regardless of debug\n  # info being enabled.\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS -gz)\nendif ()\n\ntarget_link_libraries (seastar\n  PUBLIC\n    Boost::boost\n    Boost::program_options\n    c-ares::cares\n    fmt::fmt\n    lz4::lz4\n  PRIVATE\n    ${CMAKE_DL_LIBS}\n    GnuTLS::gnutls\n    StdAtomic::atomic\n    lksctp-tools::lksctp-tools\n    protobuf::libprotobuf\n    rt::rt\n    ucontext::ucontext\n    yaml-cpp::yaml-cpp\n    Threads::Threads)\nif (CMAKE_VERSION VERSION_GREATER_EQUAL 3.26)\n  target_link_libraries (seastar\n    PRIVATE\n      \"$<BUILD_LOCAL_INTERFACE:Valgrind::valgrind>\")\nelse ()\n  target_link_libraries (seastar\n    PRIVATE\n      \"$<BUILD_INTERFACE:Valgrind::valgrind>\")\nendif ()\n\nif (Seastar_DPDK)\n  target_link_libraries (seastar\n    PRIVATE\n      DPDK::dpdk)\nendif ()\n\ninclude (TriStateOption)\ntri_state_option (${Seastar_SANITIZE}\n  DEFAULT_BUILD_TYPES \"Debug\" \"Sanitize\" \"Fuzz\"\n  CONDITION condition)\nif (condition)\n  if (NOT Sanitizers_FOUND)\n    message (FATAL_ERROR \"Sanitizers not found!\")\n  endif ()\n  set (Seastar_Sanitizers_OPTIONS ${Sanitizers_COMPILE_OPTIONS})\n  target_link_libraries (seastar\n    PUBLIC\n      $<${condition}:Sanitizers::address>\n      $<${condition}:Sanitizers::undefined_behavior>\n      $<${condition}:Sanitizers::vptr>)\nendif ()\n\n# Link with fuzzer instrumentation for Fuzz builds\nif (CMAKE_BUILD_TYPE STREQUAL \"Fuzz\")\n  find_package (Sanitizers REQUIRED COMPONENTS fuzzer)\n  target_link_libraries (seastar\n    PUBLIC Sanitizers::fuzzer)\nendif ()\n\n# We only need valgrind to find uninitialized memory uses, so disable\n# the leak sanitizer.\n# To test with valgrind run \"ctest -T memcheck\"\nset( MEMORYCHECK_COMMAND_OPTIONS \"--error-exitcode=1 --leak-check=no --trace-children=yes\" )\ninclude (CTest)\n\n#\n# To disable -Werror, pass -Wno-error to Seastar_CXX_FLAGS.\n#\n\ntarget_compile_definitions(seastar\n  PUBLIC\n  SEASTAR_API_LEVEL=${Seastar_API_LEVEL}\n  $<$<BOOL:${BUILD_SHARED_LIBS}>:SEASTAR_BUILD_SHARED_LIBS>)\n\ntarget_compile_features(seastar\n  PUBLIC\n    cxx_std_${CMAKE_CXX_STANDARD})\n\ninclude (CheckCXXCompilerFlag)\ncheck_cxx_compiler_flag (\"-Wno-maybe-uninitialized -Werror\" MaybeUninitialized_FOUND)\nif (MaybeUninitialized_FOUND)\n  target_compile_options (seastar\n    PUBLIC\n      # With std::experimental::optional it is easy to hit\n      # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88897.  We disable\n      # -Wmaybe-uninitialized in here since otherwise we would have to\n      # disable it on many types used inside optional<>.\n      -Wno-maybe-uninitialized)\nendif ()\n\nif (Seastar_SSTRING)\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_SSTRING)\nendif ()\n\nif (Seastar_DEPRECATED_OSTREAM_FORMATTERS)\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_DEPRECATED_OSTREAM_FORMATTERS)\nendif ()\n\nif (LinuxMembarrier_FOUND)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAS_MEMBARRIER)\n\n  target_link_libraries (seastar\n    PRIVATE LinuxMembarrier::membarrier)\nendif ()\n\ntri_state_option (${Seastar_ALLOC_FAILURE_INJECTION}\n  DEFAULT_BUILD_TYPES \"Dev\"\n  CONDITION condition)\nif (condition)\n  target_compile_definitions (seastar\n    PUBLIC $<${condition}:SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION>)\nendif ()\n\nif (Seastar_TASK_BACKTRACE)\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_TASK_BACKTRACE)\nendif ()\n\nif (Seastar_DEBUG_ALLOCATIONS)\n  target_compile_definitions (seastar\n    PRIVATE SEASTAR_DEBUG_ALLOCATIONS)\nendif ()\n\nif (Sanitizers_FIBER_SUPPORT)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAVE_ASAN_FIBER_SUPPORT)\nendif ()\n\nif (Seastar_ALLOC_PAGE_SIZE)\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_OVERRIDE_ALLOCATOR_PAGE_SIZE=${Seastar_ALLOC_PAGE_SIZE})\nendif ()\n\nif (Seastar_LOGGER_COMPILE_TIME_FMT)\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_LOGGER_COMPILE_TIME_FMT)\nendif ()\n\ntarget_compile_definitions (seastar\n  PUBLIC SEASTAR_SCHEDULING_GROUPS_COUNT=${Seastar_SCHEDULING_GROUPS_COUNT})\n\nif (Seastar_CXX_FLAGS)\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS ${Seastar_CXX_FLAGS})\nendif ()\n\n# When the consumer specifies Seastar_SOURCE_SUBDIR, add -ffile-prefix-map\n# rules so that gdb can resolve Seastar sources correctly.\n#\n# Without these rules, a broader -ffile-prefix-map=<project_root>=. (typically\n# passed via Seastar_CXX_FLAGS) maps CMAKE_CURRENT_BINARY_DIR to a relative\n# build path (e.g. ./build/dev/seastar).  GDB then concatenates that with the\n# source-relative DW_AT_name and looks for a non-existent path like\n# ./build/dev/seastar/./seastar/src/core/reactor_backend.cc.\n#\n# Rule 1 maps the build dir itself to \".\", fixing DW_AT_comp_dir.\n# Rule 2 maps paths *under* the build dir (generated sources) to the subdir.\n# Both rules must appear AFTER Seastar_CXX_FLAGS because both GCC and clang\n# apply the last matching -ffile-prefix-map rule.\nif (Seastar_SOURCE_SUBDIR)\n  list (APPEND Seastar_PRIVATE_CXX_FLAGS\n    -ffile-prefix-map=${CMAKE_CURRENT_BINARY_DIR}=.\n    -ffile-prefix-map=${CMAKE_CURRENT_BINARY_DIR}/=${Seastar_SOURCE_SUBDIR})\nendif ()\n\n# When using split dwarf --gdb-index is effectively required since\n# otherwise gdb is just too slow. We also want to use split dwarf in\n# as many compilation units as possible.  So while these flags don't\n# have to be public, we don't expect anyone to want to build seastar\n# with them and some client code without.\nif (Seastar_SPLIT_DWARF)\n  set (Seastar_SPLIT_DWARF_FLAG \"-Wl,--gdb-index\")\n  target_link_libraries (seastar PUBLIC\n    $<$<NOT:$<CONFIG:Dev>>:${Seastar_SPLIT_DWARF_FLAG}>)\n  target_compile_options (seastar PUBLIC\n    $<$<NOT:$<CONFIG:Dev>>:-gsplit-dwarf>)\nendif ()\n\nif (Seastar_HEAP_PROFILING)\n    set_property (\n      SOURCE \"src/core/memory.cc\"\n      PROPERTY\n      COMPILE_DEFINITIONS SEASTAR_HEAPPROF)\n    set_property (\n      SOURCE \"src/core/reactor.cc\"\n      PROPERTY\n      COMPILE_DEFINITIONS SEASTAR_HEAPPROF)\nendif ()\n\nif (Seastar_DEFERRED_ACTION_REQUIRE_NOEXCEPT)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_DEFERRED_ACTION_REQUIRE_NOEXCEPT)\nendif ()\n\nif (Seastar_DPDK)\n  if (CMAKE_SYSTEM_PROCESSOR MATCHES \"ppc64\")\n    target_compile_options (seastar\n      PUBLIC\n        -mcpu=${Seastar_DPDK_MACHINE}\n        -mtune=${Seastar_DPDK_MACHINE})\n  else()\n    target_compile_options (seastar\n      PUBLIC\n        -march=${Seastar_DPDK_MACHINE})\n  endif ()\n  target_compile_definitions (seastar\n    PUBLIC SEASTAR_HAVE_DPDK)\nendif ()\n\nif (Seastar_HWLOC)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAVE_HWLOC)\n\n  target_link_libraries (seastar\n    PRIVATE hwloc::hwloc)\nendif ()\n\nset_option_if_package_is_found (Seastar_IO_URING LibUring)\nif (Seastar_IO_URING)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAVE_URING)\n  target_link_libraries (seastar\n    PRIVATE URING::uring)\nendif ()\n\nif (Seastar_LD_FLAGS)\n  target_link_options (seastar\n    PRIVATE ${Seastar_LD_FLAGS})\nendif ()\n\nif (SystemTap-SDT_FOUND)\n  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAVE_SYSTEMTAP_SDT)\n\n  target_link_libraries (seastar\n    PRIVATE SystemTap::SDT)\nendif ()\n\ncheck_cxx_compiler_flag (\"-Werror=unused-result\" ErrorUnused_FOUND)\nif (ErrorUnused_FOUND)\n  if (Seastar_UNUSED_RESULT_ERROR)\n    target_compile_options (seastar\n      PUBLIC -Werror=unused-result)\n  else()\n    target_compile_options (seastar\n      PUBLIC -Wno-error=unused-result)\n  endif ()\nendif ()\n\ncheck_cxx_compiler_flag (\"-Wno-error=#warnings\" ErrorWarnings_FOUND)\nif (ErrorWarnings_FOUND)\n  target_compile_options (seastar\n      PRIVATE \"-Wno-error=#warnings\")\nendif ()\n\nforeach (definition\n    SEASTAR_DEBUG\n    SEASTAR_DEFAULT_ALLOCATOR\n    SEASTAR_SHUFFLE_TASK_QUEUE)\n  target_compile_definitions (seastar\n    PUBLIC\n      $<$<IN_LIST:$<CONFIG>,Debug;Sanitize;Fuzz>:${definition}>)\nendforeach ()\n\ntri_state_option (${Seastar_DEBUG_SHARED_PTR}\n  DEFAULT_BUILD_TYPES \"Debug\" \"Sanitize\" \"Fuzz\"\n  CONDITION condition)\nif (condition)\n  target_compile_definitions (seastar\n    PUBLIC\n      $<${condition}:SEASTAR_DEBUG_SHARED_PTR>)\nendif ()\n\ntri_state_option (${Seastar_DEBUG_SHARED_PTR}\n  DEFAULT_BUILD_TYPES \"Debug\" \"Sanitize\" \"Fuzz\"\n  CONDITION condition)\nif (condition)\n  target_compile_definitions (seastar\n    PUBLIC\n      $<${condition}:SEASTAR_DEBUG_PROMISE>)\nendif ()\n\ninclude (CheckLibc)\n\ntri_state_option (${Seastar_STACK_GUARDS}\n  DEFAULT_BUILD_TYPES \"Debug\" \"Sanitize\" \"Dev\" \"Fuzz\"\n  CONDITION condition)\nif (condition)\n  # check for -fstack-clash-protection together with -Werror, because\n  # otherwise clang can soft-fail (return 0 but emit a warning) instead.\n  check_cxx_compiler_flag (\"-fstack-clash-protection -Werror\" StackClashProtection_FOUND)\n  if (StackClashProtection_FOUND)\n    target_compile_options (seastar\n      PUBLIC\n        $<${condition}:-fstack-clash-protection>)\n  endif ()\n  target_compile_definitions (seastar\n    PRIVATE\n      $<${condition}:SEASTAR_THREAD_STACK_GUARDS>)\nendif ()\n\ntarget_compile_definitions (seastar\n  PUBLIC\n    $<$<IN_LIST:$<CONFIG>,Dev;Debug>:SEASTAR_TYPE_ERASE_MORE>)\n\ntarget_compile_definitions (seastar\n  PRIVATE ${Seastar_PRIVATE_COMPILE_DEFINITIONS})\n\ntarget_compile_options (seastar\n  PRIVATE ${Seastar_PRIVATE_CXX_FLAGS})\n\nset_target_properties (seastar\n  PROPERTIES\n    CXX_STANDARD ${CMAKE_CXX_STANDARD}\n    CXX_EXTENSIONS ON)\n\nadd_library (seastar_private INTERFACE)\n\ntarget_compile_definitions (seastar_private\n  INTERFACE ${Seastar_PRIVATE_COMPILE_DEFINITIONS})\n\ntarget_compile_options (seastar_private\n  INTERFACE ${Seastar_PRIVATE_CXX_FLAGS})\n\ntarget_link_libraries (seastar_private\n  INTERFACE seastar)\n\n#\n# The testing library.\n#\n\nif (Seastar_INSTALL OR Seastar_TESTING)\n  add_library (seastar_testing\n    include/seastar/testing/entry_point.hh\n    include/seastar/testing/exchanger.hh\n    include/seastar/testing/random.hh\n    include/seastar/testing/seastar_test.hh\n    include/seastar/testing/test_case.hh\n    include/seastar/testing/test_runner.hh\n    include/seastar/testing/thread_test_case.hh\n    src/testing/entry_point.cc\n    src/testing/random.cc\n    src/testing/seastar_test.cc\n    src/testing/test_runner.cc)\n\n  add_library (Seastar::seastar_testing ALIAS seastar_testing)\n\n  target_compile_definitions (seastar_testing\n    PRIVATE ${Seastar_PRIVATE_COMPILE_DEFINITIONS})\n\n  target_compile_options (seastar_testing\n    PRIVATE ${Seastar_PRIVATE_CXX_FLAGS})\n\n  target_link_libraries (seastar_testing\n    PUBLIC\n      Boost::unit_test_framework\n      Boost::dynamic_linking\n      seastar)\n\n  add_library(seastar_perf_testing\n    src/testing/random.cc\n    include/seastar/testing/perf_tests.hh\n    tests/perf/perf_tests.cc\n    tests/perf/linux_perf_event.cc)\n  add_library (Seastar::seastar_perf_testing ALIAS seastar_perf_testing)\n  target_compile_definitions (seastar_perf_testing\n    PRIVATE ${Seastar_PRIVATE_COMPILE_DEFINITIONS})\n  target_compile_options (seastar_perf_testing\n    PRIVATE ${Seastar_PRIVATE_CXX_FLAGS})\n  target_link_libraries (seastar_perf_testing\n    PUBLIC\n    seastar)\n\nendif ()\n\nif (Seastar_MODULE)\n  if (POLICY CMP0155)\n    cmake_policy (SET CMP0155 NEW)\n  endif ()\n  include (CxxModulesRules)\nendif ()\n\n#\n# The tests themselves.\n#\n\nif (Seastar_TESTING)\n  enable_testing ()\n\n  if (Seastar_EXCLUDE_TESTS_FROM_ALL)\n    set (exclude EXCLUDE_FROM_ALL)\n  else ()\n    set (exclude \"\")\n  endif ()\n\n  add_subdirectory (tests ${exclude})\nendif ()\n\n#\n# Demonstrations.\n#\n\nif (Seastar_DEMOS)\n  if (Seastar_EXCLUDE_DEMOS_FROM_ALL)\n    set (exclude EXCLUDE_FROM_ALL)\n  else ()\n    set (exclude \"\")\n  endif ()\n\n  add_subdirectory (demos ${exclude})\nendif ()\n\n#\n# Documentation.\n#\n\nif (Seastar_DOCS)\n  add_subdirectory (doc)\nendif ()\n\n#\n# Applications.\n#\n\nif (Seastar_APPS)\n  if (Seastar_EXCLUDE_APPS_FROM_ALL)\n    set (exclude EXCLUDE_FROM_ALL)\n  else ()\n    set (exclude \"\")\n  endif ()\n\n  add_subdirectory (apps ${exclude})\nendif ()\n\nif (CMAKE_BUILD_TYPE STREQUAL \"Dev\")\n  include (CheckHeaders)\n  include (CheckIncludeStyle)\n  add_custom_target (checkheaders)\n  add_custom_target (check-include-style)\n  foreach (lib seastar seastar_testing seastar_perf_testing)\n    if (TARGET ${lib})\n      seastar_check_self_contained (checkheaders ${lib}\n        INCLUDE \"\\\\.hh$\"\n        # impl.hh headers are internal implementations of .hh, so they are not\n        # compilable. let's exclude them from the files to be checked.\n        EXCLUDE \"_impl.hh$|-impl.hh$\")\n      seastar_check_include_style (check-include-style ${lib})\n    endif ()\n  endforeach ()\nendif ()\n\n#\n# Installation and export.\n#\n\nif (Seastar_INSTALL)\n  #\n  # pkg-config generation.\n  #\n  # Note that unlike the CMake \"config module\", this description is not relocatable because\n  # some dependencies do not natively support pkg-config.\n  #\n\n  # Necessary here for pkg-config.\n  include (GNUInstallDirs)\n\n  # Set paths in pkg-config files for installation.\n  set (Seastar_PKG_CONFIG_PREFIX ${CMAKE_INSTALL_PREFIX})\n  set (Seastar_PKG_CONFIG_LIBDIR ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR})\n  set (Seastar_PKG_CONFIG_SEASTAR_INCLUDE_FLAGS \"-I${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_INCLUDEDIR}\")\n\n  get_property (_is_Multi_Config GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)\n  if (_is_Multi_Config)\n    # use different library names for each config\n    set (Seastar_PC \"_$<CONFIG>.pc\")\n  else ()\n    set (Seastar_PC \".pc\")\n  endif ()\n\n  if(CMAKE_CXX_EXTENSIONS)\n    set(Seastar_CXX_COMPILE_OPTION ${CMAKE_CXX${CMAKE_CXX_STANDARD}_EXTENSION_COMPILE_OPTION})\n  else()\n    set(Seastar_CXX_COMPILE_OPTION ${CMAKE_CXX${CMAKE_CXX_STANDARD}_STANDARD_COMPILE_OPTION})\n  endif()\n\n  configure_file (\n    ${CMAKE_CURRENT_SOURCE_DIR}/pkgconfig/seastar.pc.in\n    ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-install${Seastar_PC}.in\n    @ONLY)\n\n  configure_file (\n    ${CMAKE_CURRENT_SOURCE_DIR}/pkgconfig/seastar-testing.pc.in\n    ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-testing-install.pc.in\n    @ONLY)\n\n  # Set paths in pkg-config files for direct use in the build directory.\n  set (Seastar_PKG_CONFIG_PREFIX ${CMAKE_CURRENT_BINARY_DIR})\n  set (Seastar_PKG_CONFIG_LIBDIR ${CMAKE_CURRENT_BINARY_DIR})\n  set (Seastar_PKG_CONFIG_SEASTAR_INCLUDE_FLAGS \"-I${CMAKE_CURRENT_SOURCE_DIR}/include -I${CMAKE_CURRENT_BINARY_DIR}/gen/include\")\n\n  configure_file (\n    ${CMAKE_CURRENT_SOURCE_DIR}/pkgconfig/seastar.pc.in\n    ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar${Seastar_PC}.in\n    @ONLY)\n\n  configure_file (\n    ${CMAKE_CURRENT_SOURCE_DIR}/pkgconfig/seastar-testing.pc.in\n    ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-testing.pc.in\n    @ONLY)\n\n  file (GENERATE\n    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/seastar${Seastar_PC}\n    INPUT ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar${Seastar_PC}.in)\n\n  file (GENERATE\n    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/seastar-testing.pc\n    INPUT ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-testing.pc.in)\n\n  file (GENERATE\n    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/seastar-install${Seastar_PC}\n    INPUT ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-install${Seastar_PC}.in)\n\n  file (GENERATE\n    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/seastar-testing-install.pc\n    INPUT ${CMAKE_CURRENT_BINARY_DIR}/pkgconfig/seastar-testing-install.pc.in)\n\n  include (CMakePackageConfigHelpers)\n  set (install_cmakedir ${CMAKE_INSTALL_LIBDIR}/cmake/Seastar)\n\n  install (\n    DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/include/\n    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})\n\n  install (\n    DIRECTORY ${Seastar_GEN_BINARY_DIR}/include/\n    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})\n\n  install (\n    PROGRAMS ${CMAKE_CURRENT_SOURCE_DIR}/scripts/seastar-json2code.py\n    DESTINATION ${CMAKE_INSTALL_BINDIR})\n\n  set (seastar_install_targets\n    TARGETS\n      seastar\n      seastar_testing\n      seastar_perf_testing\n    EXPORT seastar-export\n    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}\n    ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})\n\n  if (Seastar_MODULE)\n    list (APPEND seastar_install_targets\n      FILE_SET CXX_MODULES\n        DESTINATION \"${CMAKE_INSTALL_INCLUDEDIR}/cxx/modules\"\n      CXX_MODULES_BMI\n        DESTINATION \"${CMAKE_INSTALL_LIBDIR}/cxx/bmi\")\n  endif ()\n\n  install (${seastar_install_targets})\n\n  install (\n    EXPORT seastar-export\n    FILE SeastarTargets.cmake\n    NAMESPACE Seastar::\n    DESTINATION ${install_cmakedir})\n\n  write_basic_package_version_file (\n    ${CMAKE_CURRENT_BINARY_DIR}/SeastarConfigVersion.cmake\n    VERSION ${PROJECT_VERSION}\n    COMPATIBILITY ExactVersion)\n\n  configure_package_config_file (\n    ${CMAKE_CURRENT_LIST_DIR}/cmake/SeastarConfig.cmake.in\n    ${CMAKE_CURRENT_BINARY_DIR}/SeastarConfig.cmake\n    INSTALL_DESTINATION ${install_cmakedir})\n\n  install (\n    FILES\n      ${CMAKE_CURRENT_BINARY_DIR}/SeastarConfig.cmake\n      ${CMAKE_CURRENT_BINARY_DIR}/SeastarConfigVersion.cmake\n    DESTINATION ${install_cmakedir})\n\n  install (\n    FILES\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindGnuTLS.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindLinuxMembarrier.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindSanitizers.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindStdAtomic.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findc-ares.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Finddpdk.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findhwloc.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findlksctp-tools.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findlz4.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findragel.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findrt.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Finducontext.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/Findyaml-cpp.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/SeastarDependencies.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindLibUring.cmake\n      ${CMAKE_CURRENT_SOURCE_DIR}/cmake/FindSystemTap-SDT.cmake\n    DESTINATION ${install_cmakedir})\n\n  install (\n    DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/cmake/code_tests\n    DESTINATION ${install_cmakedir})\n\n  install (\n    FILES ${CMAKE_CURRENT_BINARY_DIR}/seastar-install${Seastar_PC}\n    DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig\n    RENAME seastar${Seastar_PC})\n\n  install (\n    FILES ${CMAKE_CURRENT_BINARY_DIR}/seastar-testing-install.pc\n    DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig\n    RENAME seastar-testing.pc)\n\n  if (Seastar_MODULE)\n    install (\n      TARGETS seastar\n      CXX_MODULES_BMI\n        DESTINATION \"${CMAKE_INSTALL_LIBDIR}/cxx/bmi\"\n      FILE_SET CXX_MODULES\n        DESTINATION \"${CMAKE_INSTALL_INCLUDEDIR}/cxx/modules\")\n  endif ()\n\n  #\n  # Export targets from the build tree for the user package registry.\n  #\n\n  set (export_args\n    EXPORT seastar-export\n    FILE ${CMAKE_CURRENT_BINARY_DIR}/SeastarTargets.cmake\n    NAMESPACE Seastar::)\n\n  if (Seastar_MODULE)\n    list (APPEND export_args CXX_MODULES_DIRECTORY \"include/cxx/modules\")\n  endif ()\n\n  export (${export_args})\n\n  export (PACKAGE Seastar)\n\n  #\n  # Packaging.\n  #\n\n  set (CPACK_PACKAGE_VERSION_MAJOR ${PROJECT_VERSION_MAJOR})\n  set (CPACK_PACKAGE_VERSION_MINOR ${PROJECT_VERSION_MINOR})\n  set (CPACK_PACKAGE_VERSION_PATCH ${PROJECT_VERSION_PATCH})\n\n  include (CPack)\nendif ()\n"
  },
  {
    "path": "CONTRIBUTING.md",
    "content": "# Contributing Code to Seastar\n\nThere are two ways to contribute code to Seastar:\n* send your changes as [patches](https://github.com/scylladb/scylla/wiki/Formatting-and-sending-patches) to the [mailing list](https://groups.google.com/forum/#!forum/seastar-dev).\n* alternatively, open a [github pull request](https://github.com/scylladb/seastar/pulls).\n\n# Asking questions or requesting help\n\nUse the [Seastar mailing list](https://groups.google.com/forum/#!forum/seastar-dev) for general questions and help.\n\n# Reporting an issue\n\nPlease use the [Issue Tracker](https://github.com/scylladb/seastar/issues/) to report issues. Supply as much information about your environment as possible, especially for performance problems.\n\n"
  },
  {
    "path": "HACKING.md",
    "content": "# Developing and using Seastar\n\n## Configuring the project\n\nThere are multiple ways to configure Seastar and its dependencies.\n\n### Use system-packages for most dependencies\n\nSee the instructions in [README.md](./README.md).\n\n### Download and install all external dependencies in a project-specific location\n\n- First pull the git submodules using `git submodule update --init --recursive`\n\n- Use `cmake-cooking` to prepare a development environment with all dependencies.  This allows for reproducible development environments, but means that approximately 3 GiB of dependencies get installed to `build/_cooking_`:\n\n```\n./cooking.sh\n```\n\n- The same as above, and enable DPDK support:\n\n```\n./cooking.sh -- -DSeastar_DPDK=ON\n```\n\n- Use system packages for all dependencies except `dpdk`, which is provided by `cmake-cooking` (and not yet widely available via system package-managers):\n\n```\n./cooking.sh -i dpdk\n```\n\n- Use `cmake-cooking` for all dependencies except for Boost:\n\n```\n./cooking.sh -e Boost\n```\n\n- The same, but compile in \"release\" mode:\n\n```\n./cooking.sh -e Boost -t Release\n```\n\n## Using an IDE with CMake support\n\nIf you use `configure.py` or `cooking.sh` to to configure Seastar, then the easiest way to use an IDE (such as Qt Creator, or CLion) for development is to instruct the IDE, when it invokes CMake, to include the following option:\n\n```\n-DCMAKE_PREFIX_PATH=${source_dir}/build/_cooking/installed\n```\n\nwhere `${source_dir}` is the root of the Seastar source tree on your file-system.\n\nThis will allow the IDE to also index Seastar's dependencies.\n\n## Building the project\n\n```\ncd $my_build_dir\nninja\n```\n\nIf you used `configure.py` to configure Seastar, then the build directory will be `build/$mode`. For example, `build/release`.\n\nIf you use `cooking.sh`, then the build directory will just be `build`.\n\n## Running tests\n\nMake sure you are in the \"build\" directory.\n\n- Run unit tests:\n\n```\nninja test_unit\n```\n\n- Run all tests:\n\n```\nninja test\n```\n\n- Build and run a specific test:\n\n```\nninja test_unit_thread_run\n```\n\n\n## Building documentation\n\nMake sure you are in the \"build\" directory.\n\n- Build all documentation:\n\n```\nninja docs\n```\n\n- Build the tutorial in HTML form:\n\n```\nninja doc_tutorial_html\n```\n\n- Build the tutorial in HTML form (one file per chapter):\n\n```\nninja doc_tutorial_html_split\n```\n\n- Build the Doxygen documentation:\n\n```\nninja doc_api\n```\n\n## Installing the project\n\nChoose the install path:\n\nWith `configure.py`:\n\n```\n./configure.py --mode=release --prefix=/my/install/path\n```\n\nWith `cooking.sh`:\n\n```\n./cooking.sh -- -DCMAKE_INSTALL_PREFIX=/my/install/path\n```\n\n```\nninja -C build install\n```\n\n## Using Seastar in an application\n\n### CMake\n\nOnce Seastar has been installed, it is sufficient to add a dependency on Seastar with\n\n```\nfind_package (Seastar ${VERSION} REQUIRED)\n\nadd_executable (my_program\n  my_program.cc)\n\ntarget_link_libraries (my_program\n  PRIVATE Seastar::seastar)\n```\n\nwhere `VERSION` is the desired version.\n\nIf you'd like to use `cmake-cooking` to set up a development environment which includes Seastar and its dependencies (a \"recipe\"), you can include Seastar as follows:\n\n```\ncooking_ingredient (Seastar\n  COOKING_RECIPE <DEFAULT>\n  COOKING_CMAKE_ARGS\n    -DSeastar_APPS=OFF\n    -DSeastar_DEMOS=OFF\n    -DSeastar_DOCS=OFF\n    -DSeastar_TESTING=OFF\n  EXTERNAL_PROJECT_ARGS\n    SOURCE_DIR ${MY_SEASTAR_SOURCE_DIR})\n```\n\n### pkg-config\n\nSeastar includes a `seastar.pc` file. It can be used from both the\ninstall and build directories.\n\nCompiling a single file:\n```\ng++ foo.cc -o foo $(pkg-config --libs --cflags --static /path/to/seastar.pc)\n```\n\nCompiling multiple files:\n```\n# Compiling sources into object files\ng++ -c $(pkg-config --cflags /path/to/seastar.pc) foo.cc -o foo.o\ng++ -c $(pkg-config --cflags /path/to/seastar.pc) bar.cc -o bar.o\n\n# Linking object files into an executable\ng++ -o foo_bar foo.o bar.o $(pkg-config --libs --static /path/to/seastar.pc)\n```\n\nThe `--static` flag is needed to include transitive (private) dependencies of `libseastar.a`.\n\n## Development Tools\n\n### Pre-commit hooks\n\nThis project uses [pre-commit](https://pre-commit.com/) to enforce some basic checks. These\nchecks run in CI, but you can also run them locally as a pre-commit hook as follows.\n\n#### Installation\n\n[Install pre-commit](https://pre-commit.com/#install), following those instructions or\nperhaps using `uv` to avoid polluting your global environment:\n\n```\nuv tool install pre-commit\n```\n\n#### Setup\n\nInstall the git hooks:\n\n```\npre-commit install\n```\n\nThis will run the configured hooks automatically on every commit.\n\n#### Manual execution\n\nRun hooks on all files:\n\n```\npre-commit run --all-files\n```\n\nRun hooks on staged files only:\n\n```\npre-commit run\n```\n"
  },
  {
    "path": "LICENSE",
    "content": "\n                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n"
  },
  {
    "path": "NOTICE",
    "content": "Seastar Framework\nCopyright 2015 Cloudius Systems\n\nThis works contains software from the OSv project (http://osv.io), licensed\nunder the BSD license.\n\nThis work contains software from the DPDK project (http://dpdk.org), licensed\nunder the BSD license.  The software is under the dpdk/ directory.\n\nThis work contains software from the Android Open Source Project,\nlicensed under the Apache2 license.\nThe software is in the include/seastar/util/sampler.hh file.\n"
  },
  {
    "path": "README-DPDK.md",
    "content": "Seastar and DPDK\n================\n\nSeastar uses the Data Plane Development Kit to drive NIC hardware directly.  This\nprovides an enormous performance boost.\n\nTo enable DPDK, specify `--enable-dpdk` to `./configure.py`, and `--dpdk-pmd` as a\nrun-time parameter.  This will use the DPDK package provided as a git submodule with the\nseastar sources.\n\nPlease note, if `--enable-dpdk` is used to build DPDK on an aarch64 machine, you need to\nspecify [target architecture](https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html) with optional\n[feature modifiers](https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html#aarch64-feature-modifiers)\nwith the `--cflags` option as well, like:\n```console\n$ ./configure.py --mode debug --enable-dpdk --cflags='-march=armv8-a+crc+crypto'\n```\n\nTo use your own self-compiled DPDK package, follow this procedure:\n\n1. Setup host to compile DPDK:\n   - Ubuntu\n     `sudo apt-get install -y build-essential linux-image-extra-$(uname -r)`\n2. Prepare a DPDK SDK:\n   - Download the latest DPDK release: `wget https://fast.dpdk.org/rel/dpdk-23.07.tar.xz`\n   - Untar it.\n   - Follow the [Quick Start Guide](https://core.dpdk.org/doc/quick-start/)\n   - Pass `-Dmbuf_refcnt_atomic=false` to meson.\n3. Modify the CMake cache (`CMakeCache.txt`) to inform CMake of the location of the installed DPDK SDK.\n"
  },
  {
    "path": "README.md",
    "content": "Seastar\n=======\n\n[![Test](https://github.com/scylladb/seastar/actions/workflows/tests.yaml/badge.svg)](https://github.com/scylladb/seastar/actions/workflows/tests.yaml)\n[![Version](https://img.shields.io/github/tag/scylladb/seastar.svg?label=version&colorB=green)](https://github.com/scylladb/seastar/releases)\n[![License: Apache2](https://img.shields.io/github/license/scylladb/seastar.svg)](https://github.com/scylladb/seastar/blob/master/LICENSE)\n[![n00b issues](https://img.shields.io/github/issues/scylladb/seastar/n00b.svg?colorB=green)](https://github.com/scylladb/seastar/labels/n00b)\n\nIntroduction\n------------\n\nSeaStar is an event-driven framework allowing you to write non-blocking,\nasynchronous code in a relatively straightforward manner (once understood).\nIt is based on [futures](http://en.wikipedia.org/wiki/Futures_and_promises).\n\nBuilding Seastar\n--------------------\n\nFor more details and alternative work-flows, read [HACKING.md](./HACKING.md).\n\nAssuming that you would like to use system packages (RPMs or DEBs) for Seastar's dependencies, first install them:\n\n```\n$ sudo ./install-dependencies.sh\n```\n\nthen configure (in \"release\" mode):\n\n```\n$ ./configure.py --mode=release\n```\nthen compile:\n\n```\n$ ninja -C build/release\n```\n\nIn case there are compilation issues, especially like ```g++: internal compiler error: Killed (program cc1plus)```\ntry giving more memory to gcc, either by limiting the amount of threads ( -j1 ) and/or allowing at least 4g ram to your\nmachine.\n\nIf you're missing a dependency of Seastar, then it is possible to have the configuration process fetch a version of the dependency locally for development.\n\nFor example, to fetch `fmt` locally, configure Seastar like this:\n\n```\n$ ./configure.py --mode=dev --cook fmt\n```\n\n`--cook` can be repeated many times for selecting multiple dependencies.\n\n\nBuild modes\n----------------------------------------------------------------------------\n\nThe configure.py script is a wrapper around cmake. The --mode argument\nmaps to CMAKE_BUILD_TYPE, and supports the following modes\n\n|          | CMake mode          | Debug info | Optimi&shy;zations | Sanitizers   | Allocator | Checks   | Use for                                |\n| -------- | ------------------- | ---------- | ------------------ |------------- | --------- | -------- | -------------------------------------- |\n| debug    | `Debug`             | Yes        | `-O0`              | ASAN, UBSAN  | System    | All      | gdb                                    |\n| release  | `RelWithDebInfo`    | Yes        | `-O3`              | None         | Seastar   | Asserts  | production                             |\n| dev      | `Dev` (Custom)      | No         | `-O1`              | None         | Seastar   | Asserts  | build and test cycle                   |\n| sanitize | `Sanitize` (Custom) | Yes        | `-Os`              | ASAN, UBSAN  | System    | All      | second level of tests, track down bugs |\n\nNote that seastar is more sensitive to allocators and optimizations than\nusual. A quick rule of the thumb of the relative performances is that\nrelease is 2 times faster than dev, 150 times faster than sanitize and\n300 times faster than debug.\n\nUsing Seastar from its build directory (without installation)\n----------------------------------------------------------------------------\n\nIt's possible to consume Seastar directly from its build directory with CMake or `pkg-config`.\n\nWe'll assume that the Seastar repository is located in a directory at `$seastar_dir`.\n\n\nVia `pkg-config`:\n\n```\n$ g++ my_app.cc $(pkg-config --libs --cflags --static $seastar_dir/build/release/seastar.pc) -o my_app\n```\n\nand with CMake using the `Seastar` package:\n\n\n`CMakeLists.txt` for `my_app`:\n\n```\nset (CMAKE_CXX_STANDARD 23)\n\nfind_package (Seastar REQUIRED)\n\nadd_executable (my_app\n  my_app.cc)\n\ntarget_link_libraries (my_app\n  Seastar::seastar)\n```\n\n```\n$ mkdir $my_app_dir/build\n$ cd $my_app_dir/build\n$ cmake -DCMAKE_PREFIX_PATH=\"$seastar_dir/build/release;$seastar_dir/build/release/_cooking/installed\" -DCMAKE_MODULE_PATH=$seastar_dir/cmake $my_app_dir\n```\n\nThe `CMAKE_PREFIX_PATH` values ensure that CMake can locate Seastar and its compiled submodules. The `CMAKE_MODULE_PATH` value ensures that CMake can uses Seastar's CMake scripts for locating its dependencies.\n\nUsing an installed Seastar\n--------------------------------\n\nYou can also consume Seastar after it has been installed to the file-system.\n\n**Important:**\n\n- Seastar works with a customized version of DPDK, so by default builds and installs the DPDK submodule to `$build_dir/_cooking/installed`\n\nFirst, configure the installation path:\n\n```\n$ ./configure.py --mode=release --prefix=/usr/local\n```\n\nthen run the `install` target:\n\n```\n$ ninja -C build/release install\n```\n\nthen consume it from `pkg-config`:\n\n```\n$ g++ my_app.cc $(pkg-config --libs --cflags --static seastar) -o my_app\n```\n\nor consume it with the same `CMakeLists.txt` as before but with a simpler CMake invocation:\n\n```\n$ cmake ..\n```\n\n(If Seastar has not been installed to a \"standard\" location like `/usr` or `/usr/local`, then you can invoke CMake with `-DCMAKE_PREFIX_PATH=$my_install_root`.)\n\nThere are also instructions for building on any host that supports [Docker](doc/building-docker.md).\n\nUse of the [DPDK](http://dpdk.org) is [optional](doc/building-dpdk.md).\n\n#### Seastar's C++ standard: C++20 or C++23\n\nSeastar supports both C++20, and C++23. The build defaults to the latest\nstandard supported by your compiler, but can be explicitly selected with\nthe `--c++-standard` configure option, e.g., `--c++-standard=20`,\nor if using CMake directly, by setting on the `CMAKE_CXX_STANDARD` CMake\nvariable.\n\nSee the [compatibity statement](doc/compatibility.md) for more information.\n\nGetting started\n---------------\n\nThere is a [mini tutorial](doc/mini-tutorial.md) and a [more comprehensive one](doc/tutorial.md).\n\nThe documentation is available on the [web](http://docs.seastar.io/master/index.html).\n\n\nResources\n---------\n\n* Seasatar Development Mailing List: Discuss challenges, propose improvements with\n  sending code contributions (patches), and get help from experienced developers.\n  Subscribe or browse archives: [here](https://groups.google.com/forum/#!forum/seastar-dev)\n  (or email seastar-dev@googlegroups.com).\n* GitHub Discussions: For more casual conversations and quick questions, consider\n  using the Seastar project's [discussions on Github](https://github.com/scylladb/seastar/discussions).\n* Issue Tracker: File bug reports on the project's [issue tracker](https://github.com/scylladb/seastar/issues).\n\nLearn more about Seastar on the main [project website](http://seastar.io).\n\nThe Native TCP/IP Stack\n-----------------------\n\nSeastar comes with its own [userspace TCP/IP stack](doc/native-stack.md) for better performance.\n\nRecommended hardware configuration for SeaStar\n----------------------------------------------\n\n* CPUs - As much as you need. SeaStar is highly friendly for multi-core and NUMA\n* NICs - As fast as possible, we recommend 10G or 40G cards. It's possible to use\n       1G too but you may be limited by their capacity.\n       In addition, the more hardware queue per cpu the better for SeaStar.\n       Otherwise we have to emulate that in software.\n* Disks - Fast SSDs with high number of IOPS.\n* Client machines - Usually a single client machine can't load our servers.\n       Both memaslap (memcached) and WRK (httpd) cannot over load their matching\n       server counter parts. We recommend running the client on different machine\n       than the servers and use several of them.\n\nProjects using Seastar\n----------------------------------------------\n\n* [cpv-cql-driver](https://github.com/cpv-project/cpv-cql-driver): C++ driver for Cassandra/Scylla based on seastar framework\n* [cpv-framework](https://github.com/cpv-project/cpv-framework): A web framework written in c++ based on seastar framework\n* [redpanda](https://vectorized.io/): A Kafka replacement for mission critical systems\n* [Scylla](https://github.com/scylladb/scylla): A fast and reliable NoSQL data store compatible with Cassandra and DynamoDB\n* [smf](https://github.com/smfrpc/smf): The fastest RPC in the West\n* [Ceph - Crimson](https://github.com/ceph/ceph): Next-generation OSD (Object Storage Daemon) implementation based on the Seastar framework\n"
  },
  {
    "path": "apps/CMakeLists.txt",
    "content": "# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\n# Logical target for all applications.\nadd_custom_target (apps)\n\nmacro (seastar_add_app name)\n  set (args ${ARGN})\n\n  cmake_parse_arguments (\n    parsed_args\n    \"\"\n    \"\"\n    \"SOURCES\"\n    ${args})\n\n  set (target app_${name})\n  add_executable (${target} ${parsed_args_SOURCES})\n\n  target_include_directories (${target}\n    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})\n\n  target_link_libraries (${target}\n    PRIVATE seastar_private)\n\n  set_target_properties (${target}\n    PROPERTIES\n      OUTPUT_NAME ${name})\n\n  add_dependencies (apps ${target})\nendmacro ()\n\nadd_subdirectory (httpd)\nadd_subdirectory (io_tester)\nadd_subdirectory (rpc_tester)\nadd_subdirectory (iotune)\nif (${Seastar_API_LEVEL} GREATER_EQUAL 9)\n  add_subdirectory (memcached)\nendif ()\nadd_subdirectory (seawreck)\n"
  },
  {
    "path": "apps/httpd/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nseastar_generate_swagger (\n  TARGET app_httpd_swagger\n  VAR app_httpd_swagger_files\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/demo.json\n  OUT_DIR ${CMAKE_CURRENT_BINARY_DIR})\n\nseastar_add_app (httpd\n  SOURCES\n    ${app_httpd_swagger_files}\n    main.cc)\n\ntarget_include_directories (app_httpd\n  PRIVATE ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_dependencies (app_httpd app_httpd_swagger)\n"
  },
  {
    "path": "apps/httpd/demo.json",
    "content": "{\n    \"apiVersion\": \"0.0.1\",\n    \"swaggerVersion\": \"1.2\",\n    \"basePath\": \"{{Protocol}}://{{Host}}\",\n    \"resourcePath\": \"/hello\",\n    \"produces\": [\n        \"application/json\"\n    ],\n    \"apis\": [\n        {\n            \"path\": \"/hello/world/{var1}/{var2}\",\n            \"operations\": [\n                {\n                    \"method\": \"GET\",\n                    \"summary\": \"Returns the number of seconds since the system was booted\",\n                    \"type\": \"long\",\n                    \"nickname\": \"hello_world\",\n                    \"produces\": [\n                        \"application/json\"\n                    ],\n                    \"parameters\": [\n                    {\n                     \"name\":\"var2\",\n                     \"description\":\"Full path of file or directory\",\n                     \"required\":true,\n                     \"allowMultiple\":true,\n                     \"type\":\"string\",\n                     \"paramType\":\"path\"\n                  },\n                  {\n                     \"name\":\"var1\",\n                     \"description\":\"Full path of file or directory\",\n                     \"required\":true,\n                     \"allowMultiple\":false,\n                     \"type\":\"string\",\n                     \"paramType\":\"path\"\n                  },\n                    {\n                     \"name\":\"query_enum\",\n                     \"description\":\"The operation to perform\",\n                     \"required\":true,\n                     \"allowMultiple\":false,\n                     \"type\":\"string\",\n                     \"paramType\":\"query\",\n                     \"enum\":[\"VAL1\", \"VAL2\", \"VAL3\"]\n                  \t}\n                    ]\n                }\n            ]\n        }\n    ],\n    \"models\" : {\n        \"my_object\": {\n           \"id\": \"my_object\",\n           \"description\": \"Demonstrate an object\",\n               \"properties\": {\n                \"var1\": {\n                    \"type\": \"string\",\n                    \"description\": \"The first parameter in the path\"\n                },\n                \"var2\": {\n                    \"type\": \"string\",\n                    \"description\": \"The second parameter in the path\"\n                },\n                \"enum_var\" : {\n                    \"type\": \"string\",\n                    \"description\": \"Demonstrate an enum returned, note this is not the same enum type of the request\",\n                    \"enum\":[\"VAL1\", \"VAL2\", \"VAL3\"]\n                }\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "apps/httpd/main.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#include <memory>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/handlers.hh>\n#include <seastar/http/function_handlers.hh>\n#include <seastar/http/file_handler.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include \"demo.json.hh\"\n#include <seastar/http/api_docs.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/core/print.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/net/api.hh>\n#include \"../lib/stop_signal.hh\"\n\nnamespace bpo = boost::program_options;\n\nusing namespace seastar;\nusing namespace httpd;\n\nclass handl : public httpd::handler_base {\npublic:\n    virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n        rep->_content = \"hello\";\n        rep->done(\"html\");\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n};\n\nvoid set_routes(routes& r) {\n    function_handler* h1 = new function_handler([](const_req req) {\n        return \"hello\";\n    });\n    function_handler* h2 = new function_handler([](std::unique_ptr<http::request> req) {\n        return make_ready_future<json::json_return_type>(\"json-future\");\n    });\n    r.add(operation_type::GET, url(\"/\"), h1);\n    r.add(operation_type::GET, url(\"/jf\"), h2);\n    r.add(operation_type::GET, url(\"/file\").remainder(\"path\"),\n            new directory_handler(\"/\"));\n    r.add(operation_type::GET, url(\"/shard\"), new function_handler([] (const_req req) {\n        return seastar::sstring(fmt::format(\"{}\", seastar::this_shard_id()));\n    }));\n    demo_json::hello_world.set(r, [] (const_req req) {\n        demo_json::my_object obj;\n        obj.var1 = req.param.at(\"var1\");\n        obj.var2 = req.param.at(\"var2\");\n        demo_json::ns_hello_world::query_enum v = demo_json::ns_hello_world::str2query_enum(req.get_query_param(\"query_enum\"));\n        // This demonstrate enum conversion\n        obj.enum_var = v;\n        return obj;\n    });\n}\n\nint main(int ac, char** av) {\n    app_template app;\n\n    app.add_options()(\"port\", bpo::value<uint16_t>()->default_value(10000), \"HTTP Server port\");\n    app.add_options()(\"load-balancing-algorithm\",\n            bpo::value<std::string>()->default_value(\"connection_distribution\"),\n            \"Load balancing algorithm: connection_distribution, port, fixed\");\n    app.add_options()(\"prometheus_port\", bpo::value<uint16_t>()->default_value(9180), \"Prometheus port. Set to zero in order to disable.\");\n    app.add_options()(\"prometheus_address\", bpo::value<sstring>()->default_value(\"0.0.0.0\"), \"Prometheus address\");\n    app.add_options()(\"prometheus_prefix\", bpo::value<sstring>()->default_value(\"seastar_httpd\"), \"Prometheus metrics prefix\");\n\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            seastar_apps_lib::stop_signal stop_signal;\n            auto&& config = app.configuration();\n            httpd::http_server_control prometheus_server;\n            bool prometheus_started = false;\n\n            auto stop_prometheus = defer([&] () noexcept {\n                if (prometheus_started) {\n                    std::cout << \"Stoppping Prometheus server\" << std::endl;  // This can throw, but won't.\n                    prometheus_server.stop().get();\n                }\n            });\n\n            uint16_t pport = config[\"prometheus_port\"].as<uint16_t>();\n            if (pport) {\n                prometheus::config pctx;\n                net::inet_address prom_addr(config[\"prometheus_address\"].as<sstring>());\n\n                pctx.prefix = config[\"prometheus_prefix\"].as<sstring>();\n\n                std::cout << \"starting prometheus API server\" << std::endl;\n                prometheus_server.start(\"prometheus\").get();\n\n                prometheus::start(prometheus_server, pctx).get();\n\n                prometheus_started = true;\n\n                prometheus_server.listen(socket_address{prom_addr, pport}).handle_exception([prom_addr, pport] (auto ep) {\n                    std::cerr << seastar::format(\"Could not start Prometheus API server on {}:{}: {}\\n\", prom_addr, pport, ep);\n                    return make_exception_future<>(ep);\n                }).get();\n\n            }\n\n            uint16_t port = config[\"port\"].as<uint16_t>();\n            auto server = std::make_unique<http_server_control>();\n            auto rb = make_shared<api_registry_builder>(\"apps/httpd/\");\n            server->start().get();\n\n            auto stop_server = defer([&] () noexcept {\n                std::cout << \"Stoppping HTTP server\" << std::endl; // This can throw, but won't.\n                server->stop().get();\n            });\n\n            server->set_routes(set_routes).get();\n            server->set_routes([rb](routes& r){rb->set_api_doc(r);}).get();\n            server->set_routes([rb](routes& r) {rb->register_function(r, \"demo\", \"hello world application\");}).get();\n\n            auto lba_str = config[\"load-balancing-algorithm\"].as<std::string>();\n            server_socket::load_balancing_algorithm lba;\n            if (lba_str == \"connection_distribution\") {\n                lba = server_socket::load_balancing_algorithm::connection_distribution;\n            } else if (lba_str == \"port\") {\n                lba = server_socket::load_balancing_algorithm::port;\n            } else if (lba_str == \"fixed\") {\n                lba = server_socket::load_balancing_algorithm::fixed;\n            } else {\n                throw std::runtime_error(\"Invalid load balancing algorithm: \" + lba_str);\n            }\n\n            listen_options lo;\n            lo.lba = lba;\n            server->listen(socket_address{net::inet_address{}, port}, lo).get();\n\n            std::cout << \"Seastar HTTP server listening on port \" << port << \" ...\\n\";\n\n            stop_signal.wait().get();\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "apps/io_tester/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nseastar_add_app (io_tester\n  SOURCES io_tester.cc)\n\nseastar_add_app (ioinfo\n  SOURCES ioinfo.cc)\n\ntarget_link_libraries (app_io_tester\n  PRIVATE yaml-cpp::yaml-cpp)\n\ntarget_link_libraries (app_ioinfo\n  PRIVATE yaml-cpp::yaml-cpp)\n"
  },
  {
    "path": "apps/io_tester/conf.yaml",
    "content": "- name: big_writes\n  shards: all\n  type: overwrite\n  shard_info:\n    parallelism: 10\n    reqsize: 256kB\n    shares: 10\n    think_time: 0\n\n- name: latency_reads\n  shards: [0]\n  type: randread\n  data_size: 1GB\n  shard_info:\n    parallelism: 1\n    reqsize: 512\n    shares: 100\n    think_time: 1000us\n\n- name: cpu_hog\n  shards: [0]\n  type: cpu\n  shard_info:\n    parallelism: 1\n    execution_time: 90us\n    think_time: 10us\n\n- name: unlinking\n  shards: all\n  type: unlink\n  data_size: 2GB\n  files_count: 5000\n  shard_info:\n    parallelism: 10\n    think_time: 10us\n"
  },
  {
    "path": "apps/io_tester/io_tester.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/later.hh>\n#include <chrono>\n#include <optional>\n#include <ranges>\n#include <utility>\n#include <unordered_set>\n#include <vector>\n#include <boost/range/irange.hpp>\n#include <boost/algorithm/string.hpp>\n\n#pragma GCC diagnostic push\n// see https://github.com/boostorg/accumulators/pull/54\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n#include <boost/accumulators/accumulators.hpp>\n#include <boost/accumulators/statistics/stats.hpp>\n#include <boost/accumulators/statistics/max.hpp>\n#include <boost/accumulators/statistics/mean.hpp>\n#include <boost/accumulators/statistics/p_square_quantile.hpp>\n#include <boost/accumulators/statistics/extended_p_square.hpp>\n#include <boost/accumulators/statistics/extended_p_square_quantile.hpp>\n#pragma GCC diagnostic pop\n#include <boost/range/adaptor/filtered.hpp>\n#include <boost/range/adaptor/map.hpp>\n#include <boost/array.hpp>\n#include <iomanip>\n#include <random>\n#include <yaml-cpp/yaml.h>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\nusing namespace boost::accumulators;\n\nstatic constexpr uint64_t extent_size_hint_alignment{1u << 20}; // 1MB\n\nstatic auto random_seed = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::system_clock::now().time_since_epoch()).count();\nstatic thread_local std::default_random_engine random_generator(random_seed);\n\nclass context;\nenum class request_type { seqread, overwrite, randread, randwrite, append, cpu, unlink };\n\nnamespace std {\n\ntemplate <>\nstruct hash<request_type> {\n    size_t operator() (const request_type& type) const {\n        return static_cast<size_t>(type);\n    }\n};\n\n}\n\nauto allocate_and_fill_buffer(size_t buffer_size) {\n    constexpr size_t alignment{4096u};\n    auto buffer = allocate_aligned_buffer<char>(buffer_size, alignment);\n\n    std::uniform_int_distribution<int> fill('@', '~');\n    memset(buffer.get(), fill(random_generator), buffer_size);\n\n    return buffer;\n}\n\nfuture<file> create_and_fill_file(sstring name, uint64_t fsize, open_flags flags, file_open_options options, bool fill, bool fallocate) {\n    auto f = co_await open_file_dma(name, flags, options);\n    uint64_t pre_truncate_size = co_await f.size();\n    co_await f.truncate(fsize);\n    if (fallocate) {\n        co_await f.allocate(0, fsize);\n    }\n    if (!fill || (pre_truncate_size >= fsize)) {\n        co_return f;\n    }\n\n    const uint64_t buffer_size{256ul << 10};\n    const uint64_t additional_iteration = (fsize % buffer_size == 0) ? 0 : 1;\n    const uint64_t buffers_count{static_cast<uint64_t>(fsize / buffer_size) + additional_iteration};\n    auto buffers_range = std::views::iota(UINT64_C(0), buffers_count);\n    co_await max_concurrent_for_each(buffers_range.begin(), buffers_range.end(), 64, [f, buffer_size] (auto buffer_id) mutable {\n            auto source_buffer = allocate_and_fill_buffer(buffer_size);\n            auto write_position = buffer_id * buffer_size;\n            return do_with(std::move(source_buffer), [f, write_position, buffer_size] (const auto& buffer) mutable {\n                return f.dma_write(write_position, buffer.get(), buffer_size).discard_result();\n            });\n    });\n    co_await f.flush();\n    co_return f;\n}\n\nfuture<> busyloop_sleep(std::chrono::steady_clock::time_point until, std::chrono::steady_clock::time_point now) {\n    return do_until([until] {\n        return std::chrono::steady_clock::now() >= until;\n    }, [] {\n        return yield();\n    });\n}\n\ntemplate <typename Clock>\nfuture<> timer_sleep(std::chrono::steady_clock::time_point until, std::chrono::steady_clock::time_point now) {\n    return seastar::sleep<Clock>(std::chrono::duration_cast<std::chrono::microseconds>(until - now));\n}\n\nusing sleep_fn = std::function<future<>(std::chrono::steady_clock::time_point until, std::chrono::steady_clock::time_point now)>;\n\nclass pause_distribution {\npublic:\n\n    virtual std::chrono::duration<double> get() = 0;\n\n    template <typename Dur>\n    Dur get_as() {\n        return std::chrono::duration_cast<Dur>(get());\n    }\n\n    virtual ~pause_distribution() {}\n};\n\nusing pause_fn = std::function<std::unique_ptr<pause_distribution>(std::chrono::duration<double>)>;\n\nclass uniform_process : public pause_distribution {\n    std::chrono::duration<double> _pause;\n\npublic:\n    uniform_process(std::chrono::duration<double> period)\n            : _pause(period)\n    {\n    }\n\n    std::chrono::duration<double> get() override {\n        return _pause;\n    }\n};\n\nstd::unique_ptr<pause_distribution> make_uniform_pause(std::chrono::duration<double> d) {\n    return std::make_unique<uniform_process>(d);\n}\n\nclass poisson_process : public pause_distribution {\n    std::random_device _rd;\n    std::mt19937 _rng;\n    std::exponential_distribution<double> _exp;\n\npublic:\n    poisson_process(std::chrono::duration<double> period)\n            : _rng(_rd())\n            , _exp(1.0 / period.count())\n    {\n    }\n\n    std::chrono::duration<double> get() override {\n        return std::chrono::duration<double>(_exp(_rng));\n    }\n};\n\nstd::unique_ptr<pause_distribution> make_poisson_pause(std::chrono::duration<double> d) {\n    return std::make_unique<poisson_process>(d);\n}\n\nstruct byte_size {\n    uint64_t size;\n};\n\nstruct duration_time {\n    std::chrono::duration<float> time;\n};\n\nclass shard_config {\n    std::unordered_set<unsigned> _shards;\npublic:\n    shard_config()\n        : _shards(boost::copy_range<std::unordered_set<unsigned>>(boost::irange(0u, smp::count))) {}\n    shard_config(std::unordered_set<unsigned> s) : _shards(std::move(s)) {}\n\n    bool is_set(unsigned cpu) const {\n        return _shards.count(cpu);\n    }\n};\n\nstruct shard_info {\n    unsigned parallelism = 0;\n    unsigned rps = 0;\n    unsigned batch = 1;\n    unsigned limit = std::numeric_limits<unsigned>::max();\n    unsigned shares = 10;\n    std::string sched_class = \"\";\n    uint64_t request_size = 4 << 10;\n    uint64_t bandwidth = 0;\n    std::chrono::duration<float> think_time = 0ms;\n    std::chrono::duration<float> think_after = 0ms;\n    std::chrono::duration<float> execution_time = 1ms;\n    seastar::scheduling_group scheduling_group = seastar::default_scheduling_group();\n    bool vectorized = false;\n    unsigned iov_count = 1;\n};\n\nstruct options {\n    bool dsync = false;\n    ::sleep_fn sleep_fn = timer_sleep<lowres_clock>;\n    ::pause_fn pause_fn = make_uniform_pause;\n    // the value passed as a hint for allocated extent size\n    // if not specified, then file_size is used as a hint\n    std::optional<uint64_t> extent_allocation_size_hint;\n    bool pre_allocate_blocks = false;\n    std::optional<uint64_t> sloppy_size_hint;\n};\n\nclass class_data;\n\nstruct job_config {\n    std::string name;\n    request_type type;\n    shard_config shard_placement;\n    ::shard_info shard_info;\n    ::options options;\n    // size of each individual file. Every class and every shard have its file, so in a normal\n    // system with many shards we'll naturally have many files and that will push the data out\n    // of the disk's cache. An exception to that rule is unlink_class_data, that creates files_count\n    // files with file_size/files_count.\n    uint64_t file_size;\n    // the number of files to create and unlink by unlink_class_data per shard\n    // remaining operations utilize only one file per shard\n    std::optional<uint64_t> files_count;\n    uint64_t offset_in_bdev;\n    std::unique_ptr<class_data> gen_class_data();\n};\n\nstruct sched_group_config {\n    std::string name;\n    std::string parent;\n    unsigned shares;\n\n    bool is_supergroup = false;\n};\n\nstd::array<double, 4> quantiles = { 0.5, 0.95, 0.99, 0.999};\nstd::array<double, 2> quantiles_short = { 0.5, 0.9 };\nstatic bool keep_files = false;\n\nfuture<> maybe_remove_file(sstring fname) {\n    return keep_files ? make_ready_future<>() : remove_file(fname);\n}\n\nfuture<> maybe_close_file(file& f) {\n    return f ? f.close() : make_ready_future<>();\n}\n\nclass class_data {\nprotected:\n    using accumulator_type = accumulator_set<double, stats<tag::extended_p_square_quantile(quadratic), tag::mean, tag::max>>;\n\n    job_config _config;\n    uint64_t _alignment;\n\n    seastar::scheduling_group _sg;\n\n    size_t _data = 0;\n    std::chrono::duration<float> _total_duration;\n\n    std::chrono::steady_clock::time_point _start = {};\n    accumulator_type _latencies;\n    uint64_t _requests = 0;\n    file _file;\n    bool _think = false;\n    ::sleep_fn _sleep_fn = timer_sleep<lowres_clock>;\n    timer<> _thinker;\n\n    virtual future<> do_start(sstring dir, directory_entry_type type) = 0;\n    virtual future<size_t> issue_request(char *buf, io_intent* intent) = 0;\npublic:\n    class_data(job_config cfg)\n        : _config(std::move(cfg))\n        , _alignment(_config.shard_info.request_size >= 4096 ? 4096 : 512)\n        , _sg(cfg.shard_info.scheduling_group)\n        , _latencies(extended_p_square_probabilities = quantiles)\n        , _sleep_fn(_config.options.sleep_fn)\n        , _thinker([this] { think_tick(); })\n    {\n        if (_config.shard_info.think_after > 0us) {\n            _thinker.arm(std::chrono::duration_cast<std::chrono::microseconds>(_config.shard_info.think_after));\n        } else if (_config.shard_info.think_time > 0us) {\n            _think = true;\n        }\n    }\n\n    virtual ~class_data() = default;\n\nprivate:\n\n    void think_tick() {\n        if (_think) {\n            _think = false;\n            _thinker.arm(std::chrono::duration_cast<std::chrono::microseconds>(_config.shard_info.think_after));\n        } else {\n            _think = true;\n            _thinker.arm(std::chrono::duration_cast<std::chrono::microseconds>(_config.shard_info.think_time));\n        }\n    }\n\n    future<> issue_request(char* buf, io_intent* intent, std::chrono::steady_clock::time_point start, std::chrono::steady_clock::time_point stop) {\n        return issue_request(buf, intent).then([this, start, stop] (auto size) {\n            auto now = std::chrono::steady_clock::now();\n            if (now < stop) {\n                this->add_result(size, std::chrono::duration_cast<std::chrono::microseconds>(now - start));\n            }\n            return make_ready_future<>();\n        });\n    }\n\n    future<> issue_requests_in_parallel(std::chrono::steady_clock::time_point stop) {\n        return parallel_for_each(std::views::iota(0u, parallelism()), [this, stop] (auto dummy) mutable {\n            auto bufptr = allocate_aligned_buffer<char>(this->req_size(), _alignment);\n            auto buf = bufptr.get();\n            return do_until([this, stop] { return std::chrono::steady_clock::now() > stop || requests() > limit(); }, [this, buf, stop] () mutable {\n                auto start = std::chrono::steady_clock::now();\n                return issue_request(buf, nullptr, start, stop).then([this] {\n                    return think();\n                });\n            }).finally([bufptr = std::move(bufptr)] {});\n        });\n    }\n\n    future<> issue_requests_at_rate(std::chrono::steady_clock::time_point stop) {\n        return do_with(io_intent{}, 0u, [this, stop] (io_intent& intent, unsigned& in_flight) {\n            return parallel_for_each(std::views::iota(0u, parallelism()), [this, stop, &intent, &in_flight] (auto dummy) mutable {\n                auto bufptr = allocate_aligned_buffer<char>(this->req_size(), _alignment);\n                auto buf = bufptr.get();\n                auto pause = std::chrono::duration_cast<std::chrono::microseconds>(1s) / rps();\n                auto pause_dist = _config.options.pause_fn(pause);\n                return seastar::sleep((pause / parallelism()) * dummy).then([this, buf, stop, pause = pause_dist.get(), &intent, &in_flight] () mutable {\n                    return do_until([this, stop] { return std::chrono::steady_clock::now() > stop || requests() > limit(); }, [this, buf, stop, pause, &intent, &in_flight] () mutable {\n                        auto start = std::chrono::steady_clock::now();\n                        in_flight++;\n                        return parallel_for_each(std::views::iota(0u, batch()), [this, buf, &intent, start, stop] (auto dummy) {\n                            return issue_request(buf, &intent, start, stop);\n                        }).then([this, start, pause] {\n                            auto now = std::chrono::steady_clock::now();\n                            auto p = pause->template get_as<std::chrono::microseconds>();\n                            auto next = start + p;\n\n                            if (next > now) {\n                                return this->_sleep_fn(next, now);\n                            } else {\n                                // probably the system cannot keep-up with this rate\n                                return make_ready_future<>();\n                            }\n                        }).handle_exception_type([] (const cancelled_error&) {\n                            // expected\n                        }).finally([&in_flight] {\n                            in_flight--;\n                        });\n                    });\n                }).finally([bufptr = std::move(bufptr), pause = std::move(pause_dist)] {});\n            }).then([&intent, &in_flight] {\n                intent.cancel();\n                return do_until([&in_flight] { return in_flight == 0; }, [] { return seastar::sleep(100ms /* ¯\\_(ツ)_/¯ */); });\n            });\n        });\n    }\n\npublic:\n    future<> issue_requests(std::chrono::steady_clock::time_point stop) {\n        _start = std::chrono::steady_clock::now();\n        return with_scheduling_group(_sg, [this, stop] {\n            if (rps() == 0) {\n                return issue_requests_in_parallel(stop);\n            } else {\n                return issue_requests_at_rate(stop);\n            }\n        }).then([this] {\n            _total_duration = std::chrono::steady_clock::now() - _start;\n        });\n    }\n\n    future<> think() {\n        if (_think) {\n            return seastar::sleep(std::chrono::duration_cast<std::chrono::microseconds>(_config.shard_info.think_time));\n        } else {\n            return make_ready_future<>();\n        }\n    }\n    // Generate the test file(s) for reads and writes alike. It is much simpler to just generate one file per job instead of expecting\n    // job dependencies between creators and consumers. Removal of files is an exception - it creates multiple files during startup to\n    // unlink them. So every job (a class in a shard) will have its own file(s) and will operate differently depending on the type:\n    //\n    // sequential reads  : will read the file from pos = 0 onwards, back to 0 on EOF\n    // overwrites        : will write the file from pos = 0 onwards, back to 0 on EOF\n    // random reads      : will read the file at random positions, between 0 and EOF\n    // random writes     : will overwrite the file at a random position, between 0 and EOF\n    // append            : will write to the file from pos = EOF onwards, always appending to the end.\n    // unlink            : will unlink files created at the beginning of the execution\n    // cpu               : CPU-only load, file is not created.\n    future<> start(sstring dir, directory_entry_type type) {\n        return do_start(dir, type).then([this] {\n            if (this_shard_id() == 0 && _config.shard_info.bandwidth != 0) {\n                return _sg.update_io_bandwidth(_config.shard_info.bandwidth);\n            } else {\n                return make_ready_future<>();\n            }\n        });\n    }\n\n    future<> stop() {\n        return stop_hook().finally([this] {\n            return maybe_close_file(_file);\n        });\n    }\n\n    const sstring name() const {\n        return _config.name;\n    }\n\nprotected:\n    sstring type_str() const {\n        auto base = std::unordered_map<request_type, sstring>{\n            { request_type::seqread, \"SEQ READ\" },\n            { request_type::overwrite, \"OVERWRITE\" },\n            { request_type::randread, \"RAND READ\" },\n            { request_type::randwrite, \"RAND WRITE\" },\n            { request_type::append , \"APPEND\" },\n            { request_type::cpu , \"CPU\" },\n            { request_type::unlink, \"UNLINK\" },\n        }[_config.type];\n        if (_config.shard_info.vectorized) {\n            return format(\"{} (vectorized, {} iovecs)\", base, _config.shard_info.iov_count);\n        }\n        return base;\n    }\n\n    request_type req_type() const {\n        return _config.type;\n    }\n\n    sstring think_time() const {\n        if (_config.shard_info.think_time == std::chrono::duration<float>(0)) {\n            return \"NO think time\";\n        } else {\n            return format(\"{:d} us think time\", std::chrono::duration_cast<std::chrono::microseconds>(_config.shard_info.think_time).count());\n        }\n    }\n\n    size_t req_size() const {\n        return _config.shard_info.request_size;\n    }\n\n    unsigned parallelism() const {\n        return _config.shard_info.parallelism;\n    }\n\n    unsigned rps() const {\n        return _config.shard_info.rps;\n    }\n\n    unsigned batch() const {\n        return _config.shard_info.batch;\n    }\n\n    unsigned limit() const noexcept {\n        return _config.shard_info.limit;\n    }\n\n    unsigned shares() const {\n        return _config.shard_info.shares;\n    }\n\n    std::chrono::duration<float> total_duration() const {\n        return _total_duration;\n    }\n\n    uint64_t file_size_mb() const {\n        return _config.file_size >> 20;\n    }\n\n    uint64_t total_data() const {\n        return _data;\n    }\n\n    uint64_t max_latency() const {\n        return max(_latencies);\n    }\n\n    uint64_t average_latency() const {\n        return mean(_latencies);\n    }\n\n    uint64_t quantile_latency(double q) const {\n        return quantile(_latencies, quantile_probability = q);\n    }\n\n    uint64_t requests() const noexcept {\n        return _requests;\n    }\n\n    void add_result(size_t data, std::chrono::microseconds latency) {\n        _data += data;\n        _latencies(latency.count());\n        _requests++;\n    }\n\npublic:\n    virtual void emit_results(YAML::Emitter& out) = 0;\n    virtual future<> stop_hook() {\n        return make_ready_future<>();\n    }\n};\n\nclass io_class_data : public class_data {\n    uint64_t _next_seq_pos = 0;\n    uint64_t _offset = 0;\n    unsigned _overflows = 0;\n    std::uniform_int_distribution<uint32_t> _pos_distribution;\nprotected:\n    bool _is_dev_null = false;\n    timer<> _queue_length_timer;\n    accumulator_type _disk_queue_lengths;\n\n    future<size_t> on_io_completed(future<size_t> f) {\n        if (!_is_dev_null) {\n            return f;\n        }\n\n        return f.then([this] (auto size_f) {\n            return make_ready_future<size_t>(this->req_size());\n        });\n    }\n\n    bool is_random() const {\n        return (req_type() == request_type::randread) || (req_type() == request_type::randwrite);\n    }\n\n    uint64_t get_pos() {\n        uint64_t pos;\n        if (is_random()) {\n            pos = _pos_distribution(random_generator) * req_size();\n        } else {\n            pos = _next_seq_pos;\n            _next_seq_pos += req_size();\n            if (_next_seq_pos >= _config.file_size) {\n                _overflows++;\n                if (req_type() != request_type::append) {\n                    _next_seq_pos = 0;\n                }\n            }\n        }\n        return pos + _offset;\n    }\n\npublic:\n    io_class_data(job_config cfg)\n            : class_data(std::move(cfg))\n            , _pos_distribution(0,  _config.file_size / _config.shard_info.request_size)\n            , _queue_length_timer([this] { update_queue_length(); })\n            , _disk_queue_lengths(extended_p_square_probabilities = quantiles_short)\n    {}\n\n    future<> do_start(sstring path, directory_entry_type type) override {\n        return do_start_path(std::move(path), type).then([this] {\n            _queue_length_timer.arm(std::chrono::steady_clock::now() + 500ms, 200ms);\n        });\n    }\n\nprivate:\n    void update_queue_length() {\n        unsigned qlen = get_one_metrics(\"io_queue_disk_queue_length\").value();\n        _disk_queue_lengths(qlen);\n    }\n\n    future<> do_start_path(sstring path, directory_entry_type type) {\n        if (type == directory_entry_type::directory) {\n            return do_start_on_directory(path);\n        }\n\n        if (type == directory_entry_type::block_device) {\n            return do_start_on_bdev(path);\n        }\n\n        if (type == directory_entry_type::char_device && path == \"/dev/null\") {\n            return do_start_on_dev_null();\n        }\n\n        throw std::runtime_error(format(\"Unsupported storage. {} should be directory or block device\", path));\n    }\n\n    future<> do_start_on_directory(sstring dir) {\n        auto fname = format(\"{}/test-{}-{:d}\", dir, name(), this_shard_id());\n        auto flags = open_flags::rw | open_flags::create;\n        if (_config.options.dsync) {\n            flags |= open_flags::dsync;\n        }\n        file_open_options options;\n        options.extent_allocation_size_hint = _config.options.extent_allocation_size_hint.value_or(_config.file_size);\n        if (_config.options.sloppy_size_hint.has_value()) {\n            options.sloppy_size = true;\n            options.sloppy_size_hint = *_config.options.sloppy_size_hint;\n        }\n        options.append_is_unlikely = (req_type() != request_type::append);\n\n        return create_and_fill_file(fname, _config.file_size, flags, options, req_type() != request_type::append, _config.options.pre_allocate_blocks).then([this](file f) {\n            _file = std::move(f);\n            return make_ready_future<>();\n        }).then([fname] {\n            // If keep_files == false, then the file shall not exist after the execution.\n            // After the following function call the usage of the file is valid until `this->_file` object is closed.\n            return maybe_remove_file(fname);\n        });\n    }\n\n    future<> do_start_on_bdev(sstring name) {\n        auto flags = open_flags::rw;\n        if (_config.options.dsync) {\n            flags |= open_flags::dsync;\n        }\n\n        return open_file_dma(name, flags).then([this] (auto f) {\n            _file = std::move(f);\n            return _file.size().then([this] (uint64_t size) {\n                auto shard_area_size = align_down<uint64_t>(size / smp::count, 1 << 20);\n                if (_config.offset_in_bdev + _config.file_size > shard_area_size) {\n                    throw std::runtime_error(\"Data doesn't fit the blockdevice\");\n                }\n                _offset = shard_area_size * this_shard_id() + _config.offset_in_bdev;\n                return make_ready_future<>();\n            });\n        });\n    }\n\n    future<> do_start_on_dev_null() {\n        file_open_options options;\n        options.append_is_unlikely = true;\n        return open_file_dma(\"/dev/null\", open_flags::rw, std::move(options)).then([this] (auto f) {\n            _file = std::move(f);\n            _is_dev_null = true;\n            return make_ready_future<>();\n        });\n    }\n\n    std::optional<double> get_one_metrics(sstring m_name, bool check_class_metrics = true) {\n        const auto& values = seastar::metrics::impl::get_value_map();\n        const auto& mf = values.find(m_name);\n        SEASTAR_ASSERT(mf != values.end());\n        for (auto&& mi : mf->second) {\n            if (check_class_metrics) {\n                auto&& cname = mi.first.labels().find(\"class\");\n                if (cname == mi.first.labels().end() || cname->second.value() != name()) {\n                    continue;\n                }\n            }\n            return mi.second->get_function()().d();\n        }\n        return {};\n    }\n\n    void emit_one_metrics(YAML::Emitter& out, sstring m_name, bool check_class_metrics = true) {\n        if (auto v = get_one_metrics(m_name, check_class_metrics); v.has_value()) {\n            out << YAML::Key << m_name << YAML::Value << *v;\n        }\n    }\n\n    void emit_metrics(YAML::Emitter& out) {\n        emit_one_metrics(out, \"io_queue_total_exec_sec\");\n        emit_one_metrics(out, \"io_queue_total_delay_sec\");\n        emit_one_metrics(out, \"io_queue_total_operations\");\n        emit_one_metrics(out, \"io_queue_starvation_time_sec\");\n        emit_one_metrics(out, \"io_queue_consumption\");\n        emit_one_metrics(out, \"io_queue_adjusted_consumption\");\n        emit_one_metrics(out, \"io_queue_activations\");\n        emit_one_metrics(out, \"reactor_aio_outsizes\", false);\n    }\n\npublic:\n    virtual void emit_results(YAML::Emitter& out) override {\n        auto throughput_kbs = (total_data() >> 10) / total_duration().count();\n        auto iops = requests() / total_duration().count();\n        out << YAML::Key << \"throughput\" << YAML::Value << throughput_kbs << YAML::Comment(\"kB/s\");\n        out << YAML::Key << \"IOPS\" << YAML::Value << iops;\n        out << YAML::Key << \"latencies\" << YAML::Comment(\"usec\");\n        out << YAML::BeginMap;\n        out << YAML::Key << \"average\" << YAML::Value << average_latency();\n        for (auto& q: quantiles) {\n            out << YAML::Key << fmt::format(\"p{}\", q) << YAML::Value << quantile_latency(q);\n        }\n        out << YAML::Key << \"max\" << YAML::Value << max_latency();\n        out << YAML::EndMap;\n        out << YAML::Key << \"stats\" << YAML::BeginMap;\n        out << YAML::Key << \"total_requests\" << YAML::Value << requests();\n        emit_metrics(out);\n        out << YAML::Key << \"disk_queue_length\";\n        out << YAML::BeginMap;\n        for (auto& q: quantiles_short) {\n            out << YAML::Key << fmt::format(\"p{}\", q) << YAML::Value << (unsigned)quantile(_disk_queue_lengths, quantile_probability = q);\n        }\n        out << YAML::Key << \"max\" << YAML::Value << (unsigned)max(_disk_queue_lengths);\n        out << YAML::EndMap;\n        out << YAML::Key << \"file_size_overflows\" << YAML::Value << _overflows;\n        out << YAML::EndMap;\n    }\n};\n\nclass read_io_class_data : public io_class_data {\npublic:\n    read_io_class_data(job_config cfg) : io_class_data(std::move(cfg)) {}\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        auto f = _file.dma_read(this->get_pos(), buf, this->req_size(), intent);\n        return on_io_completed(std::move(f));\n    }\n};\n\nclass write_io_class_data : public io_class_data {\npublic:\n    write_io_class_data(job_config cfg) : io_class_data(std::move(cfg)) {}\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        auto f = _file.dma_write(this->get_pos(), buf, this->req_size(), intent);\n        return on_io_completed(std::move(f));\n    }\n};\n\nclass vectorized_read_io_class_data : public io_class_data {\n    std::vector<std::unique_ptr<char[], free_deleter>> _buffers;\n    std::vector<iovec> _iovecs;\n    size_t _segment_size;\n\npublic:\n    vectorized_read_io_class_data(job_config cfg)\n            : io_class_data(std::move(cfg))\n            , _segment_size(_config.shard_info.request_size / _config.shard_info.iov_count)\n    {\n        if (_config.shard_info.request_size % _config.shard_info.iov_count != 0) {\n            throw std::runtime_error(format(\"request_size {} must be evenly divisible by iov_count {}\",\n                _config.shard_info.request_size, _config.shard_info.iov_count));\n        }\n        if (_segment_size < _alignment || _segment_size % _alignment != 0) {\n            throw std::runtime_error(format(\"segment_size {} must be at least {} and aligned to {}\",\n                _segment_size, _alignment, _alignment));\n        }\n        allocate_buffers();\n    }\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        auto pos = this->get_pos();\n        std::vector<iovec> iovs = _iovecs;\n        auto f = _file.dma_read(pos, std::move(iovs), intent);\n        return on_io_completed(std::move(f));\n    }\n\nprivate:\n    void allocate_buffers() {\n        _buffers.reserve(_config.shard_info.iov_count);\n        _iovecs.reserve(_config.shard_info.iov_count);\n\n        for (unsigned i = 0; i < _config.shard_info.iov_count; ++i) {\n            auto buf = allocate_aligned_buffer<char>(_segment_size, _alignment);\n            _iovecs.push_back(iovec{ buf.get(), _segment_size });\n            _buffers.push_back(std::move(buf));\n        }\n    }\n};\n\nclass vectorized_write_io_class_data : public io_class_data {\n    std::vector<std::unique_ptr<char[], free_deleter>> _buffers;\n    std::vector<iovec> _iovecs;\n    size_t _segment_size;\n\npublic:\n    vectorized_write_io_class_data(job_config cfg)\n            : io_class_data(std::move(cfg))\n            , _segment_size(_config.shard_info.request_size / _config.shard_info.iov_count)\n    {\n        if (_config.shard_info.request_size % _config.shard_info.iov_count != 0) {\n            throw std::runtime_error(format(\"request_size {} must be evenly divisible by iov_count {}\",\n                _config.shard_info.request_size, _config.shard_info.iov_count));\n        }\n        if (_segment_size < _alignment || _segment_size % _alignment != 0) {\n            throw std::runtime_error(format(\"segment_size {} must be at least {} and aligned to {}\",\n                _segment_size, _alignment, _alignment));\n        }\n        allocate_buffers();\n    }\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        auto pos = this->get_pos();\n        std::vector<iovec> iovs = _iovecs;\n        auto f = _file.dma_write(pos, std::move(iovs), intent);\n        return on_io_completed(std::move(f));\n    }\n\nprivate:\n    void allocate_buffers() {\n        _buffers.reserve(_config.shard_info.iov_count);\n        _iovecs.reserve(_config.shard_info.iov_count);\n\n        for (unsigned i = 0; i < _config.shard_info.iov_count; ++i) {\n            auto buf = allocate_and_fill_buffer(_segment_size);\n            _iovecs.push_back(iovec{ buf.get(), _segment_size });\n            _buffers.push_back(std::move(buf));\n        }\n    }\n};\n\nclass unlink_class_data : public class_data {\nprivate:\n    sstring _dir_path{};\n    uint64_t _file_id_to_remove{0u};\n\npublic:\n    unlink_class_data(job_config cfg) : class_data(std::move(cfg)) {\n        if (!_config.files_count.has_value()) {\n            throw std::runtime_error(\"request_type::unlink requires specifying 'files_count'\");\n        }\n    }\n\n    future<> do_start(sstring path, directory_entry_type type) override {\n        if (type == directory_entry_type::directory) {\n            return do_start_on_directory(path);\n        }\n        throw std::runtime_error(format(\"Unsupported storage. {} should be directory\", path));\n    }\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        if (all_files_removed()) {\n            fmt::print(\"[WARNING]: Cannot issue request in unlink_class_data! All files have been removed for shard_id={}\\n\"\n                       \"[WARNING]: Please create more files or adjust the frequency of unlinks.\", this_shard_id());\n            return make_ready_future<size_t>(0u);\n        }\n\n        const auto fname = get_filename(_file_id_to_remove);\n        ++_file_id_to_remove;\n\n        return remove_file(fname).then([]{\n            return make_ready_future<size_t>(0u);\n        });\n    }\n\n    void emit_results(YAML::Emitter& out) override {\n        const auto iops = requests() / total_duration().count();\n        out << YAML::Key << \"IOPS\" << YAML::Value << iops;\n        out << YAML::Key << \"latencies\" << YAML::Comment(\"usec\");\n        out << YAML::BeginMap;\n        out << YAML::Key << \"average\" << YAML::Value << average_latency();\n        out << YAML::Key << \"max\" << YAML::Value << max_latency();\n        out << YAML::EndMap;\n        out << YAML::Key << \"stats\" << YAML::BeginMap;\n        out << YAML::Key << \"total_requests\" << YAML::Value << requests();\n        out << YAML::EndMap;\n    }\n\nprivate:\n    future<> stop_hook() override {\n        if (all_files_removed() || keep_files) {\n            return make_ready_future<>();\n        }\n\n        return max_concurrent_for_each(std::views::iota(_file_id_to_remove, files_count()), max_concurrency(), [this] (uint64_t file_id) {\n            const auto fname = get_filename(file_id);\n            return remove_file(fname);\n        });\n    }\n\n    uint64_t files_count() const {\n        return *_config.files_count;\n    }\n\n    uint64_t max_concurrency() const {\n        // When we have many files it is easy to exceed the limit of open file descriptors.\n        // To avoid that the limit is divided between shards (leaving some room for other jobs).\n        return static_cast<uint64_t>((1024u / smp::count) * 0.8);\n    }\n\n    bool all_files_removed() const {\n        return files_count() <= _file_id_to_remove;\n    }\n\n    sstring get_filename(uint64_t file_id) const {\n        return format(\"{}/test-{}-shard-{:d}-file-{}\", _dir_path, name(), this_shard_id(), file_id);\n    }\n\n    future<> do_start_on_directory(sstring path) {\n        _dir_path = std::move(path);\n\n        return max_concurrent_for_each(std::views::iota(UINT64_C(0), files_count()), max_concurrency(), [this] (uint64_t file_id) {\n            const auto fname = get_filename(file_id);\n            const auto fsize = align_up<uint64_t>(_config.file_size / files_count(), extent_size_hint_alignment);\n            const auto flags = open_flags::rw | open_flags::create;\n\n            file_open_options options;\n            options.extent_allocation_size_hint = _config.options.extent_allocation_size_hint.value_or(fsize);\n            options.append_is_unlikely = true;\n\n            return create_and_fill_file(fname, fsize, flags, options, true, false).then([](file f) {\n                return do_with(std::move(f), [] (auto& f) {\n                    return f.close();\n                });\n            });\n        });\n    }\n};\n\nclass cpu_class_data : public class_data {\npublic:\n    cpu_class_data(job_config cfg) : class_data(std::move(cfg)) {}\n\n    future<> do_start(sstring dir, directory_entry_type type) override {\n        return make_ready_future<>();\n    }\n\n    future<size_t> issue_request(char *buf, io_intent* intent) override {\n        // We do want the execution time to be a busy loop, and not just a bunch of\n        // continuations until our time is up: by doing this we can also simulate the behavior\n        // of I/O continuations in the face of reactor stalls.\n        auto start  = std::chrono::steady_clock::now();\n        do {\n        } while ((std::chrono::steady_clock::now() - start) < _config.shard_info.execution_time);\n        return make_ready_future<size_t>(1);\n    }\n\n    virtual void emit_results(YAML::Emitter& out) override {\n        auto throughput = total_data() / total_duration().count();\n        out << YAML::Key << \"throughput\" << YAML::Value << throughput;\n    }\n};\n\nstd::unique_ptr<class_data> job_config::gen_class_data() {\n    if (type == request_type::cpu) {\n        return std::make_unique<cpu_class_data>(*this);\n    } else if (type == request_type::unlink) {\n        return std::make_unique<unlink_class_data>(*this);\n    } else if ((type == request_type::seqread) || (type == request_type::randread)) {\n        if (shard_info.vectorized) {\n            return std::make_unique<vectorized_read_io_class_data>(*this);\n        }\n        return std::make_unique<read_io_class_data>(*this);\n    } else {\n        if (shard_info.vectorized) {\n            return std::make_unique<vectorized_write_io_class_data>(*this);\n        }\n        return std::make_unique<write_io_class_data>(*this);\n    }\n}\n\n/// YAML parsing functions\nnamespace YAML {\ntemplate<>\nstruct convert<byte_size> {\n    static bool decode(const Node& node, byte_size& bs) {\n        auto str = node.as<std::string>();\n        unsigned shift = 0;\n        if (str.back() == 'B') {\n            str.pop_back();\n            shift = std::unordered_map<char, unsigned>{\n                { 'k', 10 },\n                { 'M', 20 },\n                { 'G', 30 },\n            }[str.back()];\n            str.pop_back();\n        }\n        bs.size = (boost::lexical_cast<size_t>(str) << shift);\n        return bs.size >= 512;\n    }\n};\n\ntemplate<>\nstruct convert<duration_time> {\n    static bool decode(const Node& node, duration_time& dt) {\n        auto str = node.as<std::string>();\n        if (str == \"0\") {\n            dt.time = 0ns;\n            return true;\n        }\n        if (str.back() != 's') {\n            return false;\n        }\n        str.pop_back();\n        std::unordered_map<char, std::chrono::duration<float>> unit = {\n            { 'n', 1ns },\n            { 'u', 1us },\n            { 'm', 1ms },\n        };\n\n        if (unit.count(str.back())) {\n            auto u = str.back();\n            str.pop_back();\n            dt.time = (boost::lexical_cast<size_t>(str) * unit[u]);\n        } else {\n            dt.time = (boost::lexical_cast<size_t>(str) * 1s);\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<shard_config> {\n    static bool decode(const Node& node, shard_config& shards) {\n        try {\n            auto str = node.as<std::string>();\n            return (str == \"all\");\n        } catch (YAML::TypedBadConversion<std::string>& e) {\n            shards = shard_config(boost::copy_range<std::unordered_set<unsigned>>(node.as<std::vector<unsigned>>()));\n            return true;\n        }\n        return false;\n    }\n};\n\ntemplate<>\nstruct convert<request_type> {\n    static bool decode(const Node& node, request_type& rt) {\n        static std::unordered_map<std::string, request_type> mappings = {\n            { \"seqread\", request_type::seqread },\n            { \"overwrite\", request_type::overwrite},\n            { \"randread\", request_type::randread },\n            { \"randwrite\", request_type::randwrite },\n            { \"append\", request_type::append},\n            { \"cpu\", request_type::cpu},\n            { \"unlink\", request_type::unlink },\n        };\n        auto reqstr = node.as<std::string>();\n        if (!mappings.count(reqstr)) {\n            return false;\n        }\n        rt = mappings[reqstr];\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<shard_info> {\n    static bool decode(const Node& node, shard_info& sl) {\n        if (node[\"parallelism\"]) {\n            sl.parallelism = node[\"parallelism\"].as<unsigned>();\n        }\n        if (node[\"rps\"]) {\n            sl.rps = node[\"rps\"].as<unsigned>();\n        }\n        if (node[\"batch\"]) {\n            sl.batch = node[\"batch\"].as<unsigned>();\n        }\n        if (node[\"limit\"]) {\n            sl.limit = node[\"limit\"].as<unsigned>();\n        }\n\n        if (node[\"shares\"]) {\n            sl.shares = node[\"shares\"].as<unsigned>();\n        } else if (node[\"class\"]) {\n            sl.sched_class = node[\"class\"].as<std::string>();\n        }\n        if (node[\"bandwidth\"]) {\n            sl.bandwidth = node[\"bandwidth\"].as<byte_size>().size;\n        }\n        if (node[\"reqsize\"]) {\n            sl.request_size = node[\"reqsize\"].as<byte_size>().size;\n        }\n        if (node[\"think_time\"]) {\n            sl.think_time = node[\"think_time\"].as<duration_time>().time;\n        }\n        if (node[\"think_after\"]) {\n            sl.think_after = node[\"think_after\"].as<duration_time>().time;\n        }\n        if (node[\"execution_time\"]) {\n            sl.execution_time = node[\"execution_time\"].as<duration_time>().time;\n        }\n        if (node[\"vectorized\"]) {\n            sl.vectorized = node[\"vectorized\"].as<bool>();\n        }\n        if (node[\"iov_count\"]) {\n            sl.iov_count = node[\"iov_count\"].as<unsigned>();\n            if (sl.iov_count == 0) {\n                throw std::runtime_error(\"iov_count must be at least 1\");\n            }\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<options> {\n    static bool decode(const Node& node, options& op) {\n        if (node[\"dsync\"]) {\n            op.dsync = node[\"dsync\"].as<bool>();\n        }\n        if (node[\"sleep_type\"]) {\n            auto st = node[\"sleep_type\"].as<std::string>();\n            if (st == \"busyloop\") {\n                op.sleep_fn = busyloop_sleep;\n            } else if (st == \"lowres\") {\n                op.sleep_fn = timer_sleep<lowres_clock>;\n            } else if (st == \"steady\") {\n                op.sleep_fn = timer_sleep<std::chrono::steady_clock>;\n            } else {\n                throw std::runtime_error(seastar::format(\"Unknown sleep_type {}\", st));\n            }\n        }\n        if (node[\"pause_distribution\"]) {\n            auto pd = node[\"pause_distribution\"].as<std::string>();\n            if (pd == \"uniform\") {\n                op.pause_fn = make_uniform_pause;\n            } else if (pd == \"poisson\") {\n                op.pause_fn = make_poisson_pause;\n            } else {\n                throw std::runtime_error(seastar::format(\"Unknown pause_distribution {}\", pd));\n            }\n        }\n        // By default the file size is used as the allocation hint.\n        // However, certain tests may require using a specific value (e.g. 32MB).\n        if (node[\"extent_allocation_size_hint\"]) {\n            op.extent_allocation_size_hint = node[\"extent_allocation_size_hint\"].as<byte_size>().size;\n        }\n        if (node[\"sloppy_size_hint\"]) {\n            op.sloppy_size_hint = node[\"sloppy_size_hint\"].as<byte_size>().size;\n        }\n        if (node[\"fallocate\"]) {\n            op.pre_allocate_blocks = node[\"fallocate\"].as<bool>();\n        }\n\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<job_config> {\n    static bool decode(const Node& node, job_config& cl) {\n        cl.name = node[\"name\"].as<std::string>();\n        cl.type = node[\"type\"].as<request_type>();\n        cl.shard_placement = node[\"shards\"].as<shard_config>();\n        // The data_size is used to divide the available (and effectively\n        // constant) disk space between workloads. Each shard inside the\n        // workload thus uses its portion of the assigned space.\n        if (node[\"data_size\"]) {\n            const uint64_t per_shard_bytes = node[\"data_size\"].as<byte_size>().size / smp::count;\n            cl.file_size = align_up<uint64_t>(per_shard_bytes, extent_size_hint_alignment);\n        } else if (cl.type == request_type::append) {\n            cl.file_size = 0;\n        } else {\n            cl.file_size = 1ull << 30; // 1G by default\n        }\n\n        // By default a job may create 0 or 1 file.\n        // That is not the case for unlink_class_data - it creates multiple\n        // files that are unlinked during the execution.\n        if (node[\"files_count\"]) {\n            cl.files_count = node[\"files_count\"].as<uint64_t>();\n        }\n\n        if (node[\"shard_info\"]) {\n            cl.shard_info = node[\"shard_info\"].as<shard_info>();\n        }\n        if (node[\"options\"]) {\n            cl.options = node[\"options\"].as<options>();\n        }\n\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<sched_group_config> {\n    static bool decode(const Node& node, sched_group_config& sc) {\n        sc.name = node[\"name\"].as<std::string>();\n        sc.shares = node[\"shares\"].as<unsigned>();\n        if (node[\"parent\"]) {\n            sc.parent = node[\"parent\"].as<std::string>();\n        }\n        return true;\n    }\n};\n}\n\n/// Each shard has one context, and the context is responsible for creating the classes that should\n/// run in this shard.\nclass context {\n    std::vector<std::unique_ptr<class_data>> _cl;\n\n    sstring _dir;\n    directory_entry_type _type;\n    std::chrono::seconds _duration;\n\n    semaphore _finished;\npublic:\n    context(sstring dir, directory_entry_type dtype, std::vector<job_config> req_config, unsigned duration)\n            : _cl(boost::copy_range<std::vector<std::unique_ptr<class_data>>>(req_config\n                | boost::adaptors::filtered([] (auto& cfg) { return cfg.shard_placement.is_set(this_shard_id()); })\n                | boost::adaptors::transformed([] (auto& cfg) { return cfg.gen_class_data(); })\n            ))\n            , _dir(dir)\n            , _type(dtype)\n            , _duration(duration)\n            , _finished(0)\n    {}\n\n    future<> stop() {\n        return parallel_for_each(_cl, [] (std::unique_ptr<class_data>& cl) {\n            return cl->stop();\n        });\n    }\n\n    future<> start() {\n        return parallel_for_each(_cl, [this] (std::unique_ptr<class_data>& cl) {\n            return cl->start(_dir, _type);\n        });\n    }\n\n    future<> issue_requests() {\n        return parallel_for_each(_cl.begin(), _cl.end(), [this] (std::unique_ptr<class_data>& cl) {\n            return cl->issue_requests(std::chrono::steady_clock::now() + _duration).finally([this] {\n                _finished.signal(1);\n            });\n        });\n    }\n\n    future<> emit_results(YAML::Emitter& out) {\n        return _finished.wait(_cl.size()).then([this, &out] {\n            for (auto& cl: _cl) {\n                out << YAML::Key << cl->name();\n                out << YAML::BeginMap;\n                cl->emit_results(out);\n                out << YAML::EndMap;\n            }\n            out << YAML::Key << \"statistics\";\n            out << YAML::BeginMap;\n            out << YAML::Key << \"aio_retries\" << YAML::Value << engine().get_io_stats().aio_retries;\n            out << YAML::EndMap;\n            return make_ready_future<>();\n        });\n    }\n};\n\nstatic void show_results(sharded<context>& ctx) {\n    YAML::Emitter out;\n    out << YAML::BeginDoc;\n    out << YAML::BeginSeq;\n    for (unsigned i = 0; i < smp::count; ++i) {\n        out << YAML::BeginMap;\n        out << YAML::Key << \"shard\" << YAML::Value << i;\n        ctx.invoke_on(i, [&out] (auto& c) {\n            return c.emit_results(out);\n        }).get();\n        out << YAML::EndMap;\n    }\n    out << YAML::EndSeq;\n    out << YAML::EndDoc;\n    std::cout << out.c_str();\n}\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n\n    app_template app;\n    auto opt_add = app.add_options();\n    opt_add\n        (\"storage\", bpo::value<sstring>()->default_value(\".\"), \"directory or block device where to execute the test\")\n        (\"duration\", bpo::value<unsigned>()->default_value(10), \"for how long (in seconds) to run the test\")\n        (\"conf\", bpo::value<sstring>()->default_value(\"./conf.yaml\"), \"YAML file containing benchmark specification\")\n        (\"keep-files\", bpo::value<bool>()->default_value(false), \"keep test files, next run may re-use them\")\n        (\"sched-groups\", bpo::value<sstring>(), \"YAML file containing scheduling groups configuration\")\n    ;\n\n    sharded<context> ctx;\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            auto& opts = app.configuration();\n            auto& storage = opts[\"storage\"].as<sstring>();\n\n            auto st_type = engine().file_type(storage).get();\n\n            if (!st_type) {\n                throw std::runtime_error(format(\"Unknown storage {}\", storage));\n            }\n\n            if (*st_type == directory_entry_type::directory) {\n                auto fs = file_system_at(storage).get();\n                if (fs != fs_type::xfs) {\n                    std::cout << \"WARNING!!! This is a performance test. \" << storage << \" is not on XFS\" << std::endl;\n                }\n            }\n\n            keep_files = opts[\"keep-files\"].as<bool>();\n            auto& duration = opts[\"duration\"].as<unsigned>();\n            auto& yaml = opts[\"conf\"].as<sstring>();\n            YAML::Node doc = YAML::LoadFile(yaml);\n            auto reqs = doc.as<std::vector<job_config>>();\n\n            struct sched_class {\n                seastar::scheduling_group sg;\n                seastar::scheduling_supergroup ssg;\n            };\n            std::unordered_map<std::string, sched_class> sched_classes;\n\n            if (opts.contains(\"sched-groups\")) {\n                auto& yaml = opts[\"sched-groups\"].as<sstring>();\n                YAML::Node doc = YAML::LoadFile(yaml);\n                auto sched_config = doc.as<std::vector<sched_group_config>>();\n\n                // First, find out which sched_group_config-s are supergroups\n                for (auto& sg : sched_config) {\n                    if (sg.parent.empty()) {\n                        continue;\n                    }\n\n                    auto parent = std::find_if(sched_config.begin(), sched_config.end(), [&sg] (auto& s) { return s.name == sg.parent; });\n                    if (parent == sched_config.end()) {\n                        throw std::runtime_error(fmt::format(\"Unknown parent sched group {} for {} in config\", sg.parent, sg.name));\n                    }\n                    parent->is_supergroup = true;\n                }\n\n                // Second, create the supergroups\n                parallel_for_each(sched_config, [&sched_classes] (auto& g) {\n                    if (!g.is_supergroup) {\n                        return make_ready_future<>();\n                    }\n\n                    fmt::print(\"Creating {} supergroup\\n\", g.name);\n                    return seastar::create_scheduling_supergroup(g.shares).then([&g, &sched_classes] (seastar::scheduling_supergroup ssg) {\n                        sched_classes.insert(std::make_pair(g.name, sched_class { .ssg = ssg }));\n                    });\n                }).get();\n\n                // Finally, create the sched groups\n                parallel_for_each(sched_config, [&sched_classes] (auto& g) {\n                    if (g.is_supergroup) {\n                        return make_ready_future<>();\n                    }\n\n                    seastar::scheduling_supergroup parent;\n                    if (!g.parent.empty()) {\n                        fmt::print(\"Creating {}.{} group\\n\", g.parent, g.name);\n                        parent = sched_classes.at(g.parent).ssg;\n                    } else {\n                        fmt::print(\"Creating {} group\\n\", g.name);\n                    }\n                    return seastar::create_scheduling_group(g.name, g.name, g.shares, parent).then([&g, &sched_classes] (seastar::scheduling_group sg) {\n                        sched_classes.insert(std::make_pair(g.name, sched_class { .sg = sg }));\n                    });\n                }).get();\n            }\n\n            parallel_for_each(reqs, [&sched_classes] (auto& r) {\n                if (r.shard_info.sched_class != \"\") {\n                    return make_ready_future<>();\n                }\n\n                fmt::print(\"Creating {} group (by job name)\\n\", r.name);\n                return seastar::create_scheduling_group(r.name, r.shard_info.shares).then([&r, &sched_classes] (seastar::scheduling_group sg) {\n                    sched_classes.insert(std::make_pair(r.name, sched_class {\n                        .sg = sg,\n                    }));\n                });\n            }).get();\n\n            for (job_config& r : reqs) {\n                auto cname = r.shard_info.sched_class != \"\" ? r.shard_info.sched_class : r.name;\n                fmt::print(\"Job {} -> sched class {}\\n\", r.name, cname);\n                auto& sc = sched_classes.at(cname);\n                r.shard_info.scheduling_group = sc.sg;\n            }\n\n            if (*st_type == directory_entry_type::block_device) {\n                uint64_t off = 0;\n                for (job_config& r : reqs) {\n                    r.offset_in_bdev = off;\n                    off += r.file_size;\n                }\n            }\n\n            ctx.start(storage, *st_type, reqs, duration).get();\n            internal::at_exit([&ctx] {\n                return ctx.stop();\n            });\n            std::cout << \"Creating initial files...\" << std::endl;\n            ctx.invoke_on_all([] (auto& c) {\n                return c.start();\n            }).get();\n            std::cout << \"Starting evaluation...\" << std::endl;\n            ctx.invoke_on_all([] (auto& c) {\n                return c.issue_requests();\n            }).get();\n            show_results(ctx);\n            ctx.stop().get();\n        }).or_terminate();\n    });\n}\n"
  },
  {
    "path": "apps/io_tester/ioinfo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n#include <iostream>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/util/closeable.hh>\n#include <yaml-cpp/yaml.h>\n\nusing namespace seastar;\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n\n    app_template app;\n    auto opt_add = app.add_options();\n    opt_add\n        (\"directory\", bpo::value<sstring>()->default_value(\".\"), \"directory to work on\")\n        (\"max-reqsize\", bpo::value<unsigned>()->default_value(128u * 1024u), \"maximum request size in bytes used when calculating capacity (default: 128kB)\")\n    ;\n\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            auto& opts = app.configuration();\n            auto& storage = opts[\"directory\"].as<sstring>();\n            auto max_reqsz = opts[\"max-reqsize\"].as<unsigned>();\n\n            YAML::Emitter out;\n            out << YAML::BeginDoc;\n            out << YAML::BeginMap;\n\n            engine().open_file_dma(storage + \"/tempfile\", open_flags::rw | open_flags::create | open_flags::exclusive).then([&] (file f) {\n                return with_closeable(std::move(f), [&out, &storage, max_reqsz] (file& f) {\n                    return remove_file(storage + \"/tempfile\").then([&out, &f] {\n                        out << YAML::Key << \"disk_read_max_length\" << YAML::Value << f.disk_read_max_length();\n                        out << YAML::Key << \"disk_write_max_length\" << YAML::Value << f.disk_write_max_length();\n                    }).then([&out, &f, max_reqsz] {\n                        return f.stat().then([&out, max_reqsz] (auto st) {\n                            auto& ioq = engine().get_io_queue(st.st_dev);\n                            auto& cfg = ioq.get_config();\n\n                            out << YAML::Key << \"device\" << YAML::Value << st.st_dev;\n                            out << YAML::Key << \"io_latency_goal_ms\" << YAML::Value <<\n                                    std::chrono::duration_cast<std::chrono::duration<double, std::milli>>(ioq.get_io_latency_goal()).count();\n                            out << YAML::Key << \"io_queue\" << YAML::BeginMap;\n                            out << YAML::Key << \"id\" << YAML::Value << ioq.id();\n                            out << YAML::Key << \"req_count_rate\" << YAML::Value << cfg.req_count_rate;\n                            out << YAML::Key << \"blocks_count_rate\" << YAML::Value << cfg.blocks_count_rate;\n                            out << YAML::Key << \"disk_req_write_to_read_multiplier\" << YAML::Value << cfg.disk_req_write_to_read_multiplier;\n                            out << YAML::Key << \"disk_blocks_write_to_read_multiplier\" << YAML::Value << cfg.disk_blocks_write_to_read_multiplier;\n                            out << YAML::EndMap;\n\n                            out << YAML::Key << \"fair_queue\" << YAML::BeginMap;\n                            out << YAML::Key << \"capacities\" << YAML::BeginMap;\n                            for (size_t sz = 512; sz <= max_reqsz; sz <<= 1) {\n                                out << YAML::Key << sz << YAML::BeginMap;\n                                out << YAML::Key << \"read\" << YAML::Value << ioq.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::read_idx, sz));\n                                out << YAML::Key << \"write\" << YAML::Value << ioq.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::write_idx, sz));\n                                out << YAML::EndMap;\n                            }\n                            out << YAML::EndMap;\n\n                            const auto& fg = internal::get_throttler(ioq, internal::io_direction_and_length::write_idx);\n                            out << YAML::Key << \"per_tick_grab_threshold\" << YAML::Value << fg.per_tick_grab_threshold();\n\n                            const auto& tb = fg.token_bucket();\n                            out << YAML::Key << \"token_bucket\" << YAML::BeginMap;\n                            out << YAML::Key << \"limit\" << YAML::Value << tb.limit();\n                            out << YAML::Key << \"rate\" << YAML::Value << tb.rate();\n                            out << YAML::Key << \"threshold\" << YAML::Value << tb.threshold();\n                            out << YAML::EndMap;\n\n                            out << YAML::EndMap;\n                        });\n                    });\n                });\n            }).get();\n\n            out << YAML::EndMap;\n            out << YAML::EndDoc;\n            std::cout << out.c_str();\n        });\n    });\n}\n"
  },
  {
    "path": "apps/io_tester/sched.yaml",
    "content": "- name: statement\n  shares: 1000\n\n- name: maintenance\n  shares: 100\n\n- name: compaction\n  shares: 200\n  parent: maintenance\n\n- name: backup\n  shares: 100\n  parent: maintenance\n"
  },
  {
    "path": "apps/iotune/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nseastar_add_app (iotune\n  SOURCES iotune.cc)\n\ntarget_link_libraries (app_iotune\n  PRIVATE\n    yaml-cpp::yaml-cpp)\n"
  },
  {
    "path": "apps/iotune/iotune.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB\n *\n * The goal of this program is to allow a user to properly configure the Seastar I/O\n * scheduler.\n */\n#include <iostream>\n#include <chrono>\n#include <optional>\n#include <random>\n#include <memory>\n#include <map>\n#include <ranges>\n#include <vector>\n#include <cmath>\n#include <sys/vfs.h>\n#include <sys/sysmacros.h>\n#include <boost/program_options.hpp>\n#include <boost/iterator/counting_iterator.hpp>\n#include <fstream>\n#include <wordexp.h>\n#include <yaml-cpp/yaml.h>\n#include <fmt/printf.h>\n#include <fmt/ranges.h>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/fsqual.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/read_first_line.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\nnamespace fs = std::filesystem;\n\nlogger iotune_logger(\"iotune\");\n\nusing iotune_clock = std::chrono::steady_clock;\nstatic thread_local std::default_random_engine random_generator(std::chrono::duration_cast<std::chrono::nanoseconds>(iotune_clock::now().time_since_epoch()).count());\n\nvoid check_device_properties(fs::path dev_sys_file) {\n    auto sched_file = dev_sys_file / \"queue\" / \"scheduler\";\n    auto sched_string = read_first_line(sched_file);\n    auto beg = sched_string.find('[');\n    size_t len = sched_string.size();\n    if (beg == sstring::npos) {\n        beg = 0;\n    } else {\n        auto end = sched_string.find(']');\n        if (end != sstring::npos) {\n            len = end - beg - 1;\n        }\n        beg++;\n    }\n    auto scheduler = sched_string.substr(beg, len);\n    if ((scheduler != \"noop\") && (scheduler != \"none\")) {\n        iotune_logger.warn(\"Scheduler for {} set to {}. It is recommend to set it to noop before evaluation so as not to skew the results.\",\n                sched_file.string(), scheduler);\n    }\n\n    auto nomerges_file = dev_sys_file / \"queue\" / \"nomerges\";\n    auto nomerges = read_first_line_as<unsigned>(nomerges_file);\n    if (nomerges != 2u) {\n        iotune_logger.warn(\"nomerges for {} set to {}. It is recommend to set it to 2 before evaluation so that merges are disabled. Results can be skewed otherwise.\",\n                nomerges_file.string(), nomerges);\n    }\n\n    auto write_cache_file = dev_sys_file / \"queue\" / \"write_cache\";\n    auto write_cache = read_first_line_as<std::string>(write_cache_file);\n    if (write_cache == \"write back\") {\n        iotune_logger.warn(\"write_cache for {} is set to write back. Some disks have poor implementation of this mode, pay attention to the measurements accuracy.\",\n                write_cache_file.string());\n    }\n}\n\nstruct evaluation_directory {\n    sstring _name;\n    // We know that if we issue more than this, they will be blocked on linux anyway.\n    unsigned _max_iodepth = 0;\n    unsigned _force_io_depth;\n    uint64_t _available_space;\n    uint64_t _min_data_transfer_size = 512;\n    unsigned _disks_per_array = 0;\n    std::optional<uint64_t> _reported_physical_block_size;\n\n    void scan_device(unsigned dev_maj, unsigned dev_min) {\n        scan_device(fmt::format(\"{}:{}\", dev_maj, dev_min));\n    }\n\n    void scan_device(std::string dev_str) {\n        scan_device(fs::path(\"/sys/dev/block\") / dev_str);\n    }\n\n    void scan_device(fs::path sys_file) {\n        try {\n            sys_file = fs::canonical(sys_file);\n            bool is_leaf = true;\n            if (fs::exists(sys_file / \"slaves\")) {\n                for (auto& dev : fs::directory_iterator(sys_file / \"slaves\")) {\n                    is_leaf = false;\n                    scan_device(read_first_line(dev.path() / \"dev\"));\n                }\n            }\n\n            // our work is done if not leaf. We'll tune the leaves\n            if (!is_leaf) {\n                return;\n            }\n\n            if (fs::exists(sys_file / \"partition\")) {\n                scan_device(sys_file.remove_filename());\n            } else {\n                check_device_properties(sys_file);\n                auto queue_dir = sys_file / \"queue\";\n                auto disk_min_io_size = read_first_line_as<uint64_t>(queue_dir / \"minimum_io_size\");\n\n                _min_data_transfer_size = std::max(_min_data_transfer_size, disk_min_io_size);\n                _max_iodepth += read_first_line_as<uint64_t>(queue_dir / \"nr_requests\");\n\n                // Read physical_block_size from sysfs for comparison with detected value\n                if (!_reported_physical_block_size) {\n                    auto pbs_file = queue_dir / \"physical_block_size\";\n                    if (fs::exists(pbs_file)) {\n                        _reported_physical_block_size = read_first_line_as<uint64_t>(pbs_file);\n                    }\n                }\n\n                _disks_per_array++;\n            }\n        } catch (std::system_error& se) {\n            iotune_logger.error(\"Error while parsing sysfs. Will continue with guessed values: {}\", se.what());\n            _max_iodepth = 128;\n        }\n        if (_force_io_depth != 0) {\n            _max_iodepth = _force_io_depth;\n        }\n        _disks_per_array = std::max(_disks_per_array, 1u);\n    }\npublic:\n    evaluation_directory(sstring name, unsigned force_io_depth)\n        : _name(name)\n        , _force_io_depth(force_io_depth)\n        , _available_space(fs::space(fs::path(_name)).available)\n    {}\n\n    unsigned max_iodepth() const {\n        return _max_iodepth;\n    }\n\n    fs::path path() const {\n        return fs::path(_name);\n    }\n\n    const sstring& name() const {\n        return _name;\n    }\n\n    unsigned disks_per_array() const {\n        return _disks_per_array;\n    }\n\n    uint64_t minimum_io_size() const {\n        return _min_data_transfer_size;\n    }\n\n    std::optional<uint64_t> reported_physical_block_size() const {\n        return _reported_physical_block_size;\n    }\n\n    future<> discover_directory() {\n        return seastar::async([this] {\n            auto f = open_directory(_name).get();\n            auto st = f.stat().get();\n            f.close().get();\n\n            scan_device(major(st.st_dev), minor(st.st_dev));\n        });\n    }\n\n    uint64_t available_space() const {\n        return _available_space;\n    }\n};\n\nstruct io_rates {\n    float bytes_per_sec = 0;\n    float iops = 0;\n    io_rates operator+(const io_rates& a) const {\n        return io_rates{bytes_per_sec + a.bytes_per_sec, iops + a.iops};\n    }\n\n    io_rates& operator+=(const io_rates& a) {\n        bytes_per_sec += a.bytes_per_sec;\n        iops += a.iops;\n        return *this;\n    }\n};\n\nstruct row_stats {\n    size_t points;\n    double average;\n    double stdev;\n\n    float stdev_percents() const {\n        return points > 0 ? stdev / average : 0.0;\n    }\n};\n\ntemplate <typename T>\nstatic row_stats get_row_stats_for(const std::vector<T>& v) {\n    if (v.size() == 0) {\n        return row_stats{0, 0.0, 0.0};\n    }\n\n    double avg = std::accumulate(v.begin(), v.end(), 0.0) / v.size();\n    double stdev = std::sqrt(std::transform_reduce(v.begin(), v.end(), 0.0,\n                std::plus<double>(), [avg] (auto& v) -> double { return (v - avg) * (v - avg); }) / v.size());\n\n    return row_stats{ v.size(), avg, stdev };\n}\n\nclass invalid_position : public std::exception {\npublic:\n    virtual const char* what() const noexcept {\n        return \"file access position invalid\";\n    }\n};\n\nstruct position_generator {\n    virtual uint64_t get_pos() = 0;\n    virtual bool is_sequential() const = 0;\n    virtual ~position_generator() {}\n};\n\nclass sequential_issuer : public position_generator {\n    size_t _buffer_size;\n    uint64_t _position = 0;\n    uint64_t _size_limit;\npublic:\n    sequential_issuer(size_t buffer_size, uint64_t size_limit)\n        : _buffer_size(buffer_size)\n        , _size_limit(size_limit)\n    {}\n\n    virtual bool is_sequential() const {\n        return true;\n    }\n\n    virtual uint64_t get_pos() {\n        if (_position >= _size_limit) {\n            // Wrap around if reaching EOF. The write bandwidth is lower,\n            // and we also split the write bandwidth among shards, while we\n            // read only from shard 0, so shard 0's file may not be large\n            // enough to read from.\n            _position = 0;\n        }\n        auto pos = _position;\n        _position += _buffer_size;\n        return pos;\n    }\n};\n\nclass random_issuer : public position_generator {\n    size_t _buffer_size;\n    uint64_t _last_position;\n    std::uniform_int_distribution<uint64_t> _pos_distribution;\npublic:\n    random_issuer(size_t buffer_size, uint64_t last_position)\n        : _buffer_size(buffer_size)\n        , _last_position(last_position)\n        , _pos_distribution(0, (last_position / buffer_size) - 1)\n    {}\n\n    virtual bool is_sequential() const {\n        return false;\n    }\n\n    virtual uint64_t get_pos() {\n        uint64_t pos = _pos_distribution(random_generator) * _buffer_size;\n        if (pos >= _last_position) {\n            throw invalid_position();\n        }\n        return pos;\n    }\n};\n\nclass request_issuer {\npublic:\n    virtual future<size_t> issue_request(uint64_t pos, char* buf, uint64_t size) = 0;\n    virtual ~request_issuer() {}\n};\n\n\nclass write_request_issuer : public request_issuer {\n    file _file;\npublic:\n    explicit write_request_issuer(file f) : _file(f) {}\n    future<size_t> issue_request(uint64_t pos, char* buf, uint64_t size) override {\n        return _file.dma_write(pos, buf, size);\n    }\n};\n\nclass read_request_issuer : public request_issuer {\n    file _file;\npublic:\n    explicit read_request_issuer(file f) : _file(f) {}\n    future<size_t> issue_request(uint64_t pos, char* buf, uint64_t size) override {\n        return _file.dma_read(pos, buf, size);\n    }\n};\n\nstd::chrono::duration<double>\nwarmup_period(std::chrono::duration<double> duration) {\n    auto min_warmup = 3s;\n    auto five_percent = duration * 0.05;\n    return std::max(min_warmup, std::chrono::duration_cast<std::chrono::seconds>(five_percent));\n}\n\nclass io_worker {\n    class requests_rate_meter {\n        std::vector<unsigned> _rates;\n        // Passed in rates might be reused so we locally track in our own vector\n        // (such that we can modify that list) and append later.\n        std::vector<unsigned>& _parent_rates;\n        const unsigned& _requests;\n        unsigned _prev_requests = 0;\n        std::chrono::duration<double> _duration;\n        timer<> _tick;\n        bool _with_warmup;\n\n        static constexpr auto period = 1s;\n\n    public:\n        requests_rate_meter(std::chrono::duration<double> duration, std::vector<unsigned>& rates, const unsigned& requests, bool with_warmup)\n            : _rates()\n            , _parent_rates(rates)\n            , _requests(requests)\n            , _duration(duration)\n            , _tick([this] {\n                _rates.push_back(_requests - _prev_requests);\n                _prev_requests = _requests;\n            })\n            , _with_warmup(with_warmup)\n        {\n            _rates.reserve(256); // ~2 minutes\n            if (duration > 4 * period) {\n                _tick.arm_periodic(period);\n            }\n        }\n\n        ~requests_rate_meter() {\n            if (_tick.armed()) {\n                _tick.cancel();\n            } else {\n                _rates.push_back(_requests);\n            }\n\n            // Drop samples that got measured during the warm up period.\n            // We drop an extra sample. This is because the one second timer\n            // logic is really independent of when we start counting requests in\n            // `issue_request` itself so the first bucket might not be a full\n            // measurement and cause skew otherwise.\n            size_t samples_to_drop = _with_warmup ? warmup_period(_duration) / period + 1 : 0;\n            _rates.erase(_rates.begin(), _rates.begin() + std::min(samples_to_drop, _rates.size()));\n            _parent_rates.insert(_parent_rates.end(), _rates.begin(), _rates.end());\n        }\n    };\n\n    uint64_t _bytes = 0;\n    uint64_t _max_offset = 0;\n    unsigned _requests = 0;\n    size_t _buffer_size;\n    std::chrono::time_point<iotune_clock, std::chrono::duration<double>> _start_measuring;\n    std::chrono::time_point<iotune_clock, std::chrono::duration<double>> _end_measuring;\n    std::chrono::time_point<iotune_clock, std::chrono::duration<double>> _end_load;\n    // track separately because in the sequential case we may exhaust the file before _duration\n    std::chrono::time_point<iotune_clock, std::chrono::duration<double>> _last_time_seen;\n\n    requests_rate_meter _rr_meter;\n    std::unique_ptr<position_generator> _pos_impl;\n    std::unique_ptr<request_issuer> _req_impl;\npublic:\n    bool is_sequential() const {\n        return _pos_impl->is_sequential();\n    }\n\n    bool should_stop() const {\n        return iotune_clock::now() >= _end_load;\n    }\n\n    io_worker(size_t buffer_size, std::chrono::duration<double> duration, std::unique_ptr<request_issuer> reqs, std::unique_ptr<position_generator> pos, std::vector<unsigned>& rates, bool with_warmup = true)\n        : _buffer_size(buffer_size)\n        , _start_measuring(iotune_clock::now() + (with_warmup ? std::chrono::duration<double>(warmup_period(duration)) : 0s))\n        , _end_measuring(_start_measuring + duration)\n        , _end_load(_end_measuring + 10ms)\n        , _last_time_seen(_start_measuring)\n        , _rr_meter(duration, rates, _requests, with_warmup)\n        , _pos_impl(std::move(pos))\n        , _req_impl(std::move(reqs))\n    {}\n\n    std::unique_ptr<char[], free_deleter> get_buffer() {\n        return allocate_aligned_buffer<char>(_buffer_size, _buffer_size);\n    }\n\n    future<> issue_request(char* buf) {\n        uint64_t pos = _pos_impl->get_pos();\n        return _req_impl->issue_request(pos, buf, _buffer_size).then([this, pos] (size_t size) {\n            auto now = iotune_clock::now();\n            _max_offset = std::max(_max_offset, pos + size);\n            if ((now > _start_measuring) && (now < _end_measuring)) {\n                _last_time_seen = now;\n                _bytes += size;\n                _requests++;\n            }\n        });\n    }\n\n    uint64_t max_offset() const noexcept { return _max_offset; }\n\n    io_rates get_io_rates() const {\n        io_rates rates;\n        auto t = _last_time_seen - _start_measuring;\n        if (!t.count()) {\n            throw std::runtime_error(\"No data collected\");\n        }\n        rates.bytes_per_sec = _bytes / t.count();\n        rates.iops = _requests / t.count();\n        return rates;\n    }\n};\n\nclass test_file {\npublic:\n    enum class pattern { sequential, random };\n    enum class access_type { read, write };\nprivate:\n    fs::path _dirpath;\n    uint64_t _file_size;\n    file _file;\n    uint64_t _forced_random_write_io_buffer_size;\n    uint64_t _forced_random_read_io_buffer_size;\n\n    std::unique_ptr<position_generator> get_position_generator(size_t buffer_size, pattern access_pattern) {\n        if (access_pattern == pattern::sequential) {\n            return std::make_unique<sequential_issuer>(buffer_size, _file_size);\n        } else {\n            return std::make_unique<random_issuer>(buffer_size, _file_size);\n        }\n    }\n\n    uint64_t calculate_buffer_size(pattern access_pattern, uint64_t buffer_size, uint64_t operation_alignment, access_type io_type) const {\n        if (access_pattern == pattern::random) {\n            if(io_type == access_type::write && _forced_random_write_io_buffer_size != 0u) {\n                return _forced_random_write_io_buffer_size;\n            }\n            if(io_type == access_type::read && _forced_random_read_io_buffer_size != 0u) {\n                return _forced_random_read_io_buffer_size;\n            }\n        }\n\n        return std::max(buffer_size, operation_alignment);\n    }\n\npublic:\n    test_file(const ::evaluation_directory& dir, uint64_t maximum_size, uint64_t random_write_io_buffer_size, uint64_t random_read_io_buffer_size)\n        : _dirpath(dir.path() / fs::path(fmt::format(\"ioqueue-discovery-{}\", this_shard_id())))\n        , _file_size(maximum_size)\n        , _forced_random_write_io_buffer_size(random_write_io_buffer_size)\n        , _forced_random_read_io_buffer_size(random_read_io_buffer_size)\n    {}\n\n    future<> create_data_file() {\n        // XFS likes access in many directories better.\n        return make_directory(_dirpath.string()).then([this] {\n            auto testfile = _dirpath / fs::path(\"testfile\");\n            file_open_options options;\n            options.extent_allocation_size_hint = _file_size;\n            return open_file_dma(testfile.string(), open_flags::rw | open_flags::create, std::move(options)).then([this, testfile] (file file) {\n                _file = file;\n                if (this_shard_id() == 0) {\n                    iotune_logger.info(\"Filesystem parameters: read alignment {}, write alignment {}\", _file.disk_read_dma_alignment(), _file.disk_write_dma_alignment());\n                }\n                return remove_file(testfile.string()).then([this] {\n                    return remove_file(_dirpath.string());\n                });\n            }).then([this] {\n                return _file.truncate(_file_size);\n            });\n        });\n    }\n\n    future<io_rates> do_workload(std::unique_ptr<io_worker> worker_ptr, unsigned max_os_concurrency, bool update_file_size = false) {\n        if (update_file_size) {\n            _file_size = 0;\n        }\n\n        auto worker = worker_ptr.get();\n        auto concurrency = std::views::iota(0u, max_os_concurrency);\n        return parallel_for_each(std::move(concurrency), [worker] (unsigned idx) {\n            auto bufptr = worker->get_buffer();\n            auto buf = bufptr.get();\n            return do_until([worker] { return worker->should_stop(); }, [buf, worker] {\n                return worker->issue_request(buf);\n            }).finally([alive = std::move(bufptr)] {});\n        }).then_wrapped([this, worker = std::move(worker_ptr), update_file_size] (future<> f) {\n            try {\n                f.get();\n            } catch (invalid_position& ip) {\n                // expected if sequential. Example: reading and the file ended.\n                if (!worker->is_sequential()) {\n                    throw;\n                }\n            }\n\n            if (update_file_size) {\n                _file_size = worker->max_offset();\n            }\n            return make_ready_future<io_rates>(worker->get_io_rates());\n        });\n    }\n\n    future<io_rates> read_workload(size_t buffer_size, pattern access_pattern, unsigned max_os_concurrency, std::chrono::duration<double> duration, std::vector<unsigned>& rates, bool _ = true) {\n        buffer_size = calculate_buffer_size(access_pattern, buffer_size, _file.disk_read_dma_alignment(), access_type::read);\n        auto worker = std::make_unique<io_worker>(buffer_size, duration, std::make_unique<read_request_issuer>(_file), get_position_generator(buffer_size, access_pattern), rates);\n        return do_workload(std::move(worker), max_os_concurrency);\n    }\n\n    future<io_rates> write_workload(size_t buffer_size, pattern access_pattern, unsigned max_os_concurrency, std::chrono::duration<double> duration, std::vector<unsigned>& rates, bool with_warmup = true) {\n        buffer_size = calculate_buffer_size(access_pattern, buffer_size, _file.disk_write_dma_alignment(), access_type::write);\n        auto worker = std::make_unique<io_worker>(buffer_size, duration, std::make_unique<write_request_issuer>(_file), get_position_generator(buffer_size, access_pattern), rates, with_warmup);\n        bool update_file_size = worker->is_sequential();\n        return do_workload(std::move(worker), max_os_concurrency, update_file_size).then([this] (io_rates r) {\n            return _file.flush().then([r = std::move(r)] () mutable {\n                return make_ready_future<io_rates>(std::move(r));\n            });\n        });\n    }\n\n    future<> stop() {\n        return _file ? _file.close() : make_ready_future<>();\n    }\n};\n\nclass iotune_multi_shard_context {\n    ::evaluation_directory _test_directory;\n    uint64_t _random_write_io_buffer_size;\n    uint64_t _random_read_io_buffer_size;\n\n    unsigned per_shard_io_depth() const {\n        auto iodepth = _test_directory.max_iodepth() / smp::count;\n        if (this_shard_id() < _test_directory.max_iodepth() % smp::count) {\n            iodepth++;\n        }\n        return std::min(iodepth, 128u);\n    }\n    seastar::sharded<test_file> _iotune_test_file;\n\n    std::vector<unsigned> serial_rates;\n    seastar::sharded<std::vector<unsigned>> sharded_rates;\n\npublic:\n    future<> stop() {\n        return _iotune_test_file.stop().then([this] { return sharded_rates.stop(); });\n    }\n\n    future<> start() {\n       const auto maximum_size = (_test_directory.available_space() / (2 * smp::count));\n       return _iotune_test_file.start(_test_directory, maximum_size, _random_write_io_buffer_size, _random_read_io_buffer_size).then([this] {\n           return sharded_rates.start();\n       });\n    }\n\n    future<row_stats> get_serial_rates() {\n        row_stats ret = get_row_stats_for<unsigned>(serial_rates);\n        serial_rates.clear();\n        return make_ready_future<row_stats>(ret);\n    }\n\n    future<row_stats> get_sharded_worst_rates() {\n        return sharded_rates.map_reduce0([] (std::vector<unsigned>& rates) {\n            row_stats ret = get_row_stats_for<unsigned>(rates);\n            rates.clear();\n            return ret;\n        }, row_stats{0, 0.0, 0.0},\n        [] (const row_stats& res, row_stats lres) {\n            return res.stdev < lres.stdev ? lres : res;\n        });\n    }\n\n    future<> create_data_file() {\n        return _iotune_test_file.invoke_on_all([] (test_file& tf) {\n            return tf.create_data_file();\n        });\n    }\n\n    future<io_rates> write_sequential_data(unsigned shard, size_t buffer_size, std::chrono::duration<double> duration) {\n        return _iotune_test_file.invoke_on(shard, [this, buffer_size, duration] (test_file& tf) {\n            return tf.write_workload(buffer_size, test_file::pattern::sequential, 4 * _test_directory.disks_per_array(), duration, serial_rates, false);\n        });\n    }\n\n    future<> warm_up_sequential_data(size_t buffer_size, std::chrono::duration<double> duration) {\n        return _iotune_test_file.invoke_on_all([this, buffer_size, duration] (test_file& tf) {\n            return tf.write_workload(buffer_size, test_file::pattern::sequential, 4 * _test_directory.disks_per_array(), duration, sharded_rates.local(), false).then([this] (io_rates r) {\n                sharded_rates.local().clear();\n                return make_ready_future<>();\n            });\n        });\n    }\n\n    future<io_rates> read_sequential_data(unsigned shard, size_t buffer_size, std::chrono::duration<double> duration) {\n        return _iotune_test_file.invoke_on(shard, [this, buffer_size, duration] (test_file& tf) {\n            return tf.read_workload(buffer_size, test_file::pattern::sequential, 4 * _test_directory.disks_per_array(), duration, serial_rates);\n        });\n    }\n\n    future<io_rates> write_random_data(size_t buffer_size, std::chrono::duration<double> duration) {\n        return _iotune_test_file.map_reduce0([buffer_size, this, duration] (test_file& tf) {\n            const auto shard_io_depth = per_shard_io_depth();\n            if (shard_io_depth == 0) {\n                return make_ready_future<io_rates>();\n            } else {\n                return tf.write_workload(buffer_size, test_file::pattern::random, shard_io_depth, duration, sharded_rates.local());\n            }\n        }, io_rates(), std::plus<io_rates>());\n    }\n\n    future<io_rates> read_random_data(size_t buffer_size, std::chrono::duration<double> duration) {\n        return _iotune_test_file.map_reduce0([buffer_size, this, duration] (test_file& tf) {\n            const auto shard_io_depth = per_shard_io_depth();\n            if (shard_io_depth == 0) {\n                return make_ready_future<io_rates>();\n            } else {\n                return tf.read_workload(buffer_size, test_file::pattern::random, shard_io_depth, duration, sharded_rates.local());\n            }\n        }, io_rates(), std::plus<io_rates>());\n    }\n\nprivate:\n    template <typename Fn>\n    future<uint64_t> saturate(float rate_threshold, size_t buffer_size, std::chrono::duration<double> duration, Fn&& workload) {\n        return _iotune_test_file.invoke_on(0, [this, rate_threshold, buffer_size, duration, workload] (test_file& tf) {\n            return (tf.*workload)(buffer_size, test_file::pattern::sequential, 1, duration, serial_rates, true).then([this, rate_threshold, buffer_size, duration, workload] (io_rates rates) {\n                serial_rates.clear();\n                if (rates.bytes_per_sec < rate_threshold) {\n                    // The throughput with the given buffer-size is already \"small enough\", so\n                    // return back its previous value\n                    return make_ready_future<uint64_t>(buffer_size * 2);\n                } else {\n                    return saturate(rate_threshold, buffer_size / 2, duration, workload);\n                }\n            });\n        });\n    }\n\npublic:\n    future<uint64_t> saturate_write(float rate_threshold, size_t buffer_size, std::chrono::duration<double> duration) {\n        return saturate(rate_threshold, buffer_size, duration, &test_file::write_workload);\n    }\n\n    future<uint64_t> saturate_read(float rate_threshold, size_t buffer_size, std::chrono::duration<double> duration) {\n        return saturate(rate_threshold, buffer_size, duration, &test_file::read_workload);\n    }\n\n    // Detect physical block size by testing write performance at different alignments.\n    // Returns the smallest alignment where write performance is good (no RMW penalty).\n    // Tests alignments from 512 bytes up to 8192 bytes.\n    //\n    // The detection works by measuring write IOPS at each alignment. If writes smaller\n    // than the physical block size are issued, the disk must perform read-modify-write\n    // (RMW) operations, which reduces IOPS. Once we reach the physical block size,\n    // IOPS should plateau as larger alignments don't provide additional benefit.\n    future<uint64_t> detect_physical_block_size(std::chrono::duration<double> duration, std::optional<uint64_t> reported_pbs = std::nullopt) {\n        // Test alignments: 512, 1024, 2048, 4096, 8192\n        // This covers most common physical block sizes\n        std::vector<uint64_t> test_alignments = {512, 1024, 2048, 4096, 8192};\n        std::map<uint64_t, float> alignment_to_iops;\n\n        iotune_logger.info(\"Detecting physical block size by testing write performance at different alignments\");\n\n        for (auto alignment : test_alignments) {\n            // Perform a short write test at this alignment (5% of total test duration)\n            auto rates = co_await write_random_data(alignment, duration * 0.05);\n            alignment_to_iops[alignment] = rates.iops;\n\n            iotune_logger.info(\"Alignment {} bytes: {} IOPS\", alignment, uint64_t(rates.iops));\n\n            // Clear rates after each test\n            co_await sharded_rates.invoke_on_all([] (std::vector<unsigned>& rates) {\n                rates.clear();\n                return make_ready_future<>();\n            });\n        }\n\n        // Find the smallest alignment where performance plateaus.\n        // We look for the point where increasing alignment doesn't significantly improve IOPS.\n        // This indicates we've reached or exceeded the physical block size.\n        //\n        // Algorithm: Find the first alignment where the next larger alignment doesn't\n        // improve IOPS by more than 5%. This is the physical block size.\n        uint64_t detected_size = test_alignments.back(); // Default to largest if no plateau found\n        float max_iops = 0;\n\n        for (size_t i = 0; i < test_alignments.size() - 1; ++i) {\n            auto current_alignment = test_alignments[i];\n            auto next_alignment = test_alignments[i + 1];\n            auto current_iops = alignment_to_iops[current_alignment];\n            auto next_iops = alignment_to_iops[next_alignment];\n\n            max_iops = std::max(max_iops, current_iops);\n\n            // If the next larger alignment doesn't improve IOPS by more than 5%,\n            // we've found the physical block size\n            if (next_iops <= current_iops * 1.05) {\n                detected_size = current_alignment;\n                iotune_logger.info(\"Performance plateau detected at {} bytes\", detected_size);\n                break;\n            }\n        }\n\n        // If we didn't find a plateau, use the alignment with the best IOPS\n        if (detected_size == test_alignments.back()) {\n            for (const auto& [alignment, iops] : alignment_to_iops) {\n                if (iops >= max_iops * 0.95) {  // Within 5% of max\n                    detected_size = std::min(detected_size, alignment);\n                }\n            }\n            iotune_logger.info(\"No clear plateau, using alignment with best performance: {} bytes\", detected_size);\n        }\n\n        iotune_logger.info(\"Detected physical block size: {} bytes\", detected_size);\n\n        // Compare with driver-reported value if available\n        if (reported_pbs) {\n            if (detected_size == *reported_pbs) {\n                iotune_logger.info(\"Detected value matches driver-reported value ({} bytes)\", *reported_pbs);\n            } else {\n                iotune_logger.warn(\"Detected value differs from driver-reported value ({} bytes)\", *reported_pbs);\n                iotune_logger.warn(\"The disk may be lying about its physical block size\");\n                iotune_logger.warn(\"Using empirically detected value {} bytes to avoid write amplification\", detected_size);\n            }\n        }\n\n        co_return detected_size;\n    }\n\n    iotune_multi_shard_context(::evaluation_directory dir, uint64_t random_write_io_buffer_size, uint64_t random_read_io_buffer_size)\n        : _test_directory(dir)\n        , _random_write_io_buffer_size(random_write_io_buffer_size)\n        , _random_read_io_buffer_size(random_read_io_buffer_size)\n    {}\n};\n\nstruct disk_descriptor {\n    std::string mountpoint;\n    uint64_t read_iops;\n    uint64_t read_bw;\n    uint64_t write_iops;\n    uint64_t write_bw;\n    std::optional<uint64_t> read_sat_len;\n    std::optional<uint64_t> write_sat_len;\n    std::optional<uint64_t> physical_block_size;\n};\n\nvoid string_to_file(sstring conf_file, sstring buf) {\n    auto f = file_desc::open(conf_file, O_WRONLY | O_CLOEXEC | O_CREAT | O_TRUNC, 0664);\n    auto ret = f.write(buf.data(), buf.size());\n    if (!ret || (*ret != buf.size())) {\n        throw std::runtime_error(fmt::format(\"Can't write {}: {}\", conf_file, *ret));\n    }\n}\n\nvoid write_configuration_file(sstring conf_file, std::string format, sstring properties_file) {\n    sstring buf;\n    if (format == \"seastar\") {\n        buf = fmt::format(\"io-properties-file={}\\n\", properties_file);\n    } else {\n        buf = fmt::format(\"SEASTAR_IO=\\\"--io-properties-file={}\\\"\\n\", properties_file);\n    }\n    string_to_file(conf_file, buf);\n}\n\nvoid write_property_file(sstring conf_file, std::vector<disk_descriptor> disk_descriptors) {\n    YAML::Emitter out;\n    out << YAML::BeginMap;\n    out << YAML::Key << \"disks\";\n    out << YAML::BeginSeq;\n    for (auto& desc : disk_descriptors) {\n        out << YAML::BeginMap;\n        out << YAML::Key << \"mountpoint\" << YAML::Value << desc.mountpoint;\n        out << YAML::Key << \"read_iops\" << YAML::Value << desc.read_iops;\n        out << YAML::Key << \"read_bandwidth\" << YAML::Value << desc.read_bw;\n        out << YAML::Key << \"write_iops\" << YAML::Value << desc.write_iops;\n        out << YAML::Key << \"write_bandwidth\" << YAML::Value << desc.write_bw;\n        if (desc.read_sat_len) {\n            out << YAML::Key << \"read_saturation_length\" << YAML::Value << *desc.read_sat_len;\n        }\n        if (desc.write_sat_len) {\n            out << YAML::Key << \"write_saturation_length\" << YAML::Value << *desc.write_sat_len;\n        }\n        if (desc.physical_block_size) {\n            out << YAML::Key << \"physical_block_size\" << YAML::Value << *desc.physical_block_size;\n        }\n        out << YAML::EndMap;\n    }\n    out << YAML::EndSeq;\n    out << YAML::EndMap;\n    out << YAML::Newline;\n\n    string_to_file(conf_file, sstring(out.c_str(), out.size()));\n}\n\n// Returns the mountpoint of a path. It works by walking backwards from the canonical path\n// (absolute, with symlinks resolved), until we find a point that crosses a device ID.\nfs::path mountpoint_of(sstring filename) {\n    fs::path mnt_candidate = fs::canonical(fs::path(filename));\n    std::optional<dev_t> candidate_id = {};\n    auto current = mnt_candidate;\n    do {\n        auto f = open_directory(current.string()).get();\n        auto st = f.stat().get();\n        if ((candidate_id) && (*candidate_id != st.st_dev)) {\n            return mnt_candidate;\n        }\n        mnt_candidate = current;\n        candidate_id = st.st_dev;\n        current = current.parent_path();\n    } while (mnt_candidate != current);\n\n    return mnt_candidate;\n}\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n    bool fs_check = false;\n\n    app_template::config app_cfg;\n    app_cfg.name = \"IOTune\";\n\n    app_template app(std::move(app_cfg));\n    auto opt_add = app.add_options();\n    opt_add\n        (\"evaluation-directory\", bpo::value<std::vector<sstring>>()->required(), \"directory where to execute the evaluation\")\n        (\"properties-file\", bpo::value<sstring>(), \"path in which to write the YAML file\")\n        (\"options-file\", bpo::value<sstring>(), \"path in which to write the legacy conf file\")\n        (\"duration\", bpo::value<unsigned>()->default_value(120), \"time, in seconds, for which to run the test\")\n        (\"format\", bpo::value<sstring>()->default_value(\"seastar\"), \"Configuration file format (seastar | envfile)\")\n        (\"fs-check\", bpo::bool_switch(&fs_check), \"perform FS check only\")\n        (\"detect-physical-block-size\", bpo::bool_switch(), \"detect physical block size (off by default; can be slow)\")\n        (\"accuracy\", bpo::value<unsigned>()->default_value(3), \"acceptable deviation of measurements (percents)\")\n        (\"saturation\", bpo::value<sstring>()->default_value(\"\"), \"measure saturation lengths (read | write | both) (this is very slow!)\")\n        (\"random-write-io-buffer-size\", bpo::value<unsigned>()->default_value(0), \"force buffer size for random write\")\n        (\"random-read-io-buffer-size\", bpo::value<unsigned>()->default_value(0), \"force buffer size for random read\")\n        (\"force-io-depth\", bpo::value<unsigned>()->default_value(0), \"force io depth to a certain size (overriding auto detection logic)\")\n        (\"get-best-iops-with-buffer-sizes\", bpo::value<std::vector<uint64_t>>()->default_value(std::vector<uint64_t>{}, \"\"), \"run IOPS tests with buffer sizes passed as arg and choose the best combination\")\n        (\"sequential_read_buffer_size\", bpo::value<uint64_t>()->default_value(1 << 20), \"Set the sequential buffer size used for read bandwidth calculation. Default is 1MiB\")\n        (\"sequential_write_buffer_size\", bpo::value<uint64_t>()->default_value(1 << 20), \"Set the sequential buffer size used for write bandwidth calculation. Default is 1MiB\")\n    ;\n\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            auto& configuration = app.configuration();\n            auto eval_dirs = configuration[\"evaluation-directory\"].as<std::vector<sstring>>();\n            auto format = configuration[\"format\"].as<sstring>();\n            auto duration = std::chrono::duration<double>(configuration[\"duration\"].as<unsigned>() * 1s);\n            auto accuracy = configuration[\"accuracy\"].as<unsigned>();\n            auto saturation = configuration[\"saturation\"].as<sstring>();\n            auto random_write_io_buffer_size = configuration[\"random-write-io-buffer-size\"].as<unsigned>();\n            auto random_read_io_buffer_size = configuration[\"random-read-io-buffer-size\"].as<unsigned>();\n            auto force_io_depth = configuration[\"force-io-depth\"].as<unsigned>();\n            auto buffer_sizes_for_best_iops = configuration[\"get-best-iops-with-buffer-sizes\"].as<std::vector<uint64_t>>();\n            auto detect_pbs = configuration[\"detect-physical-block-size\"].as<bool>();\n            if (!buffer_sizes_for_best_iops.empty() && (random_read_io_buffer_size || random_write_io_buffer_size)) {\n                iotune_logger.error(\"--get-best-iops-with-buffer-sizes is set, but random-read-io-buffer-size or random-write-io-buffer-size are also set. \"\n                                    \"The options are mutually exclusive, please unset one of them.\");\n                return 1;\n            }\n            auto sequential_read_buffer_size = configuration[\"sequential_read_buffer_size\"].as<uint64_t>();\n            auto sequential_write_buffer_size = configuration[\"sequential_write_buffer_size\"].as<uint64_t>();\n\n            bool read_saturation, write_saturation;\n            if (saturation == \"\") {\n                read_saturation = false;\n                write_saturation = false;\n            } else if (saturation == \"both\") {\n                read_saturation = true;\n                write_saturation = true;\n            } else if (saturation == \"read\") {\n                read_saturation = true;\n                write_saturation = false;\n            } else if (saturation == \"write\") {\n                read_saturation = false;\n                write_saturation = true;\n            } else {\n                fmt::print(\"Bad --saturation value\\n\");\n                return 1;\n            }\n\n            std::vector<disk_descriptor> disk_descriptors;\n            std::unordered_map<sstring, sstring> mountpoint_map;\n            // We want to evaluate once per mountpoint, but we still want to write in one of the\n            // directories that we were provided - we may not have permissions to write into the\n            // mountpoint itself. If we are passed more than one directory per mountpoint, we don't\n            // really care to which one we write, so this simple hash will do.\n            for (auto& eval_dir : eval_dirs) {\n                mountpoint_map[mountpoint_of(eval_dir).string()] = eval_dir;\n            }\n            for (auto eval: mountpoint_map) {\n                auto mountpoint = eval.first;\n                auto eval_dir = eval.second;\n\n                if (!filesystem_has_good_aio_support(eval_dir, false)) {\n                    iotune_logger.error(\"Linux AIO is not supported by filesystem at {}\", eval_dir);\n                    return 1;\n                }\n\n                auto rec = 10000000000ULL;\n                auto avail = fs_avail(eval_dir).get();\n                if (avail < rec) {\n                    uint64_t val;\n                    const char* units;\n                    if (avail >= 1000000000) {\n                        val = (avail + 500000000) / 1000000000;\n                        units = \"GB\";\n                    } else if (avail >= 1000000) {\n                        val = (avail + 500000) / 1000000;\n                        units = \"MB\";\n                    } else {\n                        val = avail;\n                        units = \"bytes\";\n                    }\n                    iotune_logger.warn(\"Available space on filesystem at {}: {} {}: is less than recommended: {} GB\",\n                                       eval_dir, val, units, rec / 1000000000ULL);\n                }\n\n                iotune_logger.info(\"{} passed sanity checks\", eval_dir);\n                if (fs_check) {\n                    continue;\n                }\n\n                // Directory is the same object for all tests.\n                ::evaluation_directory test_directory(eval_dir, force_io_depth);\n                test_directory.discover_directory().get();\n                iotune_logger.info(\"Disk parameters: max_iodepth={} disks_per_array={} minimum_io_size={}\",\n                        test_directory.max_iodepth(), test_directory.disks_per_array(), test_directory.minimum_io_size());\n\n                if (test_directory.max_iodepth() < smp::count) {\n                    iotune_logger.warn(\"smp::count={} is greater than max_iodepth={} - shards above max_io_depth \"\n                                       \"will be ignored during random read and random write measurements\",\n                                       smp::count, test_directory.max_iodepth());\n                }\n\n                if (random_write_io_buffer_size != 0u) {\n                    iotune_logger.info(\"Forcing buffer_size={} for random write IO!\", random_write_io_buffer_size);\n                }\n                if (random_read_io_buffer_size != 0u) {\n                    iotune_logger.info(\"Forcing buffer_size={} for random read IO!\", random_read_io_buffer_size);\n                }\n\n                ::iotune_multi_shard_context iotune_tests(test_directory, random_write_io_buffer_size, random_read_io_buffer_size);\n                iotune_tests.start().get();\n                auto stop = defer([&iotune_tests] () noexcept {\n                    try {\n                        iotune_tests.stop().get();\n                    } catch (...) {\n                        fmt::print(\"Error occurred during iotune context shutdown: {}\", std::current_exception());\n                        abort();\n                    }\n                });\n\n                row_stats rates;\n                auto accuracy_msg = [accuracy, &rates] {\n                    auto stdev = rates.stdev_percents() * 100.0;\n                    return (accuracy == 0 || stdev > accuracy) ? fmt::format(\" (deviation {}%)\", int(round(stdev))) : std::string(\"\");\n                };\n\n                iotune_tests.create_data_file().get();\n\n                fmt::print(\"Warming up the disk for sequential write job...\\n\");\n                iotune_tests.warm_up_sequential_data(sequential_write_buffer_size, warmup_period(duration)).get();\n\n                fmt::print(\"Starting Evaluation. This may take a while...\\n\");\n                fmt::print(\"Measuring sequential write bandwidth: \");\n                std::cout.flush();\n                io_rates write_bw;\n                for (unsigned shard = 0; shard < smp::count; ++shard) {\n                    write_bw += iotune_tests.write_sequential_data(shard, sequential_write_buffer_size, duration * 0.70 / smp::count).get();\n                }\n                write_bw.bytes_per_sec /= smp::count;\n                rates = iotune_tests.get_serial_rates().get();\n                fmt::print(\"{} MiB/s{}\\n\", uint64_t(write_bw.bytes_per_sec / (1024 * 1024)), accuracy_msg());\n\n                std::optional<uint64_t> write_sat;\n\n                if (write_saturation) {\n                    fmt::print(\"Measuring write saturation length: \");\n                    std::cout.flush();\n                    write_sat = iotune_tests.saturate_write(write_bw.bytes_per_sec * (1.0 - rates.stdev_percents()), sequential_write_buffer_size/2, duration * 0.70).get();\n                    fmt::print(\"{}\\n\", *write_sat);\n                }\n\n                fmt::print(\"Measuring sequential read bandwidth: \");\n                std::cout.flush();\n                auto read_bw = iotune_tests.read_sequential_data(0, sequential_read_buffer_size, duration * 0.1).get();\n                rates = iotune_tests.get_serial_rates().get();\n                fmt::print(\"{} MiB/s{}\\n\", uint64_t(read_bw.bytes_per_sec / (1024 * 1024)), accuracy_msg());\n\n                std::optional<uint64_t> read_sat;\n\n                if (read_saturation) {\n                    fmt::print(\"Measuring read saturation length: \");\n                    std::cout.flush();\n                    read_sat = iotune_tests.saturate_read(read_bw.bytes_per_sec * (1.0 - rates.stdev_percents()), sequential_read_buffer_size/2, duration * 0.1).get();\n                    fmt::print(\"{}\\n\", *read_sat);\n                }\n\n                struct disk_descriptor desc;\n                desc.mountpoint = mountpoint;\n                desc.read_bw = read_bw.bytes_per_sec;\n                desc.read_sat_len = read_sat;\n                desc.write_bw = write_bw.bytes_per_sec;\n                desc.write_sat_len = write_sat;\n\n                if (buffer_sizes_for_best_iops.empty()) {\n                    buffer_sizes_for_best_iops = {test_directory.minimum_io_size()};\n                } else {\n                    fmt::print(\"Evaluating various buffer sizes for choosing the best IOPS: {}\\n\", buffer_sizes_for_best_iops);\n                    std::cout.flush();\n                }\n                io_rates best_write_iops, best_read_iops;\n                for (uint64_t buffer_size : buffer_sizes_for_best_iops) {\n                    fmt::print(\"Measuring random write IOPS: \");\n                    std::cout.flush();\n                    auto write_iops = iotune_tests.write_random_data(buffer_size, duration * 0.1).get();\n                    rates = iotune_tests.get_sharded_worst_rates().get();\n                    fmt::print(\"{} IOPS{}\\n\", uint64_t(write_iops.iops), accuracy_msg());\n\n                    fmt::print(\"Measuring random read IOPS: \");\n                    std::cout.flush();\n                    auto read_iops = iotune_tests.read_random_data(buffer_size, duration * 0.1).get();\n                    rates = iotune_tests.get_sharded_worst_rates().get();\n                    fmt::print(\"{} IOPS{}\\n\", uint64_t(read_iops.iops), accuracy_msg());\n\n                    best_write_iops.iops = std::max(best_write_iops.iops, write_iops.iops);\n                    best_read_iops.iops = std::max(best_read_iops.iops, read_iops.iops);\n                }\n\n                desc.read_iops = best_read_iops.iops;\n                desc.write_iops = best_write_iops.iops;\n\n                // Optionally detect physical block size (can be slow, so off by default)\n                // Only write physical_block_size if empirically detected, not kernel-reported value.\n                // Kernel-reported value (sysfs) is the same as ioctl(BLKPBSZGET), so writing it\n                // to io_properties.yaml would just create a circular override with no benefit.\n                // The override is only useful when the disk lies and we detect the real value.\n                if (detect_pbs) {\n                    fmt::print(\"Detecting physical block size: \");\n                    std::cout.flush();\n                    auto detected_pbs = iotune_tests.detect_physical_block_size(duration, test_directory.reported_physical_block_size()).get();\n                    fmt::print(\"{} bytes\\n\", detected_pbs);\n                    desc.physical_block_size = detected_pbs;\n                }\n\n                disk_descriptors.push_back(std::move(desc));\n            }\n\n            if (fs_check) {\n                return 0;\n            }\n\n            auto file = \"properties file\";\n            try {\n                if (configuration.count(\"properties-file\")) {\n                    fmt::print(\"Writing result to {}\\n\", configuration[\"properties-file\"].as<sstring>());\n                    write_property_file(configuration[\"properties-file\"].as<sstring>(), disk_descriptors);\n                }\n\n                file = \"configuration file\";\n                if (configuration.count(\"options-file\")) {\n                    fmt::print(\"Writing result to {}\\n\", configuration[\"options-file\"].as<sstring>());\n                    write_configuration_file(configuration[\"options-file\"].as<sstring>(), format, configuration[\"properties-file\"].as<sstring>());\n                }\n            } catch (...) {\n                iotune_logger.error(\"Exception when writing {}: {}.\\nPlease add the above values manually to your seastar command line.\", file, std::current_exception());\n                return 1;\n            }\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "apps/lib/stop_signal.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sharded.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/signal.hh>\n\n/// Seastar apps lib namespace\n\nnamespace seastar_apps_lib {\n\n\n/// \\brief Futurized SIGINT/SIGTERM signals handler class\n///\n/// Seastar-style helper class that allows easy waiting for SIGINT/SIGTERM signals\n/// from your app.\n///\n/// Example:\n/// \\code\n/// #include <seastar/apps/lib/stop_signal.hh>\n/// ...\n/// int main() {\n/// ...\n/// seastar::thread th([] {\n///    seastar_apps_lib::stop_signal stop_signal;\n///    <some code>\n///    stop_signal.wait().get();  // this will wait till we receive SIGINT or SIGTERM signal\n/// });\n/// \\endcode\nclass stop_signal {\n    bool _caught = false;\n    seastar::condition_variable _cond;\nprivate:\n    void signaled() {\n        if (_caught) {\n            return;\n        }\n        _caught = true;\n        _cond.broadcast();\n    }\npublic:\n    stop_signal() {\n        seastar::handle_signal(SIGINT, [this] { signaled(); });\n        seastar::handle_signal(SIGTERM, [this] { signaled(); });\n    }\n    ~stop_signal() {\n        // There's no way to unregister a handler yet, so register a no-op handler instead.\n        seastar::handle_signal(SIGINT, [] {});\n        seastar::handle_signal(SIGTERM, [] {});\n    }\n    seastar::future<> wait() {\n        return _cond.wait([this] { return _caught; });\n    }\n    bool stopping() const {\n        return _caught;\n    }\n};\n}"
  },
  {
    "path": "apps/memcached/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nset (Seastar_APP_MEMCACHED_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})\nset (Seastar_APP_MEMCACHED_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})\n\nseastar_generate_ragel (\n  TARGET app_memcached_ascii\n  VAR app_memcached_ascii_file\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/ascii.rl\n  OUT_FILE ${CMAKE_CURRENT_BINARY_DIR}/ascii.hh)\n\nseastar_add_app (memcached\n  SOURCES\n    ${app_memcached_ascii_file}\n    memcache.cc\n    memcached.hh)\n\ntarget_include_directories (app_memcached\n  PRIVATE ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_dependencies (app_memcached app_memcached_ascii)\n\n#\n# Tests.\n#\n\nif (Seastar_TESTING)\n  add_subdirectory (tests)\nendif ()\n"
  },
  {
    "path": "apps/memcached/ascii.rl",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/ragel.hh>\n#include \"memcached.hh\"\n#include <memory>\n#include <algorithm>\n#include <functional>\n\nusing namespace seastar;\n\n%%{\n\nmachine memcache_ascii_protocol;\n\naccess _fsm_;\n\naction mark {\n    g.mark_start(p);\n}\n\naction start_blob {\n    g.mark_start(p);\n    _size_left = _size;\n}\n\naction advance_blob {\n    auto len = std::min((uint32_t)(pe - p), _size_left);\n    _size_left -= len;\n    p += len;\n    if (_size_left == 0) {\n        _blob = str();\n        p--;\n        fret;\n    }\n    p--;\n}\n\ncrlf = '\\r\\n';\nsp = ' ';\nu32 = digit+ >{ _u32 = 0; } ${ _u32 *= 10; _u32 += fc - '0'; };\nu64 = digit+ >{ _u64 = 0; } ${ _u64 *= 10; _u64 += fc - '0'; };\nkey = [^ ]+ >mark %{ _key = memcache::item_key(str()); };\nflags = digit+ >mark %{ _flags_str = str(); };\nexpiration = u32 %{ _expiration = _u32; };\nsize = u32 >mark %{ _size = _u32; _size_str = str(); };\nblob := any+ >start_blob $advance_blob;\nmaybe_noreply = (sp \"noreply\" @{ _noreply = true; })? >{ _noreply = false; };\nmaybe_expiration = (sp expiration)? >{ _expiration = 0; };\nversion_field = u64 %{ _version = _u64; };\n\ninsertion_params = sp key sp flags sp expiration sp size maybe_noreply (crlf @{ fcall blob; } ) crlf;\nset = \"set\" insertion_params @{ _state = state::cmd_set; };\nadd = \"add\" insertion_params @{ _state = state::cmd_add; };\nreplace = \"replace\" insertion_params @{ _state = state::cmd_replace; };\ncas = \"cas\" sp key sp flags sp expiration sp size sp version_field maybe_noreply (crlf @{ fcall blob; } ) crlf @{ _state = state::cmd_cas; };\nget = \"get\" (sp key %{ _keys.emplace_back(std::move(_key)); })+ crlf @{ _state = state::cmd_get; };\ngets = \"gets\" (sp key %{ _keys.emplace_back(std::move(_key)); })+ crlf @{ _state = state::cmd_gets; };\ndelete = \"delete\" sp key maybe_noreply crlf @{ _state = state::cmd_delete; };\nflush = \"flush_all\" maybe_expiration maybe_noreply crlf @{ _state = state::cmd_flush_all; };\nversion = \"version\" crlf @{ _state = state::cmd_version; };\nstats = \"stats\" crlf @{ _state = state::cmd_stats; };\nstats_hash = \"stats hash\" crlf @{ _state = state::cmd_stats_hash; };\nincr = \"incr\" sp key sp u64 maybe_noreply crlf @{ _state = state::cmd_incr; };\ndecr = \"decr\" sp key sp u64 maybe_noreply crlf @{ _state = state::cmd_decr; };\nmain := (add | replace | set | get | gets | delete | flush | version | cas | stats | incr | decr\n    | stats_hash) >eof{ _state = state::eof; };\n\nprepush {\n    prepush();\n}\n\npostpop {\n    postpop();\n}\n\n}%%\n\nclass memcache_ascii_parser : public ragel_parser_base<memcache_ascii_parser> {\n    %% write data nofinal noprefix;\npublic:\n    enum class state {\n        error,\n        eof,\n        cmd_set,\n        cmd_cas,\n        cmd_add,\n        cmd_replace,\n        cmd_get,\n        cmd_gets,\n        cmd_delete,\n        cmd_flush_all,\n        cmd_version,\n        cmd_stats,\n        cmd_stats_hash,\n        cmd_incr,\n        cmd_decr,\n    };\n    state _state;\n    uint32_t _u32;\n    uint64_t _u64;\n    memcache::item_key _key;\n    sstring _flags_str;\n    uint32_t _expiration;\n    uint32_t _size;\n    sstring _size_str;\n    uint32_t _size_left;\n    uint64_t _version;\n    sstring _blob;\n    bool _noreply;\n    std::vector<memcache::item_key> _keys;\npublic:\n    void init() {\n        init_base();\n        _state = state::error;\n        _keys.clear();\n        %% write init;\n    }\n\n    char* parse(char* p, char* pe, char* eof) {\n        sstring_builder::guard g(_builder, p, pe);\n        auto str = [this, &g, &p] { g.mark_end(p); return get_str(); };\n#ifdef __clang__\n#pragma clang diagnostic push\n#pragma clang diagnostic ignored \"-Wmisleading-indentation\"\n#endif\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wimplicit-fallthrough\"\n        %% write exec;\n#pragma GCC diagnostic pop\n#ifdef __clang__\n#pragma clang diagnostic pop\n#endif\n        if (_state != state::error) {\n            return p;\n        }\n        if (p != pe) {\n            p = pe;\n            return p;\n        }\n        return nullptr;\n    }\n    bool eof() const {\n        return _state == state::eof;\n    }\n};\n"
  },
  {
    "path": "apps/memcached/memcache.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014-2015 Cloudius Systems\n */\n\n#include <boost/intrusive/unordered_set.hpp>\n#include <boost/intrusive/list.hpp>\n#include <boost/intrusive_ptr.hpp>\n#include <boost/lexical_cast.hpp>\n#include <iostream>\n#include <iomanip>\n#include <sstream>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/timer-set.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/units.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/bitops.hh>\n#include <seastar/core/slab.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/print.hh>\n#include <seastar/net/api.hh>\n#include <seastar/util/memory-data-source.hh>\n#include <seastar/util/memory-data-sink.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/log.hh>\n#include \"ascii.hh\"\n#include \"memcached.hh\"\n#include <unistd.h>\n\n#define PLATFORM \"seastar\"\n#define VERSION \"v1.0\"\n#define VERSION_STRING PLATFORM \" \" VERSION\n\nusing namespace seastar;\nusing namespace net;\n\nusing jagged_array_of_buffers = std::vector<std::vector<seastar::temporary_buffer<char>>>;\nnamespace seastar { void append_buffers(jagged_array_of_buffers& vb, std::span<seastar::temporary_buffer<char>> bufs); }\n\nnamespace memcache {\n\nnamespace bi = boost::intrusive;\n\nstatic constexpr double default_slab_growth_factor = 1.25;\nstatic constexpr uint64_t default_slab_page_size = 1UL*MB;\nstatic constexpr uint64_t default_per_cpu_slab_size = 0UL; // zero means reclaimer is enabled.\nstatic __thread slab_allocator<item>* slab;\nstatic thread_local std::unique_ptr<slab_allocator<item>> slab_holder;\n\ntemplate<typename T>\nusing optional = std::optional<T>;\n\nusing clock_type = lowres_clock;\n\n//\n// \"Expiration\" is a uint32_t value.\n// The minimal value of _time is when \"expiration\" is set to (seconds_in_a_month\n// + 1).\n// In this case _time will have a value of\n//\n// (seconds_in_a_month + 1 - Wall_Clock_Time_Since_Epoch)\n//\n// because lowres_clock now() initialized to zero when the application starts.\n//\n// We will use a timepoint at LLONG_MIN to represent a \"never expire\" value\n// since it will not collide with the minimum _time value mentioned above for\n// about 290 thousand years to come.\n//\nstatic constexpr clock_type::time_point never_expire_timepoint = clock_type::time_point(clock_type::duration::min());\n\nstruct expiration {\n    using time_point = clock_type::time_point;\n    using duration   = time_point::duration;\n\n    static constexpr uint32_t seconds_in_a_month = 60U * 60 * 24 * 30;\n    time_point _time = never_expire_timepoint;\n\n    expiration() {}\n\n    expiration(clock_type::duration wc_to_clock_type_delta, uint32_t s) {\n        using namespace std::chrono;\n\n        static_assert(sizeof(clock_type::duration::rep) >= 8, \"clock_type::duration::rep must be at least 8 bytes wide\");\n\n        if (s == 0U) {\n            return; // means never expire.\n        } else if (s <= seconds_in_a_month) {\n            _time = clock_type::now() + seconds(s); // from delta\n        } else {\n            //\n            // seastar::reactor supports only a monotonic clock at the moment\n            // therefore this may make the elements with the absolute expiration\n            // time expire at the wrong time if the wall clock has been updated\n            // during the expiration period. However the original memcached has\n            // the same weakness.\n            //\n            // TODO: Fix this when a support for system_clock-based timers is\n            // added to the seastar::reactor.\n            //\n            _time = time_point(seconds(s) + wc_to_clock_type_delta); // from real time\n        }\n    }\n\n    bool ever_expires() {\n        return _time != never_expire_timepoint;\n    }\n\n    time_point to_time_point() {\n        return _time;\n    }\n};\n\nclass item : public slab_item_base {\npublic:\n    using version_type = uint64_t;\n    using time_point = expiration::time_point;\n    using duration = expiration::duration;\n    static constexpr uint8_t field_alignment = alignof(void*);\nprivate:\n    using hook_type = bi::unordered_set_member_hook<>;\n    // TODO: align shared data to cache line boundary\n    version_type _version;\n    hook_type _cache_link;\n    bi::list_member_hook<> _timer_link;\n    size_t _key_hash;\n    expiration _expiry;\n    uint32_t _value_size;\n    uint32_t _slab_page_index;\n    uint16_t _ref_count;\n    uint8_t _key_size;\n    uint8_t _ascii_prefix_size;\n    char _data[]; // layout: data=key, (data+key_size)=ascii_prefix, (data+key_size+ascii_prefix_size)=value.\n    friend class cache;\npublic:\n    item(uint32_t slab_page_index, item_key&& key, sstring&& ascii_prefix,\n         sstring&& value, expiration expiry, version_type version = 1)\n        : _version(version)\n        , _key_hash(key.hash())\n        , _expiry(expiry)\n        , _value_size(value.size())\n        , _slab_page_index(slab_page_index)\n        , _ref_count(0U)\n        , _key_size(key.key().size())\n        , _ascii_prefix_size(ascii_prefix.size())\n    {\n        SEASTAR_ASSERT(_key_size <= std::numeric_limits<uint8_t>::max());\n        SEASTAR_ASSERT(_ascii_prefix_size <= std::numeric_limits<uint8_t>::max());\n        // storing key\n        memcpy(_data, key.key().c_str(), _key_size);\n        // storing ascii_prefix\n        memcpy(_data + align_up(_key_size, field_alignment), ascii_prefix.c_str(), _ascii_prefix_size);\n        // storing value\n        memcpy(_data + align_up(_key_size, field_alignment) + align_up(_ascii_prefix_size, field_alignment),\n               value.c_str(), _value_size);\n    }\n\n    item(const item&) = delete;\n    item(item&&) = delete;\n\n    clock_type::time_point get_timeout() {\n        return _expiry.to_time_point();\n    }\n\n    version_type version() {\n        return _version;\n    }\n\n    const std::string_view key() const {\n        return std::string_view(_data, _key_size);\n    }\n\n    const std::string_view ascii_prefix() const {\n        const char *p = _data + align_up(_key_size, field_alignment);\n        return std::string_view(p, _ascii_prefix_size);\n    }\n\n    const std::string_view value() const {\n        const char *p = _data + align_up(_key_size, field_alignment) +\n            align_up(_ascii_prefix_size, field_alignment);\n        return std::string_view(p, _value_size);\n    }\n\n    size_t key_size() const {\n        return _key_size;\n    }\n\n    size_t ascii_prefix_size() const {\n        return _ascii_prefix_size;\n    }\n\n    size_t value_size() const {\n        return _value_size;\n    }\n\n    optional<uint64_t> data_as_integral() {\n        auto str = value().data();\n        if (str[0] == '-') {\n            return {};\n        }\n\n        auto len = _value_size;\n\n        // Strip trailing space\n        while (len && str[len - 1] == ' ') {\n            len--;\n        }\n\n        try {\n            return {boost::lexical_cast<uint64_t>(str, len)};\n        } catch (const boost::bad_lexical_cast& e) {\n            return {};\n        }\n    }\n\n    // needed by timer_set\n    bool cancel() {\n        return false;\n    }\n\n    // Methods required by slab allocator.\n    uint32_t get_slab_page_index() const {\n        return _slab_page_index;\n    }\n    bool is_unlocked() const {\n        return _ref_count == 1;\n    }\n\n    friend bool operator==(const item &a, const item &b) {\n         return (a._key_hash == b._key_hash) &&\n            (a._key_size == b._key_size) &&\n            (memcmp(a._data, b._data, a._key_size) == 0);\n    }\n\n    friend std::size_t hash_value(const item &i) {\n        return i._key_hash;\n    }\n\n    friend inline void intrusive_ptr_add_ref(item* it) {\n        SEASTAR_ASSERT(it->_ref_count >= 0);\n        ++it->_ref_count;\n        if (it->_ref_count == 2) {\n            slab->lock_item(it);\n        }\n    }\n\n    friend inline void intrusive_ptr_release(item* it) {\n        --it->_ref_count;\n        if (it->_ref_count == 1) {\n            slab->unlock_item(it);\n        } else if (it->_ref_count == 0) {\n            slab->free(it);\n        }\n        SEASTAR_ASSERT(it->_ref_count >= 0);\n    }\n\n    friend struct item_key_cmp;\n};\n\nstruct item_key_cmp\n{\nprivate:\n    bool compare(const item_key& key, const item& it) const {\n        return (it._key_hash == key.hash()) &&\n            (it._key_size == key.key().size()) &&\n            (memcmp(it._data, key.key().c_str(), it._key_size) == 0);\n    }\npublic:\n    bool operator()(const item_key& key, const item& it) const {\n        return compare(key, it);\n    }\n\n    bool operator()(const item& it, const item_key& key) const {\n        return compare(key, it);\n    }\n};\n\nusing item_ptr = foreign_ptr<boost::intrusive_ptr<item>>;\n\nstruct cache_stats {\n    size_t _get_hits {};\n    size_t _get_misses {};\n    size_t _set_adds {};\n    size_t _set_replaces {};\n    size_t _cas_hits {};\n    size_t _cas_misses {};\n    size_t _cas_badval {};\n    size_t _delete_misses {};\n    size_t _delete_hits {};\n    size_t _incr_misses {};\n    size_t _incr_hits {};\n    size_t _decr_misses {};\n    size_t _decr_hits {};\n    size_t _expired {};\n    size_t _evicted {};\n    size_t _bytes {};\n    size_t _resize_failure {};\n    size_t _size {};\n    size_t _reclaims{};\n\n    void operator+=(const cache_stats& o) {\n        _get_hits += o._get_hits;\n        _get_misses += o._get_misses;\n        _set_adds += o._set_adds;\n        _set_replaces += o._set_replaces;\n        _cas_hits += o._cas_hits;\n        _cas_misses += o._cas_misses;\n        _cas_badval += o._cas_badval;\n        _delete_misses += o._delete_misses;\n        _delete_hits += o._delete_hits;\n        _incr_misses += o._incr_misses;\n        _incr_hits += o._incr_hits;\n        _decr_misses += o._decr_misses;\n        _decr_hits += o._decr_hits;\n        _expired += o._expired;\n        _evicted += o._evicted;\n        _bytes += o._bytes;\n        _resize_failure += o._resize_failure;\n        _size += o._size;\n        _reclaims += o._reclaims;\n    }\n};\n\nenum class cas_result {\n    not_found, stored, bad_version\n};\n\nstruct remote_origin_tag {\n    template <typename T>\n    static inline\n    T move_if_local(T& ref) {\n        return ref;\n    }\n};\n\nstruct local_origin_tag {\n    template <typename T>\n    static inline\n    T move_if_local(T& ref) {\n        return std::move(ref);\n    }\n};\n\nstruct item_insertion_data {\n    item_key key;\n    sstring ascii_prefix;\n    sstring data;\n    expiration expiry;\n};\n\nclass cache {\nprivate:\n    using cache_type = bi::unordered_set<item,\n        bi::member_hook<item, item::hook_type, &item::_cache_link>,\n        bi::power_2_buckets<true>,\n        bi::constant_time_size<true>>;\n    using cache_iterator = typename cache_type::iterator;\n    static constexpr size_t initial_bucket_count = 1 << 10;\n    static constexpr float load_factor = 0.75f;\n    size_t _resize_up_threshold = load_factor * initial_bucket_count;\n    std::vector<cache_type::bucket_type> _buckets;\n    cache_type _cache;\n    seastar::timer_set<item, &item::_timer_link> _alive;\n    timer<clock_type> _timer;\n    // delta in seconds between the current values of a wall clock and a clock_type clock\n    clock_type::duration _wc_to_clock_type_delta;\n    cache_stats _stats;\n    timer<clock_type> _flush_timer;\nprivate:\n    size_t item_size(item& item_ref) {\n        constexpr size_t field_alignment = alignof(void*);\n        return sizeof(item) +\n            align_up(item_ref.key_size(), field_alignment) +\n            align_up(item_ref.ascii_prefix_size(), field_alignment) +\n            item_ref.value_size();\n    }\n\n    size_t item_size(item_insertion_data& insertion) {\n        constexpr size_t field_alignment = alignof(void*);\n        auto size = sizeof(item) +\n            align_up(insertion.key.key().size(), field_alignment) +\n            align_up(insertion.ascii_prefix.size(), field_alignment) +\n            insertion.data.size();\n#ifdef __DEBUG__\n        static bool print_item_footprint = true;\n        if (print_item_footprint) {\n            print_item_footprint = false;\n            std::cout << __FUNCTION__ << \": \" << size << \"\\n\";\n            std::cout << \"sizeof(item)      \" << sizeof(item) << \"\\n\";\n            std::cout << \"key.size          \" << insertion.key.key().size() << \"\\n\";\n            std::cout << \"value.size        \" << insertion.data.size() << \"\\n\";\n            std::cout << \"ascii_prefix.size \" << insertion.ascii_prefix.size() << \"\\n\";\n        }\n#endif\n        return size;\n    }\n\n    template <bool IsInCache = true, bool IsInTimerList = true, bool Release = true>\n    void erase(item& item_ref) {\n        if (IsInCache) {\n            _cache.erase(_cache.iterator_to(item_ref));\n        }\n        if (IsInTimerList) {\n            if (item_ref._expiry.ever_expires()) {\n                _alive.remove(item_ref);\n            }\n        }\n        _stats._bytes -= item_size(item_ref);\n        if (Release) {\n            // memory used by item shouldn't be freed when slab is replacing it with another item.\n            intrusive_ptr_release(&item_ref);\n        }\n    }\n\n    void expire() {\n        using namespace std::chrono;\n\n        //\n        // Adjust the delta on every timer event to minimize an error caused\n        // by a wall clock adjustment.\n        //\n        _wc_to_clock_type_delta =\n            duration_cast<clock_type::duration>(clock_type::now().time_since_epoch() - system_clock::now().time_since_epoch());\n\n        auto exp = _alive.expire(clock_type::now());\n        while (!exp.empty()) {\n            auto item = &*exp.begin();\n            exp.pop_front();\n            erase<true, false>(*item);\n            _stats._expired++;\n        }\n        _timer.arm(_alive.get_next_timeout());\n    }\n\n    inline\n    cache_iterator find(const item_key& key) {\n        return _cache.find(key, std::hash<item_key>(), item_key_cmp());\n    }\n\n    template <typename Origin>\n    inline\n    cache_iterator add_overriding(cache_iterator i, item_insertion_data& insertion) {\n        auto& old_item = *i;\n        uint64_t old_item_version = old_item._version;\n\n        erase(old_item);\n\n        size_t size = item_size(insertion);\n        auto new_item = slab->create(size, Origin::move_if_local(insertion.key), Origin::move_if_local(insertion.ascii_prefix),\n            Origin::move_if_local(insertion.data), insertion.expiry, old_item_version + 1);\n        intrusive_ptr_add_ref(new_item);\n\n        auto insert_result = _cache.insert(*new_item);\n        SEASTAR_ASSERT(insert_result.second);\n        if (insertion.expiry.ever_expires() && _alive.insert(*new_item)) {\n            _timer.rearm(new_item->get_timeout());\n        }\n        _stats._bytes += size;\n        return insert_result.first;\n    }\n\n    template <typename Origin>\n    inline\n    void add_new(item_insertion_data& insertion) {\n        size_t size = item_size(insertion);\n        auto new_item = slab->create(size, Origin::move_if_local(insertion.key), Origin::move_if_local(insertion.ascii_prefix),\n            Origin::move_if_local(insertion.data), insertion.expiry);\n        intrusive_ptr_add_ref(new_item);\n        auto& item_ref = *new_item;\n        _cache.insert(item_ref);\n        if (insertion.expiry.ever_expires() && _alive.insert(item_ref)) {\n            _timer.rearm(item_ref.get_timeout());\n        }\n        _stats._bytes += size;\n        maybe_rehash();\n    }\n\n    void maybe_rehash() {\n        if (_cache.size() >= _resize_up_threshold) {\n            auto new_size = _cache.bucket_count() * 2;\n            std::vector<cache_type::bucket_type> old_buckets;\n            try {\n                old_buckets = std::exchange(_buckets, std::vector<cache_type::bucket_type>(new_size));\n            } catch (const std::bad_alloc& e) {\n                _stats._resize_failure++;\n                return;\n            }\n            _cache.rehash(typename cache_type::bucket_traits(_buckets.data(), new_size));\n            _resize_up_threshold = _cache.bucket_count() * load_factor;\n        }\n    }\npublic:\n    cache(uint64_t per_cpu_slab_size, uint64_t slab_page_size)\n        : _buckets(initial_bucket_count)\n        , _cache(cache_type::bucket_traits(_buckets.data(), initial_bucket_count))\n    {\n        using namespace std::chrono;\n\n        _wc_to_clock_type_delta =\n            duration_cast<clock_type::duration>(clock_type::now().time_since_epoch() - system_clock::now().time_since_epoch());\n\n        _timer.set_callback([this] { expire(); });\n        _flush_timer.set_callback([this] { flush_all(); });\n\n        // initialize per-thread slab allocator.\n        slab_holder = std::make_unique<slab_allocator<item>>(default_slab_growth_factor, per_cpu_slab_size, slab_page_size,\n                [this](item& item_ref) { erase<true, true, false>(item_ref); _stats._evicted++; });\n        slab = slab_holder.get();\n#ifdef __DEBUG__\n        static bool print_slab_classes = true;\n        if (print_slab_classes) {\n            print_slab_classes = false;\n            slab->print_slab_classes();\n        }\n#endif\n    }\n\n    ~cache() {\n       flush_all();\n    }\n\n    void flush_all() {\n        _flush_timer.cancel();\n        _cache.erase_and_dispose(_cache.begin(), _cache.end(), [this] (item* it) {\n            erase<false, true>(*it);\n        });\n    }\n\n    void flush_at(uint32_t time) {\n        auto expiry = expiration(get_wc_to_clock_type_delta(), time);\n        _flush_timer.rearm(expiry.to_time_point());\n    }\n\n    template <typename Origin = local_origin_tag>\n    bool set(item_insertion_data& insertion) {\n        auto i = find(insertion.key);\n        if (i != _cache.end()) {\n            add_overriding<Origin>(i, insertion);\n            _stats._set_replaces++;\n            return true;\n        } else {\n            add_new<Origin>(insertion);\n            _stats._set_adds++;\n            return false;\n        }\n    }\n\n    template <typename Origin = local_origin_tag>\n    bool add(item_insertion_data& insertion) {\n        auto i = find(insertion.key);\n        if (i != _cache.end()) {\n            return false;\n        }\n\n        _stats._set_adds++;\n        add_new<Origin>(insertion);\n        return true;\n    }\n\n    template <typename Origin = local_origin_tag>\n    bool replace(item_insertion_data& insertion) {\n        auto i = find(insertion.key);\n        if (i == _cache.end()) {\n            return false;\n        }\n\n        _stats._set_replaces++;\n        add_overriding<Origin>(i, insertion);\n        return true;\n    }\n\n    bool remove(const item_key& key) {\n        auto i = find(key);\n        if (i == _cache.end()) {\n            _stats._delete_misses++;\n            return false;\n        }\n        _stats._delete_hits++;\n        auto& item_ref = *i;\n        erase(item_ref);\n        return true;\n    }\n\n    item_ptr get(const item_key& key) {\n        auto i = find(key);\n        if (i == _cache.end()) {\n            _stats._get_misses++;\n            return nullptr;\n        }\n        _stats._get_hits++;\n        auto& item_ref = *i;\n        return item_ptr(&item_ref);\n    }\n\n    template <typename Origin = local_origin_tag>\n    cas_result cas(item_insertion_data& insertion, item::version_type version) {\n        auto i = find(insertion.key);\n        if (i == _cache.end()) {\n            _stats._cas_misses++;\n            return cas_result::not_found;\n        }\n        auto& item_ref = *i;\n        if (item_ref._version != version) {\n            _stats._cas_badval++;\n            return cas_result::bad_version;\n        }\n        _stats._cas_hits++;\n        add_overriding<Origin>(i, insertion);\n        return cas_result::stored;\n    }\n\n    size_t size() {\n        return _cache.size();\n    }\n\n    size_t bucket_count() {\n        return _cache.bucket_count();\n    }\n\n    cache_stats stats() {\n        _stats._size = size();\n        return _stats;\n    }\n\n    template <typename Origin = local_origin_tag>\n    std::pair<item_ptr, bool> incr(item_key& key, uint64_t delta) {\n        auto i = find(key);\n        if (i == _cache.end()) {\n            _stats._incr_misses++;\n            return {item_ptr{}, false};\n        }\n        auto& item_ref = *i;\n        _stats._incr_hits++;\n        auto value = item_ref.data_as_integral();\n        if (!value) {\n            return {boost::intrusive_ptr<item>(&item_ref), false};\n        }\n        item_insertion_data insertion {\n            .key = Origin::move_if_local(key),\n            .ascii_prefix = sstring(item_ref.ascii_prefix().data(), item_ref.ascii_prefix_size()),\n            .data = to_sstring(*value + delta),\n            .expiry = item_ref._expiry\n        };\n        i = add_overriding<local_origin_tag>(i, insertion);\n        return {boost::intrusive_ptr<item>(&*i), true};\n    }\n\n    template <typename Origin = local_origin_tag>\n    std::pair<item_ptr, bool> decr(item_key& key, uint64_t delta) {\n        auto i = find(key);\n        if (i == _cache.end()) {\n            _stats._decr_misses++;\n            return {item_ptr{}, false};\n        }\n        auto& item_ref = *i;\n        _stats._decr_hits++;\n        auto value = item_ref.data_as_integral();\n        if (!value) {\n            return {boost::intrusive_ptr<item>(&item_ref), false};\n        }\n        item_insertion_data insertion {\n            .key = Origin::move_if_local(key),\n            .ascii_prefix = sstring(item_ref.ascii_prefix().data(), item_ref.ascii_prefix_size()),\n            .data = to_sstring(*value - std::min(*value, delta)),\n            .expiry = item_ref._expiry\n        };\n        i = add_overriding<local_origin_tag>(i, insertion);\n        return {boost::intrusive_ptr<item>(&*i), true};\n    }\n\n    std::pair<unsigned, foreign_ptr<lw_shared_ptr<std::string>>> print_hash_stats() {\n        static constexpr unsigned bits = sizeof(size_t) * 8;\n        size_t histo[bits + 1] {};\n        size_t max_size = 0;\n        unsigned max_bucket = 0;\n\n        for (size_t i = 0; i < _cache.bucket_count(); i++) {\n            size_t size = _cache.bucket_size(i);\n            unsigned bucket;\n            if (size == 0) {\n                bucket = 0;\n            } else {\n                bucket = bits - count_leading_zeros(size);\n            }\n            max_bucket = std::max(max_bucket, bucket);\n            max_size = std::max(max_size, size);\n            histo[bucket]++;\n        }\n\n        std::stringstream ss;\n\n        ss << \"size: \" << _cache.size() << \"\\n\";\n        ss << \"buckets: \" << _cache.bucket_count() << \"\\n\";\n        ss << \"load: \" << format(\"{:.2f}\", (double)_cache.size() / _cache.bucket_count()) << \"\\n\";\n        ss << \"max bucket occupancy: \" << max_size << \"\\n\";\n        ss << \"bucket occupancy histogram:\\n\";\n\n        for (unsigned i = 0; i < (max_bucket + 2); i++) {\n            ss << \"  \";\n            if (i == 0) {\n                ss << \"0: \";\n            } else if (i == 1) {\n                ss << \"1: \";\n            } else {\n                ss << (1 << (i - 1)) << \"+: \";\n            }\n            ss << histo[i] << \"\\n\";\n        }\n        return {this_shard_id(), make_foreign(make_lw_shared<std::string>(ss.str()))};\n    }\n\n    future<> stop() { return make_ready_future<>(); }\n    clock_type::duration get_wc_to_clock_type_delta() { return _wc_to_clock_type_delta; }\n};\n\nclass sharded_cache {\nprivate:\n    sharded<cache>& _peers;\n\n    inline\n    unsigned get_cpu(const item_key& key) {\n        return std::hash<item_key>()(key) % smp::count;\n    }\npublic:\n    sharded_cache(sharded<cache>& peers) : _peers(peers) {}\n\n    future<> flush_all() {\n        return _peers.invoke_on_all(&cache::flush_all);\n    }\n\n    future<> flush_at(uint32_t time) {\n        return _peers.invoke_on_all(&cache::flush_at, time);\n    }\n\n    auto get_wc_to_clock_type_delta() { return _peers.local().get_wc_to_clock_type_delta(); }\n\n    // The caller must keep @insertion live until the resulting future resolves.\n    future<bool> set(item_insertion_data& insertion) {\n        auto cpu = get_cpu(insertion.key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<bool>(_peers.local().set(insertion));\n        }\n        return _peers.invoke_on(cpu, &cache::set<remote_origin_tag>, std::ref(insertion));\n    }\n\n    // The caller must keep @insertion live until the resulting future resolves.\n    future<bool> add(item_insertion_data& insertion) {\n        auto cpu = get_cpu(insertion.key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<bool>(_peers.local().add(insertion));\n        }\n        return _peers.invoke_on(cpu, &cache::add<remote_origin_tag>, std::ref(insertion));\n    }\n\n    // The caller must keep @insertion live until the resulting future resolves.\n    future<bool> replace(item_insertion_data& insertion) {\n        auto cpu = get_cpu(insertion.key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<bool>(_peers.local().replace(insertion));\n        }\n        return _peers.invoke_on(cpu, &cache::replace<remote_origin_tag>, std::ref(insertion));\n    }\n\n    // The caller must keep @key live until the resulting future resolves.\n    future<bool> remove(const item_key& key) {\n        auto cpu = get_cpu(key);\n        return _peers.invoke_on(cpu, &cache::remove, std::ref(key));\n    }\n\n    // The caller must keep @key live until the resulting future resolves.\n    future<item_ptr> get(const item_key& key) {\n        auto cpu = get_cpu(key);\n        return _peers.invoke_on(cpu, &cache::get, std::ref(key));\n    }\n\n    // The caller must keep @insertion live until the resulting future resolves.\n    future<cas_result> cas(item_insertion_data& insertion, item::version_type version) {\n        auto cpu = get_cpu(insertion.key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<cas_result>(_peers.local().cas(insertion, version));\n        }\n        return _peers.invoke_on(cpu, &cache::cas<remote_origin_tag>, std::ref(insertion), std::move(version));\n    }\n\n    future<cache_stats> stats() {\n        return _peers.map_reduce(adder<cache_stats>(), &cache::stats);\n    }\n\n    // The caller must keep @key live until the resulting future resolves.\n    future<std::pair<item_ptr, bool>> incr(item_key& key, uint64_t delta) {\n        auto cpu = get_cpu(key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<std::pair<item_ptr, bool>>(\n                _peers.local().incr<local_origin_tag>(key, delta));\n        }\n        return _peers.invoke_on(cpu, &cache::incr<remote_origin_tag>, std::ref(key), std::move(delta));\n    }\n\n    // The caller must keep @key live until the resulting future resolves.\n    future<std::pair<item_ptr, bool>> decr(item_key& key, uint64_t delta) {\n        auto cpu = get_cpu(key);\n        if (this_shard_id() == cpu) {\n            return make_ready_future<std::pair<item_ptr, bool>>(\n                _peers.local().decr(key, delta));\n        }\n        return _peers.invoke_on(cpu, &cache::decr<remote_origin_tag>, std::ref(key), std::move(delta));\n    }\n\n    future<> print_hash_stats(output_stream<char>& out) {\n        return _peers.map_reduce([&out] (std::pair<unsigned, foreign_ptr<lw_shared_ptr<std::string>>> data) mutable {\n            return out.write(\"=== CPU \" + std::to_string(data.first) + \" ===\\r\\n\")\n                .then([&out, str = std::move(data.second)] {\n                    return out.write(*str);\n                });\n            }, &cache::print_hash_stats);\n    }\n};\n\nstruct system_stats {\n    uint32_t _curr_connections {};\n    uint32_t _total_connections {};\n    uint64_t _cmd_get {};\n    uint64_t _cmd_set {};\n    uint64_t _cmd_flush {};\n    clock_type::time_point _start_time;\npublic:\n    system_stats() {\n        _start_time = clock_type::time_point::max();\n    }\n    system_stats(clock_type::time_point start_time)\n        : _start_time(start_time) {\n    }\n    system_stats self() {\n        return *this;\n    }\n    void operator+=(const system_stats& other) {\n        _curr_connections += other._curr_connections;\n        _total_connections += other._total_connections;\n        _cmd_get += other._cmd_get;\n        _cmd_set += other._cmd_set;\n        _cmd_flush += other._cmd_flush;\n        _start_time = std::min(_start_time, other._start_time);\n    }\n    future<> stop() { return make_ready_future<>(); }\n};\n\nclass ascii_protocol {\nprivate:\n    using this_type = ascii_protocol;\n    sharded_cache& _cache;\n    sharded<system_stats>& _system_stats;\n    memcache_ascii_parser _parser;\n    item_key _item_key;\n    item_insertion_data _insertion;\n    std::vector<item_ptr> _items;\nprivate:\n    static constexpr const char *msg_crlf = \"\\r\\n\";\n    static constexpr const char *msg_error = \"ERROR\\r\\n\";\n    static constexpr const char *msg_stored = \"STORED\\r\\n\";\n    static constexpr const char *msg_not_stored = \"NOT_STORED\\r\\n\";\n    static constexpr const char *msg_end = \"END\\r\\n\";\n    static constexpr const char *msg_value = \"VALUE \";\n    static constexpr const char *msg_deleted = \"DELETED\\r\\n\";\n    static constexpr const char *msg_not_found = \"NOT_FOUND\\r\\n\";\n    static constexpr const char *msg_ok = \"OK\\r\\n\";\n    static constexpr const char *msg_version = \"VERSION \" VERSION_STRING \"\\r\\n\";\n    static constexpr const char *msg_exists = \"EXISTS\\r\\n\";\n    static constexpr const char *msg_stat = \"STAT \";\n    static constexpr const char *msg_out_of_memory = \"SERVER_ERROR Out of memory allocating new item\\r\\n\";\n    static constexpr const char *msg_error_non_numeric_value = \"CLIENT_ERROR cannot increment or decrement non-numeric value\\r\\n\";\nprivate:\n    static void append(std::vector<temporary_buffer<char>>& bufs, const char* buf, size_t size) {\n        if (size) {\n            bufs.emplace_back(const_cast<char*>(buf), size, deleter());\n        }\n    }\n\n    static void append(std::vector<temporary_buffer<char>>& bufs, const char* s) { append(bufs, s, strlen(s)); }\n    static void append(std::vector<temporary_buffer<char>>& bufs, const std::string_view& s) { append(bufs, s.data(), s.size()); }\n\n    template <bool WithVersion>\n    static void serialize(std::vector<temporary_buffer<char>>& bufs, item_ptr item) {\n        if (!item) {\n            return;\n        }\n\n        append(bufs, \"VALUE \");\n        append(bufs, item->key());\n        append(bufs, item->ascii_prefix());\n\n        if (WithVersion) {\n            append(bufs, \" \");\n            bufs.emplace_back(temporary_buffer<char>::copy_of(to_sstring(item->version())));\n        }\n\n        append(bufs, msg_crlf);\n        append(bufs, item->value());\n        bufs.emplace_back(const_cast<char*>(msg_crlf), strlen(msg_crlf), make_deleter([item = std::move(item)]{}));\n    }\n\n    template <bool WithVersion>\n    future<> handle_get(output_stream<char>& out) {\n        _system_stats.local()._cmd_get++;\n        if (_parser._keys.size() == 1) {\n            return _cache.get(_parser._keys[0]).then([&out] (auto item) -> future<> {\n                std::vector<temporary_buffer<char>> bufs;\n                this_type::serialize<WithVersion>(bufs, std::move(item));\n                append(bufs, msg_end);\n                return out.write(std::span<temporary_buffer<char>>(bufs));\n            });\n        } else {\n            _items.clear();\n            return parallel_for_each(_parser._keys.begin(), _parser._keys.end(), [this] (const auto& key) {\n                return _cache.get(key).then([this] (auto item) {\n                    _items.emplace_back(std::move(item));\n                });\n            }).then([this, &out] () {\n                std::vector<temporary_buffer<char>> bufs;\n                for (auto& item : _items) {\n                    serialize<WithVersion>(bufs, std::move(item));\n                }\n                append(bufs, msg_end);\n                return out.write(std::span<temporary_buffer<char>>(bufs));\n            });\n        }\n    }\n\n    template <typename Value>\n    static future<> print_stat(output_stream<char>& out, const char* key, Value value) {\n        return out.write(msg_stat)\n                .then([&out, key] { return out.write(key); })\n                .then([&out] { return out.write(\" \"); })\n                .then([&out, value] { return out.write(to_sstring(value)); })\n                .then([&out] { return out.write(msg_crlf); });\n    }\n\n    future<> print_stats(output_stream<char>& out) {\n        return _cache.stats().then([this, &out] (auto stats) {\n            return _system_stats.map_reduce(adder<system_stats>(), &system_stats::self)\n                .then([&out, all_cache_stats = std::move(stats)] (auto all_system_stats) -> future<> {\n                    auto now = clock_type::now();\n                    auto total_items = all_cache_stats._set_replaces + all_cache_stats._set_adds\n                        + all_cache_stats._cas_hits;\n                    return print_stat(out, \"pid\", getpid())\n                        .then([&out, uptime = now - all_system_stats._start_time] {\n                            return print_stat(out, \"uptime\",\n                                std::chrono::duration_cast<std::chrono::seconds>(uptime).count());\n                        }).then([now, &out] {\n                            return print_stat(out, \"time\",\n                                std::chrono::duration_cast<std::chrono::seconds>(now.time_since_epoch()).count());\n                        }).then([&out] {\n                            return print_stat(out, \"version\", VERSION_STRING);\n                        }).then([&out] {\n                            return print_stat(out, \"pointer_size\", sizeof(void*)*8);\n                        }).then([&out, v = all_system_stats._curr_connections] {\n                            return print_stat(out, \"curr_connections\", v);\n                        }).then([&out, v = all_system_stats._total_connections] {\n                            return print_stat(out, \"total_connections\", v);\n                        }).then([&out, v = all_system_stats._curr_connections] {\n                            return print_stat(out, \"connection_structures\", v);\n                        }).then([&out, v = all_system_stats._cmd_get] {\n                            return print_stat(out, \"cmd_get\", v);\n                        }).then([&out, v = all_system_stats._cmd_set] {\n                            return print_stat(out, \"cmd_set\", v);\n                        }).then([&out, v = all_system_stats._cmd_flush] {\n                            return print_stat(out, \"cmd_flush\", v);\n                        }).then([&out] {\n                            return print_stat(out, \"cmd_touch\", 0);\n                        }).then([&out, v = all_cache_stats._get_hits] {\n                            return print_stat(out, \"get_hits\", v);\n                        }).then([&out, v = all_cache_stats._get_misses] {\n                            return print_stat(out, \"get_misses\", v);\n                        }).then([&out, v = all_cache_stats._delete_misses] {\n                            return print_stat(out, \"delete_misses\", v);\n                        }).then([&out, v = all_cache_stats._delete_hits] {\n                            return print_stat(out, \"delete_hits\", v);\n                        }).then([&out, v = all_cache_stats._incr_misses] {\n                            return print_stat(out, \"incr_misses\", v);\n                        }).then([&out, v = all_cache_stats._incr_hits] {\n                            return print_stat(out, \"incr_hits\", v);\n                        }).then([&out, v = all_cache_stats._decr_misses] {\n                            return print_stat(out, \"decr_misses\", v);\n                        }).then([&out, v = all_cache_stats._decr_hits] {\n                            return print_stat(out, \"decr_hits\", v);\n                        }).then([&out, v = all_cache_stats._cas_misses] {\n                            return print_stat(out, \"cas_misses\", v);\n                        }).then([&out, v = all_cache_stats._cas_hits] {\n                            return print_stat(out, \"cas_hits\", v);\n                        }).then([&out, v = all_cache_stats._cas_badval] {\n                            return print_stat(out, \"cas_badval\", v);\n                        }).then([&out] {\n                            return print_stat(out, \"touch_hits\", 0);\n                        }).then([&out] {\n                            return print_stat(out, \"touch_misses\", 0);\n                        }).then([&out] {\n                            return print_stat(out, \"auth_cmds\", 0);\n                        }).then([&out] {\n                            return print_stat(out, \"auth_errors\", 0);\n                        }).then([&out] {\n                            return print_stat(out, \"threads\", smp::count);\n                        }).then([&out, v = all_cache_stats._size] {\n                            return print_stat(out, \"curr_items\", v);\n                        }).then([&out, v = total_items] {\n                            return print_stat(out, \"total_items\", v);\n                        }).then([&out, v = all_cache_stats._expired] {\n                            return print_stat(out, \"seastar.expired\", v);\n                        }).then([&out, v = all_cache_stats._resize_failure] {\n                            return print_stat(out, \"seastar.resize_failure\", v);\n                        }).then([&out, v = all_cache_stats._evicted] {\n                            return print_stat(out, \"evictions\", v);\n                        }).then([&out, v = all_cache_stats._bytes] {\n                            return print_stat(out, \"bytes\", v);\n                        }).then([&out] {\n                            return out.write(msg_end);\n                        });\n                });\n        });\n    }\npublic:\n    ascii_protocol(sharded_cache& cache, sharded<system_stats>& system_stats)\n        : _cache(cache)\n        , _system_stats(system_stats)\n    {}\n\n    void prepare_insertion() {\n        _insertion = item_insertion_data{\n            .key = std::move(_parser._key),\n            .ascii_prefix = make_sstring(\" \", _parser._flags_str, \" \", _parser._size_str),\n            .data = std::move(_parser._blob),\n            .expiry = expiration(_cache.get_wc_to_clock_type_delta(), _parser._expiration)\n        };\n    }\n\n    future<> handle(input_stream<char>& in, output_stream<char>& out) {\n        _parser.init();\n        return in.consume(_parser).then([this, &out] () -> future<> {\n            switch (_parser._state) {\n                case memcache_ascii_parser::state::eof:\n                    return make_ready_future<>();\n\n                case memcache_ascii_parser::state::error:\n                    return out.write(msg_error);\n\n                case memcache_ascii_parser::state::cmd_set:\n                {\n                    _system_stats.local()._cmd_set++;\n                    prepare_insertion();\n                    auto f = _cache.set(_insertion);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (...) {\n                        return out.write(msg_stored);\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_cas:\n                {\n                    _system_stats.local()._cmd_set++;\n                    prepare_insertion();\n                    auto f = _cache.cas(_insertion, _parser._version);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (auto result) {\n                        switch (result) {\n                            case cas_result::stored:\n                                return out.write(msg_stored);\n                            case cas_result::not_found:\n                                return out.write(msg_not_found);\n                            case cas_result::bad_version:\n                                return out.write(msg_exists);\n                            default:\n                                std::abort();\n                        }\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_add:\n                {\n                    _system_stats.local()._cmd_set++;\n                    prepare_insertion();\n                    auto f = _cache.add(_insertion);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (bool added) {\n                        return out.write(added ? msg_stored : msg_not_stored);\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_replace:\n                {\n                    _system_stats.local()._cmd_set++;\n                    prepare_insertion();\n                    auto f = _cache.replace(_insertion);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (auto replaced) {\n                        return out.write(replaced ? msg_stored : msg_not_stored);\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_get:\n                    return handle_get<false>(out);\n\n                case memcache_ascii_parser::state::cmd_gets:\n                    return handle_get<true>(out);\n\n                case memcache_ascii_parser::state::cmd_delete:\n                {\n                    auto f = _cache.remove(_parser._key);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (bool removed) {\n                        return out.write(removed ? msg_deleted : msg_not_found);\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_flush_all:\n                {\n                    _system_stats.local()._cmd_flush++;\n                    if (_parser._expiration) {\n                        auto f = _cache.flush_at(_parser._expiration);\n                        if (_parser._noreply) {\n                            return f;\n                        }\n                        return std::move(f).then([&out] {\n                            return out.write(msg_ok);\n                        });\n                    } else {\n                        auto f = _cache.flush_all();\n                        if (_parser._noreply) {\n                            return f;\n                        }\n                        return std::move(f).then([&out] {\n                            return out.write(msg_ok);\n                        });\n                    }\n                }\n\n                case memcache_ascii_parser::state::cmd_version:\n                    return out.write(msg_version);\n\n                case memcache_ascii_parser::state::cmd_stats:\n                    return print_stats(out);\n\n                case memcache_ascii_parser::state::cmd_stats_hash:\n                    return _cache.print_hash_stats(out);\n\n                case memcache_ascii_parser::state::cmd_incr:\n                {\n                    auto f = _cache.incr(_parser._key, _parser._u64);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (auto result) {\n                        auto item = std::move(result.first);\n                        if (!item) {\n                            return out.write(msg_not_found);\n                        }\n                        auto incremented = result.second;\n                        if (!incremented) {\n                            return out.write(msg_error_non_numeric_value);\n                        }\n                        return out.write(item->value().data(), item->value_size()).then([&out] {\n                            return out.write(msg_crlf);\n                        });\n                    });\n                }\n\n                case memcache_ascii_parser::state::cmd_decr:\n                {\n                    auto f = _cache.decr(_parser._key, _parser._u64);\n                    if (_parser._noreply) {\n                        return std::move(f).discard_result();\n                    }\n                    return std::move(f).then([&out] (auto result) {\n                        auto item = std::move(result.first);\n                        if (!item) {\n                            return out.write(msg_not_found);\n                        }\n                        auto decremented = result.second;\n                        if (!decremented) {\n                            return out.write(msg_error_non_numeric_value);\n                        }\n                        return out.write(item->value().data(), item->value_size()).then([&out] {\n                            return out.write(msg_crlf);\n                        });\n                    });\n                }\n            };\n            std::abort();\n        }).then_wrapped([this, &out] (auto&& f) -> future<> {\n            // FIXME: then_wrapped() being scheduled even though no exception was triggered has a\n            // performance cost of about 2.6%. Not using it means maintainability penalty.\n            try {\n                f.get();\n            } catch (std::bad_alloc& e) {\n                if (_parser._noreply) {\n                    return make_ready_future<>();\n                }\n                return out.write(msg_out_of_memory);\n            }\n            return make_ready_future<>();\n        });\n    };\n};\n\nclass udp_server {\npublic:\n    static const size_t default_max_datagram_size = 1400;\nprivate:\n    std::optional<future<>> _task;\n    sharded_cache& _cache;\n    sharded<system_stats>& _system_stats;\n    udp_channel _chan;\n    uint16_t _port;\n    size_t _max_datagram_size = default_max_datagram_size;\n\n    struct header {\n        packed<uint16_t> _request_id;\n        packed<uint16_t> _sequence_number;\n        packed<uint16_t> _n;\n        packed<uint16_t> _reserved;\n\n        template<typename Adjuster>\n        auto adjust_endianness(Adjuster a) {\n            return a(_request_id, _sequence_number, _n);\n        }\n    } __attribute__((packed));\n\n    struct connection {\n        ipv4_addr _src;\n        uint16_t _request_id;\n        input_stream<char> _in;\n        jagged_array_of_buffers _out_bufs;\n        output_stream<char> _out;\n        ascii_protocol _proto;\n\n        static output_stream_options make_opts() noexcept {\n            output_stream_options opts;\n            opts.trim_to_size = true;\n            return opts;\n        }\n\n        connection(ipv4_addr src, input_stream<char>&& in, size_t out_size,\n                sharded_cache& c, sharded<system_stats>& system_stats)\n            : _src(src)\n            , _in(std::move(in))\n            , _out(output_stream<char>(data_sink(std::make_unique<util::basic_memory_data_sink<jagged_array_of_buffers>>(_out_bufs)), out_size, make_opts()))\n            , _proto(c, system_stats)\n        {}\n\n        future<> respond(udp_channel& chan) {\n            int i = 0;\n            return do_for_each(_out_bufs.begin(), _out_bufs.end(), [this, i, &chan] (std::vector<temporary_buffer<char>>& bufs) mutable {\n                header* out_hdr = reinterpret_cast<header*>(bufs.front().get_write());\n                out_hdr->_request_id = _request_id;\n                out_hdr->_sequence_number = i++;\n                out_hdr->_n = _out_bufs.size();\n                *out_hdr = hton(*out_hdr);\n                return chan.send(_src, bufs);\n            });\n        }\n    };\n\npublic:\n    static constexpr size_t header_size = sizeof(header);\n\n    udp_server(sharded_cache& c, sharded<system_stats>& system_stats, uint16_t port = 11211)\n         : _cache(c)\n         , _system_stats(system_stats)\n         , _port(port)\n    {}\n\n    void set_max_datagram_size(size_t max_datagram_size) {\n        _max_datagram_size = max_datagram_size;\n    }\n\nprivate:\n    input_stream<char> as_input_stream(std::span<temporary_buffer<char>> bufs) {\n        if (bufs.size() == 1) {\n            return util::as_input_stream(std::move(bufs[0]));\n        }\n\n        std::vector<temporary_buffer<char>> stable_bufs;\n        stable_bufs.reserve(bufs.size());\n        std::ranges::move(bufs, std::back_inserter(stable_bufs));\n        return util::as_input_stream(std::move(stable_bufs));\n    }\n\npublic:\n    void start() {\n        _chan = make_bound_datagram_channel({_port});\n        // Run in the background.\n        _task = keep_doing([this] {\n            return _chan.receive().then([this](datagram dgram) {\n                input_stream<char> in = as_input_stream(dgram.get_buffers());\n                auto conn = make_lw_shared<connection>(dgram.get_src(), std::move(in),\n                      _max_datagram_size - sizeof(header), _cache, _system_stats);\n                return conn->_in.read_exactly(sizeof(header)).then([this, conn] (auto h) mutable {\n                    if (h.size() < sizeof(header)) {\n                        // dropping invalid packet\n                        return make_ready_future<>();\n                    }\n                    header hdr = ntoh(*reinterpret_cast<const header*>(h.get()));\n                    conn->_request_id = hdr._request_id;\n\n                    if (hdr._n != 1 || hdr._sequence_number != 0) {\n                        return conn->_out.write(\"CLIENT_ERROR only single-datagram requests supported\\r\\n\").then([this, conn] {\n                            return conn->_out.flush().then([this, conn] {\n                                return conn->respond(_chan).then([conn] {});\n                            });\n                        });\n                    }\n\n                    return conn->_proto.handle(conn->_in, conn->_out).then([this, conn]() mutable {\n                        return conn->_out.flush().then([this, conn] {\n                            return conn->respond(_chan).then([conn] {});\n                        });\n                    });\n                });\n            });\n        });\n    };\n\n    future<> stop() {\n        _chan.shutdown_input();\n        _chan.shutdown_output();\n        return _task->handle_exception([](std::exception_ptr e) {\n            std::cerr << \"exception in udp_server \" << e << '\\n';\n        });\n    }\n};\n\nclass tcp_server {\nprivate:\n    std::optional<future<>> _task;\n    lw_shared_ptr<seastar::server_socket> _listener;\n    sharded_cache& _cache;\n    sharded<system_stats>& _system_stats;\n    uint16_t _port;\n    struct connection {\n        connected_socket _socket;\n        socket_address _addr;\n        input_stream<char> _in;\n        output_stream<char> _out;\n        ascii_protocol _proto;\n        sharded<system_stats>& _system_stats;\n        connection(connected_socket&& socket, socket_address addr, sharded_cache& c, sharded<system_stats>& system_stats)\n            : _socket(std::move(socket))\n            , _addr(addr)\n            , _in(_socket.input())\n            , _out(_socket.output())\n            , _proto(c, system_stats)\n            , _system_stats(system_stats)\n        {\n            _system_stats.local()._curr_connections++;\n            _system_stats.local()._total_connections++;\n        }\n        ~connection() {\n            _system_stats.local()._curr_connections--;\n        }\n    };\npublic:\n    tcp_server(sharded_cache& cache, sharded<system_stats>& system_stats, uint16_t port = 11211)\n        : _cache(cache)\n        , _system_stats(system_stats)\n        , _port(port)\n    {}\n\n    void start() {\n        listen_options lo;\n        lo.reuse_address = true;\n        _listener = make_lw_shared<seastar::server_socket>(seastar::listen(make_ipv4_address({_port}), lo));\n        // Run in the background until eof has reached on the input connection.\n        _task = keep_doing([this] {\n            return _listener->accept().then([this] (accept_result ar) mutable {\n                connected_socket fd = std::move(ar.connection);\n                socket_address addr = std::move(ar.remote_address);\n                auto conn = make_lw_shared<connection>(std::move(fd), addr, _cache, _system_stats);\n                (void)do_until([conn] { return conn->_in.eof(); }, [conn] {\n                    return conn->_proto.handle(conn->_in, conn->_out).then([conn] {\n                        return conn->_out.flush();\n                    });\n                }).finally([conn] {\n                    return conn->_out.close().finally([conn]{});\n                });\n            });\n        });\n    }\n\n    future<> stop() {\n        _listener->abort_accept();\n        return _task->handle_exception([](std::exception_ptr e) {\n            std::cerr << \"exception in tcp_server \" << e << '\\n';\n        });\n    }\n};\n\nclass stats_printer {\nprivate:\n    timer<> _timer;\n    sharded_cache& _cache;\npublic:\n    stats_printer(sharded_cache& cache)\n        : _cache(cache) {}\n\n    void start() {\n        _timer.set_callback([this] {\n            (void)_cache.stats().then([] (auto stats) {\n                auto gets_total = stats._get_hits + stats._get_misses;\n                auto get_hit_rate = gets_total ? ((double)stats._get_hits * 100 / gets_total) : 0;\n                auto sets_total = stats._set_adds + stats._set_replaces;\n                auto set_replace_rate = sets_total ? ((double)stats._set_replaces * 100/ sets_total) : 0;\n                std::cout << \"items: \" << stats._size << \" \"\n                    << std::setprecision(2) << std::fixed\n                    << \"get: \" << stats._get_hits << \"/\" << gets_total << \" (\" << get_hit_rate << \"%) \"\n                    << \"set: \" << stats._set_replaces << \"/\" << sets_total << \" (\" <<  set_replace_rate << \"%)\";\n                std::cout << std::endl;\n            });\n        });\n        _timer.arm_periodic(std::chrono::seconds(1));\n    }\n\n    future<> stop() { return make_ready_future<>(); }\n};\n\n} /* namespace memcache */\n\nnamespace seastar {\nvoid append_buffers(jagged_array_of_buffers& vb, std::span<temporary_buffer<char>> bufs) {\n    std::vector<temporary_buffer<char>> stable;\n    stable.reserve(bufs.size() + 1);\n    stable.emplace_back(memcache::udp_server::header_size);\n    std::ranges::move(bufs, std::back_inserter(stable));\n    vb.emplace_back(std::move(stable));\n}\n}\n\nint main(int ac, char** av) {\n    sharded<memcache::cache> cache_peers;\n    memcache::sharded_cache cache(cache_peers);\n    sharded<memcache::system_stats> system_stats;\n    sharded<memcache::udp_server> udp_server;\n    sharded<memcache::tcp_server> tcp_server;\n    memcache::stats_printer stats(cache);\n\n    namespace bpo = boost::program_options;\n    app_template app;\n    app.add_options()\n        (\"max-datagram-size\", bpo::value<int>()->default_value(memcache::udp_server::default_max_datagram_size),\n             \"Maximum size of UDP datagram\")\n        (\"max-slab-size\", bpo::value<uint64_t>()->default_value(memcache::default_per_cpu_slab_size/MB),\n             \"Maximum memory to be used for items (value in megabytes) (reclaimer is disabled if set)\")\n        (\"slab-page-size\", bpo::value<uint64_t>()->default_value(memcache::default_slab_page_size/MB),\n             \"Size of slab page (value in megabytes)\")\n        (\"stats\",\n             \"Print basic statistics periodically (every second)\")\n        (\"port\", bpo::value<uint16_t>()->default_value(11211),\n             \"Specify UDP and TCP ports for memcached server to listen on\")\n        ;\n\n    return app.run_deprecated(ac, av, [&] {\n        internal::at_exit([&] { return tcp_server.stop(); });\n        internal::at_exit([&] { return udp_server.stop(); });\n        internal::at_exit([&] { return cache_peers.stop(); });\n        internal::at_exit([&] { return system_stats.stop(); });\n\n        auto&& config = app.configuration();\n        uint16_t port = config[\"port\"].as<uint16_t>();\n        uint64_t per_cpu_slab_size = config[\"max-slab-size\"].as<uint64_t>() * MB;\n        uint64_t slab_page_size = config[\"slab-page-size\"].as<uint64_t>() * MB;\n        return cache_peers.start(std::move(per_cpu_slab_size), std::move(slab_page_size)).then([&system_stats] {\n            return system_stats.start(memcache::clock_type::now());\n        }).then([&] {\n            std::cout << PLATFORM << \" memcached \" << VERSION << \"\\n\";\n            return make_ready_future<>();\n        }).then([&, port] {\n            return tcp_server.start(std::ref(cache), std::ref(system_stats), port);\n        }).then([&tcp_server] {\n            return tcp_server.invoke_on_all(&memcache::tcp_server::start);\n        }).then([&, port] {\n            if (engine().net().has_per_core_namespace()) {\n                return udp_server.start(std::ref(cache), std::ref(system_stats), port);\n            } else {\n                return udp_server.start_single(std::ref(cache), std::ref(system_stats), port);\n            }\n        }).then([&] {\n            return udp_server.invoke_on_all(&memcache::udp_server::set_max_datagram_size,\n                    (size_t)config[\"max-datagram-size\"].as<int>());\n        }).then([&] {\n            return udp_server.invoke_on_all(&memcache::udp_server::start);\n        }).then([&stats, start_stats = config.count(\"stats\")] {\n            if (start_stats) {\n                stats.start();\n            }\n        });\n    });\n}\n"
  },
  {
    "path": "apps/memcached/memcached.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n\nnamespace memcache {\n\nusing namespace seastar;\n\nclass item;\nclass cache;\n\nclass item_key {\nprivate:\n    sstring _key;\n    size_t _hash;\npublic:\n    item_key() = default;\n    item_key(item_key&) = default;\n    item_key(sstring key)\n        : _key(key)\n        , _hash(std::hash<sstring>()(key))\n    {}\n    item_key(item_key&& other)\n        : _key(std::move(other._key))\n        , _hash(other._hash)\n    {\n        other._hash = 0;\n    }\n    size_t hash() const {\n        return _hash;\n    }\n    const sstring& key() const {\n        return _key;\n    }\n    bool operator==(const item_key& other) const {\n        return other._hash == _hash && other._key == _key;\n    }\n    void operator=(item_key&& other) {\n        _key = std::move(other._key);\n        _hash = other._hash;\n        other._hash = 0;\n    }\n};\n\n}\n\nnamespace std {\n\ntemplate <>\nstruct hash<memcache::item_key> {\n    size_t operator()(const memcache::item_key& key) {\n        return key.hash();\n    }\n};\n\n} /* namespace std */\n"
  },
  {
    "path": "apps/memcached/tests/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nif (Seastar_EXECUTE_ONLY_FAST_TESTS)\n  set (memcached_test_args --fast)\nelse ()\n  set (memcached_test_args \"\")\nendif ()\n\nadd_custom_target (app_memcached_test_memcached_run\n  DEPENDS\n    ${memcached_app}\n    ${CMAKE_CURRENT_SOURCE_DIR}/test.py\n    ${CMAKE_CURRENT_SOURCE_DIR}/test_memcached.py\n  COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/test.py --memcached $<TARGET_FILE:app_memcached> ${memcached_test_args}\n  USES_TERMINAL)\n\nadd_test (\n  NAME Seastar.app.memcached.memcached\n  COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target app_memcached_test_memcached_run)\n\nset_tests_properties (Seastar.app.memcached.memcached\n  PROPERTIES\n    TIMEOUT ${Seastar_TEST_TIMEOUT}\n    ENVIRONMENT \"${Seastar_TEST_ENVIRONMENT}\")\n\nadd_executable (app_memcached_test_ascii\n  test_ascii_parser.cc)\n\nadd_dependencies (app_memcached_test_ascii app_memcached)\n\ntarget_include_directories (app_memcached_test_ascii\n  PRIVATE\n    ${CMAKE_CURRENT_SOURCE_DIR}\n    ${Seastar_APP_MEMCACHED_BINARY_DIR}\n    ${Seastar_APP_MEMCACHED_SOURCE_DIR})\n\ntarget_compile_definitions (app_memcached_test_ascii\n  PRIVATE SEASTAR_TESTING_MAIN)\n\ntarget_link_libraries (app_memcached_test_ascii\n  PRIVATE\n    seastar_private\n    seastar_testing)\n\nadd_custom_target (app_memcached_test_ascii_run\n  DEPENDS app_memcached_test_ascii\n  COMMAND app_memcached_test_ascii -- -c 2\n  USES_TERMINAL)\n\nadd_test (\n  NAME Seastar.app.memcached.ascii\n  COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target app_memcached_test_ascii_run)\n\nset_tests_properties (Seastar.app.memcached.ascii\n  PROPERTIES\n    TIMEOUT ${Seastar_TEST_TIMEOUT}\n    ENVIRONMENT \"${Seastar_TEST_ENVIRONMENT}\")\n"
  },
  {
    "path": "apps/memcached/tests/test.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\nimport time\nimport sys\nimport os\nimport argparse\nimport subprocess\n\nDIR_PATH = os.path.dirname(os.path.realpath(__file__))\n\ndef run(args, cmd):\n    mc = subprocess.Popen([args.memcached, '--smp=2'])\n    print('Memcached started.')\n    try:\n        cmdline = [DIR_PATH + '/test_memcached.py'] + cmd\n        if args.fast:\n            cmdline.append('--fast')\n        print('Running: ' + ' '.join(cmdline))\n        subprocess.check_call(cmdline)\n    finally:\n        print('Killing memcached...')\n        mc.terminate();\n        mc.wait()\n        print('Memcached killed.')\n\nif __name__ == \"__main__\":\n    parser = argparse.ArgumentParser(description=\"Seastar test runner\")\n    parser.add_argument('--fast',  action=\"store_true\", help=\"Run only fast tests\")\n    parser.add_argument('--memcached', required=True, help='Path of the memcached executable')\n    args = parser.parse_args()\n\n    run(args, [])\n    run(args, ['-U'])\n"
  },
  {
    "path": "apps/memcached/tests/test_ascii_parser.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <iostream>\n#include <limits>\n#include <seastar/testing/test_case.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/util/memory-data-source.hh>\n#include \"ascii.hh\"\n#include <seastar/core/loop.hh>\n\nusing namespace seastar;\nusing namespace net;\nusing namespace memcache;\n\nusing parser_type = memcache_ascii_parser;\n\nstatic std::vector<temporary_buffer<char>> make_packet(std::vector<std::string> chunks, size_t buffer_size) {\n    std::vector<temporary_buffer<char>> ret;\n    for (auto&& chunk : chunks) {\n        size_t size = chunk.size();\n        for (size_t pos = 0; pos < size; pos += buffer_size) {\n            auto now = std::min(pos + buffer_size, chunk.size()) - pos;\n            ret.emplace_back(chunk.data() + pos, now);\n        }\n    }\n    return ret;\n}\n\nstatic auto parse(std::vector<temporary_buffer<char>>&& bufs) {\n    auto is = make_lw_shared<input_stream<char>>(util::as_input_stream(std::move(bufs)));\n    auto parser = make_lw_shared<parser_type>();\n    parser->init();\n    return is->consume(*parser).then([is, parser] {\n        return make_ready_future<lw_shared_ptr<parser_type>>(parser);\n    });\n}\n\nauto for_each_fragment_size = [] (auto&& func) {\n    static std::vector<int> buffer_sizes = { 100000, 1000, 100, 10, 5, 2, 1 };\n    return do_for_each(buffer_sizes.begin(), buffer_sizes.end(), [func] (size_t buffer_size) {\n        return func([buffer_size] (std::vector<std::string> chunks) {\n            return make_packet(chunks, buffer_size);\n        });\n    });\n};\n\nSEASTAR_TEST_CASE(test_set_command_is_parsed) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return parse(make_packet({\"set key 1 2 3\\r\\nabc\\r\\n\"})).then([] (auto p) {\n            BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n            BOOST_REQUIRE(p->_flags_str == \"1\");\n            BOOST_REQUIRE(p->_expiration == 2);\n            BOOST_REQUIRE(p->_size == 3);\n            BOOST_REQUIRE(p->_size_str == \"3\");\n            BOOST_REQUIRE(p->_key.key() == \"key\");\n            BOOST_REQUIRE(p->_blob == \"abc\");\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_empty_data_is_parsed) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return parse(make_packet({\"set key 1 2 0\\r\\n\\r\\n\"})).then([] (auto p) {\n            BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n            BOOST_REQUIRE(p->_flags_str == \"1\");\n            BOOST_REQUIRE(p->_expiration == 2);\n            BOOST_REQUIRE(p->_size == 0);\n            BOOST_REQUIRE(p->_size_str == \"0\");\n            BOOST_REQUIRE(p->_key.key() == \"key\");\n            BOOST_REQUIRE(p->_blob == \"\");\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_superflous_data_is_an_error) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return parse(make_packet({\"set key 0 0 0\\r\\nasd\\r\\n\"})).then([] (auto p) {\n            BOOST_REQUIRE(p->_state == parser_type::state::error);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_not_enough_data_is_an_error) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return parse(make_packet({\"set key 0 0 3\\r\\n\"})).then([] (auto p) {\n            BOOST_REQUIRE(p->_state == parser_type::state::error);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_u32_parsing) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return make_ready_future<>().then([make_packet] {\n            return parse(make_packet({\"set key 0 0 0\\r\\n\\r\\n\"})).then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_flags_str == \"0\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 12345 0 0\\r\\n\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_flags_str == \"12345\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key -1 0 0\\r\\n\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::error);\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 1-1 0 0\\r\\n\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::error);\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key \" + std::to_string(std::numeric_limits<uint32_t>::max()) + \" 0 0\\r\\n\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_flags_str == to_sstring(std::numeric_limits<uint32_t>::max()));\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_parsing_of_split_data) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return make_ready_future<>()\n                .then([make_packet] {\n            return parse(make_packet({\"set key 11\", \"1 222 3\\r\\nasd\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 11\", \"1 22\", \"2 3\", \"\\r\\nasd\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set k\", \"ey 11\", \"1 2\", \"2\", \"2 3\", \"\\r\\nasd\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 111 222 3\\r\\n\", \"asd\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 111 222 3\\r\\na\", \"sd\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 111 222 3\\r\\nasd\", \"\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"set key 111 222 3\\r\\nasd\\r\", \"\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key\");\n                BOOST_REQUIRE(p->_flags_str == \"111\");\n                BOOST_REQUIRE(p->_expiration == 222);\n                BOOST_REQUIRE(p->_size == 3);\n                BOOST_REQUIRE(p->_size_str == \"3\");\n                BOOST_REQUIRE(p->_blob == \"asd\");\n            });\n        });\n    });\n}\n\nstatic std::vector<sstring> as_strings(std::vector<item_key>& keys) {\n    std::vector<sstring> v;\n    for (auto&& key : keys) {\n        v.push_back(key.key());\n    }\n    return v;\n}\n\nSEASTAR_TEST_CASE(test_get_parsing) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return make_ready_future<>()\n                .then([make_packet] {\n            return parse(make_packet({\"get key1\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_get);\n                BOOST_REQUIRE_EQUAL(as_strings(p->_keys), std::vector<sstring>({\"key1\"}));\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"get key1 key2\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_get);\n                BOOST_REQUIRE_EQUAL(as_strings(p->_keys), std::vector<sstring>({\"key1\", \"key2\"}));\n            });\n        }).then([make_packet] {\n            return parse(make_packet({\"get key1 key2 key3\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_get);\n                BOOST_REQUIRE_EQUAL(as_strings(p->_keys), std::vector<sstring>({\"key1\", \"key2\", \"key3\"}));\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_catches_errors_in_get) {\n    return for_each_fragment_size([] (auto make_packet) {\n        return make_ready_future<>()\n                .then([make_packet] {\n            return parse(make_packet({\"get\\r\\n\"}))\n                    .then([] (auto p) {\n                BOOST_REQUIRE(p->_state == parser_type::state::error);\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_parser_returns_eof_state_when_no_command_follows) {\n    return for_each_fragment_size([] (auto make_packet) {\n        auto p = make_shared<parser_type>();\n        auto is = make_shared<input_stream<char>>(util::as_input_stream(make_packet({\"get key\\r\\n\"})));\n        p->init();\n        return is->consume(*p).then([p] {\n            BOOST_REQUIRE(p->_state == parser_type::state::cmd_get);\n        }).then([is, p] {\n            p->init();\n            return is->consume(*p).then([p, is] {\n                BOOST_REQUIRE(p->_state == parser_type::state::eof);\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_incomplete_command_is_an_error) {\n    return for_each_fragment_size([] (auto make_packet) {\n        auto p = make_shared<parser_type>();\n        auto is = make_shared<input_stream<char>>(util::as_input_stream(make_packet({\"get\"})));\n        p->init();\n        return is->consume(*p).then([p] {\n            BOOST_REQUIRE(p->_state == parser_type::state::error);\n        }).then([is, p] {\n            p->init();\n            return is->consume(*p).then([p, is] {\n                BOOST_REQUIRE(p->_state == parser_type::state::eof);\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_multiple_requests_in_one_stream) {\n    return for_each_fragment_size([] (auto make_packet) {\n        auto p = make_shared<parser_type>();\n        auto is = make_shared<input_stream<char>>(util::as_input_stream(make_packet({\"set key1 1 1 5\\r\\ndata1\\r\\nset key2 2 2 6\\r\\ndata2+\\r\\n\"})));\n        p->init();\n        return is->consume(*p).then([p] {\n            BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n            BOOST_REQUIRE(p->_key.key() == \"key1\");\n            BOOST_REQUIRE(p->_flags_str == \"1\");\n            BOOST_REQUIRE(p->_expiration == 1);\n            BOOST_REQUIRE(p->_size == 5);\n            BOOST_REQUIRE(p->_size_str == \"5\");\n            BOOST_REQUIRE(p->_blob == \"data1\");\n        }).then([is, p] {\n            p->init();\n            return is->consume(*p).then([p, is] {\n                BOOST_REQUIRE(p->_state == parser_type::state::cmd_set);\n                BOOST_REQUIRE(p->_key.key() == \"key2\");\n                BOOST_REQUIRE(p->_flags_str == \"2\");\n                BOOST_REQUIRE(p->_expiration == 2);\n                BOOST_REQUIRE(p->_size == 6);\n                BOOST_REQUIRE(p->_size_str == \"6\");\n                BOOST_REQUIRE(p->_blob == \"data2+\");\n            });\n        });\n    });\n}\n"
  },
  {
    "path": "apps/memcached/tests/test_memcached.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\nfrom contextlib import contextmanager\nimport socket\nimport struct\nimport sys\nimport random\nimport argparse\nimport time\nimport re\nimport unittest\n\nserver_addr = None\ncall = None\nargs = None\n\nclass TimeoutError(Exception):\n    pass\n\n@contextmanager\ndef tcp_connection(timeout=1):\n    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n    s.settimeout(timeout)\n    s.connect(server_addr)\n    def call(msg):\n        s.send(msg.encode())\n        return s.recv(16*1024)\n    yield call\n    s.close()\n\ndef slow(f):\n    def wrapper(self):\n        if args.fast:\n            raise unittest.SkipTest('Slow')\n        return f(self)\n    return wrapper\n\ndef recv_all(s):\n    m = b''\n    while True:\n        data = s.recv(1024)\n        if not data:\n            break\n        m += data\n    return m\n\ndef tcp_call(msg, timeout=1):\n    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n    s.settimeout(timeout)\n    s.connect(server_addr)\n    s.send(msg.encode())\n    s.shutdown(socket.SHUT_WR)\n    data = recv_all(s)\n    s.close()\n    return data\n\ndef udp_call_for_fragments(msg, timeout=1):\n    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n    sock.settimeout(timeout)\n    this_req_id = random.randint(-32768, 32767)\n\n    datagram = struct.pack(\">hhhh\", this_req_id, 0, 1, 0) + msg.encode()\n    sock.sendto(datagram, server_addr)\n\n    messages = {}\n    n_determined = None\n    while True:\n        data, addr = sock.recvfrom(1500)\n        req_id, seq, n, res = struct.unpack_from(\">hhhh\", data)\n        content = data[8:]\n\n        if n_determined and n_determined != n:\n            raise Exception('Inconsitent number of total messages, %d and %d' % (n_determined, n))\n        n_determined = n\n\n        if req_id != this_req_id:\n            raise Exception('Invalid request id: ' + req_id + ', expected ' + this_req_id)\n\n        if seq in messages:\n            raise Exception('Duplicate message for seq=' + seq)\n\n        messages[seq] = content\n        if len(messages) == n:\n            break\n\n    for k, v in sorted(messages.items(), key=lambda e: e[0]):\n        yield v\n\n    sock.close()\n\ndef udp_call(msg, **kwargs):\n    return b''.join(udp_call_for_fragments(msg, **kwargs))\n\nclass MemcacheTest(unittest.TestCase):\n    def set(self, key, value, flags=0, expiry=0):\n        self.assertEqual(call('set %s %d %d %d\\r\\n%s\\r\\n' % (key, flags, expiry, len(value), value)), b'STORED\\r\\n')\n\n    def delete(self, key):\n        self.assertEqual(call('delete %s\\r\\n' % key), b'DELETED\\r\\n')\n\n    def assertHasKey(self, key):\n        resp = call('get %s\\r\\n' % key)\n        if not resp.startswith(('VALUE %s' % key).encode()):\n            self.fail('Key \\'%s\\' should be present, but got: %s' % (key, resp.decode()))\n\n    def assertNoKey(self, key):\n        resp = call('get %s\\r\\n' % key)\n        if resp != b'END\\r\\n':\n            self.fail('Key \\'%s\\' should not be present, but got: %s' % (key, resp.decode()))\n\n    def setKey(self, key):\n        self.set(key, 'some value')\n\n    def getItemVersion(self, key):\n        m = re.match(r'VALUE %s \\d+ \\d+ (?P<version>\\d+)' % key, call('gets %s\\r\\n' % key).decode())\n        return int(m.group('version'))\n\n    def getStat(self, name, call_fn=None):\n        if not call_fn: call_fn = call\n        resp = call_fn('stats\\r\\n').decode()\n        m = re.search(r'STAT %s (?P<value>.+)' % re.escape(name), resp, re.MULTILINE)\n        return m.group('value')\n\n    def flush(self):\n        self.assertEqual(call('flush_all\\r\\n'), b'OK\\r\\n')\n\n    def tearDown(self):\n        self.flush()\n\nclass TcpSpecificTests(MemcacheTest):\n    def test_recovers_from_errors_in_the_stream(self):\n        with tcp_connection() as conn:\n            self.assertEqual(conn('get\\r\\n'), b'ERROR\\r\\n')\n            self.assertEqual(conn('get key\\r\\n'), b'END\\r\\n')\n\n    def test_incomplete_command_results_in_error(self):\n        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n        s.connect(server_addr)\n        s.send(b'get')\n        s.shutdown(socket.SHUT_WR)\n        self.assertEqual(recv_all(s), b'ERROR\\r\\n')\n        s.close()\n\n    def test_stream_closed_results_in_error(self):\n        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n        s.connect(server_addr)\n        s.shutdown(socket.SHUT_WR)\n        self.assertEqual(recv_all(s), b'')\n        s.close()\n\n    def test_unsuccesful_parsing_does_not_leave_data_behind(self):\n        with tcp_connection() as conn:\n            self.assertEqual(conn('set key 0 0 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n            self.assertRegex(conn('delete a b c\\r\\n'), b'^(CLIENT_)?ERROR.*\\r\\n$')\n            self.assertEqual(conn('get key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n            self.assertEqual(conn('delete key\\r\\n'), b'DELETED\\r\\n')\n\n    def test_flush_all_no_reply(self):\n        self.assertEqual(call('flush_all noreply\\r\\n'), b'')\n\n    def test_set_no_reply(self):\n        self.assertEqual(call('set key 0 0 5 noreply\\r\\nhello\\r\\nget key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n        self.delete('key')\n\n    def test_delete_no_reply(self):\n        self.setKey('key')\n        self.assertEqual(call('delete key noreply\\r\\nget key\\r\\n'), b'END\\r\\n')\n\n    def test_add_no_reply(self):\n        self.assertEqual(call('add key 0 0 1 noreply\\r\\na\\r\\nget key\\r\\n'), b'VALUE key 0 1\\r\\na\\r\\nEND\\r\\n')\n        self.delete('key')\n\n    def test_replace_no_reply(self):\n        self.assertEqual(call('set key 0 0 1\\r\\na\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('replace key 0 0 1 noreply\\r\\nb\\r\\nget key\\r\\n'), b'VALUE key 0 1\\r\\nb\\r\\nEND\\r\\n')\n        self.delete('key')\n\n    def test_cas_noreply(self):\n        self.assertNoKey('key')\n        self.assertEqual(call('cas key 0 0 1 1 noreply\\r\\na\\r\\n'), b'')\n        self.assertNoKey('key')\n\n        self.assertEqual(call('add key 0 0 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        version = self.getItemVersion('key')\n\n        self.assertEqual(call('cas key 1 0 5 %d noreply\\r\\naloha\\r\\n' % (version + 1)), b'')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n\n        self.assertEqual(call('cas key 1 0 5 %d noreply\\r\\naloha\\r\\n' % (version)), b'')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 1 5\\r\\naloha\\r\\nEND\\r\\n')\n\n        self.delete('key')\n\n    @slow\n    def test_connection_statistics(self):\n        with tcp_connection() as conn:\n            # Issue some command to ensure server has fully processed this\n            # connection, otherwise our stats query will race with the server\n            # userspace accepting the connection and registering it\n            conn('version\\r\\n')\n            curr_connections = int(self.getStat('curr_connections', call_fn=conn))\n            total_connections = int(self.getStat('total_connections', call_fn=conn))\n            with tcp_connection() as conn2:\n                conn2('version\\r\\n')\n                self.assertEqual(curr_connections + 1, int(self.getStat('curr_connections', call_fn=conn)))\n                self.assertEqual(total_connections + 1, int(self.getStat('total_connections', call_fn=conn)))\n            self.assertEqual(total_connections + 1, int(self.getStat('total_connections', call_fn=conn)))\n            time.sleep(0.1)\n            self.assertEqual(curr_connections, int(self.getStat('curr_connections', call_fn=conn)))\n\nclass UdpSpecificTests(MemcacheTest):\n    def test_large_response_is_split_into_mtu_chunks(self):\n        max_datagram_size = 1400\n        data = '1' * (max_datagram_size*3)\n        self.set('key', data)\n\n        chunks = list(udp_call_for_fragments('get key\\r\\n'))\n\n        for chunk in chunks:\n            self.assertLessEqual(len(chunk), max_datagram_size)\n\n        self.assertEqual(b''.join(chunks).decode(),\n            'VALUE key 0 %d\\r\\n%s\\r\\n' \\\n            'END\\r\\n' % (len(data), data))\n\n        self.delete('key')\n\nclass TestCommands(MemcacheTest):\n    def test_basic_commands(self):\n        self.assertEqual(call('get key\\r\\n'), b'END\\r\\n')\n        self.assertEqual(call('set key 0 0 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n        self.assertEqual(call('delete key\\r\\n'), b'DELETED\\r\\n')\n        self.assertEqual(call('delete key\\r\\n'), b'NOT_FOUND\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'END\\r\\n')\n\n    def test_error_handling(self):\n        self.assertEqual(call('get\\r\\n'), b'ERROR\\r\\n')\n\n    @slow\n    def test_expiry(self):\n        self.assertEqual(call('set key 0 1 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n        time.sleep(2)\n        self.assertEqual(call('get key\\r\\n'), b'END\\r\\n')\n\n    @slow\n    def test_expiry_at_epoch_time(self):\n        expiry = int(time.time()) + 2\n        self.assertEqual(call('set key 0 %d 5\\r\\nhello\\r\\n' % expiry), b'STORED\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 5\\r\\nhello\\r\\nEND\\r\\n')\n        time.sleep(3)\n        self.assertEqual(call('get key\\r\\n'), b'END\\r\\n')\n\n    def test_multiple_keys_in_get(self):\n        self.assertEqual(call('set key1 0 0 2\\r\\nv1\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('set key 0 0 2\\r\\nv2\\r\\n'), b'STORED\\r\\n')\n        resp = call('get key1 key\\r\\n')\n        self.assertRegex(resp, b'^(VALUE key1 0 2\\r\\nv1\\r\\nVALUE key 0 2\\r\\nv2\\r\\nEND\\r\\n)|(VALUE key 0 2\\r\\nv2\\r\\nVALUE key1 0 2\\r\\nv1\\r\\nEND\\r\\n)$')\n        self.delete(\"key\")\n        self.delete(\"key1\")\n\n    def test_flush_all(self):\n        self.set('key', 'value')\n        self.assertEqual(call('flush_all\\r\\n'), b'OK\\r\\n')\n        self.assertNoKey('key')\n\n    def test_keys_set_after_flush_remain(self):\n        self.assertEqual(call('flush_all\\r\\n'), b'OK\\r\\n')\n        self.setKey('key')\n        self.assertHasKey('key')\n        self.delete('key')\n\n    @slow\n    def test_flush_all_with_timeout_flushes_all_keys_even_those_set_after_flush(self):\n        self.setKey('key')\n        self.assertEqual(call('flush_all 2\\r\\n'), b'OK\\r\\n')\n        self.assertHasKey('key')\n        self.setKey('key2')\n        time.sleep(3)\n        self.assertNoKey('key')\n        self.assertNoKey('key2')\n\n    @slow\n    def test_subsequent_flush_is_merged(self):\n        self.setKey('key')\n        self.assertEqual(call('flush_all 2\\r\\n'), b'OK\\r\\n') # Can flush in anything between 1-2\n        self.assertEqual(call('flush_all 4\\r\\n'), b'OK\\r\\n') # Can flush in anything between 3-4\n        time.sleep(3)\n        self.assertHasKey('key')\n        self.setKey('key2')\n        time.sleep(4)\n        self.assertNoKey('key')\n        self.assertNoKey('key2')\n\n    @slow\n    def test_immediate_flush_cancels_delayed_flush(self):\n        self.assertEqual(call('flush_all 2\\r\\n'), b'OK\\r\\n')\n        self.assertEqual(call('flush_all\\r\\n'), b'OK\\r\\n')\n        self.setKey('key')\n        time.sleep(1)\n        self.assertHasKey('key')\n        self.delete('key')\n\n    @slow\n    def test_flushing_in_the_past(self):\n        self.setKey('key1')\n        time.sleep(1)\n        self.setKey('key2')\n        key2_time = int(time.time())\n        self.assertEqual(call('flush_all %d\\r\\n' % (key2_time - 1)), b'OK\\r\\n')\n        time.sleep(1)\n        self.assertNoKey(\"key1\")\n        self.assertNoKey(\"key2\")\n\n    @slow\n    def test_memcache_does_not_crash_when_flushing_with_already_expred_items(self):\n        self.assertEqual(call('set key1 0 2 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        time.sleep(1)\n        self.assertEqual(call('flush_all\\r\\n'), b'OK\\r\\n')\n\n    def test_response_spanning_many_datagrams(self):\n        key1_data = '1' * 1000\n        key2_data = '2' * 1000\n        key3_data = '3' * 1000\n        self.set('key1', key1_data)\n        self.set('key2', key2_data)\n        self.set('key3', key3_data)\n\n        resp = call('get key1 key2 key3\\r\\n').decode()\n\n        pattern = '^VALUE (?P<v1>.*?\\r\\n.*?)\\r\\nVALUE (?P<v2>.*?\\r\\n.*?)\\r\\nVALUE (?P<v3>.*?\\r\\n.*?)\\r\\nEND\\r\\n$'\n        self.assertRegex(resp, pattern)\n\n        m = re.match(pattern, resp)\n        self.assertEqual(set([m.group('v1'), m.group('v2'), m.group('v3')]),\n            set(['key1 0 %d\\r\\n%s' % (len(key1_data), key1_data),\n                'key2 0 %d\\r\\n%s' % (len(key2_data), key2_data),\n                'key3 0 %d\\r\\n%s' % (len(key3_data), key3_data)]))\n\n        self.delete('key1')\n        self.delete('key2')\n        self.delete('key3')\n\n    def test_version(self):\n        self.assertRegex(call('version\\r\\n'), b'^VERSION .*\\r\\n$')\n\n    def test_add(self):\n        self.assertEqual(call('add key 0 0 1\\r\\na\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('add key 0 0 1\\r\\na\\r\\n'), b'NOT_STORED\\r\\n')\n        self.delete('key')\n\n    def test_replace(self):\n        self.assertEqual(call('add key 0 0 1\\r\\na\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('replace key 0 0 1\\r\\na\\r\\n'), b'STORED\\r\\n')\n        self.delete('key')\n        self.assertEqual(call('replace key 0 0 1\\r\\na\\r\\n'), b'NOT_STORED\\r\\n')\n\n    def test_cas_and_gets(self):\n        self.assertEqual(call('cas key 0 0 1 1\\r\\na\\r\\n'), b'NOT_FOUND\\r\\n')\n        self.assertEqual(call('add key 0 0 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        version = self.getItemVersion('key')\n\n        self.assertEqual(call('set key 1 0 5\\r\\nhello\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('gets key\\r\\n').decode(), 'VALUE key 1 5 %d\\r\\nhello\\r\\nEND\\r\\n' % (version + 1))\n\n        self.assertEqual(call('cas key 0 0 5 %d\\r\\nhello\\r\\n' % (version)), b'EXISTS\\r\\n')\n        self.assertEqual(call('cas key 0 0 5 %d\\r\\naloha\\r\\n' % (version + 1)), b'STORED\\r\\n')\n        self.assertEqual(call('gets key\\r\\n').decode(), 'VALUE key 0 5 %d\\r\\naloha\\r\\nEND\\r\\n' % (version + 2))\n\n        self.delete('key')\n\n    def test_curr_items_stat(self):\n        self.assertEqual(0, int(self.getStat('curr_items')))\n        self.setKey('key')\n        self.assertEqual(1, int(self.getStat('curr_items')))\n        self.delete('key')\n        self.assertEqual(0, int(self.getStat('curr_items')))\n\n    def test_how_stats_change_with_different_commands(self):\n        get_count = int(self.getStat('cmd_get'))\n        set_count = int(self.getStat('cmd_set'))\n        flush_count = int(self.getStat('cmd_flush'))\n        total_items = int(self.getStat('total_items'))\n        get_misses = int(self.getStat('get_misses'))\n        get_hits = int(self.getStat('get_hits'))\n        cas_hits = int(self.getStat('cas_hits'))\n        cas_badval = int(self.getStat('cas_badval'))\n        cas_misses = int(self.getStat('cas_misses'))\n        delete_misses = int(self.getStat('delete_misses'))\n        delete_hits = int(self.getStat('delete_hits'))\n        curr_connections = int(self.getStat('curr_connections'))\n        incr_hits = int(self.getStat('incr_hits'))\n        incr_misses = int(self.getStat('incr_misses'))\n        decr_hits = int(self.getStat('decr_hits'))\n        decr_misses = int(self.getStat('decr_misses'))\n\n        call('get key\\r\\n')\n        get_count += 1\n        get_misses += 1\n\n        call('gets key\\r\\n')\n        get_count += 1\n        get_misses += 1\n\n        call('set key1 0 0 1\\r\\na\\r\\n')\n        set_count += 1\n        total_items += 1\n\n        call('get key1\\r\\n')\n        get_count += 1\n        get_hits += 1\n\n        call('add key1 0 0 1\\r\\na\\r\\n')\n        set_count += 1\n\n        call('add key2 0 0 1\\r\\na\\r\\n')\n        set_count += 1\n        total_items += 1\n\n        call('replace key1 0 0 1\\r\\na\\r\\n')\n        set_count += 1\n        total_items += 1\n\n        call('replace key3 0 0 1\\r\\na\\r\\n')\n        set_count += 1\n\n        call('cas key4 0 0 1 1\\r\\na\\r\\n')\n        set_count += 1\n        cas_misses += 1\n\n        call('cas key1 0 0 1 %d\\r\\na\\r\\n' % self.getItemVersion('key1'))\n        set_count += 1\n        get_count += 1\n        get_hits += 1\n        cas_hits += 1\n        total_items += 1\n\n        call('cas key1 0 0 1 %d\\r\\na\\r\\n' % (self.getItemVersion('key1') + 1))\n        set_count += 1\n        get_count += 1\n        get_hits += 1\n        cas_badval += 1\n\n        call('delete key1\\r\\n')\n        delete_hits += 1\n\n        call('delete key1\\r\\n')\n        delete_misses += 1\n\n        call('incr num 1\\r\\n')\n        incr_misses += 1\n        call('decr num 1\\r\\n')\n        decr_misses += 1\n\n        call('set num 0 0 1\\r\\n0\\r\\n')\n        set_count += 1\n        total_items += 1\n\n        call('incr num 1\\r\\n')\n        incr_hits += 1\n        call('decr num 1\\r\\n')\n        decr_hits += 1\n\n        self.flush()\n        flush_count += 1\n\n        self.assertEqual(get_count, int(self.getStat('cmd_get')))\n        self.assertEqual(set_count, int(self.getStat('cmd_set')))\n        self.assertEqual(flush_count, int(self.getStat('cmd_flush')))\n        self.assertEqual(total_items, int(self.getStat('total_items')))\n        self.assertEqual(get_hits, int(self.getStat('get_hits')))\n        self.assertEqual(get_misses, int(self.getStat('get_misses')))\n        self.assertEqual(cas_misses, int(self.getStat('cas_misses')))\n        self.assertEqual(cas_hits, int(self.getStat('cas_hits')))\n        self.assertEqual(cas_badval, int(self.getStat('cas_badval')))\n        self.assertEqual(delete_misses, int(self.getStat('delete_misses')))\n        self.assertEqual(delete_hits, int(self.getStat('delete_hits')))\n        self.assertEqual(0, int(self.getStat('curr_items')))\n        self.assertEqual(curr_connections, int(self.getStat('curr_connections')))\n        self.assertEqual(incr_misses, int(self.getStat('incr_misses')))\n        self.assertEqual(incr_hits, int(self.getStat('incr_hits')))\n        self.assertEqual(decr_misses, int(self.getStat('decr_misses')))\n        self.assertEqual(decr_hits, int(self.getStat('decr_hits')))\n\n    def test_incr(self):\n        self.assertEqual(call('incr key 0\\r\\n'), b'NOT_FOUND\\r\\n')\n\n        self.assertEqual(call('set key 0 0 1\\r\\n0\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('incr key 0\\r\\n'), b'0\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 1\\r\\n0\\r\\nEND\\r\\n')\n\n        self.assertEqual(call('incr key 1\\r\\n'), b'1\\r\\n')\n        self.assertEqual(call('incr key 2\\r\\n'), b'3\\r\\n')\n        self.assertEqual(call('incr key %d\\r\\n' % (pow(2, 64) - 1)), b'2\\r\\n')\n        self.assertEqual(call('incr key %d\\r\\n' % (pow(2, 64) - 3)), b'18446744073709551615\\r\\n')\n        self.assertRegex(call('incr key 1\\r\\n').decode(), r'0(\\w+)?\\r\\n')\n\n        self.assertEqual(call('set key 0 0 2\\r\\n1 \\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('incr key 1\\r\\n'), b'2\\r\\n')\n\n        self.assertEqual(call('set key 0 0 2\\r\\n09\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('incr key 1\\r\\n'), b'10\\r\\n')\n\n    def test_decr(self):\n        self.assertEqual(call('decr key 0\\r\\n'), b'NOT_FOUND\\r\\n')\n\n        self.assertEqual(call('set key 0 0 1\\r\\n7\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('decr key 1\\r\\n'), b'6\\r\\n')\n        self.assertEqual(call('get key\\r\\n'), b'VALUE key 0 1\\r\\n6\\r\\nEND\\r\\n')\n\n        self.assertEqual(call('decr key 6\\r\\n'), b'0\\r\\n')\n        self.assertEqual(call('decr key 2\\r\\n'), b'0\\r\\n')\n\n        self.assertEqual(call('set key 0 0 2\\r\\n20\\r\\n'), b'STORED\\r\\n')\n        self.assertRegex(call('decr key 11\\r\\n').decode(), r'^9( )?\\r\\n$')\n\n        self.assertEqual(call('set key 0 0 3\\r\\n100\\r\\n'), b'STORED\\r\\n')\n        self.assertRegex(call('decr key 91\\r\\n').decode(), r'^9(  )?\\r\\n$')\n\n        self.assertEqual(call('set key 0 0 2\\r\\n1 \\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('decr key 1\\r\\n'), b'0\\r\\n')\n\n        self.assertEqual(call('set key 0 0 2\\r\\n09\\r\\n'), b'STORED\\r\\n')\n        self.assertEqual(call('decr key 1\\r\\n'), b'8\\r\\n')\n\n    def test_incr_and_decr_on_invalid_input(self):\n        error_msg = b'CLIENT_ERROR cannot increment or decrement non-numeric value\\r\\n'\n        for cmd in ['incr', 'decr']:\n            for value in ['', '-1', 'a', '0x1', '18446744073709551616']:\n                self.assertEqual(call('set key 0 0 %d\\r\\n%s\\r\\n' % (len(value), value)), b'STORED\\r\\n')\n                prev = call('get key\\r\\n')\n                self.assertEqual(call(cmd + ' key 1\\r\\n'), error_msg, \"cmd=%s, value=%s\" % (cmd, value))\n                self.assertEqual(call('get key\\r\\n'), prev)\n                self.delete('key')\n\ndef wait_for_memcache_tcp(timeout=4):\n    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n    timeout_at = time.time() + timeout\n    while True:\n        if time.time() >= timeout_at:\n            raise TimeoutError()\n        try:\n            s.connect(server_addr)\n            s.close()\n            break\n        except ConnectionRefusedError:\n            time.sleep(0.1)\n\n\ndef wait_for_memcache_udp(timeout=4):\n    timeout_at = time.time() + timeout\n    while True:\n        if time.time() >= timeout_at:\n            raise TimeoutError()\n        try:\n            udp_call('version\\r\\n', timeout=0.2)\n            break\n        except socket.timeout:\n            pass\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser(description=\"memcache protocol tests\")\n    parser.add_argument('--server', '-s', action=\"store\", help=\"server adddress in <host>:<port> format\", default=\"localhost:11211\")\n    parser.add_argument('--udp', '-U', action=\"store_true\", help=\"Use UDP protocol\")\n    parser.add_argument('--fast',  action=\"store_true\", help=\"Run only fast tests\")\n    args = parser.parse_args()\n\n    host, port = args.server.split(':')\n    server_addr = (host, int(port))\n\n    if args.udp:\n        call = udp_call\n        wait_for_memcache_udp()\n    else:\n        call = tcp_call\n        wait_for_memcache_tcp()\n\n    runner = unittest.TextTestRunner()\n    loader = unittest.TestLoader()\n    suite = unittest.TestSuite()\n    suite.addTest(loader.loadTestsFromTestCase(TestCommands))\n    if args.udp:\n        suite.addTest(loader.loadTestsFromTestCase(UdpSpecificTests))\n    else:\n        suite.addTest(loader.loadTestsFromTestCase(TcpSpecificTests))\n    result = runner.run(suite)\n    if not result.wasSuccessful():\n        sys.exit(1)\n"
  },
  {
    "path": "apps/rpc_tester/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2022 Scylladb, Ltd.\n#\n\nseastar_add_app (rpc_tester\n  SOURCES rpc_tester.cc)\n\ntarget_link_libraries (app_rpc_tester\n  PRIVATE yaml-cpp::yaml-cpp)\n"
  },
  {
    "path": "apps/rpc_tester/rpc_tester.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#include <iostream>\n#include <vector>\n#include <chrono>\n#include <random>\n#include <ranges>\n#include <yaml-cpp/yaml.h>\n#include <fmt/core.h>\n#pragma GCC diagnostic push\n// see https://github.com/boostorg/accumulators/pull/54\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n#include <boost/accumulators/accumulators.hpp>\n#include <boost/accumulators/statistics/stats.hpp>\n#include <boost/accumulators/statistics/max.hpp>\n#include <boost/accumulators/statistics/mean.hpp>\n#include <boost/accumulators/statistics/p_square_quantile.hpp>\n#include <boost/accumulators/statistics/extended_p_square.hpp>\n#include <boost/accumulators/statistics/extended_p_square_quantile.hpp>\n#pragma GCC diagnostic pop\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/rpc/rpc.hh>\n#include <seastar/util/assert.hh>\n\nusing namespace seastar;\nusing namespace boost::accumulators;\nusing namespace std::chrono_literals;\n\nstruct serializer {};\n\ntemplate <typename T, typename Output>\ninline\nvoid write_arithmetic_type(Output& out, T v) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    return out.write(reinterpret_cast<const char*>(&v), sizeof(T));\n}\n\ntemplate <typename T, typename Input>\ninline\nT read_arithmetic_type(Input& in) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    T v;\n    in.read(reinterpret_cast<char*>(&v), sizeof(T));\n    return v;\n}\n\ntemplate <typename Output>\ninline void write(serializer, Output& output, int32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, int64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, double v) { return write_arithmetic_type(output, v); }\ntemplate <typename Input>\ninline int32_t read(serializer, Input& input, rpc::type<int32_t>) { return read_arithmetic_type<int32_t>(input); }\ntemplate <typename Input>\ninline uint32_t read(serializer, Input& input, rpc::type<uint32_t>) { return read_arithmetic_type<uint32_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<uint64_t>) { return read_arithmetic_type<uint64_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<int64_t>) { return read_arithmetic_type<int64_t>(input); }\ntemplate <typename Input>\ninline double read(serializer, Input& input, rpc::type<double>) { return read_arithmetic_type<double>(input); }\n\ntemplate <typename Output>\ninline void write(serializer, Output& out, const sstring& v) {\n    write_arithmetic_type(out, uint32_t(v.size()));\n    out.write(v.c_str(), v.size());\n}\n\ntemplate <typename Input>\ninline sstring read(serializer, Input& in, rpc::type<sstring>) {\n    auto size = read_arithmetic_type<uint32_t>(in);\n    sstring ret = uninitialized_string(size);\n    in.read(ret.data(), size);\n    return ret;\n}\n\nusing payload_t = std::vector<uint64_t>;\n\ntemplate <typename Output>\ninline void write(serializer, Output& out, const payload_t& v) {\n    write_arithmetic_type(out, uint32_t(v.size()));\n    out.write((const char*)v.data(), v.size() * sizeof(payload_t::value_type));\n}\n\ntemplate <typename Input>\ninline payload_t read(serializer, Input& in, rpc::type<payload_t>) {\n    auto size = read_arithmetic_type<uint32_t>(in);\n    payload_t ret;\n    ret.resize(size);\n    in.read((char*)ret.data(), size * sizeof(payload_t::value_type));\n    return ret;\n}\n\nclass pause_distribution {\npublic:\n\n    virtual std::chrono::duration<double> get() = 0;\n\n    template <typename Dur>\n    Dur get_as() {\n        return std::chrono::duration_cast<Dur>(get());\n    }\n\n    virtual ~pause_distribution() {}\n};\n\nclass steady_process : public pause_distribution {\n    std::chrono::duration<double> _pause;\npublic:\n    steady_process(std::chrono::duration<double> period) : _pause(period) { }\n    std::chrono::duration<double> get() override { return _pause; }\n};\n\nstd::unique_ptr<pause_distribution> make_steady_pause(std::chrono::duration<double> d) {\n    return std::make_unique<steady_process>(d);\n}\n\nclass uniform_process : public pause_distribution {\n    std::random_device _rd;\n    std::mt19937 _rng;\n    std::uniform_real_distribution<double> _range;\n\npublic:\n    uniform_process(std::chrono::duration<double> min, std::chrono::duration<double> max)\n            : _rng(_rd()) , _range(min.count(), max.count()) { }\n\n    std::chrono::duration<double> get() override {\n        return std::chrono::duration<double>(_range(_rng));\n    }\n};\n\nstruct duration_range {\n    std::chrono::duration<double> min;\n    std::chrono::duration<double> max;\n};\n\nstd::unique_ptr<pause_distribution> make_uniform_pause(duration_range range) {\n    return std::make_unique<uniform_process>(range.min, range.max);\n}\n\nstruct client_config {\n    bool nodelay = true;\n};\n\nstruct server_config {\n    bool nodelay = true;\n};\n\nstruct job_config {\n    std::string name;\n    std::string type;\n    std::string verb;\n    unsigned parallelism;\n    unsigned shares = 100;\n    std::chrono::duration<double> exec_time;\n    std::optional<duration_range> exec_time_range;\n    std::optional<std::chrono::duration<double>> sleep_time;\n    std::optional<duration_range> sleep_time_range;\n    std::optional<std::chrono::duration<double>> timeout;\n    size_t payload;\n\n    bool client = false;\n    bool server = false;\n\n    std::chrono::seconds duration;\n    std::string sg_name;\n    scheduling_group sg = default_scheduling_group();\n};\n\nstruct config {\n    client_config client;\n    server_config server;\n    std::vector<job_config> jobs;\n};\n\nstruct duration_time {\n    std::chrono::duration<float> time;\n};\n\nstruct byte_size {\n    uint64_t size;\n};\n\nnamespace YAML {\n\ntemplate<>\nstruct convert<client_config> {\n    static bool decode(const Node& node, client_config& cfg) {\n        if (node[\"nodelay\"]) {\n            cfg.nodelay = node[\"nodelay\"].as<bool>();\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<server_config> {\n    static bool decode(const Node& node, server_config& cfg) {\n        if (node[\"nodelay\"]) {\n            cfg.nodelay = node[\"nodelay\"].as<bool>();\n        }\n        return true;\n    }\n};\n\ntemplate <>\nstruct convert<job_config> {\n    static bool decode(const Node& node, job_config& cfg) {\n        cfg.name = node[\"name\"].as<std::string>();\n        cfg.type = node[\"type\"].as<std::string>();\n        cfg.parallelism = node[\"parallelism\"].as<unsigned>();\n        if (cfg.type == \"rpc\" || cfg.type == \"rpc_streaming\") {\n            cfg.verb = node[\"verb\"].as<std::string>();\n            cfg.payload = node[\"payload\"].as<byte_size>().size;\n            cfg.client = true;\n            if (node[\"sleep_time\"]) {\n                cfg.sleep_time = node[\"sleep_time\"].as<duration_time>().time;\n            }\n\n            if (cfg.type == \"rpc\" && node[\"timeout\"]) {\n                cfg.timeout = node[\"timeout\"].as<duration_time>().time;\n            }\n        } else if (cfg.type == \"cpu\") {\n            if (node[\"execution_time\"]) {\n                cfg.exec_time = node[\"execution_time\"].as<duration_time>().time;\n            } else {\n                duration_range r;\n                r.min = node[\"execution_time_min\"].as<duration_time>().time;\n                r.max = node[\"execution_time_max\"].as<duration_time>().time;\n                cfg.exec_time_range = r;\n            }\n            if (node[\"sleep_time\"]) {\n                cfg.sleep_time = node[\"sleep_time\"].as<duration_time>().time;\n            } else if (node[\"sleep_time_min\"] && node[\"sleep_time_max\"]) {\n                duration_range r;\n                r.min = node[\"sleep_time_min\"].as<duration_time>().time;\n                r.max = node[\"sleep_time_max\"].as<duration_time>().time;\n                cfg.sleep_time_range = r;\n            }\n            cfg.client = !node[\"side\"] || (node[\"side\"].as<std::string>() == \"client\");\n            cfg.server = !node[\"side\"] || (node[\"side\"].as<std::string>() == \"server\");\n        }\n        if (node[\"shares\"]) {\n            cfg.shares = node[\"shares\"].as<unsigned>();\n        }\n        if (node[\"sched_group\"]) {\n            cfg.sg_name = node[\"sched_group\"].as<std::string>();\n        } else {\n            cfg.sg_name = cfg.name;\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<config> {\n    static bool decode(const Node& node, config& cfg) {\n        if (node[\"client\"]) {\n            cfg.client = node[\"client\"].as<client_config>();\n        }\n        if (node[\"server\"]) {\n            cfg.server = node[\"server\"].as<server_config>();\n        }\n        if (node[\"jobs\"]) {\n            cfg.jobs = node[\"jobs\"].as<std::vector<job_config>>();\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<duration_time> {\n    static bool decode(const Node& node, duration_time& dt) {\n        auto str = node.as<std::string>();\n        if (str == \"0\") {\n            dt.time = 0ns;\n            return true;\n        }\n        if (str.back() != 's') {\n            return false;\n        }\n        str.pop_back();\n\n        std::chrono::duration<double> unit;\n        if (str.back() == 'm') {\n            unit = 1ms;\n            str.pop_back();\n        } else if (str.back() == 'u') {\n            unit = 1us;\n            str.pop_back();\n        } else if (str.back() == 'n') {\n            unit = 1ns;\n            str.pop_back();\n        } else {\n            unit = 1s;\n        }\n\n        dt.time = boost::lexical_cast<size_t>(str) * unit;\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<byte_size> {\n    static bool decode(const Node& node, byte_size& bs) {\n        auto str = node.as<std::string>();\n        unsigned shift = 0;\n        if (str.back() == 'B') {\n            str.pop_back();\n            if (str.back() != 'k') {\n                return false;\n            }\n            str.pop_back();\n            shift = 10;\n        }\n        bs.size = (boost::lexical_cast<size_t>(str) << shift);\n        return bs.size;\n    }\n};\n\n} // YAML namespace\n\nenum class rpc_verb : int32_t {\n    HELLO = 0,\n    BYE = 1,\n    ECHO = 2,\n    WRITE = 3,\n    STREAM_BIDIRECTIONAL = 4,\n    STREAM_UNIDIRECTIONAL = 5,\n};\n\nusing rpc_protocol = rpc::protocol<serializer, rpc_verb>;\nstatic std::array<double, 4> quantiles = { 0.5, 0.95, 0.99, 0.999};\n\nclass job {\npublic:\n    virtual std::string name() const = 0;\n    virtual future<> run() = 0;\n    virtual void emit_result(YAML::Emitter& out) const = 0;\n    virtual ~job() {}\n};\n\nclass job_rpc : public job {\n    using accumulator_type = accumulator_set<double, stats<tag::extended_p_square_quantile(quadratic), tag::mean, tag::max>>;\n\n    job_config _cfg;\n    socket_address _caddr;\n    client_config _ccfg;\n    rpc_protocol& _rpc;\n    std::unique_ptr<rpc_protocol::client> _client;\n    std::function<future<>(unsigned)> _call;\n    std::chrono::steady_clock::time_point _stop;\n    uint64_t _total_messages = 0;\n    accumulator_type _latencies;\n\n    future<> call_echo(unsigned dummy) {\n        auto cln = _rpc.make_client<uint64_t(uint64_t)>(rpc_verb::ECHO);\n        if (_cfg.timeout) {\n            return cln(*_client, std::chrono::duration_cast<seastar::rpc::rpc_clock_type::duration>(*_cfg.timeout), dummy).discard_result();\n        } else {\n            return cln(*_client, dummy).discard_result();\n        }\n    }\n\n    future<> call_write(unsigned dummy, const payload_t& pl) {\n        return _rpc.make_client<uint64_t(payload_t)>(rpc_verb::WRITE)(*_client, pl).then([exp = pl.size()] (auto res) {\n            SEASTAR_ASSERT(res == exp);\n            return make_ready_future<>();\n        });\n    }\n\npublic:\n    job_rpc(job_config cfg, rpc_protocol& rpc, client_config ccfg, socket_address caddr)\n            : _cfg(cfg)\n            , _caddr(std::move(caddr))\n            , _ccfg(ccfg)\n            , _rpc(rpc)\n            , _stop(std::chrono::steady_clock::now() + _cfg.duration)\n            , _latencies(extended_p_square_probabilities = quantiles)\n    {\n        if (_cfg.verb == \"echo\") {\n            _call = [this] (unsigned x) { return call_echo(x); };\n        } else if (_cfg.verb == \"write\") {\n            payload_t payload;\n            payload.resize(_cfg.payload / sizeof(payload_t::value_type), 0);\n            _call = [this, payload = std::move(payload)] (unsigned x) { return call_write(x, payload); };\n        } else if (_cfg.verb == \"vecho\") {\n            _call = [this] (unsigned x) {\n                fmt::print(\"{}.{} send echo\\n\", this_shard_id(), x);\n                return call_echo(x).then([x] {\n                        fmt::print(\"{}.{} got response\\n\", this_shard_id(), x);\n                }).handle_exception([x] (auto ex) {\n                        fmt::print(\"{}.{} got error {}\\n\", this_shard_id(), x, ex);\n                });\n            };\n        } else {\n            throw std::runtime_error(\"unknown verb\");\n        }\n    }\n\n    virtual std::string name() const override { return _cfg.name; }\n\n    virtual future<> run() override {\n      return with_scheduling_group(_cfg.sg, [this] {\n        rpc::client_options co;\n        co.tcp_nodelay = _ccfg.nodelay;\n        co.isolation_cookie = _cfg.sg_name;\n        _client = std::make_unique<rpc_protocol::client>(_rpc, co, _caddr);\n        return parallel_for_each(std::views::iota(0u, _cfg.parallelism), [this] (auto dummy) {\n          auto f = make_ready_future<>();\n          if (_cfg.sleep_time) {\n              // Do initial small delay to de-synchronize fibers\n              f = seastar::sleep(std::chrono::duration_cast<std::chrono::nanoseconds>(*_cfg.sleep_time / _cfg.parallelism * dummy));\n          }\n          return std::move(f).then([this, dummy] {\n            return do_until([this] {\n                return std::chrono::steady_clock::now() > _stop;\n            }, [this, dummy] {\n                _total_messages++;\n                auto now = std::chrono::steady_clock::now();\n                return _call(dummy).then([this, start = now] {\n                    std::chrono::microseconds lat = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now() - start);\n                    _latencies(lat.count());\n                }).then([this] {\n                    if (_cfg.sleep_time) {\n                        return seastar::sleep(std::chrono::duration_cast<std::chrono::nanoseconds>(*_cfg.sleep_time));\n                    } else {\n                        return make_ready_future<>();\n                    }\n                });\n            });\n          });\n        }).finally([this] {\n            return _client->stop();\n        });\n      });\n    }\n\n    virtual void emit_result(YAML::Emitter& out) const override {\n        out << YAML::Key << \"messages\" << YAML::Value << _total_messages;\n        out << YAML::Key << \"latencies\" << YAML::Comment(\"usec\");\n        out << YAML::BeginMap;\n        out << YAML::Key << \"average\" << YAML::Value << (uint64_t)mean(_latencies);\n        for (auto& q: quantiles) {\n            out << YAML::Key << fmt::format(\"p{}\", q) << YAML::Value << (uint64_t)quantile(_latencies, quantile_probability = q);\n        }\n        out << YAML::Key << \"max\" << YAML::Value << (uint64_t)max(_latencies);\n        out << YAML::EndMap;\n    }\n};\n\nclass job_rpc_streaming : public job {\n    job_config _cfg;\n    socket_address _caddr;\n    client_config _ccfg;\n    rpc_protocol& _rpc;\n    std::unique_ptr<rpc_protocol::client> _client;\n    std::function<future<>(unsigned, const payload_t&)> _call;\n    std::chrono::steady_clock::time_point _stop;\n    uint64_t _total_messages = 0;\n    uint64_t _payload_size_bytes = 0;\n    std::chrono::steady_clock::time_point _start_time{};\n    std::chrono::duration<double> _total_duration{0.0};\n    payload_t _payload;\n\npublic:\n    job_rpc_streaming(job_config cfg, rpc_protocol& rpc, client_config ccfg, socket_address caddr)\n            : _cfg(cfg)\n            , _caddr(std::move(caddr))\n            , _ccfg(ccfg)\n            , _rpc(rpc)\n            , _stop(std::chrono::steady_clock::now() + _cfg.duration)\n            , _payload_size_bytes(_cfg.payload)\n            , _payload(_cfg.payload / sizeof(payload_t::value_type), 0) {\n        if (_cfg.verb == \"bidirectional\") {\n            _call = [this] (unsigned worker_id, const payload_t& payload) {\n                return run_streaming_worker(worker_id, payload, rpc_verb::STREAM_BIDIRECTIONAL);\n            };\n        } else if (_cfg.verb == \"unidirectional\") {\n            _call = [this] (unsigned worker_id, const payload_t& payload) {\n                return run_streaming_worker(worker_id, payload, rpc_verb::STREAM_UNIDIRECTIONAL);\n            };\n        } else {\n            throw std::runtime_error(\"unknown verb, should be 'bidirectional' or 'unidirectional'\");\n        }\n    }\n\n    virtual std::string name() const override { return _cfg.name; }\n\nprivate:\n    future<> stream_data(rpc::sink<payload_t> sink, const payload_t& payload) {\n        std::exception_ptr error;\n        try {\n            // Send data through the sink until stop time\n            while (std::chrono::steady_clock::now() <= _stop) {\n                ++_total_messages;\n                co_await sink(payload);\n                if (_cfg.sleep_time) {\n                    co_await seastar::sleep(std::chrono::duration_cast<std::chrono::nanoseconds>(*_cfg.sleep_time));\n                }\n            }\n        } catch (...) {\n            error = std::current_exception();\n        }\n\n        auto flush_future = co_await coroutine::as_future(sink.flush());\n        if (flush_future.failed() && !error) [[unlikely]] {\n            error = flush_future.get_exception();\n        }\n        co_await sink.close();\n\n        if (error) {\n            std::rethrow_exception(error);\n        }\n    }\n\n    // Streaming worker:\n    // - client sends payload_t\n    // - in bidirectional case: server echoes back payload_t\n    // - in unidirectional case: server only sends EOS at the end\n    future<> run_streaming_worker(unsigned worker_id, const payload_t& payload, enum rpc_verb verb) {\n        auto sink = co_await _client->make_stream_sink<serializer, payload_t>();\n\n        auto rpc_call = _rpc.make_client<rpc::source<payload_t>(rpc::sink<payload_t>)>(verb);\n        auto source = co_await rpc_call(*_client, sink);\n\n        auto sender = stream_data(std::move(sink), payload);\n\n        auto receiver = repeat([src = std::move(source)] () mutable {\n            return src().then([] (std::optional<std::tuple<payload_t>> data) {\n                if (!data) {\n                    // EOS from server\n                    return stop_iteration::yes;\n                }\n                return stop_iteration::no;\n            });\n        });\n\n        co_await when_all(std::move(sender), std::move(receiver));\n    }\n\n    future<> run_worker_with_delay(unsigned worker_id, const payload_t& payload) {\n        if (_cfg.sleep_time) {\n            auto delay = std::chrono::duration_cast<std::chrono::nanoseconds>(*_cfg.sleep_time / _cfg.parallelism * worker_id);\n            co_await seastar::sleep(delay);\n        }\n        co_await _call(worker_id, payload);\n    }\n\npublic:\n    virtual future<> run() override {\n        return with_scheduling_group(_cfg.sg, [this] {\n            rpc::client_options co;\n            co.tcp_nodelay = _ccfg.nodelay;\n            co.isolation_cookie = _cfg.sg_name;\n            _client = std::make_unique<rpc_protocol::client>(_rpc, co, _caddr);\n            _start_time = std::chrono::steady_clock::now();\n\n            return parallel_for_each(std::views::iota(0u, _cfg.parallelism), [this] (unsigned worker_id) {\n                return run_worker_with_delay(worker_id, _payload);\n            }).finally([this] {\n                _total_duration = std::chrono::steady_clock::now() - _start_time;\n                return _client->stop();\n            });\n        });\n    }\n\n    virtual void emit_result(YAML::Emitter& out) const override {\n        out << YAML::Key << \"messages\" << YAML::Value << _total_messages;\n\n        auto total_bytes = _total_messages * _payload_size_bytes;\n        if (_total_duration.count() > 0) {\n            double throughput = total_bytes / _total_duration.count();\n            out << YAML::Key << \"throughput\" << YAML::Value << throughput << YAML::Comment(\"B/s\");\n\n            double messages_per_sec = _total_messages / _total_duration.count();\n            out << YAML::Key << \"messages per second\" << YAML::Value << messages_per_sec;\n        } else {\n            out << YAML::Key << \"throughput\" << YAML::Value << \"inf\" << YAML::Comment(\"kB/s\");\n            out << YAML::Key << \"messages per second\" << YAML::Value << \"inf\";\n        }\n    }\n\n    static future<> process_bi_source(rpc::source<payload_t> source, rpc::sink<payload_t> sink) {\n        uint64_t total_messages = 0, total_payload = 0;\n        std::exception_ptr error;\n\n        try {\n            while (true) {\n                auto data = co_await source();\n                if (!data) {\n                    break;\n                }\n                ++total_messages;\n                auto received_data = std::move(std::get<0>(*data));\n                total_payload += received_data.size() * sizeof(payload_t::value_type);\n                // Send data back to client\n                co_await sink(received_data);\n            }\n        } catch (...) {\n            error = std::current_exception();\n        }\n\n        auto flush_future = co_await coroutine::as_future(sink.flush());\n        if (flush_future.failed() && !error) [[unlikely]] {\n            error = flush_future.get_exception();\n        }\n        co_await sink.close();\n\n        if (error) {\n            std::rethrow_exception(error);\n        }\n\n        fmt::print(\"Server received total {} messages on bidirectional stream, total payload: {} bytes\\n\", total_messages, total_payload);\n    }\n\n    static future<> process_uni_source(rpc::source<payload_t> source, rpc::sink<uint64_t> sink) {\n        uint64_t total_messages = 0, total_payload = 0;\n        std::exception_ptr error;\n\n        try {\n            while (true) {\n                auto data = co_await source();\n                if (!data) {\n                    // We need to have some kind of synchronization with client, so they don't close the main RPC connection\n                    // until server received EOS (client -> server connection was closed). If we don't do that, server might\n                    // receive `seastar::rpc::stream_closed` exception on reading from the source, as the stream is terminated\n                    // on connection close.\n                    // In order to do that, we create a connection to the other side - from server to client, and keep it\n                    // open until client sends EOS - then, by closing this connection, server also sends EOS.\n                    // This gives client the guarantee that server received all data before closing the main RPC connection.\n                    break;\n                }\n                ++total_messages;\n                total_payload += std::get<0>(*data).size() * sizeof(payload_t::value_type);\n            }\n        } catch (...) {\n            error = std::current_exception();\n        }\n\n        auto flush_future = co_await coroutine::as_future(sink.flush());\n        if (flush_future.failed() && !error) [[unlikely]] {\n            error = flush_future.get_exception();\n        }\n        co_await sink.close();\n\n        if (error) {\n            std::rethrow_exception(error);\n        }\n\n        fmt::print(\"Server received total {} messages on unidirectional stream, total payload: {} bytes\\n\", total_messages, total_payload);\n    }\n};\n\nclass job_cpu : public job {\n    job_config _cfg;\n    std::chrono::steady_clock::time_point _stop;\n    uint64_t _total_invocations = 0;\n    std::unique_ptr<pause_distribution> _pause;\n    std::unique_ptr<pause_distribution> _sleep;\n\n    std::unique_ptr<pause_distribution> make_pause() {\n        if (_cfg.exec_time_range) {\n            return make_uniform_pause(*_cfg.exec_time_range);\n        } else {\n            return make_steady_pause(_cfg.exec_time);\n        }\n    }\n\n    std::unique_ptr<pause_distribution> make_sleep() {\n        if (_cfg.sleep_time) {\n            return make_steady_pause(*_cfg.sleep_time);\n        }\n        if (_cfg.sleep_time_range) {\n            return make_uniform_pause(*_cfg.sleep_time_range);\n        }\n        return nullptr;\n    }\n\npublic:\n    job_cpu(job_config cfg)\n            : _cfg(cfg)\n            , _pause(make_pause())\n            , _sleep(make_sleep())\n    {\n    }\n\n    virtual std::string name() const override { return _cfg.name; }\n    virtual void emit_result(YAML::Emitter& out) const override {\n        out << YAML::Key << \"total\" << YAML::Value << _total_invocations;\n    }\n\n    virtual future<> run() override {\n        _stop = std::chrono::steady_clock::now() + _cfg.duration;\n        return with_scheduling_group(_cfg.sg, [this] {\n          return parallel_for_each(std::views::iota(0u, _cfg.parallelism), [this] (auto dummy) {\n            return do_until([this] {\n                return std::chrono::steady_clock::now() > _stop;\n            }, [this] {\n                _total_invocations++;\n                auto start  = std::chrono::steady_clock::now();\n                auto pause = _pause->get();\n                while ((std::chrono::steady_clock::now() - start) < pause);\n                if (!_sleep) {\n                    return make_ready_future<>();\n                } else {\n                    auto sleep = std::chrono::duration_cast<std::chrono::nanoseconds>(_sleep->get());\n                    return seastar::sleep(sleep);\n                }\n            });\n          });\n        });\n    }\n};\n\nclass context {\n    std::unique_ptr<rpc_protocol> _rpc;\n    std::unique_ptr<rpc_protocol::server> _server;\n    std::unique_ptr<rpc_protocol::client> _client;\n    promise<> _bye;\n    promise<> _server_jobs;\n    config _cfg;\n    std::vector<std::unique_ptr<job>> _jobs;\n    std::unordered_map<std::string, scheduling_group> _sched_groups;\n\n    std::unique_ptr<job> make_job(job_config cfg, std::optional<socket_address> caddr) {\n        if (cfg.type == \"rpc\") {\n            return std::make_unique<job_rpc>(cfg, *_rpc, _cfg.client, *caddr);\n        }\n        if (cfg.type == \"rpc_streaming\") {\n            return std::make_unique<job_rpc_streaming>(cfg, *_rpc, _cfg.client, *caddr);\n        }\n        if (cfg.type == \"cpu\") {\n            return std::make_unique<job_cpu>(cfg);\n        }\n\n        throw std::runtime_error(\"unknown job type\");\n    }\n\n    future<> run_jobs() {\n        return parallel_for_each(_jobs, [] (auto& job) {\n            return job->run();\n        });\n    }\n\n    rpc::isolation_config isolate_connection(std::string group_name) {\n        rpc::isolation_config cfg;\n        if (group_name != \"\") {\n            cfg.sched_group = _sched_groups[group_name];\n        }\n        return cfg;\n    }\n\npublic:\n    context(std::optional<socket_address> laddr, std::optional<socket_address> caddr, uint16_t port, config cfg, std::unordered_map<std::string, scheduling_group> groups)\n            : _rpc(std::make_unique<rpc_protocol>(serializer{}))\n            , _cfg(cfg)\n            , _sched_groups(std::move(groups))\n    {\n        _rpc->register_handler(rpc_verb::HELLO, [this] {\n            fmt::print(\"Got HELLO message from client\\n\");\n            run_jobs().discard_result().forward_to(std::move(_server_jobs));\n        });\n        _rpc->register_handler(rpc_verb::BYE, [this] {\n            fmt::print(\"Got BYE message from client, exiting\\n\");\n            _bye.set_value();\n        });\n        _rpc->register_handler(rpc_verb::ECHO, [] (uint64_t val) {\n            return make_ready_future<uint64_t>(val);\n        });\n        _rpc->register_handler(rpc_verb::WRITE, [] (payload_t val) {\n            return make_ready_future<uint64_t>(val.size());\n        });\n        _rpc->register_handler(rpc_verb::STREAM_BIDIRECTIONAL, [] (rpc::source<payload_t> source) {\n            // Create sink for server->client direction\n            auto sink = source.make_sink<serializer, payload_t>();\n\n            (void)job_rpc_streaming::process_bi_source(std::move(source), sink);\n\n            return sink;\n        });\n        _rpc->register_handler(rpc_verb::STREAM_UNIDIRECTIONAL, [] (rpc::source<payload_t> source) {\n            auto sink = source.make_sink<serializer, uint64_t>();\n\n            (void)job_rpc_streaming::process_uni_source(std::move(source), sink);\n\n            return sink;\n        });\n\n\n        if (laddr) {\n            rpc::server_options so;\n            so.tcp_nodelay = _cfg.server.nodelay;\n            so.streaming_domain = rpc::streaming_domain_type(1);\n            rpc::resource_limits limits;\n            limits.isolate_connection = [this] (sstring cookie) { return isolate_connection(cookie); };\n            _server = std::make_unique<rpc_protocol::server>(*_rpc, so, *laddr, limits);\n\n            for (auto&& jc : _cfg.jobs) {\n                if (jc.server) {\n                    _jobs.push_back(make_job(jc, {}));\n                }\n            }\n        }\n\n        if (caddr) {\n            rpc::client_options co;\n            co.tcp_nodelay = _cfg.client.nodelay;\n            _client = std::make_unique<rpc_protocol::client>(*_rpc, co, *caddr);\n\n            for (auto&& jc : _cfg.jobs) {\n                if (jc.client) {\n                    _jobs.push_back(make_job(jc, *caddr));\n                }\n            }\n        }\n    }\n\n    future<> start() {\n        if (_client) {\n            return _rpc->make_client<void()>(rpc_verb::HELLO)(*_client);\n        }\n\n        return make_ready_future<>();\n    }\n\n    future<> stop() {\n        if (_client) {\n            return _rpc->make_client<void()>(rpc_verb::BYE)(*_client).finally([this] {\n                return _client->stop();\n            });\n        }\n\n        if (_server) {\n            return _server->stop();\n        }\n\n        return make_ready_future<>();\n    }\n\n    future<> run() {\n        if (_client) {\n            return run_jobs();\n        }\n\n        if (_server) {\n            return when_all(_bye.get_future(), _server_jobs.get_future()).discard_result();\n        }\n\n        return make_ready_future<>();\n    }\n\n    future<> emit_result(YAML::Emitter& out) const {\n        for (const auto& job : _jobs) {\n            out << YAML::Key << job->name();\n            out << YAML::BeginMap;\n            job->emit_result(out);\n            out << YAML::EndMap;\n        }\n\n        return make_ready_future<>();\n    }\n};\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n\n    app_template app;\n    auto opt_add = app.add_options();\n    opt_add\n        (\"listen\", bpo::value<sstring>()->default_value(\"\"), \"address to start server on\")\n        (\"connect\", bpo::value<sstring>()->default_value(\"\"), \"address to connect client to\")\n        (\"port\", bpo::value<int>()->default_value(9123), \"port to listen on or connect to\")\n        (\"conf\", bpo::value<sstring>()->default_value(\"./conf.yaml\"), \"config with jobs and options\")\n        (\"duration\", bpo::value<unsigned>()->default_value(30), \"duration in seconds\")\n    ;\n\n    sharded<context> ctx;\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            auto& opts = app.configuration();\n            auto& listen = opts[\"listen\"].as<sstring>();\n            auto& connect = opts[\"connect\"].as<sstring>();\n            auto& port = opts[\"port\"].as<int>();\n            auto& conf = opts[\"conf\"].as<sstring>();\n            auto duration = std::chrono::seconds(opts[\"duration\"].as<unsigned>());\n\n            std::optional<socket_address> laddr;\n            if (listen != \"\") {\n                if (listen[0] == '.' || listen[0] == '/') {\n                    unix_domain_addr addr(listen);\n                    laddr.emplace(std::move(addr));\n                } else {\n                    ipv4_addr addr(listen, port);\n                    laddr.emplace(std::move(addr));\n                }\n            }\n            std::optional<socket_address> caddr;\n            if (connect != \"\") {\n                if (connect[0] == '.' || connect[0] == '/') {\n                    unix_domain_addr addr(connect);\n                    caddr.emplace(std::move(addr));\n                } else {\n                    ipv4_addr addr(connect, port);\n                    caddr.emplace(std::move(addr));\n                }\n            }\n\n            YAML::Node doc = YAML::LoadFile(conf);\n            auto cfg = doc.as<config>();\n            std::unordered_map<std::string, scheduling_group> groups;\n\n            for (auto&& jc : cfg.jobs) {\n                jc.duration = duration;\n                if (groups.count(jc.sg_name) == 0) {\n                    fmt::print(\"Make sched group {}, {} shares\\n\", jc.sg_name, jc.shares);\n                    groups[jc.sg_name] = create_scheduling_group(jc.sg_name, jc.shares).get();\n                }\n                jc.sg = groups[jc.sg_name];\n            }\n\n            ctx.start(laddr, caddr, port, cfg, groups).get();\n            ctx.invoke_on_all(&context::start).get();\n            ctx.invoke_on_all(&context::run).get();\n\n            YAML::Emitter out;\n            out << YAML::BeginDoc;\n            out << YAML::BeginSeq;\n            for (unsigned i = 0; i < smp::count; i++) {\n                out << YAML::BeginMap;\n                out << YAML::Key << \"shard\" << YAML::Value << i;\n                ctx.invoke_on(i, [&out] (auto& c) {\n                    return c.emit_result(out);\n                }).get();\n                out << YAML::EndMap;\n            }\n            out << YAML::EndSeq;\n            out << YAML::EndDoc;\n            std::cout << out.c_str();\n\n            ctx.stop().get();\n        });\n    });\n}\n"
  },
  {
    "path": "apps/rpc_tester/sample-conf.yaml",
    "content": "client:\n  nodelay: # bool, whether or not to set tcp_nodelay option\nserver:\n  nodelay: # bool, whether or not to set tcp_nodelay option\njobs:\n  - name: # any parseable string\n    type: rpc\n    verb: # string, one of: echo, vecho, write\n    parallelism: # number of verbs to send simultaneously\n    shares: # sched group shares (100 by default)\n    payload: # number of bytes in the payload for write verb, accepts kB suffix\n    sleep_time: # optional inactivity pause between sending messages\n    timeout: # optional rpc send timeout duration\n  - name: # any parseable string\n    type: rpc_streaming\n    verb: # string, one of: bidirectional, unidirectional\n           # - bidirectional: client sends payload_t, server responds asynchronously with received data\n           # - unidirectional: client sends payload_t, server consumes without responding\n    parallelism: # number of streaming workers to run simultaneously\n    shares: # sched group shares (100 by default)\n    payload: # number of bytes in the payload sent from client to server, accepts kB suffix\n    sleep_time: # optional inactivity pause between sending messages\n  - name:\n    type: cpu\n    execution_time: # time in [0-9]+[mun]?s format\n    sleep_time: # optional inactivity pause between burning cpu\n    side: # optional, 'client' or 'server' to specify which side to run on\n"
  },
  {
    "path": "apps/seawreck/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nseastar_add_app (seawreck\n  SOURCES seawreck.cc)\n"
  },
  {
    "path": "apps/seawreck/seawreck.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/http/response_parser.hh>\n#include <seastar/net/api.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/semaphore.hh>\n#include <chrono>\n\nusing namespace seastar;\n\ntemplate <typename... Args>\nvoid http_debug(const char* fmt, Args&&... args) {\n#if HTTP_DEBUG\n    print(fmt, std::forward<Args>(args)...);\n#endif\n}\n\nclass http_client {\nprivate:\n    unsigned _duration;\n    unsigned _conn_per_core;\n    unsigned _reqs_per_conn;\n    std::vector<connected_socket> _sockets;\n    semaphore _conn_connected{0};\n    semaphore _conn_finished{0};\n    timer<> _run_timer;\n    bool _timer_based;\n    bool _timer_done{false};\n    uint64_t _total_reqs{0};\npublic:\n    http_client(unsigned duration, unsigned total_conn, unsigned reqs_per_conn)\n        : _duration(duration)\n        , _conn_per_core(total_conn / smp::count)\n        , _reqs_per_conn(reqs_per_conn)\n        , _run_timer([this] { _timer_done = true; })\n        , _timer_based(reqs_per_conn == 0) {\n    }\n\n    class connection {\n    private:\n        connected_socket _fd;\n        input_stream<char> _read_buf;\n        output_stream<char> _write_buf;\n        http_response_parser _parser;\n        http_client* _http_client;\n        uint64_t _nr_done{0};\n    public:\n        connection(connected_socket&& fd, http_client* client)\n            : _fd(std::move(fd))\n            , _read_buf(_fd.input())\n            , _write_buf(_fd.output())\n            , _http_client(client){\n        }\n\n        uint64_t nr_done() {\n            return _nr_done;\n        }\n\n        future<> do_req() {\n            return _write_buf.write(\"GET / HTTP/1.1\\r\\nHost: 127.0.0.1:10000\\r\\n\\r\\n\").then([this] {\n                return _write_buf.flush();\n            }).then([this] {\n                _parser.init();\n                return _read_buf.consume(_parser).then([this] {\n                    // Read HTTP response header first\n                    if (_parser.eof()) {\n                        return make_ready_future<>();\n                    }\n                    auto _rsp = _parser.get_parsed_response();\n                    auto it = _rsp->_headers.find(\"Content-Length\");\n                    if (it == _rsp->_headers.end()) {\n                        fmt::print(\"Error: HTTP response does not contain: Content-Length\\n\");\n                        return make_ready_future<>();\n                    }\n                    auto content_len = std::stoi(it->second);\n                    http_debug(\"Content-Length = %d\\n\", content_len);\n                    // Read HTTP response body\n                    return _read_buf.read_exactly(content_len).then([this] (temporary_buffer<char> buf) {\n                        _nr_done++;\n                        http_debug(\"%s\\n\", buf.get());\n                        if (_http_client->done(_nr_done)) {\n                            return make_ready_future();\n                        } else {\n                            return do_req();\n                        }\n                    });\n                });\n            });\n        }\n    };\n\n    future<uint64_t> total_reqs() {\n        fmt::print(\"Requests on cpu {:2d}: {:d}\\n\", this_shard_id(), _total_reqs);\n        return make_ready_future<uint64_t>(_total_reqs);\n    }\n\n    bool done(uint64_t nr_done) {\n        if (_timer_based) {\n            return _timer_done;\n        } else {\n            return nr_done >= _reqs_per_conn;\n        }\n    }\n\n    future<> connect(ipv4_addr server_addr) {\n        // Establish all the TCP connections first\n        for (unsigned i = 0; i < _conn_per_core; i++) {\n            // Connect in the background, signal _conn_connected when done.\n            (void)seastar::connect(make_ipv4_address(server_addr)).then([this] (connected_socket fd) {\n                _sockets.push_back(std::move(fd));\n                http_debug(\"Established connection %6d on cpu %3d\\n\", _conn_connected.current(), this_shard_id());\n                _conn_connected.signal();\n            }).or_terminate();\n        }\n        return _conn_connected.wait(_conn_per_core);\n    }\n\n    future<> run() {\n        // All connected, start HTTP request\n        http_debug(\"Established all %6d tcp connections on cpu %3d\\n\", _conn_per_core, this_shard_id());\n        if (_timer_based) {\n            _run_timer.arm(std::chrono::seconds(_duration));\n        }\n        for (auto&& fd : _sockets) {\n            auto conn = new connection(std::move(fd), this);\n            // Run in the background, signal _conn_finished when done.\n            (void)conn->do_req().then_wrapped([this, conn] (auto&& f) {\n                http_debug(\"Finished connection %6d on cpu %3d\\n\", _conn_finished.current(), this_shard_id());\n                _total_reqs += conn->nr_done();\n                _conn_finished.signal();\n                delete conn;\n                // FIXME: should _conn_finished.signal be called only after this?\n                // nothing seems to synchronize with this background work.\n                try {\n                    f.get();\n                } catch (std::exception& ex) {\n                    fmt::print(\"http request error: {}\\n\", ex.what());\n                }\n            });\n        }\n\n        // All finished\n        return _conn_finished.wait(_conn_per_core);\n    }\n    future<> stop() {\n        return make_ready_future();\n    }\n};\n\nnamespace bpo = boost::program_options;\n\nint main(int ac, char** av) {\n    app_template::config app_cfg;\n    app_cfg.auto_handle_sigint_sigterm = false;\n    app_template app(std::move(app_cfg));\n\n    app.add_options()\n        (\"server,s\", bpo::value<std::string>()->default_value(\"192.168.66.100:10000\"), \"Server address\")\n        (\"conn,c\", bpo::value<unsigned>()->default_value(100), \"total connections\")\n        (\"reqs,r\", bpo::value<unsigned>()->default_value(0), \"reqs per connection\")\n        (\"duration,d\", bpo::value<unsigned>()->default_value(10), \"duration of the test in seconds)\");\n\n    return app.run(ac, av, [&app] () -> future<int> {\n        auto& config = app.configuration();\n        auto server = config[\"server\"].as<std::string>();\n        auto reqs_per_conn = config[\"reqs\"].as<unsigned>();\n        auto total_conn= config[\"conn\"].as<unsigned>();\n        auto duration = config[\"duration\"].as<unsigned>();\n\n        if (total_conn % smp::count != 0) {\n            fmt::print(\"Error: conn needs to be n * cpu_nr\\n\");\n            return make_ready_future<int>(-1);\n        }\n\n        auto http_clients = new sharded<http_client>;\n\n        // Start http requests on all the cores\n        auto started = steady_clock_type::now();\n        fmt::print(\"========== http_client ============\\n\");\n        fmt::print(\"Server: {}\\n\", server);\n        fmt::print(\"Connections: {:d}\\n\", total_conn);\n        fmt::print(\"Requests/connection: {}\\n\", reqs_per_conn == 0 ? \"dynamic (timer based)\" : std::to_string(reqs_per_conn));\n        return http_clients->start(std::move(duration), std::move(total_conn), std::move(reqs_per_conn)).then([http_clients, server] {\n            return http_clients->invoke_on_all(&http_client::connect, ipv4_addr{server});\n        }).then([http_clients] {\n            return http_clients->invoke_on_all(&http_client::run);\n        }).then([http_clients] {\n            return http_clients->map_reduce(adder<uint64_t>(), &http_client::total_reqs);\n        }).then([http_clients, started] (auto total_reqs) {\n           // All the http requests are finished\n           auto finished = steady_clock_type::now();\n           auto elapsed = finished - started;\n           auto secs = static_cast<double>(elapsed.count() / 1000000000.0);\n           fmt::print(\"Total cpus: {:d}\\n\", smp::count);\n           fmt::print(\"Total requests: {:d}\\n\", total_reqs);\n           fmt::print(\"Total time: {:f}\\n\", secs);\n           fmt::print(\"Requests/sec: {:f}\\n\", static_cast<double>(total_reqs) / secs);\n           fmt::print(\"==========     done     ============\\n\");\n           return http_clients->stop().then([http_clients] {\n               // FIXME: If we call engine().exit(0) here to exit when\n               // requests are done. The tcp connection will not be closed\n               // properly, becasue we exit too earily and the FIN packets are\n               // not exchanged.\n                delete http_clients;\n                return make_ready_future<int>(0);\n           });\n        });\n    });\n}\n"
  },
  {
    "path": "cmake/CheckGcc107852.cmake",
    "content": "include (CheckCXXSourceCompiles)\ninclude (CMakePushCheckState)\n\ncmake_push_check_state (RESET)\n\n# these options are included by -Wall, which is in turn included by\n# Seastar_PRIVATE_CXX_FLAGS, which is not applied to CMAKE_CXX_FLAGS, so\n# let's apply them explicitly.\nset (CMAKE_REQUIRED_FLAGS \"-Werror=stringop-overflow -Werror=array-bound\")\nset (CMAKE_REQUIRED_LIBRARIES fmt::fmt)\n\n# see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107852\ncheck_cxx_source_compiles (\"\n#include <fmt/ranges.h>\n\nint main() {\n    float grades[] = {3.14};\n    fmt::print(\\\"{}\\\", grades);\n}\"\n  Cxx_Compiler_BZ107852_Free\n  FAIL_REGEX \"is out of the bounds\")\n\ncmake_pop_check_state ()\n\n"
  },
  {
    "path": "cmake/CheckHeaders.cmake",
    "content": "# seastar_check_self_contained() checks if the headers listed as the source of\n# a target are self-contained.\n#\n# Header files should be self-contained. In general, the source files should\n# not have to adhere to special conditions to include them. For instance,\n# they don't need to include other header files for using a header file, or\n# to define certain macro(s) for using it. But the macros are allowed to be\n# used to its behavior though.\n#\n# seastar_check_self_contained() is created to perform a minimal check on the\n# specified set of header files by compiling each of them.\n#\n# Please note, there are chances that a symbol declaration could be included\n# indirectly by an (indirectly) included header file even if that symbol is not\n# a part of the public interface of that header file, so this dependency is\n# a little bit fragile. seastar_check_self_contained()\" does not warn at seeing\n# the indirect dependency, it just check if the preprocessed header file\n# *contains* the declarations of the symbols so that any source file including\n# it can be compiled as well. The check performed by the CMake function allows\n# the indirect inclusion of the used symbols. For instance, if \"juno.h\"\n# references a symbol named \"Jupiter\", which is declared in \"jupiter.h\". So,\n# strictly speaking, \"juno.h\" should include \"jupiter.h\" for accessing this\n# symbol. But \"juno.h\" happens to include \"solar.h\", which in turn includes\n# \"jupiter.h\". So \"seastar_check_self_contained()\" accepts \"juno.h\", while a\n# tool like iwyu would complain at seeing it.\n#\n# You can also use CMAKE_CXX_INCLUDE_WHAT_YOU_USE for using an external tool\n# for performing a similar check. see\n# https://cmake.org/cmake/help/latest/prop_tgt/LANG_INCLUDE_WHAT_YOU_USE.html\n\nfunction (seastar_check_self_contained target library)\n  cmake_parse_arguments (\n    parsed_args\n    \"\"\n    \"\"\n    \"EXCLUDE;INCLUDE\"\n    ${ARGN})\n\n  get_target_property (sources ${library} SOURCES)\n  list (FILTER sources INCLUDE REGEX \"${parsed_args_INCLUDE}\")\n  list (FILTER sources EXCLUDE REGEX \"${parsed_args_EXCLUDE}\")\n  foreach (fn ${sources})\n    get_filename_component (file_ext ${fn} EXT)\n    if (NOT file_ext STREQUAL \".hh\")\n      message (SEND_ERROR \"Only headers are checked if they are self-contained, while ${fn} is not a header.\")\n    elseif (IS_ABSOLUTE ${fn})\n      # the header specified with absolute path is likely to be generated, this\n      # is not our focus at this moment.\n      continue ()\n    endif ()\n    get_filename_component (file_dir ${fn} DIRECTORY)\n    list (APPEND includes \"${file_dir}\")\n    set (src_dir \"${CMAKE_BINARY_DIR}/${target}/${file_dir}\")\n    file (MAKE_DIRECTORY \"${src_dir}\")\n    get_filename_component (file_name ${fn} NAME)\n    set (src \"${src_dir}/${file_name}.cc\")\n    # CMake refuses to compile .hh files, so we need to rename them first.\n    add_custom_command (\n      OUTPUT ${src}\n      DEPENDS ${fn}\n      # silence \"-Wpragma-once-outside-header\"\n      COMMAND sed\n            -e \"s/^#pragma once//\"\n            \"${fn}\" > \"${src}\"\n      WORKING_DIRECTORY \"${CMAKE_CURRENT_SOURCE_DIR}\"\n      VERBATIM)\n    list (APPEND srcs \"${src}\")\n  endforeach ()\n\n  if (NOT srcs)\n    # library's SOURCES does not contain any header\n    return ()\n  endif ()\n\n  set (check_lib \"${target}-${library}\")\n  add_library (${check_lib} EXCLUDE_FROM_ALL)\n  target_sources (${check_lib}\n    PRIVATE ${srcs})\n  # use ${library} as an interface library by consuming all of its\n  # compile time options\n  get_target_property (libraries ${library} LINK_LIBRARIES)\n  if (libraries)\n    target_link_libraries (${check_lib}\n      PRIVATE ${libraries})\n  endif ()\n\n  # if header includes other header files with relative path,\n  # we should satisfy it.\n  list (REMOVE_DUPLICATES includes)\n  target_include_directories (${check_lib}\n    PRIVATE ${includes})\n  get_target_property (includes ${library} INCLUDE_DIRECTORIES)\n  if (includes)\n    target_include_directories (${check_lib}\n      PRIVATE ${includes})\n  endif ()\n\n  get_target_property (compile_options ${library} COMPILE_OPTIONS)\n  if (compile_options)\n    target_compile_options (${check_lib}\n      PRIVATE ${compile_options})\n  endif ()\n  # symbols in header file should always be referenced, but these\n  # are just pure headers, so unused variables should be tolerated.\n  target_compile_options (${check_lib}\n    PRIVATE\n      -Wno-unused-const-variable\n      -Wno-unused-function\n      -Wno-unused-variable)\n\n  get_target_property (compile_definitions ${library} COMPILE_DEFINITIONS)\n  if (compile_definitions)\n    target_compile_definitions (${check_lib}\n      PRIVATE ${compile_definitions})\n  endif ()\n\n  add_dependencies (${target} ${check_lib})\nendfunction ()\n"
  },
  {
    "path": "cmake/CheckIncludeStyle.cmake",
    "content": "# seastar_check_include_style() enforces that all source and header files under\n# specified directories include the headers with predefined list of prefixes\n# with angle brackets instead of quotes.\n\nfind_package (Python3 COMPONENTS Interpreter)\n\nfunction (seastar_check_include_style target library)\n  get_target_property (sources ${library} SOURCES)\n  set (check-target \"${target}-${library}\")\n  add_custom_target(\"${check-target}\"\n    COMMAND Python3::Interpreter ${CMAKE_CURRENT_LIST_DIR}/cmake/check-seastar-include-style.py ${sources}\n    WORKING_DIRECTORY \"${CMAKE_CURRENT_SOURCE_DIR}\"\n    COMMENT \"Checking include directive styles for ${library} source files\"\n    USES_TERMINAL)\n  add_dependencies (${target} ${check-target})\nendfunction ()\n"
  },
  {
    "path": "cmake/CheckLibc.cmake",
    "content": "# check for the bits in different standard C library implementations we\n# care about\n\ninclude (CheckCXXSourceCompiles)\nfile (READ ${CMAKE_CURRENT_LIST_DIR}/code_tests/stdout_test.cc _stdout_test_code)\ncheck_cxx_source_compiles (\"${_stdout_test_code}\" Stdout_Can_Be_Used_As_Identifier)\nif (Stdout_Can_Be_Used_As_Identifier)\n  # \"stdout\" is defined as a macro by the C++ standard, so we cannot assume\n  # that the macro is always expanded into an identifier which can be re-used\n  # to name a enumerator in the declaration of an enumeration.\n  target_compile_definitions (seastar\n    PUBLIC\n      SEASTAR_LOGGER_TYPE_STDOUT)\nendif ()\n\ncheck_cxx_source_compiles (\"\n#include <string.h>\n\nint main() {\n    char buf;\n    char* a = strerror_r(1, &buf, 0);\n    static_cast<void>(a);\n}\"\n  Strerror_R_Returns_Char_P)\nif (Strerror_R_Returns_Char_P)\n  # define SEASTAR_STRERROR_R_CHAR_P if strerror_r() is GNU-specific version,\n  # which returns a \"char*\" not \"int\".\n  target_compile_definitions (seastar\n    PRIVATE\n      SEASTAR_STRERROR_R_CHAR_P)\nendif ()\n\ninclude (CheckFunctionExists)\n\ncheck_function_exists (pthread_attr_setaffinity_np\n  Pthread_Attr_Setaffinity_Np)\nif (Pthread_Attr_Setaffinity_Np)\n  target_compile_definitions (seastar\n    PRIVATE\n    SEASTAR_PTHREAD_ATTR_SETAFFINITY_NP)\nendif ()\n"
  },
  {
    "path": "cmake/CheckP2582R1.cmake",
    "content": "include (CheckCXXSourceCompiles)\ninclude (CMakePushCheckState)\n\ncmake_push_check_state (RESET)\n\nset (CMAKE_REQUIRED_FLAGS \"-std=c++23\")\n\n# check if the compiler implements the inherited vs non-inherited guide\n# tiebreaker specified by P2582R1, see https://wg21.link/P2582R1\ncheck_cxx_source_compiles (\"\ntemplate <typename... T> struct B {\n    B(T...) {}\n};\n\ntemplate <typename... T> struct C : public B<T...> {\n    using B<T...>::B;\n    C(B<T...>) {}\n};\n\nB(int) -> B<char>;\nC c2(42);\n\nint main() {}\n\"\n  Cxx_Compiler_IMPLEMENTS_P2581R1)\n\ncmake_pop_check_state ()\n"
  },
  {
    "path": "cmake/CxxModulesRules.cmake",
    "content": "if (CMAKE_CXX_COMPILER_ID STREQUAL \"Clang\")\n  if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 16)\n    message (FATAL_ERROR \"C++20 module needs Clang++-16 or up\")\n  endif ()\nelseif (CMAKE_CXX_COMPILER_ID STREQUAL \"GNU\")\n  if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 14)\n    message (FATAL_ERROR \"C++20 module needs g++-14 or up\")\n  endif ()\nelse ()\n  message (FATAL_ERROR \"Unsupported compiler: ${CMAKE_CXX_COMPILER_ID}\")\nendif ()\n\nif (CMAKE_VERSION VERSION_GREATER_EQUAL 3.28)\n  # CMake 3.28 has official support of C++20 modules\nelseif (CMAKE_VERSION VERSION_GREATER_EQUAL 3.27)\n  set (CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API \"aa1f7df0-828a-4fcd-9afc-2dc80491aca7\")\nelseif (CMAKE_VERSION VERSION_GREATER_EQUAL 3.26)\n  set (CMAKE_EXPERIMENTAL_CXX_SCANDEP_SOURCE \"\")\n  set (CMAKE_EXPERIMENTAL_CXX_MODULE_DYNDEP 1)\n  set (CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API \"2182bf5c-ef0d-489a-91da-49dbc3090d2a\")\nendif ()\n\nset (CMAKE_CXX_STANDARD_REQUIRED ON)\n# C++ extension does work with C++ module support so far\nset (CMAKE_CXX_EXTENSIONS OFF)\n"
  },
  {
    "path": "cmake/FindGnuTLS.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_check_modules (PC_GnuTLS QUIET gnutls)\n\nfind_library (GnuTLS_LIBRARY\n  NAMES gnutls\n  HINTS\n    ${PC_GnuTLS_LIBDIR}\n    ${PC_GnuTLS_LIBRARY_DIRS})\n\nfind_path (GnuTLS_INCLUDE_DIR\n  NAMES gnutls/gnutls.h\n  HINTS\n    ${PC_GnuTLS_INCLUDEDIR}\n    ${PC_GnuTLS_INCLUDE_DIRS})\n\nmark_as_advanced (\n  GnuTLS_LIBRARY\n  GnuTLS_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (GnuTLS\n  REQUIRED_VARS\n    GnuTLS_LIBRARY\n    GnuTLS_INCLUDE_DIR\n  VERSION_VAR PC_GnuTLS_VERSION)\n\nif (GnuTLS_FOUND)\n  set (GnuTLS_LIBRARIES ${GnuTLS_LIBRARY})\n  set (GnuTLS_INCLUDE_DIRS ${GnuTLS_INCLUDE_DIR})\n  if (NOT (TARGET GnuTLS::gnutls))\n    add_library (GnuTLS::gnutls UNKNOWN IMPORTED)\n\n    set_target_properties (GnuTLS::gnutls\n      PROPERTIES\n        IMPORTED_LOCATION ${GnuTLS_LIBRARY}\n        INTERFACE_INCLUDE_DIRECTORIES ${GnuTLS_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/FindLibUring.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2022 ScyllaDB\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_check_modules (PC_URING QUIET liburing)\n\nfind_library (URING_LIBRARY\n  NAMES uring\n  HINTS\n    ${PC_URING_LIBDIR}\n    ${PC_URING_LIBRARY_DIRS})\n\nfind_path (URING_INCLUDE_DIR\n  NAMES liburing.h\n  HINTS\n    ${PC_URING_INCLUDEDIR}\n    ${PC_URING_INCLUDE_DIRS})\n\nif (URING_INCLUDE_DIR)\n  include (CheckStructHasMember)\n  include (CMakePushCheckState)\n  cmake_push_check_state (RESET)\n  list(APPEND CMAKE_REQUIRED_INCLUDES ${URING_INCLUDE_DIR})\n  CHECK_STRUCT_HAS_MEMBER (\"struct io_uring\" features liburing.h\n    HAVE_IOURING_FEATURES LANGUAGE CXX)\n  cmake_pop_check_state ()\nendif ()\n\nmark_as_advanced (\n  URING_LIBRARY\n  URING_INCLUDE_DIR\n  HAVE_IOURING_FEATURES)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (LibUring\n  REQUIRED_VARS\n    URING_LIBRARY\n    URING_INCLUDE_DIR\n    HAVE_IOURING_FEATURES\n  VERSION_VAR PC_URING_VERSION)\n\nif (LibUring_FOUND)\n  set (URING_LIBRARIES ${URING_LIBRARY})\n  set (URING_INCLUDE_DIRS ${URING_INCLUDE_DIR})\n  if (NOT (TARGET URING::uring))\n    add_library (URING::uring UNKNOWN IMPORTED)\n\n    set_target_properties (URING::uring\n      PROPERTIES\n        IMPORTED_LOCATION ${URING_LIBRARY}\n        INTERFACE_INCLUDE_DIRECTORIES ${URING_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/FindLinuxMembarrier.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_path (LinuxMembarrier_INCLUDE_DIR\n  NAMES linux/membarrier.h)\n\ninclude (CheckCXXSourceCompiles)\nfile (READ ${CMAKE_CURRENT_LIST_DIR}/code_tests/LinuxMembarrier_test.cc _linuxmembarrier_test_code)\ncheck_cxx_source_compiles (\"${_linuxmembarrier_test_code}\" LinuxMembarrier_FOUND)\n\nif (LinuxMembarrier_FOUND)\n  set (LinuxMembarrier_INCLUDE_DIRS ${LinuxMembarrier_INCLUDE_DIR})\nendif ()\n\nif (LinuxMembarrier_FOUND AND NOT (TARGET LinuxMembarrier::membarrier))\n  add_library (LinuxMembarrier::membarrier INTERFACE IMPORTED)\n\n  set_target_properties (LinuxMembarrier::membarrier\n    PROPERTIES\n      INTERFACE_INCLUDE_DIRECTORIES ${LinuxMembarrier_INCLUDE_DIRS})\nendif ()\n"
  },
  {
    "path": "cmake/FindPthreadSetName.cmake",
    "content": "include (CheckSymbolExists)\ninclude (CMakePushCheckState)\n\ncmake_push_check_state (RESET)\nset (CMAKE_REQUIRED_FLAGS \"-pthread\")\nset (CMAKE_REQUIRED_DEFINITIONS \"-D_GNU_SOURCE\")\ncheck_symbol_exists (pthread_setname_np pthread.h HAVE_PTHREAD_SETNAME_NP)\ncmake_pop_check_state ()\n\nfind_package_handle_standard_args (PthreadSetName\n  FOUND_VAR PthreadSetName_FOUND\n  REQUIRED_VARS\n    HAVE_PTHREAD_SETNAME_NP)\n"
  },
  {
    "path": "cmake/FindSanitizers.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\ncmake_policy (PUSH)\ncmake_policy (SET CMP0057 NEW)\n\nif(NOT Sanitizers_FIND_COMPONENTS)\n  set(Sanitizers_FIND_COMPONENTS\n    address\n    undefined_behavior\n    vptr)\nendif()\n\nforeach (component ${Sanitizers_FIND_COMPONENTS})\n  string (TOUPPER ${component} COMPONENT)\n  set (compile_options \"Sanitizers_${COMPONENT}_COMPILE_OPTIONS\")\n  if (component STREQUAL \"address\")\n    list (APPEND ${compile_options} -fsanitize=address)\n  elseif (component STREQUAL \"undefined_behavior\")\n    list (APPEND ${compile_options} -fsanitize=undefined)\n  elseif (component STREQUAL \"vptr\")\n    # since Clang version 21, -fsanitize=undefined no longer implies vptr,\n    # so we enable it explicitly\n    list (APPEND ${compile_options} -fsanitize=vptr)\n  elseif (component STREQUAL \"fuzzer\")\n    # fuzzer-no-link provides instrumentation without the libfuzzer main\n    list (APPEND ${compile_options} -fsanitize=fuzzer-no-link)\n  elseif (component STREQUAL \"fuzzer_main\")\n    # fuzzer provides the full fuzzer with main function\n    list (APPEND ${compile_options} -fsanitize=fuzzer)\n  else ()\n    message (FATAL_ERROR \"Unsupported sanitizer: ${component}\")\n  endif ()\n  list(APPEND Sanitizers_COMPILE_OPTIONS \"${${compile_options}}\")\n  unset (compile_options)\nendforeach ()\n\ninclude(CheckCXXSourceCompiles)\ninclude(CMakePushCheckState)\n\n# -fsanitize=address cannot be combined with -fsanitize=thread, so let's test\n# the combination of the compiler options.\ncmake_push_check_state()\nstring (REPLACE \";\" \" \" CMAKE_REQUIRED_FLAGS \"${Sanitizers_COMPILE_OPTIONS}\")\ncheck_cxx_source_compiles(\"int main() {}\"\n  Sanitizers_SUPPORTED)\nif (Sanitizers_SUPPORTED)\n  if (\"address\" IN_LIST Sanitizers_FIND_COMPONENTS)\n    file (READ ${CMAKE_CURRENT_LIST_DIR}/code_tests/Sanitizers_fiber_test.cc _sanitizers_fiber_test_code)\n    check_cxx_source_compiles (\"${_sanitizers_fiber_test_code}\"\n      Sanitizers_FIBER_SUPPORT)\n  endif ()\nendif ()\ncmake_pop_check_state()\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (Sanitizers\n  REQUIRED_VARS\n    Sanitizers_COMPILE_OPTIONS\n    Sanitizers_SUPPORTED)\n\nif (Sanitizers_FOUND)\n  foreach (component ${Sanitizers_FIND_COMPONENTS})\n    string (TOUPPER ${component} COMPONENT)\n    set (library Sanitizers::${component})\n    if (NOT TARGET ${library})\n      add_library (${library} INTERFACE IMPORTED)\n      set_target_properties (${library}\n        PROPERTIES\n          INTERFACE_COMPILE_OPTIONS \"${Sanitizers_${COMPONENT}_COMPILE_OPTIONS}\"\n          INTERFACE_LINK_LIBRARIES \"${Sanitizers_${COMPONENT}_COMPILE_OPTIONS}\")\n    endif ()\n  endforeach ()\nendif ()\n\ncmake_policy (POP)\n"
  },
  {
    "path": "cmake/FindStdAtomic.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2019 Scylladb, Ltd.\n#\n\nfunction (_stdatomic_can_link var)\n  include (CheckCXXSourceCompiles)\n  set (test_code \"int main() {}\")\n  set (CMAKE_REQUIRED_LIBRARIES -latomic)\n  check_cxx_source_compiles (\"${test_code}\" ${var})\nendfunction ()\n\n_stdatomic_can_link (StdAtomic_EXPLICIT_LINK)\n\n#\n# If linking against `-latomic` is successful, then do it unconditionally.\n#\n\nif (StdAtomic_EXPLICIT_LINK)\n  set (StdAtomic_LIBRARY_NAME atomic)\n  set (StdAtomic_LIBRARIES -l${StdAtomic_LIBRARY_NAME})\n  include (FindPackageHandleStandardArgs)\n\n  find_package_handle_standard_args (StdAtomic\n    REQUIRED_VARS StdAtomic_LIBRARIES)\nendif ()\n\nif (NOT (TARGET StdAtomic::atomic))\n  add_library (StdAtomic::atomic INTERFACE IMPORTED)\n\n  set_target_properties (StdAtomic::atomic\n    PROPERTIES\n      INTERFACE_LINK_LIBRARIES \"${StdAtomic_LIBRARIES}\")\nendif ()\n"
  },
  {
    "path": "cmake/FindSystemTap-SDT.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2023 Scylladb, Ltd.\n#\n\nfind_path (SystemTap-SDT_INCLUDE_DIR\n  NAMES sys/sdt.h)\n\nmark_as_advanced (\n  SystemTap-SDT_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (SystemTap-SDT\n  REQUIRED_VARS SystemTap-SDT_INCLUDE_DIR)\n\nif (NOT TARGET SystemTap::SDT)\n  add_library (SystemTap::SDT INTERFACE IMPORTED)\n  set_target_properties (SystemTap::SDT\n    PROPERTIES\n      INTERFACE_INCLUDE_DIRECTORIES ${SystemTap-SDT_INCLUDE_DIR})\nendif ()\n"
  },
  {
    "path": "cmake/FindValgrind.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2022 Kefu Chai ( tchaikov@gmail.com )\n#\n\n\nfind_package (PkgConfig REQUIRED)\n\npkg_check_modules (PC_valgrind QUIET valgrind)\n\nfind_path (Valgrind_INCLUDE_DIR\n  NAMES valgrind/valgrind.h\n  HINTS\n    ${PC_valgrind_INCLUDEDIR}\n    ${PC_valgrind_INCLUDE_DIRS})\n\nmark_as_advanced (\n  Valgrind_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (Valgrind\n  REQUIRED_VARS\n    Valgrind_INCLUDE_DIR)\n\nif (Valgrind_FOUND)\n  set (Valgrind_INCLUDE_DIRS ${Valgrind_INCLUDE_DIR})\n\n  if (NOT (TARGET Valgrind::valgrind))\n    add_library (Valgrind::valgrind INTERFACE IMPORTED)\n\n    set_target_properties (Valgrind::valgrind\n      PROPERTIES\n        INTERFACE_INCLUDE_DIRECTORIES ${Valgrind_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/Findc-ares.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_check_modules (PC_c-ares QUIET libcares)\n\nfind_library (c-ares_LIBRARY\n  NAMES cares\n  HINTS\n    ${PC_c-ares_LIBDIR}\n    ${PC_c-ares_LIBRARY_DIRS})\n\nfind_path (c-ares_INCLUDE_DIR\n  NAMES ares_dns.h\n  HINTS\n    ${PC_c-ares_INCLUDEDIR}\n    ${PC_c-ares_INCLUDE_DIRS})\n\nif (c-ares_INCLUDE_DIR)\n  foreach (v MAJOR MINOR PATCH)\n    file(STRINGS \"${c-ares_INCLUDE_DIR}/ares_version.h\" ares_VERSION_LINE\n      REGEX \"^#define[ \\t]+ARES_VERSION_${v}[ \\t]+[0-9]+$\")\n    if (ares_VERSION_LINE MATCHES \"ARES_VERSION_${v} ([0-9]+)\")\n      set (c-ares_VERSION_${v} \"${CMAKE_MATCH_1}\")\n    endif ()\n    unset (ares_VERSION_LINE)\n  endforeach ()\n  set (c-ares_VERSION ${c-ares_VERSION_MAJOR}.${c-ares_VERSION_MINOR}.${c-ares_VERSION_PATCH})\nendif ()\n\nmark_as_advanced (\n  c-ares_LIBRARY\n  c-ares_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (c-ares\n  REQUIRED_VARS\n    c-ares_LIBRARY\n    c-ares_INCLUDE_DIR\n  VERSION_VAR c-ares_VERSION)\n\nif (c-ares_FOUND)\n  set (c-ares_LIBRARIES ${c-ares_LIBRARY})\n  set (c-ares_INCLUDE_DIRS ${c-ares_INCLUDE_DIR})\n  if (NOT (TARGET c-ares::cares))\n    add_library (c-ares::cares UNKNOWN IMPORTED)\n\n    set_target_properties (c-ares::cares\n      PROPERTIES\n        IMPORTED_LOCATION ${c-ares_LIBRARY}\n        INTERFACE_INCLUDE_DIRECTORIES ${c-ares_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/Finddpdk.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\npkg_check_modules (dpdk_PC libdpdk)\n\n# we cannot use ${dpdk_PC_STATIC_LDFLAGS} directly, because we want to\n# export DPDK as a bundle of static libraries, so need to find the\n# individual paths to all .a files\nfind_path (dpdk_INCLUDE_DIR\n  NAMES rte_atomic.h\n  HINTS\n    ${dpdk_PC_INCLUDE_DIRS}\n  PATH_SUFFIXES\n    dpdk)\n\nif (dpdk_INCLUDE_DIR AND EXISTS \"${dpdk_INCLUDE_DIR}/rte_build_config.h\")\n  file (STRINGS \"${dpdk_INCLUDE_DIR}/rte_build_config.h\" rte_mbuf_refcnt_atomic\n    REGEX \"^#define[ \\t ]+RTE_MBUF_REFCNT_ATOMIC\")\n  if (rte_mbuf_refcnt_atomic)\n    message (WARNING\n      \"DPDK is configured with RTE_MBUF_REFCNT_ATOMIC enabled, \"\n      \"please disable this option and recompile DPDK for better performance.\")\n  endif ()\nendif ()\n\nset(rte_libs\n  bus_pci\n  bus_vdev\n  cfgfile\n  cmdline\n  cryptodev\n  eal\n  ethdev\n  hash\n  kvargs\n  mbuf\n  mempool\n  mempool_ring\n  net\n  net_bnxt\n  net_cxgbe\n  net_e1000\n  net_ena\n  net_enic\n  net_i40e\n  net_ixgbe\n  net_nfp\n  net_qede\n  net_ring\n  net_sfc\n  net_vmxnet3\n  pci\n  rcu\n  ring\n  security\n  telemetry\n  timer)\n# sfc_efx driver can only build on x86 and aarch64\nif (CMAKE_SYSTEM_PROCESSOR MATCHES \"amd64|x86_64|aarch64\")\n  list (APPEND rte_libs\n    common_sfc_efx)\nendif ()\n\nlist (APPEND dpdk_REQUIRED\n  dpdk_INCLUDE_DIR)\n\n# we prefer static library over the shared library, so just find the\n# static libraries first.\nset (_cmake_find_library_suffixes_saved ${CMAKE_FIND_LIBRARY_SUFFIXES})\nset (CMAKE_FIND_LIBRARY_SUFFIXES\n  ${CMAKE_STATIC_LIBRARY_SUFFIX}\n  ${CMAKE_SHARED_LIBRARY_SUFFIX})\n\nforeach (lib ${rte_libs})\n  string(TOUPPER ${lib} upper_lib)\n  set(library_name \"dpdk_${upper_lib}_LIBRARY\")\n  find_library (${library_name}\n    NAME rte_${lib}\n    HINTS\n      ${dpdk_PC_STATIC_LIBRARY_DIRS})\n  list (APPEND dpdk_REQUIRED\n    ${library_name})\n  list (APPEND dpdk_LIBRARIES\n    ${library_name})\n\n  if (NOT ${library_name})\n    continue()\n  endif ()\n\n  set (library_path ${${library_name}})\n  list (APPEND _dpdk_linker_files ${library_path})\n  set (dpdk_lib dpdk::${lib})\n  list (APPEND _dpdk_libraries ${dpdk_lib})\n\n  if (dpdk_INCLUDE_DIR AND NOT (TARGET ${dpdk_lib}))\n    add_library (${dpdk_lib} UNKNOWN IMPORTED)\n    set_target_properties (${dpdk_lib}\n      PROPERTIES\n        IMPORTED_LOCATION ${library_path}\n        INTERFACE_INCLUDE_DIRECTORIES ${dpdk_INCLUDE_DIR})\n  endif ()\nendforeach ()\n\n# restore the previous saved suffixes\nset (CMAKE_FIND_LIBRARY_SUFFIXES ${_cmake_find_library_suffixes_saved})\n\ninclude (FindPackageHandleStandardArgs)\nfind_package_handle_standard_args (dpdk\n  REQUIRED_VARS\n    ${dpdk_REQUIRED})\n\n# DPDK's build system adds certain dependencies conditionally based on what's available\n# at build time. While most libraries from dpdk_PC_LIBRARIES are handled through the\n# rte_libs logic elsewhere, external dependencies ('bsd' and 'numa' in this case) are\n# explicitly handled below. This foreach loop checks if these specific libraries are\n# present in dpdk_PC_LIBRARIES and adds them to the dpdk_dependencies list if found.\nforeach (lib \"bsd\" \"numa\")\n  if (lib IN_LIST dpdk_PC_STATIC_LIBRARIES)\n    list (APPEND dpdk_dependencies ${lib})\n  endif()\nendforeach ()\n\n# As of DPDK 23.07, if libarchive-dev is present, it will make DPDK depend on the library.\n# Unfortunately DPDK also has a bug in its .pc file generation and will not include libarchive\n# dependency under any circumstance. Accordingly, the dependency is added explicitly if libarchive\n# exists.\npkg_check_modules (libarchive_PC QUIET libarchive)\nlist(APPEND dpdk_dependencies ${libarchive_PC_LIBRARIES})\n\nif (dpdk_FOUND AND NOT (TARGET dpdk))\n  get_filename_component (library_suffix \"${dpdk_EAL_LIBRARY}\" LAST_EXT)\n  # strictly speaking, we should have being using check_c_compiler_flag()\n  # here, but we claim Seastar as a project written in CXX language, and\n  # C is not enabled, so CXX is used here instead.\n  include(CheckCXXCompilerFlag)\n  check_cxx_compiler_flag(\"-Wno-volatile\" _warning_supported_volatile)\n  if(_warning_supported_volatile)\n    # include/generic/rte_spinlock.h increments volatiled-qualified type with\n    # \"++\". but this is deprecated by GCC, so silence it.\n    set(compile_options\n      INTERFACE_COMPILE_OPTIONS \"-Wno-volatile\")\n  endif()\n  if (library_suffix STREQUAL CMAKE_STATIC_LIBRARY_SUFFIX)\n    # No pmd driver code will be pulled in without \"--whole-archive\". To\n    # avoid exposing that to seastar users, combine dpdk into a single\n    # .o file.\n    set (dpdk_object_path \"${CMAKE_BINARY_DIR}/dpdk.o\")\n    add_custom_command (\n      OUTPUT ${dpdk_object_path}\n      COMMAND ${CMAKE_CXX_COMPILER}\n        -r # create a relocatable object\n        -o ${dpdk_object_path}\n        -Wl,--whole-archive ${_dpdk_linker_files}\n      DEPENDS\n        ${_dpdk_linker_files})\n    add_custom_target (dpdk_object\n      DEPENDS ${dpdk_object_path})\n\n    add_library (dpdk OBJECT IMPORTED)\n    add_dependencies (dpdk dpdk_object)\n    set_target_properties (dpdk\n      PROPERTIES\n        INTERFACE_INCLUDE_DIRECTORIES ${dpdk_INCLUDE_DIR}\n        INTERFACE_LINK_LIBRARIES \"${dpdk_dependencies}\"\n        IMPORTED_OBJECTS ${dpdk_object_path}\n        ${compile_options})\n    # we include dpdk in seastar already, but we need to pull in the\n    # dependency libraries linked by dpdk\n    list(TRANSFORM dpdk_dependencies PREPEND \"-l\" OUTPUT_VARIABLE dpdk_LIBRARIES)\n    add_library (DPDK::dpdk ALIAS dpdk)\n  else ()\n    set (dpdk_LIBRARIES ${dpdk_PC_LDFLAGS})\n    add_library (DPDK::dpdk INTERFACE IMPORTED)\n    set_target_properties (DPDK::dpdk\n      PROPERTIES\n        INTERFACE_INCLUDE_DIRECTORIES \"${dpdk_PC_INCLUDE_DIRS}\"\n        INTERFACE_LINK_LIBRARIES \"${_dpdk_libraries};${dpdk_dependencies}\"\n        ${compile_options})\n  endif()\nendif ()\n"
  },
  {
    "path": "cmake/Findhwloc.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_search_module (PC_hwloc QUIET hwloc)\n\nfind_library (hwloc_LIBRARY\n  NAMES hwloc\n  HINTS\n    ${PC_hwloc_LIBDIR}\n    ${PC_hwloc_LIBRARY_DIRS})\n\nfind_path (hwloc_INCLUDE_DIR\n  NAMES hwloc.h\n  HINTS\n    ${PC_hwloc_INCLUDEDIR}\n    ${PC_hwloc_INCLUDE_DIRS})\n\nmark_as_advanced (\n  hwloc_LIBRARY\n  hwloc_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (hwloc\n  REQUIRED_VARS\n    hwloc_LIBRARY\n    hwloc_INCLUDE_DIR\n  VERSION_VAR hwloc_VERSION)\n\nif (hwloc_FOUND)\n  set (hwloc_LIBRARIES ${hwloc_LIBRARY})\n  set (hwloc_INCLUDE_DIRS ${hwloc_INCLUDE_DIR})\n  if (NOT (TARGET hwloc::hwloc))\n    add_library (hwloc::hwloc UNKNOWN IMPORTED)\n\n    set_target_properties (hwloc::hwloc\n      PROPERTIES\n        IMPORTED_LOCATION ${hwloc_LIBRARY}\n        INTERFACE_INCLUDE_DIRECTORIES ${hwloc_INCLUDE_DIRS}\n        INTERFACE_SYSTEM_INCLUDE_DIRECTORIES ${hwloc_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/Findlksctp-tools.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_library (lksctp-tools_LIBRARY\n  NAMES sctp)\n\nfind_path (lksctp-tools_INCLUDE_DIR\n  NAMES netinet/sctp.h)\n\nmark_as_advanced (\n  lksctp-tools_LIBRARY\n  lksctp-tools_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (lksctp-tools\n  REQUIRED_VARS\n    lksctp-tools_LIBRARY\n    lksctp-tools_INCLUDE_DIR)\n\nset (lksctp-tools_LIBRARIES ${lksctp-tools_LIBRARY})\nset (lksctp-tools_INCLUDE_DIRS ${lksctp-tools_INCLUDE_DIR})\n\nif (lksctp-tools_FOUND AND NOT (TARGET lksctp-tools::lksctp-tools))\n  add_library (lksctp-tools::lksctp-tools UNKNOWN IMPORTED)\n\n  set_target_properties (lksctp-tools::lksctp-tools\n    PROPERTIES\n      IMPORTED_LOCATION ${lksctp-tools_LIBRARIES}\n      INTERFACE_INCLUDE_DIRECTORIES ${lksctp-tools_INCLUDE_DIRS})\nendif ()\n\nmark_as_advanced (\n  lksctp-tools_INCLUDE_DIR\n  lksctp-tools_LIBRARY)\n"
  },
  {
    "path": "cmake/Findlz4.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_search_module (PC_lz4 QUIET liblz4)\n\nfind_library (lz4_LIBRARY\n  NAMES lz4\n  HINTS\n    ${PC_lz4_LIBDIR}\n    ${PC_lz4_LIBRARY_DIRS})\n\nfind_path (lz4_INCLUDE_DIR\n  NAMES lz4.h\n  HINTS\n    ${PC_lz4_INCLUDEDIR}\n    ${PC_lz4_INCLUDE_DIRS})\n\nmark_as_advanced (\n  lz4_LIBRARY\n  lz4_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (lz4\n  REQUIRED_VARS\n    lz4_LIBRARY\n    lz4_INCLUDE_DIR\n  VERSION_VAR PC_lz4_VERSION)\n\nif (lz4_FOUND)\n  set (CMAKE_REQUIRED_LIBRARIES ${lz4_LIBRARY})\n\n  set (lz4_LIBRARIES ${lz4_LIBRARY})\n  set (lz4_INCLUDE_DIRS ${lz4_INCLUDE_DIR})\n\n  if (NOT (TARGET lz4::lz4))\n    add_library (lz4::lz4 UNKNOWN IMPORTED)\n\n    set_target_properties (lz4::lz4\n      PROPERTIES\n        IMPORTED_LOCATION ${lz4_LIBRARY}\n        INTERFACE_INCLUDE_DIRECTORIES ${lz4_INCLUDE_DIRS})\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/Findragel.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_program (\n  ragel_RAGEL_EXECUTABLE\n  ragel)\nif (NOT ragel_RAGEL_EXECUTABLE)\n  message (FATAL_ERROR \"ragel is required for processing .rl source files!\")\nendif ()\n\nmark_as_advanced (ragel_RAGEL_EXECUTABLE)\n\nset (_ragel_version_pattern \"[0-9]+\\\\.[0-9]+\\\\.[0-9]+(\\\\.[0-9]+)?\")\n\nif (ragel_RAGEL_EXECUTABLE)\n  set (ragel_FOUND ON)\n\n  execute_process (COMMAND ${ragel_RAGEL_EXECUTABLE} -v\n    OUTPUT_VARIABLE _ragel_version_output)\n\n  if (${_ragel_version_output} MATCHES \"version (${_ragel_version_pattern})\")\n    set (ragel_VERSION ${CMAKE_MATCH_1})\n  endif ()\nendif ()\n\nfind_package_handle_standard_args (ragel\n  REQUIRED_VARS ragel_RAGEL_EXECUTABLE\n  VERSION_VAR ragel_VERSION)\n"
  },
  {
    "path": "cmake/Findrt.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nset (_rt_test_source ${CMAKE_CURRENT_LIST_DIR}/code_tests/rt_test.cc)\n\n# Try to compile without the library first.\ntry_compile (rt_NO_EXPLICIT_LINK\n  ${CMAKE_CURRENT_BINARY_DIR}\n  SOURCES ${_rt_test_source})\n\nif (rt_NO_EXPLICIT_LINK)\n  set (rt_FOUND yes)\nelse ()\n  # The `rt` library is required.\n\n  try_compile (_rt_test\n    ${CMAKE_CURRENT_BINARY_DIR}\n    SOURCES ${_rt_test_source}\n    LINK_LIBRARIES rt)\n\n  if (_rt_test)\n    set (rt_LIBRARY_NAME rt)\n    set (rt_LIBRARIES -l${rt_LIBRARY_NAME})\n  endif ()\n\n  include (FindPackageHandleStandardArgs)\n\n  find_package_handle_standard_args (rt\n    REQUIRED_VARS rt_LIBRARIES)\nendif ()\n\nif (rt_FOUND AND NOT (TARGET rt::rt))\n  add_library (rt::rt INTERFACE IMPORTED)\n\n  set_target_properties (rt::rt\n    PROPERTIES\n      INTERFACE_LINK_LIBRARIES \"${rt_LIBRARIES}\")\nendif ()\n"
  },
  {
    "path": "cmake/Finducontext.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2023 Scylladb, Ltd.\n#\n\n# Try to compile without the library first.\ninclude (CheckFunctionExists)\ncheck_function_exists (getcontext\n  ucontext_NO_EXPLICIT_LINK)\n\nif (ucontext_NO_EXPLICIT_LINK)\n  set (ucontext_FOUND yes)\nelse ()\n  # The `libucontext` library is required.\n  find_package (PkgConfig QUIET REQUIRED)\n  pkg_check_modules (PC_ucontext QUIET ucontext)\n  find_library (ucontext_LIBRARY\n    NAMES ucontext\n    HINTS\n      ${PC_ucontext_LIBDIR}\n      ${PC_ucontext_LIBRARY_DIRS})\n  mark_as_advanced (ucontext_LIBRARY)\n  include (FindPackageHandleStandardArgs)\n  find_package_handle_standard_args (ucontext\n    REQUIRED_VARS ucontext_LIBRARY)\nendif ()\n\nif (ucontext_FOUND AND NOT (TARGET ucontext::ucontext))\n  add_library (ucontext::ucontext INTERFACE IMPORTED)\n\n  set_target_properties (ucontext::ucontext\n    PROPERTIES\n      INTERFACE_LINK_LIBRARIES \"${ucontext_LIBRARY}\")\nendif ()\n"
  },
  {
    "path": "cmake/Findyaml-cpp.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfind_package (PkgConfig REQUIRED)\n\npkg_search_module (PC_yaml-cpp QUIET yaml-cpp)\n\nfind_library (yaml-cpp_LIBRARY_RELEASE\n NAMES yaml-cpp\n HINTS\n   ${PC_yaml-cpp_LIBDIR}\n   ${PC_yaml-cpp_LIBRARY_DIRS})\n\nfind_library (yaml-cpp_LIBRARY_DEBUG\n NAMES yaml-cppd\n HINTS\n   ${PC_yaml-cpp_LIBDIR}\n   ${PC_yaml-cpp_LIBRARY_DIRS})\n\ninclude (SelectLibraryConfigurations)\nselect_library_configurations (yaml-cpp)\n\nfind_path (yaml-cpp_INCLUDE_DIR\n  NAMES yaml-cpp/yaml.h\n  PATH_SUFFIXES yaml-cpp\n  HINTS\n    ${PC_yaml-cpp_INCLUDEDIR}\n    ${PC_yaml-cpp_INCLUDE_DIRS})\n\nmark_as_advanced (\n  yaml-cpp_LIBRARY_RELEASE\n  yaml-cpp_LIBRARY_DEBUG\n  yaml-cpp_INCLUDE_DIR)\n\ninclude (FindPackageHandleStandardArgs)\n\nfind_package_handle_standard_args (yaml-cpp\n  REQUIRED_VARS\n    yaml-cpp_LIBRARY\n    yaml-cpp_INCLUDE_DIR\n  VERSION_VAR yaml-cpp_VERSION)\n\nif (yaml-cpp_FOUND)\n  set (yaml-cpp_LIBRARIES ${yaml-cpp_LIBRARY})\n  set (yaml-cpp_INCLUDE_DIRS ${yaml-cpp_INCLUDE_DIR})\n  if (NOT (TARGET yaml-cpp::yaml-cpp))\n    add_library (yaml-cpp::yaml-cpp UNKNOWN IMPORTED)\n\n    set_target_properties (yaml-cpp::yaml-cpp\n      PROPERTIES\n        INTERFACE_INCLUDE_DIRECTORIES ${yaml-cpp_INCLUDE_DIRS})\n    if (EXISTS \"${yaml-cpp_LIBRARY}\")\n      set_target_properties (yaml-cpp::yaml-cpp\n        PROPERTIES\n          IMPORTED_LOCATION \"${yaml-cpp_LIBRARY}\")\n    endif ()\n    foreach (build \"RELEASE\" \"DEBUG\")\n      if (yaml-cpp_LIBRARY_${build})\n        set_property (TARGET yaml-cpp::yaml-cpp APPEND PROPERTY\n          IMPORTED_CONFIGURATIONS \"${build}\")\n        set_target_properties (yaml-cpp::yaml-cpp PROPERTIES\n          IMPORTED_LOCATION_${build} \"${yaml-cpp_LIBRARY_${build}}\")\n      endif ()\n    endforeach ()\n  endif ()\nendif ()\n"
  },
  {
    "path": "cmake/SeastarConfig.cmake.in",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nlist (APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})\n\nif (CMAKE_CXX_STANDARD)\n  set (current_cxx_standard ${CMAKE_CXX_STANDARD})\nelse ()\n  set (current_cxx_standard ${CMAKE_CXX_STANDARD_DEFAULT})\nendif ()\n\nif (NOT (current_cxx_standard STREQUAL @CMAKE_CXX_STANDARD@))\n  message(WARNING\n    \"C++ Standard mismatch detected:\n- Seastar was compiled with: C++@CMAKE_CXX_STANDARD@\n- This project is configured to use: C++${current_cxx_standard}\nThis mismatch may lead to build failures due to differences in the supported \\\nfeatures of these two standards. Please adjust your project's C++ standard to \\\nmatch Seastar.\")\nendif ()\n\nif (NOT ((CMAKE_CXX_COMPILER_ID STREQUAL @CMAKE_CXX_COMPILER_ID@) AND\n         (CMAKE_CXX_COMPILER_VERSION VERSION_EQUAL @CMAKE_CXX_COMPILER_VERSION@)))\n  message (WARNING\n    \"Compiler mismatch detected:\n- Seastar was compiled with: @CMAKE_CXX_COMPILER_ID@ @CMAKE_CXX_COMPILER_VERSION@ \\\n- This project is configured to use: ${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION}\nThis mismatch may lead to build failures due to differences in supported features and ABI compatibility. \\\nPlease configure your project to use the same C++ compiler to match Seastar.\")\nendif ()\n\n#\n# Dependencies.\n#\n\ninclude (SeastarDependencies)\nset (Seastar_DPDK @Seastar_DPDK@)\nset (Seastar_IO_URING @Seastar_IO_URING@)\nset (Seastar_HWLOC @Seastar_HWLOC@)\nseastar_find_dependencies ()\n\nif (NOT TARGET Seastar::seastar)\n  include (\"${CMAKE_CURRENT_LIST_DIR}/SeastarTargets.cmake\")\nendif ()\n"
  },
  {
    "path": "cmake/SeastarDependencies.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2019 Scylladb, Ltd.\n#\n\ninclude(CMakeParseArguments)\n\n# This is required because cmake-boost may return to Boost_{component}_LIBRARY:\n# - /usr/lib64/libboost_foo.so\n# - Boost::foo\n# While pkgconf's .pc file consumers expect argument which can be passed as\n# part of the command line arguments\nset (Boost_NO_BOOST_CMAKE ON)\n\n# for including the fix of https://github.com/boostorg/test/pull/252\nset (_seastar_boost_version 1.73.0)\n\n# This is the minimum version of Boost we need the CMake-bundled `FindBoost.cmake` to know about.\nfind_package (Boost ${_seastar_boost_version})\nif (Boost_VERSION_STRING VERSION_LESS 1.81.0)\n  set_target_properties (Boost::boost PROPERTIES\n    INTERFACE_COMPILE_DEFINITIONS \"BOOST_NO_CXX98_FUNCTION_BASE\")\nendif ()\n\nif (CMAKE_FIND_PACKAGE_NAME)\n  # used inside find_package(Seastar)\n  include (CMakeFindDependencyMacro)\n\n  macro (seastar_find_dep package)\n    cmake_parse_arguments(args \"REQUIRED\" \"\" \"\" ${ARGN})\n    if (arg_REQUIRED)\n      find_dependency (${package} ${arg_UNPARSED_ARGUMENTS})\n    else ()\n      # some packages are not REQUIRED, so we just check for them instead of\n      # populating \"REQUIRED\" from the original find_package() call.\n      find_package (${package} ${ARGN})\n    endif ()\n  endmacro ()\nelse()\n  macro (seastar_find_dep package)\n    # used when configuring Seastar\n    find_package (${package} ${ARGN})\n  endmacro ()\nendif ()\n\nmacro (seastar_find_dependencies)\n  #\n  # List of Seastar dependencies that is meant to be used\n  # both in Seastar configuration and by clients which\n  # consume Seastar via SeastarConfig.cmake.\n  #\n  # `unit_test_framework` is not required in the case we are building Seastar\n  # without the testing library, however the component is always specified as required\n  # to keep the CMake code minimalistic and easy-to-use.\n  seastar_find_dep (Boost ${_seastar_boost_version} REQUIRED\n    COMPONENTS\n      filesystem\n      program_options\n      thread\n      unit_test_framework)\n  seastar_find_dep (c-ares 1.13 REQUIRED)\n  if (c-ares_VERSION VERSION_GREATER_EQUAL 1.33.0 AND c-ares_VERSION VERSION_LESS 1.34.1)\n    # https://github.com/scylladb/seastar/issues/2472\n    message (FATAL_ERROR\n      \"c-ares ${c-ares_VERSION} is not supported. \"\n      \"Seastar requires c-ares version <1.33 or >=1.34.1 \")\n  endif ()\n\n  if (Seastar_DPDK)\n    seastar_find_dep (dpdk)\n  endif()\n  seastar_find_dep (fmt 8.1.1 REQUIRED)\n  seastar_find_dep (lz4 1.7.3 REQUIRED)\n  seastar_find_dep (GnuTLS 3.3.26 REQUIRED)\n  if (Seastar_IO_URING)\n    seastar_find_dep (LibUring 2.0 REQUIRED)\n  endif()\n  seastar_find_dep (LinuxMembarrier)\n  seastar_find_dep (Sanitizers)\n  seastar_find_dep (StdAtomic REQUIRED)\n  seastar_find_dep (SystemTap-SDT)\n  if (Seastar_HWLOC)\n    seastar_find_dep (hwloc 1.11.2 REQUIRED)\n  endif()\n  seastar_find_dep (lksctp-tools REQUIRED)\n  seastar_find_dep (rt REQUIRED)\n  seastar_find_dep (ucontext REQUIRED)\n  seastar_find_dep (yaml-cpp REQUIRED\n    VERSION 0.5.1)\n\n  # workaround for https://gitlab.kitware.com/cmake/cmake/-/issues/25079\n  # since protobuf v22.0, it started using abseil, see\n  # https://github.com/protocolbuffers/protobuf/releases/tag/v22.0 .\n  # but due to https://gitlab.kitware.com/cmake/cmake/-/issues/25079,\n  # CMake's FindProtobuf does add this linkage yet. fortunately,\n  # ProtobufConfig.cmake provided by protobuf defines this linkage. so we try\n  # the CMake package configuration file first, and fall back to CMake's\n  # FindProtobuf module.\n  find_package (Protobuf QUIET CONFIG)\n  if (Protobuf_FOUND AND Protobuf_VERSION VERSION_GREATER_EQUAL 2.5.0)\n    # do it again, so the message is printed when the package is found\n    seastar_find_dep (Protobuf CONFIG REQUIRED)\n  else ()\n    seastar_find_dep (Protobuf 2.5.0 REQUIRED)\n  endif ()\n\nendmacro ()\n"
  },
  {
    "path": "cmake/TriStateOption.cmake",
    "content": "# the \"option()\" defined by CMake represents a boolean. but somtimes, we want\n# to enable/disable it depending on the CMAKE_BUILD_TYPE, if user leaves the\n# option unset.\nfunction (tri_state_option option)\n  cmake_parse_arguments (\n    parsed_args\n    \"\"\n    \"CONDITION\"\n    \"DEFAULT_BUILD_TYPES\"\n    ${ARGN})\n\n  get_property(is_multi_config GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)\n  if (is_multi_config)\n    set (all_build_types ${CMAKE_CONFIGURATION_TYPES})\n  else ()\n    set (all_build_types ${CMAKE_BUILD_TYPE})\n  endif ()\n\n  # generic boolean values passed as string, potentially from configure.py\n  set (True_STRING_VALUES \"ON\" \"yes\" \"Yes\" \"YES\" \"true\" \"True\" \"TRUE\")\n  set (Default_STRING_VALUES \"DEFAULT\" \"default\" \"Default\")\n\n  if (\"${option}\" IN_LIST True_STRING_VALUES)\n    set (enabled_types ${all_build_types})\n  elseif (\"${option}\" IN_LIST Default_STRING_VALUES)\n    set (enabled_types ${parsed_args_DEFAULT_BUILD_TYPES})\n  else ()\n    set (enabled_types \"\")\n  endif ()\n\n  if (is_multi_config)\n    set (${parsed_args_CONDITION} \"$<IN_LIST:$<CONFIG>,${enabled_types}>\" PARENT_SCOPE)\n  elseif (CMAKE_BUILD_TYPE IN_LIST enabled_types)\n    set (${parsed_args_CONDITION} 1 PARENT_SCOPE)\n  else ()\n    set (${parsed_args_CONDITION} 0 PARENT_SCOPE)\n  endif ()\nendfunction ()\n"
  },
  {
    "path": "cmake/check-seastar-include-style.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nimport fileinput\nimport os.path\nimport re\nimport sys\n\n\ndef check_includes(files, dirname):\n    # Check for include directives with quotes for specified dirname\n    incorrect_include = re.compile(rf'#include\\s+\"({dirname}/[^\\\"]+)\"')\n    num_errors = 0\n    for line in fileinput.input(files=files, encoding=\"utf-8\"):\n        # Look for #include \\\"seastar/...\\\" pattern\n        if matched := incorrect_include.match(line):\n            location = f\"{fileinput.filename()}:{fileinput.lineno()}\"\n            header = matched.group(1)\n            print(f\"{location}: warning: please include seastar headers using: #include <{header}>\")\n            num_errors += 1\n    return num_errors\n\n\ndef main():\n    # If any incorrect includes are found, fail the check\n    files = [fn for fn in sys.argv[1:] if os.path.exists(fn)]\n    if check_includes(files, \"seastar\") > 0:\n        sys.exit(1)\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "cmake/code_tests/LinuxMembarrier_test.cc",
    "content": "extern \"C\" {\n#include <linux/membarrier.h>\n}\n\nint main() {\n    int x = MEMBARRIER_CMD_PRIVATE_EXPEDITED | MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED;\n    (void)x;\n}\n"
  },
  {
    "path": "cmake/code_tests/Sanitizers_fiber_test.cc",
    "content": "#include <cstddef>\n\nextern \"C\" {\n    void __sanitizer_start_switch_fiber(void**, const void*, size_t);\n    void __sanitizer_finish_switch_fiber(void*, const void**, size_t*);\n}\n\nint main() {\n    __sanitizer_start_switch_fiber(nullptr, nullptr, 0);\n    __sanitizer_finish_switch_fiber(nullptr, nullptr, nullptr);\n}\n"
  },
  {
    "path": "cmake/code_tests/Source_location_default_argument.cc",
    "content": "#include<source_location>\n\nint test_source_location(int line,\n                         std::source_location loc = std::source_location::current()) {\n    return line == loc.line() ? 0 : 1;\n}\n\nint main() {\n    return test_source_location(__LINE__);\n}\n"
  },
  {
    "path": "cmake/code_tests/Source_location_test.cc",
    "content": "#if __has_include(<source_location>)\n#include <source_location>\n#endif\n\n#ifdef __cpp_lib_source_location\nusing source_location = std::source_location;\n#elif __has_include(<experimental/source_location>)\n#include <experimental/source_location>\nusing source_location = std::experimental::source_location;\n#endif\n\n#if defined(__cpp_lib_source_location) || defined(__cpp_lib_experimental_source_location)\nstruct format_info {\n    format_info(source_location loc = source_location::current()) noexcept\n        : loc(loc)\n    { }\n    source_location loc;\n};\n#else\nstruct format_info { };\n#endif\n\nint main()\n{\n    format_info fi;\n}\n"
  },
  {
    "path": "cmake/code_tests/rt_test.cc",
    "content": "extern \"C\" {\n#include <signal.h>\n#include <time.h>\n}\n\nint main() {\n    timer_t td;\n    struct sigevent sev;\n    timer_create(CLOCK_MONOTONIC, &sev, &td);\n}\n"
  },
  {
    "path": "cmake/code_tests/stdout_test.cc",
    "content": "#include <cstdio>\n\nenum class logger_type {\n  stdout,\n  stderr,\n};\n\nint main() {}\n"
  },
  {
    "path": "coding-style.md",
    "content": "# Seastar Coding Style\n\n## Files\n\nHeader files have the `.hh` extension, source files use the `.cc` extension. All files must have a license and copyright blurb. Use `#pragma once` instead of an include guard.\n\nHeader files which contain a public part of the interface of Seastar go in the `include` directory. Internal header and source files which are private to the implementation go in the `src` directory.\n\n## Whitespace\n\nUse spaces only; NEVER tabs. Rationale: tabs render differently on each system.\n\nAn _indent_ is four spaces. A double indent is eight spaces, a half-indent is two spaces.\n\n## Naming\n\nWe follow the C++ and Boost naming conventions: class names, variables, functions, and concepts are `words_separated_by_whitespace`.\n\nPrivate data members are prefixed by an underscore:\n\n```c++\nclass my_class {\n    int _a_member;\npublic:\n    void foo() {\n        _a_member = 3;\n    }\n};\n```\n\nThink of the leading underscore as a shorthand for `this->`.\n\nTemplate parameters use `CamelCase`\n\nNote: because the Concept Technical Specification used CamelCase for concepts,\nsome Seastar concepts also use CamelCase. These will be gradually deprecated\nand replaced with snake_case names. New concepts should use snake_case.\n\n## Including header files\n\nIn any file, to include a public header file (one in the `include` directory), use an absolute path with `<>` like this:\n\n```c++\n#include <seastar/core/future.hh>\n```\n\nIn any private file, to include a private header file (one in the `src` directory), use an absolute path with `\"\"` like this:\n\n```c++\n#include \"core/future_impl.hh\"\n```\n\nHeader files in Seastar must be self-contained, i.e., each can be included without having to include specific other headers first. To verify that your change did not break this property, run `ninja checkheaders` in the build directory.\n\n## Braced blocks\n\nAll nested scopes are braced, even when the language allows omitting the braces (such as an if-statement), this makes patches simpler and is more consistent. The opening brace is merged with the line that opens the scope (class definition, function definition, if statement, etc.) and the body is indented.\n\n```c++\nvoid a_function() {\n    if (some condition) {\n        stmt;\n    } else {\n        stmt;\n    }\n}\n```\n\nAn exception is namespaces -- the body is _not_ indented, to prevent files that are almost 100% whitespace left margin.\n\nWhen making a change, if you need to insert an indentation level, you can temporarily break the rules by inserting a half-indent, so that the patch is easily reviewable:\n\n```c++\nvoid a_function() {\n  while (something) {   // new line - half indent\n    if (some condition) {\n        stmt;\n    } else {\n        stmt;\n    }\n  }                      // new line\n}\n```\n\nA follow-up patch can restore the indents without any functional changes.\n\n## Function parameters\n\nAvoid output parameters; use return values instead.  In/out parameters are tricky, but in some cases they are relatively standard, such as serialization/deserialization.\n\nIf a function accepts a lambda or an `std::function`, make it the last argument, so that it can be easily provided inline:\n\n```c++\ntemplate <typename Func>\nint function_accepting_a_lambda(int a, int b, Func func);\n\nint f() {\n    return function_accepting_a_lambda(2, 3, [] (int x, int y) {\n        return x + y;\n    });\n}\n```\n\n## Complex return types\n\nIf a function returns a complicated return type, put its return type on a separate line, otherwise it becomes hard to see where the return type ends and where the function name begins:\n\n```c++\ntemplate <typename T1, T2>\ntemplate <typename T3, T4>\nstd::vector<typename a_struct<T1, T2>::some_nested_class<T3, T4>>  // I'm the return type\na_struct<T1, T2>::a_function(T3 a, T4 b) {                         // And I'm the function name\n    // ...\n}\n```\n\n## Whitespace around operators\n\nWhitespace around operators should match their precedence: high precedence = no spaces, low precedence = add spaces:\n\n```c++\n     return *a + *b;  // good\n     return * a+* b;  // bad\n```\n\n`if`, `while`, `return` (and `template`) are not function calls, so they get a space after the keyword.\n\n## Long lines\n\nIf a line becomes excessively long (>160 characters?), or is just complicated, break it into two or more lines.  The second (and succeeding lines) are _continuation lines_, and have a double indent:\n\n```c++\n    if ((some_condition && some_other_condition)\n            || (more complicated stuff here...)   // continuation line, double indent\n            || (even more complicated stuff)) {   // another continuation line\n        do_something();  // back to single indent\n    }\n```\n\nOf course, long lines or complex conditions may indicate that refactoring is in order.\n\n## Generic lambdas and types\n\nGeneric lambdas (`[] (auto param)`) are discouraged where the type is known. Generic\nlambdas reduce the compiler's and other tools' ability to reason about the code.\nIn case the actual type of `param` doesn't match the programmers expectations,\nthe compiler will only detect an error in the lambda body, or perhaps\neven lower down the stack if more generic functions are called. In the case of an\nIDE, most of its functionality is disabled in a generic lambda, since it can't\nassume anything about that parameter.\n\nOf course, when there is a need to support multiple types, genericity is the correct\ntool. Even then, type parameters should be constrained with concepts, in order to\ncatch type mismatches early rather than deep in the instantiation chain.\n\n\n"
  },
  {
    "path": "configure.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\nimport argparse\nimport os\nimport re\nimport seastar_cmake\nfrom shutil import which\nimport subprocess\nimport sys\nimport tempfile\n\ntempfile.tempdir = \"./build/tmp\"\n\n\ndef add_tristate(arg_parser, name, dest, help, default=None):\n    arg_parser.add_argument('--enable-' + name, dest=dest, action='store_true', default=default,\n                            help='Enable ' + help + ' [default]' if default else '')\n    arg_parser.add_argument('--disable-' + name, dest=dest, action='store_false', default=None,\n                            help='Disable ' + help)\n\n\ndef try_compile(compiler, source='', flags=[]):\n    return try_compile_and_link(compiler, source, flags=flags + ['-c'])\n\n\ndef ensure_tmp_dir_exists():\n    if not os.path.exists(tempfile.tempdir):\n        os.makedirs(tempfile.tempdir)\n\n\ndef try_compile_and_link(compiler, source='', flags=[]):\n    ensure_tmp_dir_exists()\n    with tempfile.NamedTemporaryFile() as sfile:\n        ofd, ofile = tempfile.mkstemp()\n        os.close(ofd)\n        try:\n            sfile.file.write(bytes(source, 'utf-8'))\n            sfile.file.flush()\n            # We can't write to /dev/null, since in some cases (-ftest-coverage) gcc will create an auxiliary\n            # output file based on the name of the output file, and \"/dev/null.gcsa\" is not a good name\n            return subprocess.call([compiler, '-x', 'c++', '-o', ofile, sfile.name] + flags,\n                                   stdout=subprocess.DEVNULL,\n                                   stderr=subprocess.DEVNULL) == 0\n        finally:\n            if os.path.exists(ofile):\n                os.unlink(ofile)\n\n\ndef standard_supported(standard, compiler='g++'):\n    return try_compile(compiler=compiler, source='', flags=['-std=' + standard])\n\n\ndef find_compiler_cache(preference):\n    \"\"\"\n    Find a compiler cache based on the preference.\n\n    Args:\n        preference: One of 'auto', 'sccache', 'ccache', 'none', or a path to a binary.\n\n    Returns:\n        Path to the compiler cache binary, or empty string if not found/disabled.\n    \"\"\"\n    if preference == 'none':\n        return ''\n\n    if preference == 'auto':\n        # Prefer sccache over ccache\n        for cache in ['sccache', 'ccache']:\n            path = which(cache)\n            if path:\n                return path\n        return ''\n\n    if preference in ('sccache', 'ccache'):\n        path = which(preference)\n        if path:\n            return path\n        print(f\"Warning: {preference} not found on PATH, disabling compiler cache\")\n        return ''\n\n    # Assume it's a path to a binary\n    if os.path.isfile(preference) and os.access(preference, os.X_OK):\n        return preference\n\n    print(f\"Warning: compiler cache '{preference}' not found or not executable, disabling compiler cache\")\n    return ''\n\n\ndef find_compiler(name):\n    \"\"\"\n    Find a compiler by name, skipping ccache wrapper directories.\n\n    This is useful when using sccache to avoid double-caching through ccache.\n\n    Args:\n        name: The compiler name (e.g., 'clang++', 'clang', 'gcc')\n\n    Returns:\n        Path to the compiler, skipping ccache directories, or None if not found.\n    \"\"\"\n    ccache_dirs = {'/usr/lib/ccache', '/usr/lib64/ccache'}\n    for path_dir in os.environ.get('PATH', '').split(os.pathsep):\n        # Skip ccache wrapper directories\n        if os.path.realpath(path_dir) in ccache_dirs or path_dir in ccache_dirs:\n            continue\n        candidate = os.path.join(path_dir, name)\n        if os.path.isfile(candidate) and os.access(candidate, os.X_OK):\n            return candidate\n    return None\n\n\ndef resolve_compilers_for_compiler_cache(args, compiler_cache):\n    \"\"\"\n    When using a compiler cache, resolve compiler paths to avoid ccache directories.\n\n    This prevents double-caching when ccache symlinks are in PATH.\n\n    Args:\n        args: The argument namespace with cc and cxx attributes.\n        compiler_cache: Path to the compiler cache binary, or empty string.\n    \"\"\"\n    if not compiler_cache:\n        return\n    if not os.path.isabs(args.cxx):\n        real_cxx = find_compiler(args.cxx)\n        if real_cxx:\n            args.cxx = real_cxx\n    if not os.path.isabs(args.cc):\n        real_cc = find_compiler(args.cc)\n        if real_cc:\n            args.cc = real_cc\n\n\narg_parser = argparse.ArgumentParser('Configure seastar')\narg_parser.add_argument('--mode', action='store', choices=seastar_cmake.SUPPORTED_MODES + ['all'], default='all')\narg_parser.add_argument('--build-root', action='store', default=seastar_cmake.DEFAULT_BUILD_ROOT, type=str,\n                        help='The name of the build root build directoy: using a different name allows multiple '\n                        'configurations to co-exist in the same repository')\narg_parser.add_argument('--cflags', action = 'store', dest='user_cflags', default='',\n                        help='Extra flags for the C++ compiler')\narg_parser.add_argument('--ldflags', action='store', dest='user_ldflags', default='',\n                        help='Extra flags for the linker')\narg_parser.add_argument('--optflags', action='store', dest='user_optflags', default='',\n                        help='Extra optimization flags for the release mode')\narg_parser.add_argument('--api-level', action='store', dest='api_level', default='9',\n                        help='Compatibility API level (9=latest)')\narg_parser.add_argument('--compiler', action='store', dest='cxx', default='g++',\n                        help='C++ compiler path')\narg_parser.add_argument('--c-compiler', action='store', dest='cc', default='gcc',\n                        help='C compiler path (for bundled libraries such as dpdk)')\narg_parser.add_argument('--compiler-cache', dest='compiler_cache', default='auto',\n                        help=\"Use a compiler cache: 'auto' (prefer sccache over ccache), 'sccache', 'ccache', 'none' to disable, or a path to a compiler cache binary\")\narg_parser.add_argument('--c++-standard', action='store', dest='cpp_standard', default='',\n                        help='C++ standard to build with')\narg_parser.add_argument('--cook', action='append', dest='cook', default=[],\n                        help='Supply this dependency locally for development via `cmake-cooking` (can be repeated)')\narg_parser.add_argument('--verbose', dest='verbose', action='store_true', help='Make configure output more verbose.')\narg_parser.add_argument('--scheduling-groups-count', action='store', dest='scheduling_groups_count', default='16',\n                        help='Number of available scheduling groups in the reactor')\n\nadd_tristate(\n    arg_parser,\n    name='dpdk',\n    dest='dpdk',\n    help='DPDK support')\nadd_tristate(\n    arg_parser,\n    name='cxx-modules',\n    dest='cxx_modules',\n    help='build as C++20 module')\nadd_tristate(\n    arg_parser,\n    name='hwloc',\n    dest='hwloc',\n    help='hwloc support')\nadd_tristate(\n    arg_parser,\n    name='alloc-failure-injector',\n    dest='alloc_failure_injection',\n    help='allocation failure injection')\nadd_tristate(\n    arg_parser,\n    name='task-backtrace',\n    dest='task_backtrace',\n    help='Collect backtrace at deferring points')\nadd_tristate(\n    arg_parser,\n    name='unused-result-error',\n    dest=\"unused_result_error\",\n    help='Make [[nodiscard]] violations an error')\nadd_tristate(\n    arg_parser,\n    name='debug-shared-ptr',\n    dest=\"debug_shared_ptr\",\n    help='Debug shared_ptr')\nadd_tristate(\n    arg_parser,\n    name='io_uring',\n    dest='io_uring',\n    help='Support io_uring via liburing')\narg_parser.add_argument('--allocator-page-size', dest='alloc_page_size', type=int, help='override allocator page size')\narg_parser.add_argument('--without-tests', dest='exclude_tests', action='store_true', help='Do not build tests by default')\narg_parser.add_argument('--without-apps', dest='exclude_apps', action='store_true', help='Do not build applications by default')\narg_parser.add_argument('--without-demos', dest='exclude_demos', action='store_true', help='Do not build demonstrations by default')\narg_parser.add_argument('--split-dwarf', dest='split_dwarf', action='store_true', default=False,\n                        help='use of split dwarf (https://gcc.gnu.org/wiki/DebugFission) to speed up linking')\narg_parser.add_argument('--compile-commands-json', dest='cc_json', action='store_true',\n                        help='Generate a compile_commands.json file for integration with clangd and other tools.')\narg_parser.add_argument('--heap-profiling', dest='heap_profiling', action='store_true', default=False, help='Enable heap profiling')\narg_parser.add_argument('--dpdk-machine', default='native', help='Specify the target architecture')\nadd_tristate(arg_parser, name='deferred-action-require-noexcept', dest='deferred_action_require_noexcept', help='noexcept requirement for deferred actions', default=True)\narg_parser.add_argument('--prefix', dest='install_prefix', default='/usr/local', help='Root installation path of Seastar files')\nargs = arg_parser.parse_args()\n\n\ndef identify_best_standard(cpp_standards, compiler):\n    \"\"\"Returns the first C++ standard accepted by the compiler in the sequence,\n    assuming the \"best\" standards appear first.\n\n    If no standards are accepted, we fail configure.py. There is not point\n    of letting the user attempt to build with a standard that is known not\n    to be supported.\n    \"\"\"\n    for std in cpp_standards:\n        if standard_supported('c++{}'.format(std), compiler):\n            return std\n    raise Exception(f\"{compiler} does not seem to support any of Seastar's preferred C++ standards - {cpp_standards}. Please upgrade your compiler.\")\n\n\nif not args.cpp_standard:\n    cpp_standards = ['23', '20']\n    args.cpp_standard = identify_best_standard(cpp_standards, compiler=args.cxx)\n\n# Resolve compiler cache\ncompiler_cache = find_compiler_cache(args.compiler_cache)\nresolve_compilers_for_compiler_cache(args, compiler_cache)\n\n\nMODES = seastar_cmake.SUPPORTED_MODES if args.mode == 'all' else [args.mode]\n\n# For convenience.\ntr = seastar_cmake.translate_arg\n\nMODE_TO_CMAKE_BUILD_TYPE = {'release': 'RelWithDebInfo', 'debug': 'Debug', 'dev': 'Dev', 'sanitize': 'Sanitize', 'fuzz': 'Fuzz'}\n\n\ndef get_valid_ingredients():\n    \"\"\"Extract valid ingredient names from cooking_recipe.cmake.\"\"\"\n    recipe_path = os.path.join(seastar_cmake.ROOT_PATH, 'cooking_recipe.cmake')\n    with open(recipe_path, 'r') as f:\n        content = f.read()\n    # Match cooking_ingredient(name or cooking_ingredient (name\n    matches = re.findall(r'cooking_ingredient\\s*\\(\\s*([\\w-]+)', content)\n    return set(matches)\n\n\ndef configure_mode(mode):\n    BUILD_PATH = seastar_cmake.build_path(mode, build_root=args.build_root)\n\n    CFLAGS = seastar_cmake.convert_strings_to_cmake_list(\n        args.user_cflags,\n        args.user_optflags if seastar_cmake.is_release_mode(mode) else '')\n\n    LDFLAGS = seastar_cmake.convert_strings_to_cmake_list(args.user_ldflags)\n\n    TRANSLATED_ARGS = [\n        '-DCMAKE_BUILD_TYPE={}'.format(MODE_TO_CMAKE_BUILD_TYPE[mode]),\n        '-DCMAKE_CXX_COMPILER={}'.format(args.cxx),\n        '-DCMAKE_CXX_STANDARD={}'.format(args.cpp_standard),\n        '-DCMAKE_CXX_COMPILER_LAUNCHER={}'.format(compiler_cache),\n        '-DCMAKE_INSTALL_PREFIX={}'.format(args.install_prefix),\n        '-DCMAKE_EXPORT_COMPILE_COMMANDS={}'.format('yes' if args.cc_json else 'no'),\n        '-DBUILD_SHARED_LIBS={}'.format('yes' if mode in ('debug', 'dev') else 'no'),\n        '-DSeastar_API_LEVEL={}'.format(args.api_level),\n        '-DSeastar_SCHEDULING_GROUPS_COUNT={}'.format(args.scheduling_groups_count),\n        tr(args.exclude_tests, 'EXCLUDE_TESTS_FROM_ALL'),\n        tr(args.exclude_apps, 'EXCLUDE_APPS_FROM_ALL'),\n        tr(args.exclude_demos, 'EXCLUDE_DEMOS_FROM_ALL'),\n        tr(CFLAGS, 'CXX_FLAGS'),\n        tr(LDFLAGS, 'LD_FLAGS'),\n        tr(args.cxx_modules, 'MODULE'),\n        tr(args.dpdk, 'DPDK'),\n        tr(args.dpdk_machine, 'DPDK_MACHINE'),\n        tr(args.hwloc, 'HWLOC', value_when_none='yes'),\n        tr(args.io_uring, 'IO_URING', value_when_none=None),\n        tr(args.alloc_failure_injection, 'ALLOC_FAILURE_INJECTION', value_when_none='DEFAULT'),\n        tr(args.task_backtrace, 'TASK_BACKTRACE'),\n        tr(args.alloc_page_size, 'ALLOC_PAGE_SIZE'),\n        tr(args.split_dwarf, 'SPLIT_DWARF'),\n        tr(args.heap_profiling, 'HEAP_PROFILING'),\n        tr(args.deferred_action_require_noexcept, 'DEFERRED_ACTION_REQUIRE_NOEXCEPT'),\n        tr(args.unused_result_error, 'UNUSED_RESULT_ERROR'),\n        tr(args.debug_shared_ptr, 'DEBUG_SHARED_PTR', value_when_none='default'),\n    ]\n\n    ingredients_to_cook = set(args.cook)\n\n    if ingredients_to_cook:\n        valid_ingredients = get_valid_ingredients()\n        invalid = ingredients_to_cook - valid_ingredients\n        if invalid:\n            print(f\"error: unknown ingredient(s): {', '.join(sorted(invalid))}\")\n            print(f\"valid ingredients: {', '.join(sorted(valid_ingredients))}\")\n            sys.exit(1)\n\n    if args.dpdk:\n        ingredients_to_cook.add('dpdk')\n\n    # Generate a new build by pointing to the source directory.\n    if ingredients_to_cook:\n        # the C compiler is only used when building ingredients.\n        TRANSLATED_ARGS.append(f'-DCMAKE_C_COMPILER={args.cc}')\n\n        # We need to use cmake-cooking for some dependencies.\n        inclusion_arguments = []\n\n        for ingredient in ingredients_to_cook:\n            inclusion_arguments.extend(['-i', ingredient])\n\n        ARGS = seastar_cmake.COOKING_BASIC_ARGS + inclusion_arguments\n        if args.user_cflags:\n            ARGS += ['-s', f'CXXFLAGS={args.user_cflags}']\n        if args.user_ldflags:\n            ARGS += ['-s', f'LDFLAGS={args.user_ldflags}']\n        ARGS += ['-d', BUILD_PATH, '--']\n        dir = seastar_cmake.ROOT_PATH\n    else:\n        # When building without cooked dependencies, we can invoke cmake directly. We can't call\n        # cooking.sh, because without any -i parameters, it will try to build\n        # everything.\n        root_relative_to_build = os.path.relpath(seastar_cmake.ROOT_PATH, BUILD_PATH)\n        ARGS = ['cmake', '-G', 'Ninja', root_relative_to_build]\n        dir = BUILD_PATH\n    # filter out empty args, their values are actually \"guess\",\n    # CMake should be able to figure it out.\n    ARGS += filter(lambda arg: arg, TRANSLATED_ARGS)\n    if args.verbose:\n        print(\"Running CMake in '{}' ...\".format(dir))\n        print(\" \\\\\\n  \".join(ARGS))\n    os.makedirs(BUILD_PATH, exist_ok=True)\n    subprocess.check_call(ARGS, shell=False, cwd=dir)\n\n\nfor mode in MODES:\n    configure_mode(mode)\n"
  },
  {
    "path": "cooking.sh",
    "content": "#!/bin/bash\n\n#\n# Copyright 2018 Jesse Haber-Kucharsky\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\n#\n# This is cmake-cooking v0.10.0\n# The home of cmake-cooking is https://github.com/hakuch/CMakeCooking\n#\n\nset -e\n\nCMAKE=${CMAKE:-cmake}\n\ninvoked_args=(\"$@\")\nsource_dir=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\ninitial_wd=$(pwd)\nmemory_file=\"${initial_wd}/.cooking_memory\"\n\nrecipe=\"${source_dir}/cooking_recipe.cmake\"\ndeclare -a excluded_ingredients\ndeclare -a included_ingredients\nbuild_dir=\"${initial_wd}/build\"\nbuild_type=\"Debug\"\n# Depends on `build_dir`.\ningredients_dir=\"\"\ngenerator=\"Ninja\"\nlist_only=\"\"\nnested=\"\"\n\nusage() {\n    cat <<EOF\n\nFetch, configure, build, and install dependencies (\"ingredients\") for a CMake project\nin a local and repeatable development environment.\n\nUsage: $0 [OPTIONS]\n\nwhere OPTIONS are:\n\n-a\n-r RECIPE\n-e INGREDIENT\n-i INGREDIENT\n-d BUILD_DIR (=${build_dir})\n-p INGREDIENTS_DIR (=${build_dir}/_cooking/installed)\n-t BUILD_TYPE (=${build_type})\n-g GENERATOR (=${generator})\n-s VAR=VALUE\n-f EXPORT_DIR\n-l\n-h\n\nBy default, cmake-cooking reads a file called 'cooking_recipe.cmake'.\n\nIf neither [-i] nor [-e] are specified with a recipe ([-r]), then all ingredients of the recipe\nwill be fetched and built.\n\n[-i] and [-e] are mutually-exclusive options: only provide one.\n\nOption details:\n\n-a\n\n    Invoke 'cooking.sh' with the arguments that were provided to it last time, instead\n    of the arguments provided.\n\n-r RECIPE\n\n    Instead of reading the recipe in a file called 'cooking_recipe.cmake', follow the recipe\n    in the named file.\n\n    If the recipe file is a relative path, it is interpretted relative to the source directory\n    of the project.\n\n-e INGREDIENT\n\n    Exclude an ingredient from a recipe. This option can be supplied many times.\n\n    For example, if a recipe consists of 'apple', 'banana', 'carrot', and 'donut', then\n\n        ./cooking.sh -r dev -e apple -e donut\n\n    will prepare 'banana' and 'carrot' but not prepare 'apple' and 'donut'.\n\n    If an ingredient is excluded, then it is assumed that all ingredients that depend on it\n    can satisfy that dependency in some other way from the system (ie, the dependency is\n    removed internally).\n\n-i INGREDIENT\n\n   Include an ingredient from a recipe, ignoring the others. This option can be supplied\n   many times.\n\n   Similar to [-e], but the opposite.\n\n   For example, if a recipe consists of 'apple', 'banana', 'carrot', and 'donut' then\n\n       ./cooking.sh -r dev -i apple -i donut\n\n   will prepare 'apple' and 'donut' but not prepare 'banana' and 'carrot'.\n\n   If an ingredient is not in the \"include-list\", then it is assumed that all\n   ingredients that are in the list and which depend on it can satisfy that dependency\n   in some other way from the system.\n\n-d BUILD_DIR (=${build_dir})\n\n   Configure the project and build it in the named directory.\n\n-p INGREDIENTS_DIR (=${build_dir}/_cooking/installed)\n\n   Install compiled ingredients into this directory.\n\n-t BUILD_TYPE (=${build_type})\n\n   Configure all ingredients and the project with the named CMake build-type.\n   An example build type is \"Release\".\n\n-g GENERATOR (=${generator})\n\n    Use the named CMake generator for building all ingredients and the project.\n    An example generator is \"Unix Makfiles\".\n\n-s VAR=VALUE\n\n   Set an environmental variable 'VAR' to the value 'VALUE' during the invocation of CMake.\n\n-f EXPORT_DIR\n\n   If provided, and the project is successfully configured, then the tree of installed ingredients\n   is exported to this directory (the actual files: not symbolic links).\n\n   This option requires rsync.\n\n   This may be useful for preparing continuous integration environments, but it is not\n   recommended for distribution or release purposes (since this would be counter\n   to the goal of cmake-cooking).\n\n-l\n\n    Only list available ingredients for a given recipe, and don't do anything else.\n\n-h\n\n    Show this help information and exit.\n\nEOF\n}\n\nparse_assignment() {\n    local var\n    local value\n    IFS='=' read -r var value <<< \"${1}\"\n    export \"${var}\"=\"${value}\"\n}\n\nyell_include_exclude_mutually_exclusive() {\n    echo \"Cooking: [-e] and [-i] are mutually exclusive options!\" >&2\n}\n\nwhile getopts \"ar:e:i:d:p:t:g:s:f:lhx\" arg; do\n    case \"${arg}\" in\n        a)\n            if [ ! -f \"${memory_file}\" ]; then\n                echo \"No previous invocation found to recall!\" >&2\n                exit 1\n            fi\n\n            source \"${memory_file}\"\n            run_previous && exit 0\n            ;;\n        r)\n            if [[ \"${OPTARG}\" = /* ]]; then\n                recipe=${OPTARG}\n            else\n                recipe=\"${source_dir}/${OPTARG}\"\n            fi\n            ;;\n        e)\n            if [[ ${#included_ingredients[@]} -ne 0 ]]; then\n                yell_include_exclude_mutually_exclusive\n                exit 1\n            fi\n\n            excluded_ingredients+=(${OPTARG})\n            ;;\n        i)\n            if [[ ${#excluded_ingredients[@]} -ne 0 ]]; then\n                yell_include_exclude_mutually_exclusive\n                exit 1\n            fi\n\n            included_ingredients+=(${OPTARG})\n            ;;\n        d) build_dir=$(realpath \"${OPTARG}\") ;;\n        p) ingredients_dir=$(realpath \"${OPTARG}\") ;;\n        t) build_type=${OPTARG} ;;\n        g) generator=${OPTARG} ;;\n        s) parse_assignment \"${OPTARG}\" ;;\n        f) export_dir=$(realpath \"${OPTARG}\") ;;\n        l) list_only=\"1\" ;;\n        h) usage; exit 0 ;;\n        x) nested=\"1\" ;;\n        *) usage; exit 1 ;;\n    esac\ndone\n\nshift $((OPTIND - 1))\n\ncooking_dir=\"${build_dir}/_cooking\"\ncache_file=\"${build_dir}/CMakeCache.txt\"\ningredients_ready_file=\"${cooking_dir}/ready.txt\"\n\nif [ -z \"${ingredients_dir}\" ]; then\n    ingredients_dir=\"${cooking_dir}/installed\"\nfi\n\nmkdir -p \"${build_dir}\"\n\ncat <<'EOF' > \"${build_dir}/Cooking.cmake\"\n#\n# Copyright 2018 Jesse Haber-Kucharsky\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\n#\n# This file was generated by cmake-cooking v0.10.0\n# The home of cmake-cooking is https://github.com/hakuch/CMakeCooking\n#\n\nmacro (project name)\n  set (_cooking_dir ${CMAKE_CURRENT_BINARY_DIR}/_cooking)\n\n  if (CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_SOURCE_DIR)\n    set (_cooking_root ON)\n  else ()\n    set (_cooking_root OFF)\n  endif ()\n\n  find_program (Cooking_STOW_EXECUTABLE\n    stow\n    \"Executable path of GNU Stow.\")\n\n  if (NOT Cooking_STOW_EXECUTABLE)\n    message (FATAL_ERROR \"Cooking: GNU Stow is required!\")\n  endif ()\n\n  set (Cooking_INGREDIENTS_DIR\n    ${_cooking_dir}/installed\n    CACHE\n    PATH\n    \"Directory where ingredients will be installed.\")\n\n  set (Cooking_EXCLUDED_INGREDIENTS\n   \"\"\n   CACHE\n   STRING\n   \"Semicolon-separated list of ingredients that are not provided by Cooking.\")\n\n  set (Cooking_INCLUDED_INGREDIENTS\n   \"\"\n   CACHE\n   STRING\n   \"Semicolon-separated list of ingredients that are provided by Cooking.\")\n\n  option (Cooking_LIST_ONLY\n    \"Available ingredients will be listed and nothing will be installed.\"\n    OFF)\n\n  set (Cooking_RECIPE \"\" CACHE STRING \"Configure ${name}'s dependencies according to the named recipe.\")\n\n  if ((NOT DEFINED Cooking_EXCLUDED_INGREDIENTS) OR (Cooking_EXCLUDED_INGREDIENTS STREQUAL \"\"))\n    set (_cooking_is_excluding OFF)\n  else ()\n    set (_cooking_is_excluding ON)\n  endif ()\n\n  if ((NOT DEFINED Cooking_INCLUDED_INGREDIENTS) OR (Cooking_INCLUDED_INGREDIENTS STREQUAL \"\"))\n    set (_cooking_is_including OFF)\n  else ()\n    set (_cooking_is_including ON)\n  endif ()\n\n  if (_cooking_is_excluding AND _cooking_is_including)\n    message (\n      FATAL_ERROR\n      \"Cooking: The EXCLUDED_INGREDIENTS and INCLUDED_INGREDIENTS lists are mutually exclusive options!\")\n  endif ()\n\n  if (_cooking_root)\n    _project (${name} ${ARGN})\n\n    if (NOT (\"${Cooking_RECIPE}\" STREQUAL \"\"))\n      add_custom_target (_cooking_ingredients)\n\n      set (_cooking_ready_marker_file ${_cooking_dir}/ready.txt)\n\n      add_custom_command (\n        OUTPUT ${_cooking_ready_marker_file}\n        DEPENDS _cooking_ingredients\n        COMMAND ${CMAKE_COMMAND} -E touch ${_cooking_ready_marker_file})\n\n      add_custom_target (_cooking_ingredients_ready\n        DEPENDS ${_cooking_ready_marker_file})\n\n      set (_cooking_local_synchronize_marker_file ${Cooking_INGREDIENTS_DIR}/.cooking_local_synchronize)\n\n      add_custom_command (\n        OUTPUT ${_cooking_local_synchronize_marker_file}\n        COMMAND ${CMAKE_COMMAND} -E touch ${_cooking_local_synchronize_marker_file})\n\n      add_custom_target (_cooking_marked_for_local_synchronization\n        DEPENDS ${_cooking_local_synchronize_marker_file})\n\n      list (APPEND CMAKE_PREFIX_PATH ${Cooking_INGREDIENTS_DIR})\n      include (${Cooking_RECIPE})\n\n      if (NOT EXISTS ${_cooking_ready_marker_file})\n        return ()\n      endif ()\n    endif ()\n  endif ()\nendmacro ()\n\nfunction (_cooking_set_union x y var)\n  set (r ${${x}})\n\n  foreach (e ${${y}})\n    list (APPEND r ${e})\n  endforeach ()\n\n  list (REMOVE_DUPLICATES r)\n  set (${var} ${r} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_set_difference x y var)\n  set (r ${${x}})\n\n  foreach (e ${${y}})\n    if (${e} IN_LIST ${x})\n       list (REMOVE_ITEM r ${e})\n    endif ()\n  endforeach ()\n\n  set (${var} ${r} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_set_intersection x y var)\n  set (r \"\")\n\n  foreach (e ${${y}})\n    if (${e} IN_LIST ${x})\n      list (APPEND r ${e})\n    endif ()\n  endforeach ()\n\n  list (REMOVE_DUPLICATES r)\n  set (${var} ${r} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_query_by_key list key var)\n  list (FIND ${list} ${key} index)\n\n  if (${index} EQUAL \"-1\")\n    set (value NOTFOUND)\n  else ()\n    math (EXPR value_index \"${index} + 1\")\n    list (GET ${list} ${value_index} value)\n  endif ()\n\n  set (${var} ${value} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_populate_ep_parameter)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"EXTERNAL_PROJECT_ARGS_LIST;PARAMETER;DEFAULT_VALUE\"\n    \"\"\n    ${ARGN})\n\n  string (TOLOWER ${pa_PARAMETER} parameter_lower)\n  _cooking_query_by_key (${pa_EXTERNAL_PROJECT_ARGS_LIST} ${pa_PARAMETER} ${parameter_lower})\n  set (value ${${parameter_lower}})\n  set (var_name _cooking_${parameter_lower})\n  set (ep_var_name _cooking_ep_${parameter_lower})\n\n  if (NOT value)\n    set (${var_name} ${pa_DEFAULT_VALUE} PARENT_SCOPE)\n    set (${ep_var_name} ${pa_PARAMETER} ${pa_DEFAULT_VALUE} PARENT_SCOPE)\n  else ()\n    set (${var_name} ${value} PARENT_SCOPE)\n    set (${ep_var_name} \"\" PARENT_SCOPE)\n  endif ()\nendfunction ()\n\nfunction (_cooking_define_listing_targets)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"NAME;SOURCE_DIR;RECIPE\"\n    \"REQUIRES\"\n    ${ARGN})\n\n  set (stale_file ${Cooking_INGREDIENTS_DIR}/.cooking_stale_ingredient_${pa_NAME})\n\n  add_custom_command (\n    OUTPUT ${stale_file}\n    COMMAND ${CMAKE_COMMAND} -E touch ${stale_file})\n\n  add_custom_target (_cooking_ingredient_${pa_NAME}_stale\n    DEPENDS ${stale_file})\n\n  set (commands\n    COMMAND\n    ${CMAKE_COMMAND} -E touch ${Cooking_INGREDIENTS_DIR}/.cooking_ingredient_${pa_NAME})\n\n  if (pa_RECIPE)\n    if (pa_RECIPE STREQUAL <DEFAULT>)\n      set (recipe_args \"\")\n    else ()\n      set (recipe_args -r ${pa_RECIPE})\n    endif ()\n\n    list (INSERT commands 0\n      COMMAND\n      ${pa_SOURCE_DIR}/cooking.sh\n      ${recipe_args}\n      -p ${Cooking_INGREDIENTS_DIR}\n      -g ${CMAKE_GENERATOR}\n      -x\n      -l)\n  endif ()\n\n  add_custom_command (\n    OUTPUT ${Cooking_INGREDIENTS_DIR}/.cooking_ingredient_${pa_NAME}\n    DEPENDS\n      _cooking_ingredient_${pa_NAME}_stale\n      ${stale_file}\n    ${commands})\n\n  add_custom_target (_cooking_ingredient_${pa_NAME}_listed\n    DEPENDS ${Cooking_INGREDIENTS_DIR}/.cooking_ingredient_${pa_NAME})\n\n  foreach (d ${pa_REQUIRES})\n    add_dependencies (_cooking_ingredient_${pa_NAME}_listed _cooking_ingredient_${d}_listed)\n  endforeach ()\n\n  add_dependencies (_cooking_ingredients _cooking_ingredient_${pa_NAME}_listed)\nendfunction ()\n\nfunction (_cooking_adjust_requirements)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"IS_EXCLUDING;IS_INCLUDING;OUTPUT_LIST\"\n    \"REQUIREMENTS\"\n    ${ARGN})\n\n  if (pa_IS_EXCLUDING)\n    # Strip out any dependencies that are excluded.\n    _cooking_set_difference (\n      pa_REQUIREMENTS\n      Cooking_EXCLUDED_INGREDIENTS\n      pa_REQUIREMENTS)\n  elseif (_cooking_is_including)\n    # Eliminate dependencies that have not been included.\n    _cooking_set_intersection (\n      pa_REQUIREMENTS\n      Cooking_INCLUDED_INGREDIENTS\n      pa_REQUIREMENTS)\n  endif ()\n\n  set (${pa_OUTPUT_LIST} ${pa_REQUIREMENTS} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_populate_ep_depends)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"\"\n    \"REQUIREMENTS\"\n    ${ARGN})\n\n  if (pa_REQUIREMENTS)\n    set (value DEPENDS)\n\n    foreach (d ${pa_REQUIREMENTS})\n      list (APPEND value ingredient_${d})\n    endforeach ()\n  else ()\n    set (value \"\")\n  endif ()\n\n  set (_cooking_ep_depends ${value} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_prepare_restrictions_arguments)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"IS_EXCLUDING;IS_INCLUDING;OUTPUT_LIST\"\n    \"REQUIREMENTS\"\n    ${ARGN})\n\n  set (args \"\")\n\n  if (pa_IS_INCLUDING)\n    _cooking_set_difference (\n      Cooking_INCLUDED_INGREDIENTS\n      pa_REQUIREMENTS\n      included)\n\n    foreach (x ${included})\n      list (APPEND args -i ${x})\n    endforeach ()\n  elseif (pa_IS_EXCLUDING)\n    _cooking_set_union (\n      Cooking_EXCLUDED_INGREDIENTS\n      pa_REQUIREMENTS\n      excluded)\n\n    foreach (x ${excluded})\n      list (APPEND args -e ${x})\n    endforeach ()\n  else ()\n    foreach (x ${pa_REQUIREMENTS})\n      list (APPEND args -e ${x})\n    endforeach ()\n  endif ()\n\n  set (${pa_OUTPUT_LIST} ${args} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_determine_common_cmake_args output)\n  string (REPLACE \";\" \":::\" prefix_path_with_colons \"${CMAKE_PREFIX_PATH}\")\n\n  set (cmake_args \"-G\" \"${CMAKE_GENERATOR}\")\n\n  if (CMAKE_CXX_FLAGS)\n    list(APPEND cmake_args -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS})\n  endif ()\n  if (CMAKE_C_FLAGS)\n    list(APPEND cmake_args -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS})\n  endif ()\n\n  list (APPEND cmake_args\n    -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>\n    -DCMAKE_PREFIX_PATH=${prefix_path_with_colons}\n    -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}\n    -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}\n    -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}\n    -DBUILD_SHARED_LIBS=${BUILD_SHARED_LIBS})\n\n  set (${output} ${cmake_args}\n    PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_populate_ep_configure_command)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"IS_EXCLUDING;IS_INCLUDING;RECIPE;EXTERNAL_PROJECT_ARGS_LIST\"\n    \"REQUIREMENTS;CMAKE_ARGS;COOKING_CMAKE_ARGS\"\n    ${ARGN})\n\n  if (pa_RECIPE)\n    if (pa_RECIPE STREQUAL <DEFAULT>)\n      set (recipe_args \"\")\n    else ()\n      set (recipe_args -r ${pa_RECIPE})\n    endif ()\n\n    _cooking_prepare_restrictions_arguments (\n      IS_EXCLUDING ${pa_IS_EXCLUDING}\n      IS_INCLUDING ${pa_IS_INCLUDING}\n      REQUIREMENTS ${pa_REQUIREMENTS}\n      OUTPUT_LIST restrictions_args)\n\n    set (value\n      CONFIGURE_COMMAND\n      <SOURCE_DIR>/cooking.sh\n      ${recipe_args}\n      -d <BINARY_DIR>\n      -p ${Cooking_INGREDIENTS_DIR}\n      -g ${CMAKE_GENERATOR}\n      -x\n      ${restrictions_args}\n      --\n      ${pa_COOKING_CMAKE_ARGS})\n  elseif (NOT (CONFIGURE_COMMAND IN_LIST ${pa_EXTERNAL_PROJECT_ARGS_LIST}))\n    set (value\n      CONFIGURE_COMMAND\n      ${CMAKE_COMMAND}\n      ${pa_CMAKE_ARGS}\n      <SOURCE_DIR>)\n  else ()\n    set (value \"\")\n  endif ()\n\n  set (_cooking_ep_configure_command ${value} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_populate_ep_build_command ep_args_list)\n  if (NOT (BUILD_COMMAND IN_LIST ${ep_args_list}))\n    set (value BUILD_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR>)\n  else ()\n    set (value \"\")\n  endif ()\n\n  set (_cooking_ep_build_command ${value} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_populate_ep_install_command ep_args_list)\n  if (NOT (INSTALL_COMMAND IN_LIST ${ep_args_list}))\n    set (value INSTALL_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target install)\n  else ()\n    set (value \"\")\n  endif ()\n\n  set (_cooking_ep_install_command ${value} PARENT_SCOPE)\nendfunction ()\n\nfunction (_cooking_define_ep)\n  cmake_parse_arguments (\n    pa\n    \"\"\n    \"NAME;SOURCE_DIR;BINARY_DIR;EXTERNAL_PROJECT_ARGS_LIST;RECIPE;INGREDIENT_DIR;STOW_DIR;LOCAL_RECONFIGURE;LOCAL_REBUILD\"\n    \"DEPENDS;CONFIGURE_COMMAND;BUILD_COMMAND;INSTALL_COMMAND;CMAKE_ARGS\"\n    ${ARGN})\n\n  string (REPLACE \"<DISABLE>\" \"\" forwarded_ep_args \"${${pa_EXTERNAL_PROJECT_ARGS_LIST}}\")\n  set (ep_name ingredient_${pa_NAME})\n  include (ExternalProject)\n\n  set (stamp_dir ${pa_INGREDIENT_DIR}/stamp)\n\n  ExternalProject_add (${ep_name}\n    DEPENDS ${pa_DEPENDS}\n    SOURCE_DIR ${pa_SOURCE_DIR}\n    BINARY_DIR ${pa_BINARY_DIR}\n    CONFIGURE_COMMAND ${pa_CONFIGURE_COMMAND}\n    BUILD_COMMAND ${pa_BUILD_COMMAND}\n    INSTALL_COMMAND ${pa_INSTALL_COMMAND}\n    PREFIX ${pa_INGREDIENT_DIR}\n    STAMP_DIR ${stamp_dir}\n    INSTALL_DIR ${pa_STOW_DIR}/${pa_NAME}\n    CMAKE_ARGS ${pa_CMAKE_ARGS}\n    LIST_SEPARATOR :::\n    STEP_TARGETS install\n    \"${forwarded_ep_args}\")\n\n  set (stow_marker_file ${Cooking_INGREDIENTS_DIR}/.cooking_ingredient_${pa_NAME})\n  set (lock_file ${Cooking_INGREDIENTS_DIR}/.cooking_stow.lock)\n\n  add_custom_command (\n    OUTPUT ${stow_marker_file}\n    DEPENDS\n      ${ep_name}-install\n      ${stamp_dir}/ingredient_${pa_NAME}-install\n    COMMAND\n      flock\n      --wait 30\n      ${lock_file}\n      ${Cooking_STOW_EXECUTABLE}\n      -t ${Cooking_INGREDIENTS_DIR}\n      -d ${pa_STOW_DIR}\n      ${pa_NAME}\n    COMMAND ${CMAKE_COMMAND} -E touch ${stow_marker_file})\n\n  add_custom_target (_cooking_ingredient_${pa_NAME}_stowed\n    DEPENDS ${stow_marker_file})\n\n  if (pa_RECIPE)\n    set (reconfigure_marker_file ${Cooking_INGREDIENTS_DIR}/.cooking_reconfigure_ingredient_${pa_NAME})\n\n    add_custom_command (\n      OUTPUT ${reconfigure_marker_file}\n      COMMAND ${CMAKE_COMMAND} -E touch ${reconfigure_marker_file})\n\n    add_custom_target (_cooking_ingredient_${pa_NAME}_marked_for_reconfigure\n      DEPENDS ${reconfigure_marker_file})\n\n    ExternalProject_add_step (${ep_name}\n      cooking-reconfigure\n      DEPENDERS configure\n      DEPENDS ${reconfigure_marker_file}\n      COMMAND ${CMAKE_COMMAND} -E echo_append)\n\n    ExternalProject_add_stepdependencies (${ep_name}\n      cooking-reconfigure\n      _cooking_ingredient_${pa_NAME}_marked_for_reconfigure)\n  endif ()\n\n  foreach (d ${pa_DEPENDS})\n    ExternalProject_add_stepdependencies (${ep_name}\n      configure\n      _cooking_${d}_stowed)\n  endforeach ()\n\n  add_dependencies (_cooking_ingredients _cooking_ingredient_${pa_NAME}_stowed)\n\n  if (pa_LOCAL_RECONFIGURE OR pa_LOCAL_REBUILD)\n    if (pa_LOCAL_RECONFIGURE)\n      set (step configure)\n    else ()\n      set (step build)\n    endif ()\n\n    ExternalProject_add_step (${ep_name}\n      cooking-local-${step}\n      DEPENDERS ${step}\n      DEPENDS ${_cooking_local_synchronize_marker_file}\n      COMMAND ${CMAKE_COMMAND} -E echo_append)\n\n    ExternalProject_add_stepdependencies (${ep_name}\n      cooking-local-${step}\n      _cooking_marked_for_local_synchronization)\n  endif ()\nendfunction ()\n\nmacro (cooking_ingredient name)\n  set (_cooking_args \"${ARGN}\")\n\n  if ((_cooking_is_excluding AND (${name} IN_LIST Cooking_EXCLUDED_INGREDIENTS))\n      OR (_cooking_is_including AND (NOT (${name} IN_LIST Cooking_INCLUDED_INGREDIENTS))))\n    # Nothing.\n  else ()\n    set (_cooking_ingredient_dir ${_cooking_dir}/ingredient/${name})\n\n    cmake_parse_arguments (\n      _cooking_pa\n      \"LOCAL_RECONFIGURE;LOCAL_REBUILD\"\n      \"COOKING_RECIPE\"\n      \"CMAKE_ARGS;COOKING_CMAKE_ARGS;EXTERNAL_PROJECT_ARGS;REQUIRES\"\n      ${_cooking_args})\n\n    _cooking_populate_ep_parameter (\n      EXTERNAL_PROJECT_ARGS_LIST _cooking_pa_EXTERNAL_PROJECT_ARGS\n      PARAMETER SOURCE_DIR\n      DEFAULT_VALUE ${_cooking_ingredient_dir}/src)\n\n    _cooking_populate_ep_parameter (\n      EXTERNAL_PROJECT_ARGS_LIST _cooking_pa_EXTERNAL_PROJECT_ARGS\n      PARAMETER BINARY_DIR\n      DEFAULT_VALUE ${_cooking_ingredient_dir}/build)\n\n    _cooking_populate_ep_parameter (\n      EXTERNAL_PROJECT_ARGS_LIST _cooking_pa_EXTERNAL_PROJECT_ARGS\n      PARAMETER BUILD_IN_SOURCE\n      DEFAULT_VALUE OFF)\n\n    if (_cooking_build_in_source)\n       set (_cooking_ep_binary_dir \"\")\n    endif ()\n\n    if (Cooking_LIST_ONLY)\n      _cooking_define_listing_targets (\n        NAME ${name}\n        SOURCE_DIR ${_cooking_source_dir}\n        RECIPE ${_cooking_pa_COOKING_RECIPE}\n        REQUIRES ${_cooking_pa_REQUIRES})\n    else ()\n      _cooking_adjust_requirements (\n        IS_EXCLUDING ${_cooking_is_excluding}\n        IS_INCLUDING ${_cooking_is_including}\n        REQUIREMENTS ${_cooking_pa_REQUIRES}\n        OUTPUT_LIST _cooking_pa_REQUIRES)\n\n      _cooking_populate_ep_depends (\n        REQUIREMENTS ${_cooking_pa_REQUIRES})\n\n      _cooking_determine_common_cmake_args (_cooking_common_cmake_args)\n\n      _cooking_populate_ep_configure_command (\n        IS_EXCLUDING ${_cooking_is_excluding}\n        IS_INCLUDING ${_cooking_is_including}\n        RECIPE ${_cooking_pa_COOKING_RECIPE}\n        REQUIREMENTS ${_cooking_pa_REQUIRES}\n        EXTERNAL_PROJECT_ARGS_LIST _cooking_pa_EXTERNAL_PROJECT_ARGS\n        CMAKE_ARGS\n          ${_cooking_common_cmake_args}\n          ${_cooking_pa_CMAKE_ARGS}\n        COOKING_CMAKE_ARGS\n          ${_cooking_common_cmake_args}\n          ${_cooking_pa_COOKING_CMAKE_ARGS})\n\n      _cooking_populate_ep_build_command (_cooking_pa_EXTERNAL_PROJECT_ARGS)\n      _cooking_populate_ep_install_command (_cooking_pa_EXTERNAL_PROJECT_ARGS)\n\n      _cooking_define_ep (\n        NAME ${name}\n        RECIPE ${_cooking_pa_COOKING_RECIPE}\n        DEPENDS ${_cooking_ep_depends}\n        SOURCE_DIR ${_cooking_ep_source_dir}\n        BINARY_DIR ${_cooking_ep_binary_dir}\n        CONFIGURE_COMMAND ${_cooking_ep_configure_command}\n        BUILD_COMMAND ${_cooking_ep_build_command}\n        INSTALL_COMMAND ${_cooking_ep_install_command}\n        INGREDIENT_DIR ${_cooking_ingredient_dir}\n        STOW_DIR ${_cooking_dir}/stow\n        CMAKE_ARGS ${_cooking_common_cmake_args}\n        EXTERNAL_PROJECT_ARGS_LIST _cooking_pa_EXTERNAL_PROJECT_ARGS\n        LOCAL_RECONFIGURE ${_cooking_pa_LOCAL_RECONFIGURE}\n        LOCAL_REBUILD ${_cooking_pa_LOCAL_REBUILD})\n    endif ()\n  endif ()\nendmacro ()\nEOF\n\ncmake_cooking_args=(\n    \"-DCooking_INGREDIENTS_DIR=${ingredients_dir}\"\n    \"-DCooking_RECIPE=${recipe}\"\n)\n\n#\n# Remove any `Cooking.cmake` file from the source directory. We now generate this file in the build directory, and old\n# copies will cause conflicts.\n#\n\nold_cooking_file=\"${source_dir}/cmake/Cooking.cmake\"\n\nif [ -f \"${old_cooking_file}\" ]; then\n    grep 'This file was generated by cmake-cooking' \"${old_cooking_file}\" > /dev/null && rm \"${old_cooking_file}\"\nfi\n\n#\n# Clean-up from a previous run.\n#\n\nif [ -e \"${ingredients_ready_file}\" ]; then\n    rm \"${ingredients_ready_file}\"\nfi\n\nif [ -e \"${cache_file}\" ]; then\n    rm \"${cache_file}\"\nfi\n\nif [ -d \"${ingredients_dir}\" -a -z \"${nested}\" ]; then\n    rm -r --preserve-root \"${ingredients_dir}\"\nfi\n\nmkdir -p \"${ingredients_dir}\"\n\n#\n# Validate recipe.\n#\n\nif [ -n \"${recipe}\" ]; then\n    if [ ! -f \"${recipe}\" ]; then\n        echo \"Cooking: The '${recipe}' recipe does not exist!\" >&2\n        exit 1\n    fi\nfi\n\n#\n# Prepare lists of included and excluded ingredients.\n#\n\nif [ -n \"${excluded_ingredients}\" ] && [ -z \"${list_only}\" ]; then\n    cmake_cooking_args+=(\n        -DCooking_EXCLUDED_INGREDIENTS=$(printf \"%s;\" \"${excluded_ingredients[@]}\")\n        -DCooking_INCLUDED_INGREDIENTS=\n    )\nfi\n\nif [ -n \"${included_ingredients}\" ] && [ -z \"${list_only}\" ]; then\n    cmake_cooking_args+=(\n        -DCooking_EXCLUDED_INGREDIENTS=\n        -DCooking_INCLUDED_INGREDIENTS=$(printf \"%s;\" \"${included_ingredients[@]}\")\n    )\nfi\n\n#\n# Configure and build ingredients.\n#\n\nmkdir -p \"${cooking_dir}\"/stow\ntouch \"${cooking_dir}\"/stow/.stow\ncd \"${build_dir}\"\n\ndeclare -a build_args\n\nif [ \"${generator}\" == \"Ninja\" ]; then\n    build_args+=(-v)\nfi\n\nif [ -n \"${list_only}\" ]; then\n    cmake_cooking_args+=(\"-DCooking_LIST_ONLY=ON\")\nfi\n\n${CMAKE} -DCMAKE_BUILD_TYPE=\"${build_type}\" \"${cmake_cooking_args[@]}\" -G \"${generator}\" \"${source_dir}\" \"${@}\"\n\nif [ -n \"${recipe}\" ]; then\n    ${CMAKE} --build . --target _cooking_ingredients_ready -- \"${build_args[@]}\"\n\n    #\n    # Report what we've done (if we're not nested).\n    #\n\n    if [ -z \"${nested}\" ]; then\n        ingredients=($(find \"${ingredients_dir}\" -name '.cooking_ingredient_*' -printf '%f\\n' | sed -r 's/\\.cooking_ingredient_(.+)/\\1/'))\n\n        if [ -z \"${list_only}\" ]; then\n            printf \"\\nCooking: Installed the following ingredients:\\n\"\n        else\n            printf \"\\nCooking: The following ingredients are necessary for this recipe:\\n\"\n        fi\n\n        for ingredient in \"${ingredients[@]}\"; do\n            echo \"  - ${ingredient}\"\n        done\n\n        printf '\\n'\n    fi\n\n    if [ -n \"${list_only}\" ]; then\n        exit 0\n    fi\n\n    #\n    # Configure the project, expecting all requirements satisfied.\n    #\n\n    ${CMAKE} -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON \"${@}\" .\n\n    #\n    # Optionally export the installed files.\n    #\n\n    if [ -n \"${export_dir}\" ]; then\n        rsync \"${ingredients_dir}/\" \"${export_dir}\" -a --copy-links\n        printf \"\\nCooking: Exported ingredients to ${export_dir}\\n\"\n    fi\nfi\n\n#\n# Save invocation information.\n#\n\ncd \"${initial_wd}\"\n\ncat <<EOF > \"${memory_file}\"\nrun_previous() {\n    \"${0}\" ${invoked_args[@]@Q}\n}\nEOF\n"
  },
  {
    "path": "cooking_recipe.cmake",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\n#\n# Useful definitions for `cmake -E env`.\n#\n\nset (amended_PATH PATH=${Cooking_INGREDIENTS_DIR}/bin:$ENV{PATH})\nset (PKG_CONFIG_PATH PKG_CONFIG_PATH=${Cooking_INGREDIENTS_DIR}/lib/pkgconfig)\n\n#\n# Some Autotools ingredients need this information because they don't use pkgconfig.\n#\n\nset (autotools_ingredients_flags\n  CFLAGS=-I${Cooking_INGREDIENTS_DIR}/include\n  CXXFLAGS=-I${Cooking_INGREDIENTS_DIR}/include\n  LDFLAGS=-L${Cooking_INGREDIENTS_DIR}/lib)\n\n#\n# Some Autotools projects amend the info file instead of making a package-specific one.\n# This doesn't play nicely with GNU Stow.\n#\n# Just append the name of the ingredient, like\n#\n#     ${info_dir}/gmp\n#\n\nset (info_dir --infodir=<INSTALL_DIR>/share/info)\n\n#\n# Build-concurrency.\n#\n\ncmake_host_system_information (\n  RESULT build_concurrency_factor\n  QUERY NUMBER_OF_LOGICAL_CORES)\n\nset (make_command make -j ${build_concurrency_factor})\n\n#\n# All the ingredients.\n#\n\n##\n## Dependencies of dependencies of dependencies.\n##\n\ncooking_ingredient (gmp\n  EXTERNAL_PROJECT_ARGS\n    URL https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2\n    URL_MD5 8ddbb26dc3bd4e2302984debba1406a5\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR> ${info_dir}/gmp\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\n##\n## Dependencies of dependencies.\n##\n\ncooking_ingredient (colm\n  EXTERNAL_PROJECT_ARGS\n    URL http://www.colm.net/files/colm/colm-0.13.0.6.tar.gz\n    URL_MD5 16aaf566cbcfe9a06154e094638ac709\n    # This is upsetting.\n    BUILD_IN_SOURCE YES\n    CONFIGURE_COMMAND ./configure --prefix=<INSTALL_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (libpciaccess\n  EXTERNAL_PROJECT_ARGS\n    URL https://www.x.org/releases/individual/lib/libpciaccess-0.13.4.tar.gz\n    URL_MD5 cc1fad87da60682af1d5fa43a5da45a4\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (nettle\n  REQUIRES gmp\n  EXTERNAL_PROJECT_ARGS\n    URL https://ftp.gnu.org/gnu/nettle/nettle-3.4.tar.gz\n    URL_MD5 dc0f13028264992f58e67b4e8915f53d\n    CONFIGURE_COMMAND\n      <SOURCE_DIR>/configure\n      --prefix=<INSTALL_DIR>\n      --srcdir=<SOURCE_DIR>\n      --libdir=<INSTALL_DIR>/lib\n      ${info_dir}/nettle\n      ${autotools_ingredients_flags}\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\n# A dependency of DPDK.\ncooking_ingredient (numactl\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/numactl/numactl/releases/download/v2.0.12/numactl-2.0.12.tar.gz\n    URL_MD5 2ba9777d78bfd7d408a387e53bc33ebc\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (zlib\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/madler/zlib/releases/download/v1.2.13/zlib-1.2.13.tar.gz\n    URL_MD5 9b8aa094c4e5765dabf4da391f00d15c\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\n##\n## Private and private/public dependencies.\n##\n\nif (CMAKE_CXX_COMPILER_ID STREQUAL \"GNU\")\n  set (boost_toolset gcc)\nelseif (CMAKE_CXX_COMPILER_ID STREQUAL \"Clang\")\n  set (boost_toolset clang)\nelse ()\n  set(boost_toolset \"cook_cxx\")\nendif ()\nset (boost_user_config \"${CMAKE_CURRENT_BINARY_DIR}/cook_boost.jam\")\nif (CMAKE_C_FLAGS)\n  string (JOIN \" <cflags>\" boost_cflags\n    \"<cflags>${CMAKE_C_FLAGS}\")\nendif ()\nif (CMAKE_CXX_FLAGS)\n  string (JOIN \" <cxxflags>\" boost_cxxflags\n    \"<cxxflags>${CMAKE_CXX_FLAGS}\")\nendif ()\nfile (WRITE \"${boost_user_config}\"\n  \"using ${boost_toolset}\"\n  \" : \" # toolset's version\n  \" : ${CMAKE_CXX_COMPILER}\"\n  \" : ${boost_cflags}${boost_cxxflags} <cxxflags>-std=c++${CMAKE_CXX_STANDARD}\"\n  \" ;\\n\")\n\ncooking_ingredient (Boost\n  EXTERNAL_PROJECT_ARGS\n    URL https://archives.boost.io/release/1.81.0/source/boost_1_81_0.tar.bz2\n    URL_HASH SHA256=71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa\n    PATCH_COMMAND\n      ./bootstrap.sh\n      --prefix=<INSTALL_DIR>\n      --with-libraries=atomic,chrono,date_time,filesystem,program_options,system,test,thread\n      --with-toolset=${boost_toolset}\n    CONFIGURE_COMMAND <DISABLE>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND\n      ${CMAKE_COMMAND} -E chdir <SOURCE_DIR>\n      ./b2\n      -j ${build_concurrency_factor}\n      --layout=system\n      --build-dir=<BINARY_DIR>\n      --user-config=${boost_user_config}\n      install\n      toolset=${boost_toolset}\n      variant=debug\n      link=shared\n      threading=multi\n      hardcode-dll-paths=true\n      dll-path=<INSTALL_DIR>/lib)\n\ncooking_ingredient (GnuTLS\n  REQUIRES\n    gmp\n    nettle\n  EXTERNAL_PROJECT_ARGS\n    URL https://www.gnupg.org/ftp/gcrypt/gnutls/v3.7/gnutls-3.7.11.tar.xz\n    URL_MD5 dd8c16b17f1d37fca203e756e981a957\n    CONFIGURE_COMMAND\n     ${CMAKE_COMMAND} -E env ${PKG_CONFIG_PATH}\n      <SOURCE_DIR>/configure\n      --prefix=<INSTALL_DIR>\n      --srcdir=<SOURCE_DIR>\n      --with-included-unistring\n      --with-included-libtasn1\n      --without-p11-kit\n      # https://lists.gnupg.org/pipermail/gnutls-help/2016-February/004085.html\n      --disable-non-suiteb-curves\n      --disable-doc\n      ${autotools_ingredients_flags}\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (Protobuf\n  REQUIRES zlib\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/protocolbuffers/protobuf/releases/download/v21.11//protobuf-cpp-3.21.11.tar.gz\n    URL_MD5 e2cf711edae444bba0da199bc034e031\n  CMAKE_ARGS\n    -Dprotobuf_BUILD_TESTS=OFF)\n\ncooking_ingredient (hwloc\n  REQUIRES\n    numactl\n    libpciaccess\n  EXTERNAL_PROJECT_ARGS\n    URL https://download.open-mpi.org/release/hwloc/v2.2/hwloc-2.2.0.tar.gz\n    URL_MD5 762c93cdca3249eed4627c4a160192bd\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (ragel\n  REQUIRES colm\n  EXTERNAL_PROJECT_ARGS\n    URL http://www.colm.net/files/ragel/ragel-6.10.tar.gz\n    URL_MD5 748cae8b50cffe9efcaa5acebc6abf0d\n    PATCH_COMMAND\n      sed -i \"s/ CHAR_M/ SCHAR_M/g\" ragel/common.cpp\n    # This is upsetting.\n    BUILD_IN_SOURCE YES\n    CONFIGURE_COMMAND\n      ${CMAKE_COMMAND} -E env ${amended_PATH}\n      ./configure\n      --prefix=<INSTALL_DIR>\n      # This is even more upsetting.\n      ${autotools_ingredients_flags}\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (lksctp-tools\n  EXTERNAL_PROJECT_ARGS\n    URL https://sourceforge.net/projects/lksctp/files/lksctp-tools/lksctp-tools-1.0.16.tar.gz\n    URL_MD5 708bb0b5a6806ad6e8d13c55b067518e\n    PATCH_COMMAND ./bootstrap\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\ncooking_ingredient (yaml-cpp\n  CMAKE_ARGS\n    -DYAML_CPP_BUILD_TESTS=OFF\n    -DYAML_BUILD_SHARED_LIBS=ON\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.7.0.tar.gz\n    URL_MD5 74d646a3cc1b5d519829441db96744f0)\n\n##\n## Public dependencies.\n##\n\ncooking_ingredient (c-ares\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/c-ares/c-ares/releases/download/v1.32.3/c-ares-1.32.3.tar.gz\n    URL_MD5 d5ed5967bc3a74191c051ce81ffe02fc\n    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --srcdir=<SOURCE_DIR>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} install)\n\nset (dpdk_args\n  --default-library=static\n  -Dc_args=\"-Wno-error\"\n  -Denable_docs=false\n  -Denable_apps=dpdk-testpmd\n  -Dtests=false\n  -Dexamples=\n  -Dmbuf_refcnt_atomic=false\n  -Dmax_memseg_lists=8192\n  -Ddisable_drivers=\"net/softnic,net/bonding\"\n  -Ddisable_libs=\"jobstats,power,port,table,pipeline,member\"\n  -Dcpu_instruction_set=${Seastar_DPDK_MACHINE})\n\nif (CMAKE_BUILD_TYPE STREQUAL Debug)\n  list (APPEND dpdk_args -Dbuildtype=debug)\nendif ()\n\nfind_program (Meson_EXECUTABLE\n  meson)\nif (NOT Meson_EXECUTABLE)\n  message (FATAL_ERROR \"Cooking: Meson is required!\")\nendif ()\n\nfind_program (Ninja_EXECUTABLE\n  ninja)\nif (NOT Ninja_EXECUTABLE)\n  message (FATAL_ERROR \"Cooking: Ninja is required!\")\nendif ()\n\ncooking_ingredient (dpdk\n  EXTERNAL_PROJECT_ARGS\n    SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/dpdk\n    CONFIGURE_COMMAND\n      env CC=${CMAKE_C_COMPILER} ${Meson_EXECUTABLE} setup ${dpdk_args} --prefix=<INSTALL_DIR> <BINARY_DIR> <SOURCE_DIR>\n    BUILD_COMMAND\n      ${Ninja_EXECUTABLE} -C <BINARY_DIR>\n    INSTALL_COMMAND\n      ${Ninja_EXECUTABLE} -C <BINARY_DIR> install)\n\ncooking_ingredient (fmt\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/fmtlib/fmt/archive/11.2.0.tar.gz\n    URL_MD5 2f3701cada629ca455c3388d1089f5bd\n  CMAKE_ARGS\n    -DFMT_DOC=OFF\n    -DFMT_TEST=OFF)\n\ncooking_ingredient (liburing\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/axboe/liburing/archive/liburing-2.1.tar.gz\n    URL_MD5 78f13d9861b334b9a9ca0d12cf2a6d3c\n    CONFIGURE_COMMAND\n      ${CMAKE_COMMAND} -E env CC=${CMAKE_C_COMPILER} CXX=${CMAKE_CXX_COMPILER}\n      <SOURCE_DIR>/configure --prefix=<INSTALL_DIR>\n    BUILD_COMMAND <DISABLE>\n    BUILD_BYPRODUCTS \"<SOURCE_DIR>/src/liburing.a\"\n    BUILD_IN_SOURCE ON\n    INSTALL_COMMAND ${make_command} -s install)\n\ncooking_ingredient (lz4\n  EXTERNAL_PROJECT_ARGS\n    URL https://github.com/lz4/lz4/archive/v1.8.0.tar.gz\n    URL_MD5 6247bf0e955899969d1600ff34baed6b\n    # This is upsetting.\n    BUILD_IN_SOURCE ON\n    CONFIGURE_COMMAND <DISABLE>\n    BUILD_COMMAND <DISABLE>\n    INSTALL_COMMAND ${make_command} PREFIX=<INSTALL_DIR> install)\n"
  },
  {
    "path": "debug/task-latency.stap",
    "content": "#!/usr/bin/stap\n\n# usage: task_latency.stap process_name latency_threshold_ms\n\nglobal start_time\n\nprobe process(@1).mark(\"reactor_run_tasks_single_start\") {\n    start_time[tid()] = gettimeofday_us()\n}\n\nprobe process(@1).mark(\"reactor_run_tasks_single_end\") {\n    delete start_time[tid()]\n}\n\nprobe timer.profile {\n    if ([tid()] in start_time) {\n        now = gettimeofday_us()\n        start = start_time[tid()]\n        if ((now - start) > $2 * 1000) {\n            printf(\"detected tasks running for >%sms\\n\", @2)\n            print_usyms(ubacktrace())\n\t}\n    }\n}\n"
  },
  {
    "path": "demos/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\n# Logical target for all demos.\nadd_custom_target (demos)\n\nmacro (seastar_add_demo name)\n  set (args ${ARGN})\n\n  cmake_parse_arguments (\n    parsed_args\n    \"\"\n    \"\"\n    \"SOURCES\"\n    ${args})\n\n  set (target demo_${name})\n  add_executable (${target} ${parsed_args_SOURCES})\n\n  target_include_directories (${target}\n    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})\n\n  target_link_libraries (${target}\n    PRIVATE\n      Boost::program_options\n      seastar_private)\n\n  set_target_properties (${target}\n    PROPERTIES\n      OUTPUT_NAME ${name}_demo)\n\n  add_dependencies (demos ${target})\nendmacro ()\n\nseastar_add_demo (block_discard\n  SOURCES block_discard_demo.cc)\n\nif (${Seastar_API_LEVEL} GREATER_EQUAL 3)\n  seastar_add_demo (coroutines\n    SOURCES coroutines_demo.cc)\nendif ()\n\nseastar_add_demo (hello-world\n  SOURCES hello-world.cc)\n\nseastar_add_demo (websocket_server\n  SOURCES websocket_server_demo.cc)\n\nseastar_add_demo (websocket_client\n  SOURCES websocket_client_demo.cc)\n\nseastar_add_demo (echo\n  SOURCES echo_demo.cc)\n\nseastar_add_demo (ip\n  SOURCES ip_demo.cc)\n\nseastar_add_demo (line_count\n  SOURCES line_count_demo.cc)\n\nseastar_add_demo (l3\n  SOURCES l3_demo.cc)\n\nseastar_add_demo (rpc\n  SOURCES rpc_demo.cc)\n\nseastar_add_demo (scheduling_group\n  SOURCES scheduling_group_demo.cc)\n\nseastar_add_demo (tcp\n  SOURCES tcp_demo.cc)\n\nseastar_add_demo (tcp_sctp_client\n  SOURCES tcp_sctp_client_demo.cc)\n\nseastar_add_demo (tcp_sctp_server\n  SOURCES tcp_sctp_server_demo.cc)\n\nseastar_add_demo (tls_echo_server\n  SOURCES\n    tls_echo_server.hh\n    tls_echo_server_demo.cc)\n\nseastar_add_demo (tls_simple_client\n  SOURCES\n    tls_echo_server.hh\n    tls_simple_client_demo.cc)\n\nseastar_add_demo (udp_client\n  SOURCES udp_client_demo.cc)\n\nseastar_add_demo (udp_server\n  SOURCES udp_server_demo.cc)\n\nseastar_add_demo (udp_zero_copy\n  SOURCES udp_zero_copy_demo.cc)\n\nseastar_add_demo (sharded_parameter\n  SOURCES sharded_parameter_demo.cc)\n\nseastar_add_demo (file\n  SOURCES file_demo.cc)\n\nseastar_add_demo (tutorial_examples\n  SOURCES tutorial_examples.cc)\n\nseastar_add_demo (http_client\n  SOURCES http_client_demo.cc)\n\nif (Seastar_MODULE)\n  add_executable (hello_cxx_module)\n  target_sources (hello_cxx_module\n    PRIVATE\n      hello-cxx-module.cc)\n  target_link_libraries (hello_cxx_module\n    PRIVATE\n      seastar)\n  set_target_properties (hello_cxx_module\n    PROPERTIES\n      CXX_STANDARD ${CMAKE_CXX_STANDARD}\n      CXX_EXTENSIONS ON)\nendif ()\n"
  },
  {
    "path": "demos/block_discard_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <algorithm>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/util/assert.hh>\n#include <iostream>\n\nusing namespace seastar;\n\nnamespace bpo = boost::program_options;\n\nstruct file_test {\n    file_test(file&& f) : f(std::move(f)) {}\n    file f;\n    semaphore sem = { 0 };\n};\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n        (\"dev\", bpo::value<std::string>(), \"e.g. --dev /dev/sdb\")\n        ;\n\n    return app.run_deprecated(ac, av, [&app] {\n        static constexpr auto max = 10000;\n        auto&& config = app.configuration();\n        auto filepath = config[\"dev\"].as<std::string>();\n\n        return open_file_dma(filepath, open_flags::rw | open_flags::create).then([] (file f) {\n            auto ft = new file_test{std::move(f)};\n\n            // Discard asynchronously, siganl when done.\n            (void)ft->f.stat().then([ft] (struct stat st) mutable {\n                SEASTAR_ASSERT(S_ISBLK(st.st_mode));\n                auto offset = 0;\n                auto length = max * 4096;\n                return ft->f.discard(offset, length).then([ft] () mutable {\n                    ft->sem.signal();\n                });\n            });\n\n            // Wait and exit.\n            (void)ft->sem.wait().then([ft] () mutable {\n                return ft->f.flush();\n            }).then([ft] () mutable {\n                std::cout << \"done\\n\";\n                delete ft;\n                engine().exit(0);\n            });\n        });\n    });\n}\n"
  },
  {
    "path": "demos/coroutines_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n#include <iostream>\n\n#include <seastar/util/std-compat.hh>\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] () -> seastar::future<> {\n        std::cout << \"this is a completely useless program\\nplease stand by...\\n\";\n        auto f = seastar::coroutine::parallel_for_each(std::vector<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }, [] (int i) -> seastar::future<> {\n            co_await seastar::sleep(std::chrono::seconds(i));\n            std::cout << i << \"\\n\";\n        });\n\n        auto file = co_await seastar::open_file_dma(\"useless_file.txt\", seastar::open_flags::create | seastar::open_flags::wo);\n        auto out = co_await seastar::make_file_output_stream(file);\n        seastar::sstring str = \"nothing to see here, move along now\\n\";\n        co_await out.write(str);\n        co_await out.flush();\n        co_await out.close();\n\n        bool all_exist = true;\n        std::vector<seastar::sstring> filenames = { \"useless_file.txt\", \"non_existing\" };\n        co_await seastar::coroutine::parallel_for_each(filenames, [&all_exist] (const seastar::sstring& name) -> seastar::future<> {\n            all_exist &= co_await seastar::file_exists(name);\n        });\n        std::cout << (all_exist ? \"\" : \"not \") << \"all files exist\" << std::endl;\n\n        co_await std::move(f);\n        std::cout << \"done\\n\";\n    });\n}\n"
  },
  {
    "path": "demos/echo_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#include <seastar/net/virtio.hh>\n#include <seastar/net/dpdk.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/native-stack.hh>\n#include <iostream>\n#include <utility>\n#include <algorithm>\n\nusing namespace seastar;\nusing namespace net;\n\nvoid dump_packet(const packet& p) {\n    std::cout << \"rx:\";\n    auto f = p.frag(0);\n    for (unsigned i = 0; i < std::min(f.size, size_t(30)); ++i) {\n        char x[4];\n        std::sprintf(x, \" %02x\", uint8_t(f.base[i]));\n        std::cout << x;\n    }\n    std::cout << \"\\n\";\n}\n\nfuture<> echo_packet(net::qp& netif, packet p) {\n    auto f = p.frag(0);\n    if (f.size < sizeof(eth_hdr)) {\n        return make_ready_future<>();\n    }\n    auto pos = 0;\n    auto eh = reinterpret_cast<eth_hdr*>(f.base + pos);\n    pos += sizeof(*eh);\n    *eh = ntoh(*eh);\n    if (eh->eth_proto != 0x0800) {\n        return make_ready_future<>();\n    }\n    auto iph = reinterpret_cast<ip_hdr*>(f.base + pos);\n    *iph = ntoh(*iph);\n    pos += iph->ihl * 4;\n    if (iph->ver != 4 || iph->ihl < 5 || iph->ip_proto != 1) {\n        return make_ready_future<>();\n    }\n    auto ip_len = iph->len;\n    auto icmph = reinterpret_cast<icmp_hdr*>(f.base + pos);\n    if (icmph->type != icmp_hdr::msg_type::echo_request) {\n        return make_ready_future<>();\n    }\n    auto icmp_len = ip_len - iph->ihl * 4;\n    std::swap(eh->src_mac, eh->dst_mac);\n    std::swap(iph->src_ip, iph->dst_ip);\n    icmph->type = icmp_hdr::msg_type::echo_reply;\n    icmph->csum = 0;\n    *iph = hton(*iph);\n    *eh = hton(*eh);\n    icmph->csum = ip_checksum(icmph, icmp_len);\n    iph->csum = 0;\n    iph->csum = ip_checksum(iph, iph->ihl * 4);\n    return netif.send(std::move(p));\n}\n\n#ifdef SEASTAR_HAVE_DPDK\nvoid usage()\n{\n    std::cout<<\"Usage: echotest [-virtio|-dpdk]\"<<std::endl;\n    std::cout<<\"   -virtio - use virtio backend (default)\"<<std::endl;\n    std::cout<<\"   -dpdk   - use dpdk-pmd backend\"<<std::endl;\n}\n#endif\n\nint main(int ac, char** av) {\n    std::unique_ptr<net::device> dnet;\n    net::qp* vnet;\n\n    native_stack_options opts;\n\n#ifdef SEASTAR_HAVE_DPDK\n    if (ac > 2) {\n        usage();\n        return -1;\n    }\n\n    if ((ac == 1) || !std::strcmp(av[1], \"-virtio\")) {\n        dnet = create_virtio_net_device(opts.virtio_opts, opts.lro);\n    } else if (!std::strcmp(av[1], \"-dpdk\")) {\n        dnet = create_dpdk_net_device();\n    } else {\n        usage();\n        return -1;\n    }\n#else\n    dnet = create_virtio_net_device(opts.virtio_opts, opts.lro);\n#endif // SEASTAR_HAVE_DPDK\n\n    auto qp = dnet->init_local_queue(opts, 0);\n    vnet = qp.get();\n    dnet->set_local_queue(std::move(qp));\n    future<> rx_done =\n        dnet->receive([vnet] (packet p) {\n            return echo_packet(*vnet, std::move(p));\n        });\n    engine().run();\n    return 0;\n}\n\n\n"
  },
  {
    "path": "demos/file_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n\n// Demonstration of seastar::with_file\n\n#include <cstring>\n#include <limits>\n#include <random>\n\n#include <seastar/core/app-template.hh>\n\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/tmp_file.hh>\n\nusing namespace seastar;\nnamespace bpo = boost::program_options;\nusing namespace std::chrono_literals;\n\nconstexpr size_t aligned_size = 4096;\n\nfuture<> verify_data_file(file& f, temporary_buffer<char>& rbuf, const temporary_buffer<char>& wbuf) {\n    return f.dma_read(0, rbuf.get_write(), aligned_size).then([&rbuf, &wbuf] (size_t count) {\n        SEASTAR_ASSERT(count == aligned_size);\n        fmt::print(\"    verifying {} bytes\\n\", count);\n        SEASTAR_ASSERT(!memcmp(rbuf.get(), wbuf.get(), aligned_size));\n    });\n}\n\nfuture<file> open_data_file(sstring meta_filename, temporary_buffer<char>& rbuf) {\n    fmt::print(\"    retrieving data filename from {}\\n\", meta_filename);\n    return with_file(open_file_dma(meta_filename, open_flags::ro), [&rbuf] (file& f) {\n        return f.dma_read(0, rbuf.get_write(), aligned_size).then([&rbuf] (size_t count) {\n            SEASTAR_ASSERT(count == aligned_size);\n            auto data_filename = sstring(rbuf.get());\n            fmt::print(\"    opening {}\\n\", data_filename);\n            return open_file_dma(data_filename, open_flags::ro);\n        });\n    });\n}\n\nfuture<> demo_with_file() {\n    fmt::print(\"Demonstrating with_file():\\n\");\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto rnd = std::mt19937(std::random_device()());\n        auto dist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n        auto wbuf = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n        sstring meta_filename = (t.get_path() / \"meta_file\").native();\n        sstring data_filename = (t.get_path() / \"data_file\").native();\n\n        // `with_file` is used to create/open `filename` just around the call to `dma_write`\n        auto write_to_file = [] (const sstring filename, temporary_buffer<char>& wbuf) {\n            auto count = with_file(open_file_dma(filename, open_flags::rw | open_flags::create), [&wbuf] (file& f) {\n                return f.dma_write(0, wbuf.get(), aligned_size);\n            }).get();\n            SEASTAR_ASSERT(count == aligned_size);\n        };\n\n        // print the data_filename into the write buffer\n        std::fill(wbuf.get_write(), wbuf.get_write() + aligned_size, 0);\n        std::copy(data_filename.cbegin(), data_filename.cend(), wbuf.get_write());\n\n        // and write it to `meta_filename`\n        fmt::print(\"  writing \\\"{}\\\" into {}\\n\", data_filename, meta_filename);\n\n        write_to_file(meta_filename, wbuf);\n\n        // now write some random data into data_filename\n        fmt::print(\"  writing random data into {}\\n\", data_filename);\n        std::generate(wbuf.get_write(), wbuf.get_write() + aligned_size, [&dist, &rnd] { return dist(rnd); });\n\n        write_to_file(data_filename, wbuf);\n\n        // verify the data via meta_filename\n        fmt::print(\"  verifying data...\\n\");\n        auto rbuf = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n\n        with_file(open_data_file(meta_filename, rbuf), [&rbuf, &wbuf] (file& f) {\n            return verify_data_file(f, rbuf, wbuf);\n        }).get();\n    });\n}\n\nfuture<> demo_with_file_close_on_failure() {\n    fmt::print(\"\\nDemonstrating with_file_close_on_failure():\\n\");\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto rnd = std::mt19937(std::random_device()());\n        auto dist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n        auto wbuf = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n        sstring meta_filename = (t.get_path() / \"meta_file\").native();\n        sstring data_filename = (t.get_path() / \"data_file\").native();\n\n        // with_file_close_on_failure will close the opened file only if\n        // `make_file_output_stream` returns an error. Otherwise, in the error-free path,\n        // the opened file is moved to `file_output_stream` that in-turn closes it\n        // when the stream is closed.\n        auto make_output_stream = [] (std::string_view filename) {\n            return with_file_close_on_failure(open_file_dma(filename, open_flags::rw | open_flags::create), [] (file f) {\n                return make_file_output_stream(std::move(f), aligned_size);\n            });\n        };\n\n        // writes the buffer one byte at a time, to demonstrate output stream\n        auto write_to_stream = [] (output_stream<char>& o, const temporary_buffer<char>& wbuf) {\n            return seastar::do_for_each(wbuf, [&o] (char c) {\n                return o.write(&c, 1);\n            }).finally([&o] {\n                return o.close();\n            });\n        };\n\n        // print the data_filename into the write buffer\n        std::fill(wbuf.get_write(), wbuf.get_write() + aligned_size, 0);\n        std::copy(data_filename.cbegin(), data_filename.cend(), wbuf.get_write());\n\n        // and write it to `meta_filename`\n        fmt::print(\"  writing \\\"{}\\\" into {}\\n\", data_filename, meta_filename);\n\n        // with_file_close_on_failure will close the opened file only if\n        // `make_file_output_stream` returns an error. Otherwise, in the error-free path,\n        // the opened file is moved to `file_output_stream` that in-turn closes it\n        // when the stream is closed.\n        output_stream<char> meta_out = make_output_stream(meta_filename).get();\n\n        write_to_stream(meta_out, wbuf).get();\n\n        // now write some random data into data_filename\n        fmt::print(\"  writing random data into {}\\n\", data_filename);\n        std::generate(wbuf.get_write(), wbuf.get_write() + aligned_size, [&dist, &rnd] { return dist(rnd); });\n\n        output_stream<char> data_out = make_output_stream(data_filename).get();\n\n        write_to_stream(data_out, wbuf).get();\n\n        // verify the data via meta_filename\n        fmt::print(\"  verifying data...\\n\");\n        auto rbuf = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n\n        with_file(open_data_file(meta_filename, rbuf), [&rbuf, &wbuf] (file& f) {\n            return verify_data_file(f, rbuf, wbuf);\n        }).get();\n    });\n}\n\nstatic constexpr size_t half_aligned_size = aligned_size / 2;\n\nfuture<> demo_with_io_intent(std::chrono::duration<double> d) {\n    fmt::print(\"\\nDemonstrating demo_with_io_intent():\\n\");\n    return tmp_dir::do_with_thread([=] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n\n        auto rnd = std::mt19937(std::random_device()());\n        auto dist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n\n        auto wbuf = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n        fmt::print(\"  writing random data into {}\\n\", filename);\n        std::generate(wbuf.get_write(), wbuf.get_write() + aligned_size, [&dist, &rnd] { return dist(rnd); });\n\n        f.dma_write(0, wbuf.get(), aligned_size).get();\n\n        auto wbuf_n = temporary_buffer<char>::aligned(aligned_size, aligned_size);\n        fmt::print(\"  starting to overwrite {} with other random data in two steps\\n\", filename);\n        std::generate(wbuf_n.get_write(), wbuf_n.get_write() + aligned_size, [&dist, &rnd] { return dist(rnd); });\n\n        io_intent intent;\n        auto f1 = f.dma_write(0, wbuf_n.get(), half_aligned_size);\n        auto f2 = f.dma_write(half_aligned_size, wbuf_n.get() + half_aligned_size, half_aligned_size, &intent);\n\n        if (d >= std::chrono::duration<double>::zero()) {\n            sleep(std::chrono::duration_cast<std::chrono::microseconds>(d)).get();\n        }\n\n        fmt::print(\"  cancel the 2nd overwriting\\n\");\n        intent.cancel();\n\n        fmt::print(\"  wait for overwriting IOs to complete\\n\");\n        f1.get();\n\n        bool cancelled = false;\n        size_t nwritten = 0;\n        try {\n            nwritten = f2.get();\n            // The file::dma_write doesn't preemt, but if it\n            // suddenly will, the 2nd write will pass before\n            // the intent would be cancelled\n            fmt::print(\"    2nd write won the race with cancellation, written: {}\\n\", nwritten);\n        } catch (cancelled_error& ex) {\n            cancelled = true;\n        }\n\n        fmt::print(\"  cancel: {}, verifying data...\\n\", cancelled);\n        auto rbuf = allocate_aligned_buffer<unsigned char>(aligned_size, aligned_size);\n        f.dma_read(0, rbuf.get(), aligned_size).get();\n\n        // First part of the buffer must coincide with the overwritten data\n        SEASTAR_ASSERT(!memcmp(rbuf.get(), wbuf_n.get(), half_aligned_size));\n\n        if (cancelled) {\n            // Second part -- with the old data ...\n            SEASTAR_ASSERT(!memcmp(rbuf.get() + half_aligned_size, wbuf.get() + half_aligned_size, half_aligned_size));\n        } else {\n            // ... or with new if the cancellation didn't happen\n            SEASTAR_ASSERT(!memcmp(rbuf.get() + half_aligned_size, wbuf_n.get() + half_aligned_size, nwritten));\n        }\n    });\n}\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n            (\"wait\", bpo::value<std::int64_t>()->default_value(-1), \"wait milliseconds before cancelling the IO intent, not sleep by default\")\n    ;\n    return app.run(ac, av, [&] {\n        auto&& config = app.configuration();\n        auto duration = std::chrono::duration<double>(config[\"wait\"].as<std::int64_t>() * 1ms);\n        return demo_with_file().then([=] {\n            return demo_with_file_close_on_failure().then([=] {\n                return demo_with_io_intent(duration);\n            });\n        });\n    });\n}\n"
  },
  {
    "path": "demos/hello-cxx-module.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB Ltd.\n */\n\nimport seastar;\n\nusing namespace seastar;\nlogger applog(\"app\");\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] () -> future<> {\n        applog.info(\"Hello world!\");\n        return make_ready_future<>();\n    });\n}\n"
  },
  {
    "path": "demos/hello-world.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB Ltd.\n */\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/util/log.hh>\n\nusing namespace seastar;\nlogger applog(\"app\");\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] () -> future<> {\n        applog.info(\"Hello world!\");\n        return make_ready_future<>();\n    });\n}\n"
  },
  {
    "path": "demos/http_client_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB Ltd.\n */\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/http/client.hh>\n#include <seastar/http/request.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/net/tls.hh>\n\nusing namespace seastar;\nnamespace bpo = boost::program_options;\n\nstruct printer {\n    future<consumption_result<char>> operator() (temporary_buffer<char> buf) {\n        if (buf.empty()) {\n            return make_ready_future<consumption_result<char>>(stop_consuming(std::move(buf)));\n        }\n        fmt::print(\"{}\", sstring(buf.get(), buf.size()));\n        return make_ready_future<consumption_result<char>>(continue_consuming());\n    }\n};\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n            (\"https\", bpo::bool_switch(), \"Use HTTPS on port 443 (if off -- use HTTP on port 80)\")\n            (\"host\", bpo::value<std::string>(), \"Host to connect\")\n            (\"path\", bpo::value<std::string>(), \"Path to query upon\")\n            (\"method\", bpo::value<std::string>()->default_value(\"GET\"), \"Method to use\")\n            (\"file\", bpo::value<std::string>(), \"File to get body from (no body if missing)\")\n    ;\n\n\n    return app.run(ac, av, [&] {\n        auto&& config = app.configuration();\n        auto host = config[\"host\"].as<std::string>();\n        auto path = config[\"path\"].as<std::string>();\n        auto method = config[\"method\"].as<std::string>();\n        auto body = config.count(\"file\") == 0 ? std::string(\"\") : config[\"file\"].as<std::string>();\n        auto https = config[\"https\"].as<bool>();\n\n        return seastar::async([=] {\n            net::hostent e = net::dns::get_host_by_name(host, net::inet_address::family::INET).get();\n            std::unique_ptr<http::experimental::client> cln;\n            if (https) {\n                auto certs = ::make_shared<tls::certificate_credentials>();\n                certs->set_system_trust().get();\n                fmt::print(\"{} {}:443{}\\n\", method, e.addr_entries.front().addr, path);\n                cln = std::make_unique<http::experimental::client>(socket_address(e.addr_entries.front().addr, 443), std::move(certs), host);\n            } else {\n                fmt::print(\"{} {}:80{}\\n\", method, e.addr_entries.front().addr, path);\n                cln = std::make_unique<http::experimental::client>(socket_address(e.addr_entries.front().addr, 80));\n            }\n            auto req = http::request::make(method, host, path);\n            if (body != \"\") {\n                future<file> f = open_file_dma(body, open_flags::ro);\n                req.write_body(\"txt\", [ f = std::move(f) ] (output_stream<char>&& out) mutable {\n                    return seastar::async([f = std::move(f), out = std::move(out)] () mutable {\n                        auto in = make_file_input_stream(f.get());\n                        copy(in, out).get();\n                        out.flush().get();\n                        out.close().get();\n                        in.close().get();\n                    });\n                });\n            }\n            cln->make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                fmt::print(\"Reply status {}\\n--------8<--------\\n\", rep._status);\n                return seastar::async([in = std::move(in)] () mutable {\n                    in.consume(printer{}).get();\n                    in.close().get();\n                });\n            }).get();\n\n            cln->close().get();\n        }).handle_exception([](auto ep) {\n            fmt::print(\"Error: {}\", ep);\n        });\n    });\n}\n"
  },
  {
    "path": "demos/ip_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/arp.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/net.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/net/virtio.hh>\n#include <seastar/net/native-stack.hh>\n#include <seastar/core/aligned_buffer.hh>\n\nusing namespace seastar;\nusing namespace net;\n\nint main(int ac, char** av) {\n    native_stack_options opts;\n\n    auto vnet = create_virtio_net_device(opts.virtio_opts, opts.lro);\n    vnet->set_local_queue(vnet->init_local_queue(opts, 0));\n\n    interface netif(std::move(vnet));\n    ipv4 inet(&netif);\n    inet.set_host_address(ipv4_address(\"192.168.122.2\"));\n    engine().run();\n    return 0;\n}\n\n\n\n"
  },
  {
    "path": "demos/l3_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/net.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/net/virtio.hh>\n#include <seastar/net/native-stack.hh>\n#include <iostream>\n\nusing namespace seastar;\nusing namespace net;\n\nvoid dump_arp_packets(l3_protocol& proto) {\n    // FIXME: ignored future\n    (void)proto.receive([] (packet p, ethernet_address from) {\n        std::cout << \"seen arp packet\\n\";\n        return make_ready_future<>();\n    }, [] (forward_hash& out_hash_data, packet& p, size_t off) {return false;});\n}\n\nint main(int ac, char** av) {\n    native_stack_options opts;\n\n    auto vnet = create_virtio_net_device(opts.virtio_opts, opts.lro);\n    interface netif(std::move(vnet));\n    l3_protocol arp(&netif, eth_protocol_num::arp, []{ return std::optional<l3_protocol::l3packet>(); });\n    dump_arp_packets(arp);\n    engine().run();\n    return 0;\n}\n\n"
  },
  {
    "path": "demos/line_count_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n// Demonstration of file_input_stream.  Don't expect stellar performance\n// since no read-ahead or caching is done yet.\n\n#include <seastar/core/fstream.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <fmt/printf.h>\n#include <algorithm>\n#include <iostream>\n\nusing namespace seastar;\n\nstruct reader {\npublic:\n    reader(file f)\n            : is(make_file_input_stream(std::move(f), file_input_stream_options{1 << 16, 1})) {\n    }\n\n    input_stream<char> is;\n    size_t count = 0;\n\n    // for input_stream::consume():\n    using unconsumed_remainder = std::optional<temporary_buffer<char>>;\n    future<unconsumed_remainder> operator()(temporary_buffer<char> data) {\n        if (data.empty()) {\n            return make_ready_future<unconsumed_remainder>(std::move(data));\n        } else {\n            count += std::count(data.begin(), data.end(), '\\n');\n            // FIXME: last line without \\n?\n            return make_ready_future<unconsumed_remainder>();\n        }\n    }\n};\n\nint main(int ac, char** av) {\n    app_template app;\n    namespace bpo = boost::program_options;\n    app.add_positional_options({\n        { \"file\", bpo::value<std::string>(), \"File to process\", 1 },\n    });\n    return app.run(ac, av, [&app] {\n        auto fname = app.configuration()[\"file\"].as<std::string>();\n        return open_file_dma(fname, open_flags::ro).then([] (file f) {\n            auto r = make_shared<reader>(std::move(f));\n            return r->is.consume(*r).then([r] {\n               fmt::print(\"{:d} lines\\n\", r->count);\n               return r->is.close().then([r] {});\n            });\n        }).then_wrapped([] (future<> f) -> future<int> {\n            try {\n                f.get();\n                return make_ready_future<int>(0);\n            } catch (std::exception& ex) {\n                std::cout << ex.what() << \"\\n\";\n                return make_ready_future<int>(1);\n            } catch (...) {\n                std::cout << \"unknown exception\\n\";\n                return make_ready_future<int>(1);\n            }\n        });\n    });\n}\n\n"
  },
  {
    "path": "demos/rpc_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#include <cmath>\n#include <iostream>\n#include <ranges>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/rpc/rpc.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/rpc/lz4_compressor.hh>\n#include <seastar/util/log.hh>\n#include <seastar/core/loop.hh>\n\nusing namespace seastar;\n\nstruct serializer {\n};\n\ntemplate <typename T, typename Output>\ninline\nvoid write_arithmetic_type(Output& out, T v) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    return out.write(reinterpret_cast<const char*>(&v), sizeof(T));\n}\n\ntemplate <typename T, typename Input>\ninline\nT read_arithmetic_type(Input& in) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    T v;\n    in.read(reinterpret_cast<char*>(&v), sizeof(T));\n    return v;\n}\n\ntemplate <typename Output>\ninline void write(serializer, Output& output, int32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, int64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, double v) { return write_arithmetic_type(output, v); }\ntemplate <typename Input>\ninline int32_t read(serializer, Input& input, rpc::type<int32_t>) { return read_arithmetic_type<int32_t>(input); }\ntemplate <typename Input>\ninline uint32_t read(serializer, Input& input, rpc::type<uint32_t>) { return read_arithmetic_type<uint32_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<uint64_t>) { return read_arithmetic_type<uint64_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<int64_t>) { return read_arithmetic_type<int64_t>(input); }\ntemplate <typename Input>\ninline double read(serializer, Input& input, rpc::type<double>) { return read_arithmetic_type<double>(input); }\n\ntemplate <typename Output>\ninline void write(serializer, Output& out, const sstring& v) {\n    write_arithmetic_type(out, uint32_t(v.size()));\n    out.write(v.c_str(), v.size());\n}\n\ntemplate <typename Input>\ninline sstring read(serializer, Input& in, rpc::type<sstring>) {\n    auto size = read_arithmetic_type<uint32_t>(in);\n    sstring ret = uninitialized_string(size);\n    in.read(ret.data(), size);\n    return ret;\n}\n\nnamespace bpo = boost::program_options;\nusing namespace std::chrono_literals;\n\nclass mycomp : public rpc::compressor::factory {\n    const sstring _name = \"LZ4\";\npublic:\n    virtual const sstring& supported() const override {\n        fmt::print(\"supported called\\n\");\n        return _name;\n    }\n    virtual std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server) const override {\n        fmt::print(\"negotiate called with {}\\n\", feature);\n        return feature == _name ? std::make_unique<rpc::lz4_compressor>() : nullptr;\n    }\n};\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n                    (\"port\", bpo::value<uint16_t>()->default_value(10000), \"RPC server port\")\n                    (\"server\", bpo::value<std::string>(), \"Server address\")\n                    (\"compress\", bpo::value<bool>()->default_value(false), \"Compress RPC traffic\");\n    std::cout << \"start \";\n    rpc::protocol<serializer> myrpc(serializer{});\n    static std::unique_ptr<rpc::protocol<serializer>::server> server;\n    static std::unique_ptr<rpc::protocol<serializer>::client> client;\n    static double x = 30.0;\n\n    static logger log(\"rpc_demo\");\n    myrpc.set_logger(&log);\n\n    return app.run_deprecated(ac, av, [&] {\n        auto&& config = app.configuration();\n        uint16_t port = config[\"port\"].as<uint16_t>();\n        bool compress = config[\"compress\"].as<bool>();\n        static mycomp mc;\n        auto test1 = myrpc.register_handler(1, [x = 0](int i) mutable { fmt::print(\"test1 count {:d} got {:d}\\n\", ++x, i); });\n        auto test2 = myrpc.register_handler(2, [](int a, int b){ fmt::print(\"test2 got {:d} {:d}\\n\", a, b); return make_ready_future<int>(a+b); });\n        auto test3 = myrpc.register_handler(3, [](double x){ fmt::print(\"test3 got {:f}\\n\", x); return std::make_unique<double>(sin(x)); });\n        auto test4 = myrpc.register_handler(4, [](){ fmt::print(\"test4 throw!\\n\"); throw std::runtime_error(\"exception!\"); });\n        auto test5 = myrpc.register_handler(5, [](){ fmt::print(\"test5 no wait\\n\"); return rpc::no_wait; });\n        auto test6 = myrpc.register_handler(6, [](const rpc::client_info& info, int x){ fmt::print(\"test6 client {}, {:d}\\n\", inet_ntoa(info.addr.as_posix_sockaddr_in().sin_addr), x); });\n        auto test8 = myrpc.register_handler(8, [](){ fmt::print(\"test8 sleep for 2 sec\\n\"); return sleep(2s); });\n        auto test13 = myrpc.register_handler(13, [](){ fmt::print(\"test13 sleep for 1 msec\\n\"); return sleep(1ms); });\n        auto test_message_to_big = myrpc.register_handler(14, [](sstring payload){ fmt::print(\"test message to bit, should not get here\"); });\n\n        if (config.count(\"server\")) {\n            std::cout << \"client\" << std::endl;\n            auto test7 = myrpc.make_client<long (long a, long b)>(7);\n            auto test9 = myrpc.make_client<long (long a, long b)>(9); // do not send optional\n            auto test9_1 = myrpc.make_client<long (long a, long b, int c)>(9); // send optional\n            auto test9_2 = myrpc.make_client<long (long a, long b, int c, long d)>(9); // send more data than handler expects\n            auto test10 = myrpc.make_client<long ()>(10); // receive less then replied\n            auto test10_1 = myrpc.make_client<future<rpc::tuple<long, int>> ()>(10); // receive all\n            auto test11 = myrpc.make_client<future<rpc::tuple<long, rpc::optional<int>>> ()>(11); // receive more then replied\n            auto test12 = myrpc.make_client<void (int sleep_ms, sstring payload)>(12); // large payload vs. server limits\n            auto test_nohandler = myrpc.make_client<void ()>(100000000); // non existing verb\n            auto test_nohandler_nowait = myrpc.make_client<rpc::no_wait_type ()>(100000000); // non existing verb, no_wait call\n            rpc::client_options co;\n            if (compress) {\n                co.compressor_factory = &mc;\n            }\n\n            client = std::make_unique<rpc::protocol<serializer>::client>(myrpc, co, ipv4_addr{config[\"server\"].as<std::string>()});\n\n            auto f = test8(*client, 1500ms).then_wrapped([](future<> f) {\n                try {\n                    f.get();\n                    printf(\"test8 should not get here!\\n\");\n                } catch (rpc::timeout_error&) {\n                    printf(\"test8 timeout!\\n\");\n                }\n            });\n            for (auto i = 0; i < 100; i++) {\n                fmt::print(\"iteration={:d}\\n\", i);\n                (void)test1(*client, 5).then([] (){ fmt::print(\"test1 ended\\n\");});\n                (void)test2(*client, 1, 2).then([] (int r) { fmt::print(\"test2 got {:d}\\n\", r); });\n                (void)test3(*client, x).then([](double x) { fmt::print(\"sin={:f}\\n\", x); });\n                (void)test4(*client).then_wrapped([](future<> f) {\n                    try {\n                        f.get();\n                        fmt::print(\"test4 your should not see this!\\n\");\n                    } catch (std::runtime_error& x){\n                        fmt::print(\"test4 {}\\n\", x.what());\n                    }\n                });\n                (void)test5(*client).then([] { fmt::print(\"test5 no wait ended\\n\"); });\n                (void)test6(*client, 1).then([] { fmt::print(\"test6 ended\\n\"); });\n                (void)test7(*client, 5, 6).then([] (long r) { fmt::print(\"test7 got {:d}\\n\", r); });\n                (void)test9(*client, 1, 2).then([] (long r) { fmt::print(\"test9 got {:d}\\n\", r); });\n                (void)test9_1(*client, 1, 2, 3).then([] (long r) { fmt::print(\"test9.1 got {:d}\\n\", r); });\n                (void)test9_2(*client, 1, 2, 3, 4).then([] (long r) { fmt::print(\"test9.2 got {:d}\\n\", r); });\n                (void)test10(*client).then([] (long r) { fmt::print(\"test10 got {:d}\\n\", r); });\n                (void)test10_1(*client).then([] (rpc::tuple<long, int> r) { fmt::print(\"test10_1 got {:d} and {:d}\\n\", std::get<0>(r), std::get<1>(r)); });\n                (void)test11(*client).then([] (rpc::tuple<long, rpc::optional<int> > r) { fmt::print(\"test11 got {:d} and {:d}\\n\", std::get<0>(r), bool(std::get<1>(r))); });\n                (void)test_nohandler(*client).then_wrapped([](future<> f) {\n                    try {\n                        f.get();\n                        fmt::print(\"test_nohandler your should not see this!\\n\");\n                    } catch (rpc::unknown_verb_error& x){\n                        fmt::print(\"test_nohandle no such verb\\n\");\n                    } catch (...) {\n                        fmt::print(\"incorrect exception!\\n\");\n                    }\n                });\n                (void)test_nohandler_nowait(*client);\n                auto c = make_lw_shared<rpc::cancellable>();\n                (void)test13(*client, *c).then_wrapped([](future<> f) {\n                    try {\n                        f.get();\n                        fmt::print(\"test13 shold not get here\\n\");\n                    } catch(rpc::canceled_error&) {\n                        fmt::print(\"test13 canceled\\n\");\n                    } catch(...) {\n                        fmt::print(\"test13 wrong exception\\n\");\n                    }\n                });\n                c->cancel();\n                (void)test13(*client, *c).then_wrapped([](future<> f) {\n                    try {\n                        f.get();\n                        fmt::print(\"test13 shold not get here\\n\");\n                    } catch(rpc::canceled_error&) {\n                        fmt::print(\"test13 canceled\\n\");\n                    } catch(...) {\n                        fmt::print(\"test13 wrong exception\\n\");\n                    }\n                });\n                (void)sleep(500us).then([c] { c->cancel(); });\n                (void)test_message_to_big(*client, uninitialized_string(10'000'001)).then_wrapped([](future<> f) {\n                    try {\n                        f.get();\n                        fmt::print(\"test message to big shold not get here\\n\");\n                    } catch(std::runtime_error& err) {\n                        fmt::print(\"test message to big get error {}\\n\", err.what());\n                    } catch(...) {\n                        fmt::print(\"test message to big wrong exception\\n\");\n                    }\n                });\n            }\n            // delay a little for a time-sensitive test\n            (void)sleep(400ms).then([test12] () mutable {\n                // server is configured for 10MB max, throw 25MB worth of requests at it.\n                auto now = rpc::rpc_clock_type::now();\n                return parallel_for_each(std::views::iota(0, 25), [test12, now] (int idx) mutable {\n                    return test12(*client, 100, uninitialized_string(1'000'000)).then([idx, now] {\n                        auto later = rpc::rpc_clock_type::now();\n                        auto delta = std::chrono::duration_cast<std::chrono::milliseconds>(later - now);\n                        fmt::print(\"idx {:d} completed after {:d} ms\\n\", idx, delta.count());\n                    });\n                }).then([now] {\n                    auto later = rpc::rpc_clock_type::now();\n                    auto delta = std::chrono::duration_cast<std::chrono::milliseconds>(later - now);\n                    fmt::print(\"test12 completed after {:d} ms (should be ~300)\\n\", delta.count());\n                });\n            });\n            (void)f.finally([] {\n                return sleep(1s).then([] {\n                    return client->stop().then([] {\n                        return engine().exit(0);\n                    });\n                });\n            });\n        } else {\n            std::cout << \"server on port \" << port << std::endl;\n            myrpc.register_handler(7, [](long a, long b) mutable {\n                auto p = make_lw_shared<promise<>>();\n                auto t = make_lw_shared<timer<>>();\n                fmt::print(\"test7 got {:d} {:d}\\n\", a, b);\n                auto f = p->get_future().then([a, b, t] {\n                    fmt::print(\"test7 calc res\\n\");\n                    return a - b;\n                });\n                t->set_callback([p = std::move(p)] () mutable { p->set_value(); });\n                t->arm(1s);\n                return f;\n            });\n            myrpc.register_handler(9, [] (long a, long b, rpc::optional<int> c) {\n                long r = 2;\n                fmt::print(\"test9 got {:d} {:d} \", a, b);\n                if (c) {\n                    fmt::print(\"{:d}\", c.value());\n                    r++;\n                }\n                fmt::print(\"\\n\");\n                return r;\n            });\n            myrpc.register_handler(10, [] {\n                fmt::print(\"test 10\\n\");\n                return make_ready_future<rpc::tuple<long, int>>(rpc::tuple<long, int>(1, 2));\n            });\n            myrpc.register_handler(11, [] {\n                fmt::print(\"test 11\\n\");\n                return 1ul;\n            });\n            myrpc.register_handler(12, [] (int sleep_ms, sstring payload) {\n                return sleep(std::chrono::milliseconds(sleep_ms)).then([] {\n                    return make_ready_future<>();\n                });\n            });\n\n            rpc::resource_limits limits;\n            limits.bloat_factor = 1;\n            limits.basic_request_size = 0;\n            limits.max_memory = 10'000'000;\n            rpc::server_options so;\n            if (compress) {\n                so.compressor_factory = &mc;\n            }\n            server = std::make_unique<rpc::protocol<serializer>::server>(myrpc, so, ipv4_addr{port}, limits);\n        }\n    });\n\n}\n"
  },
  {
    "path": "demos/scheduling_group_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylla DB Ltd\n */\n\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/util/defer.hh>\n#include <fmt/printf.h>\n#include <chrono>\n#include <cmath>\n#include <ranges>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\ntemplate <typename Func, typename Duration>\nfuture<>\ncompute_intensive_task(Duration duration, unsigned& counter, Func func) {\n    auto end = std::chrono::steady_clock::now() + duration;\n    while (std::chrono::steady_clock::now() < end) {\n        func();\n    }\n    ++counter;\n    return make_ready_future<>();\n}\n\nfuture<>\nheavy_task(unsigned& counter) {\n    return compute_intensive_task(1ms, counter, [] {\n        static thread_local double x = 1;\n        x = std::exp(x) / 3;\n    });\n}\n\nfuture<>\nlight_task(unsigned& counter) {\n    return compute_intensive_task(100us, counter, [] {\n        static thread_local double x = 0.1;\n        x = std::log(x + 1);\n    });\n}\n\nfuture<>\nmedium_task(unsigned& counter) {\n    return compute_intensive_task(400us, counter, [] {\n        static thread_local double x = 0.1;\n        x = std::cos(x);\n    });\n}\n\nusing done_func = std::function<bool ()>;\n\nfuture<>\nrun_compute_intensive_tasks(seastar::scheduling_group sg, done_func done, unsigned concurrency, unsigned& counter, std::function<future<> (unsigned& counter)> task) {\n    return seastar::async([task = std::move(task), sg, concurrency, done, &counter] () mutable {\n        while (!done()) {\n            parallel_for_each(std::views::iota(0u, concurrency), [task, sg, &counter] (unsigned i) mutable {\n                return with_scheduling_group(sg, [task, &counter] {\n                    return task(counter);\n                });\n            }).get();\n            thread::maybe_yield();\n        }\n    });\n}\n\nfuture<>\nrun_compute_intensive_tasks_in_threads(seastar::scheduling_group sg, done_func done, unsigned concurrency, unsigned& counter, std::function<future<> (unsigned& counter)> task) {\n    auto attr = seastar::thread_attributes();\n    attr.sched_group = sg;\n    return parallel_for_each(std::views::iota(0u, concurrency), [attr, done, &counter, task] (unsigned i) {\n        return seastar::async(attr, [done, &counter, task] {\n            while (!done()) {\n                task(counter).get();\n                thread::maybe_yield();\n            }\n        });\n    });\n}\n\nfuture<>\nrun_with_duty_cycle(float utilization, std::chrono::steady_clock::duration period, done_func done, std::function<future<> (done_func done)> task) {\n    return seastar::async([=] {\n        bool duty_toggle = true;\n        auto t0 = std::chrono::steady_clock::now();\n        condition_variable cv;\n        timer<> tmr_on([&] { duty_toggle = true; cv.signal(); });\n        timer<> tmr_off([&] { duty_toggle = false; });\n        tmr_on.arm(t0, period);\n        tmr_off.arm(t0 + std::chrono::duration_cast<decltype(t0)::duration>(period * utilization), period);\n        auto combined_done = [&] {\n            return done() || !duty_toggle;\n        };\n        while (!done()) {\n            while (!combined_done()) {\n                task(std::cref(combined_done)).get();\n                thread::maybe_yield();\n            }\n            cv.wait([&] {\n                return done() || duty_toggle;\n            }).get();\n        }\n        tmr_on.cancel();\n        tmr_off.cancel();\n    });\n}\n\n#include <fenv.h>\n\ntemplate <typename T>\nauto var_fn(T& var) {\n    return [&var] { return var; };\n}\n\nint main(int ac, char** av) {\n    app_template app;\n    return app.run(ac, av, [] {\n        return seastar::async([] {\n            auto sg100 = seastar::create_scheduling_group(\"sg100\", 100).get();\n            auto ksg100 = seastar::defer([&] () noexcept { seastar::destroy_scheduling_group(sg100).get(); });\n            auto sg20 = seastar::create_scheduling_group(\"sg20\", 20).get();\n            auto ksg20 = seastar::defer([&] () noexcept { seastar::destroy_scheduling_group(sg20).get(); });\n            auto sg50 = seastar::create_scheduling_group(\"sg50\", 50).get();\n            auto ksg50 = seastar::defer([&] () noexcept { seastar::destroy_scheduling_group(sg50).get(); });\n\n            bool done = false;\n            auto end = timer<>([&done] {\n                done = true;\n            });\n\n            end.arm(10s);\n            unsigned ctr100 = 0, ctr20 = 0, ctr50 = 0;\n            fmt::print(\"running three scheduling groups with 100% duty cycle each:\\n\");\n            when_all(\n                    run_compute_intensive_tasks(sg100, var_fn(done), 5, ctr100, heavy_task),\n                    run_compute_intensive_tasks(sg20, var_fn(done), 3, ctr20, light_task),\n                    run_compute_intensive_tasks_in_threads(sg50, var_fn(done), 2, ctr50, medium_task)\n                    ).get();\n            fmt::print(\"{:10} {:15} {:10} {:12} {:8}\\n\", \"shares\", \"task_time (us)\", \"executed\", \"runtime (ms)\", \"vruntime\");\n            fmt::print(\"{:10d} {:15d} {:10d} {:12d} {:8.2f}\\n\", 100, 1000, ctr100, ctr100 * 1000 / 1000, ctr100 * 1000 / 1000 / 100.);\n            fmt::print(\"{:10d} {:15d} {:10d} {:12d} {:8.2f}\\n\", 20, 100, ctr20, ctr20 * 100 / 1000, ctr20 * 100 / 1000 / 20.);\n            fmt::print(\"{:10d} {:15d} {:10d} {:12d} {:8.2f}\\n\", 50, 400, ctr50, ctr50 * 400 / 1000, ctr50 * 400 / 1000 / 50.);\n            fmt::print(\"\\n\");\n\n            fmt::print(\"running two scheduling groups with 100%/50% duty cycles (period=1s:\\n\");\n            unsigned ctr100_2 = 0, ctr50_2 = 0;\n            done = false;\n            end.arm(10s);\n            when_all(\n                    run_compute_intensive_tasks(sg50, var_fn(done), 5, ctr50_2, heavy_task),\n                    run_with_duty_cycle(0.5, 1s, var_fn(done), [=, &ctr100_2] (done_func done) {\n                        return run_compute_intensive_tasks(sg100, done, 4, ctr100_2, heavy_task);\n                    })\n            ).get();\n            fmt::print(\"{:10} {:10} {:15} {:10} {:12} {:8}\\n\", \"shares\", \"duty\", \"task_time (us)\", \"executed\", \"runtime (ms)\", \"vruntime\");\n            fmt::print(\"{:10d} {:10d} {:15d} {:10d} {:12d} {:8.2f}\\n\", 100, 50, 1000, ctr100_2, ctr100_2 * 1000 / 1000, ctr100_2 * 1000 / 1000 / 100.);\n            fmt::print(\"{:10d} {:10d} {:15d} {:10d} {:12d} {:8.2f}\\n\", 50, 100, 400, ctr50_2, ctr50_2 * 1000 / 1000, ctr50_2 * 1000 / 1000 / 50.);\n\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "demos/sharded_parameter_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n\n// Demonstration of seastar::sharded_parameter\n\n#include <seastar/core/sharded.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/closeable.hh>\n\n// This is some service that we wish to run on all shards.\nclass service_one {\n    int _capacity = 7;\npublic:\n    // Pretend that this int is some important resource.\n    int get_capacity() const { return _capacity; }\n};\n\n// Another service that we run on all shards, that depends on service_one.\nclass service_two {\n    int _resource_allocation;\npublic:\n    service_two(service_one& s1, int resource_allocation) : _resource_allocation(resource_allocation) {}\n    int get_resource_allocation() const { return _resource_allocation; }\n};\n\nint main(int ac, char** av) {\n    seastar::app_template app;\n    return app.run(ac, av, [&] {\n        // sharded<> setup code is typically run in a seastar::thread\n        return seastar::async([&] {\n\n            // Launch service_one\n            seastar::sharded<service_one> s1;\n            s1.start().get();\n            auto stop_s1 = seastar::deferred_stop(s1);\n\n            auto calculate_half_capacity = [] (service_one& s1) {\n                return s1.get_capacity() / 2;\n            };\n\n            // Launch service_two, passing it per-shard dependencies from s1\n            seastar::sharded<service_two> s2;\n            // Start s2, passing two parameters to service_two's constructor\n            s2.start(\n                    // Each service_two instance will get a reference to a service_one instance on the same shard\n                    std::ref(s1),\n                    // This calculation will be performed on each shard\n                    seastar::sharded_parameter(calculate_half_capacity, std::ref(s1))\n            ).get();\n            auto stop_s2 = seastar::deferred_stop(s2);\n\n            s2.invoke_on_all([] (service_two& s2) {\n                SEASTAR_ASSERT(s2.get_resource_allocation() == 3);\n            }).get();\n        });\n    });\n}\n"
  },
  {
    "path": "demos/tcp_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/ip.hh>\n#include <seastar/net/virtio.hh>\n#include <seastar/net/tcp.hh>\n#include <seastar/net/native-stack.hh>\n#include <seastar/core/reactor.hh>\n#include <fmt/printf.h>\n\nusing namespace seastar;\nusing namespace net;\n\nstruct tcp_test {\n    ipv4& inet;\n    using tcp = net::tcp<ipv4_traits>;\n    tcp::listener _listener;\n    struct connection {\n        tcp::connection tcp_conn;\n        explicit connection(tcp::connection tc) : tcp_conn(std::move(tc)) {}\n        void run() {\n            // Read packets and echo back in the background.\n            (void)tcp_conn.wait_for_data().then([this] {\n                auto p = tcp_conn.read();\n                if (!p.len()) {\n                    tcp_conn.close_write();\n                    return;\n                }\n                fmt::print(\"read {:d} bytes\\n\", p.len());\n                auto v = std::move(p).release();\n                (void)tcp_conn.send(std::span(v));\n                run();\n            });\n        }\n    };\n    tcp_test(ipv4& inet) : inet(inet), _listener(inet.get_tcp().listen(10000)) {}\n    void run() {\n        // Run all connections in the background.\n        (void)_listener.accept().then([this] (tcp::connection conn) {\n            (new connection(std::move(conn)))->run();\n            run();\n        });\n    }\n};\n\nint main(int ac, char** av) {\n    native_stack_options opts;\n\n    auto vnet = create_virtio_net_device(opts.virtio_opts, opts.lro);\n    interface netif(std::move(vnet));\n    ipv4 inet(&netif);\n    inet.set_host_address(ipv4_address(\"192.168.122.2\"));\n    tcp_test tt(inet);\n    (void)engine().when_started().then([&tt] { tt.run(); });\n    engine().run();\n}\n\n\n"
  },
  {
    "path": "demos/tcp_sctp_client_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <iostream>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/units.hh>\n\nusing namespace seastar;\nusing namespace net;\nusing namespace std::chrono_literals;\n\nstatic int rx_msg_size = 4_KiB;\nstatic int tx_msg_total_size = 100_MiB;\nstatic int tx_msg_size = 4_KiB;\nstatic int tx_msg_nr = tx_msg_total_size / tx_msg_size;\nstatic std::string str_txbuf(tx_msg_size, 'X');\n\nclass client;\nsharded<client> clients;\n\ntransport protocol = transport::TCP;\n\nclass client {\nprivate:\n    static constexpr unsigned _pings_per_connection = 10000;\n    unsigned _total_pings;\n    unsigned _concurrent_connections;\n    ipv4_addr _server_addr;\n    std::string _test;\n    lowres_clock::time_point _earliest_started = lowres_clock::time_point::max();\n    lowres_clock::time_point _latest_finished = lowres_clock::time_point::min();\n    size_t _processed_bytes;\n    unsigned _num_reported;\npublic:\n    class connection {\n        connected_socket _fd;\n        input_stream<char> _read_buf;\n        output_stream<char> _write_buf;\n        size_t _bytes_read = 0;\n        size_t _bytes_write = 0;\n    public:\n        connection(connected_socket&& fd)\n            : _fd(std::move(fd))\n            , _read_buf(_fd.input())\n            , _write_buf(_fd.output()) {}\n\n        future<> do_read() {\n            return _read_buf.read_exactly(rx_msg_size).then([this] (temporary_buffer<char> buf) {\n                _bytes_read += buf.size();\n                if (buf.size() == 0) {\n                    return make_ready_future();\n                } else {\n                    return do_read();\n                }\n            });\n        }\n\n        future<> do_write(int end) {\n            if (end == 0) {\n                return make_ready_future();\n            }\n            return _write_buf.write(str_txbuf).then([this] {\n                _bytes_write += tx_msg_size;\n                return _write_buf.flush();\n            }).then([this, end] {\n                return do_write(end - 1);\n            });\n        }\n\n        future<> ping(int times) {\n            return _write_buf.write(\"ping\").then([this] {\n                return _write_buf.flush();\n            }).then([this, times] {\n                return _read_buf.read_exactly(4).then([this, times] (temporary_buffer<char> buf) {\n                    if (buf.size() != 4) {\n                        fmt::print(std::cerr, \"illegal packet received: {}\\n\", buf.size());\n                        return make_ready_future();\n                    }\n                    auto str = std::string(buf.get(), buf.size());\n                    if (str != \"pong\") {\n                        fmt::print(std::cerr, \"illegal packet received: {}\\n\", buf.size());\n                        return make_ready_future();\n                    }\n                    if (times > 0) {\n                        return ping(times - 1);\n                    } else {\n                        return make_ready_future();\n                    }\n                });\n            });\n        }\n\n        future<size_t> rxrx() {\n            return _write_buf.write(\"rxrx\").then([this] {\n                return _write_buf.flush();\n            }).then([this] {\n                return do_write(tx_msg_nr).then([this] {\n                    return _write_buf.close();\n                }).then([this] {\n                    return make_ready_future<size_t>(_bytes_write);\n                });\n            });\n        }\n\n        future<size_t> txtx() {\n            return _write_buf.write(\"txtx\").then([this] {\n                return _write_buf.flush();\n            }).then([this] {\n                return do_read().then([this] {\n                    return make_ready_future<size_t>(_bytes_read);\n                });\n            });\n        }\n    };\n\n    future<> ping_test(connection *conn) {\n        auto started = lowres_clock::now();\n        return conn->ping(_pings_per_connection).then([started] {\n            auto finished = lowres_clock::now();\n            (void)clients.invoke_on(0, &client::ping_report, started, finished);\n        });\n    }\n\n    future<> rxrx_test(connection *conn) {\n        auto started = lowres_clock::now();\n        return conn->rxrx().then([started] (size_t bytes) {\n            auto finished = lowres_clock::now();\n            (void)clients.invoke_on(0, &client::rxtx_report, started, finished, bytes);\n        });\n    }\n\n    future<> txtx_test(connection *conn) {\n        auto started = lowres_clock::now();\n        return conn->txtx().then([started] (size_t bytes) {\n            auto finished = lowres_clock::now();\n            (void)clients.invoke_on(0, &client::rxtx_report, started, finished, bytes);\n        });\n    }\n\n    void ping_report(lowres_clock::time_point started, lowres_clock::time_point finished) {\n        if (_earliest_started > started)\n            _earliest_started = started;\n        if (_latest_finished < finished)\n            _latest_finished = finished;\n        if (++_num_reported == _concurrent_connections) {\n            auto elapsed = _latest_finished - _earliest_started;\n            auto usecs = std::chrono::duration_cast<std::chrono::microseconds>(elapsed).count();\n            auto secs = static_cast<double>(usecs) / static_cast<double>(1000 * 1000);\n            fmt::print(std::cout, \"========== ping ============\\n\");\n            fmt::print(std::cout, \"Server: {}\\n\", _server_addr);\n            fmt::print(std::cout,\"Connections: {}\\n\", _concurrent_connections);\n            fmt::print(std::cout, \"Total PingPong: {}\\n\", _total_pings);\n            fmt::print(std::cout, \"Total Time(Secs): {}\\n\", secs);\n            fmt::print(std::cout, \"Requests/Sec: {}\\n\",\n                static_cast<double>(_total_pings) / secs);\n            (void)clients.stop().then([] {\n                engine().exit(0);\n            });\n        }\n    }\n\n    void rxtx_report(lowres_clock::time_point started, lowres_clock::time_point finished, size_t bytes) {\n        if (_earliest_started > started)\n            _earliest_started = started;\n        if (_latest_finished < finished)\n            _latest_finished = finished;\n        _processed_bytes += bytes;\n        if (++_num_reported == _concurrent_connections) {\n            auto elapsed = _latest_finished - _earliest_started;\n            auto usecs = std::chrono::duration_cast<std::chrono::microseconds>(elapsed).count();\n            auto secs = static_cast<double>(usecs) / static_cast<double>(1000 * 1000);\n            fmt::print(std::cout, \"========== {} ============\\n\", _test);\n            fmt::print(std::cout, \"Server: {}\\n\", _server_addr);\n            fmt::print(std::cout, \"Connections: {}\\n\", _concurrent_connections);\n            fmt::print(std::cout, \"Bytes Received(MiB): {}\\n\", _processed_bytes / 1_MiB);\n            fmt::print(std::cout, \"Total Time(Secs): {}\\n\", secs);\n            fmt::print(std::cout, \"Bandwidth(Gbits/Sec): {}\\n\",\n                static_cast<double>((_processed_bytes * 8)) / (1000 * 1000 * 1000) / secs);\n            (void)clients.stop().then([] {\n                engine().exit(0);\n            });\n        }\n    }\n\n    future<> start(ipv4_addr server_addr, std::string test, unsigned ncon) {\n        _server_addr = server_addr;\n        _concurrent_connections = ncon * smp::count;\n        _total_pings = _pings_per_connection * _concurrent_connections;\n        _test = test;\n\n        for (unsigned i = 0; i < ncon; i++) {\n            socket_address local = socket_address(::sockaddr_in{AF_INET, INADDR_ANY, {0}});\n            (void)connect(make_ipv4_address(server_addr), local, protocol).then([this, test] (connected_socket fd) {\n                auto conn = new connection(std::move(fd));\n                (void)(this->*tests.at(test))(conn).then_wrapped([conn] (auto&& f) {\n                    delete conn;\n                    try {\n                        f.get();\n                    } catch (std::exception& ex) {\n                        fmt::print(std::cerr, \"request error: {}\\n\", ex.what());\n                    }\n                });\n            });\n        }\n        return make_ready_future();\n    }\n    future<> stop() {\n        return make_ready_future();\n    }\n\n    typedef future<> (client::*test_fn)(connection *conn);\n    static const std::map<std::string, test_fn> tests;\n};\n\nnamespace bpo = boost::program_options;\n\nint main(int ac, char ** av) {\n    app_template app;\n    app.add_options()\n        (\"server\", bpo::value<std::string>()->required(), \"Server address\")\n        (\"test\", bpo::value<std::string>()->default_value(\"ping\"), \"test type(ping | rxrx | txtx)\")\n        (\"conn\", bpo::value<unsigned>()->default_value(16), \"nr connections per cpu\")\n        (\"proto\", bpo::value<std::string>()->default_value(\"tcp\"), \"transport protocol tcp|sctp\")\n        ;\n\n    return app.run_deprecated(ac, av, [&app] {\n        auto&& config = app.configuration();\n        auto server = config[\"server\"].as<std::string>();\n        auto test = config[\"test\"].as<std::string>();\n        auto ncon = config[\"conn\"].as<unsigned>();\n        auto proto = config[\"proto\"].as<std::string>();\n\n        if (proto == \"tcp\") {\n            protocol = transport::TCP;\n        } else if (proto == \"sctp\") {\n            protocol = transport::SCTP;\n        } else {\n            fmt::print(std::cerr, \"Error: --proto=tcp|sctp\\n\");\n            return engine().exit(1);\n        }\n\n        if (!client::tests.count(test)) {\n            fmt::print(std::cerr, \"Error: -test=ping | rxrx | txtx\\n\");\n            return engine().exit(1);\n        }\n\n        (void)clients.start().then([server, test, ncon] () {\n            return clients.invoke_on_all(&client::start, ipv4_addr{server}, test, ncon);\n        });\n    });\n}\n\nconst std::map<std::string, client::test_fn> client::tests = {\n        {\"ping\", &client::ping_test},\n        {\"rxrx\", &client::rxrx_test},\n        {\"txtx\", &client::txtx_test},\n};\n\n"
  },
  {
    "path": "demos/tcp_sctp_server_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/closeable.hh>\n#include <vector>\n#include <iostream>\n#include \"../apps/lib/stop_signal.hh\"\n\nusing namespace seastar;\n\nstatic std::string str_ping{\"ping\"};\nstatic std::string str_txtx{\"txtx\"};\nstatic std::string str_rxrx{\"rxrx\"};\nstatic std::string str_pong{\"pong\"};\nstatic std::string str_unknow{\"unknow cmd\"};\nstatic int tx_msg_total_size = 100 * 1024 * 1024;\nstatic int tx_msg_size = 4 * 1024;\nstatic int tx_msg_nr = tx_msg_total_size / tx_msg_size;\nstatic int rx_msg_size = 4 * 1024;\nstatic std::string str_txbuf(tx_msg_size, 'X');\nstatic bool enable_tcp = false;\nstatic bool enable_sctp = false;\n\nclass tcp_server {\n    std::vector<server_socket> _tcp_listeners;\n    std::vector<server_socket> _sctp_listeners;\n    std::optional<future<>> _tcp_task;\n    std::optional<future<>> _sctp_task;\n\npublic:\n    future<> listen(ipv4_addr addr) {\n        if (enable_tcp) {\n            listen_options lo;\n            lo.proto = transport::TCP;\n            lo.reuse_address = true;\n            _tcp_listeners.push_back(seastar::listen(make_ipv4_address(addr), lo));\n            _tcp_task = do_accepts(_tcp_listeners);\n        }\n\n        if (enable_sctp) {\n            listen_options lo;\n            lo.proto = transport::SCTP;\n            lo.reuse_address = true;\n            _sctp_listeners.push_back(seastar::listen(make_ipv4_address(addr), lo));\n            _sctp_task = do_accepts(_sctp_listeners);\n        }\n        return make_ready_future<>();\n    }\n\n    future<> stop() {\n        co_await do_stop(_tcp_listeners, _tcp_task);\n        co_await do_stop(_sctp_listeners, _sctp_task);\n    }\n\n    future<> do_accepts(std::vector<server_socket>& listeners) {\n        int which = listeners.size() - 1;\n        // Accept in the background.\n        return listeners[which].accept().then([this, &listeners] (accept_result ar) mutable {\n            connected_socket fd = std::move(ar.connection);\n            socket_address addr = std::move(ar.remote_address);\n            auto conn = new connection(*this, std::move(fd), addr);\n            (void)conn->process().then_wrapped([conn] (auto&& f) {\n                delete conn;\n                try {\n                    f.get();\n                } catch (std::exception& ex) {\n                    std::cout << \"request error \" << ex.what() << \"\\n\";\n                }\n            });\n            return do_accepts(listeners);\n        }).then_wrapped([] (auto&& f) {\n            try {\n                f.get();\n            } catch (std::exception& ex) {\n                std::cout << \"accept failed: \" << ex.what() << \"\\n\";\n            }\n        });\n    }\n\n    static future<> do_stop(std::vector<server_socket>& listeners, std::optional<future<>>& task) {\n        for (auto& listener : listeners) {\n            listener.abort_accept();\n        }\n        if (auto fut = std::exchange(task, {})) {\n            co_await std::move(*fut);\n        }\n    }\n\n    class connection {\n        connected_socket _fd;\n        input_stream<char> _read_buf;\n        output_stream<char> _write_buf;\n    public:\n        connection(tcp_server& server, connected_socket&& fd, socket_address addr)\n            : _fd(std::move(fd))\n            , _read_buf(_fd.input())\n            , _write_buf(_fd.output()) {}\n        future<> process() {\n             return read();\n        }\n        future<> read() {\n            if (_read_buf.eof()) {\n                return make_ready_future();\n            }\n            // Expect 4 bytes cmd from client\n            size_t n = 4;\n            return _read_buf.read_exactly(n).then([this] (temporary_buffer<char> buf) {\n                if (buf.size() == 0) {\n                    return make_ready_future();\n                }\n                auto cmd = std::string(buf.get(), buf.size());\n                // pingpong test\n                if (cmd == str_ping) {\n                    return _write_buf.write(str_pong).then([this] {\n                        return _write_buf.flush();\n                    }).then([this] {\n                        return this->read();\n                    });\n                // server tx test\n                } else if (cmd == str_txtx) {\n                    return tx_test();\n                // server tx test\n                } else if (cmd == str_rxrx) {\n                    return rx_test();\n                // unknow test\n                } else {\n                    return _write_buf.write(str_unknow).then([this] {\n                        return _write_buf.flush();\n                    }).then([] {\n                        return make_ready_future();\n                    });\n                }\n            });\n        }\n        future<> do_write(int end) {\n            if (end == 0) {\n                return make_ready_future<>();\n            }\n            return _write_buf.write(str_txbuf).then([this] {\n                return _write_buf.flush();\n            }).then([this, end] {\n                return do_write(end - 1);\n            });\n        }\n        future<> tx_test() {\n            return do_write(tx_msg_nr).then([this] {\n                return _write_buf.close();\n            }).then([] {\n                return make_ready_future<>();\n            });\n        }\n        future<> do_read() {\n            return _read_buf.read_exactly(rx_msg_size).then([this] (temporary_buffer<char> buf) {\n                if (buf.size() == 0) {\n                    return make_ready_future();\n                } else {\n                    return do_read();\n                }\n            });\n        }\n        future<> rx_test() {\n            return do_read().then([] {\n                return make_ready_future<>();\n            });\n        }\n    };\n};\n\nnamespace bpo = boost::program_options;\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n        (\"port\", bpo::value<uint16_t>()->default_value(10000), \"TCP server port\")\n        (\"tcp\", bpo::value<std::string>()->default_value(\"yes\"), \"tcp listen\")\n        (\"sctp\", bpo::value<std::string>()->default_value(\"no\"), \"sctp listen\") ;\n    return app.run(ac, av, [&] {\n        return async([&app] {\n            seastar_apps_lib::stop_signal stop_signal;\n\n            auto&& config = app.configuration();\n            uint16_t port = config[\"port\"].as<uint16_t>();\n            enable_tcp = config[\"tcp\"].as<std::string>() == \"yes\";\n            enable_sctp = config[\"sctp\"].as<std::string>() == \"yes\";\n            if (!enable_tcp && !enable_sctp) {\n                fmt::print(std::cerr, \"Error: no protocols enabled. Use \\\"--tcp yes\\\" and/or \\\"--sctp yes\\\" to enable\\n\");\n                return 1;\n            }\n            sharded<tcp_server> server;\n            server.start().get();\n            auto stop_server = deferred_stop(server);\n            // Start listening in the background.\n            server.invoke_on_all(&tcp_server::listen, ipv4_addr{port}).get();\n            fmt::print(\"Seastar TCP server listening on port {} ...\\n\", port);\n\n            stop_signal.wait().get();\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "demos/tls_echo_server.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 ScyllaDB\n */\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/util/log.hh>\n#include <iostream>\n\nusing namespace seastar;\n\nstruct streams {\n    connected_socket s;\n    input_stream<char> in;\n    output_stream<char> out;\n\n    streams(connected_socket cs) : s(std::move(cs)), in(s.input()), out(s.output())\n    {}\n};\n\nclass echoserver {\n    server_socket _socket;\n    shared_ptr<tls::server_credentials> _certs;\n    seastar::gate _gate;\n    bool _stopped = false;\n    bool _verbose = false;\npublic:\n    echoserver(bool verbose = false)\n            : _certs(make_shared<tls::server_credentials>(make_shared<tls::dh_params>()))\n            , _verbose(verbose)\n    {}\n\n    future<> listen(socket_address addr, sstring crtfile, sstring keyfile, tls::client_auth ca = tls::client_auth::NONE) {\n        _certs->set_client_auth(ca);\n        return _certs->set_x509_key_file(crtfile, keyfile, tls::x509_crt_format::PEM).then([this, addr] {\n            ::listen_options opts;\n            opts.reuse_address = true;\n\n            _socket = tls::listen(_certs, addr, opts);\n\n            // Listen in background.\n            (void)repeat([this] {\n                if (_stopped) {\n                    return make_ready_future<stop_iteration>(stop_iteration::yes);\n                }\n                return with_gate(_gate, [this] {\n                    return _socket.accept().then([this](accept_result ar) {\n                        ::connected_socket s = std::move(ar.connection);\n                        socket_address a = std::move(ar.remote_address);\n                        if (_verbose) {\n                            std::cout << \"Got connection from \"<< a << std::endl;\n                        }\n                        auto strms = make_lw_shared<streams>(std::move(s));\n                        return repeat([strms, this]() {\n                            return strms->in.read().then([this, strms](temporary_buffer<char> buf) {\n                                if (buf.empty()) {\n                                    if (_verbose) {\n                                        std::cout << \"EOM\" << std::endl;\n                                    }\n                                    return make_ready_future<stop_iteration>(stop_iteration::yes);\n                                }\n                                sstring tmp(buf.begin(), buf.end());\n                                if (_verbose) {\n                                    std::cout << \"Read \" << tmp.size() << \"B\" << std::endl;\n                                }\n                                return strms->out.write(tmp).then([strms]() {\n                                    return strms->out.flush();\n                                }).then([] {\n                                    return make_ready_future<stop_iteration>(stop_iteration::no);\n                                });\n                            });\n                        }).then([strms]{\n                            return strms->out.close();\n                        }).handle_exception([](auto ep) {\n                        }).finally([this, strms]{\n                            if (_verbose) {\n                                std::cout << \"Ending session\" << std::endl;\n                            }\n                            return strms->in.close();\n                        });\n                    }).handle_exception([this](auto ep) {\n                        if (!_stopped) {\n                            std::cerr << \"Error: \" << ep << std::endl;\n                        }\n                    }).then([this] {\n                        return make_ready_future<stop_iteration>(_stopped ? stop_iteration::yes : stop_iteration::no);\n                    });\n                });\n            });\n            return make_ready_future();\n        });\n    }\n\n    future<> stop() {\n        _stopped = true;\n        _socket.abort_accept();\n        return _gate.close();\n    }\n};\n"
  },
  {
    "path": "demos/tls_echo_server_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#include <cmath>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/util/closeable.hh>\n#include \"../apps/lib/stop_signal.hh\"\n#include \"tls_echo_server.hh\"\n\nusing namespace seastar;\nnamespace bpo = boost::program_options;\n\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n                    (\"port\", bpo::value<uint16_t>()->default_value(10000), \"Server port\")\n                    (\"address\", bpo::value<std::string>()->default_value(\"127.0.0.1\"), \"Server address\")\n                    (\"cert,c\", bpo::value<std::string>()->required(), \"Server certificate file\")\n                    (\"key,k\", bpo::value<std::string>()->required(), \"Certificate key\")\n                    (\"verbose,v\", bpo::value<bool>()->default_value(false)->implicit_value(true), \"Verbose\")\n                    ;\n    return app.run(ac, av, [&app] {\n        return async([&app] {\n            seastar_apps_lib::stop_signal stop_signal;\n            auto&& config = app.configuration();\n            uint16_t port = config[\"port\"].as<uint16_t>();\n            auto crt = config[\"cert\"].as<std::string>();\n            auto key = config[\"key\"].as<std::string>();\n            auto addr = config[\"address\"].as<std::string>();\n            auto verbose = config[\"verbose\"].as<bool>();\n\n            std::cout << \"Starting...\" << std::endl;\n            net::inet_address a = net::dns::resolve_name(addr).get();\n\n            ipv4_addr ia(a, port);\n\n            seastar::sharded<echoserver> server;\n            server.start(verbose).get();\n            auto stop_server = deferred_stop(server);\n\n            try {\n                server.invoke_on_all(&echoserver::listen, socket_address(ia), sstring(crt), sstring(key), tls::client_auth::NONE).get();\n            } catch (...) {\n                std::cerr << \"Error: \" << std::current_exception() << std::endl;\n                return 1;\n            }\n            std::cout << \"TLS echo server running at \" << addr << \":\" << port << std::endl;\n            stop_signal.wait().get();\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "demos/tls_simple_client_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#include <cmath>\n#include <ranges>\n\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/net/dns.hh>\n#include \"tls_echo_server.hh\"\n\nusing namespace seastar;\nnamespace bpo = boost::program_options;\n\n\nint main(int ac, char** av) {\n    app_template app;\n    app.add_options()\n                    (\"port\", bpo::value<uint16_t>()->default_value(10000), \"Remote port\")\n                    (\"address\", bpo::value<std::string>()->default_value(\"127.0.0.1\"), \"Remote address\")\n                    (\"trust,t\", bpo::value<std::string>(), \"Trust store\")\n                    (\"msg,m\", bpo::value<std::string>(), \"Message to send\")\n                    (\"bytes,b\", bpo::value<size_t>()->default_value(512), \"Use random bytes of length as message\")\n                    (\"iterations,i\", bpo::value<size_t>()->default_value(1), \"Repeat X times\")\n                    (\"read-response,r\", bpo::value<bool>()->default_value(true)->implicit_value(true), \"Read echoed message\")\n                    (\"verbose,v\", bpo::value<bool>()->default_value(false)->implicit_value(true), \"Verbose operation\")\n                    (\"check-name,c\", bpo::value<bool>()->default_value(false)->implicit_value(true), \"Check server name\")\n                    (\"server-name,s\", bpo::value<std::string>(), \"Expected server name\")\n                    ;\n\n\n    return app.run_deprecated(ac, av, [&] {\n        auto&& config = app.configuration();\n        uint16_t port = config[\"port\"].as<uint16_t>();\n        auto addr = config[\"address\"].as<std::string>();\n        auto n = config[\"bytes\"].as<size_t>();\n        auto i = config[\"iterations\"].as<size_t>();\n        auto do_read = config[\"read-response\"].as<bool>();\n        auto verbose = config[\"verbose\"].as<bool>();\n        auto check = config[\"check-name\"].as<bool>();\n\n        std::cout << \"Starting...\" << std::endl;\n\n        auto certs = ::make_shared<tls::certificate_credentials>();\n        auto f = make_ready_future();\n\n        if (config.count(\"trust\")) {\n            f = certs->set_x509_trust_file(config[\"trust\"].as<std::string>(), tls::x509_crt_format::PEM);\n        }\n\n        seastar::shared_ptr<sstring> msg;\n\n        if (config.count(\"msg\")) {\n            msg = seastar::make_shared<sstring>(config[\"msg\"].as<std::string>());\n        } else {\n            msg = seastar::make_shared<sstring>(uninitialized_string(n));\n            for (size_t i = 0; i < n; ++i) {\n                (*msg)[i] = '0' + char(::rand() % 30);\n            }\n        }\n\n        sstring server_name;\n        if (config.count(\"server-name\")) {\n            server_name = config[\"server-name\"].as<std::string>();\n        }\n        if (verbose) {\n            std::cout << \"Msg (\" << msg->size() << \"B):\" << std::endl << *msg << std::endl;\n        }\n        return f.then([=]() {\n            return net::dns::get_host_by_name(addr).then([=](net::hostent e) {\n                ipv4_addr ia(e.addr_entries.front().addr, port);\n\n                tls::tls_options options;\n                if (check) {\n                    options.server_name = server_name.empty() ? e.names.front() : server_name;\n                }\n                return tls::connect(certs, ia, options).then([=](::connected_socket s) {\n                    auto strms = ::make_lw_shared<streams>(std::move(s));\n                    auto range = std::views::iota(size_t(0), i);\n                    return do_for_each(range, [=](auto) {\n                        auto f = strms->out.write(*msg);\n                        if (!do_read) {\n                            return strms->out.close().then([f = std::move(f)]() mutable {\n                                return std::move(f);\n                            });\n                        }\n                        return f.then([=]() {\n                            return strms->out.flush().then([=] {\n                                return strms->in.read_exactly(msg->size()).then([=](temporary_buffer<char> buf) {\n                                    sstring tmp(buf.begin(), buf.end());\n                                    if (tmp != *msg) {\n                                        std::cerr << \"Got garbled message!\" << std::endl;\n                                        if (verbose) {\n                                            std::cout << \"Got (\" << tmp.size() << \") :\" << std::endl << tmp << std::endl;\n                                        }\n                                        throw std::runtime_error(\"Got garbled message!\");\n                                    }\n                                });\n                            });\n                        });\n                    }).then([strms, do_read]{\n                        return do_read ? strms->out.close() : make_ready_future<>();\n                    }).finally([strms]{\n                        return strms->in.close();\n                    });\n                });\n            }).handle_exception([](auto ep) {\n                std::cerr << \"Error: \" << ep << std::endl;\n            });\n        }).finally([] {\n            engine().exit(0);\n        });\n    });\n}\n"
  },
  {
    "path": "demos/tutorial_examples.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <iostream>\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/net/api.hh>\n\nseastar::future<> service_loop() {\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234})),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                [] (seastar::accept_result res) {\n                    std::cout << \"Accepted connection from \" << res.remote_address << \"\\n\";\n            });\n        });\n    });\n}\n\nconst char* canned_response = \"Seastar is the future!\\n\";\n\nseastar::future<> service_loop_2() {\n    seastar::listen_options lo;\n    lo.reuse_address = true;\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                    [] (seastar::accept_result res) {\n                auto s = std::move(res.connection);\n                auto out = s.output();\n                return seastar::do_with(std::move(s), std::move(out),\n                        [] (auto& s, auto& out) {\n                    return out.write(canned_response).then([&out] {\n                        return out.close();\n                    });\n                });\n            });\n        });\n    });\n}\n\nseastar::future<> handle_connection_3(seastar::connected_socket s,\n                                    seastar::socket_address a) {\n    auto out = s.output();\n    auto in = s.input();\n    return do_with(std::move(s), std::move(out), std::move(in),\n            [] (auto& s, auto& out, auto& in) {\n        return seastar::repeat([&out, &in] {\n            return in.read().then([&out] (auto buf) {\n                if (buf) {\n                    return out.write(std::move(buf)).then([&out] {\n                        return out.flush();\n                    }).then([] {\n                        return seastar::stop_iteration::no;\n                    });\n                } else {\n                    return seastar::make_ready_future<seastar::stop_iteration>(\n                            seastar::stop_iteration::yes);\n                }\n            });\n        }).then([&out] {\n            return out.close();\n        });\n    });\n}\n\nseastar::future<> service_loop_3() {\n    seastar::listen_options lo;\n    lo.reuse_address = true;\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                    [] (seastar::accept_result res) {\n                // Note we ignore, not return, the future returned by\n                // handle_connection(), so we do not wait for one\n                // connection to be handled before accepting the next one.\n                (void)handle_connection_3(std::move(res.connection), std::move(res.remote_address)).handle_exception(\n                        [] (std::exception_ptr ep) {\n                    fmt::print(stderr, \"Could not handle connection: {}\\n\", ep);\n                });\n            });\n        });\n    });\n}\n\n#include <seastar/core/app-template.hh>\n\nint main(int ac, char** av) {\n    seastar::app_template app;\n    return app.run(ac, av, [] {\n        std::cout << \"This is the tutorial examples demo.  It is not running anything but rather makes sure the tutorial examples compile\" << std::endl;\n        return seastar::make_ready_future<>();\n    });\n}\n"
  },
  {
    "path": "demos/udp_client_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/net/api.hh>\n#include <iostream>\n\nusing namespace seastar;\nusing namespace net;\nusing namespace std::chrono_literals;\n\nclass client {\nprivate:\n    udp_channel _chan;\n    uint64_t n_sent {};\n    uint64_t n_received {};\n    uint64_t n_failed {};\n    timer<> _stats_timer;\npublic:\n    void start(ipv4_addr server_addr) {\n        std::cout << \"Sending to \" << server_addr << std::endl;\n\n        _chan = make_unbound_datagram_channel(AF_INET);\n\n        _stats_timer.set_callback([this] {\n            std::cout << \"Out: \" << n_sent << \" pps, \\t\";\n            std::cout << \"Err: \" << n_failed << \" pps, \\t\";\n            std::cout << \"In: \" << n_received << \" pps\" << std::endl;\n            n_sent = 0;\n            n_received = 0;\n            n_failed = 0;\n        });\n        _stats_timer.arm_periodic(1s);\n\n        // Run sender in background.\n        (void)keep_doing([this, server_addr] {\n            return _chan.send(server_addr, \"hello!\\n\")\n                .then_wrapped([this] (auto&& f) {\n                    try {\n                        f.get();\n                        n_sent++;\n                    } catch (...) {\n                        n_failed++;\n                    }\n                });\n        });\n\n        // Run receiver in background.\n        (void)keep_doing([this] {\n            return _chan.receive().then([this] (auto) {\n                n_received++;\n            });\n        });\n    }\n};\n\nnamespace bpo = boost::program_options;\n\nint main(int ac, char ** av) {\n    client _client;\n    app_template app;\n    app.add_options()\n        (\"server\", bpo::value<std::string>(), \"Server address\")\n        ;\n    return app.run_deprecated(ac, av, [&_client, &app] {\n        auto&& config = app.configuration();\n        _client.start(config[\"server\"].as<std::string>());\n    });\n}\n"
  },
  {
    "path": "demos/udp_server_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <iostream>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/closeable.hh>\n#include \"../apps/lib/stop_signal.hh\"\n\nusing namespace seastar;\nusing namespace net;\nusing namespace std::chrono_literals;\n\nclass udp_server {\nprivate:\n    std::optional<udp_channel> _chan;\n    std::optional<future<>> _task;\n    timer<> _stats_timer;\n    uint64_t _n_sent {};\npublic:\n    void start(uint16_t port) {\n        ipv4_addr listen_addr{port};\n        _chan = make_bound_datagram_channel(listen_addr);\n\n        _stats_timer.set_callback([this] {\n            std::cout << \"Out: \" << _n_sent << \" pps\" << std::endl;\n            _n_sent = 0;\n        });\n        _stats_timer.arm_periodic(1s);\n\n        // Run server in background.\n        _task = keep_doing([this] {\n            return _chan->receive().then([this] (datagram dgram) {\n                auto bufs = dgram.get_buffers();\n                // send() grabs buffers immediately, no need to keep dgram alive for longer\n                return _chan->send(dgram.get_src(), bufs).then([this] {\n                    _n_sent++;\n                });\n            });\n        });\n    }\n    future<> stop() {\n        if (_chan) {\n            _chan->shutdown_input();\n            _chan->shutdown_output();\n        }\n        if (_task) {\n            co_await _task->handle_exception([](std::exception_ptr e) {\n                std::cerr << \"exception in udp_server: \" << e << \"\\n\";\n            });\n        }\n    }\n};\n\nnamespace bpo = boost::program_options;\n\nint main(int ac, char ** av) {\n    app_template app;\n    app.add_options()\n        (\"port\", bpo::value<uint16_t>()->default_value(10000), \"UDP server port\") ;\n   return app.run(ac, av, [&] {\n        return async([&app] {\n            seastar_apps_lib::stop_signal stop_signal;\n            auto&& config = app.configuration();\n            uint16_t port = config[\"port\"].as<uint16_t>();\n            sharded<udp_server> server;\n            if (engine().net().has_per_core_namespace()) {\n                server.start().get();\n            } else {\n                server.start_single().get();\n            }\n            auto stop_server = deferred_stop(server);\n            server.invoke_on_all(&udp_server::start, port).get();\n            std::cout << \"Seastar UDP server listening on port \" << port << \" ...\\n\";\n\n            stop_signal.wait().get();\n        });\n    });\n}\n"
  },
  {
    "path": "demos/udp_zero_copy_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/units.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/net/api.hh>\n#include <seastar/util/assert.hh>\n#include <random>\n#include <iomanip>\n#include <iostream>\n\nusing namespace seastar;\nusing namespace net;\nusing namespace std::chrono_literals;\nnamespace bpo = boost::program_options;\n\ntemplate <typename Duration>\ntypename Duration::rep to_seconds(Duration d) {\n    return std::chrono::duration_cast<std::chrono::seconds>(d).count();\n}\n\nclass server {\nprivate:\n    udp_channel _chan;\n    timer<> _stats_timer;\n    uint64_t _n_sent {};\n    size_t _chunk_size;\n    bool _copy;\n    steady_clock_type::time_point _last;\n    sstring _key;\n    size_t _packet_size = 8*KB;\n    char* _mem;\n    size_t _mem_size;\n    std::mt19937 _rnd;\n    std::random_device _randem_dev;\n    std::uniform_int_distribution<size_t> _chunk_distribution;\nprivate:\n    char* next_chunk() {\n        return _mem + _chunk_distribution(_rnd);\n    }\npublic:\n    server()\n        : _rnd(std::random_device()()) {\n    }\n    future<> send(ipv4_addr dst, std::vector<temporary_buffer<char>> bufs) {\n        return _chan.send(dst, bufs).then([this] {\n            _n_sent++;\n        });\n    }\n    void start(int chunk_size, bool copy, size_t mem_size) {\n        ipv4_addr listen_addr{10000};\n        _chan = make_bound_datagram_channel(listen_addr);\n\n        std::cout << \"Listening on \" << listen_addr << std::endl;\n\n        _last = steady_clock_type::now();\n        _stats_timer.set_callback([this] {\n            auto now = steady_clock_type::now();\n            std::cout << \"Out: \"\n                << std::setprecision(2) << std::fixed\n                << (double)_n_sent / to_seconds(now - _last)\n                << \" pps\" << std::endl;\n            _last = now;\n            _n_sent = 0;\n        });\n        _stats_timer.arm_periodic(1s);\n\n        _chunk_size = chunk_size;\n        _copy = copy;\n        _key = sstring(new char[64], 64);\n\n        _mem = new char[mem_size];\n        _mem_size = mem_size;\n\n        _chunk_distribution = std::uniform_int_distribution<size_t>(0, _mem_size - _chunk_size * 3);\n\n        SEASTAR_ASSERT(3 * _chunk_size <= _packet_size);\n\n        // Run sender in background.\n        (void)keep_doing([this] {\n            return _chan.receive().then([this] (datagram dgram) {\n                auto chunk = next_chunk();\n                std::vector<temporary_buffer<char>> bufs;\n                bufs.reserve(3);\n                for (unsigned int i = 0; i < bufs.size(); i++) {\n                    if (_copy) {\n                        bufs.emplace_back(temporary_buffer<char>(chunk, _chunk_size));\n                    } else {\n                        bufs.emplace_back(temporary_buffer<char>(chunk, _chunk_size, deleter()));\n                    }\n                    chunk += _chunk_size;\n                }\n                return send(dgram.get_src(), std::move(bufs));\n            });\n        });\n    }\n};\n\nint main(int ac, char ** av) {\n    server s;\n    app_template app;\n    app.add_options()\n        (\"chunk-size\", bpo::value<int>()->default_value(1024),\n             \"Chunk size\")\n        (\"mem-size\", bpo::value<int>()->default_value(512),\n             \"Memory pool size in MiB\")\n        (\"copy\", \"Copy data rather than send via zero-copy\")\n        ;\n    return app.run_deprecated(ac, av, [&app, &s] {\n        auto&& config = app.configuration();\n        auto chunk_size = config[\"chunk-size\"].as<int>();\n        auto mem_size = (size_t)config[\"mem-size\"].as<int>() * MB;\n        auto copy = config.count(\"copy\");\n        s.start(chunk_size, copy, mem_size);\n    });\n}\n"
  },
  {
    "path": "demos/websocket_client_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <seastar/websocket/client.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/log.hh>\n#include <string_view>\n\nusing namespace seastar;\nusing namespace seastar::experimental;\n\nnamespace bpo = boost::program_options;\n\nstatic logger ws_demo_logger(\"ws_demo\");\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.add_options()\n        (\"host\", bpo::value<std::string>()->default_value(\"fstream.binance.com\"),\n            \"WebSocket server host\")\n        (\"path\", bpo::value<std::string>()->default_value(\"/ws/btcusdt@trade\"),\n            \"WebSocket resource path\")\n        (\"port\", bpo::value<uint16_t>()->default_value(443),\n            \"WebSocket server port\")\n        (\"duration\", bpo::value<unsigned>()->default_value(30),\n            \"Duration in seconds to receive data\")\n        (\"tls\", bpo::value<bool>()->default_value(true),\n            \"Use TLS (wss://)\");\n    return app.run(argc, argv, [&app]() -> seastar::future<> {\n        auto&& config = app.configuration();\n        auto host = config[\"host\"].as<std::string>();\n        auto path = config[\"path\"].as<std::string>();\n        auto port = config[\"port\"].as<uint16_t>();\n        auto duration = config[\"duration\"].as<unsigned>();\n        auto use_tls = config[\"tls\"].as<bool>();\n\n        ws_demo_logger.info(\"Resolving {}...\", host);\n        net::hostent e = co_await net::dns::get_host_by_name(host,\n            net::inet_address::family::INET);\n        auto addr = socket_address(e.addr_entries.front().addr, port);\n        ws_demo_logger.info(\"Connecting to {}:{}{} ...\", host, port, path);\n\n        websocket::client ws_client;\n\n        if (use_tls) {\n            auto creds = make_shared<tls::certificate_credentials>();\n            co_await creds->set_system_trust();\n            co_await ws_client.connect(addr, std::move(creds),\n                sstring(path), sstring(host), \"\",\n                [] (input_stream<char>& in, output_stream<char>& out) -> future<> {\n                    while (true) {\n                        auto f = co_await in.read();\n                        if (f.empty()) {\n                            break;\n                        }\n\n                        std::cerr << \"receive: \" << std::string_view(f.get(), f.size()) << \"\\n\";\n                    }\n                });\n        } else {\n            co_await ws_client.connect(addr,\n                sstring(path), sstring(host), \"\",\n                [] (input_stream<char>& in, output_stream<char>& out) -> future<> {\n                    while (true) {\n                        auto f = co_await in.read();\n                        if (f.empty()) {\n                            break;\n                        }\n\n                        std::cerr << \"receive: \" << std::string_view(f.get(), f.size()) << \"\\n\";\n                    }\n\n                });\n        }\n\n        ws_demo_logger.info(\"Connected! Receiving for {} seconds...\",\n            duration);\n        co_await seastar::sleep_abortable(std::chrono::seconds(duration));\n        ws_demo_logger.info(\"Done, closing connection.\");\n        co_await ws_client.close();\n    });\n}\n"
  },
  {
    "path": "demos/websocket_server_demo.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 ScyllaDB Ltd.\n */\n\n#include <iostream>\n#include <seastar/websocket/server.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/defer.hh>\n\nusing namespace seastar;\nusing namespace seastar::experimental;\n\nnamespace bpo = boost::program_options;\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.add_options()\n        (\"port\", bpo::value<uint16_t>()->default_value(10000), \"WebSocket server port\") ;\n    app.run(argc, argv, [&app]() -> seastar::future<> {\n        auto&& config = app.configuration();\n        uint16_t port = config[\"port\"].as<uint16_t>();\n\n        return async([port] {\n            websocket::server ws;\n            ws.register_handler(\"echo\", [] (input_stream<char>& in,\n                        output_stream<char>& out) {\n                return repeat([&in, &out]() {\n                    return in.read().then([&out](temporary_buffer<char> f) {\n                        std::cerr << \"f.size(): \" << f.size() << \"\\n\";\n                        if (f.empty()) {\n                            return make_ready_future<stop_iteration>(stop_iteration::yes);\n                        } else {\n                            return out.write(std::move(f)).then([&out]() {\n                                return out.flush().then([] {\n                                    return make_ready_future<stop_iteration>(stop_iteration::no);\n                                });\n                            });\n                        }\n                    });\n                });\n            });\n            auto d = defer([&ws] () noexcept {\n                ws.stop().get();\n            });\n            ws.listen(socket_address(ipv4_addr(\"127.0.0.1\", port)));\n            std::cout << \"Listening on 127.0.0.1:\" << port << \" for 1 hour (interruptible, hit Ctrl-C to stop)...\" << std::endl;\n            seastar::sleep_abortable(std::chrono::hours(1)).handle_exception([](auto ignored) {}).get();\n            std::cout << \"Stopping the server, deepest thanks to all clients, hope we meet again\" << std::endl;\n        });\n    });\n}\n"
  },
  {
    "path": "doc/CMakeLists.txt",
    "content": "find_program (Seastar_DOXYGEN_EXECUTABLE doxygen)\nif (NOT Seastar_DOXYGEN_EXECUTABLE)\n  message (FATAL_ERROR \"doxygen is required for building document!\")\nendif ()\n\nconfigure_file (\n  ${CMAKE_CURRENT_SOURCE_DIR}/Doxyfile.in\n  ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile\n  @ONLY)\n\nconfigure_file (\n  ${CMAKE_CURRENT_SOURCE_DIR}/DoxygenLayout.xml\n  ${CMAKE_CURRENT_BINARY_DIR}/DoxygenLayout.xml\n  COPYONLY)\n\nadd_custom_target (doc_api\n  COMMAND ${Seastar_DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile)\n\nadd_custom_command (\n  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html\n  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md\n  COMMAND\n    ${CMAKE_CURRENT_SOURCE_DIR}/md2html\n    ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md\n    ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html)\n\nadd_custom_target (doc_tutorial_html\n  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html)\n\nadd_custom_command (\n  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/html/split\n  DEPENDS\n    # Necessary because file-level dependencies are not propagated for custom targets.\n    ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html\n    doc_tutorial_html\n  COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/html/split\n  COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/htmlsplit.py\n    --input ${CMAKE_CURRENT_BINARY_DIR}/html/tutorial.html\n    --output-dir ${CMAKE_CURRENT_BINARY_DIR}/html/split)\n\nadd_custom_target (doc_tutorial_html_split\n  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/html/split)\n\nadd_custom_command (\n  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf\n  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md\n  COMMAND\n    ${CMAKE_CURRENT_SOURCE_DIR}/md2pdf\n    ${CMAKE_CURRENT_SOURCE_DIR}/tutorial.md\n    ${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf)\n\nadd_custom_target (doc_tutorial_pdf\n  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/tutorial.pdf)\n\n# Logical target for all documentation.\nadd_custom_target (docs\n  DEPENDS\n    doc_api\n    doc_tutorial_html\n    doc_tutorial_html_split\n    doc_tutorial_pdf)\n"
  },
  {
    "path": "doc/Doxyfile.in",
    "content": "# Doxyfile 1.8.9.1\n\n# This file describes the settings to be used by the documentation system\n# doxygen (www.doxygen.org) for a project.\n#\n# All text after a double hash (##) is considered a comment and is placed in\n# front of the TAG it is preceding.\n#\n# All text after a single hash (#) is considered a comment and will be ignored.\n# The format is:\n# TAG = value [value, ...]\n# For lists, items can also be appended using:\n# TAG += value [value, ...]\n# Values that contain spaces should be placed between quotes (\\\" \\\").\n\n#---------------------------------------------------------------------------\n# Project related configuration options\n#---------------------------------------------------------------------------\n\n# This tag specifies the encoding used for all characters in the config file\n# that follow. The default is UTF-8 which is also the encoding used for all text\n# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv\n# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv\n# for the list of possible encodings.\n# The default value is: UTF-8.\n\nDOXYFILE_ENCODING      = UTF-8\n\n# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by\n# double-quotes, unless you are using Doxywizard) that should identify the\n# project for which the documentation is generated. This name is used in the\n# title of most generated pages and in a few other places.\n# The default value is: My Project.\n\nPROJECT_NAME           = \"Seastar\"\n\n# The PROJECT_NUMBER tag can be used to enter a project or revision number. This\n# could be handy for archiving the generated documentation or if some version\n# control system is used.\n\nPROJECT_NUMBER         =\n\n# Using the PROJECT_BRIEF tag one can provide an optional one line description\n# for a project that appears at the top of each page and should give viewer a\n# quick idea about the purpose of the project. Keep the description short.\n\nPROJECT_BRIEF          = \"High performance C++ framework for concurrent servers\"\n\n# With the PROJECT_LOGO tag one can specify a logo or an icon that is included\n# in the documentation. The maximum height of the logo should not exceed 55\n# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy\n# the logo to the output directory.\n\nPROJECT_LOGO           =\n\n# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path\n# into which the generated documentation will be written. If a relative path is\n# entered, it will be relative to the location where doxygen was started. If\n# left blank the current directory will be used.\n\nOUTPUT_DIRECTORY       = @CMAKE_CURRENT_BINARY_DIR@\n\n# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub-\n# directories (in 2 levels) under the output directory of each output format and\n# will distribute the generated files over these directories. Enabling this\n# option can be useful when feeding doxygen a huge amount of source files, where\n# putting all generated files in the same directory would otherwise causes\n# performance problems for the file system.\n# The default value is: NO.\n\nCREATE_SUBDIRS         = NO\n\n# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII\n# characters to appear in the names of generated files. If set to NO, non-ASCII\n# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode\n# U+3044.\n# The default value is: NO.\n\nALLOW_UNICODE_NAMES    = YES\n\n# The OUTPUT_LANGUAGE tag is used to specify the language in which all\n# documentation generated by doxygen is written. Doxygen will use this\n# information to generate all constant output in the proper language.\n# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese,\n# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States),\n# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian,\n# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages),\n# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian,\n# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian,\n# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish,\n# Ukrainian and Vietnamese.\n# The default value is: English.\n\nOUTPUT_LANGUAGE        = English\n\n# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member\n# descriptions after the members that are listed in the file and class\n# documentation (similar to Javadoc). Set to NO to disable this.\n# The default value is: YES.\n\nBRIEF_MEMBER_DESC      = YES\n\n# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief\n# description of a member or function before the detailed description\n#\n# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the\n# brief descriptions will be completely suppressed.\n# The default value is: YES.\n\nREPEAT_BRIEF           = YES\n\n# This tag implements a quasi-intelligent brief description abbreviator that is\n# used to form the text in various listings. Each string in this list, if found\n# as the leading text of the brief description, will be stripped from the text\n# and the result, after processing the whole list, is used as the annotated\n# text. Otherwise, the brief description is used as-is. If left blank, the\n# following values are used ($name is automatically replaced with the name of\n# the entity):The $name class, The $name widget, The $name file, is, provides,\n# specifies, contains, represents, a, an and the.\n\nABBREVIATE_BRIEF       =\n\n# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then\n# doxygen will generate a detailed section even if there is only a brief\n# description.\n# The default value is: NO.\n\nALWAYS_DETAILED_SEC    = NO\n\n# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all\n# inherited members of a class in the documentation of that class as if those\n# members were ordinary class members. Constructors, destructors and assignment\n# operators of the base classes will not be shown.\n# The default value is: NO.\n\nINLINE_INHERITED_MEMB  = YES\n\n# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path\n# before files name in the file list and in the header files. If set to NO the\n# shortest path that makes the file name unique will be used\n# The default value is: YES.\n\nFULL_PATH_NAMES        = YES\n\n# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path.\n# Stripping is only done if one of the specified strings matches the left-hand\n# part of the path. The tag can be used to show relative paths in the file list.\n# If left blank the directory from which doxygen is run is used as the path to\n# strip.\n#\n# Note that you can specify absolute paths here, but also relative paths, which\n# will be relative from the directory where doxygen is started.\n# This tag requires that the tag FULL_PATH_NAMES is set to YES.\n\nSTRIP_FROM_PATH        = @Seastar_SOURCE_DIR@/include @Seastar_BINARY_DIR@/gen/include @Seastar_SOURCE_DIR@/doc\n\n# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the\n# path mentioned in the documentation of a class, which tells the reader which\n# header file to include in order to use a class. If left blank only the name of\n# the header file containing the class definition is used. Otherwise one should\n# specify the list of include paths that are normally passed to the compiler\n# using the -I flag.\n\nSTRIP_FROM_INC_PATH    = @Seastar_SOURCE_DIR@/include @Seastar_BINARY_DIR@/gen/include\n\n# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but\n# less readable) file names. This can be useful is your file systems doesn't\n# support long names like on DOS, Mac, or CD-ROM.\n# The default value is: NO.\n\nSHORT_NAMES            = NO\n\n# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the\n# first line (until the first dot) of a Javadoc-style comment as the brief\n# description. If set to NO, the Javadoc-style will behave just like regular Qt-\n# style comments (thus requiring an explicit @brief command for a brief\n# description.)\n# The default value is: NO.\n\nJAVADOC_AUTOBRIEF      = NO\n\n# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first\n# line (until the first dot) of a Qt-style comment as the brief description. If\n# set to NO, the Qt-style will behave just like regular Qt-style comments (thus\n# requiring an explicit \\brief command for a brief description.)\n# The default value is: NO.\n\nQT_AUTOBRIEF           = NO\n\n# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a\n# multi-line C++ special comment block (i.e. a block of //! or /// comments) as\n# a brief description. This used to be the default behavior. The new default is\n# to treat a multi-line C++ comment block as a detailed description. Set this\n# tag to YES if you prefer the old behavior instead.\n#\n# Note that setting this tag to YES also means that rational rose comments are\n# not recognized any more.\n# The default value is: NO.\n\nMULTILINE_CPP_IS_BRIEF = NO\n\n# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the\n# documentation from any documented member that it re-implements.\n# The default value is: YES.\n\nINHERIT_DOCS           = YES\n\n# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new\n# page for each member. If set to NO, the documentation of a member will be part\n# of the file/class/namespace that contains it.\n# The default value is: NO.\n\nSEPARATE_MEMBER_PAGES  = NO\n\n# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen\n# uses this value to replace tabs by spaces in code fragments.\n# Minimum value: 1, maximum value: 16, default value: 4.\n\nTAB_SIZE               = 8\n\n# This tag can be used to specify a number of aliases that act as commands in\n# the documentation. An alias has the form:\n# name=value\n# For example adding\n# \"sideeffect=@par Side Effects:\\n\"\n# will allow you to put the command \\sideeffect (or @sideeffect) in the\n# documentation, which will result in a user-defined paragraph with heading\n# \"Side Effects:\". You can put \\n's in the value part of an alias to insert\n# newlines.\n\nALIASES                =\n\n# This tag can be used to specify a number of word-keyword mappings (TCL only).\n# A mapping has the form \"name=value\". For example adding \"class=itcl::class\"\n# will allow you to use the command class in the itcl::class meaning.\n\nTCL_SUBST              =\n\n# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources\n# only. Doxygen will then generate output that is more tailored for C. For\n# instance, some of the names that are used will be different. The list of all\n# members will be omitted, etc.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_FOR_C  = NO\n\n# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or\n# Python sources only. Doxygen will then generate output that is more tailored\n# for that language. For instance, namespaces will be presented as packages,\n# qualified scopes will look different, etc.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_JAVA   = NO\n\n# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran\n# sources. Doxygen will then generate output that is tailored for Fortran.\n# The default value is: NO.\n\nOPTIMIZE_FOR_FORTRAN   = NO\n\n# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL\n# sources. Doxygen will then generate output that is tailored for VHDL.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_VHDL   = NO\n\n# Doxygen selects the parser to use depending on the extension of the files it\n# parses. With this tag you can assign which parser to use for a given\n# extension. Doxygen has a built-in mapping, but you can override or extend it\n# using this tag. The format is ext=language, where ext is a file extension, and\n# language is one of the parsers supported by doxygen: IDL, Java, Javascript,\n# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran:\n# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran:\n# Fortran. In the later case the parser tries to guess whether the code is fixed\n# or free formatted code, this is the default for Fortran type files), VHDL. For\n# instance to make doxygen treat .inc files as Fortran files (default is PHP),\n# and .f files as C (default is Fortran), use: inc=Fortran f=C.\n#\n# Note: For files without extension you can use no_extension as a placeholder.\n#\n# Note that for custom extensions you also need to set FILE_PATTERNS otherwise\n# the files are not read by doxygen.\n\nEXTENSION_MAPPING      =\n\n# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments\n# according to the Markdown format, which allows for more readable\n# documentation. See http://daringfireball.net/projects/markdown/ for details.\n# The output of markdown processing is further processed by doxygen, so you can\n# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in\n# case of backward compatibilities issues.\n# The default value is: YES.\n\nMARKDOWN_SUPPORT       = YES\n\n# When enabled doxygen tries to link words that correspond to documented\n# classes, or namespaces to their corresponding documentation. Such a link can\n# be prevented in individual cases by putting a % sign in front of the word or\n# globally by setting AUTOLINK_SUPPORT to NO.\n# The default value is: YES.\n\nAUTOLINK_SUPPORT       = YES\n\n# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want\n# to include (a tag file for) the STL sources as input, then you should set this\n# tag to YES in order to let doxygen match functions declarations and\n# definitions whose arguments contain STL classes (e.g. func(std::string);\n# versus func(std::string) {}). This also make the inheritance and collaboration\n# diagrams that involve STL classes more complete and accurate.\n# The default value is: NO.\n\nBUILTIN_STL_SUPPORT    = YES\n\n# If you use Microsoft's C++/CLI language, you should set this option to YES to\n# enable parsing support.\n# The default value is: NO.\n\nCPP_CLI_SUPPORT        = NO\n\n# Set the SIP_SUPPORT tag to YES if your project consists of sip (see:\n# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen\n# will parse them like normal C++ but will assume all classes use public instead\n# of private inheritance when no explicit protection keyword is present.\n# The default value is: NO.\n\nSIP_SUPPORT            = NO\n\n# For Microsoft's IDL there are propget and propput attributes to indicate\n# getter and setter methods for a property. Setting this option to YES will make\n# doxygen to replace the get and set methods by a property in the documentation.\n# This will only work if the methods are indeed getting or setting a simple\n# type. If this is not the case, or you want to show the methods anyway, you\n# should set this option to NO.\n# The default value is: YES.\n\nIDL_PROPERTY_SUPPORT   = YES\n\n# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC\n# tag is set to YES then doxygen will reuse the documentation of the first\n# member in the group (if any) for the other members of the group. By default\n# all members of a group must be documented explicitly.\n# The default value is: NO.\n\nDISTRIBUTE_GROUP_DOC   = NO\n\n# Set the SUBGROUPING tag to YES to allow class member groups of the same type\n# (for instance a group of public functions) to be put as a subgroup of that\n# type (e.g. under the Public Functions section). Set it to NO to prevent\n# subgrouping. Alternatively, this can be done per class using the\n# \\nosubgrouping command.\n# The default value is: YES.\n\nSUBGROUPING            = YES\n\n# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions\n# are shown inside the group in which they are included (e.g. using \\ingroup)\n# instead of on a separate page (for HTML and Man pages) or section (for LaTeX\n# and RTF).\n#\n# Note that this feature does not work in combination with\n# SEPARATE_MEMBER_PAGES.\n# The default value is: NO.\n\nINLINE_GROUPED_CLASSES = NO\n\n# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions\n# with only public data fields or simple typedef fields will be shown inline in\n# the documentation of the scope in which they are defined (i.e. file,\n# namespace, or group documentation), provided this scope is documented. If set\n# to NO, structs, classes, and unions are shown on a separate page (for HTML and\n# Man pages) or section (for LaTeX and RTF).\n# The default value is: NO.\n\nINLINE_SIMPLE_STRUCTS  = YES\n\n# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or\n# enum is documented as struct, union, or enum with the name of the typedef. So\n# typedef struct TypeS {} TypeT, will appear in the documentation as a struct\n# with name TypeT. When disabled the typedef will appear as a member of a file,\n# namespace, or class. And the struct will be named TypeS. This can typically be\n# useful for C code in case the coding convention dictates that all compound\n# types are typedef'ed and only the typedef is referenced, never the tag name.\n# The default value is: NO.\n\nTYPEDEF_HIDES_STRUCT   = NO\n\n# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This\n# cache is used to resolve symbols given their name and scope. Since this can be\n# an expensive process and often the same symbol appears multiple times in the\n# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small\n# doxygen will become slower. If the cache is too large, memory is wasted. The\n# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range\n# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536\n# symbols. At the end of a run doxygen will report the cache usage and suggest\n# the optimal cache size from a speed point of view.\n# Minimum value: 0, maximum value: 9, default value: 0.\n\nLOOKUP_CACHE_SIZE      = 0\n\n#---------------------------------------------------------------------------\n# Build related configuration options\n#---------------------------------------------------------------------------\n\n# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in\n# documentation are documented, even if no documentation was available. Private\n# class members and static file members will be hidden unless the\n# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES.\n# Note: This will also disable the warnings about undocumented members that are\n# normally produced when WARNINGS is set to YES.\n# The default value is: NO.\n\nEXTRACT_ALL            = NO\n\n# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will\n# be included in the documentation.\n# The default value is: NO.\n\nEXTRACT_PRIVATE        = NO\n\n# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal\n# scope will be included in the documentation.\n# The default value is: NO.\n\nEXTRACT_PACKAGE        = NO\n\n# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be\n# included in the documentation.\n# The default value is: NO.\n\nEXTRACT_STATIC         = YES\n\n# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined\n# locally in source files will be included in the documentation. If set to NO,\n# only classes defined in header files are included. Does not have any effect\n# for Java sources.\n# The default value is: YES.\n\nEXTRACT_LOCAL_CLASSES  = NO\n\n# This flag is only useful for Objective-C code. If set to YES, local methods,\n# which are defined in the implementation section but not in the interface are\n# included in the documentation. If set to NO, only methods in the interface are\n# included.\n# The default value is: NO.\n\nEXTRACT_LOCAL_METHODS  = NO\n\n# If this flag is set to YES, the members of anonymous namespaces will be\n# extracted and appear in the documentation as a namespace called\n# 'anonymous_namespace{file}', where file will be replaced with the base name of\n# the file that contains the anonymous namespace. By default anonymous namespace\n# are hidden.\n# The default value is: NO.\n\nEXTRACT_ANON_NSPACES   = NO\n\n# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all\n# undocumented members inside documented classes or files. If set to NO these\n# members will be included in the various overviews, but no documentation\n# section is generated. This option has no effect if EXTRACT_ALL is enabled.\n# The default value is: NO.\n\nHIDE_UNDOC_MEMBERS     = NO\n\n# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all\n# undocumented classes that are normally visible in the class hierarchy. If set\n# to NO, these classes will be included in the various overviews. This option\n# has no effect if EXTRACT_ALL is enabled.\n# The default value is: NO.\n\nHIDE_UNDOC_CLASSES     = NO\n\n# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend\n# (class|struct|union) declarations. If set to NO, these declarations will be\n# included in the documentation.\n# The default value is: NO.\n\nHIDE_FRIEND_COMPOUNDS  = YES\n\n# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any\n# documentation blocks found inside the body of a function. If set to NO, these\n# blocks will be appended to the function's detailed documentation block.\n# The default value is: NO.\n\nHIDE_IN_BODY_DOCS      = NO\n\n# The INTERNAL_DOCS tag determines if documentation that is typed after a\n# \\internal command is included. If the tag is set to NO then the documentation\n# will be excluded. Set it to YES to include the internal documentation.\n# The default value is: NO.\n\nINTERNAL_DOCS          = NO\n\n# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file\n# names in lower-case letters. If set to YES, upper-case letters are also\n# allowed. This is useful if you have classes or files whose names only differ\n# in case and if your file system supports case sensitive file names. Windows\n# and Mac users are advised to set this option to NO.\n# The default value is: system dependent.\n\nCASE_SENSE_NAMES       = YES\n\n# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with\n# their full class and namespace scopes in the documentation. If set to YES, the\n# scope will be hidden.\n# The default value is: NO.\n\nHIDE_SCOPE_NAMES       = NO\n\n# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will\n# append additional text to a page's title, such as Class Reference. If set to\n# YES the compound reference will be hidden.\n# The default value is: NO.\n\nHIDE_COMPOUND_REFERENCE= NO\n\n# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of\n# the files that are included by a file in the documentation of that file.\n# The default value is: YES.\n\nSHOW_INCLUDE_FILES     = YES\n\n# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each\n# grouped member an include statement to the documentation, telling the reader\n# which file to include in order to use the member.\n# The default value is: NO.\n\nSHOW_GROUPED_MEMB_INC  = NO\n\n# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include\n# files with double quotes in the documentation rather than with sharp brackets.\n# The default value is: NO.\n\nFORCE_LOCAL_INCLUDES   = NO\n\n# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the\n# documentation for inline members.\n# The default value is: YES.\n\nINLINE_INFO            = YES\n\n# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the\n# (detailed) documentation of file and class members alphabetically by member\n# name. If set to NO, the members will appear in declaration order.\n# The default value is: YES.\n\nSORT_MEMBER_DOCS       = YES\n\n# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief\n# descriptions of file, namespace and class members alphabetically by member\n# name. If set to NO, the members will appear in declaration order. Note that\n# this will also influence the order of the classes in the class list.\n# The default value is: NO.\n\nSORT_BRIEF_DOCS        = NO\n\n# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the\n# (brief and detailed) documentation of class members so that constructors and\n# destructors are listed first. If set to NO the constructors will appear in the\n# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS.\n# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief\n# member documentation.\n# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting\n# detailed member documentation.\n# The default value is: NO.\n\nSORT_MEMBERS_CTORS_1ST = NO\n\n# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy\n# of group names into alphabetical order. If set to NO the group names will\n# appear in their defined order.\n# The default value is: NO.\n\nSORT_GROUP_NAMES       = NO\n\n# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by\n# fully-qualified names, including namespaces. If set to NO, the class list will\n# be sorted only by class name, not including the namespace part.\n# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES.\n# Note: This option applies only to the class list, not to the alphabetical\n# list.\n# The default value is: NO.\n\nSORT_BY_SCOPE_NAME     = NO\n\n# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper\n# type resolution of all parameters of a function it will reject a match between\n# the prototype and the implementation of a member function even if there is\n# only one candidate or it is obvious which candidate to choose by doing a\n# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still\n# accept a match between prototype and implementation in such cases.\n# The default value is: NO.\n\nSTRICT_PROTO_MATCHING  = NO\n\n# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo\n# list. This list is created by putting \\todo commands in the documentation.\n# The default value is: YES.\n\nGENERATE_TODOLIST      = YES\n\n# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test\n# list. This list is created by putting \\test commands in the documentation.\n# The default value is: YES.\n\nGENERATE_TESTLIST      = YES\n\n# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug\n# list. This list is created by putting \\bug commands in the documentation.\n# The default value is: YES.\n\nGENERATE_BUGLIST       = YES\n\n# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO)\n# the deprecated list. This list is created by putting \\deprecated commands in\n# the documentation.\n# The default value is: YES.\n\nGENERATE_DEPRECATEDLIST= YES\n\n# The ENABLED_SECTIONS tag can be used to enable conditional documentation\n# sections, marked by \\if <section_label> ... \\endif and \\cond <section_label>\n# ... \\endcond blocks.\n\nENABLED_SECTIONS       =\n\n# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the\n# initial value of a variable or macro / define can have for it to appear in the\n# documentation. If the initializer consists of more lines than specified here\n# it will be hidden. Use a value of 0 to hide initializers completely. The\n# appearance of the value of individual variables and macros / defines can be\n# controlled using \\showinitializer or \\hideinitializer command in the\n# documentation regardless of this setting.\n# Minimum value: 0, maximum value: 10000, default value: 30.\n\nMAX_INITIALIZER_LINES  = 30\n\n# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at\n# the bottom of the documentation of classes and structs. If set to YES, the\n# list will mention the files that were used to generate the documentation.\n# The default value is: YES.\n\nSHOW_USED_FILES        = YES\n\n# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This\n# will remove the Files entry from the Quick Index and from the Folder Tree View\n# (if specified).\n# The default value is: YES.\n\nSHOW_FILES             = YES\n\n# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces\n# page. This will remove the Namespaces entry from the Quick Index and from the\n# Folder Tree View (if specified).\n# The default value is: YES.\n\nSHOW_NAMESPACES        = YES\n\n# The FILE_VERSION_FILTER tag can be used to specify a program or script that\n# doxygen should invoke to get the current version for each file (typically from\n# the version control system). Doxygen will invoke the program by executing (via\n# popen()) the command command input-file, where command is the value of the\n# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided\n# by doxygen. Whatever the program writes to standard output is used as the file\n# version. For an example see the documentation.\n\nFILE_VERSION_FILTER    =\n\n# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed\n# by doxygen. The layout file controls the global structure of the generated\n# output files in an output format independent way. To create the layout file\n# that represents doxygen's defaults, run doxygen with the -l option. You can\n# optionally specify a file name after the option, if omitted DoxygenLayout.xml\n# will be used as the name of the layout file.\n#\n# Note that if you run doxygen from a directory containing a file called\n# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE\n# tag is left empty.\n\nLAYOUT_FILE            =\n\n# The CITE_BIB_FILES tag can be used to specify one or more bib files containing\n# the reference definitions. This must be a list of .bib files. The .bib\n# extension is automatically appended if omitted. This requires the bibtex tool\n# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info.\n# For LaTeX the style of the bibliography can be controlled using\n# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the\n# search path. See also \\cite for info how to create references.\n\nCITE_BIB_FILES         =\n\n#---------------------------------------------------------------------------\n# Configuration options related to warning and progress messages\n#---------------------------------------------------------------------------\n\n# The QUIET tag can be used to turn on/off the messages that are generated to\n# standard output by doxygen. If QUIET is set to YES this implies that the\n# messages are off.\n# The default value is: NO.\n\nQUIET                  = NO\n\n# The WARNINGS tag can be used to turn on/off the warning messages that are\n# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES\n# this implies that the warnings are on.\n#\n# Tip: Turn warnings on while writing the documentation.\n# The default value is: YES.\n\nWARNINGS               = YES\n\n# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate\n# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag\n# will automatically be disabled.\n# The default value is: YES.\n\nWARN_IF_UNDOCUMENTED   = YES\n\n# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for\n# potential errors in the documentation, such as not documenting some parameters\n# in a documented function, or documenting parameters that don't exist or using\n# markup commands wrongly.\n# The default value is: YES.\n\nWARN_IF_DOC_ERROR      = YES\n\n# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that\n# are documented, but have no documentation for their parameters or return\n# value. If set to NO, doxygen will only warn about wrong or incomplete\n# parameter documentation, but not about the absence of documentation.\n# The default value is: NO.\n\nWARN_NO_PARAMDOC       = NO\n\n# The WARN_FORMAT tag determines the format of the warning messages that doxygen\n# can produce. The string should contain the $file, $line, and $text tags, which\n# will be replaced by the file and line number from which the warning originated\n# and the warning text. Optionally the format may contain $version, which will\n# be replaced by the version of the file (if it could be obtained via\n# FILE_VERSION_FILTER)\n# The default value is: $file:$line: $text.\n\nWARN_FORMAT            = \"$file:$line: $text\"\n\n# The WARN_LOGFILE tag can be used to specify a file to which warning and error\n# messages should be written. If left blank the output is written to standard\n# error (stderr).\n\nWARN_LOGFILE           =\n\n#---------------------------------------------------------------------------\n# Configuration options related to the input files\n#---------------------------------------------------------------------------\n\n# The INPUT tag is used to specify the files and/or directories that contain\n# documented source files. You may enter file names like myfile.cpp or\n# directories like /usr/src/myproject. Separate the files or directories with\n# spaces.\n# Note: If this tag is empty the current directory is searched.\n\nINPUT                  = @Seastar_SOURCE_DIR@/include\nINPUT                 += @Seastar_BINARY_DIR@/gen/include\nINPUT                 += @Seastar_SOURCE_DIR@/doc/rpc.md\nINPUT                 += @Seastar_SOURCE_DIR@/doc/rpc-streaming.md\nINPUT                 += @Seastar_SOURCE_DIR@/doc/rpc-compression.md\nINPUT                 += @Seastar_SOURCE_DIR@/doc/compatibility.md\n\n# This tag can be used to specify the character encoding of the source files\n# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses\n# libiconv (or the iconv built into libc) for the transcoding. See the libiconv\n# documentation (see: http://www.gnu.org/software/libiconv) for the list of\n# possible encodings.\n# The default value is: UTF-8.\n\nINPUT_ENCODING         = UTF-8\n\n# If the value of the INPUT tag contains directories, you can use the\n# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and\n# *.h) to filter out the source-files in the directories. If left blank the\n# following patterns are tested:*.c, *.cc, *.cxx, *.cpp, *.c++, *.java, *.ii,\n# *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, *.hh, *.hxx, *.hpp,\n# *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, *.m, *.markdown,\n# *.md, *.mm, *.dox, *.py, *.f90, *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf,\n# *.qsf, *.as and *.js.\n\nFILE_PATTERNS          =\n\n# The RECURSIVE tag can be used to specify whether or not subdirectories should\n# be searched for input files as well.\n# The default value is: NO.\n\nRECURSIVE              = YES\n\n# The EXCLUDE tag can be used to specify files and/or directories that should be\n# excluded from the INPUT source files. This way you can easily exclude a\n# subdirectory from a directory tree whose root is specified with the INPUT tag.\n#\n# Note that relative paths are relative to the directory from which doxygen is\n# run.\n\nEXCLUDE                = build dpdk tests apps scripts\n\n# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or\n# directories that are symbolic links (a Unix file system feature) are excluded\n# from the input.\n# The default value is: NO.\n\nEXCLUDE_SYMLINKS       = NO\n\n# If the value of the INPUT tag contains directories, you can use the\n# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude\n# certain files from those directories.\n#\n# Note that the wildcards are matched against the file with absolute path, so to\n# exclude all test directories for example use the pattern */test/*\n\nEXCLUDE_PATTERNS       = test.py\n\n# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names\n# (namespaces, classes, functions, etc.) that should be excluded from the\n# output. The symbol name can be a fully qualified name, a word, or if the\n# wildcard * is used, a substring. Examples: ANamespace, AClass,\n# AClass::ANamespace, ANamespace::*Test\n#\n# Note that the wildcards are matched against the file with absolute path, so to\n# exclude all test directories use the pattern */test/*\n\nEXCLUDE_SYMBOLS        = seastar::internal seastar::coroutine::internal\n\n# The EXAMPLE_PATH tag can be used to specify one or more files or directories\n# that contain example code fragments that are included (see the \\include\n# command).\n\nEXAMPLE_PATH           = @Seastar_SOURCE_DIR@/demos @Seastar_SOURCE_DIR@/tests/unit\n\n# If the value of the EXAMPLE_PATH tag contains directories, you can use the\n# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and\n# *.h) to filter out the source-files in the directories. If left blank all\n# files are included.\n\nEXAMPLE_PATTERNS       =\n\n# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be\n# searched for input files to be used with the \\include or \\dontinclude commands\n# irrespective of the value of the RECURSIVE tag.\n# The default value is: NO.\n\nEXAMPLE_RECURSIVE      = NO\n\n# The IMAGE_PATH tag can be used to specify one or more files or directories\n# that contain images that are to be included in the documentation (see the\n# \\image command).\n\nIMAGE_PATH             =\n\n# The INPUT_FILTER tag can be used to specify a program that doxygen should\n# invoke to filter for each input file. Doxygen will invoke the filter program\n# by executing (via popen()) the command:\n#\n# <filter> <input-file>\n#\n# where <filter> is the value of the INPUT_FILTER tag, and <input-file> is the\n# name of an input file. Doxygen will then use the output that the filter\n# program writes to standard output. If FILTER_PATTERNS is specified, this tag\n# will be ignored.\n#\n# Note that the filter must not add or remove lines; it is applied before the\n# code is scanned, but not when the output code is generated. If lines are added\n# or removed, the anchors will not be placed correctly.\n\nINPUT_FILTER           =\n\n# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern\n# basis. Doxygen will compare the file name with each pattern and apply the\n# filter if there is a match. The filters are a list of the form: pattern=filter\n# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how\n# filters are used. If the FILTER_PATTERNS tag is empty or if none of the\n# patterns match the file name, INPUT_FILTER is applied.\n\nFILTER_PATTERNS        =\n\n# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using\n# INPUT_FILTER) will also be used to filter the input files that are used for\n# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES).\n# The default value is: NO.\n\nFILTER_SOURCE_FILES    = NO\n\n# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file\n# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and\n# it is also possible to disable source filtering for a specific pattern using\n# *.ext= (so without naming a filter).\n# This tag requires that the tag FILTER_SOURCE_FILES is set to YES.\n\nFILTER_SOURCE_PATTERNS =\n\n# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that\n# is part of the input, its contents will be placed on the main page\n# (index.html). This can be useful if you have a project on for instance GitHub\n# and want to reuse the introduction page also for the doxygen output.\n\nUSE_MDFILE_AS_MAINPAGE =\n\n#---------------------------------------------------------------------------\n# Configuration options related to source browsing\n#---------------------------------------------------------------------------\n\n# If the SOURCE_BROWSER tag is set to YES then a list of source files will be\n# generated. Documented entities will be cross-referenced with these sources.\n#\n# Note: To get rid of all source code in the generated output, make sure that\n# also VERBATIM_HEADERS is set to NO.\n# The default value is: NO.\n\nSOURCE_BROWSER         = NO\n\n# Setting the INLINE_SOURCES tag to YES will include the body of functions,\n# classes and enums directly into the documentation.\n# The default value is: NO.\n\nINLINE_SOURCES         = NO\n\n# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any\n# special comment blocks from generated source code fragments. Normal C, C++ and\n# Fortran comments will always remain visible.\n# The default value is: YES.\n\nSTRIP_CODE_COMMENTS    = YES\n\n# If the REFERENCED_BY_RELATION tag is set to YES then for each documented\n# function all documented functions referencing it will be listed.\n# The default value is: NO.\n\nREFERENCED_BY_RELATION = NO\n\n# If the REFERENCES_RELATION tag is set to YES then for each documented function\n# all documented entities called/used by that function will be listed.\n# The default value is: NO.\n\nREFERENCES_RELATION    = NO\n\n# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set\n# to YES then the hyperlinks from functions in REFERENCES_RELATION and\n# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will\n# link to the documentation.\n# The default value is: YES.\n\nREFERENCES_LINK_SOURCE = YES\n\n# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the\n# source code will show a tooltip with additional information such as prototype,\n# brief description and links to the definition and documentation. Since this\n# will make the HTML file larger and loading of large files a bit slower, you\n# can opt to disable this feature.\n# The default value is: YES.\n# This tag requires that the tag SOURCE_BROWSER is set to YES.\n\nSOURCE_TOOLTIPS        = YES\n\n# If the USE_HTAGS tag is set to YES then the references to source code will\n# point to the HTML generated by the htags(1) tool instead of doxygen built-in\n# source browser. The htags tool is part of GNU's global source tagging system\n# (see http://www.gnu.org/software/global/global.html). You will need version\n# 4.8.6 or higher.\n#\n# To use it do the following:\n# - Install the latest version of global\n# - Enable SOURCE_BROWSER and USE_HTAGS in the config file\n# - Make sure the INPUT points to the root of the source tree\n# - Run doxygen as normal\n#\n# Doxygen will invoke htags (and that will in turn invoke gtags), so these\n# tools must be available from the command line (i.e. in the search path).\n#\n# The result: instead of the source browser generated by doxygen, the links to\n# source code will now point to the output of htags.\n# The default value is: NO.\n# This tag requires that the tag SOURCE_BROWSER is set to YES.\n\nUSE_HTAGS              = NO\n\n# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a\n# verbatim copy of the header file for each class for which an include is\n# specified. Set to NO to disable this.\n# See also: Section \\class.\n# The default value is: YES.\n\nVERBATIM_HEADERS       = YES\n\n#---------------------------------------------------------------------------\n# Configuration options related to the alphabetical class index\n#---------------------------------------------------------------------------\n\n# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all\n# compounds will be generated. Enable this if the project contains a lot of\n# classes, structs, unions or interfaces.\n# The default value is: YES.\n\nALPHABETICAL_INDEX     = YES\n\n# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in\n# which the alphabetical index list will be split.\n# Minimum value: 1, maximum value: 20, default value: 5.\n# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.\n\nCOLS_IN_ALPHA_INDEX    = 5\n\n# In case all classes in a project start with a common prefix, all classes will\n# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag\n# can be used to specify a prefix (or a list of prefixes) that should be ignored\n# while generating the index headers.\n# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.\n\nIGNORE_PREFIX          =\n\n#---------------------------------------------------------------------------\n# Configuration options related to the HTML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output\n# The default value is: NO.\n\nGENERATE_HTML          = YES\n\n# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it.\n# The default directory is: html.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_OUTPUT            = html\n\n# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each\n# generated HTML page (for example: .htm, .php, .asp).\n# The default value is: .html.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_FILE_EXTENSION    = .html\n\n# The HTML_HEADER tag can be used to specify a user-defined HTML header file for\n# each generated HTML page. If the tag is left blank doxygen will generate a\n# standard header.\n#\n# To get valid HTML the header file that includes any scripts and style sheets\n# that doxygen needs, which is dependent on the configuration options used (e.g.\n# the setting GENERATE_TREEVIEW). It is highly recommended to start with a\n# default header using\n# doxygen -w html new_header.html new_footer.html new_stylesheet.css\n# YourConfigFile\n# and then modify the file new_header.html. See also section \"Doxygen usage\"\n# for information on how to generate the default header that doxygen normally\n# uses.\n# Note: The header is subject to change so you typically have to regenerate the\n# default header when upgrading to a newer version of doxygen. For a description\n# of the possible markers and block names see the documentation.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_HEADER            =\n\n# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each\n# generated HTML page. If the tag is left blank doxygen will generate a standard\n# footer. See HTML_HEADER for more information on how to generate a default\n# footer and what special commands can be used inside the footer. See also\n# section \"Doxygen usage\" for information on how to generate the default footer\n# that doxygen normally uses.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_FOOTER            =\n\n# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style\n# sheet that is used by each HTML page. It can be used to fine-tune the look of\n# the HTML output. If left blank doxygen will generate a default style sheet.\n# See also section \"Doxygen usage\" for information on how to generate the style\n# sheet that doxygen normally uses.\n# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as\n# it is more robust and this tag (HTML_STYLESHEET) will in the future become\n# obsolete.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_STYLESHEET        =\n\n# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined\n# cascading style sheets that are included after the standard style sheets\n# created by doxygen. Using this option one can overrule certain style aspects.\n# This is preferred over using HTML_STYLESHEET since it does not replace the\n# standard style sheet and is therefore more robust against future updates.\n# Doxygen will copy the style sheet files to the output directory.\n# Note: The order of the extra style sheet files is of importance (e.g. the last\n# style sheet in the list overrules the setting of the previous ones in the\n# list). For an example see the documentation.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_EXTRA_STYLESHEET  =\n\n# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or\n# other source files which should be copied to the HTML output directory. Note\n# that these files will be copied to the base HTML output directory. Use the\n# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these\n# files. In the HTML_STYLESHEET file, use the file name only. Also note that the\n# files will be copied as-is; there are no commands or markers available.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_EXTRA_FILES       =\n\n# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen\n# will adjust the colors in the style sheet and background images according to\n# this color. Hue is specified as an angle on a colorwheel, see\n# http://en.wikipedia.org/wiki/Hue for more information. For instance the value\n# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300\n# purple, and 360 is red again.\n# Minimum value: 0, maximum value: 359, default value: 220.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_HUE    = 220\n\n# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors\n# in the HTML output. For a value of 0 the output will use grayscales only. A\n# value of 255 will produce the most vivid colors.\n# Minimum value: 0, maximum value: 255, default value: 100.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_SAT    = 100\n\n# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the\n# luminance component of the colors in the HTML output. Values below 100\n# gradually make the output lighter, whereas values above 100 make the output\n# darker. The value divided by 100 is the actual gamma applied, so 80 represents\n# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not\n# change the gamma.\n# Minimum value: 40, maximum value: 240, default value: 80.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_GAMMA  = 80\n\n# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML\n# page will contain the date and time when the page was generated. Setting this\n# to NO can help when comparing the output of multiple runs.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_TIMESTAMP         = NO\n\n# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML\n# documentation will contain sections that can be hidden and shown after the\n# page has loaded.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_DYNAMIC_SECTIONS  = NO\n\n# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries\n# shown in the various tree structured indices initially; the user can expand\n# and collapse entries dynamically later on. Doxygen will expand the tree to\n# such a level that at most the specified number of entries are visible (unless\n# a fully collapsed tree already exceeds this amount). So setting the number of\n# entries 1 will produce a full collapsed tree by default. 0 is a special value\n# representing an infinite number of entries and will result in a full expanded\n# tree by default.\n# Minimum value: 0, maximum value: 9999, default value: 100.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_INDEX_NUM_ENTRIES = 100\n\n# If the GENERATE_DOCSET tag is set to YES, additional index files will be\n# generated that can be used as input for Apple's Xcode 3 integrated development\n# environment (see: http://developer.apple.com/tools/xcode/), introduced with\n# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a\n# Makefile in the HTML output directory. Running make will produce the docset in\n# that directory and running make install will install the docset in\n# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at\n# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html\n# for more information.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_DOCSET        = NO\n\n# This tag determines the name of the docset feed. A documentation feed provides\n# an umbrella under which multiple documentation sets from a single provider\n# (such as a company or product suite) can be grouped.\n# The default value is: Doxygen generated docs.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_FEEDNAME        = \"Doxygen generated docs\"\n\n# This tag specifies a string that should uniquely identify the documentation\n# set bundle. This should be a reverse domain-name style string, e.g.\n# com.mycompany.MyDocSet. Doxygen will append .docset to the name.\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_BUNDLE_ID       = org.doxygen.Project\n\n# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify\n# the documentation publisher. This should be a reverse domain-name style\n# string, e.g. com.mycompany.MyDocSet.documentation.\n# The default value is: org.doxygen.Publisher.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_PUBLISHER_ID    = org.doxygen.Publisher\n\n# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher.\n# The default value is: Publisher.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_PUBLISHER_NAME  = Publisher\n\n# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three\n# additional HTML index files: index.hhp, index.hhc, and index.hhk. The\n# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop\n# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on\n# Windows.\n#\n# The HTML Help Workshop contains a compiler that can convert all HTML output\n# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML\n# files are now used as the Windows 98 help format, and will replace the old\n# Windows help format (.hlp) on all Windows platforms in the future. Compressed\n# HTML files also contain an index, a table of contents, and you can search for\n# words in the documentation. The HTML workshop also contains a viewer for\n# compressed HTML files.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_HTMLHELP      = NO\n\n# The CHM_FILE tag can be used to specify the file name of the resulting .chm\n# file. You can add a path in front of the file if the result should not be\n# written to the html output directory.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nCHM_FILE               =\n\n# The HHC_LOCATION tag can be used to specify the location (absolute path\n# including file name) of the HTML help compiler (hhc.exe). If non-empty,\n# doxygen will try to run the HTML help compiler on the generated index.hhp.\n# The file has to be specified with full path.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nHHC_LOCATION           =\n\n# The GENERATE_CHI flag controls if a separate .chi index file is generated\n# (YES) or that it should be included in the master .chm file (NO).\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nGENERATE_CHI           = NO\n\n# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc)\n# and project file content.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nCHM_INDEX_ENCODING     =\n\n# The BINARY_TOC flag controls whether a binary table of contents is generated\n# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it\n# enables the Previous and Next buttons.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nBINARY_TOC             = NO\n\n# The TOC_EXPAND flag can be set to YES to add extra items for group members to\n# the table of contents of the HTML help documentation and to the tree view.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nTOC_EXPAND             = NO\n\n# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and\n# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that\n# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help\n# (.qch) of the generated HTML documentation.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_QHP           = NO\n\n# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify\n# the file name of the resulting .qch file. The path specified is relative to\n# the HTML output folder.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQCH_FILE               =\n\n# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help\n# Project output. For more information please see Qt Help Project / Namespace\n# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace).\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_NAMESPACE          = org.doxygen.Project\n\n# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt\n# Help Project output. For more information please see Qt Help Project / Virtual\n# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual-\n# folders).\n# The default value is: doc.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_VIRTUAL_FOLDER     = doc\n\n# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom\n# filter to add. For more information please see Qt Help Project / Custom\n# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-\n# filters).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_CUST_FILTER_NAME   =\n\n# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the\n# custom filter to add. For more information please see Qt Help Project / Custom\n# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-\n# filters).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_CUST_FILTER_ATTRS  =\n\n# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this\n# project's filter section matches. Qt Help Project / Filter Attributes (see:\n# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_SECT_FILTER_ATTRS  =\n\n# The QHG_LOCATION tag can be used to specify the location of Qt's\n# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the\n# generated .qhp file.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHG_LOCATION           =\n\n# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be\n# generated, together with the HTML files, they form an Eclipse help plugin. To\n# install this plugin and make it available under the help contents menu in\n# Eclipse, the contents of the directory containing the HTML and XML files needs\n# to be copied into the plugins directory of eclipse. The name of the directory\n# within the plugins directory should be the same as the ECLIPSE_DOC_ID value.\n# After copying Eclipse needs to be restarted before the help appears.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_ECLIPSEHELP   = NO\n\n# A unique identifier for the Eclipse help plugin. When installing the plugin\n# the directory name containing the HTML and XML files should also have this\n# name. Each documentation set should have its own identifier.\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES.\n\nECLIPSE_DOC_ID         = org.doxygen.Project\n\n# If you want full control over the layout of the generated HTML pages it might\n# be necessary to disable the index and replace it with your own. The\n# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top\n# of each HTML page. A value of NO enables the index and the value YES disables\n# it. Since the tabs in the index contain the same information as the navigation\n# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nDISABLE_INDEX          = NO\n\n# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index\n# structure should be generated to display hierarchical information. If the tag\n# value is set to YES, a side panel will be generated containing a tree-like\n# index structure (just like the one that is generated for HTML Help). For this\n# to work a browser that supports JavaScript, DHTML, CSS and frames is required\n# (i.e. any modern browser). Windows users are probably better off using the\n# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can\n# further fine-tune the look of the index. As an example, the default style\n# sheet generated by doxygen has an example that shows how to put an image at\n# the root of the tree instead of the PROJECT_NAME. Since the tree basically has\n# the same information as the tab index, you could consider setting\n# DISABLE_INDEX to YES when enabling this option.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_TREEVIEW      = NO\n\n# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that\n# doxygen will group on one line in the generated HTML documentation.\n#\n# Note that a value of 0 will completely suppress the enum values from appearing\n# in the overview section.\n# Minimum value: 0, maximum value: 20, default value: 4.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nENUM_VALUES_PER_LINE   = 4\n\n# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used\n# to set the initial width (in pixels) of the frame in which the tree is shown.\n# Minimum value: 0, maximum value: 1500, default value: 250.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nTREEVIEW_WIDTH         = 250\n\n# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to\n# external symbols imported via tag files in a separate window.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nEXT_LINKS_IN_WINDOW    = NO\n\n# Use this tag to change the font size of LaTeX formulas included as images in\n# the HTML documentation. When you change the font size after a successful\n# doxygen run you need to manually remove any form_*.png images from the HTML\n# output directory to force them to be regenerated.\n# Minimum value: 8, maximum value: 50, default value: 10.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nFORMULA_FONTSIZE       = 10\n\n# Use the FORMULA_TRANPARENT tag to determine whether or not the images\n# generated for formulas are transparent PNGs. Transparent PNGs are not\n# supported properly for IE 6.0, but are supported on all modern browsers.\n#\n# Note that when changing this option you need to delete any form_*.png files in\n# the HTML output directory before the changes have effect.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nFORMULA_TRANSPARENT    = YES\n\n# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see\n# http://www.mathjax.org) which uses client side Javascript for the rendering\n# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX\n# installed or if you want to formulas look prettier in the HTML output. When\n# enabled you may also need to install MathJax separately and configure the path\n# to it using the MATHJAX_RELPATH option.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nUSE_MATHJAX            = NO\n\n# When MathJax is enabled you can set the default output format to be used for\n# the MathJax output. See the MathJax site (see:\n# http://docs.mathjax.org/en/latest/output.html) for more details.\n# Possible values are: HTML-CSS (which is slower, but has the best\n# compatibility), NativeMML (i.e. MathML) and SVG.\n# The default value is: HTML-CSS.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_FORMAT         = HTML-CSS\n\n# When MathJax is enabled you need to specify the location relative to the HTML\n# output directory using the MATHJAX_RELPATH option. The destination directory\n# should contain the MathJax.js script. For instance, if the mathjax directory\n# is located at the same level as the HTML output directory, then\n# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax\n# Content Delivery Network so you can quickly see the result without installing\n# MathJax. However, it is strongly recommended to install a local copy of\n# MathJax from http://www.mathjax.org before deployment.\n# The default value is: http://cdn.mathjax.org/mathjax/latest.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_RELPATH        = http://cdn.mathjax.org/mathjax/latest\n\n# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax\n# extension names that should be enabled during MathJax rendering. For example\n# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_EXTENSIONS     =\n\n# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces\n# of code that will be used on startup of the MathJax code. See the MathJax site\n# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an\n# example see the documentation.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_CODEFILE       =\n\n# When the SEARCHENGINE tag is enabled doxygen will generate a search box for\n# the HTML output. The underlying search engine uses javascript and DHTML and\n# should work on any modern browser. Note that when using HTML help\n# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET)\n# there is already a search function so this one should typically be disabled.\n# For large projects the javascript based search engine can be slow, then\n# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to\n# search using the keyboard; to jump to the search box use <access key> + S\n# (what the <access key> is depends on the OS and browser, but it is typically\n# <CTRL>, <ALT>/<option>, or both). Inside the search box use the <cursor down\n# key> to jump into the search results window, the results can be navigated\n# using the <cursor keys>. Press <Enter> to select an item or <escape> to cancel\n# the search. The filter options can be selected when the cursor is inside the\n# search box by pressing <Shift>+<cursor down>. Also here use the <cursor keys>\n# to select a filter and <Enter> or <escape> to activate or cancel the filter\n# option.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nSEARCHENGINE           = YES\n\n# When the SERVER_BASED_SEARCH tag is enabled the search engine will be\n# implemented using a web server instead of a web client using Javascript. There\n# are two flavors of web server based searching depending on the EXTERNAL_SEARCH\n# setting. When disabled, doxygen will generate a PHP script for searching and\n# an index file used by the script. When EXTERNAL_SEARCH is enabled the indexing\n# and searching needs to be provided by external tools. See the section\n# \"External Indexing and Searching\" for details.\n# The default value is: NO.\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nSERVER_BASED_SEARCH    = NO\n\n# When EXTERNAL_SEARCH tag is enabled doxygen will no longer generate the PHP\n# script for searching. Instead the search results are written to an XML file\n# which needs to be processed by an external indexer. Doxygen will invoke an\n# external search engine pointed to by the SEARCHENGINE_URL option to obtain the\n# search results.\n#\n# Doxygen ships with an example indexer (doxyindexer) and search engine\n# (doxysearch.cgi) which are based on the open source search engine library\n# Xapian (see: http://xapian.org/).\n#\n# See the section \"External Indexing and Searching\" for details.\n# The default value is: NO.\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nEXTERNAL_SEARCH        = NO\n\n# The SEARCHENGINE_URL should point to a search engine hosted by a web server\n# which will return the search results when EXTERNAL_SEARCH is enabled.\n#\n# Doxygen ships with an example indexer (doxyindexer) and search engine\n# (doxysearch.cgi) which are based on the open source search engine library\n# Xapian (see: http://xapian.org/). See the section \"External Indexing and\n# Searching\" for details.\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nSEARCHENGINE_URL       =\n\n# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the unindexed\n# search data is written to a file for indexing by an external tool. With the\n# SEARCHDATA_FILE tag the name of this file can be specified.\n# The default file is: searchdata.xml.\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nSEARCHDATA_FILE        = searchdata.xml\n\n# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the\n# EXTERNAL_SEARCH_ID tag can be used as an identifier for the project. This is\n# useful in combination with EXTRA_SEARCH_MAPPINGS to search through multiple\n# projects and redirect the results back to the right project.\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nEXTERNAL_SEARCH_ID     =\n\n# The EXTRA_SEARCH_MAPPINGS tag can be used to enable searching through doxygen\n# projects other than the one defined by this configuration file, but that are\n# all added to the same external search index. Each project needs to have a\n# unique id set via EXTERNAL_SEARCH_ID. The search mapping then maps the id of\n# to a relative location where the documentation can be found. The format is:\n# EXTRA_SEARCH_MAPPINGS = tagname1=loc1 tagname2=loc2 ...\n# This tag requires that the tag SEARCHENGINE is set to YES.\n\nEXTRA_SEARCH_MAPPINGS  =\n\n#---------------------------------------------------------------------------\n# Configuration options related to the LaTeX output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.\n# The default value is: YES.\n\nGENERATE_LATEX         = NO\n\n# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it.\n# The default directory is: latex.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_OUTPUT           = latex\n\n# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be\n# invoked.\n#\n# Note that when enabling USE_PDFLATEX this option is only used for generating\n# bitmaps for formulas in the HTML output, but not in the Makefile that is\n# written to the output directory.\n# The default file is: latex.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_CMD_NAME         = latex\n\n# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to generate\n# index for LaTeX.\n# The default file is: makeindex.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nMAKEINDEX_CMD_NAME     = makeindex\n\n# If the COMPACT_LATEX tag is set to YES, doxygen generates more compact LaTeX\n# documents. This may be useful for small projects and may help to save some\n# trees in general.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nCOMPACT_LATEX          = NO\n\n# The PAPER_TYPE tag can be used to set the paper type that is used by the\n# printer.\n# Possible values are: a4 (210 x 297 mm), letter (8.5 x 11 inches), legal (8.5 x\n# 14 inches) and executive (7.25 x 10.5 inches).\n# The default value is: a4.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nPAPER_TYPE             = a4\n\n# The EXTRA_PACKAGES tag can be used to specify one or more LaTeX package names\n# that should be included in the LaTeX output. To get the times font for\n# instance you can specify\n# EXTRA_PACKAGES=times\n# If left blank no extra packages will be included.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nEXTRA_PACKAGES         =\n\n# The LATEX_HEADER tag can be used to specify a personal LaTeX header for the\n# generated LaTeX document. The header should contain everything until the first\n# chapter. If it is left blank doxygen will generate a standard header. See\n# section \"Doxygen usage\" for information on how to let doxygen write the\n# default header to a separate file.\n#\n# Note: Only use a user-defined header if you know what you are doing! The\n# following commands have a special meaning inside the header: $title,\n# $datetime, $date, $doxygenversion, $projectname, $projectnumber,\n# $projectbrief, $projectlogo. Doxygen will replace $title with the empty\n# string, for the replacement values of the other commands the user is referred\n# to HTML_HEADER.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_HEADER           =\n\n# The LATEX_FOOTER tag can be used to specify a personal LaTeX footer for the\n# generated LaTeX document. The footer should contain everything after the last\n# chapter. If it is left blank doxygen will generate a standard footer. See\n# LATEX_HEADER for more information on how to generate a default footer and what\n# special commands can be used inside the footer.\n#\n# Note: Only use a user-defined footer if you know what you are doing!\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_FOOTER           =\n\n# The LATEX_EXTRA_STYLESHEET tag can be used to specify additional user-defined\n# LaTeX style sheets that are included after the standard style sheets created\n# by doxygen. Using this option one can overrule certain style aspects. Doxygen\n# will copy the style sheet files to the output directory.\n# Note: The order of the extra style sheet files is of importance (e.g. the last\n# style sheet in the list overrules the setting of the previous ones in the\n# list).\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_EXTRA_STYLESHEET =\n\n# The LATEX_EXTRA_FILES tag can be used to specify one or more extra images or\n# other source files which should be copied to the LATEX_OUTPUT output\n# directory. Note that the files will be copied as-is; there are no commands or\n# markers available.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_EXTRA_FILES      =\n\n# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated is\n# prepared for conversion to PDF (using ps2pdf or pdflatex). The PDF file will\n# contain links (just like the HTML output) instead of page references. This\n# makes the output suitable for online browsing using a PDF viewer.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nPDF_HYPERLINKS         = YES\n\n# If the USE_PDFLATEX tag is set to YES, doxygen will use pdflatex to generate\n# the PDF file directly from the LaTeX files. Set this option to YES, to get a\n# higher quality PDF documentation.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nUSE_PDFLATEX           = YES\n\n# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \\batchmode\n# command to the generated LaTeX files. This will instruct LaTeX to keep running\n# if errors occur, instead of asking the user for help. This option is also used\n# when generating formulas in HTML.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_BATCHMODE        = NO\n\n# If the LATEX_HIDE_INDICES tag is set to YES then doxygen will not include the\n# index chapters (such as File Index, Compound Index, etc.) in the output.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_HIDE_INDICES     = NO\n\n# If the LATEX_SOURCE_CODE tag is set to YES then doxygen will include source\n# code with syntax highlighting in the LaTeX output.\n#\n# Note that which sources are shown also depends on other settings such as\n# SOURCE_BROWSER.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_SOURCE_CODE      = NO\n\n# The LATEX_BIB_STYLE tag can be used to specify the style to use for the\n# bibliography, e.g. plainnat, or ieeetr. See\n# http://en.wikipedia.org/wiki/BibTeX and \\cite for more info.\n# The default value is: plain.\n# This tag requires that the tag GENERATE_LATEX is set to YES.\n\nLATEX_BIB_STYLE        = plain\n\n#---------------------------------------------------------------------------\n# Configuration options related to the RTF output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_RTF tag is set to YES, doxygen will generate RTF output. The\n# RTF output is optimized for Word 97 and may not look too pretty with other RTF\n# readers/editors.\n# The default value is: NO.\n\nGENERATE_RTF           = NO\n\n# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it.\n# The default directory is: rtf.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nRTF_OUTPUT             = rtf\n\n# If the COMPACT_RTF tag is set to YES, doxygen generates more compact RTF\n# documents. This may be useful for small projects and may help to save some\n# trees in general.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nCOMPACT_RTF            = NO\n\n# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated will\n# contain hyperlink fields. The RTF file will contain links (just like the HTML\n# output) instead of page references. This makes the output suitable for online\n# browsing using Word or some other Word compatible readers that support those\n# fields.\n#\n# Note: WordPad (write) and others do not support links.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nRTF_HYPERLINKS         = NO\n\n# Load stylesheet definitions from file. Syntax is similar to doxygen's config\n# file, i.e. a series of assignments. You only have to provide replacements,\n# missing definitions are set to their default value.\n#\n# See also section \"Doxygen usage\" for information on how to generate the\n# default style sheet that doxygen normally uses.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nRTF_STYLESHEET_FILE    =\n\n# Set optional variables used in the generation of an RTF document. Syntax is\n# similar to doxygen's config file. A template extensions file can be generated\n# using doxygen -e rtf extensionFile.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nRTF_EXTENSIONS_FILE    =\n\n# If the RTF_SOURCE_CODE tag is set to YES then doxygen will include source code\n# with syntax highlighting in the RTF output.\n#\n# Note that which sources are shown also depends on other settings such as\n# SOURCE_BROWSER.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_RTF is set to YES.\n\nRTF_SOURCE_CODE        = NO\n\n#---------------------------------------------------------------------------\n# Configuration options related to the man page output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_MAN tag is set to YES, doxygen will generate man pages for\n# classes and files.\n# The default value is: NO.\n\nGENERATE_MAN           = NO\n\n# The MAN_OUTPUT tag is used to specify where the man pages will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it. A directory man3 will be created inside the directory specified by\n# MAN_OUTPUT.\n# The default directory is: man.\n# This tag requires that the tag GENERATE_MAN is set to YES.\n\nMAN_OUTPUT             = man\n\n# The MAN_EXTENSION tag determines the extension that is added to the generated\n# man pages. In case the manual section does not start with a number, the number\n# 3 is prepended. The dot (.) at the beginning of the MAN_EXTENSION tag is\n# optional.\n# The default value is: .3.\n# This tag requires that the tag GENERATE_MAN is set to YES.\n\nMAN_EXTENSION          = .3\n\n# The MAN_SUBDIR tag determines the name of the directory created within\n# MAN_OUTPUT in which the man pages are placed. If defaults to man followed by\n# MAN_EXTENSION with the initial . removed.\n# This tag requires that the tag GENERATE_MAN is set to YES.\n\nMAN_SUBDIR             =\n\n# If the MAN_LINKS tag is set to YES and doxygen generates man output, then it\n# will generate one additional man file for each entity documented in the real\n# man page(s). These additional files only source the real man page, but without\n# them the man command would be unable to find the correct page.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_MAN is set to YES.\n\nMAN_LINKS              = NO\n\n#---------------------------------------------------------------------------\n# Configuration options related to the XML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_XML tag is set to YES, doxygen will generate an XML file that\n# captures the structure of the code including all documentation.\n# The default value is: NO.\n\nGENERATE_XML           = NO\n\n# The XML_OUTPUT tag is used to specify where the XML pages will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it.\n# The default directory is: xml.\n# This tag requires that the tag GENERATE_XML is set to YES.\n\nXML_OUTPUT             = xml\n\n# If the XML_PROGRAMLISTING tag is set to YES, doxygen will dump the program\n# listings (including syntax highlighting and cross-referencing information) to\n# the XML output. Note that enabling this will significantly increase the size\n# of the XML output.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_XML is set to YES.\n\nXML_PROGRAMLISTING     = YES\n\n#---------------------------------------------------------------------------\n# Configuration options related to the DOCBOOK output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_DOCBOOK tag is set to YES, doxygen will generate Docbook files\n# that can be used to generate PDF.\n# The default value is: NO.\n\nGENERATE_DOCBOOK       = NO\n\n# The DOCBOOK_OUTPUT tag is used to specify where the Docbook pages will be put.\n# If a relative path is entered the value of OUTPUT_DIRECTORY will be put in\n# front of it.\n# The default directory is: docbook.\n# This tag requires that the tag GENERATE_DOCBOOK is set to YES.\n\nDOCBOOK_OUTPUT         = docbook\n\n# If the DOCBOOK_PROGRAMLISTING tag is set to YES, doxygen will include the\n# program listings (including syntax highlighting and cross-referencing\n# information) to the DOCBOOK output. Note that enabling this will significantly\n# increase the size of the DOCBOOK output.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_DOCBOOK is set to YES.\n\nDOCBOOK_PROGRAMLISTING = NO\n\n#---------------------------------------------------------------------------\n# Configuration options for the AutoGen Definitions output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_AUTOGEN_DEF tag is set to YES, doxygen will generate an\n# AutoGen Definitions (see http://autogen.sf.net) file that captures the\n# structure of the code including all documentation. Note that this feature is\n# still experimental and incomplete at the moment.\n# The default value is: NO.\n\nGENERATE_AUTOGEN_DEF   = NO\n\n#---------------------------------------------------------------------------\n# Configuration options related to the Perl module output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_PERLMOD tag is set to YES, doxygen will generate a Perl module\n# file that captures the structure of the code including all documentation.\n#\n# Note that this feature is still experimental and incomplete at the moment.\n# The default value is: NO.\n\nGENERATE_PERLMOD       = NO\n\n# If the PERLMOD_LATEX tag is set to YES, doxygen will generate the necessary\n# Makefile rules, Perl scripts and LaTeX code to be able to generate PDF and DVI\n# output from the Perl module output.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_PERLMOD is set to YES.\n\nPERLMOD_LATEX          = NO\n\n# If the PERLMOD_PRETTY tag is set to YES, the Perl module output will be nicely\n# formatted so it can be parsed by a human reader. This is useful if you want to\n# understand what is going on. On the other hand, if this tag is set to NO, the\n# size of the Perl module output will be much smaller and Perl will parse it\n# just the same.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_PERLMOD is set to YES.\n\nPERLMOD_PRETTY         = YES\n\n# The names of the make variables in the generated doxyrules.make file are\n# prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. This is useful\n# so different doxyrules.make files included by the same Makefile don't\n# overwrite each other's variables.\n# This tag requires that the tag GENERATE_PERLMOD is set to YES.\n\nPERLMOD_MAKEVAR_PREFIX =\n\n#---------------------------------------------------------------------------\n# Configuration options related to the preprocessor\n#---------------------------------------------------------------------------\n\n# If the ENABLE_PREPROCESSING tag is set to YES, doxygen will evaluate all\n# C-preprocessor directives found in the sources and include files.\n# The default value is: YES.\n\nENABLE_PREPROCESSING   = YES\n\n# If the MACRO_EXPANSION tag is set to YES, doxygen will expand all macro names\n# in the source code. If set to NO, only conditional compilation will be\n# performed. Macro expansion can be done in a controlled way by setting\n# EXPAND_ONLY_PREDEF to YES.\n# The default value is: NO.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nMACRO_EXPANSION        = YES\n\n# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES then\n# the macro expansion is limited to the macros specified with the PREDEFINED and\n# EXPAND_AS_DEFINED tags.\n# The default value is: NO.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nEXPAND_ONLY_PREDEF     = NO\n\n# If the SEARCH_INCLUDES tag is set to YES, the include files in the\n# INCLUDE_PATH will be searched if a #include is found.\n# The default value is: YES.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nSEARCH_INCLUDES        = YES\n\n# The INCLUDE_PATH tag can be used to specify one or more directories that\n# contain include files that are not input files but should be processed by the\n# preprocessor.\n# This tag requires that the tag SEARCH_INCLUDES is set to YES.\n\nINCLUDE_PATH           =  @Seastar_SOURCE_DIR@/include @Seastar_BINARY_DIR@/gen/include\n\n# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard\n# patterns (like *.h and *.hpp) to filter out the header-files in the\n# directories. If left blank, the patterns specified with FILE_PATTERNS will be\n# used.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nINCLUDE_FILE_PATTERNS  =\n\n# The PREDEFINED tag can be used to specify one or more macro names that are\n# defined before the preprocessor is started (similar to the -D option of e.g.\n# gcc). The argument of the tag is a list of macros of the form: name or\n# name=definition (no spaces). If the definition and the \"=\" are omitted, \"=1\"\n# is assumed. To prevent a macro definition from being undefined via #undef or\n# recursively expanded use the := operator instead of the = operator.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nPREDEFINED             = SEASTAR_API_LEVEL=8\n\n# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this\n# tag can be used to specify a list of macro names that should be expanded. The\n# macro definition that is found in the sources will be used. Use the PREDEFINED\n# tag if you want to use a different macro definition that overrules the\n# definition found in the source code.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nEXPAND_AS_DEFINED      =\n\n# If the SKIP_FUNCTION_MACROS tag is set to YES then doxygen's preprocessor will\n# remove all references to function-like macros that are alone on a line, have\n# an all uppercase name, and do not end with a semicolon. Such function macros\n# are typically used for boiler-plate code, and will confuse the parser if not\n# removed.\n# The default value is: YES.\n# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.\n\nSKIP_FUNCTION_MACROS   = YES\n\n#---------------------------------------------------------------------------\n# Configuration options related to external references\n#---------------------------------------------------------------------------\n\n# The TAGFILES tag can be used to specify one or more tag files. For each tag\n# file the location of the external documentation should be added. The format of\n# a tag file without this location is as follows:\n# TAGFILES = file1 file2 ...\n# Adding location for the tag files is done as follows:\n# TAGFILES = file1=loc1 \"file2 = loc2\" ...\n# where loc1 and loc2 can be relative or absolute paths or URLs. See the\n# section \"Linking to external documentation\" for more information about the use\n# of tag files.\n# Note: Each tag file must have a unique name (where the name does NOT include\n# the path). If a tag file is not located in the directory in which doxygen is\n# run, you must also specify the path to the tagfile here.\n\nTAGFILES               =\n\n# When a file name is specified after GENERATE_TAGFILE, doxygen will create a\n# tag file that is based on the input files it reads. See section \"Linking to\n# external documentation\" for more information about the usage of tag files.\n\nGENERATE_TAGFILE       =\n\n# If the ALLEXTERNALS tag is set to YES, all external class will be listed in\n# the class index. If set to NO, only the inherited external classes will be\n# listed.\n# The default value is: NO.\n\nALLEXTERNALS           = NO\n\n# If the EXTERNAL_GROUPS tag is set to YES, all external groups will be listed\n# in the modules index. If set to NO, only the current project's groups will be\n# listed.\n# The default value is: YES.\n\nEXTERNAL_GROUPS        = YES\n\n# If the EXTERNAL_PAGES tag is set to YES, all external pages will be listed in\n# the related pages index. If set to NO, only the current project's pages will\n# be listed.\n# The default value is: YES.\n\nEXTERNAL_PAGES         = YES\n\n# The PERL_PATH should be the absolute path and name of the perl script\n# interpreter (i.e. the result of 'which perl').\n# The default file (with absolute path) is: /usr/bin/perl.\n\nPERL_PATH              = /usr/bin/perl\n\n#---------------------------------------------------------------------------\n# Configuration options related to the dot tool\n#---------------------------------------------------------------------------\n\n# If the CLASS_DIAGRAMS tag is set to YES, doxygen will generate a class diagram\n# (in HTML and LaTeX) for classes with base or super classes. Setting the tag to\n# NO turns the diagrams off. Note that this option also works with HAVE_DOT\n# disabled, but it is recommended to install and use dot, since it yields more\n# powerful graphs.\n# The default value is: YES.\n\nCLASS_DIAGRAMS         = YES\n\n# You can define message sequence charts within doxygen comments using the \\msc\n# command. Doxygen will then run the mscgen tool (see:\n# http://www.mcternan.me.uk/mscgen/)) to produce the chart and insert it in the\n# documentation. The MSCGEN_PATH tag allows you to specify the directory where\n# the mscgen tool resides. If left empty the tool is assumed to be found in the\n# default search path.\n\nMSCGEN_PATH            =\n\n# You can include diagrams made with dia in doxygen documentation. Doxygen will\n# then run dia to produce the diagram and insert it in the documentation. The\n# DIA_PATH tag allows you to specify the directory where the dia binary resides.\n# If left empty dia is assumed to be found in the default search path.\n\nDIA_PATH               =\n\n# If set to YES the inheritance and collaboration graphs will hide inheritance\n# and usage relations if the target is undocumented or is not a class.\n# The default value is: YES.\n\nHIDE_UNDOC_RELATIONS   = YES\n\n# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is\n# available from the path. This tool is part of Graphviz (see:\n# http://www.graphviz.org/), a graph visualization toolkit from AT&T and Lucent\n# Bell Labs. The other options in this section have no effect if this option is\n# set to NO\n# The default value is: NO.\n\nHAVE_DOT               = NO\n\n# The DOT_NUM_THREADS specifies the number of dot invocations doxygen is allowed\n# to run in parallel. When set to 0 doxygen will base this on the number of\n# processors available in the system. You can set it explicitly to a value\n# larger than 0 to get control over the balance between CPU load and processing\n# speed.\n# Minimum value: 0, maximum value: 32, default value: 0.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_NUM_THREADS        = 0\n\n# When you want a differently looking font in the dot files that doxygen\n# generates you can specify the font name using DOT_FONTNAME. You need to make\n# sure dot is able to find the font, which can be done by putting it in a\n# standard location or by setting the DOTFONTPATH environment variable or by\n# setting DOT_FONTPATH to the directory containing the font.\n# The default value is: Helvetica.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_FONTNAME           = Helvetica\n\n# The DOT_FONTSIZE tag can be used to set the size (in points) of the font of\n# dot graphs.\n# Minimum value: 4, maximum value: 24, default value: 10.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_FONTSIZE           = 10\n\n# By default doxygen will tell dot to use the default font as specified with\n# DOT_FONTNAME. If you specify a different font using DOT_FONTNAME you can set\n# the path where dot can find it using this tag.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_FONTPATH           =\n\n# If the CLASS_GRAPH tag is set to YES then doxygen will generate a graph for\n# each documented class showing the direct and indirect inheritance relations.\n# Setting this tag to YES will force the CLASS_DIAGRAMS tag to NO.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nCLASS_GRAPH            = YES\n\n# If the COLLABORATION_GRAPH tag is set to YES then doxygen will generate a\n# graph for each documented class showing the direct and indirect implementation\n# dependencies (inheritance, containment, and class references variables) of the\n# class with other documented classes.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nCOLLABORATION_GRAPH    = YES\n\n# If the GROUP_GRAPHS tag is set to YES then doxygen will generate a graph for\n# groups, showing the direct groups dependencies.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nGROUP_GRAPHS           = YES\n\n# If the UML_LOOK tag is set to YES, doxygen will generate inheritance and\n# collaboration diagrams in a style similar to the OMG's Unified Modeling\n# Language.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nUML_LOOK               = NO\n\n# If the UML_LOOK tag is enabled, the fields and methods are shown inside the\n# class node. If there are many fields or methods and many nodes the graph may\n# become too big to be useful. The UML_LIMIT_NUM_FIELDS threshold limits the\n# number of items for each type to make the size more manageable. Set this to 0\n# for no limit. Note that the threshold may be exceeded by 50% before the limit\n# is enforced. So when you set the threshold to 10, up to 15 fields may appear,\n# but if the number exceeds 15, the total amount of fields shown is limited to\n# 10.\n# Minimum value: 0, maximum value: 100, default value: 10.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nUML_LIMIT_NUM_FIELDS   = 10\n\n# If the TEMPLATE_RELATIONS tag is set to YES then the inheritance and\n# collaboration graphs will show the relations between templates and their\n# instances.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nTEMPLATE_RELATIONS     = NO\n\n# If the INCLUDE_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are set to\n# YES then doxygen will generate a graph for each documented file showing the\n# direct and indirect include dependencies of the file with other documented\n# files.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nINCLUDE_GRAPH          = YES\n\n# If the INCLUDED_BY_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are\n# set to YES then doxygen will generate a graph for each documented file showing\n# the direct and indirect include dependencies of the file with other documented\n# files.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nINCLUDED_BY_GRAPH      = YES\n\n# If the CALL_GRAPH tag is set to YES then doxygen will generate a call\n# dependency graph for every global function or class method.\n#\n# Note that enabling this option will significantly increase the time of a run.\n# So in most cases it will be better to enable call graphs for selected\n# functions only using the \\callgraph command.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nCALL_GRAPH             = NO\n\n# If the CALLER_GRAPH tag is set to YES then doxygen will generate a caller\n# dependency graph for every global function or class method.\n#\n# Note that enabling this option will significantly increase the time of a run.\n# So in most cases it will be better to enable caller graphs for selected\n# functions only using the \\callergraph command.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nCALLER_GRAPH           = NO\n\n# If the GRAPHICAL_HIERARCHY tag is set to YES then doxygen will graphical\n# hierarchy of all classes instead of a textual one.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nGRAPHICAL_HIERARCHY    = YES\n\n# If the DIRECTORY_GRAPH tag is set to YES then doxygen will show the\n# dependencies a directory has on other directories in a graphical way. The\n# dependency relations are determined by the #include relations between the\n# files in the directories.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDIRECTORY_GRAPH        = YES\n\n# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images\n# generated by dot.\n# Note: If you choose svg you need to set HTML_FILE_EXTENSION to xhtml in order\n# to make the SVG files visible in IE 9+ (other browsers do not have this\n# requirement).\n# Possible values are: png, jpg, gif and svg.\n# The default value is: png.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_IMAGE_FORMAT       = png\n\n# If DOT_IMAGE_FORMAT is set to svg, then this option can be set to YES to\n# enable generation of interactive SVG images that allow zooming and panning.\n#\n# Note that this requires a modern browser other than Internet Explorer. Tested\n# and working are Firefox, Chrome, Safari, and Opera.\n# Note: For IE 9+ you need to set HTML_FILE_EXTENSION to xhtml in order to make\n# the SVG files visible. Older versions of IE do not have SVG support.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nINTERACTIVE_SVG        = NO\n\n# The DOT_PATH tag can be used to specify the path where the dot tool can be\n# found. If left blank, it is assumed the dot tool can be found in the path.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_PATH               =\n\n# The DOTFILE_DIRS tag can be used to specify one or more directories that\n# contain dot files that are included in the documentation (see the \\dotfile\n# command).\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOTFILE_DIRS           =\n\n# The MSCFILE_DIRS tag can be used to specify one or more directories that\n# contain msc files that are included in the documentation (see the \\mscfile\n# command).\n\nMSCFILE_DIRS           =\n\n# The DIAFILE_DIRS tag can be used to specify one or more directories that\n# contain dia files that are included in the documentation (see the \\diafile\n# command).\n\nDIAFILE_DIRS           =\n\n# When using plantuml, the PLANTUML_JAR_PATH tag should be used to specify the\n# path where java can find the plantuml.jar file. If left blank, it is assumed\n# PlantUML is not used or called during a preprocessing step. Doxygen will\n# generate a warning when it encounters a \\startuml command in this case and\n# will not generate output for the diagram.\n\nPLANTUML_JAR_PATH      =\n\n# When using plantuml, the specified paths are searched for files specified by\n# the !include statement in a plantuml block.\n\nPLANTUML_INCLUDE_PATH  =\n\n# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of nodes\n# that will be shown in the graph. If the number of nodes in a graph becomes\n# larger than this value, doxygen will truncate the graph, which is visualized\n# by representing a node as a red box. Note that doxygen if the number of direct\n# children of the root node in a graph is already larger than\n# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note that\n# the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH.\n# Minimum value: 0, maximum value: 10000, default value: 50.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_GRAPH_MAX_NODES    = 50\n\n# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the graphs\n# generated by dot. A depth value of 3 means that only nodes reachable from the\n# root by following a path via at most 3 edges will be shown. Nodes that lay\n# further from the root node will be omitted. Note that setting this option to 1\n# or 2 may greatly reduce the computation time needed for large code bases. Also\n# note that the size of a graph can be further restricted by\n# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction.\n# Minimum value: 0, maximum value: 1000, default value: 0.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nMAX_DOT_GRAPH_DEPTH    = 0\n\n# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent\n# background. This is disabled by default, because dot on Windows does not seem\n# to support this out of the box.\n#\n# Warning: Depending on the platform used, enabling this option may lead to\n# badly anti-aliased labels on the edges of a graph (i.e. they become hard to\n# read).\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_TRANSPARENT        = NO\n\n# Set the DOT_MULTI_TARGETS tag to YES to allow dot to generate multiple output\n# files in one run (i.e. multiple -o and -T options on the command line). This\n# makes dot run faster, but since only newer versions of dot (>1.8.10) support\n# this, this feature is disabled by default.\n# The default value is: NO.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_MULTI_TARGETS      = NO\n\n# If the GENERATE_LEGEND tag is set to YES doxygen will generate a legend page\n# explaining the meaning of the various boxes and arrows in the dot generated\n# graphs.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nGENERATE_LEGEND        = YES\n\n# If the DOT_CLEANUP tag is set to YES, doxygen will remove the intermediate dot\n# files that are used to generate the various graphs.\n# The default value is: YES.\n# This tag requires that the tag HAVE_DOT is set to YES.\n\nDOT_CLEANUP            = YES\n"
  },
  {
    "path": "doc/DoxygenLayout.xml",
    "content": "<doxygenlayout version=\"1.0\">\n  <!-- Generated by doxygen 1.8.13 -->\n  <!-- Navigation index tabs for HTML output -->\n  <navindex>\n    <tab type=\"mainpage\" visible=\"yes\" title=\"\"/>\n    <tab type=\"pages\" visible=\"yes\" title=\"\" intro=\"\"/>\n    <tab type=\"modules\" visible=\"yes\" title=\"\" intro=\"\"/>\n    <tab type=\"namespaces\" visible=\"yes\" title=\"\">\n      <tab type=\"namespacelist\" visible=\"yes\" title=\"\" intro=\"\"/>\n      <tab type=\"namespacemembers\" visible=\"yes\" title=\"\" intro=\"\"/>\n    </tab>\n    <tab type=\"classes\" visible=\"yes\" title=\"\">\n      <tab type=\"classlist\" visible=\"yes\" title=\"\" intro=\"\"/>\n      <tab type=\"classindex\" visible=\"$ALPHABETICAL_INDEX\" title=\"\"/>\n      <tab type=\"hierarchy\" visible=\"yes\" title=\"\" intro=\"\"/>\n      <tab type=\"classmembers\" visible=\"yes\" title=\"\" intro=\"\"/>\n    </tab>\n    <tab type=\"files\" visible=\"yes\" title=\"\">\n      <tab type=\"filelist\" visible=\"yes\" title=\"\" intro=\"\"/>\n      <tab type=\"globals\" visible=\"yes\" title=\"\" intro=\"\"/>\n    </tab>\n    <tab type=\"examples\" visible=\"yes\" title=\"\" intro=\"\"/>\n  </navindex>\n\n  <!-- Layout definition for a class page -->\n  <class>\n    <detaileddescription title=\"\"/>\n    <includes visible=\"$SHOW_INCLUDE_FILES\"/>\n    <inheritancegraph visible=\"$CLASS_GRAPH\"/>\n    <collaborationgraph visible=\"$COLLABORATION_GRAPH\"/>\n    <memberdecl>\n      <nestedclasses visible=\"yes\" title=\"\"/>\n      <publictypes title=\"\"/>\n      <services title=\"\"/>\n      <interfaces title=\"\"/>\n      <publicslots title=\"\"/>\n      <signals title=\"\"/>\n      <publicmethods title=\"\"/>\n      <publicstaticmethods title=\"\"/>\n      <publicattributes title=\"\"/>\n      <publicstaticattributes title=\"\"/>\n      <protectedtypes title=\"\"/>\n      <protectedslots title=\"\"/>\n      <protectedmethods title=\"\"/>\n      <protectedstaticmethods title=\"\"/>\n      <protectedattributes title=\"\"/>\n      <protectedstaticattributes title=\"\"/>\n      <packagetypes title=\"\"/>\n      <packagemethods title=\"\"/>\n      <packagestaticmethods title=\"\"/>\n      <packageattributes title=\"\"/>\n      <packagestaticattributes title=\"\"/>\n      <properties title=\"\"/>\n      <events title=\"\"/>\n      <privatetypes title=\"\"/>\n      <privateslots title=\"\"/>\n      <privatemethods title=\"\"/>\n      <privatestaticmethods title=\"\"/>\n      <privateattributes title=\"\"/>\n      <privatestaticattributes title=\"\"/>\n      <friends title=\"\"/>\n      <related title=\"\" subtitle=\"\"/>\n      <membergroups visible=\"yes\"/>\n    </memberdecl>\n    <memberdef>\n      <inlineclasses title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <services title=\"\"/>\n      <interfaces title=\"\"/>\n      <constructors title=\"\"/>\n      <functions title=\"\"/>\n      <related title=\"\"/>\n      <variables title=\"\"/>\n      <properties title=\"\"/>\n      <events title=\"\"/>\n    </memberdef>\n    <allmemberslink visible=\"yes\"/>\n    <usedfiles visible=\"$SHOW_USED_FILES\"/>\n    <authorsection visible=\"yes\"/>\n  </class>\n\n  <!-- Layout definition for a namespace page -->\n  <namespace>\n    <detaileddescription title=\"\"/>\n    <memberdecl>\n      <nestednamespaces visible=\"yes\" title=\"\"/>\n      <constantgroups visible=\"yes\" title=\"\"/>\n      <classes visible=\"yes\" title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n      <membergroups visible=\"yes\"/>\n    </memberdecl>\n    <memberdef>\n      <inlineclasses title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n    </memberdef>\n    <authorsection visible=\"yes\"/>\n  </namespace>\n\n  <!-- Layout definition for a file page -->\n  <file>\n    <detaileddescription title=\"\"/>\n    <includes visible=\"$SHOW_INCLUDE_FILES\"/>\n    <includegraph visible=\"$INCLUDE_GRAPH\"/>\n    <includedbygraph visible=\"$INCLUDED_BY_GRAPH\"/>\n    <sourcelink visible=\"yes\"/>\n    <memberdecl>\n      <classes visible=\"yes\" title=\"\"/>\n      <namespaces visible=\"yes\" title=\"\"/>\n      <constantgroups visible=\"yes\" title=\"\"/>\n      <defines title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n      <membergroups visible=\"yes\"/>\n    </memberdecl>\n    <memberdef>\n      <inlineclasses title=\"\"/>\n      <defines title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n    </memberdef>\n    <authorsection/>\n  </file>\n\n  <!-- Layout definition for a group page -->\n  <group>\n    <detaileddescription title=\"\"/>\n    <groupgraph visible=\"$GROUP_GRAPHS\"/>\n    <memberdecl>\n      <nestedgroups visible=\"yes\" title=\"\"/>\n      <dirs visible=\"yes\" title=\"\"/>\n      <files visible=\"yes\" title=\"\"/>\n      <namespaces visible=\"yes\" title=\"\"/>\n      <classes visible=\"yes\" title=\"\"/>\n      <defines title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <enumvalues title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n      <signals title=\"\"/>\n      <publicslots title=\"\"/>\n      <protectedslots title=\"\"/>\n      <privateslots title=\"\"/>\n      <events title=\"\"/>\n      <properties title=\"\"/>\n      <friends title=\"\"/>\n      <membergroups visible=\"yes\"/>\n    </memberdecl>\n    <memberdef>\n      <pagedocs/>\n      <inlineclasses title=\"\"/>\n      <defines title=\"\"/>\n      <typedefs title=\"\"/>\n      <enums title=\"\"/>\n      <enumvalues title=\"\"/>\n      <functions title=\"\"/>\n      <variables title=\"\"/>\n      <signals title=\"\"/>\n      <publicslots title=\"\"/>\n      <protectedslots title=\"\"/>\n      <privateslots title=\"\"/>\n      <events title=\"\"/>\n      <properties title=\"\"/>\n      <friends title=\"\"/>\n    </memberdef>\n    <authorsection visible=\"yes\"/>\n  </group>\n\n  <!-- Layout definition for a directory page -->\n  <directory>\n    <detaileddescription title=\"\"/>\n    <directorygraph visible=\"yes\"/>\n    <memberdecl>\n      <dirs visible=\"yes\"/>\n      <files visible=\"yes\"/>\n    </memberdecl>\n  </directory>\n</doxygenlayout>\n"
  },
  {
    "path": "doc/compatibility.md",
    "content": "Compatibility\n=============\n\nAs a library, Seastar aims to maintain backwards compatibility\nin terms of the source (application code should continue to\nbuild with newer versions of Seastar) and any binary protocols\nthat Seastar exposes (e.g. rpc).\n\nLink compatibility is not maintained - you cannot link an\napplication built with one version of Seastar with another\nversion of Seastar.\n\nLanguage standards\n==================\n\nSeastar will support the last two standards approved by the\nISO C++ committee. For example, after C++20 is released,\nSeastar supports C++17 and C++20.  Similarly, when C++23 is released,\nSeastar will support C++20 and C++23.\n\nSome features may only be enabled for newer dialects.\n\n\nPlatforms\n=========\n\nSeastar supports Linux. There is no known minimum kernel version,\nbut very old kernels might not work. Performance can be significantly\nbetter for newer kernels.\n\nFilesystem implementation quality can have significant effect on\nfile I/O performance. XFS is known to be working, ext4 may work well\ntoo. Test your filesystem and kernel versions to be sure.\n\nPatches for new platforms (e.g, Windows) are welcome.\n\n\nCompilers\n=========\n\nSeastar supports GCC and Clang. Ports to other compilers are\nwelcome.\n\nThe last two major releases of a compiler are supported (e.g.\nGCC 13 and GCC 14). Patches to support older versions are welcome,\nas long as they don't require onerous compromises.\n\nDeprecation\n===========\n\nOccasionally, we discover that we took the wrong approach with\nan API. In these cases we will offer a new API and tag the old\nAPI with the [[deprecated]] attribute. The deprecated API will\nbe removed after a transition period (which can vary depending on\nhow central the deprecated API is).\n\nBreaking changes\n================\n\nRarely, we have to make breaking changes. We try to limit those,\nbut sometimes there is no choice.\n\nTo support a transition period for breaking changes, Seastar\noffers the Seastar_API_LEVEL cmake variable (and corresponding\n--api-level configure.py option). An API level selects different\nversions of the API. For example.\n\n   - Seastar_API_LEVEL=1 selects an old version of the\n     server_socket::accept() API that returns a variadic\n     future (which is deprecated)\n   - Seastar_API_LEVEL=2 selects a new version of the\n     server_socket::accept() API that returns a non-variadic\n     future\n   - Seastar_API_LEVEL=6 makes futures non-variadic\n   - Seastar_API_LEVEL=7 unifies CPU scheduling groups and IO priority classes\n     \"while at it\" file_impl API is forced to accept io_intent argument\n   - Seastar_API_LEVEL=8 changes json_return_type to hold a noncopyable function\n     and become a move-only type\n   - Seastar_API_LEVEL=9 defines the data_sink_impl::put(span<temporary_buffer>)\n     as the new and only method to be implemented\n\nApplications can use an old API_LEVEL during a transition\nperiod, fix their code, and move to the new API_LEVEL.\n\nOld API levels only live for a transition period, so if\nyou are using an API level below the latest, you should\nupgrade quickly.\n\nNote the application should not refer to the `api_vN`\nsub-namespaces that Seastar defines as part of the API_LEVEL\nmechanism; these are internal.\n\nInternal namespace\n==================\n\nIdentifiers in the `seastar::internal` namespace are not subject\nto source level compatibility and are subject to change or removal\nwithout notice. In addition the `api_vN` sub-namespaces are also\ninternal.\n\nAccidentally exposed internal identifiers\n=========================================\n\nSome identifiers predate the internal namespace, and are only\nexposed accidentally. These can also be removed or changed. Exposed\nidentifiers are documented using doxygen, but not all exposed\nAPIs are documented. In case of doubt, ask on the mailing list.\n\n\nAPI Level History\n=================\n\n|Level|Introduced |Mandatory|Description                                   |\n|:---:|:---------:|:-------:| -------------------------------------------- |\n| 2   |  2019-07  | 2020-04 | Non-variadic futures in socket::accept()     |\n| 3   |  2020-05  | 2023-03 | make_file_data_sink() closes file and returns a future<>  |\n| 4   |  2020-06  | 2023-03 | Non-variadic futures in when_all_succeed()   |\n| 5   |  2020-08  | 2023-03 | future::get() returns std::monostate() instead of void |\n| 6   |  2020-09  | 2023-03 | future<T> instead of future<T...>            |\n| 7   |  2023-05  | 2024-09 | unified CPU/IO scheduling groups             |\n| 8   |  2025-08  |         | noncopyable function in json_return_type     |\n| 9   |  2025-08  |         | data_sink_impl new API                       |\n\n\nNote: The \"mandatory\" column indicates when backwards compatibility\nsupport for the API preceding the new level was removed.\n\nImplementation notes for API levels\n===================================\n\nAPI levels are implemented by defining internal sub-namespaces\nfor each API level: `seastar::api_v1`, `seatar::api_v2` etc. `#ifdef`s\nare used to inline the user-selected API level namespace into the\nmain `seastar` namespace, making it visible.\n\nUsually, the old API is implemented in terms of the new API to\navoid code duplication.\n\nHere is an example about the transition from API_LEVEL 1 to 2. The\ntransition from 2 to 3 and similar is analogous.\n\nUnconditionally:\n - the new API is defined in sub-namespace `api_v2`\n\nIf API_LEVEL is 2:\n - `api_v2` namespace is inlined into the `seastar` namespace\n\nIf API_LEVEL is 1:\n - the old API is defined in sub-namespace `api_v1`\n - `api_v1` is implemented in terms of `api_v2` to prevent code duplication\n - `api_v1` namespace is inlined into the `seastar` namespace\n\nAfter a transition period:\n - everthing in `api_v1` is dropped\n - `api_v2` is removed, and its contents is placed in the parent namespace\n"
  },
  {
    "path": "doc/contributing.md",
    "content": "Contributing to Seastar\n=======================\n\n# Sending Patches\nSeastar follows a patch submission similar to Linux. Send patches to seastar-dev, with a DCO signed off message. Use git send-email to send your patch.\n\nExample:\n\n1. When you commit, use \"-s \" in your git commit command, which adds a DCO signed off message. DCO is a \"Developer's Certificate of Origin\" http://elinux.org/Developer_Certificate_Of_Origin\n\nFor the commit message, you can prefix a tag for an area of the codebase the patch is addressing\n\n        git commit -s -m \"core: some descriptive commit message\"\n\n2. then send an email to the google group\n\n        git send-email <revision>..<final_revision> --to seastar-dev@googlegroups.com\n\nNOTE: for sending replies to patches, use --in-reply-to with the message ID of the original message. Also, if you are sending out a new version of the change, use git rebase and then a `git send-email` with a `-v2`, for instance, to denote that it is a second version.\n\n# Testing and Approval\nRun test.py and ensure tests are passing (at least) as well as before the patch.\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "doc/generator.md",
    "content": "# Coroutine Generator\n\n## Overview\n\nThe generator implementation is based on C++23 proposal [P2502R2](https://wg21.link/P2502R2). It provides\nasynchronous coroutine-based value generation with two variants:\n\n- **Unbuffered**: Zero-copy, one suspension per element\n- **Buffered**: Batches elements, amortizes suspension overhead\n\nUnlike `std::generator`, `operator()` is a coroutine returning\n`std::optional<reference_type>`.\n\n## Object Relationships and Control Flow\n\nThe generator implementation uses awaiters to manage control flow between producer and consumer coroutines. Each awaiter has multiple entry/exit points:\n\n### yield_awaiter lifecycle (4 entry/exit points):\n1. Created by `co_yield value` → `promise.yield_value()` creates awaiter\n2. `co_await yield_awaiter` → calls `await_suspend(producer_handle)`\n3. `await_suspend()` branches:\n   - Direct path: returns `consumer_handle` (symmetric transfer)\n   - Scheduler path: schedules consumer, returns `noop_coroutine()`\n4. `await_resume()` called:\n   - By scheduler (after scheduler path)\n   - After symmetric transfer back from consumer\n\n### next_awaiter/call_awaiter lifecycle (4 entry/exit points):\n1. Created by `co_await gen()` → `operator()` creates awaiter\n2. `co_await call_awaiter` → calls `await_suspend(consumer_handle)`\n3. `await_suspend()` branches:\n   - Direct path: returns `producer_handle` (symmetric transfer)\n   - Scheduler path: schedules producer, returns `noop_coroutine()`\n4. `await_resume()` called:\n   - By scheduler (after scheduler path)\n   - After symmetric transfer back from producer\n\n```mermaid\ngraph TB\n    subgraph Consumer[\" \"]\n        C_Exec[Consumer Executing]\n        C_Suspended[Consumer Suspended]\n        C_Resume[Consumer await_resume#40;#41;]\n\n        C_Exec -->|co_await gen#40;#41;| CallAwaiter[Create call_awaiter]\n        CallAwaiter -->|co_await| CA_Suspend[call_awaiter::await_suspend#40;#41;]\n\n        CA_Suspend -->|!need_preempt#40;#41;<br/>return producer_handle| P_Exec\n        CA_Suspend -->|need_preempt#40;#41;<br/>schedule#40;producer#41;<br/>return noop| C_Suspended\n\n        P_Resume -->|symmetric transfer| C_Resume\n        Sched -->|resume consumer| C_Resume\n        C_Resume --> C_Exec\n    end\n\n    subgraph Producer[\" \"]\n        P_Exec[Producer Executing]\n        P_Suspended[Producer Suspended]\n        P_Resume[Producer await_resume#40;#41;]\n\n        P_Exec -->|co_yield value| YieldValue[promise.yield_value#40;#41;]\n        YieldValue -->|create| YieldAwaiter[Create yield_awaiter]\n        YieldAwaiter -->|co_await| YA_Suspend[yield_awaiter::await_suspend#40;#41;]\n\n        YA_Suspend -->|!need_preempt#40;#41;<br/>return consumer_handle| C_Exec\n        YA_Suspend -->|need_preempt#40;#41;<br/>schedule#40;consumer#41;<br/>return noop| P_Suspended\n\n        C_Resume -->|symmetric transfer| P_Resume\n        Sched -->|resume producer| P_Resume\n        P_Resume --> P_Exec\n    end\n\n    Sched[Seastar Scheduler]\n\n    style C_Exec fill:#e1f5ff\n    style P_Exec fill:#ffe1f5\n    style Sched fill:#fff5e1\n    style CallAwaiter fill:#d0e0ff\n    style YieldAwaiter fill:#ffd0e0\n```\n\n## Control Flow\n\nProducer and consumer transfer control via symmetric transfer when\n`!need_preempt()`. When `need_preempt()` returns true, the target coroutine\nis scheduled via `seastar::schedule()` and the current coroutine suspends\nto `noop_coroutine()`.\n\nThe sequence diagrams below show the complete lifecycle of awaiters as\nparticipants. Awaiters are shown with their creation and destruction\npoints via activate/deactivate markers. Three separate diagrams cover\nthe different awaiter types and their behaviors.\n\n### Unbuffered Generator: yield_awaiter (Zero-Copy)\n\nThis is the most common case where `co_yield` passes an rvalue reference.\nThe promise stores only a pointer to the yielded value (zero-copy).\n\n```mermaid\nsequenceDiagram\n    participant Consumer\n    participant CallAwaiter as call_awaiter\n    participant Producer\n    participant YieldAwaiter as yield_awaiter\n    participant Scheduler\n\n    activate Consumer\n    Note over Consumer: co_await gen()\n    Consumer->>CallAwaiter: create call_awaiter\n    activate CallAwaiter\n    Note over CallAwaiter: await_ready() = false\n    Note over CallAwaiter: await_suspend(consumer)\n\n    alt !need_preempt() - symmetric transfer\n        Note over CallAwaiter: return producer_handle\n        CallAwaiter-->>Producer: symmetric transfer\n        deactivate Consumer\n        activate Producer\n    else need_preempt() - via scheduler\n        activate Consumer\n        activate CallAwaiter\n        CallAwaiter->>Scheduler: schedule(producer)\n        Note over CallAwaiter: return noop_coroutine()\n        deactivate Consumer\n        Scheduler->>Producer: resume\n        activate Producer\n    end\n\n    Note over Producer: co_yield value (rvalue)\n    Note over Producer: stores pointer only\n    Producer->>YieldAwaiter: create yield_awaiter\n    activate YieldAwaiter\n    Note over YieldAwaiter: await_ready() = false\n    Note over YieldAwaiter: await_suspend(producer)\n\n    alt !need_preempt() - symmetric transfer\n        Note over YieldAwaiter: return consumer_handle\n        YieldAwaiter-->>Consumer: symmetric transfer\n        deactivate YieldAwaiter\n        deactivate Producer\n        activate Consumer\n        Note over CallAwaiter: await_resume()\n        CallAwaiter->>Consumer: return optional<ref>\n        deactivate CallAwaiter\n    else need_preempt() - via scheduler\n        activate Producer\n        activate YieldAwaiter\n        activate CallAwaiter\n        YieldAwaiter->>Scheduler: schedule(consumer)\n        Note over YieldAwaiter: return noop_coroutine()\n        deactivate YieldAwaiter\n        deactivate Producer\n        Scheduler->>Consumer: resume\n        activate Consumer\n        Note over CallAwaiter: await_resume()\n        CallAwaiter->>Consumer: return optional<ref>\n        deactivate CallAwaiter\n    end\n```\n\n### Unbuffered Generator: copy_awaiter (Type Conversion)\n\nUsed when `co_yield` passes a `const` lvalue reference requiring type conversion.\nThe awaiter creates and stores a copy of the converted value.\n\n```mermaid\nsequenceDiagram\n    participant Consumer\n    participant CallAwaiter as call_awaiter\n    participant Producer\n    participant CopyAwaiter as copy_awaiter\n    participant Scheduler\n\n    activate Consumer\n    Note over Consumer: co_await gen()\n    Consumer->>CallAwaiter: create call_awaiter\n    activate CallAwaiter\n    Note over CallAwaiter: await_ready() = false\n    CallAwaiter-->>Producer: transfer to producer\n    deactivate Consumer\n    activate Producer\n\n    Note over Producer: co_yield const_lvalue\n    Note over Producer: needs type conversion\n    Producer->>CopyAwaiter: create copy_awaiter\n    activate CopyAwaiter\n    Note over CopyAwaiter: stores converted copy\n    Note over CopyAwaiter: sets _value_ptr\n    Note over CopyAwaiter: await_ready() = false\n    Note over CopyAwaiter: await_suspend(producer)\n\n    alt !need_preempt() - symmetric transfer\n        Note over CopyAwaiter: return consumer_handle\n        CopyAwaiter-->>Consumer: symmetric transfer\n        deactivate CopyAwaiter\n        deactivate Producer\n        activate Consumer\n        Note over CallAwaiter: await_resume()\n        CallAwaiter->>Consumer: return optional<ref>\n        deactivate CallAwaiter\n        Note over Consumer: value points to copy_awaiter._value\n    else need_preempt() - via scheduler\n        activate Producer\n        activate CopyAwaiter\n        activate CallAwaiter\n        CopyAwaiter->>Scheduler: schedule(consumer)\n        Note over CopyAwaiter: return noop_coroutine()\n        deactivate CopyAwaiter\n        deactivate Producer\n        Scheduler->>Consumer: resume\n        activate Consumer\n        Note over CallAwaiter: await_resume()\n        CallAwaiter->>Consumer: return optional<ref>\n        deactivate CallAwaiter\n        Note over Consumer: value points to copy_awaiter._value\n    end\n```\n\n### Buffered Generator: yield_awaiter (Batching)\n\nThe buffered generator accumulates elements and conditionally suspends.\n`await_ready()` can return `true` when the buffer has space and no preemption is needed.\n\n```mermaid\nsequenceDiagram\n    participant Consumer\n    participant CallAwaiter as call_awaiter\n    participant Producer\n    participant YieldAwaiter as yield_awaiter\n    participant Scheduler\n\n    activate Consumer\n    Note over Consumer: co_await gen()\n    Consumer->>CallAwaiter: create call_awaiter\n    activate CallAwaiter\n    Note over CallAwaiter: buffer empty?\n    Note over CallAwaiter: await_ready() = false\n    Note over CallAwaiter: await_suspend(consumer)\n    Note over CallAwaiter: clear buffer\n    CallAwaiter-->>Producer: transfer to producer\n    deactivate Consumer\n    activate Producer\n\n    Note over Producer: co_yield element\n    Note over Producer: push to buffer\n    Producer->>YieldAwaiter: create yield_awaiter\n    activate YieldAwaiter\n    Note over YieldAwaiter: should_suspend?\n    Note over YieldAwaiter: !can_push_more OR need_preempt\n\n    alt await_ready() = true (buffer has space, !need_preempt)\n        Note over YieldAwaiter: continue without suspend\n        deactivate YieldAwaiter\n        Note over Producer: co_yield next element\n        Note over Producer: ...continues batching\n    else await_ready() = false (buffer full OR need_preempt)\n        activate Producer\n        activate CallAwaiter\n        activate YieldAwaiter\n        Note over YieldAwaiter: await_suspend(producer)\n        alt !need_preempt() - symmetric transfer\n            Note over YieldAwaiter: return consumer_handle\n            YieldAwaiter-->>Consumer: symmetric transfer\n            deactivate YieldAwaiter\n            deactivate Producer\n            activate Consumer\n            Note over CallAwaiter: await_resume()\n            CallAwaiter->>Consumer: return buffer[0]\n            deactivate CallAwaiter\n            Note over Consumer: drain buffer without suspension\n            Note over Consumer: buffer[1], buffer[2], ...\n        else need_preempt() - via scheduler\n            activate Producer\n            activate YieldAwaiter\n            activate CallAwaiter\n            YieldAwaiter->>Scheduler: schedule(consumer)\n            Note over YieldAwaiter: return noop_coroutine()\n            deactivate YieldAwaiter\n            deactivate Producer\n            Scheduler->>Consumer: resume\n            activate Consumer\n            Note over CallAwaiter: await_resume()\n            CallAwaiter->>Consumer: return buffer[0]\n            deactivate CallAwaiter\n            Note over Consumer: drain buffer without suspension\n        end\n    end\n```\n\n## Unbuffered Generator\n\n### Characteristics\n\n- Stores pointer to value in producer's stack frame\n- No copies or moves\n- One suspension per yielded element\n\n### Usage\n\n```cpp\ngenerator<const T&> produce() {\n    T value;\n    co_yield value;  // Zero-copy: stores pointer only\n}\n```\n\nUse when element moves are expensive or latency is critical.\n\n## Buffered Generator\n\n### Characteristics\n\n- Accumulates elements in a container\n- Suspends when buffer full or need_preempt() returns true\n- Consumer drains buffer without suspensions\n\n### Usage\n\n```cpp\ngenerator<const T&, T, circular_buffer_fixed_capacity<T, 128>> produce() {\n    co_yield element;           // Individual elements\n    co_yield std::span(data);   // Ranges\n}\n```\n\nUse when throughput matters and element moves are cheap.\n\n### Buffer Measurement\n\nThe buffered variant uses a customization point object to check buffer capacity:\n\n```cpp\n// Priority 1: Member function\nstruct MemoryBuffer {\n    bool can_push_more() const {\n        return memory_used < memory_limit;\n    }\n};\n\n// Priority 2: ADL free function\nnamespace my_ns {\n    bool can_push_more(const MyContainer& c);\n}\n\n// Priority 3: Default\nreturn container.size() < container.capacity();\n```\n\n## Lifetime Guarantees\n\n### Unbuffered\n\nWhen `co_yield` evaluates an expression producing a glvalue, the object lives\nuntil the coroutine resumes. The promise stores only a pointer.\n\n### Buffered\n\nValues are moved into the buffer and have independent lifetime.\n\n## Exception Handling\n\nExceptions in the producer are caught by `promise_type::unhandled_exception()`\nand stored. On the next consumer resumption, `await_resume()` rethrows the\nexception.\n\n## Template Parameters\n\n### Single-parameter form\n\n```cpp\ngenerator<int>              // value_type=int, reference=int&&\ngenerator<const string&>    // value_type=string, reference=const string&\n```\n\n### Two-parameter form\n\n```cpp\ngenerator<string_view, string>  // Return string_view, store string\n```\n\nAllows proxy reference pattern: producer yields `string`, consumer receives\n`string_view`.\n\n## Performance\n\nPerformance characteristics measured using benchmarks in `tests/perf/coroutine_perf.cc`.\n\n### Benchmark Results\n\nEach test generates 100 integers per iteration (Release mode with `-O2`):\n\n| test                                |           runtime |    allocs |      inst |    cycles |\n| -                                   |                -: |        -: |        -: |        -: |\n| coroutine_test.unbuffered_generator |  308ns ± 0.15%    |     2.000 |   9356.47 |    1639.1 |\n| coroutine_test.buffered_generator   |  200ns ± 0.14%    |     2.000 |   6259.26 |    1069.6 |\n\n**Unbuffered generator:**\n- One suspension per element (100 suspensions per iteration)\n- Zero-copy: stores pointer to value in producer's stack frame\n- ~3.1ns per element (308ns / 100)\n\n**Buffered generator** (using `circular_buffer_fixed_capacity<int, 16>`):\n- Amortized suspensions (~6-7 suspensions per iteration with buffer size 16)\n- Moves elements into fixed-capacity buffer (zero heap allocations)\n- **35% faster than unbuffered** due to amortized suspension overhead\n- ~2.0ns per element (200ns / 100)\n\n**Important: Container choice is critical for buffered generator performance!**\n\nUsing `std::vector` instead of `circular_buffer_fixed_capacity` results in:\n- **2.6x slower performance** (524ns vs 200ns)\n- **2x more allocations** (4 vs 2) due to dynamic memory\n- **2x more instructions** (13,088 vs 6,259)\n\nThe heap allocation overhead completely negates the batching benefit. Always use fixed-capacity containers like `circular_buffer_fixed_capacity` for best performance.\n\n### Choosing Between Variants\n\nThe choice depends on multiple factors:\n\n**Use unbuffered when:**\n- Element moves are expensive (large objects, non-trivial move constructors)\n- Latency is critical (need first element ASAP)\n- Memory pressure is a concern (no buffering overhead)\n- Elements are naturally references to existing data\n\n**Use buffered when:**\n- Throughput matters more than latency\n- Elements are cheap to move (integers, small PODs)\n- Producer can generate elements in batches\n- Using a fixed-capacity container (avoids heap allocations)\n\n**Note:** The buffered variant's performance advantage becomes more pronounced with:\n- Larger element counts (more amortization)\n- Fixed-capacity containers like `circular_buffer_fixed_capacity` (no allocations)\n- Higher element generation cost in the producer\n- Natural batching in the data source\n"
  },
  {
    "path": "doc/htmlsplit.py",
    "content": "#!/usr/bin/env python3\n\n# This script takes the single-page HTML output from pandoc - tutorial.html -\n# and splits it into many pages in split/: one page index.html for the table\n# of contents, and an additional page for each chapter. We make sure that\n# links from the TOC to each chapter, and also links across chapters,\n# continue to work correctly, and also had links from each chapter back to\n# the TOC, as well as to the next and previous chapters.\n\n\n# Copyright (C) 2018 ScyllaDB.\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nfrom xml.etree import ElementTree\nimport argparse\nimport copy\nimport os\n\n# chapter number to chapter title\ntitles = {}\n# section id => chapter number\nsections = {}\n\n\ndef add_elem_to_body(tree, e):\n    body = next(tree.iterfind('./body'))\n    body.append(e)\n\n\ndef add_nav_to_body(tree, chap_num):\n    body = next(tree.iterfind('./body'))\n\n    nav = ElementTree.SubElement(body, 'div')\n    e = ElementTree.SubElement(nav, 'a',\n                               href='index.html')\n    e.text = 'Back to table of contents'\n    e.tail = '.'\n    prev_index = chap_num - 1\n    if prev_index in titles:\n        e.tail += \" Previous: \"\n        prev_title = titles[prev_index]\n        e = ElementTree.SubElement(nav, 'a',\n                                   href=f'{prev_index}.html')\n        e.text = f'{prev_index} {prev_title}'\n        e.tail = '.'\n    next_index = chap_num + 1\n    if next_index in titles:\n        e.tail += \" Next: \"\n        next_title = titles[next_index]\n        e = ElementTree.SubElement(nav, 'a',\n                                   href=f'{next_index}.html')\n        e.text = f'{next_index} {next_title}'\n        e.tail = '.'\n\n\ndef handle_toc(toc):\n    for chap in toc.iterfind('./ul/li'):\n        chap_href_elem = next(chap.iterfind('./a[@href]'))\n        chap_num_elem = next(chap_href_elem.iterfind(\n            './span[@class=\"toc-section-number\"]'))\n        # For chapters, remember the mapping from number to name in the\n        # map \"titles\", so we can use them later in links to next and\n        # previous chapter\n        chap_num = int(chap_num_elem.text)\n        titles[chap_num] = chap_num_elem.tail.strip()\n\n        # For all sections, remember the mapping from name-with-dashes\n        # to the chapter number they are in in \"sections\". We need this\n        # to support links to other sections.\n        href = chap_href_elem.get('href')\n        sections[href] = chap_num\n        for section in chap.iterfind('.//ul/li/a[@href]'):\n            href = section.get('href')\n            # replace the link to '#section' with number N.M to chapterN#section\n            if href.startswith('#'):\n                sections[href] = chap_num\n\n\ndef fix_links(e):\n    for link in e.findall('.//a[@href]'):\n        href = link.get('href')\n        if href.startswith('#') and href in sections:\n            # In a chapter we can have a link to a different subsection, which\n            # looks like <a href=\"#some-title\">Some title</A>. We need to\n            # replace this to refer to the right file after the split.\n            chap_num = sections[href]\n            link.set('href', f'{chap_num}.html{href}')\n\n\ndef remove_ns_prefix(tree):\n    prefix = '{http://www.w3.org/1999/xhtml}'\n    for e in tree.iter():\n        if e.tag.startswith(prefix):\n            e.tag = e.tag[len(prefix):]\n\n\ndef get_chap_num(element):\n    data_num = e.get('data-number')\n    if data_num:\n        return int(data_num)\n    data_num = e.findtext('./span[@class=\"header-section-number\"]')\n    if data_num:\n        return int(data_num)\n    assert data_num, \"section number not found\"\n\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--input')\nparser.add_argument('--output-dir')\nargs = parser.parse_args()\n\ntree = ElementTree.parse(args.input)\nfor e in tree.iter():\n    remove_ns_prefix(e)\ntemplate = copy.deepcopy(tree.getroot())\ntemplate_body = next(template.iterfind('./body'))\ntemplate_body.clear()\n\n# iterate through the children elements in body\n# body element is composed of\n#  - header\n#  - toc\n#  - h1,h2,p,...\n# h1 marks the beginning of a chapter\n\nchap_num = 0\nchap_tree = None\nfor e in next(tree.iterfind('./body')):\n    if e.tag == 'header':\n        template_body.append(e)\n    elif e.get('id') == 'TOC':\n        handle_toc(e)\n        fix_links(e)\n        toc_tree = ElementTree.ElementTree(copy.deepcopy(template))\n        add_elem_to_body(toc_tree, e)\n        toc_tree.write(os.path.join(args.output_dir, 'index.html'),\n                       method='html')\n    elif e.tag == 'h1':\n        assert titles\n        assert sections\n        if chap_num > 0:\n            add_nav_to_body(chap_tree, chap_num)\n            chap_tree.write(os.path.join(args.output_dir, f'{chap_num}.html'),\n                            method='html')\n        chap_num = get_chap_num(e)\n        chap_tree = ElementTree.ElementTree(copy.deepcopy(template))\n        add_nav_to_body(chap_tree, chap_num)\n        add_elem_to_body(chap_tree, e)\n    else:\n        assert chap_tree is not None\n        fix_links(e)\n        add_elem_to_body(chap_tree, e)\n\nadd_nav_to_body(chap_tree, chap_num)\nchap_tree.write(os.path.join(args.output_dir, f'{chap_num}.html'),\n                method='html')\n"
  },
  {
    "path": "doc/io-properties-file.md",
    "content": "# Specifying the I/O properties of a system\n\nThe I/O properties of a system can be specified as a YAML string, by\nusing the option --io-properties, or as a YAML file with the option\n--io-properties-file.\n\nThe expected format starts with a map of sections at the top level.\nCurrently only `disks` is supported.\n\n## The disks section\n\nInside the `disks` section, the user can specify a list of mount points.\n\nFor each mount point, 4 properties have to be specified (none are\noptional):\n\n* `read_iops`: read IOPS speed of the device\n* `read_bandwidth`: read bandwidth speed of the device\n* `write_iops`: write IOPS speed of the device\n* `write_bandwidth`: write bandwidth speed of the device\n\n\nAdditionally the following optional properties can be added:\n\n* `read_saturation_length`: read buffer length to saturate the device throughput\n* `write_saturation_length`: write buffer length to saturate the device throughput\n* `physical_block_size`: override for the physical block size of the device (in bytes).\n  This is used as the write alignment to avoid hardware-level read-modify-write operations.\n  Some devices mis-report their physical block size, so this override can be used to\n  specify the correct value\n\nThose quantities can be specified in raw form, or followed with a\nsuffix (k, M, G, or T).\n\nExample:\n\n```\ndisks:\n  - mountpoint: /var/lib/some_seastar_app\n    read_iops: 95000\n    read_bandwidth: 545M\n    write_iops: 85000\n    write_bandwidth: 510M\n    write_saturation_length: 64k\n```\n\nOptionally, instead of the \"mountpoint\" there can be the \"mountpoints\" (plural)\nentry in the list element being a list itself and listing more than one path. As\na result all the listed mountpoints will share the corresponding internal IO queue.\n\nExample:\n\n```\ndisks:\n  - mountpoints:\n      - /var/lib/some_seastar_app/sub_one\n      - /var/lib/some_seastar_app/sub_two\n    read_iops: 95000\n    ...\n```\n\nAn example when this configuration is applicable can be an LVM set of volumes\nfrom one disk each being mounted at its own path. In that case, different mount\npoints would have different (virtual) block devices, yet, they will share the\nsame physical disk and thus need to run over one shared IO queue.\n"
  },
  {
    "path": "doc/io-scheduler.md",
    "content": "IO scheduler uses rate-limiter to throttle the amount of data it dispatches into the disk.\n\n# Basic math\n\nThe scheduler's main equation that models the disk behavior is\n\n$$ \\frac {bandwidth_r} {bandwidth_{r_{max}}} +\n   \\frac {bandwidth_w} {bandwidth_{w_{max}}} +\n   \\frac {iops_r} {iops_{r_{max}}} +\n   \\frac {iops_w} {iops_{w_{max}}} \\le 1.0 $$\n\nLet's say that\n\n$$ m_b = \\frac {bandwidth_{r_{max}}} {bandwidth_{w_{max}}} \\ \\ m_o = \\frac {iops_{r_{max}}} {iops_{w_{max}}} $$\n\nSince\n\n$$ bandwidth_x = \\frac {d}{dt} bytes_x \\ \\ iops_x = \\frac {d}{dt} ios_x $$\n\nThe main equation turns into\n\n$$ \\frac {d}{dt} \\( \\frac {bytes_r + m_b \\times bytes_w} {bandwidth_{r_{max}}} + \\frac {ios_r + m_o \\times ios_w} {iops_{r_{max}}} \\) \\le 1.0 $$\n\nRequests are asigned a 2d value of _{1, bytes}_ for reads and _{m<sub>o</sub>, m<sub>b</sub> * bytes}_ for writes called \"tickets\"\n\nThe \"normalization\" operation is defined as\n\n$$ N(ticket) = \\frac {ticket_0}{iops_{r_{max}}} + \\frac {ticket_1}{bandwidth_{r_{max}}} $$\n\nWith that the main equation turns into\n\n$$ \\frac {d}{dt} \\sum_{ticket} N(ticket) \\le 1.0 $$\n\nThe time-derivative limitation is then implemented with the token bucket algorithm\n\n# Token bucket\n\nThe algorithm creates token bucket with the refill rate of _1.0_ and each request wants to carry the fractional token value of _tokens = N(ticket)_ with _ticket_ defined above\n\nThe bucket additionally requires the _limit_ parameter which is the maximum number of tokens the bucket may hold. This value is calculated using the _io_latency_goal_ parameter, to be the amount of tokens accumulated for the _io_latency_goal_ duration\n\n# Slowing down the flow\n\nLet's assume we need to reduce the rate of request run-time. This means that we want to\n\n$$ \\frac {d}{dt} \\( \\frac {bytes_r + m_b \\times bytes_w} {bandwidth_{r_{max}} \\times \\alpha} + \\frac {ios_r + m_o \\times ios_w} {iops_{r_{max}} \\times \\beta} \\) \\le 1.0 \\ \\  \\alpha \\le 1.0 \\ \\beta \\le 1.0 $$\n\nThere are many ways of selecting which of IOPS or bandwidth to reduce, the \"general\" slowing down may assume that\n\n$$ \\alpha = \\beta = \\frac {1}{\\gamma} , \\ \\ \\ \\gamma \\ge 1.0 $$\n\nThe main equation then turns into\n\n$$ \\frac {d}{dt} \\sum_{ticket} \\gamma \\times N(ticket) \\le 1.0 $$\n\nwhich in turn means, that we can just multiply the request cost (in tokens) by some number above 1.0\n\n# Detecting the slowdown\n\nLet's say that\n\n_d<sub>i</sub>_ -- the amount of requests dispatched at tick _i_\n_p<sub>i</sub>_ -- the amount of requests processed by disk at tick _i_\n_c<sub>i</sub>_ -- the amount of requests completed by reactor loop at tick _i_\n\nWe can observe _d<sub>i</sub>_ and _c<sub>i</sub>_ in the dispatcher, but _not_ the _p<sub>i</sub>_, because we don't have direct access to disks' queues\n\nAfter _n_ ticks we have\n\n_D<sub>n</sub>_ -- total amount of requests dispatched,\n_P<sub>n</sub>_ -- total amount of requests processed,\n_C<sub>n</sub>_ -- total amount of requests completed,\n\n$$ D_n = \\sum_{i=0}^n d_i $$\n\n$$ P_n = \\sum_{i=0}^n p_i $$\n\n$$ C_n = \\sum_{i=0}^n c_i $$\n\n* Disk cannot process more than it was dispatched, but it can process less \"accumulating\" a queue\n\n$$ d_i \\ge p_i   \\Rightarrow  D_n \\ge P_n$$\n\n* Reactor cannot complete more than it was processed by disk either\n\n$$ p_i \\ge c_i \\Rightarrow P_n \\ge C_n $$\n\n\nNote that _D<sub>n</sub> > P<sub>n</sub>_  means that disk is falling behind. Our goal is to make sure the disk doesn't do it and doesn't accumulate the queue, i.e. _D<sub>n</sub> = P<sub>n</sub>_, but we cannot observe _P<sub>n</sub>_ directly, only _C<sub>n</sub>_.\n\nNext\n\n$$ D_n - P_n = Qd_n \\ge 0 $$\n\n$$ P_n - C_n = Qc_n \\ge 0 $$\n\n$$ D_n - C_N = Qd_n + Qc_n \\ge 0 $$\n\n_Qd<sub>n</sub>_ is the queue accumulated in disk. We try to avoid it. _Qc<sub>n</sub>_ is the accumulated delta between processed (by disk) and completed (by reactor) requests. We cannot reliably say that it goes to zero over time, because there's always some unknown amount of processed but not yet completed requests. So the above formula contains two unknowns and we cannot solve it. Let's try the other way\n\n$$ \\frac {D_n} {P_n} = Rd_n \\ge 1 $$\n\n$$ \\frac {P_n} {C_n} = Rc_n \\ge 1 $$\n\n$$ \\frac {D_n} {C_n} = Rd_n \\times Rc_n \\ge 1 $$\n\n_Rd<sub>n</sub>_ is the ratio of dispatched to processed requests. If it's 1, we're OK, if it's greater, disk is accumulating the queue, we try to avoid it. _Rc<sub>n</sub>_ is the ratio between processed (by disk) and completed (by reactor) requests. It has a not immediately apparent, but very pretty property\n\n$$ \\lim_{n\\to\\infty} Rc_n = \\lim_{n\\to\\infty} \\frac {P_n} {C_n} = \\lim_{n\\to\\infty} \\frac {C_n + Qc_n} { C_n} = \\lim_{n\\to\\infty} \\(1 +  \\frac {Qc_n} {C_n} \\) = 1 + \\lim_{n\\to\\infty} \\frac {Qc_n} {C_n}$$\n\nThe _Qc<sub>n</sub>_ doesn't grow over time. It's non-zero, but it's upper bound by some value. Respectively\n\n$$ \\lim_{n\\to\\infty} Rc_n = 1 $$\n\n$$ \\lim_{n\\to\\infty} \\frac {D_n} {C_n} = \\lim_{n\\to\\infty} \\( Rd_n \\times Rc_n \\) =  \\lim_{n\\to\\infty} Rd_n  $$\n\nIOW -- we can say if the disk is accumulating the queue or not by observing the dispatched-to-completed (to _completed_, not _processed_) over a long enough time\n"
  },
  {
    "path": "doc/io-tester.md",
    "content": "## I/O Tester utility\n\nThe I/O Tester utility, `io_tester` generates a user-defined I/O pattern\nspanning one of multiple shards that is designed to simulate the I/O behavior\nof a complex Seastar application.\n\n# Running I/O tester:\n\nI/O tester takes the same options as Seastar, and those options may be used\nto test the behavior of I/O under the circumnstances established by those\noptions. For instance, one may adjust the `--task-quota-ms` option to see\nif that affects higher percentile latencies.\n\nAside from the usual seastar options, I/O tester accepts the following options:\n\n* `duration`: for how long to run the evaluation,\n* `storage`: a directory or a block device where to execute the test (it must be on XFS),\n* `conf`: the path to a YAML file describing the evaluation,\n* `keep-files`: a flag that indicates keeping test files - next run may re-use them.\n\n# Describing the evaluation\n\nThe evaluation is described in a YAML file that contains multiple classes.\nEach class spans jobs of similar characteristics in different shards and (for now)\nall jobs run concurrently.\n\nThe YAML file contains a list of maps where each element of the list describes a class.\nA class has some properties that are common to all elements of the class, and a nested map\nthat contain properties of a job (class instance in a shard)\n\nFor example:\n\n```\n- name: big_writes\n  type: seqread\n  shards: all\n  shard_info:\n    parallelism: 10\n    reqsize: 256kB\n    shares: 10\n    think_time: 0\n\n- name: vectorized_writes\n  type: randwrite\n  shards: all\n  shard_info:\n    parallelism: 10\n    reqsize: 64kB\n    vectorized: true\n    iov_count: 4        # 4 segments of 16kB each\n    shares: 10\n```\n\n* `name`: mandatory property, a string that identifies jobs of this class\n* `type`: mandatory property, one of seqread, seqwrite, randread, randwrite, append, cpu, unlink\n* `shards`: mandatory property, either the string \"all\" or a list of shards where this class should place jobs.\n* `data_size`: optional property, used to divide the available disk space between workloads. Each shard inside the workload uses its portion of the assigned space. If not specified 1GB is used.\n* `extent_allocation_size_hint`: optional property, allows setting the hint for allocation of extents for files. If not specified, then the size of file is used as hint.\n* `files_count`: optional property, relevant only for unlink job class - in such case it is required. Describes the number of files that need to be created during startup to be unlinked during evaluation. Describes files count per shard.\n\n> **_NOTE:_** the actual file size is always aligned to 1MB.\n> **_NOTE:_** if not properly aligned, then the extent allocation size hint is aligned to 128kB by seastar.\n\nThe properties under `shard_info` represent properties of the job that will\nbe replicated to each shard. All properties under `shard_info` are optional, and in case not specified, defaults are used.\n\n* `parallelism`: the amount of parallel requests this job will generate in a specific shard. Requests can be either active or thinking (see `think_time`)\n* `limit`: the maximum number of requests to send in this job. If not set, job sends requests throughout the whole `duration`\n* `rps`: the requests-per-second rate to apply to sending fibers. If unset, each fiber sends new request as soon as previous one completes\n* `batch`: the number of requests sent at a time when the `rps` mode is used. It's 1 by default\n* `reqsize` : (I/O loads only) the size of requests generated by this job\n* `shares` : how many shares requests in this job will have in the scheduler\n* `class`: name of the job to share the sched class with (used if `shares` is not set)\n* `think_time`: how long to wait before submitting another request in this job once one finishes.\n* `execution_time`: (cpu loads only) for how long to execute a CPU loop\n* `vectorized`: (I/O loads only) if true, use scatter-gather I/O with multiple iovec segments per request\n* `iov_count`: (I/O loads only) number of iovec segments when `vectorized` is true. The `reqsize` is split into `iov_count` uniform segments. Each segment must be at least 4096 bytes and properly aligned.\n\n# Example output\n\n```\n      Creating initial files...\n      Starting evaluation...\n      Shard  0\n       Class 0(big_writes: 10 shares, 262144-byte SEQ WRITE, 10 concurrent requests, NO think time)\n        Throughput         :   436556 KB/s\n        Lat average        :     5847 usec\n        Lat quantile=  0.5 :     2678 usec\n        Lat quantile= 0.95 :    13029 usec\n        Lat quantile= 0.99 :    20835 usec\n        Lat quantile=0.999 :   246090 usec\n        Lat max            :   450785 usec\n```\n\n# Future\n\nSome ideas for extending I/O tester:\n\n* allow properties like think time, request size, etc, to be specified as distributions instead of a fixed number\n* allow classes to have class-wide properties. For instance, we could define a class with parallelism of 100, and distribute those 100 requests over all shards in which this class is placed\n* allow some jobs to be executed sequentially in relationship to others, so we can have preparation jobs.\n* support other types, like sync, etc.\n* provide functionality similar to diskplorer.\n\n"
  },
  {
    "path": "doc/lambda-coroutine-fiasco.md",
    "content": "# The Lambda Coroutine Fiasco\n\nLambda coroutines and Seastar APIs that accept continuations interact badly. This\ndocument explains the bad interaction and how it is mitigated.\n\n## Lambda coroutines revisited\n\nA lambda coroutine is a lambda function that is also a coroutine due\nto the use of the coroutine keywords (typically co_await). A lambda\ncoroutine is notionally translated by the compiler into a struct with a\nfunction call operator:\n\n```cpp\n[captures] (arguments) -> seastar::future<> {\n    body\n    co_return result\n}\n```\n\nbecomes (more or less)\n\n```cpp\nstruct lambda {\n    captures;\n    seastar::future<> operator()(arguments) const {\n        body\n    }\n};\n```\n\n## Lambda coroutines and coroutine argument capture\n\nCoroutines, like lambdas, can capture variables from their enclosing\nscope. Additionally, coroutines capture their arguments, which can occur\nby value or by reference, depending on the argument's declaration.\n\nThe lambda's captures however are captured by reference. To understand why,\nconsider that the coroutine translation process notionally transforms a member function\n(`lambda::operator()`) to a free function:\n\n```cpp\n// before\nseastar::future<> lambda::operator()(arguments) const;\n\n// after\nseastar::future<> lambda_call_operator(const lambda& self, arguments);\n```\n\nThis transform means that the lambda structure, which contains all the captured variables,\nis itself captured by the coroutine by reference.\n\n## Interaction with Seastar APIs accepting continuations\n\nConsider a Seastar API that accepts a continuation, such as\n`seastar::future::then(Func continuation)`. The behavior\nis that `continuation` is moved or copied into a private memory\narea managed by `then()`. Sometime later, the continuation is\nexecuted (`Func::operator()`) and the memory area is freed.\nCrucially, the memory area is freed as soon as `Func::operator()`\nreturns, which can be before the future returned by it becomes\nready. However, the coroutine can access the lambda captures\nstored in this memory area after the future is returned and before\nit becomes ready. This is a use-after-free.\n\n## Solution (C++23 and up)\n\nWith \"deducing this\", the value category of the implicit object\nparameter to the lambda, which contains the captures, can be changed.\n\nA lambda of the form\n\n```cpp\n[captures...] (this auto, arguments...)\n```\n\nIs transformed to a function call operator of the form\n\n```cpp\nseastar::future<> lambda::operator()(this auto, arguments...);\n```\n\nWhich in turn is equivalent to the free function\n\n```cpp\nseastar::future<> lambda_call_operator(lambda, arguments...);\n```\n\nSince the function accepts the lambda struct containing the\ncapture groups by value, the capture groups are moved to the\ncoroutine frame and are not tied to the lifetime of the expression\nthat created the lambda.\n\n## Solution (pre C++23)\n\nThe solution is to avoid copying or moving the lambda into\nthe memory area managed by `seastar::future::then()`. Instead,\nthe lambda spends its life as a temporary. We then rely on C++\ntemporary lifetime extension rules to extend its life until the\nfuture returned is ready, at which point the captures can longer\nbe accessed.\n\n```cpp\n    co_await seastar::yield().then(seastar::coroutine::lambda([captures] () -> future<> {\n        co_await seastar::coroutine::maybe_yield();\n        // Can use `captures` here safely.\n    }));\n```\n\n`seastar::coroutine::lambda` is very similar to `std::reference_wrapper` (the\nonly difference is that it works with temporaries); it can be safely moved to\nthe memory area managed by `seastar::future::then()` since it's only used\nto call the real lambda, and then is safe to discard.\n\n## Alternative solution (pre C++23) when lifetime extension cannot be used.\n\nIf the lambda coroutine is not co_await'ed immediately, we cannot rely on\nlifetime extension and so we must name the coroutine and use `std::ref()` to\nrefer to it without copying it from the coroutine frame:\n\n```cpp\n    auto a_lambda = [captures] () -> future<> {\n        co_await seastar::coroutine::maybe_yield();\n        // Can use `captures` here safely.\n    };\n    auto f = seastar::yield().then(std::ref(a_lambda));\n    co_await std::move(f);\n```\n\n"
  },
  {
    "path": "doc/md2html",
    "content": "#!/bin/sh\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n# Copyright (C) 2018 ScyllaDB\n\ndir=$(dirname \"$0\")\n\ncase `pandoc --version` in\n\"pandoc 1.\"*)\n    SMART_OPT=--smart\n    SMART_EXT=\n    ;;\n*)\n    SMART_OPT=\n    SMART_EXT=+smart\nesac\npandoc --self-contained $SMART_OPT --toc -c ${dir}/template.css --number-sections -f markdown+pandoc_title_block+implicit_header_references$SMART_EXT -V lang=en --highlight-style tango \"$1\" -o \"$2\"\n"
  },
  {
    "path": "doc/md2pdf",
    "content": "#!/bin/sh\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n# Copyright (C) 2018 ScyllaDB\n\ndir=$(dirname \"$0\")\n\npandoc -f markdown+pandoc_title_block+implicit_header_references --highlight-style tango --template=${dir}/template.tex \"$1\" -o \"$2\"\n\n"
  },
  {
    "path": "doc/mini-tutorial.md",
    "content": "Futures and promises\n--------------------\n\nA *future* is a result of a computation that may not be available yet.\nExamples include:\n\n  * a data buffer that we are reading from the network\n  * the expiration of a timer\n  * the completion of a disk write\n  * the result computation that requires the values from\n    one or more other futures.\n\na *promise* is an object or function that provides you with a future,\nwith the expectation that it will fulfill the future.\n\nPromises and futures simplify asynchronous programming since they decouple\nthe event producer (the promise) and the event consumer (whoever uses the\nfuture).  Whether the promise is fulfilled before the future is consumed,\nor vice versa, does not change the outcome of the code.\n\nConsuming a future\n------------------\n\nYou consume a future by using its *then()* method, providing it with a\ncallback (typically a lambda).  For example, consider the following\noperation:\n\n```C++\nfuture<int> get();   // promises an int will be produced eventually\nfuture<> put(int)    // promises to store an int\n\nvoid f() {\n    get().then([] (int value) {\n        put(value + 1).then([] {\n            std::cout << \"value stored successfully\\n\";\n        });\n    });\n}\n```\n\nHere, we initiate a *get()* operation, requesting that when it completes, a\n*put()* operation will be scheduled with an incremented value.  We also\nrequest that when the *put()* completes, some text will be printed out.\n\nChaining futures\n----------------\n\nIf a *then()* lambda returns a future (call it x), then that *then()*\nwill return a future (call it y) that will receive the same value.  This\nremoves the need for nesting lambda blocks; for example the code above\ncould be rewritten as:\n\n```C++\nfuture<int> get();   // promises an int will be produced eventually\nfuture<> put(int)    // promises to store an int\n\nvoid f() {\n    get().then([] (int value) {\n        return put(value + 1);\n    }).then([] {\n        std::cout << \"value stored successfully\\n\";\n    });\n}\n```\n\nLoops\n-----\n\nLoops are achieved with a tail call; for example:\n\n```C++\nfuture<int> get();   // promises an int will be produced eventually\nfuture<> put(int)    // promises to store an int\n\nfuture<> loop_to(int end) {\n    if (value == end) {\n        return make_ready_future<>();\n    }\n    get().then([end] (int value) {\n        return put(value + 1);\n    }).then([end] {\n        return loop_to(end);\n    });\n}\n```\n\nThe *make_ready_future()* function returns a future that is already\navailable --- corresponding to the loop termination condition, where\nno further I/O needs to take place.\n\nUnder the hood\n--------------\n\nWhen the loop above runs, both *then* method calls execute immediately\n--- but without executing the bodies.  What happens is the following:\n\n1. `get()` is called, initiates the I/O operation, and allocates a\n   temporary structure (call it `f1`).\n2. The first `then()` call chains its body to `f1` and allocates\n   another temporary structure, `f2`.\n3. The second `then()` call chains its body to `f2`.\n\nAgain, all this runs immediately without waiting for anything.\n\nAfter the I/O operation initiated by `get()` completes, it calls the\ncontinuation stored in `f1`, calls it, and frees `f1`.  The continuation\ncalls `put()`, which initiates the I/O operation required to perform\nthe store, and allocates a temporary object `f12`, and chains some glue\ncode to it.\n\nAfter the I/O operation initiated by `put()` completes, it calls the\ncontinuation associated with `f12`, which simply tells it to call the\ncontinuation associated with `f2`.  This continuation simply calls\n`loop_to()`.  Both `f12` and `f2` are freed. `loop_to()` then calls\n`get()`, which starts the process all over again, allocating new versions\nof `f1` and `f2`.\n\nHandling exceptions\n-------------------\n\nIf a `.then()` clause throws an exception, the scheduler will catch it\nand cancel any dependent `.then()` clauses.  If you want to trap the\nexception, add a `.then_wrapped()` clause at the end:\n\n```C++\nfuture<buffer> receive();\nrequest parse(buffer buf);\nfuture<response> process(request req);\nfuture<> send(response resp);\n\nvoid f() {\n    receive().then([] (buffer buf) {\n        return process(parse(std::move(buf));\n    }).then([] (response resp) {\n        return send(std::move(resp));\n    }).then([] {\n        f();\n    }).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n        } catch (std::exception& e) {\n            // your handler goes here\n        }\n    });\n}\n```\n\nThe previous future is passed as a parameter to the lambda, and its value can\nbe inspected with `f.get()`. When the `get()` variable is called as a\nfunction, it will re-throw the exception that aborted processing, and you can\nthen apply any needed error handling.  It is essentially a transformation of\n\n```C++\nbuffer receive();\nrequest parse(buffer buf);\nresponse process(request req);\nvoid send(response resp);\n\nvoid f() {\n    try {\n        while (true) {\n            auto req = parse(receive());\n            auto resp = process(std::move(req));\n            send(std::move(resp));\n        }\n    } catch (std::exception& e) {\n        // your handler goes here\n    }\n}\n```\n\nNote, however, that the `.then_wrapped()` clause will be scheduled both when\nexception occurs or not. Therefore, the mere fact that `.then_wrapped()` is\nexecuted does not mean that an exception was thrown. Only the execution of the\ncatch block can guarantee that.\n\n\nThis is shown below:\n\n```C++\n\nfuture<my_type> receive();\n\nvoid f() {\n    receive().then_wrapped([] (future<my_type> f) {\n        try {\n            my_type x = f.get();\n            return do_something(x);\n        } catch (std::exception& e) {\n            // your handler goes here\n        }\n    });\n}\n```\n### Setup notes\n\nSeaStar is a high performance framework and tuned to get the best\nperformance by default. As such, we're tuned towards polling vs interrupt\ndriven. Our assumption is that applications written for SeaStar will be\nbusy handling 100,000 IOPS and beyond. Polling means that each of our\ncores will consume 100% cpu even when no work is given to it.\n\n"
  },
  {
    "path": "doc/native-stack.md",
    "content": "Seastar Native TCP/IP Stack\n---------------------------\n\nSeastar comes with a native, sharded TCP/IP stack.  Usually it is used with the [DPDK](building-dpdk.md) environment, but there are also vhost drivers for testing in a development environment.\n\nTo enable the native network stack, pass the `--network-stack native` parameter to a seastar application.\n\nTo test the native stack without dpdk, install and start the `libvirt` daemon.  This will create a bridge device named `virbr0`, which seastar will connect to.\n\nSeastar's vhost driver will need a tap device to connect to.  The scripts `scripts/tap.sh` will set up a tap device and bind it to `virbr0`:\n\n\t$ sh ./scripts/tap.sh\n\tSet 'tap0' nonpersistent\n\tbridge name\tbridge id\t\tSTP enabled\tinterfaces\n\tvirbr0\t\t8000.5254008be729\tno\t\ttap0\n\t\t\t\t\t\t\t\tvirbr0-nic\n\tvirbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500\n\t        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255\n\t        ether 52:54:00:8b:e7:29  txqueuelen 1000  (Ethernet)\n\t        RX packets 384938  bytes 21866184 (20.8 MiB)\n\t        RX errors 0  dropped 0  overruns 0  frame 0\n\t        TX packets 547098  bytes 2508723098 (2.3 GiB)\n\t        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0\n\nYou can now run a seastar application; for example, the http server:\n\n\t$ ./build/release/apps/httpd/httpd --network-stack native\n\tDHCP sending discover\n\tDHCP Got offer for 192.168.122.18\n\tDHCP sending request for 192.168.122.18\n\tDHCP Got ack on request\n\tDHCP  ip: 192.168.122.18\n\tDHCP  nm: 255.255.255.0\n\tDHCP  gw: 192.168.122.1\n\tSeastar HTTP server listening on port 10000 ...\n\nYou can now ping the IP address shown (`192.168.122.18`) or connect to it:\n\n\t$ ping 192.168.122.18\n\tPING 192.168.122.18 (192.168.122.18) 56(84) bytes of data.\n\t64 bytes from 192.168.122.18: icmp_seq=1 ttl=64 time=0.160 ms\n\t64 bytes from 192.168.122.18: icmp_seq=2 ttl=64 time=0.110 ms\n\t64 bytes from 192.168.122.18: icmp_seq=3 ttl=64 time=0.116 ms\n\t64 bytes from 192.168.122.18: icmp_seq=4 ttl=64 time=0.112 ms\n\t64 bytes from 192.168.122.18: icmp_seq=5 ttl=64 time=0.093 ms\n\t64 bytes from 192.168.122.18: icmp_seq=6 ttl=64 time=0.108 ms\n\t^C\n\t--- 192.168.122.18 ping statistics ---\n\t6 packets transmitted, 6 received, 0% packet loss, time 4999ms\n\trtt min/avg/max/mdev = 0.093/0.116/0.160/0.023 ms\n\n\t$ curl http://192.168.122.18:10000/\n\t\"hello\"\n\n"
  },
  {
    "path": "doc/network-configuration.md",
    "content": "Network Configuration\n---------------------\n\nIn order to support multiple network devices in Seastar, new Network Configuration format has to be introduced.\nNew format is Yaml based and contains a list of network devices along with its IP parameters and optional (if DPDK is used) hardware parameters.\nThe new configuration can be provided either by command line with --net-config option or by configuration file by --net-config-file.\n\n### DPDK access\nNetwork device (called port in DPDK) can be accessed by either port index ( zero based index of device shown by dpdk-setup.sh ) or its PCI address (shown by lspci, lshw tools)\n\nExample config line with pci address given:\n\n```\neth0: {pci_address: 0000:06:00.0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }\n```\n\nExample config line with port index given:\n\n```\neth0: {port_index: 0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }\n```\n\nPlease note that device name - eth0 above, is not used by DPDK itself, it remains only for configuration consistency.\nThe hardware configuration has to be specied in the same way accross all network devices, so for example if pci_address is specified for one network device, port_index cannot be specified for any other.\n\n\n## Non-DPDK access\nWhen there is no pci_address neither port_index defined then Non-DPDK access is assumed provided by libvirt deamon (see native-stack.md), eg:\n\n```\nvirbr0: { ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 }\n```\n\n## Other hardware related options\n\nThere are other hardware related optional options, some of the are common for both DPDK and non-DPDK mode\n- lro ( large receive offload ), boolean, deafult true\n- tso ( tcp segmentation offload ), boolean, default true\n- ufo ( udp fragmentation offload ), boolean, default true\n- hw-fc ( hardware flow control ), boolean, default true,\n- csum-offload ( IP checksum offload ), boolean, default true\n- ring-size ( device ring buffer size ), unsigned, default 256, libvirt only\n- event-index ( VIRTIO_RING_F_EVENT_IDX\tsupport enabled ), boolean, default true, libvirt only\n\n\n## DHCP\n\nIP configuration can be set by either IP/gateway/netmask (as seen in examples above), but also by DHCP with dhcp=true setting, eg:\n\n```\neth0: {pci_address: 0000:06:00.0, dhcp=true}\n```\n\nDHCP can be selected per network device, so it would also be perfectly valid to define dhcp for eth0, but ip/netmask/gateway for eth1.\n\n\n## Multiple devices\nConfiguration formay for multiple devices is a comma separated lists of single devices with added YAML brackets, eg:\n\n```\n{virbr0: { ip: 192.168.100.10, gateway: 192.168.100.1, netmask: 255.255.255.0 } , virbr1: { dhcp: true } }\n```\n\n\n\n"
  },
  {
    "path": "doc/network-connection-load-balancing.md",
    "content": "# Motivation\n\nIn sharded systems like seastar it is important for work to be\ndistributed equally between all shards to achieve maximum performance\nfrom the system. Networking subsystem has its part in distributing work\nequally. For instance if on a server all connections will be served by\nsingle shard only, the system will be working with the speed of this\none shard and all other shards will be underutilized.\n\n# Common ways to distribute work received over network between shards\n\nTwo common ways to distribute work between shards are:\n - do the work at a shard that received it\n - shard that does actual work depends on a data been processed\n   (one way to do it is to hash(data) % smp_count = shard,\n    another way is to bind shards to different server addresses)\n\n# Load Balancing\n\nThose two ways asks for different strategy to distribute connections\nbetween shards.  The first one will work best if each cpu will have the\nsame amount of connections (assuming each connection gets same amount of\nworks) the second will work best if data will arrive to a shard where\nit is going to be processed and actual connection distribution does\nnot matter.\n\nSeastar's posix stack supports both of those strategies. Desired\none can be chosen by specifying load balancing algorithm in\nlisten_options provided to reactor::listen() call. Available options\nare:\n\n- load_balancing_algorithm::connection_distribution\n\n  Make sure that new connection will be placed to a shard with smallest\n  amount of connections of the same type.\n\n-  load_balancing_algorithm::port\n\n   Destination shard is chosen as a function of client's local port:\n   shard = port_number % num_shards.  This allows a client to make sure that\n   a connection will be processed by a specific shard by choosing its local\n   port accordingly (the knowledge about amount of shards in the server is\n   needed and can be negotiated by different channel).\n\n- load_balancing_algorithm::fixed\n\n  Destination shard is statically configured in listen_options::fixed_cpu. This\n  allows a client to make sure that a connection to a server address will be\n  established in a specific shard, without any further negotiations.\n"
  },
  {
    "path": "doc/prometheus.md",
    "content": "# The Prometheus Protocol\n\nSeastar supports the Prometheus protocol for metrics reporting.\nSupported exposition formats are the 0.0.4 text and protocol buffer formats.\n\nMore on the formats can be found at the [Prometheus documentations](https://prometheus.io/docs/instrumenting/exposition_formats/)\n\nBy default, Seastar would listen on port `9180` and the `localhost`.\n\nSee the Seastar configuration documentation on how to change the default configuration.\n\nSeastar would reply based on the content type header, so pointing your browser to:\n`http://localhost:9180/metrics/` will return a text representation of the metrics with their documentation.\n\nStarting from Prometheus 2.0, the binary protocol is no longer supported.\nWhile seastar still supports the binary protocol, it would be deprecated in a future release.\n\n## Querying subset of the metrics\nSeastar supports querying for a subset of the metrics by their names and labels.\n\nFiltering is recommended when you only need a subset of the available metrics, especially\nin systems with many metrics. Filtering reduces the processing overhead on the Seastar\napplication, results in smaller HTTP responses, and decreases the load on your Prometheus\nserver when scraping and storing metrics.\n\n### Filtering by a metric name\nUse the `__name__` query parameter to select according to a metric name or a prefix.\n\nFor example, to get all the http metrics, point your browser to:\n`http://localhost:9180/metrics?__name__=http*` note the asterisk symbol following the http.\nFiltering by name only supports prefix matching.\n\nTo query for only the http requests served metric, point your browser to `http://localhost:9180/metrics?__name__=httpd_requests_served`\n\nYou can use either the full metric name as it appears in the output (e.g., `seastar_httpd_requests_served`)\nor the name without the prefix (e.g., `httpd_requests_served`). Both forms work identically.\n\n#### Multiple name filters\nYou can specify multiple `__name__` parameters to query for several specific metrics at once.\nA metric is included if it matches any of the specified names.\n\nFor example, to get both the http requests and connections metrics:\n`http://localhost:9180/metrics?__name__=httpd_requests_served&__name__=httpd_connections_total`\n\nThis also works with prefix matching:\n`http://localhost:9180/metrics?__name__=httpd_requests*&__name__=httpd_connections*`\n\n### Filtering by a label value\nThe Prometheus protocol uses labels to differentiate the characteristics of the thing that is being measured.\nFor example, in Seastar, it is common to report each metric per shard and add a `shard` label to the metric.\n\nYou can filter by any label using regular expressions. If you use multiple labels in your query, all conditions should be met.\nA missing label is considered an empty string. The expression should match the entire label value,\nto match a missing label, you can use `label=` or `label=^$`.\n\nHere are a few examples:\n\nTo return all metrics from shard 1 or shard 0:\nhttp://localhost:9180/metrics?shard=1|0\n\nTo get all metrics without a `service` label:\nhttp://localhost:9180/metrics?service=\n\nTo get all metrics with a `service` label equals `prometheus` and from shard `0`:\nhttp://localhost:9180/metrics?service=prometheus&shard=0\n\n## Remove the help lines\nSending the help associated with each metric on each request is an overhead.\nPrometheus itself does not use those help lines.\nSeastar supports an option to remove those lines from the metrics output using the `__help__` query parameter.\nTo remove the help lines set `__help__=false`\nfor example:\n`http://localhost:9180/metrics?__help__=false`\n\n## Aggregation\nIn Seastar, metrics can be defined with implicit aggregation by specific labels,\nwhich occurs at query time. This feature is useful, for instance to define metrics\nper shard or even more finely grained per an application-defined entity while reporting\nthem in a more aggregated manner, such as sum or histogram per server.\n\nHowever, there are times when it is necessary to inspect the fine-grained metrics.\nThis can be achieved by adding `__aggregate__=false` to the query string. For example:\n`http://localhost:9180/metrics?__aggregate__=false`\n\n### Configuring the Prometheus server for picking specific metrics\nThe [Prometheus configuration](https://prometheus.io/docs/prometheus/1.8/configuration/configuration/) describes the general Prometheus configuration.\n\nTo specify a specific metric or metrics add a `metrics_path` to the scrape config in the prometheus.yml file\n\nFor example, the following scrape config will query for all the http metrics:\n\n```\n  scrape_configs:\n    - job_name: http\n      honor_labels: true\n      metrics_path: /metrics\n      params:\n        __name__: ['http*']\n```\n\nTo query for multiple specific metrics, list them in the array:\n\n```\n  scrape_configs:\n    - job_name: selected_metrics\n      honor_labels: true\n      metrics_path: /metrics\n      params:\n        __name__: ['httpd_requests_served', 'httpd_connections_total', 'scheduler*']\n```\n\n"
  },
  {
    "path": "doc/rpc-compression.md",
    "content": "# RPC provided compression infrastructure\n\n## Compression algorithm negotiation\n\nRPC protocol only defines `COMPRESS` feature bit but does not define format of its data.\nIf application supports multiple compression algorithms it may use the data for algorithm\nnegotiation. RPC provides convenience class `multi_algo_compressor_factory` to do it\nso that each application will not have to re-implement the same logic. The class gets list\nof supported compression algorithms and send them as comma separated list in the client `COMPRESS`\nfeature payload. On receiving of the list it matches common algorithm between client and server.\nIn case there is more than one the order of algorithms in client's list is considered to be a tie\nbreaker (first algorithm wins). Once a compressor is chosen by the server, it puts the identifier of\nthis in the returned `COMPRESS` feature payload, informing the client of which algorithm should be used\nfor the connection.\n\n## Compression algorithms\n\n### `LZ4` compressor\n\nThis compressor uses LZ4 to compress and decompress RPC messages. It requires all memory buffers to be contiguous, which may force it to temporarily linearise fragmented messages. LZ4 is fast enough to often make the cost of those copies not negligible compared to the cost of the whole compression or decompression routine. Therefore, this algorithm is best suited if there is an upper bound of the message size and they are expected to fit in a single fragment of input and output memory buffers.\n\n### `LZ4_FRAGMENTED` compressor\n\nThis compressor uses LZ4 streaming interface to compress and decompress even large messages without linearising them. The LZ4 streaming routines tend to be slower than the basic ones and the general logic for handling buffers is more complex, so this compressor is best suited only when there is no clear upper bound on the message size or if the messages are expected to be fragmented.\n\nInternally, the compressor processes data in a 32 kB chunks and tries to avoid unnecessary copies as much as possible. It is therefore, recommended, that the application uses memory buffer fragment sizes that are an integral multiple of 32 kB.\n"
  },
  {
    "path": "doc/rpc-streaming.md",
    "content": "# RPC streaming\n\n## Streaming API\n\n### Sink and Source\n\nBasic element of streaming API is `rpc::sink` and `rpc::source`. The former\nis used to send data and the later is to receive it. Client and server\nhas their own pair of sink and source. `rpc::sink` and `rpc::source` are\ntemplated classes where template parameters describe a type of the data\nthat is sent/received. For instance the sink that is used to send messages\ncontaining `int` and `long` will be of a type `rpc::sink<int, long>`.  The\nopposite end of the stream will have a source of the type `rpc::source<int, long>`\nwhich will be used to receive those messages. Messages are received at a\nsource as `std::optional` containing an actual message as an `std::tuple`. Unengaged\noptional means EOS (end of stream) - the stream was closed by a peer. If\nerror happen before EOS is received a receiver cannot be sure it received all\nthe data.\n\nTo send the data using `rpc::source<int, long>` one can write (assuming `seastar::async` context):\n\n```cpp\n      while (has_data()) {\n          int data1 = get_data1();\n          long data2 = get_data2();\n          sink(data1, data2).get(); // sends data\n      }\n      sink.close().get(); // closes stream\n```\n\nTo receive:\n\n```cpp\n      while (true) {\n          std:optional<std::tuple<int, long>> data = source().get();\n          if (!data) {\n             // unengaged optional means EOS\n             break;\n          } else {\n             auto [data1, data2] = *data;\n             // process data\n          }\n      }\n```\n\n### Creating a stream\n\nTo open an RPC stream one needs RPC client to be created already. The stream\nwill be associated with the client and will be aborted if the client is closed\nbefore streaming is. Given RPC client `rc`, and a `serializer` class that models the Serializer concept (as explained in the rpc::protocol class), one creates `rpc::sink` as follows\n(again assuming `seastar::async` context):\n\n```cpp\n    rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();\n```\n\nNow the client has the sink that can be used for streaming data to\na server, but how the server will get a corresponding `rpc::source` to\nread it? For that the sink should be passed to the server by an RPC\ncall. To receive a sink a server should register an RPC handler that will\nbe used to receive it along with any auxiliary information deemed necessary.\nTo receive the sink above one may register an RPC handler like that:\n\n```cpp\n    rpc_proto.register_handler(1, [] (int aux_data, rpc::source<int, long> source) {\n    });\n```\n\nNotice that `rpc::sink` is received as an `rpc::source` since at the server\nside it will be used for receive. Now all is left to do is for the client to\ninvoke this RPC handler with aux_data and the sink.\n\nBut what about communicating in another direction: from a server to a\nclient. For that a server also has to have a sink and a client has to have\na source and since messages in this direction may be of a different type\nthan from client to server the sink and the source may be of a different\ntype as well.\n\nServer initiates creation of a communication channel in another direction.\nIt does this by creating a sink from the source it receives and returning the sink\nfrom RPC handler which will cause it to be received as a source by a client. Lets look\nat the full example where server want to send message containing sstring to a client.\n\nServer handler will look like that:\n\n```cpp\n    rpc_proto.register_handler(1, [] (int aux_data, rpc::source<int, long> source) {\n        rpc::sink<sstring> sink = source.make_sink<serializer, sstring>();\n        // use sink and source asynchronously\n        return sink;\n    });\n```\n\nClient code will be:\n\n```cpp\n   auto rpc_call = rpc_proto.make_client<rpc::source<sstring> (int, rpc::sink<int>)>(1);\n   rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();\n   rpc::source<sstring> source = rpc_call(rc, aux_data, sink).get();\n   // use sink and source here\n```\n\n## Implementation notes\n\n### RPC stream creation\n\nRPC stream is implemented as a separate TCP connection. RPC server knows that a connection\nwill be used for streaming if during RPC negotiation `Stream parent` feature is present.\nThe feature will contain ID of an RPC client that was used to create the stream.\n\nSo in the example from previous chapter:\n\n```cpp\n    rpc::sink<int, long> sink = rc.make_stream_sink<serializer, int, long>().get();\n```\n\nthe call will initiate a new TCP connection to the same server `rc` is connected to. During RPC\nprotocol negotiation this connection will have `Stream parent` feature with `rc`'s ID as a value.\n\n### Passing sink/source over RPC call\n\nWhen `rpc::sink` is sent over RPC call it is serialized as its connection ID. Server's RPC handler\nthen lookups the connection and creates an `rpc::source` from it. When RPC handler returns `rpc::sink`\nthe same happens in other direction.\n"
  },
  {
    "path": "doc/rpc.md",
    "content": "# RPC protocol\n\n## Data encoding\n\nAll integral data is encoded in little endian format.\n\n## Protocol negotiation\n\nThe negotiation works by exchanging negotiation frame immediately after connection establishment. The negotiation frame format is:\n\n    uint8_t magic[8] = SSTARRPC\n    uint32_t len\n    uint8_t data[len]\n\nThe negotiation frame data is itself composed of multiple records, one for each feature number present.  Feature numbers begin at zero and will be defined by later versions of this document.\n\n\n     struct negotiation_frame_feature_record {\n         uint32_t feature_number;\n         uint32_t len;\n         uint8_t data[len];\n     }\n\nA `negotiation_frame_feature_record` signals that an optional feature is present in the client, and can contain additional feature-specific data.  The feature number will be omitted in a server response if an optional feature is declined by the server.\n\nActual negotiation looks like this:\n\n         Client                                  Server\n    --------------------------------------------------------------------------------------------------\n    send negotiation frame\n                                        recv frame\n                                        check magic (disconnect if magic is not SSTARRPC)\n                                        send negotiation frame back\n    recv frame\n    check magic (disconnect if magic is not SSTARRPC)\n\n### Supported features\n\n#### Compression\n    feature_number:  0\n    data          :  opaque data that is passed to a compressor factory\n                     provided by an application. Compressor factory is\n                     responsible for negotiation of compression algorithm.\n\n    If compression is negotiated request and response frames are encapsulated in a compressed frame.\n\n#### Timeout propagation\n    feature_number:  1\n    data          :  none\n\n    If timeout propagation is negotiated request frame has additional 8 bytes that hold timeout value\n    for a request in milliseconds. Zero value means that timeout value was not specified.\n    If timeout is specified and server cannot handle the request in specified time frame it my choose\n    to not send the reply back (sending it back will not be an error either).\n\n#### Connection ID\n    feature_number: 2\n    uint64_t conenction_id  : RPC connection ID\n\n    Server assigns unique connection ID for each connection and sends it to a client using\n    this feature.\n\n#### Stream parent\n    feature_number: 3\n    uint64_t connection_id : RPC connection ID representing a parent of the stream\n\n    If this feature is present it means that the connection is not regular RPC connection\n    but stream connection. If parent connection is closed or aborted all streams belonging\n    to it will be closed as well.\n\n    Stream connection is a connection that allows bidirectional flow of bytes which may carry one or\n    more messages in each direction. Stream connection should be explicitly closed by both client and\n    server. Closing is done by sending special EOS frame (described below).\n\n\n#### Isolation\n    feature number: 4\n    uint32_t isolation_cookie_len\n    uint8_t isolation_cookie[len]\n\n    The `isolation_cookie` field is used by the server to select a\n    `seastar::scheduling_group` (or equivalent in another implementation) that\n    will run this connection. In the future it will also be used for rpc buffer\n    isolation, to avoid rpc traffic in one isolation group from starving another.\n\n    The server does not directly assign meaning to values of `isolation_cookie`;\n    instead, the interpretation is left to user code.\n\n#### Handler duration\n    feature number: 5\n    data: none\n\n    Asks server to send \"extended\" response that includes the handler duration time. See\n    the response frame description for more details\n\n\n##### Compressed frame format\n    uint32_t len\n    uint8_t compressed_data[len]\n\n    After compressed_data is uncompressed, it becomes a regular request, response or streaming frame.\n\n    As a special case, it is allowed to send a compressed frame of size 0 (pre-compression).\n    Such a frame will be a no-op on the receiver.\n    (This can be used as a means of communication between the compressors themselves.\n    If a compressor wants to send some metadata to its peer, it can send a no-op frame,\n    and prepend the metadata as a compressor-specific header.)\n\n## Request frame format\n    uint64_t timeout_in_ms - only present if timeout propagation is negotiated\n    uint64_t verb_type\n    int64_t msg_id\n    uint32_t len\n    uint8_t data[len]\n\nmsg_id has to be positive and may never be reused.\ndata is transparent for the protocol and serialized/deserialized by a user\n\n## Response frame format\n    int64_t msg_id\n    uint32_t len\n    uint32_t handler_duration - present if handler duration is negotiated\n    uint8_t data[len]\n\nif msg_id < 0 enclosed response contains an exception that came as a response to msg id abs(msg_id)\ndata is transparent for the protocol and serialized/deserialized by a user\n\nthe handler_duration is in microseconds, the value of 0xffffffff means that it wasn't measured\nand should be disregarded by client\n\n## Stream frame format\n   uint32_t len\n   uint8_t data[len]\n\nlen == 0xffffffff signals end of stream\ndata is transparent for the protocol and serialized/deserialized by a user\n\n## Exception encoding\n    uint32_t type\n    uint32_t len\n    uint8_t data[len]\n\n### Known exception types\n    USER = 0\n    UNKNOWN_VERB = 1\n\n#### USER exception encoding\n\n    uint32_t len\n    char[len]\n\nThis exception is sent as a reply if rpc handler throws an exception.\nIt is delivered to a caller as rpc::remote_verb_error(char[len])\n\n#### UNKNOWN_VERB exception encoding\n\n    uint64_t verb_id\n\nThis exception is sent as a response to a request with unknown verb_id, the verb id is passed back as part of the exception payload.\n\n## More formal protocol description\n\n\trequest_stream = negotiation_frame, { request | compressed_request }\n\trequest = verb_type, msg_id, len, { byte }*len\n\tcompressed_request = len, { bytes }*len\n\tresponse_stream = negotiation_frame, { response | compressed_response }\n\tresponse = reply | exception\n\tcompressed_response = len, { byte }*len\n        streaming_stream = negotiation_frame, { streaming_frame | compressed_streaming_frame }\n        streaming_frame = len, { byte }*len\n        compressed_streaming_frame = len, { byte }*len\n\treply = msg_id, len, { byte }*len\n\texception = exception_header, serialized_exception\n\texception_header = -msg_id, len\n\tserialized_exception = (user|unknown_verb)\n\tuser = len, {byte}*len\n\tunknown_verb = verb_type\n\tverb_type = uint64_t\n\tmsg_id = int64_t\n\tlen = uint32_t\n\tbyte = uint8_t\n\tnegotiation_frame = 'SSTARRPC' len32(negotiation_frame_data) negotiation_frame_data\n\tnegotiation_frame_data = negotiation_frame_feature_record*\n\tnegotiation_frame_feature_record = feature_number len {byte}*len\n\tfeature_number = uint32_t\n\nNote that replies can come in order different from requests, and some requests may not have a reply at all.\n\n"
  },
  {
    "path": "doc/shared-token-bucket.md",
    "content": "# Shared token bucket\n\n## Intro\n\nThe classical token bucket has two parameters -- rate and limit. The rate\nis the amount of tokens that are put into bucket per some period of time\n(say -- second), the limit is the maximum amount of tokens that can be\naccumulated in the bucket. The process of regeneration of tokens this way\nis called \"replenishing\" below. When a request needs to be served it should\ntry to get a certain amount of tokens from the bucket.\n\nThe shared token bucket implements the above model for seastar sharded\narchitecture. The implementation doesn't use locks and is built on atomic\narithmetics.\n\n## Theory\n\n### Rovers\n\nThe bucket is implemented as a pair of increasing counters -- tail and\nhead rovers. The consumer of tokens advances the tail rover. To replenish\ntokens into the bucket the head rover is advanced.\n\n    +--------------------------------------------------------------------->\n    ^             ^\n    tail          head\n\n    grab N tokens:\n\n    +--------------------------------------------------------------------->\n    .---> ^       ^\n          tail    head\n\n    replenish N tokens:\n\n    +--------------------------------------------------------------------->\n          ^       .---> ^\n          tail          head\n\nIt's possible that after grab the tail would overrun the head and will occur\nin front of it. This would mean that there's not enough tokens in the bucket\nand that some amount of tokens were claimed from it.\n\n    grab a lot of tokens:\n\n    +--------------------------------------------------------------------->\n          .------------ ^ --> ^\n                        head  tail\n\nTo check if the tokens were grabbed the caller needs to check if the head is\n(still) in front of the tail. This approach adds the ability for the consumers\nto line up in the queue when they all try to grab tokens from a contented\nbucket. The \"ticket lock\" model works the same way.\n\n### Capped release\n\nThe implementation additionally support so called \"capped release\". This is\nwhen tokens are not replenished from nowhere, but leak into the main bucket\nfrom another bucket into which the caller should explicitly put them. This mode\ncan be useful in cases when the token bucket guards the entrance into some\nlocation that can temporarily (or constantly, but in that case it would denote\na bucket misconfiguration) slow down and stop handling tokens at the given\nrate. To prevent token bucket from over-subscribing the guardee at those times,\nthe second bucket can be refilled with the completions coming from the latter.\n\nIn terms of rovers this is implemented with the help of a third rover called\nceiling (or ceil in the code). This ceil rover actually defines the upper\nlimit at which the head rover may point. Respectively, putting tokens into\nthe second bucket (it's called releasing below) means advancing the ceil.\n\n## Practice\n\n### API\n\nTo work with the token bucket there are 4 calls:\n\n * `grab(N)` -- grabs a certain amount of tokens from the bucket and returns\n   back the resulting \"tail\" rover value. The value is useless per-se and is\n   only needed to call the deficiency() method\n\n * `replenish(time)` -- tries to replenish tokens into the bucket. The amount\n   of replenished tokens is how many had accumulated since last replenish\n   till the `time` parameter\n\n * `release(N)` -- releases the given number of token making them available\n   for replenishment. Only works if capped release is turned on by the\n   template parameter, otherwise asserts\n\n * `deficiency(tail)` -- returns back the number of tokens that were claimed\n   from the bucket but that are not yet there. Non-zero number means that the\n   bucket is contented and the request dispatching should be delayed\n\n### Example\n\nFor example, the simple dispatch loop may look like this\n\n    while (true) {\n        request = get_next_request()\n        tail = token_bucket.grab(request.cost())\n        while (token_bucket.deficiency(tail)) {\n            yield\n        }\n        request.dispatch()\n    }\n\nAnd in the background there should run a timer calling `token_bucket.replenish(now())`\n\nAdditionally, if there's a need to cap the token bucket with the real request serving\nrate, upon request completion one should call `token_bucket.release(request.cost())`\n"
  },
  {
    "path": "doc/signal.md",
    "content": "# Signals\n\nSeastar provides an interface to handle signals natively and safely as asynchronous tasks.\n\n## Default Signal Handlers\n\nSeastar sets by default signal handlers for SIGINT/SIGTERM that call reactor::stop(). The reactor will then execute callbacks installed by reactor::at_exit().\n\nYou can disable this behavior, by setting `app_template::config::auto_handle_sigint_sigterm` to false. This flag is provided in the header [seastar/core/app-template.hh](../include/seastar/core/app-template..hh). Then, Seastar will not set signal handlers, and the default behavior of the Linux kernel will be preserved (terminate the program).\n\n### Examples\n\n```C++\n#include <seastar/core/app-template.hh>\n\nint main(int ac, char** av) {\n    seastar::app_template::config cfg;\n    cfg.auto_handle_sigint_sigterm = false;\n    seastar::app_template app(std::move(cfg));\n\n    return app.run(argc, argv, [] {\n        std::cout << \"SIGINT/SIGTERM will terminate the program\\n\";\n    });\n}\n```\n\n## Custom Signal Handler\n\nIn order to set a custom signal handler, Seastar provides a procedure called `seastar::handle_signal` in the header [seastar/core/signal.hh](../include/seastar/core/signal.hh). It registers a custom handler for the specified signal based on the configuration params.\n\nThe procedure must be called inside the `app.run()` lambda, otherwise it's UB.\n\n### Examples\n\n```C++\n#include <seastar/core/app-template.hh>\n#include <seastar/core/signal.hh>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    return app.run(argc, argv, [] {\n        seastar::handle_signal(SIGINT, [&] {\n            std::cout << \"caught sigint\\n\";\n        }, true);\n    });\n}\n```\n\n- [tests/unit/signal_test.cc](../tests/unit/signal_test.cc)\n- [apps/lib/stop_signal.hh](../apps/lib/stop_signal.hh)\n"
  },
  {
    "path": "doc/template.css",
    "content": "/* CSS style for Seastar's tutorial.\n * TODO: We also get some style for syntax highlighting inserted by our\n * use of \"highlight-style tango\" in configure.py. Perhaps we can insert\n * this style here too, for finer control.\n */\n\n/* Some defaults */\nbody {\n\tcolor: #000000;\n\tbackground: #FFFFFF;\n\tfont-size: 13pt;\n\tline-height: 1.10;\n\tfont-family: arial, sans-serif;\n\tmargin-left: 15pt;\n\tmargin-right: 15pt;\n\ttext-align: justify;\n}\n\n/* In older versions, Pandoc puts the title, author and date, if any, in its\n * own div id=\"header\". In recent versions, it uses a \"header\" tag instead.\n*/\ndiv#header, header {\n\tborder-top: 1px solid #aaa;\n\tborder-bottom: 1px solid #aaa;\n\tbackground: #F0F0C0;\n\tmargin: 10pt;\n\tmargin-left: 10%;\n\tmargin-right: 10%;\n}\n\n/* The title is in an h1.title, in the above div#header */\n.title {\n\tcolor: #000000;\n\tmargin: 5pt;\n\ttext-align: center;\n\tfont-family: serif;\n\tfont-weight: bold;\n\tfont-size: 32pt;\n}\n/* The author/date are h2.author and h3.date */\n.author, .date {\n\tcolor: #000000;\n\tmargin: 0pt;\n\ttext-align: center;\n\tfont-family: serif;\n\tfont-weight: normal;\n\tfont-size: 16pt;\n}\n\n/* table of contents is in div id=\"TOC\" in older versions, or a nav id=\"TOC\"\n * in newer versions */\ndiv#TOC, nav#TOC {\n\tborder-top: 1px solid #aaa;\n\tborder-bottom: 1px solid #aaa;\n\tbackground: #F9F9F9;\n\tmargin: 10pt;\n\tmargin-left: 20%;\n\tmargin-right: 20%;\n}\n\n\nh1, h2, h3, h4, h5, h6 {\n\tcolor: #EE3300;\n}\n\na {\n\ttext-decoration: none;\n}\na:link, a:visited {\n\tcolor: #0000CC;\n}\na:hover {\n\tcolor: #CC0000;\n\ttext-decoration: underline;\n}\n\n\n/* Multiline code snippets are wrapped in a \"code\" inside a \"pre\".\n * Inline code snippets are just in a \"code\".\n */\ncode {\n\tbackground-color: #FFFFFF;\n    /* BEGIN word wrap */\n    /* Need all the following to word wrap instead of scroll box */\n    /* This will override the overflow:auto if present */\n    white-space: pre-wrap; /* css-3 */\n    white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */\n    white-space: -pre-wrap; /* Opera 4-6 */\n    white-space: -o-pre-wrap; /* Opera 7 */\n    word-wrap: break-word; /* Internet Explorer 5.5+ */\n    /* END word wrap */\n}\npre {\n    padding: 0.5em;\n    border: 1px dotted #777;\n    margin-left:  15pt;\n    margin-right: 15pt;\n}\npre, pre > code {\n    background-color: #f8f8f8;\n}\n\n/* Fix stuff for printing, in case somebody tries to print the HTML instead\n * of getting a PDF and printing that. For example, too big fonts and big\n * margins may be a waste of paper.\n * buttondown.css has a nice trick for replacing links with the actual text\n * of the URL - might be nice to copy it one day.\n */\n@media print {\n\tbody { font-size: 11pt; }\n\ta  { color: black; background: transparent; }\n    \tpre { border: 1px solid #aaa; }\n}\n"
  },
  {
    "path": "doc/template.tex",
    "content": "% The pandoc command line (see configure.py) can communicate variables to this\n% LaTeX skelaton, by using the \"-V varname=value\" command line parameter, and\n% there are also some builtin variables defined by other options. With such a\n% variable, we can use \"$varname$\" for the value, and $if(varname)$...$endif$\n% for conditional code when this varname is defined.\n%\n% For an example of a more complete LaTeX template covering more of pandoc's\n% features, check out the example templates/default.latex in pandoc's\n% installation, or pandoc.org/demo/mytemplate.tex on their site.\n\n\\documentclass[11]{article}\n\\usepackage[a4paper, margin=1in]{geometry}\n\\usepackage{lmodern}\n\\usepackage{microtype}\n\n% In code snippets, show backtick (`, grave accent) as a short straight line,\n% not a curved quote. I could have used \\IfFileExists{upquote.sty}{...}{}\n% to make this package optional, but I really think it's important, and\n% the user in a modern Linux distribution should be able to get it easily.\n\\usepackage{upquote}\n\n% Nice header with document's name and section name, and footer\n% with page number.\n\\usepackage{fancyhdr}\n\\pagestyle{fancy}\n\\lhead{\\itshape Seastar}\n\\chead{}\n\\rhead{\\itshape{\\nouppercase{\\leftmark}}}\n\\lfoot{}\n\\cfoot{}\n\\rfoot{\\thepage}\n\n$if(highlighting-macros)$\n$highlighting-macros$\n$endif$\n\n% Support links in PDF files\n\\usepackage[]{hyperref}\n\\hypersetup{breaklinks=true,\n            bookmarks=true, % show bookmark bar (i.e, TOC sidebar) in PDF viewer\n\t    pdfstartview=FitH, % cause Adobe Acrobat to do \"fit to width\"\n            pdfauthor={ScyllaDB},\n            pdftitle={Asynchronous Programming with Seastar},\n            colorlinks=true,  % if false, uses boxed links\n            urlcolor=blue,    % color of external links\n            linkcolor=magenta % color of internal links (to other sections)\n}\n\n% Indentation went out of style with Disco music... It is especially\n% inconvenient in text which includes a lot of code snippets, etc, which\n% causes a lot of indents not preceeded by a text paragraph. So replace\n% the indentation with increased spacing between paragraphs. Note that this\n% also makes the TOC more spacious, I'm not sure it's a great idea.\n\\setlength{\\parindent}{0pt}\n\\setlength{\\parskip}{6pt plus 2pt minus 1pt}\n\n% Apparently, newer versions of pandoc use a \"\\tightlist\" command, and\n% we need to implement it.\n\\providecommand{\\tightlist}{%\n\\setlength{\\itemsep}{0pt}\\setlength{\\parskip}{0pt}}\n\n% Fancy-formatted title, but take the author list and date (if set) from the\n% preable in the markdown file.\n\\title{\\textbf{\\fontsize{0.75cm}{1.5em}\\selectfont Asynchronous Programming} \\\\ \\textbf{\\fontsize{0.75cm}{1.5em}\\selectfont with} \\\\ \\textbf{\\fontsize{3cm}{1.5em}\\selectfont Seastar}}\n\\author{$for(author)$$author$$sep$ \\and $endfor$}\n\\date{$date$}\n\n\\begin{document}\n\\maketitle\n\n% Table of contents, before the body of the document. Make the TOC clickable,\n% with links to the respective sections, but override the internal link color\n% set above (magenta) by normal black color, because the entire TOC being in\n% a different color looks rather conspicuous.\n{\n\\hypersetup{linkcolor=black}\n\\tableofcontents\n}\n\n$body$\n\n\\end{document}\n"
  },
  {
    "path": "doc/testing.md",
    "content": "# Testing\n\nSeastar leverages Boost.Test and provides facilities for developers to implement tests in coroutines.\n\n## Test Declarations\n\nThere are three categories of boost-based tests in our system:\n\n* Boost.Test Native: Tests defined using `BOOST_AUTO_TEST_CASE` and related macros from Boost.Test. For more information, see the [Boost Test documentation](https://www.boost.org/doc/libs/release/libs/test/doc/html/boost_test/utf_reference/test_org_reference.html).\n* [Coroutine](https://github.com/scylladb/seastar/blob/master/doc/tutorial.md#coroutines): Tests defined using `SEASTAR_TEST_CASE`. The test body returns a future, allowing implementation as a coroutine.\n* Coroutine with [`seastar::thread`](https://github.com/scylladb/seastar/blob/master/doc/tutorial.md#seastarthread): Tests defined using `SEASTAR_THREAD_TEST_CASE`. The test body is launched in a Seastar thread, allowing the use of Seastar coroutines. These tests should return `void`.\n\n## Choosing the Appropriate Macro\n\n* Use `SEASTAR_TEST_CASE` or `SEASTAR_THREAD_TEST_CASE` if you need to run tests in a Seastar reactor thread, typically for Seastar coroutines. `SEASTAR_THREAD_TEST_CASE` can be more convenient in some cases.\n* For Seastar facilities that don't depend on coroutines, consider using `BOOST_AUTO_TEST_CASE`.\n\n\n## Tests' organization\n\n* Tests defined with `BOOST_AUTO_TEST_CASE` are driven by Boost.Test's built-in test runner. They require defining [`BOOST_TEST_MODULE`](https://www.boost.org/doc/libs/release/libs/test/doc/html/boost_test/utf_reference/link_references/link_boost_test_module_macro.html) to include the runner in the executable. These tests support additional features like fixtures and data-driven testing.\n* `SEASTAR_TEST_CASE` and `SEASTAR_THREAD_TEST_CASE` tests only support decorators and share a common test runner, allowing them to be co-located in the same test suite.\n* Seastar tests can be co-located with \"native\" Boost tests. The `SEASTAR_TEST_CASE` / `SEASTAR_THREAD_TEST_CASE` restrictions above apply to the entire test binary, and the `BOOST_TEST_MODULE` macro must not be defined.\n\n## Adding Tests in `CMakeLists.txt`\n\nFor Seastar tests (and binaries that contain both Seastar and Boost tests):\n\n```cmake\nseastar_add_test(meow_test\n  KIND SEASTAR\n  SOURCES meow.cc)\n```\n\nor simply:\n\n```cmake\nseastar_add_test(meow_test\n  SOURCES meow.cc)\n```\n\nThe `KIND` parameter of `seastar_add_test()` function defaults to `SEASTAR`, using the Seastar test runner by defining the `SEASTAR_TESTING_MAIN` macro.\n\nFor \"native\" Boost(-only) tests:\n\n```cmake\nseastar_add_test(woof_test\n  KIND BOOST\n  SOURCE woof.cc)\n```\n\n## Fuzz Testing\n\nSeastar supports fuzz testing using [libFuzzer](https://llvm.org/docs/LibFuzzer.html) for testing non-reactor code. Fuzz tests are built only in `fuzz` mode.\n\n### Building Fuzz Tests\n\nConfigure with fuzz mode (requires Clang):\n\n```bash\n./configure.py --mode fuzz --compiler clang++ [other options]\n```\n\nBuild all fuzz tests:\n\n```bash\nninja -C build/fuzz fuzz_tests\n```\n\n### Running Fuzz Tests\n\nFuzz test executables are located in `build/fuzz/tests/fuzz/`. Run a fuzzer:\n\n```bash\n# Basic run (runs indefinitely until stopped or crash found)\n./build/fuzz/tests/fuzz/sstring_fuzz\n\n# With a corpus directory (recommended for persistent fuzzing)\nmkdir -p corpus/sstring\n./build/fuzz/tests/fuzz/sstring_fuzz corpus/sstring\n\n# Limit iterations\n./build/fuzz/tests/fuzz/sstring_fuzz -runs=10000\n\n# Limit time (in seconds)\n./build/fuzz/tests/fuzz/sstring_fuzz -max_total_time=60\n```\n\nFor more libFuzzer options, see the [libFuzzer documentation](https://llvm.org/docs/LibFuzzer.html).\n\n### Adding Fuzz Tests\n\nFuzz tests are added in `tests/fuzz/CMakeLists.txt`:\n\n```cmake\nseastar_add_fuzz_test(mycomponent\n  SOURCES mycomponent_fuzz.cc)\n```\n\nThe fuzz test source must define `LLVMFuzzerTestOneInput`:\n\n```cpp\nextern \"C\" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {\n    // Test code here using data[0..size-1] as input\n    return 0;\n}\n```\n\n"
  },
  {
    "path": "doc/tutorial.md",
    "content": "% Asynchronous Programming with Seastar\n% Nadav Har'El - nyh@ScyllaDB.com\n  Avi Kivity - avi@ScyllaDB.com\n\n# Introduction\n**Seastar**, which we introduce in this document, is a C++ library for writing highly efficient complex server applications on modern multi-core machines.\n\nTraditionally, the programming languages libraries and frameworks used for writing server applications have been divided into two distinct camps: those focusing on efficiency, and those focusing on complexity. Some frameworks are extremely efficient and yet allow building only simple applications (e.g., DPDK allows applications which process packets individually), while other frameworks allow building extremely complex applications, at the cost of run-time efficiency. Seastar is our attempt to get the best of both worlds: To create a library which allows building highly complex server applications, and yet achieve optimal performance.\n\nThe inspiration and first use case of Seastar was Scylla, a rewrite of Apache Cassandra. Cassandra is a very complex application, and yet, with Seastar we were able to re-implement it with as much as 10-fold throughput increase, as well as significantly lower and more consistent latencies.\n\nSeastar offers a complete asynchronous programming framework, which uses two concepts - **futures** and **continuations** - to uniformly represent, and handle, every type of asynchronous event, including network I/O, disk I/O, and complex combinations of other events.\n\nSince modern multi-core and multi-socket machines have steep penalties for sharing data between cores (atomic instructions, cache line bouncing and memory fences), Seastar programs use the share-nothing programming model, i.e., the available memory is divided between the cores, each core works on data in its own part of memory, and communication between cores happens via explicit message passing (which itself happens using the SMP's shared memory hardware, of course).\n\n## Asynchronous programming\nA server for a network protocol, such as the classic HTTP (Web) or SMTP (e-mail) servers, inherently deals with parallelism: Multiple clients send requests in parallel, and we cannot finish handling one request before starting to handle the next: A request may, and often does, need to block because of various reasons --- a full TCP window (i.e., a slow connection), disk I/O, or even the client holding on to an inactive connection --- and the server needs to handle other connections as well.\n\nThe most straightforward way to handle such parallel connections, employed by classic network servers such as Inetd, Apache Httpd and Sendmail, is to use a separate operating-system process per connection. This technique evolved over the years to improve its performance: At first, a new process was spawned to handle each new connection; Later, a pool of existing processes was kept and each new connection was assigned to an unemployed process from the pool; Finally, the processes were replaced by threads. However, the common idea behind all these implementations is that at each moment, each process handles exclusively a single connection. Therefore, the server code is free to use blocking system calls, such as reading or writing to a connection, or reading from disk, and if this process blocks, all is well because we have many additional processes ready to handle other connections.\n\nProgramming a server which uses a process (or a thread) per connection is known as *synchronous* programming, because the code is written linearly, and one line of code starts to run after the previous line finished. For example, the code may read a request from a socket, parse the request, and then piecemeal read a file from disk and write it back to the socket. Such code is easy to write, almost like traditional non-parallel programs. In fact, it's even possible to run an external non-parallel program to handle each request --- this is for example how Apache HTTPd ran \"CGI\" programs, the first implementation of dynamic Web-page generation.\n\n>NOTE: although the synchronous server application is written in a linear, non-parallel, fashion, behind the scenes the kernel helps ensure that everything happens in parallel and the machine's resources --- CPUs, disk and network --- are fully utilized. Beyond the process parallelism (we have multiple processes handling multiple connections in parallel), the kernel may even parallelize the work of one individual connection --- for example process an outstanding disk request (e.g., read from a disk file) in parallel with handling the network connection (send buffered-but-yet-unsent data, and buffer newly-received data until the application is ready to read it).\n\nBut synchronous, process-per-connection, server programming didn't come without disadvantages and costs. Slowly but surely, server authors realized that starting a new process is slow, context switching is slow, and each process comes with significant overheads --- most notably the size of its stack. Server and kernel authors worked hard to mitigate these overheads: They switched from processes to threads, from creating new threads to thread pools, they lowered default stack size of each thread, and increased the virtual memory size to allow more partially-utilized stacks. But still, servers with synchronous designs had unsatisfactory performance, and scaled badly as the number of concurrent connections grew. In 1999, Dan Kegel popularized \"the C10K problem\", the need of a single server to efficiently handle 10,000 concurrent connections --- most of them slow or even inactive.\n\nThe solution, which became popular in the following decade, was to abandon the cozy but inefficient synchronous server design, and switch to a new type of server design --- the *asynchronous*, or *event-driven*, server. An event-driven server has just one thread, or more accurately, one thread per CPU. This single thread runs a tight loop which, at each iteration, checks, using ```poll()``` (or the more efficient ```epoll```) for new events on many open file descriptors, e.g., sockets. For example, an event can be a socket becoming readable (new data has arrived from the remote end) or becoming writable (we can send more data on this connection). The application handles this event by doing some non-blocking operations, modifying one or more of the file descriptors, and maintaining its knowledge of the _state_ of this connection.\n\nHowever, writers of asynchronous server applications faced, and still face today, two significant challenges:\n\n* **Complexity:** Writing a simple asynchronous server is straightforward. But writing a *complex* asynchronous server is notoriously difficult. The handling of a single connection, instead of being a simple easy-to-read function call, now involves a large number of small callback functions, and a complex state machine to remember which function needs to be called when each event occurs.\n\n* **Non-blocking:** Having just one thread per core is important for the performance of the server application, because context switches are slow. However, if we only have one thread per core, the event-handling functions must _never_ block, or the core will remain idle. But some existing programming languages and frameworks leave the server author no choice but to use blocking functions, and therefore multiple threads.\nFor example, ```Cassandra``` was written as an asynchronous server application; But because disk I/O was implemented with ```mmap```ed files, which can uncontrollably block the whole thread when accessed, they are forced to run multiple threads per CPU.\n\nMoreover, when the best possible performance is desired, the server application, and its programming framework, has no choice but to also take the following into account:\n\n* **Modern Machines**: Modern machines are very different from those of just 10 years ago. They have many cores and deep memory hierarchies (from L1 caches to NUMA) which reward certain programming practices and penalizes others: Unscalable programming practices (such as taking locks) can devastate performance on many cores; Shared memory and lock-free synchronization primitives are available (i.e., atomic operations and memory-ordering fences) but are dramatically slower than operations that involve only data in a single core's cache, and also prevent the application from scaling to many cores.\n\n* **Programming Language:** High-level languages such Java, Javascript, and similar \"modern\" languages are convenient, but each comes with its own set of assumptions which conflict with the requirements listed above. These languages, aiming to be portable, also give the programmer less control over the performance of critical code. For really optimal performance, we need a programming language which gives the programmer full control, zero run-time overheads, and on the other hand --- sophisticated compile-time code generation and optimization.\n\nSeastar is a framework for writing asynchronous server applications which aims to solve all four of the above challenges: It is a framework for writing *complex* asynchronous applications involving both network and disk I/O.  The framework's fast path is entirely single-threaded (per core), scalable to many cores and minimizes the use of costly sharing of memory between cores. It is a C++14 library, giving the user sophisticated compile-time features and full control over performance, without run-time overhead.\n\n## Seastar\n\n\nSeastar is an event-driven framework allowing you to write non-blocking, asynchronous code in a relatively straightforward manner (once understood). Its APIs are based on futures.  Seastar utilizes the following concepts to achieve extreme performance:\n\n* **Cooperative micro-task scheduler**: instead of running threads, each core runs a cooperative task scheduler. Each task is typically very lightweight -- only running for as long as it takes to process the last I/O operation's result and to submit a new one.\n* **Share-nothing SMP architecture**: each core runs independently of other cores in an SMP system. Memory, data structures, and CPU time are not shared; instead, inter-core communication uses explicit message passing. A Seastar core is often termed a shard. TODO: more here https://github.com/scylladb/seastar/wiki/SMP\n* **Future based APIs**: futures allow you to submit an I/O operation and to chain tasks to be executed on completion of the I/O operation. It is easy to run multiple I/O operations in parallel - for example, in response to a request coming from a TCP connection, you can issue multiple disk I/O requests, send messages to other cores on the same system, or send requests to other nodes in the cluster, wait for some or all of the results to complete, aggregate the results, and send a response.\n* **Share-nothing TCP stack**: while Seastar can use the host operating system's TCP stack, it also provides its own high-performance TCP/IP stack built on top of the task scheduler and the share-nothing architecture. The stack provides zero-copy in both directions: you can process data directly from the TCP stack's buffers, and send the contents of your own data structures as part of a message without incurring a copy. Read more...\n* **DMA-based storage APIs**: as with the networking stack, Seastar provides zero-copy storage APIs, allowing you to DMA your data to and from your storage devices.\n\nThis tutorial is intended for developers already familiar with the C++ language, and will cover how to use Seastar to create a new application.\n\nTODO: copy text from https://github.com/scylladb/seastar/wiki/SMP\nhttps://github.com/scylladb/seastar/wiki/Networking\n\n# Getting started\n\nThe simplest Seastar program is this:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <iostream>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] {\n            std::cout << \"Hello world\\n\";\n            return seastar::make_ready_future<>();\n    });\n}\n```\n\nAs we do in this example, each Seastar program must define and run, an `app_template` object. This object starts the main event loop (the Seastar *engine*) on one or more CPUs, and then runs the given function - in this case an unnamed function, a *lambda* - once.\n\nThe `return make_ready_future<>();` causes the event loop, and the whole application, to exit immediately after printing the \"Hello World\" message. In a more typical Seastar application, we will want event loop to remain alive and process incoming packets (for example), until explicitly exited. Such applications will return a _future_ which determines when to exit the application. We will introduce futures and how to use them below. In any case, the regular C `exit()` should not be used, because it prevents Seastar or the application from cleaning up appropriately.\n\nAs shown in this example, all Seastar functions and types live in the \"`seastar`\" namespace. An user can either type this namespace prefix every time, or use shortcuts like \"`using seastar::app_template`\" or even \"`using namespace seastar`\" to avoid typing this prefix. We generally recommend to use the namespace prefixes `seastar` and `std` explicitly, and will follow this style in all the examples below.\n\nTo compile this program (it's present in the `demos/hello-world.cc` file) you can just use Docker.\n\n```\n$ docker build -t seastar-dev  -f ./docker/dev/Dockerfile .\n$ docker run -it --rm -v $(pwd):/seastar seastar-dev /seastar/configure --mode=dev --cook=c-ares\n$ docker run -it --rm -v $(pwd):/seastar seastar-dev ninja -C /seastar/build/dev\n$ docker run -it --rm -v $(pwd):/seastar seastar-dev /seastar/build/dev/demos/hello-world_demo -c1\n```\n\nWithout the docker help, first make sure you have downloaded, built, and optionally installed Seastar, and put the above program in a source file anywhere you want, let's call the file `getting-started.cc`.\n\nLinux's [pkg-config](http://www.freedesktop.org/wiki/Software/pkg-config/) is one way for easily determining the compilation and linking parameters needed for using various libraries - such as Seastar.  For example, if Seastar was built in the directory `$SEASTAR` but not installed, one can compile `getting-started.cc` with it using the command:\n```\nc++ getting-started.cc `pkg-config --cflags --libs --static $SEASTAR/build/release/seastar.pc`\n```\nThe \"`--static`\" is needed because currently, Seastar is built as a static library, so we need to tell `pkg-config` to include its dependencies in the link command (whereas, had Seastar been a shared library, it could have pulled in its own dependencies).\n\nIf Seastar _was_ installed, the `pkg-config` command line is even shorter:\n```\nc++ getting-started.cc `pkg-config --cflags --libs --static seastar`\n```\n\nAlternatively, one can easily build a Seastar program with CMake. Given the following `CMakeLists.txt`\n\n```cmake\ncmake_minimum_required (VERSION 3.5)\n\nproject (SeastarExample)\n\nfind_package (Seastar REQUIRED)\n\nadd_executable (example\n  getting-started.cc)\n\ntarget_link_libraries (example\n  PRIVATE Seastar::seastar)\n```\n\nyou can compile the example with the following commands:\n\n```none\n$ mkdir build\n$ cd build\n$ cmake ..\n$ make\n```\n\nThe program now runs as expected:\n```none\n$ ./example\nHello world\n$\n```\n\n# Threads and memory\n## Seastar threads\nAs explained in the introduction, Seastar-based programs run a single thread on each CPU. Each of these threads runs its own event loop, known as the *engine* in Seastar nomenclature. By default, the Seastar application will take over all the available cores, starting one thread per core. We can see this with the following program, printing `seastar::smp::count` which is the number of started threads:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <iostream>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] {\n            std::cout << seastar::smp::count << \"\\n\";\n            return seastar::make_ready_future<>();\n    });\n}\n```\n\nOn a machine with 4 hardware threads (two cores, and hyperthreading enabled), Seastar will by default start 4 engine threads:\n\n```none\n$ ./a.out\n4\n```\n\nEach of these 4 engine threads will be pinned (a la **taskset(1)**) to a different hardware thread. Note how, as we mentioned above, the app's initialization function is run only on one thread, so we see the ouput \"4\" only once. Later in the tutorial we'll see how to make use of all threads.\n\nThe user can pass a command line parameter, `-c`, to tell Seastar to start fewer threads than the available number of hardware threads. For example, to start Seastar on only 2 threads, the user can do:\n```none\n$ ./a.out -c2\n2\n```\nWhen the machine is configured as in the example above - two cores with two hyperthreads on each - and only two threads are requested, Seastar ensures that each thread is pinned to a different core, and we don't get the two threads competing as hyperthreads of the same core (which would, of course, damage performance).\n\nWe cannot start more threads than the number of hardware threads, as allowing this will be grossly inefficient. Trying it will result in an error:\n```none\n$ ./a.out -c5\nCould not initialize seastar: std::runtime_error (insufficient processing units)\n```\n\nThe error is an exception thrown from app.run, which was caught by seastar itself and turned into a non-zero exit code. Note that catching the exceptions this way does **not** catch exceptions thrown in the application's actual asynchronous code. We will discuss these later in this tutorial.\n\n## Seastar memory\nAs explained in the introduction, Seastar applications shard their memory. Each thread is preallocated with a large piece of memory (on the same NUMA node it is running on), and uses only that memory for its allocations (such as `malloc()` or `new`).\n\nBy default, the machine's **entire memory** except a certain reservation left for the OS (defaulting to the maximum of 1.5G or 7% of total memory) is pre-allocated for the application in this manner. This default can be changed by *either* changing the amount reserved for the OS (not used by Seastar) with the `--reserve-memory` option, or by explicitly giving the amount of memory given to the Seastar application, with the `-m` option. This amount of memory can be in bytes, or using the units \"k\", \"M\", \"G\" or \"T\". These units use the power-of-two values: \"M\" is a **mebibyte**, 2^20 (=1,048,576) bytes, not a **megabyte** (10^6 or 1,000,000 bytes).\n\nTrying to give Seastar more memory than physical memory immediately fails:\n```none\n$ ./a.out -m10T\nCouldn't start application: std::runtime_error (insufficient physical memory)\n```\n\n# Introducing futures and continuations\nFutures and continuations, which we will introduce now, are the building blocks of asynchronous programming in Seastar. Their strength lies in the ease of composing them together into a large, complex, asynchronous program, while keeping the code fairly readable and understandable.\n\nA [future](\\ref future) is a result of a computation that may not be available yet.\nExamples include:\n\n  * a data buffer that we are reading from the network\n  * the expiration of a timer\n  * the completion of a disk write\n  * the result of a computation that requires the values from\n    one or more other futures.\n\nThe type `future<int>` variable holds an int that will eventually be available - at this point might already be available, or might not be available yet. The method available() tests if a value is already available, and the method get() gets the value. The type `future<>` indicates something which will eventually complete, but not return any value.\n\nA future is usually returned by an **asynchronous function**, a function which returns a future and arranges for this future to be eventually resolved.  Because asynchronous functions _promise_ to eventually resolve the future which they returned, asynchronous functions are sometimes called \"promises\"; But we will avoid this term because it tends to confuse more than it explains.\n\nOne simple example of an asynchronous function is Seastar's function sleep():\n\n```cpp\nfuture<> sleep(std::chrono::duration<Rep, Period> dur);\n```\n\nThis function arranges a timer so that the returned future becomes available (without an associated value) when the given time duration elapses.\n\nA **continuation** is a callback (typically a lambda) to run when a future becomes available. A continuation is attached to a future with the `then()` method. Here is a simple example:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, [] {\n        std::cout << \"Sleeping... \" << std::flush;\n        using namespace std::chrono_literals;\n        return seastar::sleep(1s).then([] {\n            std::cout << \"Done.\\n\";\n        });\n    });\n}\n```\n\nIn this example we see us getting a future from `seastar::sleep(1s)`, and attaching to it a continuation which prints a \"Done.\" message. The future will become available after 1 second has passed, at which point the continuation is executed. Running this program, we indeed see the message \"Sleeping...\" immediately, and one second later the message \"Done.\" appears and the program exits.\n\nThe return value of `then()` is itself a future which is useful for chaining multiple continuations one after another, as we will explain below. But here we just note that we `return` this future from `app.run`'s function, so that the program will exit only after both the sleep and its continuation are done.\n\nTo avoid repeating the boilerplate \"app_engine\" part in every code example in this tutorial, let's create a simple main() with which we will compile the following examples. This main just calls function `future<> f()`, does the appropriate exception handling, and exits when the future returned by `f` is resolved:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/util/log.hh>\n#include <iostream>\n#include <stdexcept>\n\nextern seastar::future<> f();\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    try {\n        app.run(argc, argv, f);\n    } catch(...) {\n        std::cerr << \"Couldn't start application: \"\n                  << std::current_exception() << \"\\n\";\n        return 1;\n    }\n    return 0;\n}\n```\n\nCompiling together with this `main.cc`, the above sleep() example code becomes:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nseastar::future<> f() {\n    std::cout << \"Sleeping... \" << std::flush;\n    using namespace std::chrono_literals;\n    return seastar::sleep(1s).then([] {\n        std::cout << \"Done.\\n\";\n    });\n}\n```\n\nSo far, this example was not very interesting - there is no parallelism, and the same thing could have been achieved by the normal blocking POSIX `sleep()`. Things become much more interesting when we start several sleep() futures in parallel, and attach a different continuation to each. Futures and continuation make parallelism very easy and natural:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nseastar::future<> f() {\n    std::cout << \"Sleeping... \" << std::flush;\n    using namespace std::chrono_literals;\n    seastar::sleep(200ms).then([] { std::cout << \"200ms \" << std::flush; });\n    seastar::sleep(100ms).then([] { std::cout << \"100ms \" << std::flush; });\n    return seastar::sleep(1s).then([] { std::cout << \"Done.\\n\"; });\n}\n```\n\nEach `sleep()` and `then()` call returns immediately: `sleep()` just starts the requested timer, and `then()` sets up the function to call when the timer expires. So all three lines happen immediately and f returns. Only then, the event loop starts to wait for the three outstanding futures to become ready, and when each one becomes ready, the continuation attached to it is run. The output of the above program is of course:\n```none\n$ ./a.out\nSleeping... 100ms 200ms Done.\n```\n\n`sleep()` returns `future<>`, meaning it will complete at a future time, but once complete, does not return any value. More interesting futures do specify a value of any type (or multiple values) that will become available later. In the following example, we have a function returning a `future<int>`, and a continuation to be run once this value becomes available. Note how the continuation gets the future's value as a parameter:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nseastar::future<int> slow() {\n    using namespace std::chrono_literals;\n    return seastar::sleep(100ms).then([] { return 3; });\n}\n\nseastar::future<> f() {\n    return slow().then([] (int val) {\n        std::cout << \"Got \" << val << \"\\n\";\n    });\n}\n```\n\nThe function `slow()` deserves more explanation. As usual, this function returns a `future<int>` immediately, and doesn't wait for the sleep to complete, and the code in `f()` can chain a continuation to this future's completion. The future returned by `slow()` is itself a chain of futures: It will become ready once sleep's future becomes ready and then the value 3 is returned. We'll explain below in more details how `then()` returns a future, and how this allows *chaining* futures.\n\nThis example begins to show the convenience of the futures programming model, which allows the programmer to neatly encapsulate complex asynchronous operations. `slow()` might involve a complex asynchronous operation requiring multiple steps, but its user can use it just as easily as a simple `sleep()`, and Seastar's engine takes care of running the continuations whose futures have become ready at the right time.\n\n## Ready futures\nA future value might already be ready when `then()` is called to chain a continuation to it. This important case is optimized, and the continuation is run immediately instead of being registered to run later in the next iteration of the event loop.\n\n`make_ready_future<>` can be used to return a future which is already ready. The following example is identical to the previous one, except the promise function `fast()` returns a future which is already ready, and not one which will be ready in a second as in the previous example. The nice thing is that the consumer of the future does not care, and uses the future in the same way in both cases.\n\n```cpp\n#include <seastar/core/future.hh>\n#include <iostream>\n\nseastar::future<int> fast() {\n    return seastar::make_ready_future<int>(3);\n}\n\nseastar::future<> f() {\n    return fast().then([] (int val) {\n        std::cout << \"Got \" << val << \"\\n\";\n    });\n}\n```\n\n## Preemption and Task Quota\n\nAs described above, an existing fiber of execution will yield back to the event loop when it performs a blocking operation such as IO or sleeping, as it has no more work to do until this blocking operation completes. Should a fiber have a lot of CPU bound work to do without any intervening blocking operations, however, it is important that execution is still yielded back to the event loop periodically.\n\nThis is implemented via _preemption_: which can only occur at specific preemption points. At these points the fiber's remaining _task quota_ is checked and it has been exceeded the fiber yields. The task quota is a measure of how long tasks should be allowed to run before yielding to the event loop, and is set to 500 µs by default.\n\nIt is important not to starve the event loop, as this would starve continuations of futures that weren't ready but have since become ready, and also starve the important **polling** done by the event loop (e.g., checking whether there is new activity on the network card). For example, iterating over a large container while doing CPU-bound work without any suspension points could starve the reactor and cause a _reactor stall_, which refers to a substantial period of time (e.g., more than 20 milliseconds) during which a task does not yield.\n\nMany seastar constructs such as looping constructs have built-in preemption points. You may also insert your own preemption points by calling `seastar::maybe_yield`, which performs a preemption check. Coroutines will also perform a preemption check at each `co_await`. Note that there is _not_ a preemption check between continuations attached to a future with `then()`, so a recursive future loop without explicit preemption checks may starve the reactor.\n\n# Coroutines\n\nNote: coroutines require C++20 and a supporting compiler. Clang 10 and above is known to work.\n\nThe simplest way to write efficient asynchronous code with Seastar is to use coroutines. Coroutines don't share most of the pitfalls of traditional continuations (below), and so are the preferred way to write new code.\n\nA coroutine is a function that returns a `seastar::future<T>` and uses the `co_await` or `co_return` keywords. Coroutines are invisible to their callers and callees; they integrate with traditional Seastar code in either role. If you are not familiar with C++ coroutines, you may want to consult [A more general introduction to C++ coroutines](https://medium.com/pranayaggarwal25/coroutines-in-cpp-15afdf88e17e); this section focuses on how coroutines integrate with Seastar.\n\nHere's an example of a simple Seastar coroutine:\n\n```cpp\n#include <seastar/core/coroutine.hh>\n\nseastar::future<int> read();\nseastar::future<> write(int n);\n\nseastar::future<int> slow_fetch_and_increment() {\n    auto n = co_await read();     // #1\n    co_await seastar::sleep(1s);  // #2\n    auto new_n = n + 1;           // #3\n    co_await write(new_n);        // #4\n    co_return n;                  // #5\n}\n```\n\nIn #1, we call the `read()` function, which returns a future. The `co_await` keyword instructs Seastar to inspect the returned future. If the future is ready, then the value (an `int`) is extracted from the future and assigned to `n`. If the future is not ready, the coroutine arranges for itself to be called when the future becomes ready, and control is returned to Seastar. Once the future becomes ready, the coroutine is awakened and the value is extracted from the future and assigned to `n`.\n\nIn #2, we call `seastar::sleep()` and wait for the returned future to become ready, which it will in a second. This demonstrates that `n` is preserved across `co_await` calls, and the author of the coroutine need not arrange for storage for coroutine local variables.\n\nLine #3 demonstrates the addition operation, with which the reader is assumed to be familiar.\n\nIn #4, we call a function that returns a `seastar::future<>`. In this case, the future carries no value, and so no value is extracted and assigned.\n\nLine #5 demonstrates returning a value. The integer value is used to satisfy the `future<int>` that our caller got when calling the coroutine.\n\n## Lambda coroutines (C++ 23 and later)\n\nIn C++ 23 and later, lambda coroutines can be safely used from all contexts.\n\nIf the lambda coroutine is awaited from an outer coroutine, nothing special needs to be done:\n\n```cpp\nfuture<>\nouter_coroutine() {\n    co_await [captures...] () {\n        co_await ...;\n        co_return;\n    }();\n}\n```\n\nThis works even if the lambda is called indirectly, as long as it is captured by value and\nawaited in the same statement it is defined in, as C++ will extend the lifetime of the capture\ngroup until the semicolon.\n\nIn other cases (lambdas called asynchronously to the caller, or passed to continuations, add\n`this auto` as the first parameter to convert the signature of the generated `operator()()`\nto accept the lambda by value. This copies the capture group to the lambda coroutine frame when\nit is called.\n\n```cpp\nfuture<>\nouter_coroutine() {\n    asynchronous_function([captures...] (this auto) {\n        co_await ...;\n        co_return;\n    });\n    // The lambda coroutine can be called after outer_coroutine() returns\n    co_return;\n}\n```\n\nUsing `this auto` can add some small overhead to move objects, but is generally safer\nwhen it's not clear the lambda is awaited synchronously.\n\n## Lambda coroutines (before C++ 23)\n\nA lambda function can be a coroutine. Due to an interaction between how C++ lambda coroutines are specified and how\nSeastar coroutines work, using lambda coroutines as continuations can result in use-after-free. To avoid such problems,\ntake one of the following approaches:\n\n1. Use lambda coroutines as arguments to functions that explicitly claim support for them\n2. Wrap lambda coroutines with `seastar::coroutine::lambda()`, and ensure the lambda coroutine is fully awaited within the statement it is defined in.\n\nAn example of wrapping a lambda coroutine is:\n\n```cpp\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/maybe_yield.hh>\n\nfuture<> foo() {\n    int n = 3;\n    int m = co_await seastar::yield().then(seastar::coroutine::lambda([n] () -> future<int> {\n        co_await seastar::coroutine::maybe_yield();\n        // `n` can be safely used here\n        co_return n;\n    }));\n    assert(n == m);\n}\n```\n\nNotes:\n1. `seastar::future::then()` accepts a continuation\n2. We wrap the argument to `seastar::future::then()` with `seastar::coroutine::lambda()`\n3. We ensure evaluation of the lambda completes within the same expression using the outer `co_await`.\n\nMore information can be found in `lambda-coroutine-fiasco.md`.\n\n## Generators in coroutines\n\nSometimes, it would be convenient to model a view of `input_range` with a coroutine which emits the elements one after\nanother asynchronously. From the consumer of the view's perspective, it can retrieve the elements by `co_await`ing\nthe return value of the coroutine. From the coroutine's perspective, it is able to produce the elements multiple times\nusing `co_yield` without \"leaving\" the coroutine. A function producing a sequence of values can be named \"generator\".\nBut unlike the regular coroutine which returns a single `seastar::future<T>`, a generator should return\n`seastar::coroutine::experimental::generator<T, Container>`. Where `T` is the type of the elements, while `Container`\nis a template, which is used to store the elements. Because, underneath of Seastar's generator implementation, a\nbounded buffer is used for holding the elements not yet retrieved by the consumer, there is a design decision to make --\nwhat kind of container should be used, and what its maximum size should be. To define the bounded buffer, developers\nneed to:\n\n1. specify the type of the container's type by via the second template parameter of the `generator`\n2. specify the size of the bounded buffer by passing the size as the first parameter of the generator coroutine.\n   The type of the size have to be `seastar::coroutine::experimental::buffer_size_t`.\n\nBut there is an exception, if the buffer's size is one, we assume that the programmer is likely to use `std::optional`\nfor the bounded buffer, so it's not required to pass the maximum size of the buffer as the first parameter in this case.\nBut if a coroutine uses `std::optional` as its buffer, and its function signature still lists the size as its first\nparameter, it will not break anything. As this parameter will just be ignored by the underlying implementation.\n\nFollowing is an example\n\n```cpp\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/coroutine/generator.hh>\n\nseastar::future<Preprocessed> prepare_ingredients(Ingredients&&);\nseastar::future<Dish> cook_a_dish(Preprocessed&&);\nseastar::future<> consume_a_dish(Dish&&);\n\nseastar::coroutine::experimental::generator<Dish, seastar::circular_buffer>\nmake_dishes(coroutine::experimental::buffer_size_t max_dishes_on_table,\n            Ingredients&& ingredients) {\n    while (ingredients) {\n        auto some_ingredients = ingredients.alloc();\n        auto preprocessed = co_await prepare_ingredients(std::move(some_ingredients));\n        co_yield co_await cook_a_dish(std::move(preprocessed));\n    }\n}\n\nseastar::future<> have_a_dinner(unsigned max_dishes_on_table) {\n    Ingredients ingredients;\n    auto dishes = make_dishes(coroutine::experimental::buffer_size_t{max_dishes_on_table},\n                              std::move(ingredients));\n    while (auto dish = co_await dishes()) {\n        co_await consume_a_dish(std::move(dish));\n    }\n}\n```\n\nIn this hypothetical kitchen, a chef and a diner are working in parallel. Instead of preparing\nall dishes beforehand, the chef cooks the dishes while the diner is consuming them one after another.\nUnder most circumstances, neither the chef or the diner is blocked by its peer. The dishes are buffered\nusing the specified `seastar::circular_buffer<Dish>`. But if the diner\nis too slow so that there are `max_dishes_on_table` dishes left on the table, the chef would wait\nuntil the number of dishes is less than this setting. Please note, as explained above, despite that this\nparameter is not referenced by the coroutine's body, it is actually passed to the generator's promise\nconstructor, which in turn creates the buffer, as we are not using `std::optional` here. On the other hand,\napparently, if there is no dishes on the table, the diner would wait for new ones to be prepared by the chef.\n\nPlease note, `generator<T, Container>` is still at its early stage of developing,\nthe public interface this template is subject to change before it is stabilized enough.\n\n## Exceptions in coroutines\n\nCoroutines automatically translate exceptions to futures and back.\n\nCalling `co_await foo()`, when `foo()` returns an exceptional future, will throw the exception carried by the future.\n\nSimilarly throwing within a coroutine will cause the coroutine to return an exceptional future.\n\nExample:\n\n```cpp\n#include <seastar/core/coroutine.hh>\n\nseastar::future<> function_returning_an_exceptional_future();\n\nseastar::future<> exception_handling() {\n    try {\n        co_await function_returning_an_exceptional_future();\n    } catch (...) {\n        // exception will be handled here\n    }\n    throw 3; // will be captured by coroutine and returned as\n             // an exceptional future\n}\n```\n\nBoth `throw` and `std::rethrow_exception()` involve managing stack unwinding and exception objects, potentially\nimpacting performance. Additionally, the C++ standard permits `std::rethrow_exception()` to create a copy of the\nexception object, introducing further overhead. Fortunately, in certain cases, exceptions can also be propagated\ndirectly, without throwing or rethrowing them. There are multiple facilities for this.\n`coroutine::try_future` can be used to propagate exceptions automatically from a called asynchronous function.\nIt is analagous to rust's [try operator](https://google.github.io/comprehensive-rust/error-handling/try.html).\nIf the awaited function resolves with an exception, it is automatically propagated to the coroutine's waiter\nwithout throwing. If manual error handling is needed, one can use `coroutine::as_future` and `coroutine::exception`.\n`coroutine::as_future` is analagous to `future<>::then_wrapped()`. It returns a ready future, allowing the code to\nprobe for exception without expensive throw and catch. The exception can then be propagated directly using\nthe `coroutine::exception` wrapper. `coroutine::exception` only works for coroutines which return `future<T>`, not\n`future<>`, due to the limitations in compilers. In particular, the example below won't compile if the return type is\nchanged to `future<>`.\n\nExample:\n\n```cpp\nseastar::future<int> exception_propagating() {\n    // Will automatically propagate the exception, if there is one.\n    auto result = co_await seastar::coroutine::try_future(\n        function_returning_an_exceptional_future()\n    );\n\n    // Will extract the ready future. Will not result in throwing, even if the future is exceptional.\n    auto fut = co_await seastar::coroutine::as_future(\n        function_returning_an_exceptional_future()\n    );\n    if (fut.failed()) {\n        // Saved exception pointer can be propagated without rethrowing\n        co_return seastar::coroutine::exception(fut.get_exception());\n    }\n\n    co_return seastar::coroutine::make_exception(3); // Custom exceptions can be propagated without throwing\n}\n```\n\n## Concurrency in coroutines\n\nThe `co_await` operator allows for simple sequential execution. Multiple coroutines can execute in parallel, but each coroutine has only one outstanding computation at a time.\n\nThe `seastar::coroutine::all` class template allows a coroutine to fork into several concurrently executing sub-coroutines (or Seastar fibers, see below) and join again when they complete. Consider this example:\n\n\n```cpp\n#include <seastar/core/coroutines.hh>\n#include <seastar/coroutine/all.hh>\n\nseastar::future<int> read(int key);\n\nseastar::future<int> parallel_sum(int key1, int key2) {\n    int [a, b] = co_await seastar::coroutine::all(\n        [&] {\n            return read(key1);\n        },\n        [&] {\n            return read(key2);\n        }\n    );\n    co_return a + b;\n}\n```\n\nHere, two read() calls are launched concurrently. The coroutine is paused until both reads complete, and the values returned are assigned to `a` and `b`. If `read(key)` is an operation that involves I/O, then the concurrent execution will complete sooner than if we `co_await`ed each call separately, since I/O can be overlapped.\n\n\nNote that `all` waits for all of its sub-computations, even if some throw an exception. If an exception is thrown, it is propagated to the calling coroutine.\n\nThe `seastar::coroutine::parallel_for_each` class template allows a coroutine to fork into several concurrently executing function invocations (or Seastar fibers, see below) over a range of elements and join again when they complete. Consider this example:\n\n```cpp\n#include <seastar/core/coroutines.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n\nseastar::future<bool> all_exist(std::vector<seastar::sstring> filenames) {\n    bool res = true;\n    co_await seastar::coroutine::parallel_for_each(filenames, [&res] (const seastar::sstring& name) -> seastar::future<> {\n        res &= co_await seastar::file_exists(name);\n    });\n    co_return res;\n}\n```\n\nHere, the lambda function passed to parallel_for_each is launched concurrently for each element in the filenames vector. The coroutine is paused until all calls complete.\n\n## Breaking up long running computations\n\nSeastar is generally used for I/O, and coroutines usually launch I/O operations and consume their results, with little computation in between. But occasionally a long running computation is needed, and this risks preventing the reactor from performing I/O and scheduling other tasks.\n\nA coroutine will automatically yield in a `co_await` expression; but in a computation we do not `co_await` anything. We can use the `seastar::coroutine::maybe_yield` class in such cases:\n\n```cpp\n#include <seastar/coroutine/maybe_yield>\n\nseastar::future<int> long_loop(int n) {\n    float acc = 0;\n    for (int i = 0; i < n; ++i) {\n        acc += std::sin(float(i));\n        // Give the Seastar reactor opportunity to perform I/O or schedule\n        // other tasks.\n        co_await seastar::coroutine::maybe_yield();\n    }\n    co_return acc;\n}\n```\n\n## Bypassing preemption checks in coroutines\n\nBy default, `co_await`-ing a future performs a preemption check, and will suspend if the task quota is already depleted. However, in certain cases it might be useful to be able to assume that awaiting a ready future will not yield.\nFor such cases, it's possible to explicitly bypass the preemption check:\n\n```cpp\n#include <seastar/core/coroutine.hh>\n\nstruct resource;\nseastar::future<int> compute_always_ready(int i, resource& r);\n\nseastar::future<int> accumulate(int n, resource& important_resource) {\n    float acc = 0;\n    for (int i = 0; i < n; ++i) {\n        // This await will not yield the control, so we're sure that nobody will\n        // be able to touch important_resource while we accumulate all the results.\n        acc += co_await seastar::coroutine::without_preemption_check(compute_always_ready(i, important_resource));\n    }\n    co_return acc;\n}\n```\n\n# Continuations\n## Capturing state in continuations\n\nWe've already seen that Seastar *continuations* are lambdas, passed to the `then()` method of a future. In the examples we've seen so far, lambdas have been nothing more than anonymous functions. But C++11 lambdas have one more trick up their sleeve, which is extremely important for future-based asynchronous programming in Seastar: Lambdas can **capture** state. Consider the following example:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nseastar::future<int> incr(int i) {\n    using namespace std::chrono_literals;\n    return seastar::sleep(10ms).then([i] { return i + 1; });\n}\n\nseastar::future<> f() {\n    return incr(3).then([] (int val) {\n        std::cout << \"Got \" << val << \"\\n\";\n    });\n}\n```\n\nThe future operation `incr(i)` takes some time to complete (it needs to sleep a bit first...), and in that duration, it needs to save the `i` value it is working on. In the early event-driven programming models, the programmer needed to explicitly define an object for holding this state, and to manage all these objects. Everything is much simpler in Seastar, with C++11's lambdas: The *capture syntax* \"`[i]`\" in the above example means that the value of i, as it existed when incr() was called() is captured into the lambda. The lambda is not just a function - it is in fact an *object*, with both code and data. In essence, the compiler created for us automatically the state object, and we neither need to define it, nor to keep track of it (it gets saved together with the continuation, when the continuation is deferred, and gets deleted automatically after the continuation runs).\n\nOne implementation detail worth understanding is that when a continuation has captured state and is run immediately, this capture incurs no runtime overhead. However, when the continuation cannot be run immediately (because the future is not yet ready) and needs to be saved till later, memory needs to be allocated on the heap for this data, and the continuation's captured data needs to be copied there. This has runtime overhead, but it is unavoidable, and is very small compared to the related overhead in the threaded programming model (in a threaded program, this sort of state usually resides on the stack of the blocked thread, but the stack is much larger than our tiny capture state, takes up a lot of memory and causes a lot of cache pollution on context switches between those threads).\n\nIn the above example, we captured `i` *by value* - i.e., a copy of the value of `i` was saved into the continuation. C++ has two additional capture options: capturing by *reference* and capturing by *move*:\n\nUsing capture-by-reference in a continuation is usually a mistake, and can lead to serious bugs. For example, if in the above example we captured a reference to i, instead of copying it,\n```cpp\nseastar::future<int> incr(int i) {\n    using namespace std::chrono_literals;\n    // Oops, the \"&\" below is wrong:\n    return seastar::sleep(10ms).then([&i] { return i + 1; });\n}\n```\nthis would have meant that the continuation would contain the address of `i`, not its value. But `i` is a stack variable, and the incr() function returns immediately, so when the continuation eventually gets to run, long after incr() returns, this address will contain unrelated content.\n\nAn exception to the capture-by-reference-is-usually-a-mistake rule is the `do_with()` idiom, which we will introduce later. This idiom ensures that an object lives throughout the life of the continuation, and makes capture-by-reference possible, and very convenient.\n\nUsing capture-by-*move* in continuations is also very useful in Seastar applications. By **moving** an object into a continuation, we transfer ownership of this object to the continuation, and make it easy for the object to be automatically deleted when the continuation ends. For example, consider a traditional function taking a `std::unique_ptr<T>`.\n```cpp\nint do_something(std::unique_ptr<T> obj) {\n     // do some computation based on the contents of obj, let's say the result is 17\n     return 17;\n     // at this point, obj goes out of scope so the compiler delete()s it.\n```\nBy using unique_ptr in this way, the caller passes an object to the function, but tells it the object is now its exclusive responsibility - and when the function is done with the object, it automatically deletes it. How do we use unique_ptr in a continuation? The following won't work:\n\n```cpp\nseastar::future<int> slow_do_something(std::unique_ptr<T> obj) {\n    using namespace std::chrono_literals;\n    // The following line won't compile...\n    return seastar::sleep(10ms).then([obj] () mutable { return do_something(std::move(obj)); });\n}\n```\n\nThe problem is that a unique_ptr cannot be passed into a continuation by value, as this would require copying it, which is forbidden because it violates the guarantee that only one copy of this pointer exists. We can, however, *move* obj into the continuation:\n```cpp\nseastar::future<int> slow_do_something(std::unique_ptr<T> obj) {\n    using namespace std::chrono_literals;\n    return seastar::sleep(10ms).then([obj = std::move(obj)] () mutable {\n        return do_something(std::move(obj));\n    });\n}\n```\nHere the use of `std::move()` causes obj's move-assignment is used to move the object from the outer function into the continuation. The notion of move (*move semantics*), introduced in C++11, is similar to a shallow copy followed by invalidating the source copy (so that the two copies do not co-exist, as forbidden by unique_ptr). After moving obj into the continuation, the top-level function can no longer use it (in this case it's of course ok, because we return anyway).\n\nThe `[obj = ...]` capture syntax we used here is new to C++14. This is the main reason why Seastar requires C++14, and does not support older C++11 compilers.\n\nThe extra `() mutable` syntax was needed here because by default when C++ captures a value (in this case, the value of std::move(obj)) into a lambda, it makes this value read-only, so our lambda cannot, in this example, move it again. Adding `mutable` removes this artificial restriction.\n\n## Evaluation order considerations (C++14 only)\n\nC++14 (and below) does *not* guarantee that lambda captures in continuations will be evaluated after the futures they relate to are evaluated\n(See https://en.cppreference.com/w/cpp/language/eval_order).\n\nConsequently, avoid the programming pattern below:\n```cpp\n    return do_something(obj).then([obj = std::move(obj)] () mutable {\n        return do_something_else(std::move(obj));\n    });\n```\n\nIn the example above, `[obj = std::move(obj)]` might be evaluated before `do_something(obj)` is called, potentially leading to use-after-move of `obj`.\n\nTo guarantee the desired evaluation order, the expression above may be broken into separate statements as follows:\n```cpp\n    auto fut = do_something(obj);\n    return fut.then([obj = std::move(obj)] () mutable {\n        return do_something_else(std::move(obj));\n    });\n```\n\nThis was changed in C++17. The expression that creates the object the function `then` is called on (the future) is evaluated before all the arguments to the function, so this style is not required in C++17 and above.\n\n## Chaining continuations\nTODO: We already saw chaining example in slow() above. talk about the return from then, and returning a future and chaining more thens.\n\n# Handling exceptions\n\nAn exception thrown in a continuation is implicitly captured by the system and stored in the future. A future that stores such an exception is similar to a ready future in that it can cause its continuation to be launched, but it does not contain a value -- only the exception.\n\nCalling `.then()` on such a future skips over the continuation, and transfers the exception for the input future (the object on which `.then()` is called) to the output future (`.then()`'s return value).\n\nThis default handling parallels normal exception behavior -- if an exception is thrown in straight-line code, all following lines are skipped:\n\n```cpp\nline1();\nline2(); // throws!\nline3(); // skipped\n```\n\nis similar to\n\n```cpp\nreturn line1().then([] {\n    return line2(); // throws!\n}).then([] {\n    return line3(); // skipped\n});\n```\n\nUsually, aborting the current chain of operations and returning an exception is what's needed, but sometimes more fine-grained control is required. There are several primitives for handling exceptions:\n\n1. `.then_wrapped()`: instead of passing the values carried by the future into the continuation, `.then_wrapped()` passes the input future to the continuation. The future is guaranteed to be in ready state, so the continuation can examine whether it contains a value or an exception, and take appropriate action.\n2. `.finally()`: similar to a Java finally block, a `.finally()` continuation is executed whether or not its input future carries an exception or not. The result of the finally continuation is its input future, so `.finally()` can be used to insert code in a flow that is executed unconditionally, but otherwise does not alter the flow.\n\nThe following example illustrates usage of `then_wrapped` and `finally`:\n\n```cpp\n#include <seastar/core/future.hh>\n#include <iostream>\n#include <exception>\n\nseastar::future<> pass() {\n    std::cout << \"I passed!!!\" << std::endl;\n    return seastar::make_ready_future<>();\n}\n\nseastar::future<> fail() {\n    std::cout << \"I failed.\" << std::endl;\n    return seastar::make_exception_future<>(std::exception());\n}\n\nseastar::future<> f() {\n    return pass().then([] {\n        std::cout << \"Oh no! I'm gonna fail!\" << std::endl;\n        return fail(); // throws\n    }).then([] () { // skipped\n        std::cout << \"If I got to this place I will pass!\" << std::endl;\n        return pass();\n    }).then_wrapped([] (seastar::future<> f) {\n        if (f.failed()) {\n            std::cout << \"The input future failed!\" << std::endl;\n            return f;\n        }\n\n        std::cout << \"If I got to this place I will pass!\" << std::endl;\n        return pass();\n    }).finally([] () {\n        std::cout << \"This code will run, regardless of any exceptions\" << std::endl;\n    });\n}\n```\n\nThis time the output will be\n```\nI passed!!!\nOh no! I'm gonna fail!\nI failed.\nThe input future failed!\nThis code will run, regardless of any exceptions\nERROR [...] Exiting on unhandled exception: std::exception (std::exception)\n```\n\nTODO: Also mention handle_exception - although perhaps delay that to a later chapter?\n\n## Exceptions vs. exceptional futures\nAn asynchronous function can fail in one of two ways: It can fail immediately, by throwing an exception, or it can return a future which will eventually fail (resolve to an exception). These two modes of failure appear similar to the uninitiated, but behave differently when attempting to handle exceptions using `finally()`, `handle_exception()`, or `then_wrapped()`. For example, consider the code:\n\n```cpp\n#include <seastar/core/future.hh>\n#include <iostream>\n#include <exception>\n\nclass my_exception : public std::exception {\n    virtual const char* what() const noexcept override { return \"my exception\"; }\n};\n\nseastar::future<> fail() {\n    return seastar::make_exception_future<>(my_exception());\n}\n\nseastar::future<> f() {\n    return fail().finally([] {\n        std::cout << \"cleaning up\\n\";\n    });\n}\n```\n\nThis code will, as expected, print the \"cleaning up\" message - the asynchronous function `fail()` returns a future which resolves to a failure, and the `finally()` continuation is run despite this failure, as expected.\n\nNow consider that in the above example we had a different definition for `fail()`:\n\n```cpp\nseastar::future<> fail() {\n    throw my_exception();\n}\n```\n\nHere, `fail()` does not return a failing future. Rather, it fails to return a future at all! The exception it throws stops the entire function `f()`, and the `finally()` continuation does not not get attached to the future (which was never returned), and will never run. The \"cleaning up\" message is not printed now.\n\nWe recommend that to reduce the chance for such errors, asynchronous functions should always return a failed future rather than throw an actual exception. If the asynchronous function calls another function _before_ returning a future, and that second function might throw, it should use `try`/`catch` to catch the exception and convert it into a failed future:\n\n```cpp\nvoid inner() {\n    throw my_exception();\n}\nseastar::future<> fail() {\n    try {\n        inner();\n    } catch(...) {\n        return seastar::current_exception_as_future();\n    }\n    return seastar::make_ready_future<>();\n}\n```\n\nHere, `fail()` catches the exception thrown by `inner()`, whatever it might be, and returns a failed future with that failure. Written this way, the `finally()` continuation will be reached, and the \"cleaning up\" message printed.\n\nThe `seastar::current_exception_as_future()` function is equivalent\nto `seastar::make_exception_future(std::current_exception())`, but\nexpands to less code.\n\n>Despite this recommendation that asynchronous functions avoid throwing, some asynchronous functions do throw exceptions in addition to returning exceptional futures. A common example are functions which allocate memory and throw `std::bad_alloc` when running out of memory, instead of returning a future. The `future<> seastar::semaphore::wait()` method is one such function: It returns a future which may be exceptional if the semaphore was `broken()` or the wait timed out, but may also *throw* an exception when failing to allocate memory it needs to hold the list of waiters.\n> Therefore, unless a function --- including asynchronous functions --- is explicitly tagged \"`noexcept`\", the application should be prepared to handle exceptions thrown from it. In modern C++, code usually uses RAII to be exception-safe without sprinkling it with `try`/`catch`.  `seastar::defer()` is a RAII-based idiom that ensures that some cleanup code is run even if an exception is thrown.\n\n\nSeastar has a convenient generic function, `futurize_invoke()`, which can be useful here. `futurize_invoke(func, args...)` runs a function which may return either a future value or an immediate value, and in both cases convert the result into a future value. `futurize_invoke()` also converts an immediate exception thrown by the function, if any, into a failed future, just like we did above. So using `futurize_invoke()` we can make the above example work even if `fail()` did throw exceptions:\n\n```cpp\nseastar::future<> fail() {\n    throw my_exception();\n}\nseastar::future<> f() {\n    return seastar::futurize_invoke(fail).finally([] {\n        std::cout << \"cleaning up\\n\";\n    });\n}\n```\n\nNote that most of this discussion becomes moot if the risk of exception is inside a _continuation_. Consider the following code:\n\n```cpp\nseastar::future<> f() {\n    return seastar::sleep(1s).then([] {\n        throw my_exception();\n    }).finally([] {\n        std::cout << \"cleaning up\\n\";\n    });\n}\n```\n\nHere, the lambda function of the first continuation does throw an exception instead of returning a failed future. However, we do _not_ have the same problem as before, which only happened because an asynchronous function threw an exception _before_ returning a valid future. Here, `f()` does return a valid future immediately - the failure will only be known later, after `sleep()` resolves. The message in `finally()` will be printed. The methods which attach continuations (such as `then()` and `finally()`) run the continuation the same way, so continuation functions may return immediate values or, in this case, throw an immediate exception, and still work properly.\n\n# Lifetime management\nAn asynchronous function starts an operation which may continue long after the function returns: The function itself returns a `future<T>` almost immediately, but it may take a while until this future is resolved.\n\nWhen such an asynchronous operation needs to operate on existing objects, or to use temporary objects, we need to worry about the *lifetime* of these objects: We need to ensure that these objects do not get destroyed before the asynchronous function completes (or it will try to use the freed object and malfunction or crash), and to also ensure that the object finally get destroyed when it is no longer needed (otherwise we will have a memory leak).\nSeastar offers a variety of mechanisms for safely and efficiently keeping objects alive for the right duration. In this section we will explore these mechanisms, and when to use each mechanism.\n\n## Passing ownership to continuation\nThe most straightforward way to ensure that an object is alive when a continuation runs and is destroyed afterwards is to pass its ownership to the continuation. When continuation *owns* the object, the object will be kept until the continuation runs, and will be destroyed as soon as the continuation is not needed (i.e., it may have run, or skipped in case of exception and `then()` continuation).\n\nWe already saw above that the way for a continuation to get ownership of an object is through *capturing*:\n\n```cpp\nseastar::future<> slow_incr(int i) {\n    return seastar::sleep(10ms).then([i] { return i + 1; });\n}\n```\nHere the continuation captures the value of `i`. In other words, the continuation includes a copy of `i`. When the continuation runs 10ms later, it will have access to this value, and as soon as the continuation finishes its object is destroyed, together with its captured copy of `i`. The continuation owns this copy of `i`.\n\nCapturing by value as we did here - making a copy of the object we need in the continuation - is useful mainly for very small objects such as the integer in the previous example. Other objects are expensive to copy, or sometimes even cannot be copied. For example, the following is **not** a good idea:\n```cpp\nseastar::future<> slow_op(std::vector<int> v) {\n    // this makes another copy of v:\n    return seastar::sleep(10ms).then([v] { /* do something with v */ });\n}\n```\nThis would be inefficient - as the vector `v`, potentially very long, will be copied and the copy will be saved in the continuation. In this example, there is no reason to copy `v` - it was anyway passed to the function by value and will not be used again after capturing it into the continuation, as right after the capture, the function returns and destroys its copy of `v`.\n\nFor such cases, C++14 allows *moving* the object into the continuation:\n```cpp\nseastar::future<> slow_op(std::vector<int> v) {\n    // v is not copied again, but instead moved:\n    return seastar::sleep(10ms).then([v = std::move(v)] { /* do something with v */ });\n}\n```\nNow, instead of copying the object `v` into the continuation, it is *moved* into the continuation. The C++11-introduced move constructor moves the vector's data into the continuation and clears the original vector. Moving is a quick operation - for a vector it only requires copying a few small fields such as the pointer to the data. As before, once the continuation is dismissed the vector is destroyed - and its data array (which was moved in the move operation) is finally freed.\n\nTODO: talk about temporary_buffer as an example of an object designed to be moved in this way.\n\nIn some cases, moving the object is undesirable. For example, some code keeps references to an object or one of its fields and the references become invalid if the object is moved. In some complex objects, even the move constructor is slow. For these cases, C++ provides the useful wrapper `std::unique_ptr<T>`. A `unique_ptr<T>` object owns an object of type `T` allocated on the heap. When a `unique_ptr<T>` is moved, the object of type T is not touched at all - just the pointer to it is moved. An example of using `std::unique_ptr<T>` in capture is:\n\n```cpp\nseastar::future<> slow_op(std::unique_ptr<T> p) {\n    return seastar::sleep(10ms).then([p = std::move(p)] { /* do something with *p */ });\n}\n```\n\n`std::unique_ptr<T>` is the standard C++ mechanism for passing unique ownership of an object to a function: The object is only owned by one piece of code at a time, and ownership is transferred by moving the `unique_ptr` object. A `unique_ptr` cannot be copied: If we try to capture p by value, not by move, we will get a compilation error.\n\n## Keeping ownership at the caller\n\nThe technique we described above - giving the continuation ownership of the object it needs to work on - is powerful and safe. But often it becomes hard and verbose to use. When an asynchronous operation involves not just one continuation but a chain of continuations that each needs to work on the same object, we need to pass the ownership of the object between each successive continuation, which can become inconvenient. It is especially inconvenient when we need to pass the same object into two separate asynchronous functions (or continuations) - after we move the object into one, the object needs to be returned so it can be moved again into the second. E.g.,\n```cpp\nseastar::future<> slow_op(T o) {\n    return seastar::sleep(10ms).then([o = std::move(o)] {\n        // first continuation, doing something with o\n        ...\n        // return o so the next continuation can use it!\n        return std::move(o);\n    }).then([](T o) {\n        // second continuation, doing something with o\n        ...\n    });\n}\n```\n\nThis complexity arises because we wanted asynchronous functions and continuations to take the ownership of the objects they operated on. A simpler approach would be to have the *caller* of the asynchronous function continue to be the owner of the object, and just pass *references* to the object to the various other asynchronous functions and continuations which need the object. For example:\n\n```cpp\nseastar::future<> slow_op(T& o) {           // <-- pass by reference\n    return seastar::sleep(10ms).then([&o] {// <-- capture by reference\n        // first continuation, doing something with o\n        ...\n    }).then([&o]) {                        // <-- another capture by reference\n        // second continuation, doing something with o\n        ...\n    });\n}\n```\n\nThis approach raises a question: The caller of `slow_op` is now responsible for keeping the object `o` alive while the asynchronous code started by `slow_op` needs this object. But how will this caller know how long this object is actually needed by the asynchronous operation it started?\n\nThe most reasonable answer is that an asynchronous function may need access to its parameters until the future it returns is resolved - at which point the asynchronous code completes and no longer needs access to its parameters. We therefore recommend that Seastar code adopt the following convention:\n\n> **Whenever an asynchronous function takes a parameter by reference, the caller must ensure that the referred object lives until the future returned by the function is resolved.**\n\nNote that this is merely a convention suggested by Seastar, and unfortunately nothing in the C++ language enforces it. C++ programmers in non-Seastar programs often pass large objects to functions as a const reference just to avoid a slow copy, and assume that the called function will *not* save this reference anywhere. But in Seastar code, that is a dangerous practice because even if the asynchronous function did not intend to save the reference anywhere, it may end up doing it implicitly by passing this reference to another function and eventually capturing it in a continuation.\n\n> It would be nice if future versions of C++ could help us catch incorrect uses of references. Perhaps we could have a tag for a special kind of reference, an \"immediate reference\" which a function can use use immediately (i.e, before returning a future), but cannot be captured into a continuation.\n\nWith this convention in place, it is easy to write complex asynchronous functions functions like `slow_op` which pass the object around, by reference, until the asynchronous operation is done. But how does the caller ensure that the object lives until the returned future is resolved? The following is *wrong*:\n```cpp\nseastar::future<> f() {\n    T obj; // wrong! will be destroyed too soon!\n    return slow_op(obj);\n}\n```\nIt is wrong because the object `obj` here is local to the call of `f`, and is destroyed as soon as `f` returns a future - not when this returned future is resolved! The correct thing for a caller to do would be to create the object `obj` on the heap (so it does not get destroyed as soon as `f` returns), and then run `slow_op(obj)` and when that future resolves (i.e., with `.finally()`), destroy the object.\n\nSeastar provides a convenient idiom, `do_with()` for doing this correctly:\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(T(), [] (auto& obj) {\n        // obj is passed by reference to slow_op, and this is fine:\n        return slow_op(obj);\n    }\n}\n```\n`do_with` will *do* the given function *with* the given object alive.\n\n`do_with` saves the given object on the heap, and calls the given lambda with a reference to the new object. Finally it ensures that the new object is destroyed after the returned future is resolved. Usually, do_with is given an *rvalue*, i.e., an unnamed temporary object or an `std::move()`ed object, and `do_with` moves that object into its final place on the heap. `do_with` returns a future which resolves after everything described above is done (the lambda's future is resolved and the object is destroyed).\n\nFor convenience, `do_with` can also be given multiple objects to hold alive. For example here we create two objects and hold alive them until the future resolves:\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(T1(), T2(), [] (auto& obj1, auto& obj2) {\n        return slow_op(obj1, obj2);\n    }\n}\n```\n\nWhile `do_with` can the lifetime of the objects it holds, if the user accidentally makes copies of these objects, these copies might have the wrong lifetime. Unfortunately, a simple typo like forgetting an \"&\" can cause such accidental copies. For example, the following code is broken:\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(T(), [] (T obj) { // WRONG: should be T&, not T\n        return slow_op(obj);\n    }\n}\n```\nIn this wrong snippet, `obj` is mistakenly not a reference to the object which `do_with` allocated, but rather a copy of it - a copy which is destroyed as soon as the lambda function returns, rather than when the future it returns resolved. Such code will most likely crash because the object is used after being freed. Unfortunately the compiler will not warn about such mistakes. Users should get used to always using the type \"auto&\" with `do_with` - as in the above correct examples - to reduce the chance of such mistakes.\n\n For the same reason, the following code snippet is also wrong:\n```cpp\nseastar::future<> slow_op(T obj); // WRONG: should be T&, not T\nseastar::future<> f() {\n    return seastar::do_with(T(), [] (auto& obj) {\n        return slow_op(obj);\n    }\n}\n```\nHere, although `obj` was correctly passed to the lambda by reference, we later accidentally passed `slow_op()` a copy of it (because here `slow_op` takes the object by value, not by reference), and this copy will be destroyed as soon as `slow_op` returns, not waiting until the returned future resolves.\n\nWhen using `do_with`, always remember it requires adhering to the convention described above: The asynchronous function which we call inside `do_with` must not use the objects held by `do_with` *after* the returned future is resolved. It is a serious use-after-free bug for an asynchronous function to return a future which resolves while still having background operations using the `do_with()`ed objects.\n\nIn general, it is rarely a good idea for an asynchronous function to resolve while leaving behind background operations - even if those operations do not use the `do_with()`ed objects. Background operations that we do not wait for may cause us to run out of memory (if we don't limit their number) and make it difficult to shut down the application cleanly.\n\n\n## Sharing ownership (reference counting)\nIn the beginning of this chapter, we already noted that capturing a copy of an object into a continuation is the simplest way to ensure that the object is alive when the continuation runs and destroyed afterwards. However, complex objects are often expensive (in time and memory) to copy. Some objects cannot be copied at all, or are read-write and the continuation should modify the original object, not a new copy. The solution to all these issues are **reference counted**, a.k.a. **shared** objects:\n\nA simple example of a reference-counted object in Seastar is a `seastar::file`, an object holding an open file object (we will introduce `seastar::file` in a later section). A `file` object can be copied, but copying does not involve copying the file descriptor (let alone the file). Instead, both copies point to the same open file, and a reference count is increased by 1. When a file object is destroyed, the file's reference count is decreased by one, and only when the reference count reaches 0 the underlying file is actually closed.\n\nThe fact that `file` objects can be copied very quickly and all copies actually point to the same file, make it very convenient to pass them to asynchronous code; For example,\n\n```cpp\nseastar::future<uint64_t> slow_size(file f) {\n    return seastar::sleep(10ms).then([f] {\n        return f.size();\n    });\n}\n```\n\nNote how calling `slow_size` is as simple as calling `slow_size(f)`, passing a copy of `f`, without  needing to do anything special to ensure that `f` is only destroyed when no longer needed. That simply happens naturally when nothing refers to `f` any more.\n\nYou may wonder why `return f.size()` in the above example is safe:  Doesn't it start an asynchronous operation on `f` (the file's size may be stored on disk, so not immediately available), and `f` may be  immediately destroyed when we return and nothing keeps holding a copy of `f`? If `f` is really the last reference, that is indeed a bug, but there is another one: the file is never closed. The assumption that makes the code valid is that there is another reference to `f` that will be used to close it. The close member function holds on to the reference count of that object, so it continues to live even if nothing else keeps holding on to it. Since all futures produced by a file object complete before it is closed, all that is needed for correctness is to remember to always close files.\n\nThe reference counting has a run-time cost, but it is usually very small; It is important to remember that Seastar objects are always used by a single CPU only, so the reference-count increment and decrement operations are not the slow atomic operations often used for reference counting, but just regular CPU-local integer operations. Moreover, judicious use of `std::move()` and the compiler's optimizer can reduce the number of unnecessary back-and-forth increment and decrement of the reference count.\n\nC++11 offers a standard way of creating reference-counted shared objects - using the template `std::shared_ptr<T>`. A `shared_ptr` can be used to wrap any type into a reference-counted shared object like `seastar::file` above.  However, the standard `std::shared_ptr` was designed with multi-threaded applications in mind so it uses slow atomic increment/decrement operations for the reference count which we already noted is unnecessary in Seastar. For this reason Seastar offers its own single-threaded implementation of this template, `seastar::shared_ptr<T>`. It is similar to `std::shared_ptr<T>` except no atomic operations are used.\n\nAdditionally, Seastar also provides an even lower overhead variant of `shared_ptr`: `seastar::lw_shared_ptr<T>`. The full-featured `shared_ptr` is complicated by the need to support polymorphic types correctly (a shared object created of one class, and accessed through a pointer to a base class). It makes `shared_ptr` need to add two words to the shared object, and two words to each `shared_ptr` copy. The simplified `lw_shared_ptr` - which does **not** support polymorphic types - adds just one word in the object (the reference count) and each copy is just one word - just like copying a regular pointer. For this reason, the light-weight `seastar::lw_shared_ptr<T>` should be preferred when possible (`T` is not a polymorphic type), otherwise `seastar::shared_ptr<T>`. The slower `std::shared_ptr<T>` should never be used in sharded Seastar applications.\n\n## Saving objects on the stack\nWouldn't it be convenient if we could save objects on a stack just like we normally do in synchronous code? I.e., something like:\n```cpp\nint i = ...;\nseastar::sleep(10ms).get();\nreturn i;\n```\nSeastar allows writing such code, by using a `seastar::thread` object which comes with its own stack.  A complete example using a `seastar::thread` might look like this:\n```cpp\nseastar::future<> slow_incr(int i) {\n    return seastar::async([i] {\n        seastar::sleep(10ms).get();\n        // We get here after the 10ms of wait, i is still available.\n        return i + 1;\n    });\n}\n```\nWe present `seastar::thread`, `seastar::async()` and `seastar::future::get()` in the [seastar::thread] section.\n\n# Advanced futures\n## Futures and interruption\nTODO: A future, e.g., sleep(10s) cannot be interrupted. So if we need to, the promise needs to have a mechanism to interrupt it. Mention pipe's close feature, semaphore stop feature, etc.\n\n## Futures are single use\nTODO: Talk about if we have a `future<int>` variable, as soon as we `get()` or `then()` it, it becomes invalid - we need to store the value somewhere else. Think if there's an alternative we can suggest\n\n# Fibers\nSeastar continuations are normally short, but often chained to one another, so that one continuation does a bit of work and then schedules another continuation for later. Such chains can be long, and often even involve loopings - see the following section, \"Loops\". We call such chains \"fibers\" of execution.\n\nThese fibers are not threads - each is just a string of continuations - but they share some common requirements with traditional threads.  For example, we want to avoid one fiber getting starved while a second fiber continuously runs its continuations one after another.  As another example, fibers may want to communicate - e.g., one fiber produces data that a second fiber consumes, and we wish to ensure that both fibers get a chance to run, and that if one stops prematurely, the other doesn't hang forever.\n\nTODO: Mention fiber-related sections like loops, semaphores, gates, pipes, etc.\n\n# Loops\nA majority of time-consuming computations involve using loops. Seastar provides several primitives for expressing them in a way that composes nicely with the future/promise model. A very important aspect of Seastar loop primitives is that each iteration is followed by a preemption point, thus allowing other tasks to run in between iterations.\n\n## repeat\nA loop created with `repeat` executes its body until it receives a `stop_iteration` object, which informs if the iteration should continue (`stop_iteration::no`) or stop (`stop_iteration::yes`). Next iteration will be launched only after the first one has finished. The loop body passed to `repeat` is expected to have a `future<stop_iteration>` return type.\n```cpp\nseastar::future<int> recompute_number(int number);\n\nseastar::future<> push_until_100(seastar::lw_shared_ptr<std::vector<int>> queue, int element) {\n    return seastar::repeat([queue, element] {\n        if (queue->size() == 100) {\n            return make_ready_future<stop_iteration>(stop_iteration::yes);\n        }\n        return recompute_number(element).then([queue] (int new_element) {\n            queue->push_back(new_element);\n            return stop_iteration::no;\n        });\n    });\n}\n```\n\n## do_until\nDo until is a close relative of `repeat`, but it uses an explicitly passed condition to decide whether it should stop iterating. The above example could be expressed with `do_until` as follows:\n```cpp\nseastar::future<int> recompute_number(int number);\n\nseastar::future<> push_until_100(seastar::lw_shared_ptr<std::vector<int>> queue, int element) {\n    return seastar::do_until([queue] { return queue->size() == 100; }, [queue, element] {\n        return recompute_number(element).then([queue] (int new_element) {\n            queue->push_back(new_element);\n        });\n    });\n}\n```\nNote that the loop body is expected to return a `future<>`, which allows composing complex continuations inside the loop.\n\n## do_for_each\nA `do_for_each` is an equivalent of a `for` loop in Seastar world. It accepts a range (or a pair of iterators) and a function body, which it applies to each argument, in order, one by one. The next iteration will be launched only after the first one has finished, as was the case with `repeat`. As usual, `do_for_each` expects its loop body to return a `future<>`.\n```cpp\nseastar::future<> append(seastar::lw_shared_ptr<std::vector<int>> queue1, seastar::lw_shared_ptr<std::vector<int>> queue2) {\n    return seastar::do_for_each(queue2, [queue1] (int element) {\n        queue1->push_back(element);\n    });\n}\n\nseastar::future<> append_iota(seastar::lw_shared_ptr<std::vector<int>> queue1, int n) {\n    return seastar::do_for_each(boost::make_counting_iterator<size_t>(0), boost::make_counting_iterator<size_t>(n), [queue1] (int element) {\n        queue1->push_back(element);\n    });\n}\n```\n`do_for_each` accepts either an lvalue reference to a container or a pair of iterators. It implies that the responsibility to ensure that the container is alive during the whole loop execution belongs to the caller. If the container needs its lifetime prolonged, it can be easily achieved with `do_with`:\n```cpp\nseastar::future<> do_something(int number);\n\nseastar::future<> do_for_all(std::vector<int> numbers) {\n    // Note that the \"numbers\" vector will be destroyed as soon as this function\n    // returns, so we use do_with to guarantee it lives during the whole loop execution:\n    return seastar::do_with(std::move(numbers), [] (std::vector<int>& numbers) {\n        return seastar::do_for_each(numbers, [] (int number) {\n            return do_something(number);\n        });\n    });\n}\n```\n\n## parallel_for_each\nParallel for each is a high concurrency variant of `do_for_each`. When using `parallel_for_each`, all iterations are queued simultaneously - which means that there's no guarantee in which order they finish their operations.\n\n```cpp\nseastar::future<> flush_all_files(seastar::lw_shared_ptr<std::vector<seastar::file>> files) {\n    return seastar::parallel_for_each(files, [] (seastar::file f) {\n        // file::flush() returns a future<>\n        return f.flush();\n    });\n}\n```\n`parallel_for_each` is a powerful tool, as it allows spawning many tasks in parallel. It can be a great performance gain, but there are also caveats. First of all, too high concurrency may be troublesome - the details can be found in chapter **Limiting parallelism of loops**.\n\nTo restrict the concurrency of `parallel_for_each` by an integer number, use `max_concurrent_for_each` that is described below.\nMore details about dealing with parallelism can be found in chapter **Limiting parallelism of loops**.\n\nSecondly, take note that the order in which iterations will be executed within a `parallel_for_each` loop is arbitrary - if a strict ordering is needed, consider using `do_for_each` instead.\n\nTODO: map_reduce, as a shortcut (?) for parallel_for_each which needs to produce some results (e.g., logical_or of boolean results), so we don't need to create a lw_shared_ptr explicitly (or do_with).\n\nTODO: See seastar commit \"input_stream: Fix possible infinite recursion in consume()\" for an example on why recursion is a possible, but bad, replacement for repeat(). See also my comment on https://groups.google.com/d/msg/seastar-dev/CUkLVBwva3Y/3DKGw-9aAQAJ on why Seastar's iteration primitives should be used over tail call optimization.\n\n## max_concurrent_for_each\nMax concurrent for each is a variant of `parallel_for_each` with restricted parallelism.\nIt accepts an additional parameter - `max_concurrent` - with which, up to `max_concurrent` iterations are queued simultaneously, with no guarantee in which order they finish their operations.\n\n```cpp\nseastar::future<> flush_all_files(seastar::lw_shared_ptr<std::vector<seastar::file>> files, size_t max_concurrent) {\n    return seastar::max_concurrent_for_each(files, max_concurrent, [] (seastar::file f) {\n        return f.flush();\n    });\n}\n```\n\nDetermining the maximum concurrency limit is out of the scope of this document.\nIt should typically be derived from the actual capabilities of the system the software is running on, like the number of parallel execution units or I/O channels, so to optimize utilization of resources without overwhelming the system.\n\n# when_all: Waiting for multiple futures\nAbove we've seen `parallel_for_each()`, which starts a number of asynchronous operations, and then waits for all to complete. Seastar has another idiom, `when_all()`, for waiting for several already-existing futures to complete.\n\nThe first variant of `when_all()` is variadic, i.e., the futures are given as separate parameters, the exact number of which is known at compile time. The individual futures may have different types. For example,\n\n```cpp\n#include <seastar/core/sleep.hh>\n\nfuture<> f() {\n    using namespace std::chrono_literals;\n    future<int> slow_two = sleep(2s).then([] { return 2; });\n    return when_all(sleep(1s), std::move(slow_two),\n                    make_ready_future<double>(3.5)\n           ).discard_result();\n}\n```\n\nThis starts three futures - one which sleeps for one second (and doesn't return anything), one which sleeps for two seconds and returns the integer 2, and one which returns the double 3.5 immediately - and then waits for them. The `when_all()` function returns a future which resolves as soon as all three futures resolves, i.e., after two seconds. This future also has a value, which we shall explain below, but in this example, we simply waited for the future to resolve and discarded its value.\n\nNote that `when_all()` accept only rvalues, which can be temporaries (like the return value of an asynchronous function or `make_ready_future`) or an `std::move()`'ed variable holding a future.\n\nThe future returned by `when_all()` resolves to a tuple of futures which are already resolved, and contain the results of the three input futures. Continuing the above example,\n\n```cpp\nfuture<> f() {\n    using namespace std::chrono_literals;\n    future<int> slow_two = sleep(2s).then([] { return 2; });\n    return when_all(sleep(1s), std::move(slow_two),\n                    make_ready_future<double>(3.5)\n           ).then([] (auto tup) {\n            std::cout << std::get<0>(tup).available() << \"\\n\";\n            std::cout << std::get<1>(tup).get() << \"\\n\";\n            std::cout << std::get<2>(tup).get() << \"\\n\";\n    });\n}\n```\n\nThe output of this program (which comes after two seconds) is `1, 2, 3.5`: the first future in the tuple is available (but has no value), the second has the integer value 2, and the third a double value 3.5 - as expected.\n\nOne or more of the waited futures might resolve in an exception, but this does not change how `when_all()` works: It still waits for all the futures to resolve, each with either a value or an exception, and in the returned tuple some of the futures may contain an exception instead of a value. For example,\n\n```cpp\nfuture<> f() {\n    using namespace std::chrono_literals;\n    future<> slow_success = sleep(1s);\n    future<> slow_exception = sleep(2s).then([] { throw 1; });\n    return when_all(std::move(slow_success), std::move(slow_exception)\n           ).then([] (auto tup) {\n            std::cout << std::get<0>(tup).available() << \"\\n\";\n            std::cout << std::get<1>(tup).failed() << \"\\n\";\n            std::get<1>(tup).ignore_ready_future();\n    });\n}\n```\n\nBoth futures are `available()` (resolved), but the second has `failed()` (resulted in an exception instead of a value). Note how we called `ignore_ready_future()` on this failed future, because silently ignoring a failed future is considered a bug, and will result in an \"Exceptional future ignored\" error message. More typically, an application will log the failed future instead of ignoring it.\n\nThe above example demonstrate that `when_all()` is inconvenient and verbose to use properly. The results are wrapped in a tuple, leading to verbose tuple syntax, and uses ready futures which must all be inspected individually for an exception to avoid error messages.\n\nSo Seastar also provides an easier to use `when_all_succeed()` function. This function too returns a future which resolves when all the given futures have resolved. If all of them succeeded, it passes a tuple of the resulting values to continuation, without wrapping each of them in a future first. Sometimes, it could be tedious to unpack the tuple for consuming the resulting values. In that case, `then_unpack()` can be used in place of `then()`. `then_unpack()` unpacks the returned tuple and passes its elements to the following continuation as its parameters. If, however, one or more of the futures failed, `when_all_succeed()` resolves to a failed future, containing the exception from one of the failed futures. If more than one of the given future failed, one of those will be passed on (it is unspecified which one is chosen), and the rest will be silently ignored. For example,\n\n```cpp\nusing namespace seastar;\nfuture<> f() {\n    using namespace std::chrono_literals;\n    return when_all_succeed(sleep(1s), make_ready_future<int>(2),\n                    make_ready_future<double>(3.5)\n            ).then_unpack([] (int i, double d) {\n        std::cout << i << \" \" << d << \"\\n\";\n    });\n}\n```\n\nNote how the integer and double values held by the futures are conveniently passed, individually to the continuation. Since `sleep()` does not contain a value, it is waited for, but no third value is passed to the continuation. That also means that if we `when_all_succeed()` on several `future<>` (without a value), the result is a `future<tuple<>>`:\n\n```cpp\nusing namespace seastar;\nfuture<> f() {\n    using namespace std::chrono_literals;\n    return when_all_succeed(sleep(1s), sleep(2s), sleep(3s)).then_unpack([] {\n        return make_ready_future<>();\n    });\n}\n```\n\nThis example simply waits for 3 seconds (the maximum of 1, 2 and 3 seconds).\n\nAn example of `when_all_succeed()` with an exception:\n\n```cpp\nusing namespace seastar;\nfuture<> f() {\n    using namespace std::chrono_literals;\n    return when_all_succeed(make_ready_future<int>(2),\n                    make_exception_future<double>(\"oops\")\n            ).then_unpack([] (int i, double d) {\n        std::cout << i << \" \" << d << \"\\n\";\n    }).handle_exception([] (std::exception_ptr e) {\n        std::cout << \"exception: \" << e << \"\\n\";\n    });\n}\n```\n\nIn this example, one of the futures fails, so the result of `when_all_succeed` is a failed future, so the normal continuation is not run, and the `handle_exception()` continuation is done.\n\nTODO: also explain `when_all` and `when_all_succeed` for vectors.\n\n# Semaphores\nSeastar's semaphores are the standard computer-science semaphores, adapted for futures. A semaphore is a counter into which you can deposit units or take them away. Taking units from the counter may wait if not enough units are available.\n\n## Limiting parallelism with semaphores\nThe most common use for a semaphore in Seastar is for limiting parallelism, i.e., limiting the number of instances of some code which can run in parallel. This can be important when each of the parallel invocations uses a limited resource (e.g., memory) so letting an unlimited number of them run in parallel can exhaust this resource.\n\nConsider a case where an external source of events (e.g., an incoming network request) causes an asynchronous function ```g()``` to be called. Imagine that we want to limit the number of concurrent ```g()``` operations to 100. I.e., If g() is started when 100 other invocations are still ongoing, we want it to delay its real work until one of the other invocations has completed. We can do this with a semaphore:\n\n```cpp\nseastar::future<> g() {\n    static thread_local seastar::semaphore limit(100);\n    return limit.wait(1).then([] {\n        return slow(); // do the real work of g()\n    }).finally([] {\n        limit.signal(1);\n    });\n}\n```\n\nIn this example, the semaphore starts with the counter at 100. The asynchronous operation `slow()` is only started when we can reduce the counter by one (`wait(1)`), and when `slow()` is done, either successfully or with exception, the counter is increased back by one (```signal(1)```). This way, when 100 operations have already started their work and have not yet finished, the 101st operation will wait, until one of the ongoing operations finishes and returns a unit to the semaphore. This ensures that at each time we have at most 100 concurrent `slow()` operations running in the above code.\n\nNote how we used a ```static thread_local``` semaphore, so that all calls to ```g()``` from the same shard count towards the same limit; As usual, a Seastar application is sharded so this limit is separate per shard (CPU thread). This is usually fine, because sharded applications consider resources to be separate per shard.\n\nLuckily, the above code happens to be exception safe: `limit.wait(1)` can throw an exception when it runs out of memory (keeping a list of waiters), and in that case the semaphore counter is not decreased but the continuations below are not run so it is not increased either. `limit.wait(1)` can also return an exceptional future when the semaphore is *broken* (we'll discuss this later) but in that case the extra `signal()` call is ignored. Finally, `slow()` may also throw, or return an exceptional future, but the `finally()` ensures the semaphore is still increased.\n\nHowever, as the application code becomes more complex, it becomes harder to ensure that we never forget to call `signal()` after the operation is done, regardless of which code path or exceptions happen. As an example of what might go wrong, consider the following *buggy* code snippet, which differs subtly from the above one, and also appears, on first sight, to be correct:\n\n```cpp\nseastar::future<> g() {\n    static thread_local seastar::semaphore limit(100);\n    return limit.wait(1).then([] {\n        return slow().finally([] { limit.signal(1); });\n    });\n}\n```\n\nBut this version is **not** exception safe: Consider what happens if `slow()` throws an exception before returning a future (this is different from `slow()` returning an exceptional future - we discussed this difference in the section about exception handling). In this case, we decreased the counter, but the `finally()` will never be reached, and the counter will never be increased back. There is a way to fix this code, by replacing the call to `slow()` with `seastar::futurize_invoke(slow)`. But the point we're trying to make here is not how to fix buggy code, but rather that by using the separate `semaphore::wait()` and `semaphore::signal()` functions, you can very easily get things wrong.\n\nFor exception safety, in C++ it is generally not recommended to have separate resource acquisition and release functions.  Instead, C++ offers safer mechanisms for acquiring a resource (in this case semaphore units) and later releasing it: lambda functions, and RAII (\"resource acquisition is initialization\"):\n\nThe lambda-based solution is a function ```seastar::with_semaphore()``` which is a shortcut for the code in the examples above:\n\n```cpp\nseastar::future<> g() {\n    static thread_local seastar::semaphore limit(100);\n    return seastar::with_semaphore(limit, 1, [] {\n        return slow(); // do the real work of g()\n    });\n}\n```\n\n`with_semaphore()`, like the earlier code snippets, waits for the given number of units from the semaphore, then runs the given lambda, and when the future returned by the lambda is resolved, `with_semaphore()` returns back the units to the semaphore. `with_semaphore()` returns a future which only resolves after all these steps are done.\n\nThe function `seastar::get_units()` is more general. It provides an exception-safe alternative to `seastar::semaphore`'s separate `wait()` and `signal()` methods, based on C++'s RAII philosophy: The function returns an opaque units object, which while held, keeps the semaphore's counter decreased - and as soon as this object is destructed, the counter is increased back. With this interface you cannot forget to increase the counter, or increase it twice, or increase without decreasing: The counter will always be decreased once when the units object is created, and if that succeeded, increased when the object is destructed. When the units object is moved into a continuation, no matter how this continuation ends, when the continuation is destructed, the units object is destructed and the units are returned to the semaphore's counter. The above examples, written with `get_units()`, looks like this:\n\n```cpp\nseastar::future<> g() {\n    static thread_local semaphore limit(100);\n    return seastar::get_units(limit, 1).then([] (auto units) {\n        return slow().finally([units = std::move(units)] {});\n    });\n}\n```\n\nNote the somewhat convoluted way that `get_units()` needs to be used: The continuations must be nested because we need the `units` object to be moved to the last continuation. If `slow()` returns a future (and does not throw immediately),  the `finally()` continuation captures the `units` object until everything is done, but does not run any code.\n\nSeastar's programmers should generally avoid using the the `semaphore::wait()` and `semaphore::signal()` functions directly, and always prefer either `with_semaphore()` (when applicable) or `get_units()`.\n\n\n## Limiting resource use\nBecause semaphores support waiting for any number of units, not just 1, we can use them for more than simple limiting of the *number* of parallel invocation. For example, consider we have an asynchronous function ```using_lots_of_memory(size_t bytes)```, which uses ```bytes``` bytes of memory, and we want to ensure that not more than 1 MB of memory is used by all parallel invocations of this function --- and that additional calls are delayed until previous calls have finished. We can do this with a semaphore:\n\n```cpp\nseastar::future<> using_lots_of_memory(size_t bytes) {\n    static thread_local seastar::semaphore limit(1000000); // limit to 1MB\n    return seastar::with_semaphore(limit, bytes, [bytes] {\n        // do something allocating 'bytes' bytes of memory\n    });\n}\n```\n\nWatch out that in the above example, a call to `using_lots_of_memory(2000000)` will return a future that never resolves, because the semaphore will never contain enough units to satisfy the semaphore wait. `using_lots_of_memory()` should probably check whether `bytes` is above the limit, and throw an exception in that case. Seastar doesn't do this for you.\n\n\n## Limiting parallelism of loops\nAbove, we looked at a function `g()` which gets called by some external event, and wanted to control its parallelism. In this section, we look at parallelism of loops, which also can be controlled with semaphores.\n\nConsider the following simple loop:\n\n```cpp\n#include <seastar/core/sleep.hh>\nseastar::future<> slow() {\n    std::cerr << \".\";\n    return seastar::sleep(std::chrono::seconds(1));\n}\nseastar::future<> f() {\n    return seastar::repeat([] {\n        return slow().then([] { return seastar::stop_iteration::no; });\n    });\n}\n```\n\nThis loop runs the ```slow()``` function (taking one second to complete) without any parallelism --- the next ```slow()``` call starts only when the previous one completed. But what if we do not need to serialize the calls to ```slow()```, and want to allow multiple instances of it to be ongoing concurrently?\n\nNaively, we could achieve more parallelism, by starting the next call to ```slow()``` right after the previous call --- ignoring the future returned by the previous call to ```slow()``` and not waiting for it to resolve:\n```cpp\nseastar::future<> f() {\n    return seastar::repeat([] {\n        slow();\n        return seastar::stop_iteration::no;\n    });\n}\n```\n\nBut in this loop, there is no limit to the amount of parallelism --- millions of ```sleep()``` calls might be active in parallel, before the first one ever returned. Eventually, this loop may consume all available memory and crash.\n\nUsing a semaphore allows us to run many instances of ```slow()``` in parallel, but limit the number of these parallel instances to, in the following example, 100:\n\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {\n        return seastar::repeat([&limit] {\n            return limit.wait(1).then([&limit] {\n                seastar::futurize_invoke(slow).finally([&limit] {\n                    limit.signal(1);\n                });\n                return seastar::stop_iteration::no;\n            });\n        });\n    });\n}\n```\n\nNote how this code differs from the code we saw above for limiting the number of parallel invocations of a function `g()`:\n\n1. Here we cannot use a single `thread_local` semaphore. Each call to `f()` has its loop with parallelism of 100, so needs its own semaphore \"`limit`\", kept alive during the loop with `do_with()`.\n2. Here we do not wait for `slow()` to complete before continuing the loop, i.e., we do not `return` the future chain starting at `futurize_invoke(slow)`. The loop continues to the next iteration when a semaphore unit becomes available, while (in our example) 99 other operations might be ongoing in the background and we do not wait for them.\n\nIn the examples in this section, we cannot use the `with_semaphore()` shortcut. `with_semaphore()` returns a future which only resolves after the lambda's returned future resolves. But in the above example, the loop needs to know when just the semaphore units are available, to start the next iteration --- and not wait for the previous iteration to complete. We could not achieve that with `with_semaphore()`. But the more general exception-safe idiom, `seastar::get_units()`, can be used in this case, and is recommended:\n\n\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {\n        return seastar::repeat([&limit] {\n    \t    return seastar::get_units(limit, 1).then([] (auto units) {\n\t            slow().finally([units = std::move(units)] {});\n\t            return seastar::stop_iteration::no;\n\t        });\n        });\n    });\n}\n```\n\nThe above examples are not realistic, because they have a never-ending loop and the future returned by `f()` will never resolve. In more realistic cases, the loop has an end, and at the end of the loop we need to wait for all the background operations which the loop started. We can do this by ```wait()```ing on the original count of the semaphore: When the full count is finally available, it means that *all* the operations have completed. For example, the following loop ends after 456 iterations:\n\n```cpp\nseastar::future<> f() {\n    return seastar::do_with(seastar::semaphore(100), [] (auto& limit) {\n        return seastar::do_for_each(boost::counting_iterator<int>(0),\n                boost::counting_iterator<int>(456), [&limit] (int i) {\n            return seastar::get_units(limit, 1).then([] (auto units) {\n                slow().finally([units = std::move(units)] {});\n\t        });\n        }).finally([&limit] {\n            return limit.wait(100);\n        });\n    });\n}\n````\n\nThe last `finally` is what ensures that we wait for the last operations to complete: After the `repeat` loop ends (whether successfully or prematurely because of an exception in one of the iterations), we do a `wait(100)` to wait for the semaphore to reach its original value 100, meaning that all operations that we started have completed. Without this `finally`, the future returned by `f()` will resolve *before* all the iterations of the loop actually completed (the last 100 may still be running).\n\nIn the idiom we saw in the above example, the same semaphore is used both for limiting the number of background operations, and later to wait for all of them to complete. Sometimes, we want several different loops to use the same semaphore to limit their *total* parallelism. In that case we must use a separate mechanism for waiting for the completion of the background operations started by the loop. The most convenient way to wait for ongoing operations is using a gate, which we will describe in detail later. A typical example of a loop whose parallelism is limited by an external semaphore:\n\n```cpp\nthread_local seastar::semaphore limit(100);\nseastar::future<> f() {\n    return seastar::do_with(seastar::gate(), [] (auto& gate) {\n        return seastar::do_for_each(boost::counting_iterator<int>(0),\n                boost::counting_iterator<int>(456), [&gate] (int i) {\n            return seastar::get_units(limit, 1).then([&gate] (auto units) {\n                gate.enter();\n                seastar::futurize_invoke(slow).finally([&gate, units = std::move(units)] {\n                    gate.leave();\n                });\n\t        });\n        }).finally([&gate] {\n            return gate.close();\n        });\n    });\n}\n```\nIn this code, we use the external semaphore `limit` to limit the number of concurrent operations, but additionally have a gate specific to this loop to help us wait for all ongoing operations to complete.\n\nTODO: also allow `get_units()` or something similar on a gate, and use that instead of the explicit gate.enter/gate.leave.\n\nTODO: say something about semaphore fairness - if someone is waiting for a lot of units and later someone asks for 1 unit, will both wait or will the request for 1 unit be satisfied?\n\nTODO: say something about broken semaphores? (or in later section especially about breaking/closing/shutting down/etc?)\n\nTODO: Have a few paragraphs, or even a section, on additional uses of semaphores. One is for mutual exclusion using semaphore(1) - we need to explain why although why in Seastar we don't have multiple threads touching the same data, if code is composed of different continuations (i.e., a fiber) it can switch to a different fiber in the middle, so if data needs to be protected between two continuations, it needs a mutex. Another example is something akin to wait_all: we start with a semaphore(0), run a known number N of asynchronous functions with finally sem.signal(), and from all this return the future sem.wait(N). PERHAPS even have a separate section on mutual exclusion, where we begin with semaphore(1) but also mention shared_mutex\n\n# Pipes\nSeastar's `pipe<T>` is a mechanism to transfer data between two fibers, one producing data, and the other consuming it. It has a fixed-size buffer to ensures a balanced execution of the two fibers, because the producer fiber blocks when it writes to a full pipe, until the consumer fiber gets to run and read from the pipe.\n\nA `pipe<T>` resembles a Unix pipe, in that it has a read side, a write side, and a fixed-sized buffer between them, and supports either end to be closed independently (and EOF or broken pipe when using the other side). A `pipe<T>` object holds the reader and write sides of the pipe as two separate objects. These objects can be moved into two different fibers.  Importantly, if one of the pipe ends is destroyed (i.e., the continuations capturing it end), the other end of the pipe will stop blocking, so the other fiber will not hang.\n\nThe pipe's read and write interfaces are future-based blocking. I.e., the write() and read() methods return a future which is fulfilled when the operation is complete. The pipe is single-reader single-writer, meaning that until the future returned by read() is fulfilled, read() must not be called again (and same for write).\nNote: The pipe reader and writer are movable, but *not* copyable. It is often convenient to wrap each end in a shared pointer, so it can be copied (e.g., used in an std::function which needs to be copyable) or easily captured into multiple continuations.\n\n# Shutting down a service with a gate\nConsider an application which has some long operation `slow()`, and many such operations may be started at any time. A number of `slow()` operations may even even be active in parallel.  Now, you want to shut down this service, but want to make sure that before that, all outstanding operations are completed. Moreover, you don't want to allow new `slow()` operations to start while the shut-down is in progress.\n\nThis is the purpose of a `seastar::gate`. A gate `g` maintains an internal counter of operations in progress. We call `g.enter()` when entering an operation (i.e., before running `slow()`), and call `g.leave()` when leaving the operation (when a call to `slow()` completed). The method `g.close()` *closes the gate*, which means it forbids any further calls to `g.enter()` (such attempts will generate an exception); Moreover `g.close()` returns a future which resolves when all the existing operations have completed. In other words, when `g.close()` resolves, we know that no more invocations of `slow()` can be in progress - because the ones that already started have completed, and new ones could not have started.\n\nThe construct\n```cpp\nseastar::with_gate(g, [] { return slow(); })\n```\ncan be used as a shortcut to the idiom\n```cpp\ng.enter();\nslow().finally([&g] { g.leave(); });\n```\n\nHere is a typical example of using a gate:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <seastar/core/gate.hh>\n#include <boost/iterator/counting_iterator.hpp>\n\nseastar::future<> slow(int i) {\n    std::cerr << \"starting \" << i << \"\\n\";\n    return seastar::sleep(std::chrono::seconds(10)).then([i] {\n        std::cerr << \"done \" << i << \"\\n\";\n    });\n}\nseastar::future<> f() {\n    return seastar::do_with(seastar::gate(), [] (auto& g) {\n        return seastar::do_for_each(boost::counting_iterator<int>(1),\n                boost::counting_iterator<int>(6),\n                [&g] (int i) {\n            seastar::with_gate(g, [i] { return slow(i); });\n            // wait one second before starting the next iteration\n            return seastar::sleep(std::chrono::seconds(1));\n\t\t}).then([&g] {\n            seastar::sleep(std::chrono::seconds(1)).then([&g] {\n                // This will fail, because it will be after the close()\n                seastar::with_gate(g, [] { return slow(6); });\n            });\n            return g.close();\n        });\n    });\n}\n```\n\nIn this example, we have a function `future<> slow()` taking 10 seconds to complete. We run it in a loop 5 times, waiting 1 second between calls, and surround each call with entering and leaving the gate (using `with_gate`). After the 5th call, while all calls are still ongoing (because each takes 10 seconds to complete), we close the gate and wait for it before exiting the program. We also test that new calls cannot begin after closing the gate, by trying to enter the gate again one second after closing it.\n\nThe output of this program looks like this:\n```\nstarting 1\nstarting 2\nstarting 3\nstarting 4\nstarting 5\nWARNING: exceptional future ignored of type 'seastar::gate_closed_exception': gate closed\ndone 1\ndone 2\ndone 3\ndone 4\ndone 5\n```\n\nHere, the invocations of `slow()` were started at 1 second intervals. After the \"`starting 5`\" message, we closed the gate and another attempt to use it resulted in a `seastar::gate_closed_exception`, which we ignored and hence this message. At this point the application waits for the future returned by `g.close()`. This will happen once all the `slow()` invocations have completed: Immediately after printing \"`done 5`\", the test program stops.\n\nAs explained so far, a gate can prevent new invocations of an operation, and wait for any in-progress operations to complete. However, these in-progress operations may take a very long time to complete. Often, a long operation would like to know that a shut-down has been requested, so it could stop its work prematurely. An operation can check whether its gate was closed by calling the gate's `check()` method: If the gate is already closed, the `check()` method throws an exception (the same `seastar::gate_closed_exception` that `enter()` would throw at that point). The intent is that the exception will cause the operation calling it to stop at this point.\n\nIn the previous example code, we had an un-interruptible operation `slow()` which slept for 10 seconds. Let's replace it by a loop of 10 one-second sleeps, calling `g.check()` each second:\n\n```cpp\nseastar::future<> slow(int i, seastar::gate &g) {\n    std::cerr << \"starting \" << i << \"\\n\";\n    return seastar::do_for_each(boost::counting_iterator<int>(0),\n                                boost::counting_iterator<int>(10),\n            [&g] (int) {\n        g.check();\n        return seastar::sleep(std::chrono::seconds(1));\n    }).finally([i] {\n        std::cerr << \"done \" << i << \"\\n\";\n    });\n}\n```\n\nNow, just one second after gate is closed (after the \"starting 5\" message is printed), all the `slow()` operations notice the gate was closed, and stop. As expected, the exception stops the `do_for_each()` loop, and the `finally()` continuation is performed so we see the \"done\" messages for all five operations.\n\n\n# Introducing shared-nothing programming\n\nTODO: Explain in more detail Seastar's shared-nothing approach where the entire memory is divided up-front to cores, malloc/free and pointers only work on one core.\n\nTODO: Introduce our shared_ptr (and lw_shared_ptr) and sstring and say the standard ones use locked instructions which are unnecessary when we assume these objects (like all others) are for a single thread. Our futures and continuations do the same.\n\n\n# More about Seastar's event loop\nTODO: Mention the event loop (scheduler). remind that continuations on the same thread do not run in parallel, so do not need locks, atomic variables, etc (different threads shouldn't access the same data - more on that below). continuations obviously must not use blocking operations, or they block the whole thread.\n\nTODO: Talk about polling that we currently do, and how today even sleep() or waiting for incoming connections or whatever, takes 100% of all CPUs.\n\n# Introducing Seastar's network stack\n\nTODO: Mention the two modes of operation: Posix and native (i.e., take a L2 (Ethernet) interface (vhost or dpdk) and on top of it we built (in Seastar itself) an L3 interface (TCP/IP)).\n\nFor optimal performance, Seastar's network stack is sharded just like Seastar applications are: each shard (thread) takes responsibility for a different subset of the connections. Each incoming connection is directed to one of the threads, and after a connection is established, it continues to be handled on the same thread.\n\nIn the examples we saw earlier, `main()` ran our function `f()` only once, on the first thread. Unless the server is run with the `\"-c1\"` option (one thread only), this will mean that any connection arriving to a different thread will not be handled. So in all the examples below, we will need to run the same service loop on all cores. We can easily do this with the `smp::submit_to` function:\n\n```cpp\nseastar::future<> service_loop();\n\nseastar::future<> f() {\n    return seastar::parallel_for_each(std::views::iota(0u, seastar::smp::count),\n            [] (unsigned c) {\n        return seastar::smp::submit_to(c, service_loop);\n    });\n}\n```\n\nHere we ask each of Seastar cores (from 0 to `smp::count`-1) to run the same function `service_loop()`. Each of these invocations returns a future, and `f()` will return when all of them have returned (in the examples below, they will never return - we will discuss shutting down services in later sections).\n\nWe begin with a simple example of a TCP network server written in Seastar. This server repeatedly accepts connections on TCP port 1234, and returns an empty response:\n\n```cpp\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/net/api.hh>\n\nseastar::future<> service_loop() {\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234})),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                [] (seastar::accept_result res) {\n                    std::cout << \"Accepted connection from \" << res.remote_address << \"\\n\";\n            });\n        });\n    });\n}\n```\n\nThis code works as follows:\n\n1. The ```listen()``` call creates a ```server_socket``` object, ```listener```, which listens on TCP port 1234 (on any network interface).\n2. We use ```do_with()``` to ensure that the listener socket lives throughout the loop.\n3. To handle one connection, we call ```listener```'s  ```accept()``` method. This method returns a ```future<accept_result>```, i.e., is eventually resolved with an incoming TCP connection from a client (```accept_result.connection```) and the client's IP address and port (```accept_result.remote_address```).\n4. To repeatedly accept new connections, we use the ```keep_doing()``` loop idiom. ```keep_doing()``` runs its lambda parameter over and over, starting the next iteration as soon as the future returned by the previous iteration completes. The iterations only stop if an exception is encountered. The future returned by ```keep_doing()``` itself completes only when the iteration stops (i.e., only on exception).\n\nOutput from this server looks like the following example:\n\n```\n$ ./a.out\nAccepted connection from 127.0.0.1:47578\nAccepted connection from 127.0.0.1:47582\n...\n```\n\nIf you run the above example server immediately after killing the previous server, it often fails to start again, complaining that:\n\n```\n$ ./a.out\nprogram failed with uncaught exception: bind: Address already in use\n```\n\nThis happens because by default, Seastar refuses to reuse the local port if there are any vestiges of old connections using that port. In our silly server, because the server is the side which first closes the connection, each connection lingers for a while in the \"```TIME_WAIT```\" state after being closed, and these prevent ```listen()``` on the same port from succeeding. Luckily, we can give listen an option to work despite these remaining ```TIME_WAIT```. This option is analogous to ```socket(7)```'s ```SO_REUSEADDR``` option:\n\n```cpp\n    seastar::listen_options lo;\n    lo.reuse_address = true;\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),\n```\n\nMost servers will always turn on this ```reuse_address``` listen option. Stevens' book \"Unix Network Programming\" even says that \"All TCP servers should specify this socket option to allow the server to be restarted\". Therefore in the future Seastar should probably default to this option being on --- even if for historic reasons this is not the default in Linux's socket API.\n\nLet's advance our example server by outputting some canned response to each connection, instead of closing each connection immediately with an empty reply.\n\n```cpp\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/net/api.hh>\n\nconst char* canned_response = \"Seastar is the future!\\n\";\n\nseastar::future<> service_loop() {\n    seastar::listen_options lo;\n    lo.reuse_address = true;\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                    [] (seastar::accept_result res) {\n                auto s = std::move(res.connection);\n                auto out = s.output();\n                return seastar::do_with(std::move(s), std::move(out),\n                        [] (auto& s, auto& out) {\n                    return out.write(canned_response).then([&out] {\n                        return out.close();\n                    });\n                });\n            });\n        });\n    });\n}\n```\n\nThe new part of this code begins by taking the ```connected_socket```'s ```output()```, which returns an ```output_stream<char>``` object. On this output stream ```out``` we can write our response using the ```write()``` method. The simple-looking ```write()``` operation is in fact a complex asynchronous operation behind the scenes,  possibly causing multiple packets to be sent, retransmitted, etc., as needed. ```write()``` returns a future saying when it is ok to ```write()``` again to this output stream; This does not necessarily guarantee that the remote peer received all the data we sent it, but it guarantees that the output stream has enough buffer space (or in the TCP case, there is enough room in the TCP congestion window) to allow another write to begin.\n\nAfter ```write()```ing the response to ```out```, the example code calls ```out.close()``` and waits for the future it returns. This is necessary, because ```write()``` attempts to batch writes so might not have yet written anything to the TCP stack at this point, and only when close() concludes can we be sure that all the data we wrote to the output stream has actually reached the TCP stack --- and only at this point we may finally dispose of the ```out``` and ```s``` objects.\n\nIndeed, this server returns the expected response:\n\n```\n$ telnet localhost 1234\n...\nSeastar is the future!\nConnection closed by foreign host.\n```\n\nIn the above example we only saw writing to the socket. Real servers will also want to read from the socket. The ```connected_socket```'s ```input()``` method returns an ```input_stream<char>``` object which can be used to read from the socket. The simplest way to read from this stream is using the ```read()``` method which returns a future ```temporary_buffer<char>```, containing some more bytes read from the socket --- or an empty buffer when the remote end shut down the connection.\n\n```temporary_buffer<char>``` is a convenient and safe way to pass around byte buffers that are only needed temporarily (e.g., while processing a request). As soon as this object goes out of scope (by normal return, or exception), the memory it holds gets automatically freed. Ownership of buffer can also be transferred by ```std::move()```ing it. We'll discuss ```temporary_buffer``` in more details in a later section.\n\nLet's look at a simple example server involving both reads an writes. This is a simple echo server, as described in RFC 862: The server listens for connections from the client, and once a connection is established, any data received is simply sent back - until the client closes the connection.\n\n```cpp\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/net/api.hh>\n\nseastar::future<> handle_connection(seastar::connected_socket s) {\n    auto out = s.output();\n    auto in = s.input();\n    return do_with(std::move(s), std::move(out), std::move(in),\n            [] (auto& s, auto& out, auto& in) {\n        return seastar::repeat([&out, &in] {\n            return in.read().then([&out] (auto buf) {\n                if (buf) {\n                    return out.write(std::move(buf)).then([&out] {\n                        return out.flush();\n                    }).then([] {\n                        return seastar::stop_iteration::no;\n                    });\n                } else {\n                    return seastar::make_ready_future<seastar::stop_iteration>(\n                            seastar::stop_iteration::yes);\n                }\n            });\n        }).then([&out] {\n            return out.close();\n        });\n    });\n}\n\nseastar::future<> service_loop_3() {\n    seastar::listen_options lo;\n    lo.reuse_address = true;\n    return seastar::do_with(seastar::listen(seastar::make_ipv4_address({1234}), lo),\n            [] (auto& listener) {\n        return seastar::keep_doing([&listener] () {\n            return listener.accept().then(\n                    [] (seastar::accept_result res) {\n                // Note we ignore, not return, the future returned by\n                // handle_connection(), so we do not wait for one\n                // connection to be handled before accepting the next one.\n                (void)handle_connection(std::move(res.connection)).handle_exception(\n                        [] (std::exception_ptr ep) {\n                    fmt::print(stderr, \"Could not handle connection: {}\\n\", ep);\n                });\n            });\n        });\n    });\n}\n```\n\nThe main function ```service_loop()``` loops accepting new connections, and for each connection calls ```handle_connection()``` to handle this connection. Our ```handle_connection()``` returns a future saying when handling this connection completed, but importantly, we do ***not*** wait for this future: Remember that ```keep_doing``` will only start the next iteration when the future returned by the previous iteration is resolved. Because we want to allow parallel ongoing connections, we don't want the next ```accept()``` to wait until the previously accepted connection was closed. So we call ```handle_connection()``` to start the handling of the connection, but return nothing from the continuation, which resolves that future immediately, so ```keep_doing``` will continue to the next ```accept()```.\n\nThis demonstrates how easy it is to run parallel _fibers_ (chains of continuations) in Seastar - When a continuation runs an asynchronous function but ignores the future it returns, the asynchronous operation continues in parallel, but never waited for.\n\nIt is often a mistake to silently ignore an exception, so if the future we're ignoring might resolve with an except, it is recommended to handle this case, e.g. using a ```handle_exception()``` continuation. In our case, a failed connection is fine (e.g., the client might close its connection will we're sending it output), so we did not bother to handle the exception.\n\nThe ```handle_connection()``` function itself is straightforward --- it repeatedly calls ```read()``` read on the input stream, to receive a ```temporary_buffer``` with some data, and then moves this temporary buffer into a ```write()``` call on the output stream. The buffer will eventually be freed, automatically, when the ```write()``` is done with it. When ```read()``` eventually returns an empty buffer signifying the end of input, we stop ```repeat```'s iteration by returning a ```stop_iteration::yes```.\n\n# Sharded services\n\nIn the previous section we saw that a Seastar application usually needs to run its code on all available CPU cores. We saw that the `seastar::smp::submit_to()` function allows the main function, which initially runs only on the first core, to start the server's code on all `seastar::smp::count` cores.\n\nHowever, usually one needs not just to run code on each core, but also to have an object that contains the state of this code. Additionally, one may like to interact with those different objects, and also have a mechanism to stop the service running on the different cores.\n\nThe `seastar::sharded<T>` template provides a structured way create such a _sharded service_. It creates a separate object of type `T` in each core, and provides mechanisms to interact with those copies, to start some code on each, and finally to cleanly stop the service.\n\nTo use `seastar::sharded`, first create a class for the object holding the state of the service on a single core. For example:\n\n```cpp\n#include <seastar/core/future.hh>\n#include <iostream>\n\nclass my_service {\npublic:\n    std::string _str;\n    my_service(const std::string& str) : _str(str) { }\n    seastar::future<> run() {\n        std::cerr << \"running on \" << seastar::engine().cpu_id() <<\n            \", _str = \" << _str << \"\\n\";\n        return seastar::make_ready_future<>();\n    }\n    seastar::future<> stop() {\n        return seastar::make_ready_future<>();\n    }\n};\n```\n\nThe only mandatory method in this object is `stop()`, which will be called in each core when we want to stop the sharded service and want to wait until it stops on all cores.\n\nNow let's see how to use it:\n\n```cpp\n#include <seastar/core/sharded.hh>\n\nseastar::sharded<my_service> s;\n\nseastar::future<> f() {\n    return s.start(std::string(\"hello\")).then([] {\n        return s.invoke_on_all([] (my_service& local_service) {\n            return local_service.run();\n        });\n    }).then([] {\n        return s.stop();\n    });\n}\n```\n\nThe `s.start()` starts the service by creating a `my_service` object on each of the cores. The arguments to `s.start()`, if any (in this example, `std::string(\"hello\")`), are passed to `my_service`'s constructor.\n\nBut `s.start()` did not start running any code yet (besides the object's constructor). For that, we have the `s.invoke_on_all()` which runs the given lambda on all the cores - giving each lambda the local `my_service` object on that core. In this example, we have a `run()` method on each object, so we run that.\n\nFinally, at the end of the run we want to give the service on all cores a chance to shut down cleanly, so we call `s.stop()`. This will call the `stop()` method on each core's object, and wait for all of them to finish. Calling `s.stop()` before destroying `s` is mandatory - Seastar will warn you if you forget to do it.\n\nIn addition to `invoke_on_all()` which runs the same code on all shards, another feature a sharded service often needs is for one shard to invoke code another specific shard. This is done by calling the sharded service's `invoke_on()` method. For example:\n\n```cpp\nseastar::sharded<my_service> s;\n...\nreturn s.invoke_on(0, [] (my_service& local_service) {\n    std::cerr << \"invoked on \" << seastar::engine().cpu_id() <<\n        \", _str = \" << local_service._str << \"\\n\";\n});\n```\n\nThis runs the lambda function on shard 0, with a reference to the local `my_service` object on that shard.\n\n\n# Shutting down cleanly\n\nTODO: Handling interrupt, shutting down services, etc.\n\nMove the seastar::gate section here.\n\n# Command line options\n## Standard Seastar command-line options\nAll Seastar applications accept a standard set of command-line arguments, such as those we've already seen above: The `-c` option for controlling the number of threads used, or  `-m` for determining the amount of memory given to the application.\n\nTODO: list and explain more of these options.\n\nEvery Seastar application also accepts the `-h` (or `--help`) option, which lists and explains all the available options --- the standard Seastar ones, and the user-defined ones as explained below.\n\n## User-defined command-line options\nSeastar parses the command line options (`argv[]`) when it is passed to `app_template::run()`, looking for its own standard options. Therefore, it is not recommended that the application tries to parse `argv[]` on its own because the application might not understand some of the standard Seastar options and not be able to correctly skip them.\n\nRather, applications which want to have command-line options of their own should tell Seastar's command line parser of these additional application-specific options, and ask Seastar's command line parser to recognize them too. Seastar's command line parser is actually the Boost library's `boost::program_options`. An application adds its own option by using the `add_options()` and `add_positional_options()` methods on the `app_template` to define options, and later calling `configuration()` to retrieve the setting of these options. For example,\n\n```cpp\n#include <iostream>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    namespace bpo = boost::program_options;\n    app.add_options()\n        (\"flag\", \"some optional flag\")\n        (\"size,s\", bpo::value<int>()->default_value(100), \"size\")\n        ;\n    app.add_positional_options({\n       { \"filename\", bpo::value<std::vector<seastar::sstring>>()->default_value({}),\n         \"sstable files to verify\", -1}\n    });\n    app.run(argc, argv, [&app] {\n        auto& args = app.configuration();\n        if (args.count(\"flag\")) {\n            std::cout << \"Flag is on\\n\";\n        }\n        std::cout << \"Size is \" << args[\"size\"].as<int>() << \"\\n\";\n        auto& filenames = args[\"filename\"].as<std::vector<seastar::sstring>>();\n        for (auto&& fn : filenames) {\n            std::cout << fn << \"\\n\";\n        }\n        return seastar::make_ready_future<>();\n    });\n    return 0;\n}\n```\n\nIn this example, we add via `add_options()` two application-specific options: `--flag` is an optional parameter which doesn't take any additional arguments, and `--size` (or `-s`) takes an integer value, which defaults (if this option is missing) to 100. Additionally, we ask via `add_positional_options()` that an unlimited number of arguments that do not begin with a \"`-`\" --- the so-called _positional_ arguments --- be collected to a vector of strings under the \"filename\" option. Some example outputs from this program:\n\n```\n$ ./a.out\nSize is 100\n$ ./a.out --flag\nFlag is on\nSize is 100\n$ ./a.out --flag -s 3\nFlag is on\nSize is 3\n$ ./a.out --size 3 hello hi\nSize is 3\nhello\nhi\n$ ./a.out --filename hello --size 3 hi\nSize is 3\nhello\nhi\n```\n\n`boost::program_options` has more powerful features, such as required options, option checking and combining, various option types, and more. Please refer to Boost's documentation for more information.\n\n# Debugging a Seastar program\n## Debugging ignored exceptions\nIf a future resolves with an exception, and the application neglects to handle that exception or to explicitly ignore it, the application may have missed an important problem. This is likely to be an application bug.\n\nTherefore, Seastar prints a warning message to the log if a future is destroyed when it stores an exception that hasn't been handled.\n\nFor example, consider this code:\n```cpp\n#include <seastar/core/future.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/app-template.hh>\n\nclass myexception {};\n\nseastar::future<> g() {\n    return seastar::make_exception_future<>(myexception());\n}\n\nseastar::future<> f() {\n    g();\n    return seastar::sleep(std::chrono::seconds(1));\n}\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    app.run(argc, argv, f);\n}\n```\n\nHere, the main function `f()` calls `g()`, but doesn't do anything with the future it returns. But this future resolves with an exception, and this exception is silently ignored. So Seastar prints this warning message about the ignored exception:\n```\nWARN  2020-03-31 11:08:09,208 [shard 0] seastar - Exceptional future ignored: myexception, backtrace:   /lib64/libasan.so.5+0x6ce7f\n  0x1a64193\n  0x1a6265f\n  0xf326cc\n  0xeaf1a0\n  0xeaffe4\n  0xead7be\n  0xeb5917\n  0xee2477\n  0xec312e\n  0xec8fcf\n  0xeec765\n  0xee1b29\n  0xed9fab\n  0xec27c8\n  0xec867f\n  0xf00acc\n  0xef179d\n  0xef1824\n  0xef18b5\n  0xee4827\n  0xee470f\n  0xf00f81\n  0xebac29\n  0xeb9095\n  0xeb9174\n  0xeb925a\n  0xeb9964\n  0xebef89\n  0x10f74c3\n  0x10fb439\n  0x11005fd\n  0xec4f08\n  0xec2f43\n  0xec3461\n  0xeadabe\n  /lib64/libc.so.6+0x271a2\n  0xead52d\n```\n\nThis message says that an exceptional future was ignored, and that the type of the exception was \"`myexception`\". The type of the exception is usually not enough to pinpoint where the problem happened, so the warning message also includes the backtrace - the call chain - leading to where the exceptional future was destroyed. The backtrace is given as a list of addresses, where code in other shared libraries is written as a shared library plus offset (when ASLR is enabled, the shared libraries are mapped in a different address each time).\n\nSeastar includes a utility, `seastar-addr2line`, for translating these addresses into readable backtraces including exact method names, source files and line numbers. This utility needs the _unstripped_ executable. Typically, a stripped executable is used for production, but an unstripped copy is kept separately to be used in debugging - including `seastar-addr2line`.\n\nTo decode the backtrace, we run\n```\nseastar-addr2line -e a.out\n```\nAnd then paste the list of addresses in the warning message, and conclude with a `control-D` (it's also possible, if you want, to put the list of addresses in the `seastar-addr2line` command line). The result looks like this:\n\n```\nvoid seastar::backtrace<seastar::current_backtrace()::{lambda(seastar::frame)#1}>(seastar::current_backtrace()::{lambda(seastar::frame)#1}&&) at include/seastar/util/backtrace.hh:56\nseastar::current_backtrace() at src/util/backtrace.cc:84\nseastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at src/core/future.cc:116\nseastar::future_state_base::~future_state_base() at include/seastar/core/future.hh:335\nseastar::future_state<>::~future_state() at include/seastar/core/future.hh:414\n (inlined by) seastar::future<>::~future() at include/seastar/core/future.hh:990\nf() at test.cc:12\nstd::_Function_handler<seastar::future<> (), seastar::future<> (*)()>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:286\nstd::function<seastar::future<> ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690\nseastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const at src/core/app-template.cc:131\nstd::_Function_handler<seastar::future<int> (), seastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:286\nstd::function<seastar::future<int> ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690\nseastar::future<int> seastar::futurize<seastar::future<int> >::invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) at include/seastar/core/future.hh:1670\nauto seastar::futurize_invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) at include/seastar/core/future.hh:1754\nseastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::{lambda()#1}::operator()() at src/core/app-template.cc:120 (discriminator 4)\nstd::_Function_handler<void (), seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/9/bits/std_function.h:300\nstd::function<void ()>::operator()() const at /usr/include/c++/9/bits/std_function.h:690\nseastar::apply_helper<std::function<void ()>&, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/apply.hh:36\nauto seastar::apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/apply.hh:44\nseastar::future<> seastar::futurize<void>::apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/future.hh:1634\nauto seastar::futurize_apply<std::function<void ()>&>(std::function<void ()>&, std::tuple<>&&) at include/seastar/core/future.hh:1766\nseastar::future<>::then<std::function<void ()>, seastar::future<> >(std::function<void ()>&&)::{lambda()#1}::operator()() at include/seastar/core/future.hh:1191\nseastar::noncopyable_function<seastar::future<> ()>::direct_vtable_for<seastar::future<>::then<std::function<void ()>, seastar::future<> >(std::function<void ()>&&)::{lambda()#1}>::call(seastar::noncopyable_function<seastar::future<> ()> const*) at include/seastar/util/noncopyable_function.hh:101\nseastar::noncopyable_function<seastar::future<> ()>::operator()() const at include/seastar/util/noncopyable_function.hh:184\nseastar::apply_helper<seastar::noncopyable_function<seastar::future<> ()>, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/apply.hh:36\nauto seastar::apply<seastar::noncopyable_function<seastar::future<> ()>>(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/apply.hh:44\nseastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> ()>>(seastar::noncopyable_function<seastar::future<> ()>&&, std::tuple<>&&) at include/seastar/core/future.hh:1660\nseastar::future<>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, seastar::future_state<>&&)#1}::operator()(seastar::internal::promise_base_with_type<>, seastar::future_state<>) at include/seastar/core/future.hh:1213\nseastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, seastar::future_state<>&&)#1}>::run_and_dispose() at include/seastar/core/future.hh:509\nseastar::reactor::run_tasks(seastar::reactor::task_queue&) at src/core/reactor.cc:2124\nseastar::reactor::run_some_tasks() at src/core/reactor.cc:2539 (discriminator 2)\nseastar::reactor::run() at src/core/reactor.cc:2694\nseastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at src/core/app-template.cc:199 (discriminator 1)\nseastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at src/core/app-template.cc:115 (discriminator 2)\nseastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&) at src/core/app-template.cc:130 (discriminator 2)\nmain at test.cc:19 (discriminator 1)\n__libc_start_main at /usr/src/debug/glibc-2.30-34-g994e529a37/csu/../csu/libc-start.c:308\n_start at ??:?\n```\n\nMost of the lines at the bottom of this backtrace are not interesting, and just showing the internal details of how Seastar ended up running the main function `f()`. The only interesting part is the _first_ few lines:\n\n```\nseastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at src/core/future.cc:116\nseastar::future_state_base::~future_state_base() at include/seastar/core/future.hh:335\nseastar::future_state<>::~future_state() at include/seastar/core/future.hh:414\n (inlined by) seastar::future<>::~future() at include/seastar/core/future.hh:990\nf() at test.cc:12\n```\n\nHere we see that the warning message was printed by the `seastar::report_failed_future()` function which was called when destroying a future (`future<>::~future`) that had not been handled. The future's destructor was called in line 11 of our test code (`26.cc`), which is indeed the line where we called `g()` and ignored its result.\nThis backtrace gives us an accurate understanding of where our code destroyed an exceptional future without handling it first, which is usually helpful in solving these kinds of bugs. Note that this technique does not tell us where the exception was first created, nor what code passed around the exceptional future before it was destroyed - we just learn where the future was destroyed. To learn where the exception was originally thrown, see the next section:\n\n## Finding where an exception was thrown\nSometimes an application logs an exception, and we want to know where in the code the exception was originally thrown. Unlike languages like Java, C++ does not have a builtin method of attaching a backtrace to every exception. So Seastar provides functions which allow adding to an exception the backtrace recorded when throwing it.\n\nFor example, in the following code we throw and catch an `std::runtime_error` normally:\n\n```cpp\n#include <seastar/core/future.hh>\n#include <seastar/util/log.hh>\n#include <exception>\n#include <iostream>\n\nseastar::future<> g() {\n    return seastar::make_exception_future<>(std::runtime_error(\"hello\"));\n}\n\nseastar::future<> f() {\n    return g().handle_exception([](std::exception_ptr e) {\n        std::cerr << \"Exception: \" << e << \"\\n\";\n    });\n}\n```\nThe output is\n```\nException: std::runtime_error (hello)\n```\nFrom this output, we have no way of knowing that the exception was thrown in `g()`. We can solve this if we use `make_exception_future_with_backtrace` instead of `make_exception_future`:\n\n```\n#include <util/backtrace.hh>\nseastar::future<> g() {\n    return seastar::make_exception_future_with_backtrace<>(std::runtime_error(\"hello\"));\n}\n```\nNow the output looks like\n```\nException: seastar::internal::backtraced<std::runtime_error> (hello Backtrace:   0x678bd3\n  0x677204\n  0x67736b\n  0x678cd5\n  0x4f923c\n  0x4f9c38\n  0x4ff4d0\n...\n)\n```\nWhich, as above, can be converted to a human-readable backtrace by using the `seastar-addr2line` script.\n\nIn addition to `seastar::make_exception_future_with_backtrace()`, Seastar also provides a function `throw_with_backtrace()`, to throw an exception instead of returning an exceptional future. For example:\n```\n    seastar::throw_with_backtrace<std::runtime_error>(\"hello\");\n```\n\nIn the current implementation, both `make_exception_future_with_backtrace` and `throw_with_backtrace` require that the original exception type (in the above example, `std::runtime_error`) is a subclass of the `std::exception` class. The original exception provides a `what()` string, and the wrapped exception adds the backtrace to this string, as demonstrated above. Moreover, the wrapped exception type is a _subclass_ of the original exception type, which allows `catch(...)` code to continue filtering by the exception original type - despite the addition of the backtrace.\n\n\n## Debugging with gdb\n\n```\nhandle SIGUSR1 pass noprint\nhandle SIGALRM pass noprint\n```\n\n# Promise objects\n\nAs we already defined above, An **asynchronous function**, also called a **promise**, is a function which returns a future and arranges for this future to be eventually resolved. As we already saw, an asynchronous function is usually written in terms of other asynchronous functions, for example we saw the function `slow()` which waits for the existing asynchronous function `sleep()` to complete, and then returns 3:\n\n```cpp\nseastar::future<int> slow() {\n    using namespace std::chrono_literals;\n    return seastar::sleep(100ms).then([] { return 3; });\n}\n```\n\nThe most basic building block for writing promises is the **promise object**, an object of type `promise<T>`. A `promise<T>` has a method `future<T> get_future()` to returns a future, and a method `set_value(T)`, to resolve this future. An asynchronous function can create a promise object, return its future, and the `set_value` method to be eventually called - which will finally resolve the future it returned.\n\nIn the following example we create a promise that manages the process of printing 10 messages, once every second.\nWe start by creating an empty promise to work with. We then spin up a `seastart::thread` to perform the work we want. When the work, printing those messages, is completed we call `promise::set_value` to mark the completion of the task. Other than that we wait for the future which is generated by our promise, just like any other future.\n```cpp\n#include <seastar/core/future.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <iostream>\n\n\nseastar::future<> f() {\n    return seastar::do_with(seastar::promise<>(), [](auto& promise) {\n        (void)seastar::async([&promise]() {\n                using namespace std::chrono_literals;\n                for (int i = 0; i < 10; i++) {\n                    std::cout << i << \"...\" << std::flush;\n                    seastar::sleep(1s).wait();\n                }\n                std::cout << std::endl;\n\n                promise.set_value();\n        });\n\n        return promise.get_future();\n    });\n}\n```\n\n# Memory allocation in Seastar\n## Per-thread memory allocation\nSeastar requires that applications be sharded, i.e., that code running on different threads operate on different objects in memory. We already saw in [Seastar memory] how Seastar takes over a given amount of memory (often, most of the machine's memory) and divides it equally between the different threads. Modern multi-socket machines have non-uniform memory access (NUMA), meaning that some parts of memory are closer to some of the cores, and Seastar takes this knowledge into account when dividing the memory between threads. Currently, the division of memory between threads is static, and equal - the threads are expected to experience roughly equal amount of load and require roughly equal amounts of memory.\n\nTo achieve this per-thread allocation, Seastar redefines the C library functions `malloc()`, `free()`, and their numerous relatives --- `calloc()`, `realloc()`, `posix_memalign()`, `memalign()`, `malloc_usable_size()`, and `malloc_trim()`. It also redefines the C++ memory allocation functions, `operator new`, `operator delete`,  and all their variants (including array versions, the C++14 delete taking a size, and the C++17 variants taking required alignment).\n\nIt is important to remember that Seastar's different threads *can* see memory allocated by other threads, but they are nonetheless strongly discouraged from actually doing this. Sharing data objects between threads on modern multi-core machines results in stiff performance penalties from locks, memory barriers, and cache-line bouncing. Rather, Seastar encourages applications to avoid sharing objects between threads when possible (by *sharding* --- each thread owns a subset of the objects), and when threads do need to interact they do so with explicit message passing, with `submit_to()`, as we shall see later.\n\n## Foreign pointers\nAn object allocated on one thread will be owned by this thread, and eventually should be freed by the same thread. Freeing memory on the *wrong* thread is strongly discouraged, but is currently supported (albeit slowly) to support library code beyond Seastar's control. For example, `std::exception_ptr` allocates memory; So if we invoke an asynchronous operation on a remote thread and this operation returns an exception, when we free the returned `std::exception_ptr` this will happen on the \"wrong\" core. So Seastar allows it, but inefficiently.\n\nIn most cases objects should spend their entire life on a single thread and be used only by this thread. But in some cases we want to reassign ownership of an object which started its life on one thread, to a different thread. This can be done using a `seastar::foreign_ptr<>`. A pointer, or smart pointer, to an object is wrapped in a `seastar::foreign_ptr<P>`. This wrapper can then be moved into code running in a different thread (e.g., using `submit_to()`).\n\nThe most common use-case is a `seastar::foreign_ptr<std::unique_ptr<T>>`. The thread receiving this `foreign_ptr` will get exclusive use of the object, and when it destroys this wrapper, it will go back to the original thread to destroy the object. Note that the object is not only freed on the original shard - it is also *destroyed* (i.e., its destructor is run) there. This is often important when the object's destructor needs to access other state which belongs to the original shard - e.g., unlink itself from a container.\n\nAlthough `foreign_ptr` ensures that the object's *destructor* automatically runs on the object's home thread, it does not absolve the user from worrying where to run the object's other methods. Some simple methods, e.g., methods which just read from the object's fields, can be run on the receiving thread. However, other methods may need to access other data owned by the object's home shard, or need to prevent concurrent operations. Even if we're sure that object is now used exclusively by the receiving thread, such methods must still be run, explicitly, on the home thread:\n```\n    // fp is some foreign_ptr<>\n    return smp::submit_to(fp.get_owner_shard(), [p=fp.get()]\n        { return p->some_method(); });\n```\nSo `seastar::foreign_ptr<>` not only has functional benefits (namely, to run the destructor on the home shard), it also has *documentational* benefits - it warns the programmer to watch out every time the object is used, that this is a *foreign* pointer, and if we want to do anything non-trivial with the pointed object, we may need to do it on the home shard.\n\nAbove, we discussed the case of transferring ownership of an object to a another shard, via `seastar::foreign_ptr<std::unique_ptr<T>>`. However, sometimes the sender does not want to relinquish ownership of the object. Sometimes, it wants the remote thread to operate on its object and return with the object intact. Sometimes, it wants to send the same object to multiple shards. In such cases, `seastar::foreign_ptr<seastar::lw_shared_ptr<T>>` is useful. The user needs to watch out, of course, not to operate on the same object from multiple threads concurrently. If this cannot be ensured by program logic alone, some methods of serialization must be used - such as running the operations on the home shard with `submit_to()` as described above.\n\nNormally, a `seastar::foreign_ptr` cannot not be copied - only moved. However, when it holds a smart pointer that can be copied (namely, a `shared_ptr`), one may want to make an additional copy of that pointer and create a second `foreign_ptr`. Doing this is inefficient and asynchronous (it requires communicating with the original owner of the object to create the copies), so a method `future<foreign_ptr> copy()` needs to be explicitly used instead of the normal copy constructor.\n\n# Seastar::thread\nSeastar's programming model, using futures and continuations, is very powerful and efficient.  However, as we've already seen in examples above, it is also relatively verbose: Every time that we need to wait before proceeding with a computation, we need to write another continuation. We also need to worry about passing the data between the different continuations (using techniques like those described in the [Lifetime management] section). Simple flow-control constructs such as loops also become more involved using continuations. For example, consider this simple classical synchronous code:\n```cpp\n    std::cout << \"Hi.\\n\";\n    for (int i = 1; i < 4; i++) {\n        sleep(1);\n        std::cout << i << \"\\n\";\n    }\n```\nIn Seastar, using futures and continuations, we need to write something like this:\n```cpp\n    std::cout << \"Hi.\\n\";\n    return seastar::do_for_each(boost::counting_iterator<int>(1),\n        boost::counting_iterator<int>(4), [] (int i) {\n        return seastar::sleep(std::chrono::seconds(1)).then([i] {\n            std::cout << i << \"\\n\";\n        });\n    });\n```\n\nBut Seastar also allows, via `seastar::thread`, to write code which looks more like synchronous code. A `seastar::thread` provides an execution environment where blocking is tolerated; You can issue an asynchronous function, and wait for it in the same function, rather then establishing a callback to be called with `future<>::then()`:\n\n```cpp\n    seastar::thread th([] {\n        std::cout << \"Hi.\\n\";\n        for (int i = 1; i < 4; i++) {\n            seastar::sleep(std::chrono::seconds(1)).get();\n            std::cout << i << \"\\n\";\n        }\n    });\n```\nA `seastar::thread` is **not** a separate operating system thread. It still uses continuations, which are scheduled on Seastar's single thread (per core). It works as follows:\n\nThe `seastar::thread` allocates a 128KB stack, and runs the given function until the it *blocks* on the call to a future's `get()` method. Outside a `seastar::thread` context, `get()` may only be called on a future which is already available. But inside a thread, calling `get()` on a future which is not yet available stops running the thread function, and schedules a continuation for this future, which continues to run the thread's function (on the same saved stack) when the future becomes available.\n\nJust like normal Seastar continuations, `seastar::thread`s always run on the same core they were launched on. They are also cooperative: they are never preempted except when `seastar::future::get()` blocks or on explicit calls to `seastar::thread::yield()`.\n\nIt is worth reiterating that a `seastar::thread` is not a POSIX thread, and it can only block on Seastar futures, not on blocking system calls. The above example used `seastar::sleep()`, not the `sleep()` system call. The `seastar::thread`'s function can throw and catch exceptions normally. Remember that `get()` will throw an exception if the future resolves with an exception.\n\nIn addition to `seastar::future::get()`, we also have `seastar::future::wait()` to wait *without* fetching the future's result. This can sometimes be useful when you want to avoid throwing an exception when the future failed (as `get()` does). For example:\n```cpp\n    future<char> getchar();\n    int try_getchar() noexcept { // run this in seastar::thread context\n        future fut = get_char();\n        fut.wait();\n        if (fut.failed()) {\n            return -1;\n        } else {\n            // Here we already know that get() will return immediately,\n            // and will not throw.\n            return fut.get();\n        }\n    }\n```\n\n## Starting and ending a seastar::thread\nAfter we created a `seastar::thread` object, we need wait until it ends, using its `join()` method. We also need to keep that object alive until `join()` completes. A complete example using `seastar::thread` will therefore look like this:\n\n```cpp\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\nseastar::future<> f() {\n    seastar::thread th([] {\n        std::cout << \"Hi.\\n\";\n        for (int i = 1; i < 4; i++) {\n            seastar::sleep(std::chrono::seconds(1)).get();\n            std::cout << i << \"\\n\";\n        }\n    });\n    return do_with(std::move(th), [] (auto& th) {\n        return th.join();\n    });\n}\n```\n\nThe `seastar::async()` function provides a convenient shortcut for creating a `seastar::thread` and returning a future which resolves when the thread completes:\n```cpp\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\nseastar::future<> f() {\n    return seastar::async([] {\n        std::cout << \"Hi.\\n\";\n        for (int i = 1; i < 4; i++) {\n            seastar::sleep(std::chrono::seconds(1)).get();\n            std::cout << i << \"\\n\";\n        }\n    });\n}\n```\n\n`seastar::async()`'s lambda may return a value, and `seastar::async()` returns it when it completes. For example:\n\n```cpp\nseastar::future<seastar::sstring> read_file(sstring file_name) {\n    return seastar::async([file_name] () {  // lambda executed in a thread\n        file f = seastar::open_file_dma(file_name).get();  // get() call \"blocks\"\n        auto buf = f.dma_read(0, 512).get();  // \"block\" again\n        return seastar::sstring(buf.get(), buf.size());\n    });\n};\n```\n\nWhile `seastar::thread`s and `seastar::async()` make programming more convenient, they also add overhead beyond that of programming directly with continuations. Most notably, each `seastar::thread` requires additional memory for its stack. It is therefore not a good idea to use a `seastar::thread` to handle a highly concurrent operation. For example, if you need to handle 10,000 concurrent requests, do not use a `seastar::thread` to handle each --- use futures and continuations. But if you are writing code where you know that only a few instances will ever run concurrently, e.g., a background cleanup operation in your application, `seastar::thread` is a good match. `seastar::thread` is also great for code which doesn't care about performance --- such as test code.\n\n# Isolation of application components\nSeastar makes multi-tasking very easy - as easy as running an asynchronous function. It is therefore easy for a server to do many unrelated things in parallel. For example, a server might be in the process of answering 100 users' requests, and at the same time also be making progress on some long background operation.\n\nBut in the above example, what percentage of the CPU and disk throughput will the background operation get? How long can one of the user's requests be delayed by the background operation? Without the mechanisms we describe in this section, these questions cannot be reliably answered:\n\n* The background operation may be a very \"considerate\" single fiber, i.e., run a very short continuation and then schedule the next continuation to run later. At each point the scheduler sees 100 request-handling continuations and just one of the background continuations ready to run. The background task gets around 1% of the CPU time, and users' requests are hardly delayed.\n* On the other hand, the background operation may spawn 1,000 fibers in parallel and have 1,000 ready-to-run continuations at each time. The background operation will get about 90% of the runtime, and the continuation handling a user's request may get stuck behind 1,000 of these background continuations, and experience huge latency.\n\nComplex Seastar applications often have different components which run in parallel and have different performance objectives. In the above example we saw two components - user requests and the background operation.  The first goal of the mechanisms we describe in this section is to _isolate_ the performance of each component from the others; In other words, the throughput and latency of one component should not depend on decisions that another component makes - e.g., how many continuations it runs in parallel. The second goal is to allow the application to _control_ this isolation, e.g., in the above example allow the application to explicitly control the amount of CPU the background operation receives, so that it completes at a desired pace.\n\nIn the above examples we used CPU time as the limited resource that the different components need to share effectively. As we show later, another important shared resource is disk I/O.\n\n## Scheduling groups (CPU scheduler)\nConsider the following asynchronous function `loop()`, which loops until some shared variable `stop` becomes true. It keeps a `counter` of the number of iterations until stopping, and returns this counter when finally stopping.\n```cpp\nseastar::future<long> loop(int parallelism, bool& stop) {\n    return seastar::do_with(0L, [parallelism, &stop] (long& counter) {\n        return seastar::parallel_for_each(std::views::iota(0u, parallelism),\n            [&stop, &counter]  (unsigned c) {\n                return seastar::do_until([&stop] { return stop; }, [&counter] {\n                    ++counter;\n                    return seastar::make_ready_future<>();\n                });\n            }).then([&counter] { return counter; });\n    });\n}\n```\nThe `parallelism` parameter determines the parallelism of the silly counting operation: `parallelism=1` means we have just one loop incrementing the counter; `parallelism=10` means we start 10 loops in parallel all incrementing the same counter.\n\nWhat happens if we start two `loop()` calls in parallel and let them run for 10 seconds?\n```c++\nseastar::future<> f() {\n    return seastar::do_with(false, [] (bool& stop) {\n        seastar::sleep(std::chrono::seconds(10)).then([&stop] {\n            stop = true;\n        });\n        return seastar::when_all_succeed(loop(1, stop), loop(1, stop)).then_unpack(\n            [] (long n1, long n2) {\n                std::cout << \"Counters: \" << n1 << \", \" << n2 << \"\\n\";\n            });\n    });\n}\n```\nIt turns out that if the two `loop()` calls had the same parallelism `1`, we get roughly the same amount of work from both of them:\n```\nCounters: 3'559'635'758, 3'254'521'376\n```\nBut if for example we ran a `loop(1)` in parallel with a `loop(10)`, the result is that the `loop(10)` gets 10 times more work done:\n```\nCounters: 629'482'397, 6'320'167'297\n```\n\nWhy does the amount of work that loop(1) can do in ten seconds depends on the parallelism chosen by its competitor, and how can we solve this?\n\nThe reason this happens is as follows: When a future resolves and a continuation was linked to it, this continuation becomes ready to run. By default, Seastar's scheduler keeps a single list of ready-to-run continuations (in each shard, of course), and runs the continuations at the same order they became ready to run. In the above example, `loop(1)` always has one ready-to-run continuation, but `loop(10)`, which runs 10 loops in parallel, always has ten ready-to-run continuations. So for every continuation of `loop(1)`, Seastar's default scheduler will run 10 continuations of `loop(10)`, which is why loop(10) gets 10 times more work done.\n\nTo solve this, Seastar allows an application to define separate components known as **scheduling groups**, which each has a separate list of ready-to-run continuations. Each scheduling group gets to run its own continuations on a desired percentage of the CPU time, but the number of runnable continuations in one scheduling group does not affect the amount of CPU that another scheduling group gets. Let's look at how this is done:\n\nA scheduling group is defined by a value of type `scheduling_group`. This value is opaque, but internally it is a small integer (similar to a process ID in Linux). We use the `seastar::with_scheduling_group()` function to run code in the desired scheduling group:\n\n```cpp\nseastar::future<long>\nloop_in_sg(int parallelism, bool& stop, seastar::scheduling_group sg) {\n    return seastar::with_scheduling_group(sg, [parallelism, &stop] {\n        return loop(parallelism, stop);\n    });\n}\n```\n\nTODO: explain what `with_scheduling_group` group really does, how the group is \"inherited\" to the continuations started inside it.\n\n\nNow let's create two scheduling groups, and run `loop(1)` in the first scheduling group and `loop(10)` in the second scheduling group:\n```cpp\nseastar::future<> f() {\n    return seastar::when_all_succeed(\n            seastar::create_scheduling_group(\"loop1\", 100),\n            seastar::create_scheduling_group(\"loop2\", 100)).then_unpack(\n        [] (seastar::scheduling_group sg1, seastar::scheduling_group sg2) {\n        return seastar::do_with(false, [sg1, sg2] (bool& stop) {\n            seastar::sleep(std::chrono::seconds(10)).then([&stop] {\n                stop = true;\n            });\n            return seastar::when_all_succeed(loop_in_sg(1, stop, sg1), loop_in_sg(10, stop, sg2)).then_unpack(\n                [] (long n1, long n2) {\n                    std::cout << \"Counters: \" << n1 << \", \" << n2 << \"\\n\";\n                });\n        });\n    });\n}\n```\nHere we created two scheduling groups, `sg1` and `sg2`. Each scheduling group has an arbitrary name (which is used for diagnostic purposes only), and a number of *shares*, a number traditionally between 1 and 1000: If one scheduling group has twice the number of shares than a second scheduling group, it will get twice the amount of CPU time. In this example, we used the same number of shares (100) for both groups, so they should get equal CPU time.\n\nUnlike most objects in Seastar which are separate per shard, Seastar wants the identities and numbering of the scheduling groups to be the same on all shards, because it is important when invoking tasks on remote shards. For this reason, the function to create a scheduling group, `seastar::create_scheduling_group()`, is an asynchronous function returning a `future<scheduling_group>`.\n\nRunning the above example, with both scheduling group set up with the same number of shares (100), indeed results in both scheduling groups getting the same amount of CPU time:\n```\nCounters: 3'353'900'256, 3'350'871'461\n```\n\nNote how now both loops got the same amount of work done - despite one loop having 10 times the parallelism of the second loop.\n\nIf we change the definition of the second scheduling group to have 200 shares, twice the number of shares of the first scheduling group, we'll see the second scheduling group getting twice the amount of CPU time:\n```\nCounters: 2'273'783'385, 4'549'995'716\n```\n## Latency\nTODO: Task quota, preempt, loops with built-in preemption check, etc.\n\n## Disk I/O scheduler\nTODO\n\n## Network scheduler\nTODO: Say that not yet available. Give example of potential problem - e.g., sharing a slow WAN link.\n\n## Controllers\nTODO: Talk about how to dynamically change the number of shares, and why.\n\n## Multi-tenancy\nTODO\n\n# General application structuring tips\n\n## Initialization and cleanup\n\nIn C++, RAII is typically used to coordinate initialization and cleanup. In asyncronous programming, RAII\nis not available, but for the control plane it is useful to bring up a seastar::thread to make use of it.\n\nIn conjunction with seastar::app_template, initialization and cleanup look like so:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <iostream>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    return app.run(argc, argv, [] {\n        return seastar::async() {\n            std::cout << \"Hello world\\n\";\n            my_class my_object;\n            // ... more code\n\n            // my_object will be destroyed here\n            return 0;\n    });\n}\n```\n\nIf one needs to invoke an asynchronous function during shutdown, one can make\nuse of the seastar::defer() function to schedule an arbitrary cleanup function:\n\n```cpp\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <iostream>\n\nint main(int argc, char** argv) {\n    seastar::app_template app;\n    return app.run(argc, argv, [] {\n        return seastar::async() {\n            std::cout << \"Hello world\\n\";\n            my_class my_object;\n            // ... more code\n\n            auto cleanup = defer([] () noexcept {\n                // .get() joins the asynchronous function back\n                // into the thread.\n                call_async_cleanup_function().get();\n            });\n\n            // `cleanup` will invoke its function here, if we reached\n            // its construction point\n\n            // my_object will be destroyed here\n            return 0;\n    });\n}\n```\n\nSee also seastar::deferred_close and seastar::deferred_stop for more ways to\ncontrol cleanup.\n"
  },
  {
    "path": "doc/websocket.md",
    "content": "# WebSocket protocol implementation\n\nSeastar includes an experimental implementation of a WebSocket server.\nRefs:\nhttps://datatracker.ietf.org/doc/html/rfc6455\nhttps://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers\n\n## Handlers\n\nA WebSocket server needs a user-defined handler in order to be functiomal. WebSocket specification defines a concept of subprotocols, and Seastar WebSocket server allows registering a single handler per subprotocol.\n\nEach subprotocol has a unique name and is expected to be sent by the connecting client during handshake,\nby sending a `Sec-Websocket-Protocol` header with the chosen value.\n\nAside from specifying the chosen subprotocol name for a handler, the developer is expected to provide a function\nwhich handles the incoming stream of data and returns responses into the output stream.\n\nHere's an example of how to register a simple echo protocol:\n\n```cpp\nusing namespace seastar;\nstatic experimental::websocket::server ws;\nws.register_handler(\"echo\", [] (input_stream<char>& in, output_stream<char>& out) -> future<> {\n    while (true) {\n        auto buf = co_await in.read();\n        if (buf.empty()) {\n            co_return;\n        }\n        co_await out.write(std::move(buf));\n        co_await out.flush();\n    }\n});\n```\n\nNote: the developers should assume that the input stream provides decoded and unmasked data - so the stream should be treated as if it was backed by a TCP socket. Similarly, responses should be sent to the output stream as is, and the WebSocket server implementation will handle its proper serialization, masking and so on.\n\n## Error handling\n\nRegistered WebSocket handlers can throw arbitrary exceptions during their operation. Currently, exceptions that aren't explicitly handled within the handler will cause the established WebSocket connection to be terminated, and a proper error message will be logged.\n\n## Secure WebSocket (wss://)\n\nImplementation of Secure WebSocket standard, based on HTTPS is currently work in advanced progress. Once reviewed and merged, this section will contain documentation for it.\nRef: https://github.com/scylladb/seastar/pull/1044\n\n"
  },
  {
    "path": "docker/dev/Dockerfile",
    "content": "# syntax=docker/dockerfile:1\n\nFROM ubuntu:plucky\n\nRUN --mount=type=bind,source=./install-dependencies.sh,target=/install-dependencies.sh \\\n    apt-get update && apt-get install -y \\\n    curl \\\n    gnupg \\\n    build-essential \\\n    clang-20 \\\n    clang-19 \\\n    clang-tools-20 \\\n    gcc-15 \\\n    g++-15 \\\n    gcc-14 \\\n    g++-14 \\\n    pandoc \\\n    && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-15 15 \\\n    && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-15 15 \\\n    && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 14 \\\n    && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-14 14 \\\n    && update-alternatives --install /usr/bin/clang clang /usr/bin/clang-20 20 \\\n    && update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-20 20 \\\n    && update-alternatives --install /usr/bin/clang clang /usr/bin/clang-19 19 \\\n    && update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-19 19 \\\n    && bash ./install-dependencies.sh \\\n    && apt-get clean \\\n    && rm -rf /var/lib/apt/lists/*\n\nCMD [\"/bin/bash\"]\n"
  },
  {
    "path": "include/seastar/core/abort_on_ebadf.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n\nnamespace seastar {\n\n\n/// Determines whether seastar should throw or abort when operation made by\n/// seastar fails because the target file descriptor is not valid. This is\n/// detected when underlying system calls return EBADF or ENOTSOCK.\n/// The default behavior is to throw std::system_error.\nvoid set_abort_on_ebadf(bool do_abort);\n\n/// Queries the current setting for seastar's behavior on invalid file descriptor access.\n/// See set_abort_on_ebadf().\nbool is_abort_on_ebadf_enabled();\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/abort_on_expiry.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB.\n */\n\n#pragma once\n\n#include <seastar/core/abort_source.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/timed_out_error.hh>\n\nnamespace seastar {\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// Facility to tie a timeout with an abort source\n/// Can be used to make abortable fibers also support timeouts\ntemplate<typename Clock = lowres_clock>\nclass abort_on_expiry {\n    timer<Clock> _tr;\n    seastar::abort_source _as;\npublic:\n    using clock = Clock;\n    using time_point = typename Clock::time_point;\n    /// Creates a timer and an abort source associated with it\n    /// When the timer reaches timeout point it triggers an\n    /// abort automatically\n    abort_on_expiry(time_point timeout) : _tr([this] {\n        _as.request_abort_ex(timed_out_error{});\n    }) {\n        _tr.arm(timeout);\n    }\n    abort_on_expiry(abort_on_expiry&&) = delete;\n\n    /// \\returns abort source associated with the timeout\n    seastar::abort_source& abort_source() {\n        return _as;\n    }\n};\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/abort_source.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB.\n */\n\n#pragma once\n\n#include <seastar/util/assert.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/optimized_optional.hh>\n#include <seastar/util/std-compat.hh>\n\n#include <boost/intrusive/list.hpp>\n#include <exception>\n#include <optional>\n#include <type_traits>\n#include <utility>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// Exception thrown when an \\ref abort_source object has been\n/// notified by the \\ref abort_source::request_abort() method.\nclass abort_requested_exception : public std::exception {\npublic:\n    virtual const char* what() const noexcept override {\n        return \"abort requested\";\n    }\n};\n\n/// Facility to communicate a cancellation request to a fiber.\n/// Callbacks can be registered with the \\c abort_source, which are called\n/// atomically with a call to request_abort().\nclass abort_source {\n    using subscription_callback_type = noncopyable_function<void (const std::optional<std::exception_ptr>&) noexcept>;\n    using naive_subscription_callback_type = noncopyable_function<void() noexcept>;\n\npublic:\n    /// Represents a handle to the callback registered by a given fiber. Ending the\n    /// lifetime of the \\c subscription will unregister the callback, if it hasn't\n    /// been invoked yet.\n    class subscription : public bi::list_base_hook<bi::link_mode<bi::auto_unlink>> {\n        friend class abort_source;\n\n        subscription_callback_type _target;\n        bool _aborted = false;\n\n        explicit subscription(abort_source& as, subscription_callback_type target)\n                : _target(std::move(target)) {\n          if (!as.abort_requested()) {\n            as._subscriptions.push_back(*this);\n          }\n        }\n\n        struct naive_cb_tag {}; // to disambiguate constructors\n        explicit subscription(naive_cb_tag, abort_source& as, naive_subscription_callback_type naive_cb)\n                : _target([cb = std::move(naive_cb)] (const std::optional<std::exception_ptr>&) noexcept { cb(); }) {\n          if (!as.abort_requested()) {\n            as._subscriptions.push_back(*this);\n          }\n        }\n\n    public:\n        /// Call the subscribed callback (at most once).\n        /// This method is called by the \\ref abort_source on all listed \\ref subscription objects\n        /// when \\ref request_abort() is called.\n        /// It may be called indepdently by the user at any time, causing the \\ref subscription\n        /// to be unlinked from the \\ref abort_source subscriptions list.\n        void on_abort(const std::optional<std::exception_ptr>& ex) noexcept {\n            unlink();\n            if (!std::exchange(_aborted, true)) {\n                _target(ex);\n            }\n        }\n\n    public:\n        subscription() = default;\n\n        subscription(subscription&& other) noexcept(std::is_nothrow_move_constructible_v<subscription_callback_type>)\n                : _target(std::move(other._target))\n                , _aborted(std::exchange(other._aborted, true))\n        {\n            subscription_list_type::node_algorithms::swap_nodes(other.this_ptr(), this_ptr());\n        }\n\n        subscription& operator=(subscription&& other) noexcept(std::is_nothrow_move_assignable_v<subscription_callback_type>) {\n            if (this != &other) {\n                _target = std::move(other._target);\n                _aborted = std::exchange(other._aborted, true);\n                unlink();\n                subscription_list_type::node_algorithms::swap_nodes(other.this_ptr(), this_ptr());\n            }\n            return *this;\n        }\n\n        explicit operator bool() const noexcept {\n            return is_linked();\n        }\n    };\n\nprivate:\n    using subscription_list_type = bi::list<subscription, bi::constant_time_size<false>>;\n    subscription_list_type _subscriptions;\n    std::exception_ptr _ex;\n\n    void do_request_abort(std::optional<std::exception_ptr> ex) noexcept {\n        if (_ex) {\n            return;\n        }\n        _ex = ex.value_or(get_default_exception());\n        SEASTAR_ASSERT(_ex);\n        auto subs = std::move(_subscriptions);\n        while (!subs.empty()) {\n            subscription& s = subs.front();\n            s.on_abort(ex);\n        }\n    }\n\npublic:\n    abort_source() = default;\n    virtual ~abort_source() = default;\n\n    abort_source(abort_source&&) = default;\n    abort_source& operator=(abort_source&&) = default;\n\n    /// Delays the invocation of the callback \\c f until \\ref request_abort() is called.\n    /// \\returns \\ref optimized_optional containing a \\ref subscription that can be used to control\n    ///          the lifetime of the callback \\c f.\n    ///\n    /// Note: the returned \\ref optimized_optional evaluates to \\c true if and only if\n    /// \\ref abort_requested() is \\c false at the time \\ref subscribe is called, and therefore\n    /// the \\ref subscription is linked to the \\ref abort_source subscriptions list.\n    ///\n    /// Once \\ref request_abort() is called or the subscription's \\ref on_abort() method are called,\n    /// the callback \\c f is called (exactly once), and the \\ref subscription is unlinked from\n    /// the \\ref about_source, causing the \\ref optimized_optional to evaluate to \\c false.\n    ///\n    /// The returned \\ref optimized_optional would initially evaluate to \\c false if \\ref request_abort()\n    /// was already called. In this case, an unlinked \\ref subscription is returned as \\ref optimized_optional.\n    /// That \\ref subscription still allows the user to call \\ref on_abort() to invoke the callback \\c f.\n    template <typename Func>\n        requires (std::is_nothrow_invocable_r_v<void, Func, const std::optional<std::exception_ptr>&> ||\n                  std::is_nothrow_invocable_r_v<void, Func>)\n    [[nodiscard]]\n    optimized_optional<subscription> subscribe(Func&& f) {\n        if constexpr (std::is_invocable_v<Func, std::exception_ptr>) {\n            return { subscription(*this, std::forward<Func>(f)) };\n        } else {\n            return { subscription(subscription::naive_cb_tag{}, *this, std::forward<Func>(f)) };\n        }\n    }\n\n    /// Requests that the target operation be aborted. Current subscriptions\n    /// are invoked inline with this call with a disengaged optional<std::exception_ptr>,\n    /// and no new ones can be registered.\n    void request_abort() noexcept {\n        do_request_abort(std::nullopt);\n    }\n\n    /// Requests that the target operation be aborted with a given \\c exception_ptr.\n    /// Current subscriptions are invoked inline with this exception,\n    /// and no new ones can be registered.\n    void request_abort_ex(std::exception_ptr ex) noexcept {\n        do_request_abort(std::make_optional(std::move(ex)));\n    }\n\n    /// Requests that the target operation be aborted with a given \\c Exception object.\n    /// Current subscriptions are invoked inline with this exception, converted to std::exception_ptr,\n    /// and no new ones can be registered.\n    template <typename Exception>\n    void request_abort_ex(Exception&& e) noexcept {\n        do_request_abort(std::make_optional(std::make_exception_ptr(std::forward<Exception>(e))));\n    }\n\n    /// Returns whether an abort has been requested.\n    bool abort_requested() const noexcept {\n        return bool(_ex);\n    }\n\n\n    /// Throws a \\ref abort_requested_exception if cancellation has been requested.\n    void check() const {\n        if (abort_requested()) {\n            std::rethrow_exception(_ex);\n        }\n    }\n\n    /// Returns an exception with which an abort was requested.\n    const std::exception_ptr& abort_requested_exception_ptr() const noexcept {\n        return _ex;\n    }\n\n    /// Returns the default exception type (\\ref abort_requested_exception) for this abort source.\n    /// Overridable by derived classes.\n    virtual std::exception_ptr get_default_exception() const noexcept {\n        return make_exception_ptr(abort_requested_exception());\n    }\n};\n\n/// @}\n\n\n}\n\n#if FMT_VERSION < 100000\n// fmt v10 introduced formatter for std::exception\ntemplate <>\nstruct fmt::formatter<seastar::abort_requested_exception> : fmt::formatter<string_view> {\n    auto format(const seastar::abort_requested_exception& e, fmt::format_context& ctx) const {\n        return fmt::format_to(ctx.out(), \"{}\", e.what());\n    }\n};\n#endif\n"
  },
  {
    "path": "include/seastar/core/abortable_fifo.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/abort_source.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/util/assert.hh>\n\n#include <memory>\n#include <optional>\n#include <type_traits>\n\nnamespace seastar {\n\nnamespace internal {\n\n// Test if aborter has call operator accepting optional exception_ptr\ntemplate <typename Aborter, typename T>\nconcept aborter_ex = std::is_nothrow_invocable_r_v<void, Aborter, T&, const std::optional<std::exception_ptr>&>;\n\ntemplate <typename Aborter, typename T>\nconcept aborter = std::is_nothrow_invocable_r_v<void, Aborter, T&> || aborter_ex<Aborter, T>;\n\n// This class satisfies 'aborter' concept and is used by default\ntemplate<typename... T>\nstruct noop_aborter {\n    void operator()(T...) noexcept {};\n};\n\n\n/// Container for elements with support for cancellation of entries.\n///\n/// OnAbort is a functor which will be called with a reference to T right before it expires.\n/// T is removed and destroyed from the container immediately after OnAbort returns.\n/// OnAbort callback must not modify the container, it can only modify its argument.\n///\n/// The container can only be moved before any elements are pushed.\n///\ntemplate <typename T, typename OnAbort = noop_aborter<T>>\nrequires aborter<OnAbort, T>\nclass abortable_fifo {\nprivate:\n    struct entry {\n        std::optional<T> payload; // disengaged means that it's expired\n        optimized_optional<abort_source::subscription> sub;\n        entry(T&& payload_) : payload(std::move(payload_)) {}\n        entry(const T& payload_) : payload(payload_) {}\n        entry(T payload_, abortable_fifo& ef, abort_source& as)\n                : payload(std::move(payload_))\n                , sub(as.subscribe([this, &ef] (const std::optional<std::exception_ptr>& ex_opt) noexcept {\n                    if constexpr (aborter_ex<OnAbort, T>) {\n                        ef._on_abort(*payload, ex_opt);\n                    } else {\n                        ef._on_abort(*payload);\n                    }\n                    payload = std::nullopt;\n                    --ef._size;\n                    ef.drop_expired_front();\n                })) {}\n        entry() = default;\n        entry(entry&& x) = delete;\n        entry(const entry& x) = delete;\n    };\n\n    // If engaged, represents the first element.\n    // This is to avoid large allocations done by chunked_fifo for single-element cases.\n    // abortable_fifo is used to implement wait lists in synchronization primitives\n    // and in some uses it's common to have at most one waiter.\n    std::unique_ptr<entry> _front;\n\n    // There is an invariant that the front element is never expired.\n    chunked_fifo<entry> _list;\n    OnAbort _on_abort;\n    size_t _size = 0;\n\n    // Ensures that front() is not expired by dropping expired elements from the front.\n    void drop_expired_front() noexcept {\n        while (!_list.empty() && !_list.front().payload) {\n            _list.pop_front();\n        }\n        if (_front && !_front->payload) {\n            _front.reset();\n        }\n    }\npublic:\n    abortable_fifo() noexcept = default;\n    abortable_fifo(OnAbort on_abort) noexcept(std::is_nothrow_move_constructible_v<OnAbort>) : _on_abort(std::move(on_abort)) {}\n\n    abortable_fifo(abortable_fifo&& o) noexcept\n            : abortable_fifo(std::move(o._on_abort)) {\n        // entry objects hold a reference to this so non-empty containers cannot be moved.\n        SEASTAR_ASSERT(o._size == 0);\n    }\n\n    abortable_fifo& operator=(abortable_fifo&& o) noexcept {\n        if (this != &o) {\n            this->~abortable_fifo();\n            new (this) abortable_fifo(std::move(o));\n        }\n        return *this;\n    }\n\n    /// Checks if container contains any elements\n    ///\n    /// \\note Inside OnAbort callback, the expired element is still contained.\n    ///\n    /// \\return true if and only if there are any elements contained.\n    bool empty() const noexcept {\n        return _size == 0;\n    }\n\n    /// Equivalent to !empty()\n    explicit operator bool() const noexcept {\n        return !empty();\n    }\n\n    /// Returns a reference to the element in the front.\n    /// Valid only when !empty().\n    T& front() noexcept {\n        if (_front) {\n            return *_front->payload;\n        }\n        return *_list.front().payload;\n    }\n\n    /// Returns a reference to the element in the front.\n    /// Valid only when !empty().\n    const T& front() const noexcept {\n        if (_front) {\n            return *_front->payload;\n        }\n        return *_list.front().payload;\n    }\n\n    /// Returns the number of elements contained.\n    ///\n    /// \\note Expired elements are not contained. Expiring element is still contained when OnAbort is called.\n    size_t size() const noexcept {\n        return _size;\n    }\n\n    /// Reserves storage in the container for at least 'size' elements.\n    /// Note that expired elements may also take space when they are not in the front of the queue.\n    ///\n    /// Doesn't give any guarantees about exception safety of subsequent push_back().\n    void reserve(size_t size) {\n        return _list.reserve(size);\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will never expire.\n    void push_back(const T& payload) {\n        if (_size == 0) {\n            _front = std::make_unique<entry>(payload);\n        } else {\n            _list.emplace_back(payload);\n        }\n        ++_size;\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will never expire.\n    void push_back(T&& payload) {\n        if (_size == 0) {\n            _front = std::make_unique<entry>(std::move(payload));\n        } else {\n            _list.emplace_back(std::move(payload));\n        }\n        ++_size;\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will expire when abort source is triggered.\n    void push_back(T&& payload, abort_source& as) {\n        if (as.abort_requested()) {\n            if constexpr (aborter_ex<OnAbort, T>) {\n                _on_abort(payload, std::nullopt);\n            } else {\n                _on_abort(payload);\n            }\n            return;\n        }\n        if (_size == 0) {\n            _front = std::make_unique<entry>(std::move(payload), *this, as);\n        } else {\n            _list.emplace_back(std::move(payload), *this, as);\n        }\n        ++_size;\n    }\n\n    /// Create element in place at the back of the queue.\n    template<typename... U>\n    T& emplace_back(U&&... args) {\n        if (_size == 0) {\n            _front = std::make_unique<entry>();\n            _front->payload.emplace(std::forward<U>(args)...);\n            _size = 1;\n            return *_front->payload;\n        } else {\n            _list.emplace_back();\n            _list.back().payload.emplace(std::forward<U>(args)...);\n            ++_size;\n            return *_list.back().payload;\n        }\n    }\n\n    /// Make an element at the back of the queue abortable\n    /// The element will expire when abort source is triggered.\n    /// Valid only when !empty().\n    /// Cannot be called on an element that si already associated\n    /// with an abort source.\n    void make_back_abortable(abort_source& as) {\n        entry* e = _front.get();\n        if (!_list.empty()) {\n            e = &_list.back();\n        }\n        SEASTAR_ASSERT(!e->sub);\n        auto aborter = [this, e] (const std::optional<std::exception_ptr>& ex_opt) noexcept {\n            if constexpr (aborter_ex<OnAbort, T>) {\n                _on_abort(*e->payload, ex_opt);\n            } else {\n                _on_abort(*e->payload);\n            }\n            e->payload = std::nullopt;\n            --_size;\n            drop_expired_front();\n        };\n        if (as.abort_requested()) {\n            aborter(as.abort_requested_exception_ptr());\n            return;\n        }\n        e->sub = as.subscribe(std::move(aborter));\n    }\n\n    /// Removes the element at the front.\n    /// Can be called only if !empty().\n    void pop_front() noexcept {\n        if (_front) {\n            _front.reset();\n        } else {\n            _list.pop_front();\n        }\n        --_size;\n        drop_expired_front();\n    }\n};\n\n}\n}\n\n"
  },
  {
    "path": "include/seastar/core/alien.hh",
    "content": "// -*- mode:C++; tab-width:4; c-basic-offset:4; indent-tabs-mode:nil -*-\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 Red Hat\n */\n\n#pragma once\n\n#include <atomic>\n#include <concepts>\n#include <future>\n#include <memory>\n#include <type_traits>\n#include <boost/lockfree/queue.hpp>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/cacheline.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/metrics_registration.hh>\n\n/// \\file\n\nnamespace seastar {\n\nclass reactor;\n\n/// \\brief Integration with non-seastar applications.\nnamespace alien {\n\nclass message_queue {\n    static constexpr size_t batch_size = 128;\n    static constexpr size_t prefetch_cnt = 2;\n    struct work_item;\n    struct lf_queue_remote {\n        reactor* remote;\n    };\n    using lf_queue_base = boost::lockfree::queue<work_item*>;\n    // use inheritence to control placement order\n    struct lf_queue : lf_queue_remote, lf_queue_base {\n        lf_queue(reactor* remote)\n            : lf_queue_remote{remote}, lf_queue_base{batch_size} {}\n        void maybe_wakeup();\n    } _pending;\n    struct alignas(seastar::cache_line_size) {\n        std::atomic<size_t> value{0};\n    } _sent;\n    // keep this between two structures with statistics\n    // this makes sure that they have at least one cache line\n    // between them, so hw prefetcher will not accidentally prefetch\n    // cache line used by another cpu.\n    metrics::metric_groups _metrics;\n    struct alignas(seastar::cache_line_size) {\n        size_t _received = 0;\n        size_t _last_rcv_batch = 0;\n    };\n    struct work_item {\n        virtual ~work_item() = default;\n        virtual void process() = 0;\n    };\n    template <typename  Func>\n    struct async_work_item : work_item {\n        Func _func;\n        async_work_item(Func&& func) : _func(std::move(func)) {}\n        void process() override {\n            _func();\n        }\n    };\n    template<typename Func>\n    size_t process_queue(lf_queue& q, Func process);\n    void submit_item(std::unique_ptr<work_item> wi);\npublic:\n    message_queue(reactor *to);\n    void start();\n    void stop();\n    template <typename Func>\n    void submit(Func&& func) {\n        auto wi = std::make_unique<async_work_item<Func>>(std::forward<Func>(func));\n        submit_item(std::move(wi));\n    }\n    size_t process_incoming();\n    bool pure_poll_rx() const;\n};\n\nnamespace internal {\n\nstruct qs_deleter {\n    unsigned count;\n    qs_deleter(unsigned n = 0) : count(n) {}\n    void operator()(message_queue* qs) const;\n};\n\n}\n\n/// Represents the Seastar system from alien's point of view. In a normal\n/// system, there is just one instance, but for in-process clustering testing\n/// there may be more than one. Function such as run_on() direct messages to\n/// and (instance, shard) tuple.\nclass instance {\n    using qs = std::unique_ptr<message_queue[], internal::qs_deleter>;\npublic:\n    static qs create_qs(const std::vector<reactor*>& reactors);\n    qs _qs;\n    bool poll_queues();\n    bool pure_poll_queues();\n};\n\nnamespace internal {\n\nextern instance* default_instance;\n\n}\n\n/// Runs a function on a remote shard from an alien thread where engine() is not available.\n///\n/// \\param instance designates the Seastar instance to process the message\n/// \\param shard designates the shard to run the function on\n/// \\param func a callable to run on shard \\c t.  If \\c func is a temporary object,\n///          its lifetime will be extended by moving it.  If \\c func is a reference,\n///          the caller must guarantee that it will survive the call.\n/// \\note the func must not throw and should return \\c void. as we cannot identify the\n///          alien thread, hence we are not able to post the fulfilled promise to the\n///          message queue managed by the shard executing the alien thread which is\n///          interested to the return value. Please use \\c submit_to() instead, if\n///          \\c func throws.\ntemplate <typename Func>\nrequires std::is_nothrow_invocable_r_v<void, Func>\nvoid run_on(instance& instance, unsigned shard, Func func) {\n    instance._qs[shard].submit(std::move(func));\n}\n\nnamespace internal {\ntemplate<typename Func>\nusing return_value_t = typename futurize<std::invoke_result_t<Func>>::value_type;\n\ntemplate<typename Func,\n         bool = std::is_empty_v<return_value_t<Func>>>\nstruct return_type_of {\n    using type = void;\n    static void set(std::promise<void>& p, return_value_t<Func>&&) {\n        p.set_value();\n    }\n};\ntemplate<typename Func>\nstruct return_type_of<Func, false> {\n    using return_tuple_t = typename futurize<std::invoke_result_t<Func>>::tuple_type;\n    using type = std::tuple_element_t<0, return_tuple_t>;\n    static void set(std::promise<type>& p, return_value_t<Func>&& t) {\n        p.set_value(std::move(t));\n    }\n};\ntemplate <typename Func> using return_type_t = typename return_type_of<Func>::type;\n}\n\n/// Runs a function on a remote shard from an alien thread where engine() is not available.\n///\n/// \\param instance designates the Seastar instance to process the message\n/// \\param shard designates the shard to run the function on\n/// \\param func a callable to run on \\c shard.  If \\c func is a temporary object,\n///          its lifetime will be extended by moving it.  If \\c func is a reference,\n///          the caller must guarantee that it will survive the call.\n/// \\return whatever \\c func returns, as a \\c std::future<>\n/// \\note the caller must keep the returned future alive until \\c func returns\ntemplate<std::invocable Func, typename T = internal::return_type_t<Func>>\nstd::future<T> submit_to(instance& instance, unsigned shard, Func func) {\n    std::promise<T> pr;\n    auto fut = pr.get_future();\n    run_on(instance, shard, [pr = std::move(pr), func = std::move(func)] () mutable noexcept {\n        // std::future returned via std::promise above.\n        (void)func().then_wrapped([pr = std::move(pr)] (auto&& result) mutable {\n            try {\n                internal::return_type_of<Func>::set(pr, result.get());\n            } catch (...) {\n                pr.set_exception(std::current_exception());\n            }\n        });\n    });\n    return fut;\n}\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/align.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cstdint>\n#include <cstdlib>\n\nnamespace seastar {\n\n\ntemplate <typename T>\ninline constexpr\nT align_up(T v, T align) {\n    return (v + align - 1) & ~(align - 1);\n}\n\ntemplate <typename T>\ninline constexpr\nT* align_up(T* v, size_t align) {\n    static_assert(sizeof(T) == 1, \"align byte pointers only\");\n    return reinterpret_cast<T*>(align_up(reinterpret_cast<uintptr_t>(v), align));\n}\n\ntemplate <typename T>\ninline constexpr\nT align_down(T v, T align) {\n    return v & ~(align - 1);\n}\n\ntemplate <typename T>\ninline constexpr\nT* align_down(T* v, size_t align) {\n    static_assert(sizeof(T) == 1, \"align byte pointers only\");\n    return reinterpret_cast<T*>(align_down(reinterpret_cast<uintptr_t>(v), align));\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/aligned_buffer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n#pragma once\n#include <stdlib.h>\n#include <memory>\n\nnamespace seastar {\n\nnamespace internal {\nvoid* allocate_aligned_buffer_impl(size_t size, size_t align);\n}\n\n\nstruct free_deleter {\n    void operator()(void* p) { ::free(p); }\n};\n\ntemplate <typename CharType>\ninline\nstd::unique_ptr<CharType[], free_deleter> allocate_aligned_buffer(size_t size, size_t align) {\n    static_assert(sizeof(CharType) == 1, \"must allocate byte type\");\n    void* ret = internal::allocate_aligned_buffer_impl(size, align);\n    return std::unique_ptr<CharType[], free_deleter>(reinterpret_cast<CharType *>(ret));\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/app-template.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n#pragma once\n\n#include <boost/program_options.hpp>\n#include <functional>\n#include <chrono>\n#include <seastar/core/future.hh>\n#include <seastar/core/smp_options.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/program-options.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/scollectd.hh>\n#include <seastar/util/log-cli.hh>\n\nnamespace seastar {\n\nclass smp;\n\nnamespace alien {\n\nclass instance;\n\n}\n\nclass app_template {\npublic:\n    struct config {\n        /// The name of the application.\n        ///\n        /// Will be used in the --help output to distinguish command line args\n        /// registered by the application, as opposed to those registered by\n        /// seastar and its subsystems.\n        sstring name = \"App\";\n        /// The description of the application.\n        ///\n        /// Will be printed on the top of the --help output. Lines should be\n        /// hard-wrapped for 80 chars.\n        sstring description = \"\";\n        std::chrono::duration<double> default_task_quota = std::chrono::microseconds(500);\n        /// \\brief Handle SIGINT/SIGTERM by calling reactor::stop()\n        ///\n        /// When true, Seastar will set up signal handlers for SIGINT/SIGTERM that call\n        /// reactor::stop(). The reactor will then execute callbacks installed by\n        /// reactor::at_exit().\n        ///\n        /// When false, Seastar will not set up signal handlers for SIGINT/SIGTERM\n        /// automatically. The default behavior (terminate the program) will be kept.\n        /// You can adjust the behavior of SIGINT/SIGTERM by installing signal handlers\n        /// via seastar::handle_signal(signo, handler, once); instead.\n        bool auto_handle_sigint_sigterm = true;\n        /// Specifies the default value for linux-aio I/O control blocks. This translates\n        /// to the maximum number of sockets the shard can handle.\n        unsigned max_networking_aio_io_control_blocks = 10000;\n        /// The amount of memory that should not be used by the seastar allocator,\n        /// additional to the amount of memory already reserved for the OS.\n        /// This can be used when the application allocates some of its memory using the\n        /// seastar allocator, and some using the system allocator, in particular when it\n        /// uses the mmap system call with MAP_ANONYMOUS which is not overridden in seastar.\n        size_t reserve_additional_memory_per_shard = 0;\n        config() {}\n    };\n\n    /// Seastar configuration options\n    struct seastar_options : public program_options::option_group {\n        /// The name of the application.\n        ///\n        /// Will be used in the --help output to distinguish command line args\n        /// registered by the application, as opposed to those registered by\n        /// seastar and its subsystems.\n        sstring name = \"App\";\n        /// The description of the application.\n        ///\n        /// Will be printed on the top of the --help output. Lines should be\n        /// hard-wrapped for 80 chars.\n        sstring description = \"\";\n        /// \\brief Handle SIGINT/SIGTERM by calling reactor::stop()\n        ///\n        /// When true, Seastar will set up signal handlers for SIGINT/SIGTERM that call\n        /// reactor::stop(). The reactor will then execute callbacks installed by\n        /// reactor::at_exit().\n        ///\n        /// When false, Seastar will not set up signal handlers for SIGINT/SIGTERM\n        /// automatically. The default behavior (terminate the program) will be kept.\n        /// You can adjust the behavior of SIGINT/SIGTERM by installing signal handlers\n        /// via seastar::handle_signal(signo, handler, once); instead.\n        bool auto_handle_sigint_sigterm = true;\n        /// Configuration options for the reactor.\n        reactor_options reactor_opts;\n        /// Configuration for the metrics sub-system.\n        metrics::options metrics_opts;\n        /// Configuration options for the smp aspect of seastar.\n        smp_options smp_opts;\n        /// Configuration for the scollectd sub-system.\n        scollectd::options scollectd_opts;\n        /// Configuration for the logging sub-system.\n        log_cli::options log_opts;\n\n        seastar_options();\n    };\n\n    using configuration_reader = std::function<void (boost::program_options::variables_map&)>;\nprivate:\n    // unique_ptr to avoid pulling in alien.hh.\n    std::unique_ptr<alien::instance> _alien;\n    // reactor destruction is asynchronous, so we must let the last reactor\n    // destroy the smp instance\n    std::shared_ptr<smp> _smp;\n    seastar_options _opts;\n    boost::program_options::options_description _app_opts;\n    boost::program_options::options_description _seastar_opts;\n    boost::program_options::options_description _opts_conf_file;\n    boost::program_options::positional_options_description _pos_opts;\n    std::optional<boost::program_options::variables_map> _configuration;\n    configuration_reader _conf_reader;\n\n    configuration_reader get_default_configuration_reader();\npublic:\n    struct positional_option {\n        const char* name;\n        const boost::program_options::value_semantic* value_semantic;\n        const char* help;\n        int max_count;\n    };\npublic:\n    explicit app_template(seastar_options opts);\n    explicit app_template(config cfg = config());\n    ~app_template();\n\n    const seastar_options& options() const;\n\n    boost::program_options::options_description& get_options_description();\n    boost::program_options::options_description& get_conf_file_options_description();\n    boost::program_options::options_description_easy_init add_options();\n    void add_positional_options(std::initializer_list<positional_option> options);\n    boost::program_options::variables_map& configuration();\n    int run_deprecated(int ac, char ** av, std::function<void ()>&& func) noexcept;\n\n    void set_configuration_reader(configuration_reader conf_reader);\n\n    /// Obtains an alien::instance object that can be used to send messages\n    /// to Seastar shards from non-Seastar threads.\n    alien::instance& alien() { return *_alien; }\n\n    // Runs given function and terminates the application when the future it\n    // returns resolves. The value with which the future resolves will be\n    // returned by this function.\n    int run(int ac, char ** av, std::function<future<int> ()>&& func) noexcept;\n\n    // Like run() which takes std::function<future<int>()>, but returns\n    // with exit code 0 when the future returned by func resolves\n    // successfully.\n    int run(int ac, char ** av, std::function<future<> ()>&& func) noexcept;\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/bitops.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <concepts>\n#include <limits>\n\nnamespace seastar {\n\n\ninline\nconstexpr unsigned count_leading_zeros(unsigned x) {\n    return __builtin_clz(x);\n}\n\ninline\nconstexpr unsigned count_leading_zeros(unsigned long x) {\n    return __builtin_clzl(x);\n}\n\ninline\nconstexpr unsigned count_leading_zeros(unsigned long long x) {\n    return __builtin_clzll(x);\n}\n\ninline\nconstexpr unsigned count_trailing_zeros(unsigned x) {\n    return __builtin_ctz(x);\n}\n\ninline\nconstexpr unsigned count_trailing_zeros(unsigned long x) {\n    return __builtin_ctzl(x);\n}\n\ninline\nconstexpr unsigned count_trailing_zeros(unsigned long long x) {\n    return __builtin_ctzll(x);\n}\n\ntemplate<std::integral T>\ninline constexpr unsigned log2ceil(T n) {\n    if (n == 1) {\n        return 0;\n    }\n    return std::numeric_limits<T>::digits - count_leading_zeros(n - 1);\n}\n\ntemplate<std::integral T>\ninline constexpr unsigned log2floor(T n) {\n    return std::numeric_limits<T>::digits - count_leading_zeros(n) - 1;\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/bitset-iter.hh",
    "content": "/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n/*\n * Imported from OSv:\n *\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n * This work is open source software, licensed under the terms of the\n * BSD license as described in the LICENSE file in the top-level directory.\n */\n\n#pragma once\n\n#include <bitset>\n#include <limits>\n\nnamespace seastar {\n\nnamespace bitsets {\n\nstatic constexpr int ulong_bits = std::numeric_limits<unsigned long>::digits;\n\n/**\n * Returns the number of leading zeros in value's binary representation.\n *\n * If value == 0 the result is undefied. If T is signed and value is negative\n * the result is undefined.\n *\n * The highest value that can be returned is std::numeric_limits<T>::digits - 1,\n * which is returned when value == 1.\n */\ntemplate<typename T>\nsize_t count_leading_zeros(T value) noexcept;\n\n/**\n * Returns the number of trailing zeros in value's binary representation.\n *\n * If value == 0 the result is undefied. If T is signed and value is negative\n * the result is undefined.\n *\n * The highest value that can be returned is std::numeric_limits<T>::digits - 1.\n */\ntemplate<typename T>\nsize_t count_trailing_zeros(T value) noexcept;\n\ntemplate<>\ninline size_t count_leading_zeros<unsigned long>(unsigned long value) noexcept\n{\n    return __builtin_clzl(value);\n}\n\ntemplate<>\ninline size_t count_leading_zeros<long>(long value) noexcept\n{\n    return __builtin_clzl((unsigned long)value) - 1;\n}\n\ntemplate<>\ninline size_t count_leading_zeros<unsigned long long>(unsigned long long value) noexcept\n{\n    return __builtin_clzll(value);\n}\n\ntemplate<>\ninline size_t count_leading_zeros<long long>(long long value) noexcept\n{\n    return __builtin_clzll((unsigned long long)value) - 1;\n}\n\ntemplate<>\ninline\nsize_t count_trailing_zeros<unsigned long>(unsigned long value) noexcept\n{\n    return __builtin_ctzl(value);\n}\n\ntemplate<>\ninline\nsize_t count_trailing_zeros<long>(long value) noexcept\n{\n    return __builtin_ctzl((unsigned long)value);\n}\n\ntemplate<>\ninline\nsize_t count_trailing_zeros<unsigned long long>(unsigned long long value) noexcept\n{\n    return __builtin_ctzll(value);\n}\n\ntemplate<>\ninline\nsize_t count_trailing_zeros<long long>(long long value) noexcept\n{\n    return __builtin_ctzll((unsigned long long)value);\n}\n\n/**\n * Returns the index of the first set bit.\n * Result is undefined if bitset.any() == false.\n */\ntemplate<size_t N>\ninline size_t get_first_set(const std::bitset<N>& bitset) noexcept\n{\n    static_assert(N <= ulong_bits, \"bitset too large\");\n    return count_trailing_zeros(bitset.to_ulong());\n}\n\n/**\n * Returns the index of the last set bit in the bitset.\n * Result is undefined if bitset.any() == false.\n */\ntemplate<size_t N>\ninline size_t get_last_set(const std::bitset<N>& bitset) noexcept\n{\n    static_assert(N <= ulong_bits, \"bitset too large\");\n    return ulong_bits - 1 - count_leading_zeros(bitset.to_ulong());\n}\n\n\ntemplate<size_t N>\nclass set_iterator\n{\nprivate:\n    void advance() noexcept\n    {\n        if (_bitset.none()) {\n            _index = -1;\n        } else {\n            auto shift = get_first_set(_bitset) + 1;\n            _index += shift;\n            _bitset >>= shift;\n        }\n    }\npublic:\n    using iterator_category = std::input_iterator_tag;\n    using iterator_concept = std::input_iterator_tag;\n    using value_type = int;\n    using difference_type = std::ptrdiff_t;\n    using pointer = int*;\n    using reference = int&;\n\n    set_iterator() noexcept\n        : _bitset()\n        , _index(0) {\n    }\n\n    set_iterator(std::bitset<N> bitset, int offset = 0) noexcept\n        : _bitset(bitset)\n        , _index(offset - 1)\n    {\n        static_assert(N <= ulong_bits, \"This implementation is inefficient for large bitsets\");\n        _bitset >>= offset;\n        advance();\n    }\n\n    set_iterator& operator++() noexcept\n    {\n        advance();\n        return *this;\n    }\n\n    set_iterator operator++(int) noexcept\n    {\n        auto ret = *this;\n        advance();\n        return ret;\n    }\n\n    int operator*() const noexcept\n    {\n        return _index;\n    }\n\n    bool operator==(const set_iterator& other) const noexcept\n    {\n        return _index == other._index;\n    }\n\n    bool operator!=(const set_iterator& other) const noexcept\n    {\n        return !(*this == other);\n    }\nprivate:\n    std::bitset<N> _bitset;\n    int _index;\n};\n\ntemplate<size_t N>\nclass set_range\n{\npublic:\n    using iterator = set_iterator<N>;\n    using value_type = int;\n\n    constexpr set_range(std::bitset<N> bitset, int offset = 0) noexcept\n        : _bitset(bitset)\n        , _offset(offset)\n    {\n    }\n\n    iterator begin() const noexcept { return iterator(_bitset, _offset); }\n    iterator end() const noexcept { return iterator(0); }\nprivate:\n    std::bitset<N> _bitset;\n    int _offset;\n};\n\ntemplate<size_t N>\ninline set_range<N> for_each_set(std::bitset<N> bitset, int offset = 0) noexcept\n{\n    return set_range<N>(bitset, offset);\n}\n\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/byteorder.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <algorithm>\n#include <boost/endian/conversion.hpp>\n#include <seastar/core/unaligned.hh>\n\nnamespace seastar {\n\n\ntemplate <typename T>\ninline T cpu_to_le(T x) noexcept {\n  return boost::endian::native_to_little(x);\n}\ntemplate <typename T>\ninline T le_to_cpu(T x) noexcept {\n  return boost::endian::little_to_native(x);\n}\n\ntemplate <typename T>\ninline T cpu_to_be(T x) noexcept {\n  return boost::endian::native_to_big(x);\n}\ntemplate <typename T>\ninline T be_to_cpu(T x) noexcept {\n  return boost::endian::big_to_native(x);\n}\n\ntemplate <typename T>\ninline T cpu_to_le(const unaligned<T>& v) noexcept {\n    return cpu_to_le(T(v));\n}\n\ntemplate <typename T>\ninline T le_to_cpu(const unaligned<T>& v) noexcept {\n    return le_to_cpu(T(v));\n}\n\ntemplate <typename T>\ninline\nT\nread_le(const char* p) noexcept {\n    T datum;\n    std::copy_n(p, sizeof(T), reinterpret_cast<char*>(&datum));\n    return le_to_cpu(datum);\n}\n\ntemplate <typename T>\ninline\nvoid\nwrite_le(char* p, T datum) noexcept {\n    datum = cpu_to_le(datum);\n    std::copy_n(reinterpret_cast<const char*>(&datum), sizeof(T), p);\n}\n\ntemplate <typename T>\ninline\nT\nread_be(const char* p) noexcept {\n    T datum;\n    std::copy_n(p, sizeof(T), reinterpret_cast<char*>(&datum));\n    return be_to_cpu(datum);\n}\n\ntemplate <typename T>\ninline\nvoid\nwrite_be(char* p, T datum) noexcept {\n    datum = cpu_to_be(datum);\n    std::copy_n(reinterpret_cast<const char*>(&datum), sizeof(T), p);\n}\n\ntemplate <typename T>\ninline\nT\nconsume_be(const char*& p) noexcept {\n    auto ret = read_be<T>(p);\n    p += sizeof(T);\n    return ret;\n}\n\ntemplate <typename T>\ninline\nvoid\nproduce_be(char*& p, T datum) noexcept {\n    write_be<T>(p, datum);\n    p += sizeof(T);\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/cacheline.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 IBM.\n */\n\n#pragma once\n\n#include <cstddef>\n\nnamespace seastar {\n\n\n// Platform-dependent cache line size for alignment and padding purposes.\nconstexpr size_t cache_line_size =\n#if defined(__x86_64__) || defined(__i386__)\n    64;\n#elif defined(__s390x__) || defined(__zarch__)\n    256;\n#elif defined(__PPC64__)\n    128;\n#elif defined(__aarch64__)\n    128; // from Linux, may vary among different microarchitetures?\n#else\n#error \"cache_line_size not defined for this architecture\"\n#endif\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/checked_ptr.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n/// \\file\n/// \\brief Contains a seastar::checked_ptr class implementation.\n#include <exception>\n#include <memory>\n\n/// \\namespace seastar\nnamespace seastar {\n\n\n/// The exception thrown by a default_null_deref_action.\nclass checked_ptr_is_null_exception : public std::exception {};\n\n/// \\brief\n/// Default not engaged seastar::checked_ptr dereferencing action (functor).\n///\n/// Throws a seastar::checked_ptr_is_null_exception.\n///\nstruct default_null_deref_action {\n    /// \\throw seastar::checked_ptr_is_null_exception\n    void operator()() const {\n        throw checked_ptr_is_null_exception();\n    }\n};\n\n/// \\cond internal\n/// \\namespace seastar::internal\nnamespace internal {\n\n/// \\name seastar::checked_ptr::get() helpers\n/// Helper functions that simplify the seastar::checked_ptr::get() implementation.\n/// @{\n\n/// Invokes the get() method of a smart pointer object.\n/// \\param ptr A smart pointer object\n/// \\return A pointer to the underlying object\ntemplate <typename T>\nrequires requires (T ptr) {\n    ptr.get();\n}\ninline typename std::pointer_traits<std::remove_const_t<T>>::element_type* checked_ptr_do_get(T& ptr) {\n    return ptr.get();\n}\n\n/// Return a pointer itself for a naked pointer argument.\n/// \\param ptr A naked pointer object\n/// \\return An input naked pointer object\ntemplate <typename T>\ninline T* checked_ptr_do_get(T* ptr) noexcept {\n    return ptr;\n}\n/// @}\n}\n/// \\endcond\n\n/// \\class seastar::checked_ptr\n/// \\brief\n/// seastar::checked_ptr class is a wrapper class that may be used with any pointer type\n/// (smart like std::unique_ptr or raw pointers like int*).\n///\n/// The seastar::checked_ptr object will invoke the NullDerefAction functor if\n/// it is dereferenced when the underlying pointer is not engaged.\n///\n/// It may still be assigned, compared to other seastar::checked_ptr objects or\n/// moved without limitations.\n///\n/// The default NullDerefAction will throw a  seastar::default_null_deref_action exception.\n///\n/// \\tparam NullDerefAction a functor that is invoked when a user tries to dereference a not engaged pointer.\n///\ntemplate<typename Ptr, typename NullDerefAction = default_null_deref_action>\nrequires std::is_default_constructible_v<NullDerefAction> && requires (NullDerefAction action) {\n    NullDerefAction();\n}\nclass checked_ptr {\npublic:\n    /// Underlying element type\n    using element_type = typename std::pointer_traits<Ptr>::element_type;\n\n    /// Type of the pointer to the underlying element\n    using pointer = element_type*;\n\nprivate:\n    Ptr _ptr = nullptr;\n\nprivate:\n    /// Invokes a NullDerefAction functor if the underlying pointer is not engaged.\n    void check() const {\n        if (!_ptr) {\n            NullDerefAction()();\n        }\n    }\n\npublic:\n    checked_ptr() noexcept(noexcept(Ptr(nullptr))) = default;\n    checked_ptr(std::nullptr_t) noexcept(std::is_nothrow_default_constructible_v<checked_ptr<Ptr, NullDerefAction>>) : checked_ptr() {}\n    checked_ptr(Ptr&& ptr) noexcept(std::is_nothrow_move_constructible_v<Ptr>) : _ptr(std::move(ptr)) {}\n    checked_ptr(const Ptr& p) noexcept(std::is_nothrow_copy_constructible_v<Ptr>) : _ptr(p) {}\n\n    /// \\name Checked Methods\n    /// These methods start with invoking a NullDerefAction functor if the underlying pointer is not engaged.\n    /// @{\n\n    /// Invokes the get() method of the underlying smart pointer or returns the pointer itself for a raw pointer (const variant).\n    /// \\return The pointer to the underlying object\n    pointer get() const {\n        check();\n        return internal::checked_ptr_do_get(_ptr);\n    }\n\n    /// Gets a reference to the underlying pointer object.\n    /// \\return The underlying pointer object\n    const Ptr& operator->() const {\n        check();\n        return _ptr;\n    }\n\n    /// Gets a reference to the underlying pointer object (const variant).\n    /// \\return The underlying pointer object\n    Ptr& operator->() {\n        check();\n        return _ptr;\n    }\n\n    /// Gets the reference to the underlying object (const variant).\n    /// \\return The reference to the underlying object\n    const element_type& operator*() const {\n        check();\n        return *_ptr;\n    }\n\n    /// Gets the reference to the underlying object.\n    /// \\return The reference to the underlying object\n    element_type& operator*() {\n        check();\n        return *_ptr;\n    }\n    /// @}\n\n    /// \\name Unchecked methods\n    /// These methods may be invoked when the underlying pointer is not engaged.\n    /// @{\n\n    /// Checks if the underlying pointer is engaged.\n    /// \\return TRUE if the underlying pointer is engaged\n    explicit operator bool() const { return bool(_ptr); }\n\n    bool operator==(const checked_ptr& other) const { return _ptr == other._ptr; }\n    bool operator!=(const checked_ptr& other) const { return _ptr != other._ptr; }\n\n    /// Gets the hash value for the underlying pointer object.\n    /// \\return The hash value for the underlying pointer object\n    size_t hash() const {\n        return std::hash<Ptr>()(_ptr);\n    }\n    ///@}\n};\n\n}\n\nnamespace std {\n/// std::hash specialization for seastar::checked_ptr class\ntemplate<typename T>\nstruct hash<seastar::checked_ptr<T>> {\n    /// Get the hash value for the given seastar::checked_ptr object.\n    /// The hash will calculated using the seastar::checked_ptr::hash method.\n    /// \\param p object for hash value calculation\n    /// \\return The hash value for the given object\n    size_t operator()(const seastar::checked_ptr<T>& p) const {\n        return p.hash();\n    }\n};\n}\n"
  },
  {
    "path": "include/seastar/core/chunked_fifo.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <algorithm>\n#include <cassert>\n#include <concepts>\n#include <iterator>\n#include <type_traits>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\n// An unbounded FIFO queue of objects of type T.\n//\n// It provides operations to push items in one end of the queue, and pop them\n// from the other end of the queue - both operations are guaranteed O(1)\n// (not just amortized O(1)). The size() operation is also O(1).\n// chunked_fifo also guarantees that the largest contiguous memory allocation\n// it does is O(1). The total memory used is, of course, O(N).\n//\n// How does chunked_fifo differ from std::list<>, circular_buffer<> and\n// std::deque()?\n//\n// std::list<> can also make all the above guarantees, but is inefficient -\n// both at run speed (every operation requires an allocation), and in memory\n// use. Much more efficient than std::list<> is our circular_buffer<>, which\n// allocates a contiguous array to hold the items and only reallocates it,\n// exponentially, when the queue grows. On one test of several different\n// push/pop scenarios, circular_buffer<> was between 5 and 20 times faster\n// than std::list, and also used considerably less memory.\n// The problem with circular_buffer<> is that gives up on the last guarantee\n// we made above: circular_buffer<> allocates all the items in one large\n// contiguous allocation - that might not be possible when the memory is\n// highly fragmented.\n// std::deque<> aims to solve the contiguous allocation problem by allocating\n// smaller chunks of the queue, and keeping a list of them in an array. This\n// array is necessary to allow for O(1) random access to any element, a\n// feature which we do not need; But this array is itself contiguous so\n// std::deque<> attempts larger contiguous allocations the larger the queue\n// gets: std::deque<>'s contiguous allocation is still O(N) and in fact\n// exactly 1/64 of the size of circular_buffer<>'s contiguous allocation.\n// So it's an improvement over circular_buffer<>, but not a full solution.\n//\n// chunked_fifo<> is such a solution: it also allocates the queue in fixed-\n// size chunks (just like std::deque) but holds them in a linked list, not\n// a contiguous array, so there are no large contiguous allocations.\n//\n// Unlike std::deque<> or circular_buffer<>, chunked_fifo only provides the\n// operations needed by std::queue, i.e.,: empty(), size(), front(), back(),\n// push_back() and pop_front(). For simplicity, we do *not* implement other\n// possible operations, like inserting or deleting elements from the \"wrong\"\n// side of the queue or from the middle, nor random-access to items in the\n// middle of the queue. However, chunked_fifo does allow iterating over all\n// of the queue's elements without popping them, a feature which std::queue\n// is missing.\n//\n// Another feature of chunked_fifo which std::deque is missing is the ability\n// to control the chunk size, as a template parameter. In std::deque the\n// chunk size is undocumented and fixed - in gcc, it is always 512 bytes.\n// chunked_fifo, on the other hand, makes the chunk size (in number of items\n// instead of bytes) a template parameter; In situations where the queue is\n// expected to become very long, using a larger chunk size might make sense\n// because it will result in fewer allocations.\n//\n// chunked_fifo uses uninitialized storage for unoccupied elements, and thus\n// uses move/copy constructors instead of move/copy assignments, which are\n// less efficient.\n\ntemplate <typename T, size_t items_per_chunk = 128>\nclass chunked_fifo {\n    static_assert((items_per_chunk & (items_per_chunk - 1)) == 0,\n            \"chunked_fifo chunk size must be power of two\");\n    union maybe_item {\n        maybe_item() noexcept {}\n        ~maybe_item() {}\n        T data;\n    };\n    struct chunk {\n        maybe_item items[items_per_chunk];\n        struct chunk* next;\n        // begin and end interpreted mod items_per_chunk\n        unsigned begin;\n        unsigned end;\n\n        // the number of elements in this chunk\n        size_t size() const {\n            return end - begin;\n        }\n    };\n    // We pop from the chunk at _front_chunk. This chunk is then linked to\n    // the following chunks via the \"next\" link. _back_chunk points to the\n    // last chunk in this list, and it is where we push.\n    chunk* _front_chunk = nullptr; // where we pop\n    chunk* _back_chunk = nullptr; // where we push\n    // We want an O(1) size but don't want to maintain a size() counter\n    // because this will slow down every push and pop operation just for\n    // the rare size() call. Instead, we just keep a count of chunks (which\n    // doesn't change on every push or pop), from which we can calculate\n    // size() when needed, and still be O(1).\n    // This assumes the invariant that all middle chunks (except the front\n    // and back) are always full.\n    size_t _nchunks = 0;\n    // A list of freed chunks, to support reserve() and to improve\n    // performance of repeated push and pop, especially on an empty queue.\n    // It is a performance/memory tradeoff how many freed chunks to keep\n    // here (see save_free_chunks constant below).\n    chunk* _free_chunks = nullptr;\n    size_t _nfree_chunks = 0;\npublic:\n    using value_type = T;\n    using size_type = size_t;\n    using reference = T&;\n    using pointer = T*;\n    using const_reference = const T&;\n    using const_pointer = const T*;\n\nprivate:\n    template <bool IsConst>\n    class basic_iterator {\n        friend class chunked_fifo;\n\n    public:\n        using iterator_category = std::forward_iterator_tag;\n        using difference_type = std::ptrdiff_t;\n        using value_type = std::conditional_t<IsConst, const T, T>;\n        using pointer = value_type*;\n        using reference = value_type&;\n\n    private:\n        using chunk_t = std::conditional_t<IsConst, const chunk, chunk>;\n        chunk_t* _chunk = nullptr;\n        size_t _item_index = 0;\n\n    protected:\n        inline explicit basic_iterator(chunk_t* c) noexcept;\n        inline basic_iterator(chunk_t* c, size_t item_index) noexcept;\n\n    public:\n        basic_iterator() noexcept = default;\n        template<bool OtherIsConst>\n        requires (IsConst && !OtherIsConst)\n        inline basic_iterator(const basic_iterator<OtherIsConst>& o) noexcept\n\t  :  basic_iterator{o._chunk, o._item_index} {}\n        inline bool operator==(const basic_iterator& o) const noexcept;\n        inline bool operator!=(const basic_iterator& o) const noexcept;\n        inline pointer operator->() const noexcept;\n        inline reference operator*() const noexcept;\n        inline basic_iterator operator++(int) noexcept;\n        basic_iterator& operator++() noexcept;\n    };\n\npublic:\n    using iterator = basic_iterator<false>;\n    using const_iterator = basic_iterator<true>;\n\npublic:\n    chunked_fifo() noexcept = default;\n    chunked_fifo(chunked_fifo&& x) noexcept;\n    chunked_fifo(const chunked_fifo&);\n    ~chunked_fifo();\n    chunked_fifo& operator=(const chunked_fifo&);\n    chunked_fifo& operator=(chunked_fifo&&) noexcept;\n    inline bool operator==(const chunked_fifo& rhs) const;\n    inline void push_back(const T& data);\n    inline void push_back(T&& data);\n    T& back() noexcept;\n    const T& back() const noexcept;\n    template <typename... A>\n    inline void emplace_back(A&&... args);\n    inline T& front() const noexcept;\n    inline void pop_front() noexcept;\n    inline bool empty() const noexcept;\n    inline size_t size() const noexcept;\n    // Pop the first n elements from the fifo. Equivalent to calling pop_front()\n    // n times, though likely to be faster for n greater than 1. The fifo must\n    // contain at least n elements or the behavior is undefined.\n    void pop_front_n(size_t n) noexcept;\n    void clear() noexcept;\n    // reserve(n) ensures that at least (n - size()) further push() calls can\n    // be served without needing new memory allocation.\n    // Calling pop()s between these push()es is also allowed and does not\n    // alter this guarantee.\n    // Note that reserve() does not reduce the amount of memory already\n    // reserved - use shrink_to_fit() for that.\n    void reserve(size_t n);\n    // shrink_to_fit() frees memory held, but unused, by the queue. Such\n    // unused memory might exist after pops, or because of reserve().\n    void shrink_to_fit() noexcept;\n    inline iterator begin() noexcept;\n    inline iterator end() noexcept;\n    inline const_iterator begin() const noexcept;\n    inline const_iterator end() const noexcept;\n    inline const_iterator cbegin() const noexcept;\n    inline const_iterator cend() const noexcept;\nprivate:\n    void back_chunk_new();\n    void front_chunk_delete() noexcept;\n    inline void ensure_room_back();\n    void undo_room_back() noexcept;\n    static inline size_t mask(size_t idx) noexcept;\n\n};\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::basic_iterator(chunk_t* c) noexcept : _chunk(c), _item_index(_chunk ? _chunk->begin : 0) {\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::basic_iterator(chunk_t* c, size_t item_index) noexcept : _chunk(c), _item_index(item_index) {\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline bool\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator==(const basic_iterator& o) const noexcept {\n    return _chunk == o._chunk && _item_index == o._item_index;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline bool\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator!=(const basic_iterator& o) const noexcept {\n    return !(*this == o);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline typename chunked_fifo<T, items_per_chunk>::template basic_iterator<IsConst>::pointer\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator->() const noexcept {\n    return &_chunk->items[chunked_fifo::mask(_item_index)].data;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline typename chunked_fifo<T, items_per_chunk>::template basic_iterator<IsConst>::reference\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator*() const noexcept {\n    return _chunk->items[chunked_fifo::mask(_item_index)].data;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ninline typename chunked_fifo<T, items_per_chunk>::template basic_iterator<IsConst>\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator++(int) noexcept {\n    auto it = *this;\n    ++(*this);\n    return it;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <bool IsConst>\ntypename chunked_fifo<T, items_per_chunk>::template basic_iterator<IsConst>&\nchunked_fifo<T, items_per_chunk>::basic_iterator<IsConst>::operator++() noexcept {\n    ++_item_index;\n    if (_item_index == _chunk->end) {\n        _chunk = _chunk->next;\n        _item_index = _chunk ? _chunk->begin : 0;\n    }\n    return *this;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nchunked_fifo<T, items_per_chunk>::chunked_fifo(chunked_fifo&& x) noexcept\n        : _front_chunk(x._front_chunk)\n        , _back_chunk(x._back_chunk)\n        , _nchunks(x._nchunks)\n        , _free_chunks(x._free_chunks)\n        , _nfree_chunks(x._nfree_chunks) {\n    x._front_chunk = nullptr;\n    x._back_chunk = nullptr;\n    x._nchunks = 0;\n    x._free_chunks = nullptr;\n    x._nfree_chunks = 0;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nchunked_fifo<T, items_per_chunk>::chunked_fifo(const chunked_fifo& rhs)\n        : chunked_fifo() {\n    std::copy_n(rhs.begin(), rhs.size(), std::back_inserter(*this));\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nchunked_fifo<T, items_per_chunk>&\nchunked_fifo<T, items_per_chunk>::operator=(const chunked_fifo& rhs) {\n    if (&rhs != this) {\n        clear();\n        std::copy_n(rhs.begin(), rhs.size(), std::back_inserter(*this));\n        shrink_to_fit();\n    }\n    return *this;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nchunked_fifo<T, items_per_chunk>&\nchunked_fifo<T, items_per_chunk>::operator=(chunked_fifo&& x) noexcept {\n    if (&x != this) {\n        this->~chunked_fifo();\n        new (this) chunked_fifo(std::move(x));\n    }\n    return *this;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline size_t\nchunked_fifo<T, items_per_chunk>::mask(size_t idx) noexcept {\n    return idx & (items_per_chunk - 1);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline bool\nchunked_fifo<T, items_per_chunk>::empty() const noexcept {\n    return _front_chunk == nullptr;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline size_t\nchunked_fifo<T, items_per_chunk>::size() const noexcept{\n    if (_front_chunk == nullptr) {\n        return 0;\n    } else if (_back_chunk == _front_chunk) {\n        // Single chunk.\n        return _front_chunk->end - _front_chunk->begin;\n    } else {\n        return _front_chunk->end - _front_chunk->begin\n                +_back_chunk->end - _back_chunk->begin\n                + (_nchunks - 2) * items_per_chunk;\n    }\n}\n\ntemplate <typename T, size_t items_per_chunk>\nvoid chunked_fifo<T, items_per_chunk>::clear() noexcept {\n    pop_front_n(size());\n}\n\ntemplate <typename T, size_t items_per_chunk>\nvoid chunked_fifo<T, items_per_chunk>::pop_front_n(size_t n) noexcept {\n    while (n) {\n        SEASTAR_ASSERT(_front_chunk && \"pop_front_n n too large\");\n\n        auto target = _front_chunk;\n        unsigned delete_count = std::min(target->size(), n);\n\n        for (auto i = target->begin, e = i + delete_count; i != e; i++) {\n            target->items[mask(i)].data.~T();\n        }\n\n        target->begin += delete_count;\n        n -= delete_count;\n\n        if (target->size() == 0) {\n            front_chunk_delete();\n        }\n    }\n}\n\n\n\ntemplate <typename T, size_t items_per_chunk> void\nchunked_fifo<T, items_per_chunk>::shrink_to_fit() noexcept {\n    while (_free_chunks) {\n        auto next = _free_chunks->next;\n        delete _free_chunks;\n        _free_chunks = next;\n    }\n    _nfree_chunks = 0;\n}\n\ntemplate <typename T, size_t items_per_chunk>\nchunked_fifo<T, items_per_chunk>::~chunked_fifo() {\n    clear();\n    shrink_to_fit();\n}\n\ntemplate <typename T, size_t items_per_chunk>\nvoid\nchunked_fifo<T, items_per_chunk>::back_chunk_new() {\n    chunk *old = _back_chunk;\n    if (_free_chunks) {\n        _back_chunk = _free_chunks;\n        _free_chunks = _free_chunks->next;\n        --_nfree_chunks;\n    } else {\n        _back_chunk = new chunk;\n    }\n    _back_chunk->next = nullptr;\n    _back_chunk->begin = 0;\n    _back_chunk->end = 0;\n    if (old) {\n        old->next = _back_chunk;\n    }\n    if (_front_chunk == nullptr) {\n        _front_chunk = _back_chunk;\n    }\n    _nchunks++;\n}\n\n\ntemplate <typename T, size_t items_per_chunk>\ninline void\nchunked_fifo<T, items_per_chunk>::ensure_room_back() {\n    // If we don't have a back chunk or it's full, we need to create a new one\n    if (_back_chunk == nullptr ||\n            (_back_chunk->end - _back_chunk->begin) == items_per_chunk) {\n        back_chunk_new();\n    }\n}\n\ntemplate <typename T, size_t items_per_chunk>\nvoid\nchunked_fifo<T, items_per_chunk>::undo_room_back() noexcept {\n    // If we failed creating a new item after ensure_room_back() created a\n    // new empty chunk, we must remove it, or empty() will be incorrect\n    // (either immediately, if the fifo was empty, or when all the items are\n    // popped, if it already had items).\n    if (_back_chunk->begin == _back_chunk->end) {\n        delete _back_chunk;\n        --_nchunks;\n        if (_nchunks == 0) {\n            _back_chunk = nullptr;\n            _front_chunk = nullptr;\n        } else {\n            // Because we don't usually pop from the back, we don't have a \"prev\"\n            // pointer so we need to find the previous chunk the hard and slow\n            // way. B\n            chunk *old = _back_chunk;\n            _back_chunk = _front_chunk;\n            while (_back_chunk->next != old) {\n                _back_chunk = _back_chunk->next;\n            }\n            _back_chunk->next = nullptr;\n        }\n    }\n\n}\n\ntemplate <typename T, size_t items_per_chunk>\ntemplate <typename... Args>\ninline void\nchunked_fifo<T, items_per_chunk>::emplace_back(Args&&... args) {\n    ensure_room_back();\n    auto p = &_back_chunk->items[mask(_back_chunk->end)].data;\n    try {\n        new(p) T(std::forward<Args>(args)...);\n    } catch(...) {\n        undo_room_back();\n        throw;\n    }\n    ++_back_chunk->end;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline bool chunked_fifo<T, items_per_chunk>::operator==(const chunked_fifo& rhs) const {\n    return size() == rhs.size() && std::equal(begin(), end(), rhs.begin());\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline void\nchunked_fifo<T, items_per_chunk>::push_back(const T& data) {\n    ensure_room_back();\n    auto p = &_back_chunk->items[mask(_back_chunk->end)].data;\n    try {\n        new(p) T(data);\n    } catch(...) {\n        undo_room_back();\n        throw;\n    }\n    ++_back_chunk->end;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline void\nchunked_fifo<T, items_per_chunk>::push_back(T&& data) {\n    ensure_room_back();\n    auto p = &_back_chunk->items[mask(_back_chunk->end)].data;\n    try {\n        new(p) T(std::move(data));\n    } catch(...) {\n        undo_room_back();\n        throw;\n    }\n    ++_back_chunk->end;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nT&\nchunked_fifo<T, items_per_chunk>::back() noexcept {\n    return _back_chunk->items[mask(_back_chunk->end - 1)].data;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline\nconst T&\nchunked_fifo<T, items_per_chunk>::back() const noexcept {\n    return _back_chunk->items[mask(_back_chunk->end - 1)].data;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline T&\nchunked_fifo<T, items_per_chunk>::front() const noexcept {\n    return _front_chunk->items[mask(_front_chunk->begin)].data;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline void\nchunked_fifo<T, items_per_chunk>::front_chunk_delete() noexcept {\n    chunk *next = _front_chunk->next;\n    // Certain use cases may need to repeatedly allocate and free a chunk -\n    // an obvious example is an empty queue to which we push, and then pop,\n    // repeatedly. Another example is pushing and popping to a non-empty queue\n    // we push and pop at different chunks so we need to free and allocate a\n    // chunk every items_per_chunk operations.\n    // The solution is to keep a list of freed chunks instead of freeing them\n    // immediately. There is a performance/memory tradeoff of how many freed\n    // chunks to save: If we save them all, the queue can never shrink from\n    // its maximum memory use (this is how circular_buffer behaves).\n    // The ad-hoc choice made here is to limit the number of saved chunks to 1,\n    // but this could easily be made a configuration option.\n    static constexpr int save_free_chunks = 1;\n    if (_nfree_chunks < save_free_chunks) {\n        _front_chunk->next = _free_chunks;\n        _free_chunks = _front_chunk;\n        ++_nfree_chunks;\n    } else {\n        delete _front_chunk;\n    }\n    // If we only had one chunk, _back_chunk is gone too.\n    if (_back_chunk == _front_chunk) {\n        _back_chunk = nullptr;\n    }\n    _front_chunk = next;\n    --_nchunks;\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline void\nchunked_fifo<T, items_per_chunk>::pop_front() noexcept {\n    front().~T();\n    // If the front chunk has become empty, we need to free remove it and use\n    // the next one.\n    if (++_front_chunk->begin == _front_chunk->end) {\n        front_chunk_delete();\n    }\n}\n\ntemplate <typename T, size_t items_per_chunk>\nvoid chunked_fifo<T, items_per_chunk>::reserve(size_t n) {\n    // reserve() guarantees that (n - size()) additional push()es will\n    // succeed without reallocation:\n    if (n <= size()) {\n        return;\n    }\n    size_t need = n - size();\n    // If we already have a back chunk, it might have room for some pushes\n    // before filling up, so decrease \"need\":\n    if (_back_chunk) {\n        size_t back_chunk_n = items_per_chunk - (_back_chunk->end - _back_chunk->begin);\n        need -= std::min(back_chunk_n, need);\n    }\n    size_t needed_chunks = (need + items_per_chunk - 1) / items_per_chunk;\n    // If we already have some freed chunks saved, we need to allocate fewer\n    // additional chunks, or none at all\n    if (needed_chunks <= _nfree_chunks) {\n        return;\n    }\n    needed_chunks -= _nfree_chunks;\n    while (needed_chunks--) {\n        chunk *c = new chunk;\n        c->next = _free_chunks;\n        _free_chunks = c;\n        ++_nfree_chunks;\n    }\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::iterator\nchunked_fifo<T, items_per_chunk>::begin() noexcept {\n    return iterator(_front_chunk);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::iterator\nchunked_fifo<T, items_per_chunk>::end() noexcept {\n    return iterator(nullptr);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::const_iterator\nchunked_fifo<T, items_per_chunk>::begin() const noexcept {\n    return const_iterator(_front_chunk);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::const_iterator\nchunked_fifo<T, items_per_chunk>::end() const noexcept {\n    return const_iterator(nullptr);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::const_iterator\nchunked_fifo<T, items_per_chunk>::cbegin() const noexcept {\n    return const_iterator(_front_chunk);\n}\n\ntemplate <typename T, size_t items_per_chunk>\ninline typename chunked_fifo<T, items_per_chunk>::const_iterator\nchunked_fifo<T, items_per_chunk>::cend() const noexcept {\n    return const_iterator(nullptr);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/circular_buffer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/transfer.hh>\n#include <seastar/core/bitops.hh>\n#include <concepts>\n#include <memory>\n#include <algorithm>\n\nnamespace seastar {\n\n/// A growable double-ended queue container that can be efficiently\n/// extended (and shrunk) from both ends. Implementation is a single\n/// storage vector.\n///\n/// Similar to libstdc++'s std::deque, except that it uses a single\n/// level store, and so is more efficient for simple stored items.\n/// Similar to boost::circular_buffer_space_optimized, except it uses\n/// uninitialized storage for unoccupied elements (and thus move/copy\n/// constructors instead of move/copy assignments, which are less\n/// efficient).\n///\n/// The storage of the circular_buffer is expanded automatically in\n/// exponential increments.\n/// When adding new elements:\n/// * if size + 1 > capacity: all iterators and references are\n///     invalidated,\n/// * otherwise only the begin() or end() iterator is invalidated:\n///     * push_front() and emplace_front() will invalidate begin() and\n///     * push_back() and emplace_back() will invalidate end().\n///\n/// Removing elements never invalidates any references and only\n/// invalidates begin() or end() iterators:\n///     * pop_front() will invalidate begin() and\n///     * pop_back() will invalidate end().\n///\n/// reserve() may also invalidate all iterators and references.\ntemplate <typename T, typename Alloc = std::allocator<T>>\nclass circular_buffer {\n    struct impl : Alloc {\n        T* storage = nullptr;\n        // begin, end interpreted (mod capacity)\n        size_t begin = 0;\n        size_t end = 0;\n        size_t capacity = 0;\n\n        impl(Alloc a) noexcept : Alloc(std::move(a)) { }\n        void reset() {\n            storage = {};\n            begin = 0;\n            end = 0;\n            capacity = 0;\n        }\n    };\n    static_assert(!std::is_default_constructible_v<Alloc>\n                  || std::is_nothrow_default_constructible_v<Alloc>);\n    static_assert(std::is_nothrow_move_constructible_v<Alloc>);\n    impl _impl;\npublic:\n    using value_type = T;\n    using size_type = size_t;\n    using reference = T&;\n    using pointer = T*;\n    using const_reference = const T&;\n    using const_pointer = const T*;\npublic:\n    circular_buffer() noexcept requires std::default_initializable<Alloc> : circular_buffer(Alloc()) {}\n    circular_buffer(Alloc alloc) noexcept;\n    circular_buffer(circular_buffer&& X) noexcept;\n    circular_buffer(const circular_buffer& X) = delete;\n    ~circular_buffer();\n    circular_buffer& operator=(const circular_buffer&) = delete;\n    circular_buffer& operator=(circular_buffer&& b) noexcept;\n    void push_front(const T& data);\n    void push_front(T&& data);\n    template <typename... A>\n    void emplace_front(A&&... args);\n    void push_back(const T& data);\n    void push_back(T&& data);\n    template <typename... A>\n    void emplace_back(A&&... args);\n    T& front() noexcept;\n    const T& front() const noexcept;\n    T& back() noexcept;\n    const T& back() const noexcept;\n    void pop_front() noexcept;\n    void pop_back() noexcept;\n    bool empty() const noexcept;\n    size_t size() const noexcept;\n    size_t capacity() const noexcept;\n    void reserve(size_t);\n    void clear() noexcept;\n    T& operator[](size_t idx) noexcept;\n    const T& operator[](size_t idx) const noexcept;\n    template <typename Func>\n    void for_each(Func func);\n    // access an element, may return wrong or destroyed element\n    // only useful if you do not rely on data accuracy (e.g. prefetch)\n    T& access_element_unsafe(size_t idx) noexcept;\nprivate:\n    void expand();\n    void expand(size_t);\n    void maybe_expand(size_t nr = 1);\n    size_t mask(size_t idx) const;\n\n    template<typename CB, typename ValueType>\n    struct cbiterator {\n        using iterator_category = std::random_access_iterator_tag;\n        using value_type = ValueType;\n        using difference_type = std::ptrdiff_t;\n        using pointer = ValueType*;\n        using reference = ValueType&;\n\n        ValueType& operator*() const noexcept { return cb->_impl.storage[cb->mask(idx)]; }\n        ValueType* operator->() const noexcept { return &cb->_impl.storage[cb->mask(idx)]; }\n        // prefix\n        cbiterator<CB, ValueType>& operator++() noexcept {\n            idx++;\n            return *this;\n        }\n        // postfix\n        cbiterator<CB, ValueType> operator++(int) noexcept {\n            auto v = *this;\n            idx++;\n            return v;\n        }\n        // prefix\n        cbiterator<CB, ValueType>& operator--() noexcept {\n            idx--;\n            return *this;\n        }\n        // postfix\n        cbiterator<CB, ValueType> operator--(int) noexcept {\n            auto v = *this;\n            idx--;\n            return v;\n        }\n        cbiterator<CB, ValueType> operator+(difference_type n) const noexcept {\n            return cbiterator<CB, ValueType>(cb, idx + n);\n        }\n        cbiterator<CB, ValueType> operator-(difference_type n) const noexcept {\n            return cbiterator<CB, ValueType>(cb, idx - n);\n        }\n        cbiterator<CB, ValueType>& operator+=(difference_type n) noexcept {\n            idx += n;\n            return *this;\n        }\n        cbiterator<CB, ValueType>& operator-=(difference_type n) noexcept {\n            idx -= n;\n            return *this;\n        }\n        bool operator==(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return idx == rhs.idx;\n        }\n        bool operator!=(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return idx != rhs.idx;\n        }\n        bool operator<(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return *this - rhs < 0;\n        }\n        bool operator>(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return *this - rhs > 0;\n        }\n        bool operator>=(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return *this - rhs >= 0;\n        }\n        bool operator<=(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return *this - rhs <= 0;\n        }\n       difference_type operator-(const cbiterator<CB, ValueType>& rhs) const noexcept {\n            return idx - rhs.idx;\n        }\n        cbiterator() = default;\n    private:\n        CB* cb{nullptr};\n        size_t idx{0};\n        cbiterator(CB* b, size_t i) noexcept : cb(b), idx(i) {}\n        friend class circular_buffer;\n    };\n    friend class iterator;\n\npublic:\n    typedef cbiterator<circular_buffer, T> iterator;\n    typedef cbiterator<const circular_buffer, const T> const_iterator;\n\n    iterator begin() noexcept {\n        return iterator(this, _impl.begin);\n    }\n    const_iterator begin() const noexcept {\n        return const_iterator(this, _impl.begin);\n    }\n    iterator end() noexcept {\n        return iterator(this, _impl.end);\n    }\n    const_iterator end() const noexcept {\n        return const_iterator(this, _impl.end);\n    }\n    const_iterator cbegin() const noexcept {\n        return const_iterator(this, _impl.begin);\n    }\n    const_iterator cend() const noexcept {\n        return const_iterator(this, _impl.end);\n    }\n    iterator erase(iterator first, iterator last) noexcept;\n};\n\ntemplate <typename T, typename Alloc>\ninline\nsize_t\ncircular_buffer<T, Alloc>::mask(size_t idx) const {\n    return idx & (_impl.capacity - 1);\n}\n\ntemplate <typename T, typename Alloc>\ninline\nbool\ncircular_buffer<T, Alloc>::empty() const noexcept {\n    return _impl.begin == _impl.end;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nsize_t\ncircular_buffer<T, Alloc>::size() const noexcept {\n    return _impl.end - _impl.begin;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nsize_t\ncircular_buffer<T, Alloc>::capacity() const noexcept {\n    return _impl.capacity;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::reserve(size_t size) {\n    if (capacity() < size) {\n        // Make sure that the new capacity is a power of two.\n        expand(size_t(1) << log2ceil(size));\n    }\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::clear() noexcept {\n    erase(begin(), end());\n}\n\ntemplate <typename T, typename Alloc>\ninline\ncircular_buffer<T, Alloc>::circular_buffer(Alloc alloc) noexcept\n    : _impl(std::move(alloc)) {\n}\n\ntemplate <typename T, typename Alloc>\ninline\ncircular_buffer<T, Alloc>::circular_buffer(circular_buffer&& x) noexcept\n    : _impl(std::move(x._impl)) {\n    x._impl.reset();\n}\n\ntemplate <typename T, typename Alloc>\ninline\ncircular_buffer<T, Alloc>& circular_buffer<T, Alloc>::operator=(circular_buffer&& x) noexcept {\n    if (this != &x) {\n        this->~circular_buffer();\n        new (this) circular_buffer(std::move(x));\n    }\n    return *this;\n}\n\ntemplate <typename T, typename Alloc>\ntemplate <typename Func>\ninline\nvoid\ncircular_buffer<T, Alloc>::for_each(Func func) {\n    auto s = _impl.storage;\n    auto m = _impl.capacity - 1;\n    for (auto i = _impl.begin; i != _impl.end; ++i) {\n        func(s[i & m]);\n    }\n}\n\ntemplate <typename T, typename Alloc>\ninline\ncircular_buffer<T, Alloc>::~circular_buffer() {\n    for_each([this] (T& obj) {\n        std::allocator_traits<Alloc>::destroy(_impl, &obj);\n    });\n    _impl.deallocate(_impl.storage, _impl.capacity);\n}\n\ntemplate <typename T, typename Alloc>\nvoid\ncircular_buffer<T, Alloc>::expand() {\n    expand(std::max<size_t>(_impl.capacity * 2, 1));\n}\n\ntemplate <typename T, typename Alloc>\nvoid\ncircular_buffer<T, Alloc>::expand(size_t new_cap) {\n    auto new_storage = _impl.allocate(new_cap);\n    auto p = new_storage;\n    try {\n        for_each([this, &p] (T& obj) {\n            transfer_pass1(_impl, &obj, p);\n            p++;\n        });\n    } catch (...) {\n        while (p != new_storage) {\n            std::allocator_traits<Alloc>::destroy(_impl, --p);\n        }\n        _impl.deallocate(new_storage, new_cap);\n        throw;\n    }\n    p = new_storage;\n    for_each([this, &p] (T& obj) {\n        transfer_pass2(_impl, &obj, p++);\n    });\n    std::swap(_impl.storage, new_storage);\n    std::swap(_impl.capacity, new_cap);\n    _impl.begin = 0;\n    _impl.end = p - _impl.storage;\n    _impl.deallocate(new_storage, new_cap);\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::maybe_expand(size_t nr) {\n    if (_impl.end - _impl.begin + nr > _impl.capacity) {\n        expand();\n    }\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::push_front(const T& data) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.begin - 1)];\n    std::allocator_traits<Alloc>::construct(_impl, p, data);\n    --_impl.begin;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::push_front(T&& data) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.begin - 1)];\n    std::allocator_traits<Alloc>::construct(_impl, p, std::move(data));\n    --_impl.begin;\n}\n\ntemplate <typename T, typename Alloc>\ntemplate <typename... Args>\ninline\nvoid\ncircular_buffer<T, Alloc>::emplace_front(Args&&... args) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.begin - 1)];\n    std::allocator_traits<Alloc>::construct(_impl, p, std::forward<Args>(args)...);\n    --_impl.begin;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::push_back(const T& data) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.end)];\n    std::allocator_traits<Alloc>::construct(_impl, p, data);\n    ++_impl.end;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::push_back(T&& data) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.end)];\n    std::allocator_traits<Alloc>::construct(_impl, p, std::move(data));\n    ++_impl.end;\n}\n\ntemplate <typename T, typename Alloc>\ntemplate <typename... Args>\ninline\nvoid\ncircular_buffer<T, Alloc>::emplace_back(Args&&... args) {\n    maybe_expand();\n    auto p = &_impl.storage[mask(_impl.end)];\n    std::allocator_traits<Alloc>::construct(_impl, p, std::forward<Args>(args)...);\n    ++_impl.end;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nT&\ncircular_buffer<T, Alloc>::front() noexcept {\n    return _impl.storage[mask(_impl.begin)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nconst T&\ncircular_buffer<T, Alloc>::front() const noexcept {\n    return _impl.storage[mask(_impl.begin)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nT&\ncircular_buffer<T, Alloc>::back() noexcept {\n    return _impl.storage[mask(_impl.end - 1)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nconst T&\ncircular_buffer<T, Alloc>::back() const noexcept {\n    return _impl.storage[mask(_impl.end - 1)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::pop_front() noexcept {\n    std::allocator_traits<Alloc>::destroy(_impl, &front());\n    ++_impl.begin;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ncircular_buffer<T, Alloc>::pop_back() noexcept {\n    std::allocator_traits<Alloc>::destroy(_impl, &back());\n    --_impl.end;\n}\n\ntemplate <typename T, typename Alloc>\ninline\nT&\ncircular_buffer<T, Alloc>::operator[](size_t idx) noexcept {\n    return _impl.storage[mask(_impl.begin + idx)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nconst T&\ncircular_buffer<T, Alloc>::operator[](size_t idx) const noexcept {\n    return _impl.storage[mask(_impl.begin + idx)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\nT&\ncircular_buffer<T, Alloc>::access_element_unsafe(size_t idx) noexcept {\n    return _impl.storage[mask(_impl.begin + idx)];\n}\n\ntemplate <typename T, typename Alloc>\ninline\ntypename circular_buffer<T, Alloc>::iterator\ncircular_buffer<T, Alloc>::erase(iterator first, iterator last) noexcept {\n    static_assert(std::is_nothrow_move_assignable_v<T>, \"erase() assumes move assignment does not throw\");\n    if (first == last) {\n        return last;\n    }\n    // Move to the left or right depending on which would result in least amount of moves.\n    // This also guarantees that iterators will be stable when removing from either front or back.\n    if (std::distance(begin(), first) < std::distance(last, end())) {\n        auto new_start = std::move_backward(begin(), first, last);\n        for (auto i = begin(); i < new_start; ++i) {\n            std::allocator_traits<Alloc>::destroy(_impl, &*i);\n        }\n        _impl.begin = new_start.idx;\n        return last;\n    } else {\n        auto new_end = std::move(last, end(), first);\n        for (auto i = new_end, e = end(); i < e; ++i) {\n            std::allocator_traits<Alloc>::destroy(_impl, &*i);\n        }\n        _impl.end = new_end.idx;\n        return first;\n    }\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/circular_buffer_fixed_capacity.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n// A fixed capacity double-ended queue container that can be efficiently\n// extended (and shrunk) from both ends.  Implementation is a single\n// storage vector.\n//\n// Similar to libstdc++'s std::deque, except that it uses a single level\n// store, and so is more efficient for simple stored items.\n\n#include <algorithm>\n#include <type_traits>\n#include <cstddef>\n#include <iterator>\n#include <utility>\n#include <memory>\n\n#include <seastar/core/bitops.hh>\n\n/// \\file\n\nnamespace seastar {\n\n/// A fixed-capacity container (like boost::static_vector) that can insert\n/// and remove at both ends (like std::deque). Does not allocate.\n///\n/// Does not perform overflow checking when size exceeds capacity.\n///\n/// \\tparam T type of objects stored in the container; must be noexcept move enabled\n/// \\tparam Capacity maximum number of objects that can be stored in the container; must be a power of 2\ntemplate <typename T, size_t Capacity>\nclass circular_buffer_fixed_capacity {\n    size_t _begin = 0;\n    size_t _end = 0;\n    union maybe_storage {\n        T data;\n        maybe_storage() noexcept {}\n        ~maybe_storage() {}\n    };\n    maybe_storage _storage[Capacity];\nprivate:\n    static size_t mask(size_t idx) { return idx % Capacity; }\n    T* obj(size_t idx) { return &_storage[mask(idx)].data; }\n    const T* obj(size_t idx) const { return &_storage[mask(idx)].data; }\npublic:\n    static_assert((Capacity & (Capacity - 1)) == 0, \"capacity must be a power of two\");\n    static_assert(std::is_nothrow_move_constructible_v<T> && std::is_nothrow_move_assignable_v<T>,\n            \"circular_buffer_fixed_capacity only supports nothrow-move value types\");\n    using value_type = T;\n    using size_type = size_t;\n    using reference = T&;\n    using pointer = T*;\n    using const_reference = const T&;\n    using const_pointer = const T*;\n    using difference_type = ssize_t;\npublic:\n    template <typename ValueType>\n    class cbiterator {\n        using holder = std::conditional_t<std::is_const_v<ValueType>, const maybe_storage, maybe_storage>;\n        holder* _start;\n        size_t _idx;\n    private:\n        cbiterator(holder* start, size_t idx) noexcept : _start(start), _idx(idx) {}\n    public:\n        using iterator_category = std::random_access_iterator_tag;\n        using value_type = ValueType;\n        using difference_type = ssize_t;\n        using pointer = ValueType*;\n        using reference = ValueType&;\n    public:\n        cbiterator();\n        ValueType& operator*() const { return _start[mask(_idx)].data; }\n        ValueType* operator->() const { return &operator*(); }\n        // prefix\n        cbiterator& operator++() {\n            ++_idx;\n            return *this;\n        }\n        // postfix\n        cbiterator operator++(int) {\n            auto v = *this;\n            ++_idx;\n            return v;\n        }\n        // prefix\n        cbiterator& operator--() {\n            --_idx;\n            return *this;\n        }\n        // postfix\n        cbiterator operator--(int) {\n            auto v = *this;\n            --_idx;\n            return v;\n        }\n        cbiterator operator+(difference_type n) const {\n            return cbiterator{_start, _idx + n};\n        }\n        friend cbiterator operator+(difference_type n, cbiterator i) {\n            return i + n;\n        }\n        cbiterator operator-(difference_type n) const {\n            return cbiterator{_start, _idx - n};\n        }\n        cbiterator& operator+=(difference_type n) {\n            _idx += n;\n            return *this;\n        }\n        cbiterator& operator-=(difference_type n) {\n            _idx -= n;\n            return *this;\n        }\n        bool operator==(const cbiterator& rhs) const {\n            return _idx == rhs._idx;\n        }\n        bool operator!=(const cbiterator& rhs) const {\n            return _idx != rhs._idx;\n        }\n        bool operator<(const cbiterator& rhs) const {\n            return ssize_t(_idx - rhs._idx) < 0;\n        }\n        bool operator>(const cbiterator& rhs) const {\n            return ssize_t(_idx - rhs._idx) > 0;\n        }\n        bool operator<=(const cbiterator& rhs) const {\n            return ssize_t(_idx - rhs._idx) <= 0;\n        }\n        bool operator>=(const cbiterator& rhs) const {\n            return ssize_t(_idx - rhs._idx) >= 0;\n        }\n        difference_type operator-(const cbiterator& rhs) const {\n            return _idx - rhs._idx;\n        }\n        friend class circular_buffer_fixed_capacity;\n    };\npublic:\n    using iterator = cbiterator<T>;\n    using const_iterator = cbiterator<const T>;\npublic:\n    circular_buffer_fixed_capacity() = default;\n    circular_buffer_fixed_capacity(circular_buffer_fixed_capacity&& x) noexcept;\n    ~circular_buffer_fixed_capacity();\n    circular_buffer_fixed_capacity& operator=(circular_buffer_fixed_capacity&& x) noexcept;\n    void push_front(const T& data);\n    void push_front(T&& data);\n    template <typename... A>\n    T& emplace_front(A&&... args);\n    void push_back(const T& data);\n    void push_back(T&& data);\n    template <typename... A>\n    T& emplace_back(A&&... args);\n    T& front();\n    T& back();\n    void pop_front();\n    void pop_back();\n    bool empty() const;\n    size_t size() const;\n    size_t capacity() const;\n    T& operator[](size_t idx);\n    void clear();\n    iterator begin() {\n        return iterator(_storage, _begin);\n    }\n    const_iterator begin() const {\n        return const_iterator(_storage, _begin);\n    }\n    iterator end() {\n        return iterator(_storage, _end);\n    }\n    const_iterator end() const {\n        return const_iterator(_storage, _end);\n    }\n    const_iterator cbegin() const {\n        return const_iterator(_storage, _begin);\n    }\n    const_iterator cend() const {\n        return const_iterator(_storage, _end);\n    }\n    iterator erase(iterator first, iterator last);\n};\n\ntemplate <typename T, size_t Capacity>\ninline\nbool\ncircular_buffer_fixed_capacity<T, Capacity>::empty() const {\n    return _begin == _end;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nsize_t\ncircular_buffer_fixed_capacity<T, Capacity>::size() const {\n    return _end - _begin;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nsize_t\ncircular_buffer_fixed_capacity<T, Capacity>::capacity() const {\n    return Capacity;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\ncircular_buffer_fixed_capacity<T, Capacity>::circular_buffer_fixed_capacity(circular_buffer_fixed_capacity&& x) noexcept\n        : _begin(x._begin), _end(x._end) {\n    std::uninitialized_move(x.begin(), x.end(), begin());\n}\n\ntemplate <typename T, size_t Capacity>\ninline\ncircular_buffer_fixed_capacity<T, Capacity>&\ncircular_buffer_fixed_capacity<T, Capacity>::operator=(circular_buffer_fixed_capacity&& x) noexcept {\n    if (this != &x) {\n        this->~circular_buffer_fixed_capacity();\n        new (this) circular_buffer_fixed_capacity(std::move(x));\n    }\n    return *this;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\ncircular_buffer_fixed_capacity<T, Capacity>::~circular_buffer_fixed_capacity() {\n    clear();\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::push_front(const T& data) {\n    new (obj(_begin - 1)) T(data);\n    --_begin;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::push_front(T&& data) {\n    new (obj(_begin - 1)) T(std::move(data));\n    --_begin;\n}\n\ntemplate <typename T, size_t Capacity>\ntemplate <typename... Args>\ninline\nT&\ncircular_buffer_fixed_capacity<T, Capacity>::emplace_front(Args&&... args) {\n    auto p = new (obj(_begin - 1)) T(std::forward<Args>(args)...);\n    --_begin;\n    return *p;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::push_back(const T& data) {\n    new (obj(_end)) T(data);\n    ++_end;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::push_back(T&& data) {\n    new (obj(_end)) T(std::move(data));\n    ++_end;\n}\n\ntemplate <typename T, size_t Capacity>\ntemplate <typename... Args>\ninline\nT&\ncircular_buffer_fixed_capacity<T, Capacity>::emplace_back(Args&&... args) {\n    auto p = new (obj(_end)) T(std::forward<Args>(args)...);\n    ++_end;\n    return *p;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nT&\ncircular_buffer_fixed_capacity<T, Capacity>::front() {\n    return *obj(_begin);\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nT&\ncircular_buffer_fixed_capacity<T, Capacity>::back() {\n    return *obj(_end - 1);\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::pop_front() {\n    obj(_begin)->~T();\n    ++_begin;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::pop_back() {\n    obj(_end - 1)->~T();\n    --_end;\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nT&\ncircular_buffer_fixed_capacity<T, Capacity>::operator[](size_t idx) {\n    return *obj(_begin + idx);\n}\n\ntemplate <typename T, size_t Capacity>\ninline\ntypename circular_buffer_fixed_capacity<T, Capacity>::iterator\ncircular_buffer_fixed_capacity<T, Capacity>::erase(iterator first, iterator last) {\n    static_assert(std::is_nothrow_move_assignable_v<T>, \"erase() assumes move assignment does not throw\");\n    if (first == last) {\n        return last;\n    }\n    // Move to the left or right depending on which would result in least amount of moves.\n    // This also guarantees that iterators will be stable when removing from either front or back.\n    if (std::distance(begin(), first) < std::distance(last, end())) {\n        auto new_start = std::move_backward(begin(), first, last);\n        auto i = begin();\n        while (i < new_start) {\n            i++->~T();\n        }\n        _begin = new_start.idx;\n        return last;\n    } else {\n        auto new_end = std::move(last, end(), first);\n        auto i = new_end;\n        auto e = end();\n        while (i < e) {\n            i++->~T();\n        }\n        _end = new_end.idx;\n        return first;\n    }\n}\n\ntemplate <typename T, size_t Capacity>\ninline\nvoid\ncircular_buffer_fixed_capacity<T, Capacity>::clear() {\n    for (auto& obj : *this) {\n        obj.~T();\n    }\n    _begin = _end = 0;\n}\n\n// Return a circular_buffer power-of-2 capacity\n// For fitting an array of T entries in Capacity bytes.\ntemplate <typename T, size_t Capacity>\nrequires (sizeof(T) <= Capacity)\nconstexpr inline size_t calc_circular_buffer_capacity() {\n    return 1 << (log2ceil(Capacity) - log2ceil(sizeof(T)));\n}\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/condition-variable.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB, Ltd.\n */\n\n#pragma once\n\n#include <boost/intrusive/list.hpp>\n#include <chrono>\n#include <exception>\n#include <functional>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/loop.hh>\n\nnamespace seastar {\n\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// Exception thrown when a condition variable is broken by\n/// \\ref condition_variable::broken().\nclass broken_condition_variable : public std::exception {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept;\n};\n\n/// Exception thrown when wait() operation times out\n/// \\ref condition_variable::wait(time_point timeout).\nclass condition_variable_timed_out : public std::exception {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept;\n};\n\n/// \\brief Conditional variable.\n///\n/// This is a standard computer science condition variable sans locking,\n/// since in seastar access to variables is atomic anyway, adapted\n/// for futures.  You can wait for variable to be notified.\n///\n/// To support exceptional conditions, a \\ref broken() method\n/// is provided, which causes all current waiters to stop waiting,\n/// with an exceptional future returned.  This allows causing all\n/// fibers that are blocked on a condition variable to continue.\n/// This issimilar to POSIX's `pthread_cancel()`, with \\ref wait()\n/// acting as a cancellation point.\n\nclass condition_variable {\nprivate:\n    // the base for queue waiters. looks complicated, but this is\n    // to make it transparent once we add non-promise based nodes\n    struct waiter : public boost::intrusive::list_base_hook<boost::intrusive::link_mode<boost::intrusive::auto_unlink>> {\n        waiter() = default;\n        waiter(waiter&&) = default;\n        waiter(const waiter&) = delete;\n        waiter& operator=(const waiter&) = delete;\n        virtual ~waiter() = default;\n        void timeout() noexcept;\n\n        virtual void signal() noexcept = 0;\n        virtual void set_exception(std::exception_ptr) noexcept = 0;\n    };\n\n    struct promise_waiter : public waiter, public promise<> {\n        void signal() noexcept override {\n            set_value();\n            // note: we self-delete in either case we are woken\n            // up. See usage below: only the resulting future\n            // state is required once we've left the wait queue\n            delete this;\n        }\n        void set_exception(std::exception_ptr ep) noexcept override {\n            promise<>::set_exception(std::move(ep));\n            // see comment above\n            delete this;\n        }\n    };\n\n    struct [[nodiscard(\"must co_await a when() call\")]] awaiter : public waiter {\n        condition_variable* _cv;\n        promise<> _p;\n\n        awaiter(condition_variable* cv)\n            : _cv(cv)\n        {}\n\n        void signal() noexcept override {\n            _p.set_value();\n        }\n        void set_exception(std::exception_ptr ep) noexcept override {\n            _p.set_exception(std::move(ep));\n        }\n        auto operator co_await() {\n            if (_cv->check_and_consume_signal()) {\n                return ::seastar::internal::awaiter<false, void>(make_ready_future<>());\n            }\n            _cv->add_waiter(*this);\n            return ::seastar::internal::awaiter<false, void>(_p.get_future());\n        }\n    };\n\n    template<typename Clock, typename Duration>\n    struct [[nodiscard(\"must co_await a when() call\")]] timeout_awaiter : public awaiter, public timer<Clock> {\n        using my_type = timeout_awaiter<Clock, Duration>;\n        using time_point = std::chrono::time_point<Clock, Duration>;\n\n        time_point _timeout;\n\n        timeout_awaiter(condition_variable* cv, time_point timeout)\n            : awaiter(cv)\n            , _timeout(timeout)\n        {}\n        void signal() noexcept override {\n            this->cancel();\n            awaiter::signal();\n        }\n        void set_exception(std::exception_ptr ep) noexcept override {\n            this->cancel();\n            awaiter::set_exception(std::move(ep));\n        }\n        auto operator co_await() {\n            if (_cv->check_and_consume_signal()) {\n                return ::seastar::internal::awaiter<false, void>(make_ready_future<>());\n            }\n            this->set_callback(std::bind(&waiter::timeout, this));\n            this->arm(_timeout);\n            return awaiter::operator co_await();\n        }\n    };\n\n    template<typename Func, typename Base>\n    struct [[nodiscard(\"must co_await a when() call\")]] predicate_awaiter : public Base {\n        Func _func;\n        template<typename... Args>\n        predicate_awaiter(Func func, Args&& ...args)\n            : Base(std::forward<Args>(args)...)\n            , _func(std::move(func))\n        {}\n        void signal() noexcept override {\n            try {\n                if (_func()) {\n                    Base::signal();\n                } else {\n                    // must re-enter waiter queue\n                    // this maintains \"wait\" version\n                    // semantics of moving to back of queue\n                    // if predicate fails\n                    Base::_cv->add_waiter(*this);\n                }\n            } catch(...) {\n                Base::set_exception(std::current_exception());\n            }\n        }\n        auto operator co_await() {\n            try {\n                if (_func()) {\n                    return ::seastar::internal::awaiter<false, void>(make_ready_future<>());\n                } else {\n                    Base::_cv->check_and_consume_signal(); // clear out any signal state\n                    return Base::operator co_await();\n                }\n            } catch (...) {\n                return ::seastar::internal::awaiter<false, void>(make_exception_future(std::current_exception()));\n            }\n        }\n    };\n\n    boost::intrusive::list<waiter, boost::intrusive::constant_time_size<false>> _waiters;\n    std::exception_ptr _ex; //\"broken\" exception\n    bool _signalled = false; // set to true if signalled while no waiters\n\n    void add_waiter(waiter&) noexcept;\n    void timeout(waiter&) noexcept;\n    bool wakeup_first() noexcept;\n    bool check_and_consume_signal() noexcept;\npublic:\n    /// Constructs a condition_variable object.\n    /// Initialzie the semaphore with a default value of 0 to enusre\n    /// the first call to wait() before signal() won't be waken up immediately.\n    condition_variable() noexcept = default;\n    condition_variable(condition_variable&& rhs) noexcept = default;\n    ~condition_variable();\n\n    /// Waits until condition variable is signaled, may wake up without condition been met\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception.\n    future<> wait() noexcept {\n        if (check_and_consume_signal()) {\n            return make_ready_future();\n        }\n        auto* w = new promise_waiter;\n        auto f = w->get_future();\n        add_waiter(*w);\n        return f;\n    }\n\n    /// Waits until condition variable is signaled or timeout is reached\n    ///\n    /// \\param timeout time point at which wait will exit with a timeout\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is reached will return \\ref condition_variable_timed_out exception.\n    template<typename Clock = typename timer<>::clock, typename Duration = typename Clock::duration>\n    future<> wait(std::chrono::time_point<Clock, Duration> timeout) noexcept {\n        if (check_and_consume_signal()) {\n            return make_ready_future();\n        }\n        struct timeout_waiter : public promise_waiter, public timer<Clock> {};\n\n        auto w = std::make_unique<timeout_waiter>();\n        auto f = w->get_future();\n\n        w->set_callback(std::bind(&waiter::timeout, w.get()));\n        w->arm(timeout);\n        add_waiter(*w.release());\n        return f;\n    }\n\n    /// Waits until condition variable is signaled or timeout is reached\n    ///\n    /// \\param timeout duration after which wait will exit with a timeout\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is passed will return \\ref condition_variable_timed_out exception.\n    template<typename Rep, typename Period>\n    future<> wait(std::chrono::duration<Rep, Period> timeout) noexcept {\n        return wait(timer<>::clock::now() + timeout);\n    }\n\n    /// Waits until condition variable is notified and pred() == true, otherwise\n    /// wait again.\n    ///\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken(), may contain an exception.\n    template<typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    future<> wait(Pred&& pred) noexcept {\n        return do_until(std::forward<Pred>(pred), [this] {\n            return wait();\n        });\n    }\n\n    /// Waits until condition variable is notified and pred() == true or timeout is reached, otherwise\n    /// wait again.\n    ///\n    /// \\param timeout time point at which wait will exit with a timeout\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is reached will return \\ref condition_variable_timed_out exception.\n    template<typename Clock = typename timer<>::clock, typename Duration = typename Clock::duration, typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    future<> wait(std::chrono::time_point<Clock, Duration> timeout, Pred&& pred) noexcept {\n        return do_until(std::forward<Pred>(pred), [this, timeout] {\n            return wait(timeout);\n        });\n    }\n\n    /// Waits until condition variable is notified and pred() == true or timeout is reached, otherwise\n    /// wait again.\n    ///\n    /// \\param timeout duration after which wait will exit with a timeout\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is passed will return \\ref condition_variable_timed_out exception.\n    template<typename Rep, typename Period, typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    future<> wait(std::chrono::duration<Rep, Period> timeout, Pred&& pred) noexcept {\n        return wait(timer<>::clock::now() + timeout, std::forward<Pred>(pred));\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is signaled, may wake up without condition been met\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception.\n    awaiter when() noexcept {\n        return awaiter{this};\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is signaled or timeout is reached\n    ///\n    /// \\param timeout time point at which wait will exit with a timeout\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is reached will return \\ref condition_variable_timed_out exception.\n    template<typename Clock = typename timer<>::clock, typename Duration = typename Clock::duration>\n    timeout_awaiter<Clock, Duration> when(std::chrono::time_point<Clock, Duration> timeout) noexcept {\n        return timeout_awaiter<Clock, Duration>{this, timeout};\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is signaled or timeout is reached\n    ///\n    /// \\param timeout duration after which wait will exit with a timeout\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is passed will return \\ref condition_variable_timed_out exception.\n    template<typename Rep, typename Period>\n    auto when(std::chrono::duration<Rep, Period> timeout) noexcept {\n        return when(timer<>::clock::now() + timeout);\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is notified and pred() == true, otherwise\n    /// wait again.\n    ///\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken(), may contain an exception.\n    template<typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    auto when(Pred&& pred) noexcept {\n        return predicate_awaiter<Pred, awaiter>{std::forward<Pred>(pred), when()};\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is notified and pred() == true or timeout is reached, otherwise\n    /// wait again.\n    ///\n    /// \\param timeout time point at which wait will exit with a timeout\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is reached will return \\ref condition_variable_timed_out exception.\n    template<typename Clock = typename timer<>::clock, typename Duration = typename Clock::duration, typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    auto when(std::chrono::time_point<Clock, Duration> timeout, Pred&& pred) noexcept {\n        return predicate_awaiter<Pred, timeout_awaiter<Clock, Duration>>{std::forward<Pred>(pred), when(timeout)};\n    }\n\n    /// Coroutine/co_await only waiter.\n    /// Waits until condition variable is notified and pred() == true or timeout is reached, otherwise\n    /// wait again.\n    ///\n    /// \\param timeout duration after which wait will exit with a timeout\n    /// \\param pred predicate that checks that awaited condition is true\n    ///\n    /// \\return a future that becomes ready when \\ref signal() is called\n    ///         If the condition variable was \\ref broken() will return \\ref broken_condition_variable\n    ///         exception. If timepoint is passed will return \\ref condition_variable_timed_out exception.\n    template<typename Rep, typename Period, typename Pred>\n    requires std::is_invocable_r_v<bool, Pred>\n    auto when(std::chrono::duration<Rep, Period> timeout, Pred&& pred) noexcept {\n        return when(timer<>::clock::now() + timeout, std::forward<Pred>(pred));\n    }\n\n    /// Whether or not the condition variable currently has pending waiter(s)\n    /// The returned answer is valid until next continuation/fiber switch.\n    bool has_waiters() const noexcept {\n        return !_waiters.empty();\n    }\n\n    /// Notify variable and wake up a waiter if there is one\n    void signal() noexcept;\n\n    /// Notify variable and wake up all waiter\n    void broadcast() noexcept;\n\n    /// Signal to waiters that an error occurred.  \\ref wait() will see\n    /// an exceptional future<> containing the provided exception parameter.\n    /// The future is made available immediately.\n    void broken() noexcept;\n\n    void broken(std::exception_ptr) noexcept;\n};\n\n/// @}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/coroutine.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n#pragma once\n\n\n#include <seastar/core/future.hh>\n#include <seastar/core/make_task.hh>\n#include <seastar/coroutine/exception.hh>\n#include <seastar/util/std-compat.hh>\n\n\n#include <coroutine>\n#include <new>\n#include <cstdlib>\n#include <source_location>\n\n#if !(__GLIBC__ == 2 && __GLIBC_MINOR__ >= 43) && !(__GLIBC__ > 2)\n\nextern \"C\" {\n\nvoid free_sized(void* ptr, size_t size);\n\n}\n\n#endif\n\nnamespace seastar {\n\nnamespace internal {\n\n\ninline\nvoid\nexecute_involving_handle_destruction_in_await_suspend(std::invocable<> auto&& func) noexcept {\n#if defined(__GNUC__) && !defined(__clang__) && (__GNUC__ > 15 || (__GNUC__ == 15 && __GNUC_MINOR__ >= 2))\n    memory::scoped_critical_alloc_section _;\n    schedule(new lambda_task(current_scheduling_group(), std::forward<decltype(func)>(func)));\n#else\n    std::invoke(std::forward<decltype(func)>(func));\n#endif\n}\n\ninline\nvoid\nexecute_involving_handle_destruction_in_await_suspend(task* tsk) noexcept {\n#if defined(__GNUC__) && !defined(__clang__) && (__GNUC__ > 15 || (__GNUC__ == 15 && __GNUC_MINOR__ >= 2))\n    schedule(tsk);\n#else\n    tsk->run_and_dispose();\n#endif\n}\n\nclass coroutine_allocators {\npublic:\n    static void* operator new(size_t size) {\n        memory::scoped_critical_alloc_section _;\n        return ::malloc(size);\n    }\n    static void operator delete(void* ptr) noexcept {\n        ::free(ptr);\n    }\n#ifdef __cpp_sized_deallocation\n    static void operator delete(void* ptr, std::size_t sz) noexcept {\n        ::free_sized(ptr, sz);\n    }\n#endif\n};\n\ntemplate <typename T = void>\nclass coroutine_traits_base {\npublic:\n    class promise_type final : public seastar::task, public coroutine_allocators {\n        seastar::promise<T> _promise;\n    public:\n        promise_type() = default;\n        promise_type(promise_type&&) = delete;\n        promise_type(const promise_type&) = delete;\n\n        template<typename... U>\n        void return_value(U&&... value) {\n            _promise.set_value(std::forward<U>(value)...);\n        }\n\n        void return_value(coroutine::exception ce) noexcept {\n            _promise.set_exception(std::move(ce.eptr));\n        }\n\n        void set_exception(std::exception_ptr&& eptr) noexcept {\n            _promise.set_exception(std::move(eptr));\n        }\n\n        [[deprecated(\"Forwarding coroutine returns are deprecated as too dangerous. Use 'co_return co_await ...' until explicit syntax is available.\")]]\n        void return_value(future<T>&& fut) noexcept {\n            fut.forward_to(std::move(_promise));\n        }\n\n        void unhandled_exception() noexcept {\n            _promise.set_exception(std::current_exception());\n        }\n\n        seastar::future<T> get_return_object() noexcept {\n            return _promise.get_future();\n        }\n\n        std::suspend_never initial_suspend() noexcept { return { }; }\n        std::suspend_never final_suspend() noexcept { return { }; }\n\n        virtual void run_and_dispose() noexcept override {\n            auto handle = std::coroutine_handle<promise_type>::from_promise(*this);\n            handle.resume();\n        }\n\n        task* waiting_task() noexcept override { return _promise.waiting_task(); }\n\n        scheduling_group set_scheduling_group(scheduling_group sg) noexcept {\n            return std::exchange(this->_sg, sg);\n        }\n    };\n};\n\ntemplate <>\nclass coroutine_traits_base<> {\npublic:\n   class promise_type final : public seastar::task, public coroutine_allocators {\n        seastar::promise<> _promise;\n    public:\n        promise_type() = default;\n        promise_type(promise_type&&) = delete;\n        promise_type(const promise_type&) = delete;\n\n        void return_void() noexcept {\n            _promise.set_value();\n        }\n\n        void set_exception(std::exception_ptr&& eptr) noexcept {\n            _promise.set_exception(std::move(eptr));\n        }\n\n        void unhandled_exception() noexcept {\n            _promise.set_exception(std::current_exception());\n        }\n\n        seastar::future<> get_return_object() noexcept {\n            return _promise.get_future();\n        }\n\n        std::suspend_never initial_suspend() noexcept { return { }; }\n        std::suspend_never final_suspend() noexcept { return { }; }\n\n        virtual void run_and_dispose() noexcept override {\n            auto handle = std::coroutine_handle<promise_type>::from_promise(*this);\n            handle.resume();\n        }\n\n        task* waiting_task() noexcept override { return _promise.waiting_task(); }\n\n        scheduling_group set_scheduling_group(scheduling_group new_sg) noexcept {\n            return task::set_scheduling_group(new_sg);\n        }\n    };\n};\n\ntemplate<bool CheckPreempt, typename T>\nstruct awaiter {\n    seastar::future<T> _future;\npublic:\n    explicit awaiter(seastar::future<T>&& f) noexcept : _future(std::move(f)) { }\n\n    awaiter(const awaiter&) = delete;\n    awaiter(awaiter&&) = delete;\n\n    bool await_ready() const noexcept {\n        return _future.available() && (!CheckPreempt || !need_preempt());\n    }\n\n    template<typename U>\n    void await_suspend(std::coroutine_handle<U> hndl SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(hndl.promise());\n        if (!CheckPreempt || !_future.available()) {\n            _future.set_coroutine(hndl.promise());\n        } else {\n            schedule(&hndl.promise());\n        }\n    }\n\n    T await_resume() { return _future.get(); }\n};\n\ntemplate<bool CheckPreempt>\nstruct awaiter<CheckPreempt, void> {\n    seastar::future<> _future;\npublic:\n    explicit awaiter(seastar::future<>&& f) noexcept : _future(std::move(f)) { }\n\n    awaiter(const awaiter&) = delete;\n    awaiter(awaiter&&) = delete;\n\n    bool await_ready() const noexcept {\n        return _future.available() && (!CheckPreempt || !need_preempt());\n    }\n\n    template<typename U>\n    void await_suspend(std::coroutine_handle<U> hndl SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(hndl.promise());\n        if (!CheckPreempt || !_future.available()) {\n            _future.set_coroutine(hndl.promise());\n        } else {\n            schedule(&hndl.promise());\n        }\n    }\n\n    void await_resume() { _future.get(); }\n};\n\n} // seastar::internal\n\n\ntemplate<typename T>\nauto operator co_await(future<T>&& f) noexcept {\n    return internal::awaiter<true, T>(std::move(f));\n}\n\nnamespace coroutine {\n/// Wrapper for a future which turns off checking for preemption\n/// when awaiting it in a coroutine.\n/// If constructed from a future, co_await-ing it will bypass\n/// checking if the task quota is depleted, which means that\n/// a ready future will be handled immediately.\ntemplate<typename T> struct [[nodiscard]] without_preemption_check : public seastar::future<T> {\n    explicit without_preemption_check(seastar::future<T>&& f) noexcept : seastar::future<T>(std::move(f)) {}\n};\n\n/// Make a lambda coroutine safe for use in an outer coroutine with\n/// functions that accept continuations.\n///\n/// A lambda coroutine is not a safe parameter to a function that expects\n/// a regular Seastar continuation.\n///\n/// To use, wrap the lambda coroutine in seastar::coroutine::lambda(). The\n/// lambda coroutine must complete (co_await) in the same statement.\n///\n/// Example::\n/// ```\n///    // `future::then()` expects a continuation, so not safe for lambda\n///    // coroutines without seastar::coroutine::lambda.\n///    co_await seastar::yield().then(seastar::coroutine::lambda([captures] () -> future<> {\n///        co_await seastar::coroutine::maybe_yield();\n///        // use of `captures` here can break without seastar::coroutine::lambda.\n///    }));\n/// ```\n///\n/// \\tparam Func type of function object (typically inferred)\ntemplate <typename Func>\nclass lambda {\n    Func* _func;\npublic:\n    /// Create a lambda coroutine wrapper from a function object, to be passed\n    /// to a Seastar function that accepts a continuation.\n    explicit lambda(Func&& func) : _func(&func) {}\n    /// Calls the lambda coroutine object. Normally invoked by Seastar.\n    template <typename... Args>\n    decltype(auto) operator()(Args&&... args) const {\n        return std::invoke(*_func, std::forward<Args>(args)...);\n    }\n};\n\n}\n\n/// Wait for a future without a preemption check\n///\n/// \\param f a \\c future<> wrapped with \\c without_preemption_check\ntemplate<typename T>\nauto operator co_await(coroutine::without_preemption_check<T> f) noexcept {\n    return internal::awaiter<false, T>(std::move(f));\n}\n\n\n} // seastar\n\n\nnamespace std {\n\ntemplate<typename T, typename... Args>\nclass coroutine_traits<seastar::future<T>, Args...> : public seastar::internal::coroutine_traits_base<T> {\n};\n\n} // std\n\n"
  },
  {
    "path": "include/seastar/core/deleter.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cassert>\n#include <cstdint>\n#include <cstdlib>\n#include <new>\n#include <utility>\n\nnamespace seastar {\n\n/// \\addtogroup memory-module\n/// @{\n\n/// Provides a mechanism for managing the lifetime of a buffer.\n///\n/// A \\c deleter is an object that is used to inform the consumer\n/// of some buffer (not referenced by the deleter itself) how to\n/// delete the buffer.  This can be by calling an arbitrary function\n/// or destroying an object carried by the deleter.  Examples of\n/// a deleter's encapsulated actions are:\n///\n///  - calling \\c std::free(p) on some captured pointer, p\n///  - calling \\c delete \\c p on some captured pointer, p\n///  - decrementing a reference count somewhere\n///\n/// A deleter performs its action from its destructor.\nclass deleter final {\npublic:\n    /// \\cond internal\n    struct impl;\n    struct raw_object_tag {};\n    /// \\endcond\nprivate:\n    // if bit 0 set, point to object to be freed directly.\n    impl* _impl = nullptr;\npublic:\n    /// Constructs an empty deleter that does nothing in its destructor.\n    deleter() noexcept = default;\n    deleter(const deleter&) = delete;\n    /// Moves a deleter.\n    deleter(deleter&& x) noexcept : _impl(x._impl) { x._impl = nullptr; }\n    /// \\cond internal\n    explicit deleter(impl* i) noexcept : _impl(i) {}\n    deleter(raw_object_tag, void* object) noexcept\n        : _impl(from_raw_object(object)) {}\n    /// \\endcond\n    /// Destroys the deleter and carries out the encapsulated action.\n    ~deleter();\n    deleter& operator=(deleter&& x) noexcept;\n    deleter& operator=(deleter&) = delete;\n    /// Performs a sharing operation.  The encapsulated action will only\n    /// be carried out after both the original deleter and the returned\n    /// deleter are both destroyed.\n    ///\n    /// \\return a deleter with the same encapsulated action as this one.\n    deleter share();\n    /// Checks whether the deleter has an associated action.\n    explicit operator bool() const noexcept { return bool(_impl); }\n    /// \\cond internal\n    void reset(impl* i) {\n        this->~deleter();\n        new (this) deleter(i);\n    }\n    /// \\endcond\n    /// Appends another deleter to this deleter.  When this deleter is\n    /// destroyed, both encapsulated actions will be carried out.\n    void append(deleter d);\nprivate:\n    static bool is_raw_object(impl* i) noexcept {\n        auto x = reinterpret_cast<uintptr_t>(i);\n        return x & 1;\n    }\n    bool is_raw_object() const noexcept {\n        return is_raw_object(_impl);\n    }\n    static void* to_raw_object(impl* i) noexcept {\n        auto x = reinterpret_cast<uintptr_t>(i);\n        return reinterpret_cast<void*>(x & ~uintptr_t(1));\n    }\n    void* to_raw_object() const noexcept {\n        return to_raw_object(_impl);\n    }\n    impl* from_raw_object(void* object) noexcept {\n        auto x = reinterpret_cast<uintptr_t>(object);\n        return reinterpret_cast<impl*>(x | 1);\n    }\n};\n\n/// \\cond internal\nstruct deleter::impl {\n    unsigned refs = 1;\n    deleter next;\n    impl(deleter next) : next(std::move(next)) {}\n    virtual ~impl() {}\n};\n/// \\endcond\n\ninline\ndeleter::~deleter() {\n    if (is_raw_object()) {\n        std::free(to_raw_object());\n        return;\n    }\n    if (_impl && --_impl->refs == 0) {\n        delete _impl;\n    }\n}\n\ninline\ndeleter& deleter::operator=(deleter&& x) noexcept {\n    if (this != &x) {\n        this->~deleter();\n        new (this) deleter(std::move(x));\n    }\n    return *this;\n}\n\n/// \\cond internal\ntemplate <typename Deleter>\nstruct lambda_deleter_impl final : deleter::impl {\n    Deleter del;\n    lambda_deleter_impl(deleter next, Deleter&& del)\n        : impl(std::move(next)), del(std::move(del)) {}\n    virtual ~lambda_deleter_impl() override { del(); }\n};\n\ntemplate <typename Object>\nstruct object_deleter_impl final : deleter::impl {\n    Object obj;\n    object_deleter_impl(deleter next, Object&& obj)\n        : impl(std::move(next)), obj(std::move(obj)) {}\n};\n\ntemplate <typename Object>\ninline\nobject_deleter_impl<Object>* make_object_deleter_impl(deleter next, Object obj) {\n    return new object_deleter_impl<Object>(std::move(next), std::move(obj));\n}\n/// \\endcond\n\n\n/// Makes a \\ref deleter that encapsulates the action of\n/// destroying an object, as well as running another deleter.  The input\n/// object is moved to the deleter, and destroyed when the deleter is destroyed.\n///\n/// \\param next deleter that will become part of the new deleter's encapsulated action\n/// \\param o object whose destructor becomes part of the new deleter's encapsulated action\n/// \\related deleter\ntemplate <typename Object>\ndeleter\nmake_deleter(deleter next, Object o) {\n    return deleter(new lambda_deleter_impl<Object>(std::move(next), std::move(o)));\n}\n\n/// Makes a \\ref deleter that encapsulates the action of destroying an object.  The input\n/// object is moved to the deleter, and destroyed when the deleter is destroyed.\n///\n/// \\param o object whose destructor becomes the new deleter's encapsulated action\n/// \\related deleter\ntemplate <typename Object>\ndeleter\nmake_deleter(Object o) {\n    return make_deleter(deleter(), std::move(o));\n}\n\n/// \\cond internal\nstruct free_deleter_impl final : deleter::impl {\n    void* obj;\n    free_deleter_impl(void* obj) : impl(deleter()), obj(obj) {}\n    free_deleter_impl(const free_deleter_impl&) = delete;\n    free_deleter_impl(free_deleter_impl&&) = delete;\n    virtual ~free_deleter_impl() override { std::free(obj); }\n};\n/// \\endcond\n\ninline\ndeleter\ndeleter::share() {\n    if (!_impl) {\n        return deleter();\n    }\n    if (is_raw_object()) {\n        _impl = new free_deleter_impl(to_raw_object());\n    }\n    ++_impl->refs;\n    return deleter(_impl);\n}\n\n// Appends 'd' to the chain of deleters. Avoids allocation if possible. For\n// performance reasons the current chain should be shorter and 'd' should be\n// longer.\ninline\nvoid deleter::append(deleter d) {\n    if (!d._impl) {\n        return;\n    }\n    impl* next_impl = _impl;\n    deleter* next_d = this;\n    while (next_impl) {\n        if (next_impl == d._impl) {\n            return; // Already appended\n        }\n        if (is_raw_object(next_impl)) {\n            next_d->_impl = next_impl = new free_deleter_impl(to_raw_object(next_impl));\n        }\n\n        if (next_impl->refs != 1) {\n            next_d->_impl = next_impl = make_object_deleter_impl(deleter(next_impl), std::move(d));\n            return;\n        }\n\n        next_d = &next_impl->next;\n        next_impl = next_d->_impl;\n    }\n    next_d->_impl = d._impl;\n    d._impl = nullptr;\n}\n\n/// Makes a deleter that calls \\c std::free() when it is destroyed.\n///\n/// \\param obj object to free.\n/// \\related deleter\ninline\ndeleter\nmake_free_deleter(void* obj) {\n    if (!obj) {\n        return deleter();\n    }\n    return deleter(deleter::raw_object_tag(), obj);\n}\n\n/// Makes a deleter that calls \\c std::free() when it is destroyed, as well\n/// as invoking the encapsulated action of another deleter.\n///\n/// \\param next deleter to invoke.\n/// \\param obj object to free.\n/// \\related deleter\ninline\ndeleter\nmake_free_deleter(deleter next, void* obj) {\n    return make_deleter(std::move(next), [obj] () mutable { std::free(obj); });\n}\n\n/// \\see make_deleter(Object)\n/// \\related deleter\ntemplate <typename T>\ninline\ndeleter\nmake_object_deleter(T&& obj) {\n    return deleter{make_object_deleter_impl(deleter(), std::move(obj))};\n}\n\n/// \\see make_deleter(deleter, Object)\n/// \\related deleter\ntemplate <typename T>\ninline\ndeleter\nmake_object_deleter(deleter d, T&& obj) {\n    return deleter{make_object_deleter_impl(std::move(d), std::move(obj))};\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/disk_params.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2025 ScyllaDB\n */\n\n#pragma once\n\n#include <boost/range/adaptor/map.hpp>\n#include <chrono>\n\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/reactor_config.hh>\n\nnamespace seastar {\n\nstruct smp_options;\n\nnamespace internal {\n\nstruct disk_params {\n    std::vector<std::string> mountpoints;\n    std::vector<dev_t> devices;\n    uint64_t read_bytes_rate = std::numeric_limits<uint64_t>::max();\n    uint64_t write_bytes_rate = std::numeric_limits<uint64_t>::max();\n    uint64_t read_req_rate = std::numeric_limits<uint64_t>::max();\n    uint64_t write_req_rate = std::numeric_limits<uint64_t>::max();\n    uint64_t read_saturation_length = std::numeric_limits<uint64_t>::max();\n    uint64_t write_saturation_length = std::numeric_limits<uint64_t>::max();\n    std::optional<uint32_t> physical_block_size; // Override for disks that lie about their physical block size\n    bool duplex = false;\n    float rate_factor = 1.0;\n};\n\nclass disk_config_params {\nprivate:\n    const unsigned _max_queues;\n    unsigned _num_io_groups = 0;\n    std::unordered_map<unsigned, disk_params> _disks;\n    std::chrono::duration<double> _latency_goal;\n    std::chrono::milliseconds _stall_threshold;\n    double _flow_ratio_backpressure_threshold;\n\npublic:\n    explicit disk_config_params(unsigned max_queues) noexcept\n            : _max_queues(max_queues)\n    {}\n\n    uint64_t per_io_group(uint64_t qty, unsigned nr_groups) const noexcept {\n        return std::max(qty / nr_groups, 1ul);\n    }\n\n    unsigned num_io_groups() const noexcept { return _num_io_groups; }\n\n    std::chrono::duration<double> latency_goal() const {\n        return _latency_goal;\n    }\n\n    std::chrono::milliseconds stall_threshold() const {\n        return _stall_threshold;\n    }\n\n    double latency_goal_opt(const reactor_options& opts) const {\n        return opts.io_latency_goal_ms ?\n                opts.io_latency_goal_ms.get_value() :\n                opts.task_quota_ms.get_value() * 1.5;\n    }\n\n    void parse_config(const smp_options& smp_opts, const reactor_options& reactor_opts);\n\n    struct io_queue::config generate_config(unsigned q, unsigned nr_groups) const;\n    struct io_queue::config generate_config(const disk_params& p, unsigned q, unsigned nr_groups) const;\n\n    auto queue_ids() {\n        return boost::adaptors::keys(_disks);\n    }\n\n    const std::vector<dev_t>& queue_devices(unsigned q) const {\n        return _disks.at(q).devices;\n    }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/distributed.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sharded.hh>\n\nnamespace seastar {\n\n\ntemplate <typename Service>\nusing distributed [[deprecated(\"Use sharded<T> from sharded.hh instead\")]] = sharded<Service>;\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/do_with.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <utility>\n#include <memory>\n#include <tuple>\n\nnamespace seastar {\n\n\n/// \\cond internal\n\nnamespace internal {\n\ntemplate <typename HeldState, typename Future>\nclass do_with_state final : public continuation_base_from_future<Future>::type {\n    HeldState _held;\n    typename Future::promise_type _pr;\npublic:\n    template<typename... T>\n    explicit do_with_state(T&&... args) : _held(std::forward<T>(args)...) {}\n    virtual void run_and_dispose() noexcept override {\n        _pr.set_urgent_state(std::move(this->_state));\n        delete this;\n    }\n    task* waiting_task() noexcept override {\n        return _pr.waiting_task();\n    }\n    HeldState& data() {\n        return _held;\n    }\n    Future get_future() {\n        return _pr.get_future();\n    }\n};\n\n}\n/// \\endcond\n\nnamespace internal {\ntemplate <typename Tuple, size_t... Idx>\ninline\nauto\ncherry_pick_tuple(std::index_sequence<Idx...>, Tuple&& tuple) {\n    return std::forward_as_tuple(std::get<Idx>(std::forward<Tuple>(tuple))...);\n}\n\ntemplate <typename Tuple, typename Seq>\nstruct subtuple;\n\ntemplate <typename Tuple, size_t... Idx>\nstruct subtuple<Tuple, std::index_sequence<Idx...>> {\n    using type = std::tuple<std::decay_t<std::tuple_element_t<Idx, Tuple>>...>;\n};\n\ntemplate <typename T1, typename T2, typename... More>\ninline\nauto\ndo_with_impl(T1&& rv1, T2&& rv2, More&&... more) {\n    auto all = std::forward_as_tuple(\n            std::forward<T1>(rv1),\n            std::forward<T2>(rv2),\n            std::forward<More>(more)...);\n    constexpr size_t nr = std::tuple_size<decltype(all)>::value - 1;\n    using idx = std::make_index_sequence<nr>;\n    auto&& just_values = cherry_pick_tuple(idx(), std::move(all));\n    auto&& just_func = std::move(std::get<nr>(std::move(all)));\n    using value_tuple = typename subtuple<decltype(all), idx>::type;\n    using ret_type = decltype(std::apply(just_func, std::declval<value_tuple&>()));\n    auto task = std::apply(\n        [](auto&&... x) {\n            return std::make_unique<internal::do_with_state<value_tuple, ret_type>>(std::forward<decltype(x)>(x)...);\n        },\n        std::move(just_values));\n    auto fut = std::apply(just_func, task->data());\n    if (fut.available()) {\n        return fut;\n    }\n    auto ret = task->get_future();\n    internal::set_callback(std::move(fut), task.release());\n    return ret;\n}\n}\n\n\n\n/// \\addtogroup future-util\n/// @{\n\n/// do_with() holds a objects alive until a future completes, and\n/// allow the code involved in making the future complete to have easy\n/// access to this object.\n///\n/// do_with() takes multiple arguments: The last is a function\n/// returning a future. The other are temporary objects (rvalue). The\n/// function is given (a moved copy of) these temporary object, by\n/// reference, and it is ensured that the objects will not be\n/// destructed until the completion of the future returned by the\n/// function.\n///\n/// do_with() returns a future which resolves to whatever value the given future\n/// (returned by the given function) resolves to. This returned value must not\n/// contain references to the temporary object, as at that point the temporary\n/// is destructed.\n///\n/// \\return whatever the function returns\ntemplate <typename T1, typename T2, typename... More>\ninline\nauto\ndo_with(T1&& rv1, T2&& rv2, More&&... more) noexcept {\n    auto func = internal::do_with_impl<T1, T2, More...>;\n    return futurize_invoke(func, std::forward<T1>(rv1), std::forward<T2>(rv2), std::forward<More>(more)...);\n}\n\n/// Executes the function \\c func making sure the lock \\c lock is taken,\n/// and later on properly released.\n///\n/// \\param lock the lock, which is any object having providing a lock() / unlock() semantics.\n///        Caller must make sure that it outlives \\c func.\n/// \\param func function to be executed\n/// \\returns whatever \\c func returns\ntemplate<typename Lock, typename Func>\ninline\nauto with_lock(Lock& lock, Func&& func) {\n    return lock.lock().then([&lock, func = std::forward<Func>(func)] () mutable {\n        return futurize_invoke(func).finally([&lock] {\n            lock.unlock();\n        });\n    });\n}\n\n/// @}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/dpdk_rte.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n#pragma once\n\n#ifdef SEASTAR_HAVE_DPDK\n\n#include <bitset>\n#include <optional>\n#include <rte_config.h>\n#include <rte_ethdev.h>\n#include <rte_version.h>\n\n/*********************** Compat section ***************************************/\n// We currently support only versions 2.0 and above.\n#if (RTE_VERSION < RTE_VERSION_NUM(2,0,0,0))\n#error \"DPDK version above 2.0.0 is required\"\n#endif\n\n#if defined(RTE_MBUF_REFCNT_ATOMIC)\n#warning \"CONFIG_RTE_MBUF_REFCNT_ATOMIC should be disabled in DPDK's \" \\\n         \"config/common_linuxapp\"\n#endif\n/******************************************************************************/\n\nnamespace seastar {\n\nnamespace dpdk {\n\n// DPDK Environment Abstraction Layer\nclass eal {\npublic:\n    using cpuset = std::bitset<RTE_MAX_LCORE>;\n\n    static void init(cpuset cpus, const std::string& argv0, const std::optional<std::string>& hugepages_path, bool dpdk_pmd);\n    /**\n     * Returns the amount of memory needed for DPDK\n     * @param num_cpus Number of CPUs the application is going to use\n     *\n     * @return\n     */\n    static size_t mem_size(int num_cpus, bool hugetlbfs_membackend = true);\n    static bool initialized;\n};\n\n} // namespace dpdk\n\n}\n\n#endif // SEASTAR_HAVE_DPDK\n"
  },
  {
    "path": "include/seastar/core/enum.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n/*\n * This header file defines a hash function for enum types, using the\n * standard hash function of the underlying type (such as int). This makes\n * it possible to inherit from this type to\n */\n\n#include <type_traits>\n#include <functional>\n#include <cstddef>\n\nnamespace seastar {\n\ntemplate <typename T>\nrequires std::is_enum_v<T>\nclass enum_hash {\npublic:\n    std::size_t operator()(const T& e) const {\n        using utype = std::underlying_type_t<T>;\n        return std::hash<utype>()(static_cast<utype>(e));\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/exception_hacks.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar {\nvoid init_phdr_cache();\n}\n"
  },
  {
    "path": "include/seastar/core/execution_stage.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/function_traits.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/util/reference_wrapper.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/tuple_utils.hh>\n#include <seastar/util/std-compat.hh>\n#include <fmt/format.h>\n#include <vector>\n#include <boost/container/static_vector.hpp>\n\nnamespace seastar {\n\n/// \\defgroup execution-stages Execution Stages\n///\n/// \\brief\n/// Execution stages provide an infrastructure for processing function calls in\n/// batches in order to improve instruction cache locality.\n///\n/// When the application logic becomes more and more complex and the length\n/// of the data processing pipeline grows it may happen that the most\n/// significant bottleneck are instruction cache misses. The solution for that\n/// problem may be processing similar operations in batches so that instruction\n/// cache locality is improved at the cost of potentially higher latencies and\n/// worse data cache locality.\n///\n/// Execution stages allow batching calls to the specified function object.\n/// Every time concrete_execution_stage::operator()() is used the function call\n/// is added to the queue and a future is returned. Once the number of queued\n/// calls reaches certain threshold the stage is flushed and a task is which\n/// would execute these function calls is scheduled. Execution stages are also\n/// flushed when the reactor polls for events.\n///\n/// When calling a function that is wrapped inside execution stage it is\n/// important to remember that the actual function call will happen at some\n/// later time and it has to be guaranteed the objects passed by lvalue\n/// reference are still alive. In order to avoid accidental passing of a\n/// temporary object by lvalue reference the interface of execution stages\n/// accepts only lvalue references wrapped in reference_wrapper. It is safe to\n/// pass rvalue references, they are decayed and the objects are moved. See\n/// concrete_execution_stage::operator()() for more details.\n\n/// \\addtogroup execution-stages\n/// @{\n\n/// \\cond internal\nnamespace internal {\n\n// Execution wraps lreferences in reference_wrapper so that the caller is forced\n// to use seastar::ref(). Then when the function is actually called the\n// reference is unwrapped. However, we need to distinguish between functions\n// which argument is lvalue reference and functions that take\n// reference_wrapper<> as an argument and not unwrap the latter. To solve this\n// issue reference_wrapper_for_es type is used for wrappings done automatically\n// by execution stage.\ntemplate<typename T>\nstruct reference_wrapper_for_es : reference_wrapper<T> {\n    reference_wrapper_for_es(reference_wrapper <T> rw) noexcept\n        : reference_wrapper<T>(std::move(rw)) {}\n};\n\ntemplate<typename T>\nstruct wrap_for_es {\n    using type = T;\n};\n\ntemplate<typename T>\nstruct wrap_for_es<T&> {\n    using type = reference_wrapper_for_es<T>;\n};\n\ntemplate<typename T>\nstruct wrap_for_es<T&&> {\n    using type = T;\n};\n\ntemplate<typename T>\ndecltype(auto) unwrap_for_es(T&& object) {\n    return std::forward<T>(object);\n}\n\ntemplate<typename T>\nstd::reference_wrapper<T> unwrap_for_es(reference_wrapper_for_es<T> ref) {\n    return std::reference_wrapper<T>(ref.get());\n}\n\n}\n/// \\endcond\n\n/// Base execution stage class\nclass execution_stage {\npublic:\n    struct stats {\n        uint64_t tasks_scheduled = 0;\n        uint64_t tasks_preempted = 0;\n        uint64_t function_calls_enqueued = 0;\n        uint64_t function_calls_executed = 0;\n    };\nprotected:\n    bool _empty = true;\n    bool _flush_scheduled = false;\n    scheduling_group _sg;\n    stats _stats;\n    sstring _name;\n    metrics::metric_group _metric_group;\nprotected:\n    virtual void do_flush() noexcept = 0;\npublic:\n    explicit execution_stage(const sstring& name, scheduling_group sg = {});\n    virtual ~execution_stage();\n\n    execution_stage(const execution_stage&) = delete;\n\n    /// Move constructor\n    ///\n    /// \\warning It is illegal to move execution_stage after any operation has\n    /// been pushed to it. The only reason why the move constructor is not\n    /// deleted is the fact that C++14 does not guarantee return value\n    /// optimisation which is required by make_execution_stage().\n    execution_stage(execution_stage&&);\n\n    /// Returns execution stage name\n    const sstring& name() const noexcept { return _name; }\n\n    /// Returns execution stage usage statistics\n    const stats& get_stats() const noexcept { return _stats; }\n\n    /// Flushes execution stage\n    ///\n    /// Ensures that a task which would execute all queued operations is\n    /// scheduled. Does not schedule a new task if there is one already pending\n    /// or the queue is empty.\n    ///\n    /// \\return true if a new task has been scheduled\n    bool flush() noexcept;\n\n    /// Checks whether there are pending operations.\n    ///\n    /// \\return true if there is at least one queued operation\n    bool poll() const noexcept {\n        return !_empty;\n    }\n\n    /// Returns execution stage scheduling_group\n    scheduling_group get_scheduling_group() const noexcept { return _sg; }\n\n    /// Updates execution stage name and metric_group\n    void update_name_and_metric_group() {\n        update_name();\n        update_metric_group();\n    }\n\nprivate:\n    virtual void update_name() { /* Do nothing in the default implementation */ };\n    void update_metric_group();\n};\n\n/// \\cond internal\nnamespace internal {\n\nclass execution_stage_manager {\n    std::vector<execution_stage*> _execution_stages;\n    std::unordered_map<sstring, execution_stage*> _stages_by_name;\nprivate:\n    execution_stage_manager() = default;\n    execution_stage_manager(const execution_stage_manager&) = delete;\n    execution_stage_manager(execution_stage_manager&&) = delete;\npublic:\n    void register_execution_stage(execution_stage& stage);\n    void unregister_execution_stage(execution_stage& stage) noexcept;\n    void update_execution_stage_registration(execution_stage& old_es, execution_stage& new_es) noexcept;\n    void update_scheduling_group_name(scheduling_group sg) noexcept;\n    execution_stage* get_stage(const sstring& name);\n    bool flush() noexcept;\n    bool poll() const noexcept;\npublic:\n    static execution_stage_manager& get() noexcept;\n};\n\ntemplate<typename ReturnType, typename... Args>\nrequires std::is_nothrow_move_constructible_v<std::tuple<Args...>>\nclass concrete_execution_stage_base : public execution_stage {\n    using args_tuple = std::tuple<Args...>;\n    static_assert(std::is_nothrow_move_constructible_v<args_tuple>,\n                  \"Function arguments need to be nothrow move constructible\");\n\n    static constexpr size_t flush_threshold = 128;\n    static constexpr size_t max_queue_length = 1024;\n\n    using return_type = futurize_t<ReturnType>;\n    using promise_type = typename return_type::promise_type;\n    using input_type = typename tuple_map_types<internal::wrap_for_es, args_tuple>::type;\n\n    struct work_item {\n        input_type _in;\n        promise_type _ready;\n\n        work_item(typename internal::wrap_for_es<Args>::type... args) : _in(std::move(args)...) { }\n\n        work_item(work_item&& other) = delete;\n        work_item(const work_item&) = delete;\n        work_item(work_item&) = delete;\n    };\n    chunked_fifo<work_item, flush_threshold> _queue;\n\n    noncopyable_function<ReturnType (Args...)> _function;\nprivate:\n    auto unwrap(input_type&& in) {\n        return tuple_map(std::move(in), [] (auto&& obj) {\n            return internal::unwrap_for_es(std::forward<decltype(obj)>(obj));\n        });\n    }\n\n    virtual void do_flush() noexcept override {\n        while (!_queue.empty()) {\n            auto& wi = _queue.front();\n            auto wi_in = std::move(wi._in);\n            auto wi_ready = std::move(wi._ready);\n            _queue.pop_front();\n            futurize<ReturnType>::apply(_function, unwrap(std::move(wi_in))).forward_to(std::move(wi_ready));\n            _stats.function_calls_executed++;\n\n            if (internal::scheduler_need_preempt()) {\n                _stats.tasks_preempted++;\n                break;\n            }\n        }\n        _empty = _queue.empty();\n    }\npublic:\n    explicit concrete_execution_stage_base(const sstring& name, scheduling_group sg, noncopyable_function<ReturnType (Args...)> f)\n        : execution_stage(name, sg)\n        , _function(std::move(f))\n    {\n        _queue.reserve(flush_threshold);\n    }\n    explicit concrete_execution_stage_base(const sstring& name, noncopyable_function<ReturnType (Args...)> f)\n        : concrete_execution_stage_base(name, scheduling_group(), std::move(f)) {\n    }\n\n    /// Enqueues a call to the stage's function\n    ///\n    /// Adds a function call to the queue. Objects passed by value are moved,\n    /// rvalue references are decayed and the objects are moved, lvalue\n    /// references need to be explicitly wrapped using seastar::ref().\n    ///\n    /// Usage example:\n    /// ```\n    /// void do_something(int&, int, std::vector<int>&&);\n    /// thread_local auto stage = seastar::make_execution_stage(\"execution-stage\", do_something);\n    ///\n    /// int global_value;\n    ///\n    /// future<> func(std::vector<int> vec) {\n    ///     //return stage(global_value, 42, std::move(vec)); // fail: use seastar::ref to pass references\n    ///     return stage(seastar::ref(global_value), 42, std::move(vec)); // ok\n    /// }\n    /// ```\n    ///\n    /// \\param args arguments passed to the stage's function\n    /// \\return future containing the result of the call to the stage's function\n    return_type operator()(typename internal::wrap_for_es<Args>::type... args) {\n        if (_queue.size() >= max_queue_length) {\n            do_flush();\n        }\n        _queue.emplace_back(std::move(args)...);\n        _empty = false;\n        _stats.function_calls_enqueued++;\n        auto f = _queue.back()._ready.get_future();\n        flush();\n        return f;\n    }\n};\n\n}\n/// \\endcond\n\n/// \\brief Concrete execution stage class\n///\n/// \\note The recommended way of creating execution stages is to use\n/// make_execution_stage().\n///\n/// \\tparam ReturnType return type of the function object\n/// \\tparam Args  argument pack containing arguments to the function object, needs\n///                   to have move constructor that doesn't throw\ntemplate<typename ReturnType, typename... Args>\nrequires std::is_nothrow_move_constructible_v<std::tuple<Args...>>\nclass concrete_execution_stage final : public internal::concrete_execution_stage_base<ReturnType, Args...> {\n    using internal::concrete_execution_stage_base<ReturnType, Args...>::concrete_execution_stage_base;\n};\n\n/// \\brief Base class for execution stages with support for automatic \\ref scheduling_group inheritance\nclass inheriting_execution_stage {\npublic:\n    struct per_scheduling_group_stats {\n        scheduling_group sg;\n        execution_stage::stats stats;\n    };\n    using stats = boost::container::static_vector<per_scheduling_group_stats, max_scheduling_groups()>;\n};\n\n/// \\brief Concrete execution stage class, with support for automatic \\ref scheduling_group inheritance\n///\n/// A variation of \\ref concrete_execution_stage that inherits the \\ref scheduling_group\n/// from the caller. Each call (of `operator()`) can be in its own scheduling group.\n///\n/// \\tparam ReturnType return type of the function object\n/// \\tparam Args  argument pack containing arguments to the function object, needs\n///                   to have move constructor that doesn't throw\ntemplate<typename ReturnType, typename... Args>\nrequires std::is_nothrow_move_constructible_v<std::tuple<Args...>>\nclass inheriting_concrete_execution_stage final : public inheriting_execution_stage {\n    using return_type = futurize_t<ReturnType>;\n    using args_tuple = std::tuple<Args...>;\n\n    class per_group_stage_type final : public internal::concrete_execution_stage_base<ReturnType, Args...> {\n        static sstring format_name(const sstring& name, scheduling_group sg) {\n            return fmt::format(\"{}.{}\", name, sg.name());\n        }\n        sstring _base_name;\n    public:\n        per_group_stage_type(const sstring& base_name, scheduling_group sg, noncopyable_function<ReturnType (Args...)> f)\n            : internal::concrete_execution_stage_base<ReturnType, Args...>(per_group_stage_type::format_name(base_name, sg), sg, std::move(f))\n            , _base_name(base_name)\n        { }\n\n        per_group_stage_type(const sstring& name, noncopyable_function<ReturnType (Args...)> f)\n            : per_group_stage_type(name, scheduling_group(), std::move(f)) {\n        }\n\n        void update_name() override {\n            this->_name = per_group_stage_type::format_name(_base_name, this->_sg);\n        }\n    };\n\n    static_assert(std::is_nothrow_move_constructible_v<args_tuple>,\n                  \"Function arguments need to be nothrow move constructible\");\n\n    sstring _name;\n    noncopyable_function<ReturnType (Args...)> _function;\n    std::vector<std::optional<per_group_stage_type>> _stage_for_group{max_scheduling_groups()};\nprivate:\n    per_group_stage_type make_stage_for_group(scheduling_group sg) {\n        // We can't use std::ref(function), because reference_wrapper decays to noncopyable_function& and\n        // that selects the noncopyable_function copy constructor. Use a lambda instead.\n        auto wrapped_function = [&_function = _function] (Args... args) {\n            return _function(std::forward<Args>(args)...);\n        };\n        return per_group_stage_type(_name, sg, wrapped_function);\n    }\npublic:\n    /// Construct an inheriting concrete execution stage.\n    ///\n    /// \\param name A name for the execution stage; must be unique\n    /// \\param f Function to be called in response to operator(). The function\n    ///        call will be deferred and batched with similar calls to increase\n    ///        instruction cache hit rate.\n    inheriting_concrete_execution_stage(const sstring& name, noncopyable_function<ReturnType (Args...)> f)\n        : _name(std::move(name)),_function(std::move(f)) {\n    }\n\n    /// Enqueues a call to the stage's function\n    ///\n    /// Adds a function call to the queue. Objects passed by value are moved,\n    /// rvalue references are decayed and the objects are moved, lvalue\n    /// references need to be explicitly wrapped using seastar::ref().\n    ///\n    /// The caller's \\ref scheduling_group will be preserved across the call.\n    ///\n    /// Usage example:\n    /// ```\n    /// void do_something(int);\n    /// thread_local auto stage = seastar::inheriting_concrete_execution_stage<int>(\"execution-stage\", do_something);\n    ///\n    /// future<> func(int x) {\n    ///     return stage(x);\n    /// }\n    /// ```\n    ///\n    /// \\param args arguments passed to the stage's function\n    /// \\return future containing the result of the call to the stage's function\n    return_type operator()(typename internal::wrap_for_es<Args>::type... args) {\n        auto sg = current_scheduling_group();\n        auto sg_id = internal::scheduling_group_index(sg);\n        auto& slot = _stage_for_group[sg_id];\n        if (!slot) {\n            slot.emplace(make_stage_for_group(sg));\n        }\n        return (*slot)(std::move(args)...);\n    }\n\n    /// Returns summary of individual execution stage usage statistics\n    ///\n    /// \\returns a vector of the stats of the individual per-scheduling group\n    ///     executation stages. Each element in the vector is a pair composed of\n    ///     the scheduling group and the stats for the respective execution\n    ///     stage. Scheduling groups that have had no respective calls enqueued\n    ///     yet are omitted.\n    inheriting_execution_stage::stats get_stats() const noexcept {\n        inheriting_execution_stage::stats summary;\n        for (unsigned sg_id = 0; sg_id != _stage_for_group.size(); ++sg_id) {\n            auto sg = internal::scheduling_group_from_index(sg_id);\n            if (_stage_for_group[sg_id]) {\n                summary.push_back({sg, _stage_for_group[sg_id]->get_stats()});\n            }\n        }\n        return summary;\n    }\n};\n\n\n/// \\cond internal\nnamespace internal {\n\ntemplate <typename Ret, typename ArgsTuple>\nstruct concrete_execution_stage_helper;\n\ntemplate <typename Ret, typename... Args>\nstruct concrete_execution_stage_helper<Ret, std::tuple<Args...>> {\n    using type = concrete_execution_stage<Ret, Args...>;\n};\n\n}\n/// \\endcond\n\n/// Creates a new execution stage\n///\n/// Wraps given function object in a concrete_execution_stage. All arguments\n/// of the function object are required to have move constructors that do not\n/// throw. Function object may return a future or an immediate object or void.\n///\n/// Moving execution stages is discouraged and illegal after first function\n/// call is enqueued.\n///\n/// Usage example:\n/// ```\n/// double do_something(int);\n/// thread_local auto stage1 = seastar::make_execution_stage(\"execution-stage1\", do_something);\n///\n/// future<double> func1(int val) {\n///     return stage1(val);\n/// }\n///\n/// future<double> do_some_io(int);\n/// thread_local auto stage2 = seastar::make_execution_stage(\"execution-stage2\", do_some_io);\n///\n/// future<double> func2(int val) {\n///     return stage2(val);\n/// }\n/// ```\n///\n/// \\param name unique name of the execution stage\n/// \\param sg scheduling group to run under\n/// \\param fn function to be executed by the stage\n/// \\return concrete_execution_stage\n///\ntemplate<typename Function>\nauto make_execution_stage(const sstring& name, scheduling_group sg, Function&& fn) {\n    using traits = function_traits<Function>;\n    using ret_type = typename traits::return_type;\n    using args_as_tuple = typename traits::args_as_tuple;\n    using concrete_execution_stage = typename internal::concrete_execution_stage_helper<ret_type, args_as_tuple>::type;\n    return concrete_execution_stage(name, sg, std::forward<Function>(fn));\n}\n\n/// Creates a new execution stage (variant taking \\ref scheduling_group)\n///\n/// Wraps given function object in a concrete_execution_stage. All arguments\n/// of the function object are required to have move constructors that do not\n/// throw. Function object may return a future or an immediate object or void.\n///\n/// Moving execution stages is discouraged and illegal after first function\n/// call is enqueued.\n///\n/// Usage example:\n/// ```\n/// double do_something(int);\n/// thread_local auto stage1 = seastar::make_execution_stage(\"execution-stage1\", do_something);\n///\n/// future<double> func1(int val) {\n///     return stage1(val);\n/// }\n///\n/// future<double> do_some_io(int);\n/// thread_local auto stage2 = seastar::make_execution_stage(\"execution-stage2\", do_some_io);\n///\n/// future<double> func2(int val) {\n///     return stage2(val);\n/// }\n/// ```\n///\n/// \\param name unique name of the execution stage (variant not taking \\ref scheduling_group)\n/// \\param fn function to be executed by the stage\n/// \\return concrete_execution_stage\n///\ntemplate<typename Function>\nauto make_execution_stage(const sstring& name, Function&& fn) {\n    return make_execution_stage(name, scheduling_group(), std::forward<Function>(fn));\n}\n\n/// Creates a new execution stage from a member function\n///\n/// Wraps a pointer to member function in a concrete_execution_stage. When\n/// a function call is pushed to the stage the first argument should be a\n/// pointer to the object the function is a member of.\n///\n/// Usage example:\n/// ```\n/// struct foo {\n///     void do_something(int);\n/// };\n///\n/// thread_local auto stage = seastar::make_execution_stage(\"execution-stage\", &foo::do_something);\n///\n/// future<> func(foo& obj, int val) {\n///     return stage(&obj, val);\n/// }\n/// ```\n///\n/// \\see make_execution_stage(const sstring&, Function&&)\n/// \\param name unique name of the execution stage\n/// \\param fn member function to be executed by the stage\n/// \\return concrete_execution_stage\ntemplate<typename Ret, typename Object, typename... Args>\nconcrete_execution_stage<Ret, Object*, Args...>\nmake_execution_stage(const sstring& name, scheduling_group sg, Ret (Object::*fn)(Args...)) {\n    return concrete_execution_stage<Ret, Object*, Args...>(name, sg, std::mem_fn(fn));\n}\n\ntemplate<typename Ret, typename Object, typename... Args>\nconcrete_execution_stage<Ret, const Object*, Args...>\nmake_execution_stage(const sstring& name, scheduling_group sg, Ret (Object::*fn)(Args...) const) {\n    return concrete_execution_stage<Ret, const Object*, Args...>(name, sg, std::mem_fn(fn));\n}\n\ntemplate<typename Ret, typename Object, typename... Args>\nconcrete_execution_stage<Ret, Object*, Args...>\nmake_execution_stage(const sstring& name, Ret (Object::*fn)(Args...)) {\n    return make_execution_stage(name, scheduling_group(), fn);\n}\n\ntemplate<typename Ret, typename Object, typename... Args>\nconcrete_execution_stage<Ret, const Object*, Args...>\nmake_execution_stage(const sstring& name, Ret (Object::*fn)(Args...) const) {\n    return make_execution_stage(name, scheduling_group(), fn);\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/expiring_fifo.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <exception>\n#include <memory>\n#include <seastar/core/timer.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\ntemplate<typename T>\nstruct dummy_expiry {\n    void operator()(T&) noexcept {};\n};\n\ntemplate<typename... T>\nstruct promise_expiry {\n    void operator()(promise<T...>& pr) noexcept {\n        pr.set_exception(std::make_exception_ptr(timed_out_error()));\n    };\n};\n\n/// Container for elements with support for expiration of entries.\n///\n/// OnExpiry is a functor which will be called with a reference to T right before it expires.\n/// T is removed and destroyed from the container immediately after OnExpiry returns.\n/// OnExpiry callback must not modify the container, it can only modify its argument.\n///\n/// The container can only be moved before any elements are pushed.\n///\ntemplate <typename T, typename OnExpiry = dummy_expiry<T>, typename Clock = lowres_clock>\nclass expiring_fifo {\npublic:\n    using clock = Clock;\n    using time_point = typename Clock::time_point;\nprivate:\n    struct entry {\n        std::optional<T> payload; // disengaged means that it's expired\n        timer<Clock> tr;\n        entry(T&& payload_) : payload(std::move(payload_)) {}\n        entry(const T& payload_) : payload(payload_) {}\n        entry(T payload_, expiring_fifo& ef, time_point timeout)\n                : payload(std::move(payload_))\n                , tr([this, &ef] {\n                    ef._on_expiry(*payload);\n                    payload = std::nullopt;\n                    --ef._size;\n                    ef.drop_expired_front();\n                })\n        {\n            tr.arm(timeout);\n        }\n        entry(entry&& x) = delete;\n        entry(const entry& x) = delete;\n    };\n\n    // If engaged, represents the first element.\n    // This is to avoid large allocations done by chunked_fifo for single-element cases.\n    // expiring_fifo is used to implement wait lists in synchronization primitives\n    // and in some uses it's common to have at most one waiter.\n    std::unique_ptr<entry> _front;\n\n    // There is an invariant that the front element is never expired.\n    chunked_fifo<entry> _list;\n    OnExpiry _on_expiry;\n    size_t _size = 0;\n\n    // Ensures that front() is not expired by dropping expired elements from the front.\n    void drop_expired_front() noexcept {\n        while (!_list.empty() && !_list.front().payload) {\n            _list.pop_front();\n        }\n        if (_front && !_front->payload) {\n            _front.reset();\n        }\n    }\npublic:\n    expiring_fifo() noexcept = default;\n    expiring_fifo(OnExpiry on_expiry) noexcept(std::is_nothrow_move_constructible_v<OnExpiry>) : _on_expiry(std::move(on_expiry)) {}\n\n    expiring_fifo(expiring_fifo&& o) noexcept\n            : expiring_fifo(std::move(o._on_expiry)) {\n        // entry objects hold a reference to this so non-empty containers cannot be moved.\n        SEASTAR_ASSERT(o._size == 0);\n    }\n\n    expiring_fifo& operator=(expiring_fifo&& o) noexcept {\n        if (this != &o) {\n            this->~expiring_fifo();\n            new (this) expiring_fifo(std::move(o));\n        }\n        return *this;\n    }\n\n    /// Checks if container contains any elements\n    ///\n    /// \\note Inside OnExpiry callback, the expired element is still contained.\n    ///\n    /// \\return true if and only if there are any elements contained.\n    bool empty() const noexcept {\n        return _size == 0;\n    }\n\n    /// Equivalent to !empty()\n    explicit operator bool() const noexcept {\n        return !empty();\n    }\n\n    /// Returns a reference to the element in the front.\n    /// Valid only when !empty().\n    T& front() noexcept {\n        if (_front) {\n            return *_front->payload;\n        }\n        return *_list.front().payload;\n    }\n\n    /// Returns a reference to the element in the front.\n    /// Valid only when !empty().\n    const T& front() const noexcept {\n        if (_front) {\n            return *_front->payload;\n        }\n        return *_list.front().payload;\n    }\n\n    /// Returns the number of elements contained.\n    ///\n    /// \\note Expired elements are not contained. Expiring element is still contained when OnExpiry is called.\n    size_t size() const noexcept {\n        return _size;\n    }\n\n    /// Reserves storage in the container for at least 'size' elements.\n    /// Note that expired elements may also take space when they are not in the front of the queue.\n    ///\n    /// Doesn't give any guarantees about exception safety of subsequent push_back().\n    void reserve(size_t size) {\n        return _list.reserve(size);\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will never expire.\n    void push_back(const T& payload) {\n        if (_size == 0) {\n            _front = std::make_unique<entry>(payload);\n        } else {\n            _list.emplace_back(payload);\n        }\n        ++_size;\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will never expire.\n    void push_back(T&& payload) {\n        if (_size == 0) {\n            _front = std::make_unique<entry>(std::move(payload));\n        } else {\n            _list.emplace_back(std::move(payload));\n        }\n        ++_size;\n    }\n\n    /// Adds element to the back of the queue.\n    /// The element will expire when timeout is reached, unless it is time_point::max(), in which\n    /// case it never expires.\n    void push_back(T&& payload, time_point timeout) {\n        if (timeout == time_point::max()) {\n            push_back(std::move(payload));\n            return;\n        }\n        if (_size == 0) {\n            _front = std::make_unique<entry>(std::move(payload), *this, timeout);\n        } else {\n            _list.emplace_back(std::move(payload), *this, timeout);\n        }\n        ++_size;\n    }\n\n    /// Removes the element at the front.\n    /// Can be called only if !empty().\n    void pop_front() noexcept {\n        if (_front) {\n            _front.reset();\n        } else {\n            _list.pop_front();\n        }\n        --_size;\n        drop_expired_front();\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/fair_queue.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n#pragma once\n\n#include <boost/intrusive/slist.hpp>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/util/assert.hh>\n\n#include <chrono>\n#include <cstdint>\n#include <functional>\n#include <optional>\n#include <queue>\n#include <fmt/ostream.h>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\nnamespace testing {\nclass fair_queue_test;\n}\n\n/// \\brief describes a request that passes through the \\ref fair_queue.\n///\n/// A ticket is specified by a \\c weight and a \\c size. For example, one can specify a request of \\c weight\n/// 1 and \\c size 16kB. If the \\ref fair_queue accepts one such request per second, it will sustain 1 IOPS\n/// at 16kB/s bandwidth.\n///\n/// \\related fair_queue\nclass fair_queue_ticket {\n    uint32_t _weight = 0; ///< the total weight of these requests for capacity purposes (IOPS).\n    uint32_t _size = 0;        ///< the total effective size of these requests\npublic:\n    /// Constructs a fair_queue_ticket with a given \\c weight and a given \\c size\n    ///\n    /// \\param weight the weight of the request\n    /// \\param size the size of the request\n    fair_queue_ticket(uint32_t weight, uint32_t size) noexcept;\n    fair_queue_ticket() noexcept {}\n    fair_queue_ticket operator+(fair_queue_ticket desc) const noexcept;\n    fair_queue_ticket operator-(fair_queue_ticket desc) const noexcept;\n    /// Increase the quantity represented in this ticket by the amount represented by \\c desc\n    /// \\param desc another \\ref fair_queue_ticket whose \\c weight \\c and size will be added to this one\n    fair_queue_ticket& operator+=(fair_queue_ticket desc) noexcept;\n    /// Decreases the quantity represented in this ticket by the amount represented by \\c desc\n    /// \\param desc another \\ref fair_queue_ticket whose \\c weight \\c and size will be decremented from this one\n    fair_queue_ticket& operator-=(fair_queue_ticket desc) noexcept;\n    /// Checks if the tickets fully equals to another one\n    /// \\param desc another \\ref fair_queue_ticket to compare with\n    bool operator==(const fair_queue_ticket& desc) const noexcept;\n\n    /// \\returns true if the fair_queue_ticket represents a non-zero quantity.\n    ///\n    /// For a fair_queue ticket to be non-zero, at least one of its represented quantities need to\n    /// be non-zero\n    explicit operator bool() const noexcept;\n    bool is_non_zero() const noexcept;\n\n    friend std::ostream& operator<<(std::ostream& os, fair_queue_ticket t);\n\n    /// \\returns the normalized value of this \\ref fair_queue_ticket along a base axis\n    ///\n    /// The normalization function itself is an implementation detail, but one can expect either weight or\n    /// size to have more or less relative importance depending on which of the dimensions in the\n    /// denominator is relatively higher. For example, given this request a, and two other requests\n    /// b and c, such that that c has the same \\c weight but a higher \\c size than b, one can expect\n    /// the \\c size component of this request to play a larger role.\n    ///\n    /// It is legal for the numerator to have one of the quantities set to zero, in which case only\n    /// the other quantity is taken into consideration.\n    ///\n    /// It is however not legal for the axis to have any quantity set to zero.\n    /// \\param axis another \\ref fair_queue_ticket to be used as a a base vector against which to normalize this fair_queue_ticket.\n    float normalize(fair_queue_ticket axis) const noexcept;\n\n    /*\n     * For both dimentions checks if the first rover is ahead of the\n     * second and returns the difference. If behind returns zero.\n     */\n    friend fair_queue_ticket wrapping_difference(const fair_queue_ticket& a, const fair_queue_ticket& b) noexcept;\n};\n\n/// \\addtogroup io-module\n/// @{\n\nclass fair_queue_entry {\npublic:\n    // The capacity_t represents tokens each entry needs to get dispatched, in\n    // a 'normalized' form -- converted from floating-point to fixed-point number\n    // and scaled accrding to fair-group's token-bucket duration\n    using capacity_t = uint64_t;\n    friend class fair_queue;\n\nprivate:\n    capacity_t _capacity;\n    bi::slist_member_hook<> _hook;\n\npublic:\n    explicit fair_queue_entry(capacity_t c) noexcept\n        : _capacity(c) {}\n    using container_list_t = bi::slist<fair_queue_entry,\n            bi::constant_time_size<false>,\n            bi::cache_last<true>,\n            bi::member_hook<fair_queue_entry, bi::slist_member_hook<>, &fair_queue_entry::_hook>>;\n\n    capacity_t capacity() const noexcept { return _capacity; }\n};\n\n/// \\brief Fair queuing class\n///\n/// This is a fair queue, allowing multiple request producers to queue requests\n/// that will then be served proportionally to their classes' shares.\n///\n/// To each request, a weight can also be associated. A request of weight 1 will consume\n/// 1 share. Higher weights for a request will consume a proportionally higher amount of\n/// shares.\n///\n/// The user of this interface is expected to register multiple `priority_class_data`\n/// objects, which will each have a shares attribute.\n///\n/// Internally, each priority class may keep a separate queue of requests.\n/// Requests pertaining to a class can go through even if they are over its\n/// share limit, provided that the other classes have empty queues.\n///\n/// When the classes that lag behind start seeing requests, the fair queue will serve\n/// them first, until balance is restored. This balancing is expected to happen within\n/// a certain time window that obeys an exponential decay.\nclass fair_queue {\npublic:\n    /// \\brief Fair Queue configuration structure.\n    ///\n    /// \\sets the operation parameters of a \\ref fair_queue\n    /// \\related fair_queue\n    struct config {\n        sstring label = \"\";\n        uint64_t forgiving_factor = 0;\n    };\n\n    using class_id = unsigned int;\n    using capacity_t = fair_queue_entry::capacity_t;\n    using signed_capacity_t = std::make_signed_t<capacity_t>;\n\nprivate:\n    class priority_class_group_data;\n\n    class priority_entry {\n        friend class fair_queue;\n        friend testing::fair_queue_test;\n    protected:\n        uint32_t _shares = 0;\n        capacity_t _accumulated = 0;\n        bool _queued = false;\n        uint32_t _activations = 0;\n        priority_class_group_data* _parent = nullptr;\n        bool _plugged = true;\n\n        void unplug() noexcept;\n\n        priority_entry(uint32_t shares, priority_class_group_data* p) noexcept\n                : _shares(std::max(shares, 1u))\n                , _parent(p)\n        {}\n    public:\n        virtual fair_queue_entry* top() = 0;\n        virtual std::pair<bool, capacity_t> pop_front() = 0;\n        void wakeup(const config&) noexcept;\n\n        void update_shares(uint32_t shares) noexcept {\n            _shares = (std::max(shares, 1u));\n        }\n    };\n\n    using clock_type = std::chrono::steady_clock;\n    using priority_entry_ptr = priority_entry*;\n    struct class_compare {\n        bool operator() (const priority_entry_ptr& lhs, const priority_entry_ptr & rhs) const noexcept;\n    };\n\n    class priority_queue : public std::priority_queue<priority_entry_ptr, std::vector<priority_entry_ptr>, class_compare> {\n        using super = std::priority_queue<priority_entry_ptr, std::vector<priority_entry_ptr>, class_compare>;\n    public:\n        void reserve(size_t len) {\n            c.reserve(len);\n        }\n\n        void assert_enough_capacity() const noexcept {\n            SEASTAR_ASSERT(c.size() < c.capacity());\n        }\n    };\n\n    class priority_class_group_data final : public priority_entry {\n        friend class fair_queue;\n        friend testing::fair_queue_test;\n        priority_queue _children;\n        capacity_t _last_accumulated = 0;\n        size_t _nr_children = 0;\n\n        bool plug() noexcept;\n\n    public:\n        priority_class_group_data(uint32_t shares, priority_class_group_data* p) noexcept\n                : priority_entry(shares, p)\n        {\n            if (_parent == nullptr) {\n                _queued = true;\n            }\n        }\n        priority_class_group_data(const priority_class_group_data&) = delete;\n        priority_class_group_data(priority_class_group_data&&) = delete;\n\n        fair_queue_entry* top() override;\n        std::pair<bool, capacity_t> pop_front() override;\n\n        void reserve() {\n            _children.reserve(_nr_children + 1);\n        }\n\n        void push_from_idle(priority_entry&, const config&) noexcept;\n    };\n\n    class priority_class_data;\n\n    config _config;\n    priority_class_group_data _root;\n    std::vector<std::unique_ptr<priority_class_group_data>> _priority_groups;\n    std::vector<std::unique_ptr<priority_class_data>> _priority_classes;\n    friend testing::fair_queue_test;\n\n    // Total capacity of all requests waiting in the queue.\n    capacity_t _queued_capacity = 0;\n\n    void plug_priority_class(priority_class_data& pc) noexcept;\n    void unplug_priority_class(priority_class_data& pc) noexcept;\n    void plug_priority_group(priority_class_group_data& pg) noexcept;\n    void unplug_priority_group(priority_class_group_data& pg) noexcept;\n\npublic:\n    /// Constructs a fair queue with configuration parameters \\c cfg.\n    ///\n    /// \\param cfg an instance of the class \\ref config\n    explicit fair_queue(config cfg);\n    fair_queue(fair_queue&&) = delete;\n    ~fair_queue();\n\n    sstring label() const noexcept { return _config.label; }\n\n    /// Registers a priority class against this fair queue.\n    ///\n    /// \\param shares how many shares to create this class with\n    void register_priority_class(class_id c, uint32_t shares, std::optional<unsigned> group_idx = {});\n\n    /// Makes sure class groups exists\n    void ensure_priority_group(unsigned index, uint32_t shares);\n\n    /// Unregister a priority class.\n    ///\n    /// It is illegal to unregister a priority class that still have pending requests.\n    void unregister_priority_class(class_id c);\n\n    void update_shares_for_class(class_id c, uint32_t new_shares);\n\n    /// \\return how much resources (weight, size) are currently queued for all classes.\n    [[deprecated(\"Ticket resources are not accounted for any longer\")]]\n    fair_queue_ticket resources_currently_waiting() const { return fair_queue_ticket(); }\n\n    /// \\return the amount of resources (weight, size) currently executing\n    [[deprecated(\"Ticket resources are not accounted for any longer\")]]\n    fair_queue_ticket resources_currently_executing() const { return fair_queue_ticket(); }\n\n    /// Queue the entry \\c ent through this class' \\ref fair_queue\n    ///\n    /// The user of this interface is supposed to call \\ref notify_requests_finished when the\n    /// request finishes executing - regardless of success or failure.\n    void queue(class_id c, fair_queue_entry& ent) noexcept;\n\n    void plug_class(class_id c) noexcept;\n    void unplug_class(class_id c) noexcept;\n    void plug_class_group(unsigned index) noexcept;\n    void unplug_class_group(unsigned index) noexcept;\n\n    /// Notifies that ont request finished\n    /// \\param desc an instance of \\c fair_queue_ticket structure describing the request that just finished.\n    void notify_request_finished(fair_queue_entry::capacity_t cap) noexcept;\n    void notify_request_cancelled(fair_queue_entry& ent) noexcept;\n\n    fair_queue_entry* top();\n    void pop_front();\n\n    capacity_t queued_capacity() const noexcept { return _queued_capacity; }\n\n    capacity_t accumulated(class_id cid) const noexcept;\n    capacity_t pure_accumulated(class_id cid) const noexcept;\n    unsigned activations(class_id cid) const noexcept;\n};\n/// @}\n\n}\n\ntemplate <> struct fmt::formatter<seastar::fair_queue_ticket> : fmt::ostream_formatter {};\n"
  },
  {
    "path": "include/seastar/core/file-types.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <fcntl.h>\n#include <sys/stat.h>\n#include <type_traits>\n\nnamespace seastar {\n\n\n/// \\addtogroup fileio-module\n/// @{\n\n/// Enumeration describing how a file is to be opened.\n///\n/// \\see file::open_file_dma()\nenum class open_flags {\n    rw = O_RDWR,\n    ro = O_RDONLY,\n    wo = O_WRONLY,\n    create = O_CREAT,\n    truncate = O_TRUNC,\n    exclusive = O_EXCL,\n    dsync = O_DSYNC,\n};\n\ninline constexpr open_flags operator|(open_flags a, open_flags b) {\n    return open_flags(std::underlying_type_t<open_flags>(a) | std::underlying_type_t<open_flags>(b));\n}\n\ninline void operator|=(open_flags& a, open_flags b) {\n    a = (a | b);\n}\n\ninline constexpr open_flags operator&(open_flags a, open_flags b) {\n    return open_flags(std::underlying_type_t<open_flags>(a) & std::underlying_type_t<open_flags>(b));\n}\n\ninline void operator&=(open_flags& a, open_flags b) {\n    a = (a & b);\n}\n\n// The mask that should be used to extract only open-mode from the flags\nconstexpr open_flags open_flags_mode_mask = open_flags(3);\nstatic_assert((open_flags_mode_mask & open_flags::ro) == open_flags::ro);\nstatic_assert((open_flags_mode_mask & open_flags::rw) == open_flags::rw);\nstatic_assert((open_flags_mode_mask & open_flags::wo) == open_flags::wo);\n\n/// Enumeration describing the type of a directory entry being listed.\n///\n/// \\see file::list_directory()\nenum class directory_entry_type {\n    unknown,\n    block_device,\n    char_device,\n    directory,\n    fifo,\n    link,\n    regular,\n    socket,\n};\n\nnamespace internal::linux_abi {\n\n// From getdents(2):\n// check for 64-bit inode number\nstatic_assert(sizeof(ino_t) == 8, \"large file support not enabled\");\nstatic_assert(sizeof(off_t) == 8, \"large file support not enabled\");\n\n// From getdents(2):\nstruct linux_dirent64 {\n    ino_t          d_ino;    /* 64-bit inode number */\n    off_t          d_off;    /* 64-bit offset to next structure */\n    unsigned short d_reclen; /* Size of this dirent */\n    unsigned char  d_type;   /* File type */\n    char           d_name[]; /* Filename (null-terminated) */\n};\n\n} // internal::linux_abi namespace\n\n/// Enumeration describing the type of a particular filesystem\nenum class fs_type {\n    other,\n    xfs,\n    ext2,\n    ext3,\n    ext4,\n    btrfs,\n    hfs,\n    tmpfs,\n};\n\n// Access flags for files/directories\nenum class access_flags {\n    exists = F_OK,\n    read = R_OK,\n    write = W_OK,\n    execute = X_OK,\n\n    // alias for directory access\n    lookup = execute,\n};\n\ninline constexpr access_flags operator|(access_flags a, access_flags b) {\n    return access_flags(std::underlying_type_t<access_flags>(a) | std::underlying_type_t<access_flags>(b));\n}\n\ninline constexpr access_flags operator&(access_flags a, access_flags b) {\n    return access_flags(std::underlying_type_t<access_flags>(a) & std::underlying_type_t<access_flags>(b));\n}\n\n// Permissions for files/directories\nenum class file_permissions {\n    user_read = S_IRUSR,        // Read by owner\n    user_write = S_IWUSR,       // Write by owner\n    user_execute = S_IXUSR,     // Execute by owner\n\n    group_read = S_IRGRP,       // Read by group\n    group_write = S_IWGRP,      // Write by group\n    group_execute = S_IXGRP,    // Execute by group\n\n    others_read = S_IROTH,      // Read by others\n    others_write = S_IWOTH,     // Write by others\n    others_execute = S_IXOTH,   // Execute by others\n\n    user_permissions = user_read | user_write | user_execute,\n    group_permissions = group_read | group_write | group_execute,\n    others_permissions = others_read | others_write | others_execute,\n    all_permissions = user_permissions | group_permissions | others_permissions,\n\n    default_file_permissions = user_read | user_write | group_read | group_write | others_read | others_write, // 0666\n    default_dir_permissions = all_permissions, // 0777\n};\n\ninline constexpr file_permissions operator|(file_permissions a, file_permissions b) {\n    return file_permissions(std::underlying_type_t<file_permissions>(a) | std::underlying_type_t<file_permissions>(b));\n}\n\ninline constexpr file_permissions operator&(file_permissions a, file_permissions b) {\n    return file_permissions(std::underlying_type_t<file_permissions>(a) & std::underlying_type_t<file_permissions>(b));\n}\n\n/// @}\n\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/file.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/util/std-compat.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/generator.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/io_priority_class.hh>\n#include <seastar/core/file-types.hh>\n#include <seastar/core/circular_buffer_fixed_capacity.hh>\n\n#include <sys/statvfs.h>\n#include <sys/ioctl.h>\n#include <sys/types.h>\n#include <linux/fs.h>\n#include <sys/uio.h>\n#include <unistd.h>\n#include <chrono>\n#include <concepts>\n#include <cstdint>\n#include <functional>\n#include <optional>\n\nnamespace seastar {\n\n\n/// \\addtogroup fileio-module\n/// @{\n\n/// A directory entry being listed.\nstruct directory_entry {\n    /// Name of the file in a directory entry.  Will never be \".\" or \"..\".  Only the last component is included.\n    sstring name;\n    /// Type of the directory entry, if known.\n    std::optional<directory_entry_type> type;\n};\n\n/// Group details from the system group database\nstruct group_details {\n    sstring group_name;\n    sstring group_passwd;\n    gid_t group_id;\n    std::vector<sstring> group_members;\n};\n\n/// Filesystem object stat information\nstruct stat_data {\n    uint64_t  device_id;      // ID of device containing file\n    uint64_t  inode_number;   // Inode number\n    uint64_t  mode;           // File type and mode\n    directory_entry_type type;\n    uint64_t  number_of_links;// Number of hard links\n    uint64_t  uid;            // User ID of owner\n    uint64_t  gid;            // Group ID of owner\n    uint64_t  rdev;           // Device ID (if special file)\n    uint64_t  size;           // Total size, in bytes\n    uint64_t  block_size;     // Block size for filesystem I/O\n    uint64_t  allocated_size; // Total size of allocated storage, in bytes\n\n    std::chrono::system_clock::time_point time_accessed;  // Time of last content access\n    std::chrono::system_clock::time_point time_modified;  // Time of last content modification\n    std::chrono::system_clock::time_point time_changed;   // Time of last status change (either content or attributes)\n};\n\n/// File open options\n///\n/// Options used to configure an open file.\n///\n/// \\ref file\nstruct file_open_options {\n    uint64_t extent_allocation_size_hint = 1 << 20; ///< Allocate this much disk space when extending the file\n    bool sloppy_size = false; ///< Allow the file size not to track the amount of data written until a flush\n    uint64_t sloppy_size_hint = 1 << 20; ///< Hint as to what the eventual file size will be\n    file_permissions create_permissions = file_permissions::default_file_permissions; ///< File permissions to use when creating a file\n    bool append_is_unlikely = false; ///< Hint that user promises (or at least tries hard) not to write behind file size\n    bool durable = true; ///< If false, sacrifies file data integrity to IO performance (includes skipping flush() and dropping O_DSYNC)\n\n    // The fsxattr.fsx_extsize is 32-bit\n    static constexpr uint64_t max_extent_allocation_size_hint = 1 << 31;\n\n    // XFS ignores hints that are not aligned to the logical block size.\n    // To fulfill the requirement, we ensure that hint is aligned to 128KB (best guess).\n    static constexpr uint32_t min_extent_size_hint_alignment{128u << 10}; // 128KB\n};\n\nclass file;\nclass file_impl;\nclass io_intent;\nclass file_handle;\nclass file_data_sink_impl;\nclass file_data_source_impl;\n\n// The directory_entry size is 24 bytes (as the file name is allocated separately)\n// so the circular buffer is tuned to hold 16 entries\nconstexpr size_t list_directory_generator_buffer_size = calc_circular_buffer_capacity<directory_entry, 512>();\nusing list_directory_generator_type = coroutine::experimental::generator<directory_entry, directory_entry,\n        circular_buffer_fixed_capacity<directory_entry, list_directory_generator_buffer_size>>;\n\n// A handle that can be transported across shards and used to\n// create a dup(2)-like `file` object referring to the same underlying file\nclass file_handle_impl {\npublic:\n    virtual ~file_handle_impl() = default;\n    virtual std::unique_ptr<file_handle_impl> clone() const = 0;\n    virtual shared_ptr<file_impl> to_file() && = 0;\n};\n\nclass file_impl {\n    friend class file;\nprotected:\n    static file_impl* get_file_impl(file& f);\n    unsigned _memory_dma_alignment = 4096;\n    unsigned _disk_read_dma_alignment = 4096;\n    unsigned _disk_write_dma_alignment = 4096;\n    unsigned _disk_overwrite_dma_alignment = 4096;\n    unsigned _read_max_length = 1u << 30;\n    unsigned _write_max_length = 1u << 30;\npublic:\n    virtual ~file_impl() {}\n\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent*) = 0;\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent*) = 0;\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent*) = 0;\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent*) = 0;\n    virtual future<temporary_buffer<uint8_t>> dma_read_bulk(uint64_t offset, size_t range_size, io_intent*) = 0;\n\n    virtual future<> flush() = 0;\n    virtual future<struct stat> stat() = 0;\n    virtual future<struct stat> statat(std::string_view name, int flags = 0);\n    virtual future<> truncate(uint64_t length) = 0;\n    virtual future<> discard(uint64_t offset, uint64_t length) = 0;\n    virtual future<int> ioctl(uint64_t cmd, void* argp) noexcept;\n    virtual future<int> ioctl_short(uint64_t cmd, void* argp) noexcept;\n    virtual future<int> fcntl(int op, uintptr_t arg) noexcept;\n    virtual future<int> fcntl_short(int op, uintptr_t arg) noexcept;\n    virtual future<> allocate(uint64_t position, uint64_t length) = 0;\n    virtual future<uint64_t> size() = 0;\n    virtual future<> close() = 0;\n    virtual std::unique_ptr<file_handle_impl> dup();\n    virtual subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)> next) = 0;\n    virtual list_directory_generator_type experimental_list_directory();\n};\n\nfuture<shared_ptr<file_impl>> make_file_impl(int fd, file_open_options options, int oflags, struct stat st) noexcept;\n\n/// \\endcond\n\n/// A data file on persistent storage.\n///\n/// File objects represent uncached, unbuffered files.  As such great care\n/// must be taken to cache data at the application layer; neither seastar\n/// nor the OS will cache these file.\n///\n/// Data is transferred using direct memory access (DMA).  This imposes\n/// restrictions on file offsets and data pointers.  The former must be aligned\n/// on a 4096 byte boundary, while a 512 byte boundary suffices for the latter.\nclass file {\n    shared_ptr<file_impl> _file_impl;\npublic:\n    /// Default constructor constructs an uninitialized file object.\n    ///\n    /// A default constructor is useful for the common practice of declaring\n    /// a variable, and only assigning to it later. The uninitialized file\n    /// must not be used, or undefined behavior will result (currently, a null\n    /// pointer dereference).\n    ///\n    /// One can check whether a file object is in uninitialized state with\n    /// \\ref operator bool(); One can reset a file back to uninitialized state\n    /// by assigning file() to it.\n    file() noexcept : _file_impl(nullptr) {}\n\n    file(shared_ptr<file_impl> impl) noexcept\n            : _file_impl(std::move(impl)) {}\n\n    /// Constructs a file object from a \\ref file_handle obtained from another shard\n    explicit file(file_handle&& handle) noexcept;\n\n    /// Checks whether the file object was initialized.\n    ///\n    /// \\return false if the file object is uninitialized (default\n    /// constructed), true if the file object refers to an actual file.\n    explicit operator bool() const noexcept { return bool(_file_impl); }\n\n    /// Copies a file object.  The new and old objects refer to the\n    /// same underlying file.\n    ///\n    /// \\param x file object to be copied\n    file(const file& x) = default;\n    /// Moves a file object.\n    file(file&& x) noexcept : _file_impl(std::move(x._file_impl)) {}\n    /// Assigns a file object.  After assignent, the destination and source refer\n    /// to the same underlying file.\n    ///\n    /// \\param x file object to assign to `this`.\n    file& operator=(const file& x) noexcept = default;\n    /// Moves assigns a file object.\n    file& operator=(file&& x) noexcept = default;\n\n    // O_DIRECT reading requires that buffer, offset, and read length, are\n    // all aligned. Alignment of 4096 was necessary in the past, but no longer\n    // is - 512 is usually enough; But we'll need to use BLKSSZGET ioctl to\n    // be sure it is really enough on this filesystem. 4096 is always safe.\n    // In addition, if we start reading in things outside page boundaries,\n    // we will end up with various pages around, some of them with\n    // overlapping ranges. Those would be very challenging to cache.\n\n    /// Alignment requirement for file offsets (for reads)\n    uint64_t disk_read_dma_alignment() const noexcept {\n        return _file_impl->_disk_read_dma_alignment;\n    }\n\n    /// Alignment requirement for file offsets (for writes)\n    uint64_t disk_write_dma_alignment() const noexcept {\n        return _file_impl->_disk_write_dma_alignment;\n    }\n\n    /// Alignment requirement for file offsets (for overwrites).\n    ///\n    /// Specifies the minimum alignment for disk offsets for\n    /// overwrites (writes to a location that was previously written).\n    /// This can be smaller than \\ref disk_write_dma_alignment(), allowing\n    /// a reduction in disk bandwidth used.\n    uint64_t disk_overwrite_dma_alignment() const noexcept {\n        return _file_impl->_disk_overwrite_dma_alignment;\n    }\n\n    /// Alignment requirement for data buffers\n    uint64_t memory_dma_alignment() const noexcept {\n        return _file_impl->_memory_dma_alignment;\n    }\n\n    /// Recommended limit for read request size.\n    /// Submitting a larger request will not cause any error,\n    /// but may result in poor latencies for this and any other\n    /// concurrent requests\n    size_t disk_read_max_length() const noexcept {\n        return _file_impl->_read_max_length;\n    }\n\n    /// Recommended limit for write request size.\n    /// Submitting a larger request will not cause any error,\n    /// but may result in poor latencies for this and any other\n    /// concurrent requests\n    size_t disk_write_max_length() const noexcept {\n        return _file_impl->_write_max_length;\n    }\n\n    /**\n     * Perform a single DMA read operation.\n     *\n     * @param aligned_pos offset to begin reading at (should be aligned)\n     * @param aligned_buffer output buffer (should be aligned)\n     * @param aligned_len number of bytes to read (should be aligned)\n     * @param intent the IO intention confirmation (\\ref seastar::io_intent)\n     *\n     * Alignment is HW dependent but use 4KB alignment to be on the safe side as\n     * explained above.\n     *\n     * @return number of bytes actually read\n     *         or exceptional future in case of I/O error\n     */\n    template <typename CharType>\n    future<size_t>\n    dma_read(uint64_t aligned_pos, CharType* aligned_buffer, size_t aligned_len, io_intent* intent = nullptr) noexcept {\n        return dma_read_impl(aligned_pos, reinterpret_cast<uint8_t*>(aligned_buffer), aligned_len, intent);\n    }\n\n    /**\n     * Read the requested amount of bytes starting from the given offset.\n     *\n     * @param pos offset to begin reading from\n     * @param len number of bytes to read\n     * @param intent the IO intention confirmation (\\ref seastar::io_intent)\n     *\n     * @return temporary buffer containing the requested data.\n     *         or exceptional future in case of I/O error\n     *\n     * This function doesn't require any alignment for both \"pos\" and \"len\"\n     *\n     * @note size of the returned buffer may be smaller than \"len\" if EOF is\n     *       reached or in case of I/O error.\n     */\n    template <typename CharType>\n    future<temporary_buffer<CharType>> dma_read(uint64_t pos, size_t len, io_intent* intent = nullptr) noexcept {\n        return dma_read_impl(pos, len, intent).then([] (temporary_buffer<uint8_t> t) {\n            return temporary_buffer<CharType>(reinterpret_cast<CharType*>(t.get_write()), t.size(), t.release());\n        });\n    }\n\n    /// Error thrown when attempting to read past end-of-file\n    /// with \\ref dma_read_exactly().\n    class eof_error : public std::exception {};\n\n    /**\n     * Read the exact amount of bytes.\n     *\n     * @param pos offset in a file to begin reading from\n     * @param len number of bytes to read\n     * @param intent the IO intention confirmation (\\ref seastar::io_intent)\n     *\n     * @return temporary buffer containing the read data\n     *        or exceptional future in case an error, holding:\n     *        end_of_file_error if EOF is reached, file_io_error or\n     *        std::system_error in case of I/O error.\n     */\n    template <typename CharType>\n    future<temporary_buffer<CharType>>\n    dma_read_exactly(uint64_t pos, size_t len, io_intent* intent = nullptr) noexcept {\n        return dma_read_exactly_impl(pos, len, intent).then([] (temporary_buffer<uint8_t> t) {\n            return temporary_buffer<CharType>(reinterpret_cast<CharType*>(t.get_write()), t.size(), t.release());\n        });\n    }\n\n    /// Performs a DMA read into the specified iovec.\n    ///\n    /// \\param pos offset to read from.  Must be aligned to \\ref disk_read_dma_alignment.\n    /// \\param iov vector of address/size pairs to read into.  Addresses must be\n    ///            aligned.\n    /// \\param intent the IO intention confirmation (\\ref seastar::io_intent)\n    ///\n    /// \\return a future representing the number of bytes actually read.  A short\n    ///         read may happen due to end-of-file or an I/O error.\n    ///\n    /// Note that for this overload, \\ref disk_read_max_length corresponds to the sum of\n    /// the iovec sizes.\n    future<size_t> dma_read(uint64_t pos, std::vector<iovec> iov, io_intent* intent = nullptr) noexcept {\n        return dma_read_impl(pos, std::move(iov), intent);\n    }\n\n    /// Performs a DMA write from the specified buffer.\n    ///\n    /// \\param pos offset to write into.  Must be aligned to \\ref disk_write_dma_alignment.\n    /// \\param buffer aligned address of buffer to read from.  Buffer must exists\n    ///               until the future is made ready.\n    /// \\param len number of bytes to write.  Must be aligned.\n    /// \\param intent the IO intention confirmation (\\ref seastar::io_intent)\n    ///\n    /// \\return a future representing the number of bytes actually written.  A short\n    ///         write may happen due to an I/O error.\n    template <typename CharType>\n    future<size_t> dma_write(uint64_t pos, const CharType* buffer, size_t len, io_intent* intent = nullptr) noexcept {\n        return dma_write_impl(pos, reinterpret_cast<const uint8_t*>(buffer), len, intent);\n    }\n\n    /// Performs a DMA write to the specified iovec.\n    ///\n    /// \\param pos offset to write into.  Must be aligned to \\ref disk_write_dma_alignment.\n    /// \\param iov vector of address/size pairs to write from.  Addresses must be\n    ///            aligned.\n    /// \\param intent the IO intention confirmation (\\ref seastar::io_intent)\n    ///\n    /// \\return a future representing the number of bytes actually written.  A short\n    ///         write may happen due to an I/O error.\n    ///\n    /// Note that for this overload, \\ref disk_write_max_length corresponds to the sum of\n    /// the iovec sizes.\n    future<size_t> dma_write(uint64_t pos, std::vector<iovec> iov, io_intent* intent = nullptr) noexcept {\n        return dma_write_impl(pos, std::move(iov), intent);\n    }\n\n    /// Causes any previously written data to be made stable on persistent storage.\n    ///\n    /// Prior to a flush, written data may or may not survive a power failure.  After\n    /// a flush, data is guaranteed to be on disk.\n    future<> flush() noexcept;\n\n    /// Returns \\c stat information about the file.\n    future<struct stat> stat() noexcept;\n\n    /// Returns \\c stat information about a file in this directory.\n    ///\n    /// \\param name the name of the file relative to this directory\n    /// \\param flags optional flags (e.g., AT_SYMLINK_NOFOLLOW, see man fstatat)\n    future<struct stat> statat(std::string_view name, int flags = 0) noexcept;\n\n    /// Truncates the file to a specified length.\n    future<> truncate(uint64_t length) noexcept;\n\n    /// Preallocate disk blocks for a specified byte range.\n    ///\n    /// Requests the file system to allocate disk blocks to\n    /// back the specified range (\\c length bytes starting at\n    /// \\c position).  The range may be outside the current file\n    /// size; the blocks can then be used when appending to the\n    /// file.\n    ///\n    /// \\param position beginning of the range at which to allocate\n    ///                 blocks.\n    /// \\param length length of range to allocate.\n    /// \\return future that becomes ready when the operation completes.\n    future<> allocate(uint64_t position, uint64_t length) noexcept;\n\n    /// Discard unneeded data from the file.\n    ///\n    /// The discard operation tells the file system that a range of offsets\n    /// (which be aligned) is no longer needed and can be reused.\n    future<> discard(uint64_t offset, uint64_t length) noexcept;\n\n    /// Generic ioctl syscall support for special file handling.\n    ///\n    /// This interface is useful for many non-standard operations on seastar::file.\n    /// The examples can be - querying device or file system capabilities,\n    /// configuring special performance or access modes on devices etc.\n    /// Refer ioctl(2) man page for more details.\n    ///\n    /// \\param cmd ioctl command to be executed\n    /// \\param argp pointer to the buffer which holds the argument\n    ///\n    /// \\return a future containing the return value if any, or an exceptional future\n    ///         if the operation has failed.\n    future<int> ioctl(uint64_t cmd, void* argp) noexcept;\n\n    /// Performs a short ioctl syscall on seastar::file\n    ///\n    /// This is similar to generic \\c ioctl; the difference is, here user indicates\n    /// that this operation is a short one, and does not involve any i/o or locking.\n    /// The \\c file module will process this differently from the normal \\ref ioctl().\n    /// Use this method only if the user is sure that the operation does not involve any\n    /// blocking operation. If unsure, use the default \\ref ioctl() method.\n    /// Refer ioctl(2) man page for more details on ioctl operation.\n    ///\n    /// \\param cmd ioctl command to be executed\n    /// \\param argp pointer to the buffer which holds the argument\n    ///\n    /// \\return a future containing the return value if any, or an exceptional future\n    ///         if the operation has failed.\n    future<int> ioctl_short(uint64_t cmd, void* argp) noexcept;\n\n    /// Generic fcntl syscall support for special file handling.\n    ///\n    /// fcntl performs the operation specified by 'op' field on the file.\n    /// Some of the use cases can be - setting file status flags, advisory record locking,\n    /// managing signals, managing file leases or write hints etc.\n    /// Refer fcntl(2) man page for more details.\n    ///\n    /// \\param op the operation to be executed\n    /// \\param arg the optional argument\n    /// \\return a future containing the return value if any, or an exceptional future\n    ///         if the operation has failed\n    future<int> fcntl(int op, uintptr_t arg = 0UL) noexcept;\n\n    /// Performs a 'short' fcntl syscall on seastar::file\n    ///\n    /// This is similar to generic \\c fcntl; the difference is, here user indicates\n    /// that this operation is a short one, and does not involve any i/o or locking.\n    /// The \\c file module will process this differently from normal \\ref fcntl().\n    /// Use this only if the user is sure that the operation does not involve any\n    /// blocking operation. If unsure, use the default \\ref fcntl() method.\n    /// Refer fcntl(2) man page for more details on fcntl operation.\n    ///\n    /// \\param op the operation to be executed\n    /// \\param arg the optional argument\n    /// \\return a future containing the return value if any, or an exceptional future\n    ///         if the operation has failed\n    future<int> fcntl_short(int op, uintptr_t arg = 0UL) noexcept;\n\n    /// Set a lifetime hint for the inode corresponding to seastar::file\n    ///\n    /// Write lifetime  hints  can be used to inform the kernel about the relative\n    /// expected lifetime of writes on a given inode or via open file descriptor.\n    /// An application may use the different hint values to separate writes into different\n    /// write classes, so that multiple users or applications running on a single storage back-end\n    /// can aggregate their I/O  patterns in a consistent manner.\n    /// Refer fcntl(2) man page for more details on write lifetime hints.\n    ///\n    /// \\param hint the hint value of the stream\n    /// \\return future indicating success or failure\n    future<> set_inode_lifetime_hint(uint64_t hint) noexcept;\n\n    /// Get the lifetime hint of the inode of seastar::file which was set by\n    /// \\ref set_inode_lifetime_hint()\n    ///\n    /// Write lifetime  hints  can be used to inform the kernel about the relative\n    /// expected lifetime of writes on a given inode or via open file descriptor.\n    /// An application may use the different hint values to separate writes into different\n    /// write classes, so that multiple users or applications running on a single storage back-end\n    /// can aggregate their I/O  patterns in a consistent manner.\n    /// Refer fcntl(2) man page for more details on write lifetime hints.\n    ///\n    /// \\return the hint value of the inode\n    future<uint64_t> get_inode_lifetime_hint() noexcept;\n\n    /// Gets the file size.\n    future<uint64_t> size() const noexcept;\n\n    /// Closes the file.\n    ///\n    /// Flushes any pending operations and release any resources associated with\n    /// the file (except for stable storage). Resets the file object back to\n    /// uninitialized state as if by assigning file() to it.\n    ///\n    /// \\note\n    /// \\c close() never fails. It just reports errors and swallows them.\n    /// To ensure file data reaches stable storage, you must call \\ref flush()\n    /// before calling \\c close().\n    future<> close() noexcept;\n\n    /// Returns a directory listing, given that this file object is a directory.\n    subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)> next);\n\n    /// Returns a directory listing, given that this file object is a directory.\n    list_directory_generator_type experimental_list_directory();\n\n    /**\n     * Read a data bulk containing the provided addresses range that starts at\n     * the given offset and ends at either the address aligned to\n     * dma_alignment (4KB) or at the file end.\n     *\n     * @param offset starting address of the range the read bulk should contain\n     * @param range_size size of the addresses range\n     * @param intent the IO intention confirmation (\\ref seastar::io_intent)\n     *\n     * @return temporary buffer containing the read data bulk.\n     *        or exceptional future holding:\n     *        system_error exception in case of I/O error or eof_error when\n     *        \"offset\" is beyond EOF.\n     */\n    template <typename CharType>\n    future<temporary_buffer<CharType>>\n    dma_read_bulk(uint64_t offset, size_t range_size, io_intent* intent = nullptr) noexcept {\n        return dma_read_bulk_impl(offset, range_size, intent).then([] (temporary_buffer<uint8_t> t) {\n            return temporary_buffer<CharType>(reinterpret_cast<CharType*>(t.get_write()), t.size(), t.release());\n        });\n    }\n\n    /// \\brief Creates a handle that can be transported across shards.\n    ///\n    /// Creates a handle that can be transported across shards, and then\n    /// used to create a new shard-local \\ref file object that refers to\n    /// the same on-disk file.\n    ///\n    /// \\note Use on read-only files.\n    ///\n    file_handle dup();\nprivate:\n    future<temporary_buffer<uint8_t>>\n    dma_read_bulk_impl(uint64_t offset, size_t range_size, io_intent* intent) noexcept;\n\n    future<size_t>\n    dma_write_impl(uint64_t pos, const uint8_t* buffer, size_t len, io_intent* intent) noexcept;\n\n    future<size_t>\n    dma_write_impl(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept;\n\n    future<temporary_buffer<uint8_t>>\n    dma_read_impl(uint64_t pos, size_t len, io_intent* intent) noexcept;\n\n    future<size_t>\n    dma_read_impl(uint64_t aligned_pos, uint8_t* aligned_buffer, size_t aligned_len, io_intent* intent) noexcept;\n\n    future<size_t>\n    dma_read_impl(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept;\n\n    future<temporary_buffer<uint8_t>>\n    dma_read_exactly_impl(uint64_t pos, size_t len, io_intent* intent) noexcept;\n\n    future<uint64_t> get_lifetime_hint_impl(int op) noexcept;\n    future<> set_lifetime_hint_impl(int op, uint64_t hint) noexcept;\n\n    friend class file_impl;\n    friend class file_data_sink_impl;\n    friend class file_data_source_impl;\n};\n\n/// \\brief Helper for ensuring a file is closed after \\c func is called.\n///\n/// The file provided by the \\c file_fut future is passed to \\c func.\n///\n/// \\param file_fut A future that produces a file\n/// \\param func A function that uses a file\n/// \\returns the future returned by \\c func, or an exceptional future if either \\c file_fut or closing the file failed.\ntemplate <std::invocable<file&> Func>\nfuturize_t<std::invoke_result_t<Func, file&>> with_file(future<file> file_fut, Func func) noexcept {\n    auto f = co_await std::move(file_fut);\n    // If f.close() fails, return that as nested exception.\n    co_return co_await futurize_invoke(func, f).finally([&f] {\n        return f.close();\n    });\n}\n\n/// \\brief Helper for ensuring a file is closed if \\c func fails.\n///\n/// The file provided by the \\c file_fut future is passed to \\c func.\n/// * If func throws an exception E, the file is closed and we return\n///   a failed future with E.\n/// * If func returns a value V, the file is not closed and we return\n///   a future with V.\n/// Note that when an exception is not thrown, it is the\n/// responsibility of func to make sure the file will be closed. It\n/// can close the file itself, return it, or store it somewhere.\n///\n/// \\param file_fut A future that produces a file\n/// \\param func A function that uses a file\n/// \\returns the future returned by \\c func, or an exceptional future if \\c file_fut failed or a nested exception if closing the file failed.\ntemplate <std::invocable<file&> Func>\nfuturize_t<std::invoke_result_t<Func, file&>> with_file_close_on_failure(future<file> file_fut, Func func) noexcept {\n    auto f = co_await std::move(file_fut);\n    auto fut = co_await coroutine::as_future(futurize_invoke(func, f));\n    // If f.close() fails, return that as nested exception.\n    if (fut.failed()) {\n        fut = fut.finally([&f] {\n            return f.close();\n        });\n    }\n    co_return co_await std::move(fut);\n}\n\n/// \\example file_demo.cc\n/// A program demonstrating the use of \\ref seastar::with_file\n/// and \\ref seastar::with_file_close_on_failure\n\n/// \\brief A shard-transportable handle to a file\n///\n/// If you need to access a file (for reads only) across multiple shards,\n/// you can use the file::dup() method to create a `file_handle`, transport\n/// this file handle to another shard, and use the handle to create \\ref file\n/// object on that shard.  This is more efficient than calling open_file_dma()\n/// again.\nclass file_handle {\n    std::unique_ptr<file_handle_impl> _impl;\nprivate:\n    explicit file_handle(std::unique_ptr<file_handle_impl> impl) : _impl(std::move(impl)) {}\npublic:\n    /// Copies a file handle object\n    file_handle(const file_handle&);\n    /// Moves a file handle object\n    file_handle(file_handle&&) noexcept;\n    /// Assigns a file handle object\n    file_handle& operator=(const file_handle&);\n    /// Move-assigns a file handle object\n    file_handle& operator=(file_handle&&) noexcept;\n    /// Converts the file handle object to a \\ref file.\n    file to_file() const &;\n    /// Converts the file handle object to a \\ref file.\n    file to_file() &&;\n\n    friend class file;\n};\n\n/// @}\n\n/// An exception Cancelled IOs resolve their future into (see \\ref io_intent \"io_intent\")\nclass cancelled_error : public std::exception {\npublic:\n    virtual const char* what() const noexcept {\n        return \"cancelled\";\n    }\n};\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/format.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <fmt/format.h>\n\nnamespace seastar {\n\n/**\n * Evaluate the formatted string in a native fmt library format\n *\n * @param fmt format string with the native fmt library syntax\n * @param a positional parameters\n *\n * @return sstring object with the result of applying the given positional\n *         parameters on a given format string.\n */\ntemplate <typename... A>\nsstring\nformat(fmt::format_string<A...> fmt, A&&... a) {\n    fmt::memory_buffer out;\n    fmt::format_to(fmt::appender(out), fmt, std::forward<A>(a)...);\n    return sstring{out.data(), out.size()};\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/fsnotify.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <sys/inotify.h>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n\nnamespace seastar::experimental {\n\n\n/// \\defgroup fsnotifier FileSystem Notifier\n///\n/// Seastar provides an API which can be used to monitor filesystem modifications.\n///\n/// \\addtogroup fsnotifier\n/// @{\n\n/// \\brief Filesystem modification notifier.\n///\n/// This is a thin wrapper around inotify(see http://man7.org/linux/man-pages/man7/inotify.7.html),\n/// which is the de-facto light-weight filesystem modification watch\n/// interface in Linux and can be used to log filesystem activities,\n/// reload configurations, etc.\n///\n/// The wrapper provides a buffered read of events, and RAII handling\n/// of watches themselves.\n///\n/// \\note Note that this is an experimental feature, and thus any tweaks that\n///       are backward incompatible can be made before finally freezing it.\n///       Besides, the impl currently does not (yet) handle re-writing watches.\nclass fsnotifier {\n    class impl;\n    shared_ptr<impl> _impl;\npublic:\n    class watch;\n    friend class watch;\n\n    /// \\brief Flags of events supported by \\ref fsnotifier.\n    ///\n    /// \\note Note that the flags are bit-matched with inotify and they can\n    ///       be calculated using the bitwise AND and bitwise OR operators\n    ///       (including the assignment form) directly to take intersection\n    ///       or union.\n    enum class flags : uint32_t {\n        access = IN_ACCESS,             // File was accessed (e.g., read(2), execve(2)).\n        attrib = IN_ATTRIB,             // Metadata changed—for example, permissions, timestamps, extended attributes\n        close_write = IN_CLOSE_WRITE,   // File opened for writing was closed.\n        close_nowrite = IN_CLOSE_NOWRITE,// File or directory not opened for writing was closed.\n        create_child = IN_CREATE,       // File/directory created in watched directory\n        delete_child = IN_DELETE,       // File/directory deleted from watched directory.\n        delete_self = IN_DELETE_SELF,   // Watched file/directory was itself deleted.  (This event\n                                        // also occurs if an object is moved to another filesystem)\n        modify = IN_MODIFY,             // File was modified (e.g., write(2), truncate(2)).\n        move_self = IN_MOVE_SELF,       // Watched file/directory was itself moved.\n        move_from = IN_MOVED_FROM,      // Generated for the directory containing the old filename\n                                        // when a file is renamed.\n        move_to = IN_MOVED_TO,          // Generated for the directory containing the new filename\n                                        // when a file is renamed.\n        open = IN_OPEN,                 // File was opened\n        close = IN_CLOSE,               // close_write|close_nowrite\n        move = IN_MOVE,                 // move_from|move_to\n        oneshot = IN_ONESHOT,           // listen for only a single notification, after which the\n                                        // token will be invalid\n        ignored = IN_IGNORED,           // generated when a token or the file being watched is deleted\n        onlydir = IN_ONLYDIR,           // Watch pathname only if it is a directory; the error ENOT‐\n                                        // DIR results if pathname is not a directory.  Using this\n                                        // flag provides an application with a race-free way of\n                                        // ensuring that the monitored object is a directory.\n    };\n\n    /// \\brief Token of a watch point.\n    using watch_token = int32_t;\n    /// \\brief Unique sequence number of associating related events.\n    ///\n    /// \\note The sequence number is used to connect related events.\n    ///       Currently, it is used only for the rename events(i.e.,\n    ///       move_from and move_to), and is 0 for other types.\n    using sequence_no = uint32_t;\n\n    /// \\brief Simple RAII wrapper around a \\ref fsnotifier::watch_token\n    ///\n    /// The events of the path will be unregistered automatically on\n    /// destruction of the \\ref watch.\n    class watch {\n    public:\n        ~watch();\n        watch(watch&&) noexcept;\n        watch& operator=(watch&&) noexcept;\n\n        /// Reset the watch point.\n        ///\n        /// \\note Note that this operation won't unregister the event for\n        ///       the path, but simply releases resources used internally.\n        watch_token release();\n\n        /// Cast this watch point to a watch token.\n        operator watch_token() const {\n            return _token;\n        }\n\n        /// Get the token of this watch point.\n        watch_token token() const {\n            return _token;\n        }\n\n    private:\n        friend class fsnotifier;\n        watch(shared_ptr<impl>, watch_token);\n        watch_token _token;\n        shared_ptr<impl> _impl;\n    };\n\n    fsnotifier();\n    ~fsnotifier();\n\n    fsnotifier(fsnotifier&&);\n    fsnotifier& operator=(fsnotifier&&);\n\n    /// \\brief Monitor events specified in mask for the give path.\n    ///\n    /// \\param path path of the file or directory to monitor for.\n    /// \\param mask events of interest.\n    /// \\return a future that becomes ready when the underlying\n    ///         inotify_add_watch(2) call completes.\n    future<watch> create_watch(const sstring& path, flags mask);\n\n    /// \\brief A wrapper around inotify_event.\n    struct event {\n        // matches source watch\n        watch_token id;\n        // event(s) generated\n        flags mask;\n        sequence_no seq; // event correlation -> move_from+move_to\n        sstring name; // optional file name, in case of move_from/to\n    };\n\n    /// Wait for events.\n    ///\n    /// \\return a future that becomes ready when registered events occur.\n    future<std::vector<event>> wait() const;\n\n    /// Shutdown the notifier and abort any waiting events.\n    ///\n    /// \\note After shutdown, all watches are invalidated,\n    ///       and no new ones can be created.\n    void shutdown();\n\n    /// Check if the notifier is activated.\n    bool active() const;\n\n    /// Equivalent to \\ref active().\n    operator bool() const {\n        return active();\n    }\n};\n\n/// Take the union of two events.\ninline fsnotifier::flags operator|(fsnotifier::flags a, fsnotifier::flags b) {\n    return fsnotifier::flags(std::underlying_type_t<fsnotifier::flags>(a) | std::underlying_type_t<fsnotifier::flags>(b));\n}\n\n/// Take the union of two events, assignment form.\ninline void operator|=(fsnotifier::flags& a, fsnotifier::flags b) {\n    a = (a | b);\n}\n\n/// Take the intersection of two events.\ninline fsnotifier::flags operator&(fsnotifier::flags a, fsnotifier::flags b) {\n    return fsnotifier::flags(std::underlying_type_t<fsnotifier::flags>(a) & std::underlying_type_t<fsnotifier::flags>(b));\n}\n\n/// Take the intersection of two events, assignment form.\ninline void operator&=(fsnotifier::flags& a, fsnotifier::flags b) {\n    a = (a & b);\n}\n\n/// @}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/fsqual.hh",
    "content": "/*\n * Copyright 2017 ScyllaDB\n */\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\n\nbool filesystem_has_good_aio_support(sstring directory, bool verbose = false);\n\n}\n"
  },
  {
    "path": "include/seastar/core/fstream.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n/// \\file\n\n// File <-> streams adapters\n//\n// Seastar files are block-based due to the reliance on DMA - you must read\n// on sector boundaries.  The adapters in this file provide a byte stream\n// interface to files, while retaining the zero-copy characteristics of\n// seastar files.\n#include <seastar/core/file.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/internal/api-level.hh>\n\n#include <cstdint>\n\nnamespace seastar {\n\n\nclass file_input_stream_history {\n    static constexpr uint64_t window_size = 4 * 1024 * 1024;\n    struct window {\n        uint64_t total_read = 0;\n        uint64_t unused_read = 0;\n    };\n    window current_window;\n    window previous_window;\n    unsigned read_ahead = 1;\n\n    friend class file_data_source_impl;\n};\n\n/// Data structure describing options for opening a file input stream\nstruct file_input_stream_options {\n    size_t buffer_size = 8192;    ///< I/O buffer size\n    unsigned read_ahead = 0;      ///< Maximum number of extra read-ahead operations\n    lw_shared_ptr<file_input_stream_history> dynamic_adjustments = { }; ///< Input stream history, if null dynamic adjustments are disabled\n};\n\n/// \\brief Creates an input_stream to read a portion of a file.\n///\n/// \\param file File to read; multiple streams for the same file may coexist\n/// \\param offset Starting offset to read from (no alignment restrictions)\n/// \\param len Maximum number of bytes to read; the stream will stop at end-of-file\n///            even if `offset + len` is beyond end-of-file.\n/// \\param options A set of options controlling the stream.\n///\n/// \\note Multiple input streams may exist concurrently for the same file.\ninput_stream<char> make_file_input_stream(\n        file file, uint64_t offset, uint64_t len, file_input_stream_options options = {});\n\n/// \\brief Create an input_stream for a given file, reading starting at a given\n///        position of the given file, with the specified options.\n/// \\param file File to read; multiple streams for the same file may coexist\n/// \\param offset Starting offset to read from (no alignment restrictions)\n///\n/// \\note Multiple fibers of execution (continuations) may safely open\n///       multiple input streams concurrently for the same file.\ninput_stream<char> make_file_input_stream(\n        file file, uint64_t offset, file_input_stream_options = {});\n\n/// Create an input_stream for a given file, with the specified options\n/// \\param file File to read; multiple streams for the same file may coexist\n///\n/// \\note Multiple fibers of execution (continuations) may safely open\n///       multiple input streams concurrently for the same file.\ninput_stream<char> make_file_input_stream(\n        file file, file_input_stream_options = {});\n\n/// Create a data_source for reading the given offset:len range from the file\ndata_source make_file_data_source(file, uint64_t offset, uint64_t len, file_input_stream_options);\n\n/// Create a data_source for reading the whole file from start to end\ndata_source make_file_data_source(file, file_input_stream_options);\n\nstruct file_output_stream_options {\n    // For small files, setting preallocation_size can make it impossible for XFS to find\n    // an aligned extent. On the other hand, without it, XFS will divide the file into\n    // file_size/buffer_size extents. To avoid fragmentation, we set the default buffer_size\n    // to 64k (so each extent will be a minimum of 64k) and preallocation_size to 0 (to avoid\n    // extent allocation problems).\n    //\n    // Large files should increase both buffer_size and preallocation_size.\n    unsigned buffer_size = 65536;\n    unsigned preallocation_size = 0; ///< Preallocate extents. For large files, set to a large number (a few megabytes) to reduce fragmentation\n    unsigned write_behind = 1; ///< Number of buffers to write in parallel\n};\n\n/// Create an output_stream for writing starting at the position zero of a\n/// newly created file.\n/// NOTE: flush() should be the last thing to be called on a file output stream.\n/// Closes the file if the stream creation fails.\nfuture<output_stream<char>> make_file_output_stream(\n        file file,\n        uint64_t buffer_size = 8192) noexcept;\n\n/// Create an output_stream for writing starting at the position zero of a\n/// newly created file.\n/// NOTE: flush() should be the last thing to be called on a file output stream.\n/// Closes the file if the stream creation fails.\nfuture<output_stream<char>> make_file_output_stream(\n        file file,\n        file_output_stream_options options) noexcept;\n\n/// Create a data_sink for writing starting at the position zero of a\n/// newly created file.\n/// Closes the file if the sink creation fails.\nfuture<data_sink> make_file_data_sink(file, file_output_stream_options) noexcept;\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/function_traits.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <tuple>\n#include <functional>\n\nnamespace seastar {\n\ntemplate<typename T>\nstruct function_traits;\n\ntemplate<typename Ret, typename... Args>\nstruct function_traits<Ret(Args...)>\n{\n    using return_type = Ret;\n    using args_as_tuple = std::tuple<Args...>;\n    using signature = Ret (Args...);\n\n    static constexpr std::size_t arity = sizeof...(Args);\n\n    template <std::size_t N>\n    struct arg\n    {\n        static_assert(N < arity, \"no such parameter index.\");\n        using type = typename std::tuple_element<N, std::tuple<Args...>>::type;\n    };\n};\n\ntemplate<typename Ret, typename... Args>\nstruct function_traits<Ret(*)(Args...)> : public function_traits<Ret(Args...)>\n{};\n\ntemplate <typename T, typename Ret, typename... Args>\nstruct function_traits<Ret(T::*)(Args...)> : public function_traits<Ret(Args...)>\n{};\n\ntemplate <typename T, typename Ret, typename... Args>\nstruct function_traits<Ret(T::*)(Args...) const> : public function_traits<Ret(Args...)>\n{};\n\ntemplate<typename Ret, typename... Args>\nstruct function_traits<Ret(*)(Args...) noexcept> : public function_traits<Ret(Args...)>\n{};\n\ntemplate <typename T, typename Ret, typename... Args>\nstruct function_traits<Ret(T::*)(Args...) noexcept> : public function_traits<Ret(Args...)>\n{};\n\ntemplate <typename T, typename Ret, typename... Args>\nstruct function_traits<Ret(T::*)(Args...) const noexcept> : public function_traits<Ret(Args...)>\n{};\n\ntemplate <typename T>\nstruct function_traits : public function_traits<decltype(&T::operator())>\n{};\n\ntemplate<typename T>\nstruct function_traits<T&> : public function_traits<std::remove_reference_t<T>>\n{};\n\ntemplate<typename T>\nstruct function_traits<std::reference_wrapper<T>> : public function_traits<T>\n{};\n\n}\n"
  },
  {
    "path": "include/seastar/core/future-util.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/core/with_timeout.hh>\n#include <seastar/util/later.hh>\n"
  },
  {
    "path": "include/seastar/core/future.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cassert>\n#include <concepts>\n#include <cstdlib>\n#include <cstring>\n#include <functional>\n#include <memory>\n#include <stdexcept>\n#include <type_traits>\n#include <utility>\n#include <source_location>\n\n#include <seastar/core/task.hh>\n#include <seastar/core/thread_impl.hh>\n#include <seastar/core/function_traits.hh>\n#include <seastar/core/shard_id.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/critical_alloc_section.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/log-level.hh>\n\nnamespace seastar {\n\nstruct nested_exception : public std::exception {\n    std::exception_ptr inner;\n    std::exception_ptr outer;\n    nested_exception(std::exception_ptr inner, std::exception_ptr outer) noexcept;\n    nested_exception(nested_exception&&) noexcept;\n    nested_exception(const nested_exception&) noexcept;\n    [[noreturn]] void rethrow_nested() const;\n    virtual const char* what() const noexcept override;\n};\n\n/// \\defgroup future-module Futures and Promises\n///\n/// \\brief\n/// Futures and promises are the basic tools for asynchronous\n/// programming in seastar.  A future represents a result that\n/// may not have been computed yet, for example a buffer that\n/// is being read from the disk, or the result of a function\n/// that is executed on another cpu.  A promise object allows\n/// the future to be eventually resolved by assigning it a value.\n///\n/// \\brief\n/// Another way to look at futures and promises are as the reader\n/// and writer sides, respectively, of a single-item, single use\n/// queue.  You read from the future, and write to the promise,\n/// and the system takes care that it works no matter what the\n/// order of operations is.\n///\n/// \\brief\n/// The normal way of working with futures is to chain continuations\n/// to them.  A continuation is a block of code (usually a lamdba)\n/// that is called when the future is assigned a value (the future\n/// is resolved); the continuation can then access the actual value.\n///\n\n/// \\defgroup future-module-impl Implementation overview\n/// \\ingroup future-module\n///\n/// A future has a stored value. Semantically, the value is a\n/// std::optional<std::variant<T, std::exception_ptr>>. The actual\n/// type of the value in the implementation is future_state<T>.\n///\n/// A future without an initial value can be created by first creating\n/// a promise and then calling promise::get_future. The promise also\n/// stores a future_state<T> in case promise::set_value is called\n/// before get_future.\n///\n/// In addition to the future_state<T>, the promise and the future\n/// point to each other and the pointers are updated when either is\n/// moved.\n///\n/// If a future is consumed by future::then before the future is\n/// ready, a continuation is dynamically allocated. The continuation\n/// also has a future_state<T>, but unlinke a future it is never\n/// moved.\n///\n/// After a future creates a continuation, the corresponding promise\n/// points to the newly allocated continuation. When\n/// promise::set_value is called, the continuation is ready and is\n/// scheduled.\n///\n/// A promise then consists of\n/// * A future_state<T> for use when there is no corresponding future\n///   or continuation (_local_state).\n/// * A pointer to a future to allow updates when the promise is moved\n///  (_future).\n/// * A pointer to the continuation (_task).\n/// * A pointer to future_state<T> (_state) that can point to\n///   1. The future_state<T> in the promise itself\n///   2. The future_state<T> in the future\n///   2. The future_state<T> in the continuation\n///\n/// A special case is when a future blocks inside a thread. In that\n/// case we still need a continuation, but that continuation doesn't\n/// need a future_state<T> since the original future still exists on\n/// the stack.\n///\n/// So the valid states for a promise are:\n///\n/// 1. A newly created promise. _state points to _local_state and\n///    _task and _future are null.\n/// 2. After get_future is called. _state points to the state in the\n///    future, _future points to the future and _task is null.\n/// 3. The future has been consumed by future::then. Now the _state\n///    points to the state in the continuation, _future is null and\n///    _task points to the continuation.\n/// 4. A call to future::get is blocked in a thread. This is a mix of\n///    cases 2 and 3. Like 2, there is a valid future and _future and\n///    _state point to the future and its state. Like 3, there is a\n///    valid continuation and _task points to it, but that\n///    continuation has no state of its own.\n\n/// \\defgroup future-util Future Utilities\n/// \\ingroup future-module\n///\n/// \\brief\n/// These utilities are provided to help perform operations on futures.\n\n\n/// \\addtogroup future-module\n/// @{\ntemplate <class T = void>\nclass promise;\n\ntemplate <class T>\nclass future;\n\ntemplate <typename... T>\nclass shared_future;\n\nstruct future_state_base;\n\n/// \\brief Creates a \\ref future in an available, value state.\n///\n/// Creates a \\ref future object that is already resolved.  This\n/// is useful when it is determined that no I/O needs to be performed\n/// to perform a computation (for example, because the data is cached\n/// in some buffer).\ntemplate <typename T = void, typename... A>\nfuture<T> make_ready_future(A&&... value) noexcept;\n\n\n/// \\brief Returns a ready \\ref future that is already resolved.\ntemplate<typename T>\ninline\nfuture<std::remove_cv_t<std::remove_reference_t<T>>> as_ready_future(T&& v) noexcept {\n    return make_ready_future<std::remove_cv_t<std::remove_reference_t<T>>>(\n        std::forward<T>(v));\n}\n\n/// \\brief Creates a \\ref future in an available, failed state.\n///\n/// Creates a \\ref future object that is already resolved in a failed\n/// state.  This is useful when no I/O needs to be performed to perform\n/// a computation (for example, because the connection is closed and\n/// we cannot read from it).\ntemplate <typename T = void>\nfuture<T> make_exception_future(std::exception_ptr&& value) noexcept;\n\ntemplate <typename T = void, typename Exception>\nfuture<T> make_exception_future(Exception&& ex) noexcept;\n\ntemplate <typename T = void>\nfuture<T> make_exception_future(const std::exception_ptr& ex) noexcept {\n    return make_exception_future<T>(std::exception_ptr(ex));\n}\n\ntemplate <typename T = void>\nfuture<T> make_exception_future(std::exception_ptr& ex) noexcept {\n    return make_exception_future<T>(static_cast<const std::exception_ptr&>(ex));\n}\n\ntemplate <typename T = void>\nfuture<T> make_exception_future(const std::exception_ptr&& ex) noexcept {\n    // as ex is const, we cannot move it, but can copy it.\n    return make_exception_future<T>(std::exception_ptr(ex));\n}\n/// \\cond internal\nvoid engine_exit(std::exception_ptr eptr = {});\n\n/// \\endcond\n\n/// \\brief Exception type for broken promises\n///\n/// When a promise is broken, i.e. a promise object with an attached\n/// continuation is destroyed before setting any value or exception, an\n/// exception of `broken_promise` type is propagated to that abandoned\n/// continuation.\nstruct broken_promise : std::logic_error {\n    broken_promise();\n};\n\n/// \\brief Returns std::current_exception() wrapped in a future\n///\n/// This is equivalent to\n/// make_exception_future(std::current_exception()), but expands to\n/// less code.\ntemplate <typename T = void>\nfuture<T> current_exception_as_future() noexcept;\n\nextern template\nfuture<void> current_exception_as_future() noexcept;\n\nnamespace internal {\ntemplate <class T = void>\nclass promise_base_with_type;\nclass promise_base;\n\nstruct monostate {};\n\ntemplate <typename... T>\nstruct future_stored_type;\n\ntemplate <>\nstruct future_stored_type<> {\n    using type = monostate;\n};\n\ntemplate <typename T>\nstruct future_stored_type<T> {\n    using type = std::conditional_t<std::is_void_v<T>, internal::monostate, T>;\n};\n\ntemplate <typename... T>\nusing future_stored_type_t = typename future_stored_type<T...>::type;\n\ntemplate<typename T>\nusing future_tuple_type_t = std::conditional_t<std::is_same_v<T, monostate>, std::tuple<>, std::tuple<T>>;\n\n// It doesn't seem to be possible to use std::tuple_element_t with an empty tuple. There is an static_assert in it that\n// fails the build even if it is in the non enabled side of std::conditional.\ntemplate <typename T>\nstruct get0_return_type;\n\ntemplate <>\nstruct get0_return_type<internal::monostate> {\n    using type = void;\n    static type get0(internal::monostate) { }\n};\n\ntemplate <typename T>\nstruct get0_return_type {\n    using type = T;\n    static T get0(T&& v) { return std::move(v); }\n};\n\ntemplate<typename T>\nusing maybe_wrap_ref = std::conditional_t<std::is_reference_v<T>, std::reference_wrapper<std::remove_reference_t<T>>, T>;\n\n/// \\brief Wrapper for keeping uninitialized values of non default constructible types.\n///\n/// This is similar to a std::optional<T>, but it doesn't know if it is holding a value or not, so the user is\n/// responsible for calling constructors and destructors.\n\ntemplate <typename T>\nstruct uninitialized_wrapper {\n    using tuple_type = future_tuple_type_t<T>;\n    [[no_unique_address]] union any {\n        any() noexcept {}\n        ~any() {}\n        // T can be a reference, so wrap it.\n        [[no_unique_address]] maybe_wrap_ref<T> value;\n    } _v;\n\npublic:\n    uninitialized_wrapper() noexcept = default;\n    template<typename... U>\n    requires (!std::same_as<std::tuple<std::remove_cv_t<U>...>, std::tuple<tuple_type>>)\n    void\n    uninitialized_set(U&&... vs) {\n        new (&_v.value) maybe_wrap_ref<T>(T(std::forward<U>(vs)...));\n    }\n    void uninitialized_set(tuple_type&& v) {\n        uninitialized_set(std::move(std::get<0>(v)));\n    }\n    void uninitialized_set(const tuple_type& v) {\n        uninitialized_set(std::get<0>(v));\n    }\n    maybe_wrap_ref<T>& uninitialized_get() {\n        return _v.value;\n    }\n    const maybe_wrap_ref<T>& uninitialized_get() const {\n        return _v.value;\n    }\n};\n\ntemplate <>\nstruct uninitialized_wrapper<internal::monostate> {\n    [[no_unique_address]] internal::monostate _v;\npublic:\n    uninitialized_wrapper() noexcept = default;\n    void uninitialized_set() {\n    }\n    void uninitialized_set(internal::monostate) {\n    }\n    void uninitialized_set(std::tuple<>&& v) {\n    }\n    void uninitialized_set(const std::tuple<>& v) {\n    }\n    internal::monostate& uninitialized_get() {\n        return _v;\n    }\n    const internal::monostate& uninitialized_get() const {\n        return _v;\n    }\n};\n\ntemplate <typename T>\nstruct is_trivially_move_constructible_and_destructible {\n    static constexpr bool value = std::is_trivially_move_constructible_v<T> && std::is_trivially_destructible_v<T>;\n};\n\ntemplate <bool... v>\nstruct all_true : std::false_type {};\n\ntemplate <>\nstruct all_true<> : std::true_type {};\n\ntemplate <bool... v>\nstruct all_true<true, v...> : public all_true<v...> {};\n\ntemplate<typename T>\nstruct is_tuple_effectively_trivially_move_constructible_and_destructible_helper;\n\ntemplate <typename... T>\nstruct is_tuple_effectively_trivially_move_constructible_and_destructible_helper<std::tuple<T...>> {\n    static constexpr bool value = all_true<is_trivially_move_constructible_and_destructible<T>::value...>::value;\n};\n\ntemplate <typename T>\nstatic constexpr bool is_tuple_effectively_trivially_move_constructible_and_destructible =\n    is_tuple_effectively_trivially_move_constructible_and_destructible_helper<T>::value;\n\n}\n\n// Helper to convert lvalue reference functions to rvalue.\n// This is used early in then()-alike entry points to convert lvalue references into\n// rvalues (which are used to recursively call back into the same entry point) to\n// avoid having to handle both lvalues and rvalues downstream: we convert them all\n// to rvalues. This helper handles both references to callable objects such as lambdas,\n// copying them, and references to functions, decaying them to function pointers.\ntemplate <typename Func>\nstatic constexpr auto func_to_rvalue(Func&& func) {\n    static_assert(std::is_lvalue_reference_v<Func>, \"call with lvalue reference\");\n    if constexpr (std::is_function_v<std::remove_reference_t<Func>>) {\n        // return decayed function pointer\n        return +func;\n    } else {\n        // return a copy (used as an rvalue by the called)\n        return std::forward<Func>(func);\n    }\n}\n\n\n//\n// A future/promise pair maintain one logical value (a future_state).\n// There are up to three places that can store it, but only one is\n// active at any time.\n//\n// - in the promise _local_state member variable\n//\n//   This is necessary because a promise is created first and there\n//   would be nowhere else to put the value.\n//\n// - in the future _state variable\n//\n//   This is used anytime a future exists and then has not been called\n//   yet. This guarantees a simple access to the value for any code\n//   that already has a future.\n//\n// - in the task associated with the .then() clause (after .then() is called,\n//   if a value was not set)\n//\n//\n// The promise maintains a pointer to the state, which is modified as\n// the state moves to a new location due to events (such as .then() or\n// get_future being called) or due to the promise or future being\n// moved around.\n//\n\n// non templated base class to reduce code duplication\nstruct future_state_base {\n    static_assert(sizeof(std::exception_ptr) == sizeof(void*), \"exception_ptr not a pointer\");\n    enum class state : uintptr_t {\n         invalid = 0,\n         future = 1,\n         // the substate is intended to decouple the run-time prevention\n         // for duplicative result extraction (calling e.g. then() twice\n         // ends up in abandoned()) from the wrapped object's destruction\n         // handling which is orchestrated by future_state. Instead of\n         // creating a temporary future_state just for the sake of setting\n         // the \"invalid\" in the source instance, result_unavailable can\n         // be set to ensure future_state_base::available() returns false.\n         result_unavailable = 2,\n         result = 3,\n         exception_min = 4,  // or anything greater\n    };\n    union any {\n        any() noexcept { st = state::future; }\n        any(state s) noexcept { st = s; }\n        void set_exception(std::exception_ptr&& e) noexcept {\n            new (&ex) std::exception_ptr(std::move(e));\n            assert(st >= state::exception_min);\n        }\n        any(std::exception_ptr&& e) noexcept {\n            set_exception(std::move(e));\n        }\n        // From a users' perspective, a result_unavailable is not valid\n        bool valid() const noexcept { return st != state::invalid && st != state::result_unavailable; }\n        bool available() const noexcept { return st == state::result || st >= state::exception_min; }\n        bool failed() const noexcept { return __builtin_expect(st >= state::exception_min, false); }\n        void check_failure() noexcept;\n        ~any() noexcept { }\n        std::exception_ptr take_exception() noexcept {\n            std::exception_ptr ret(std::move(ex));\n            // Unfortunately in libstdc++ ~exception_ptr is defined out of line. We know that it does nothing for\n            // moved out values, so we omit calling it. This is critical for the code quality produced for this\n            // function. Without the out of line call, gcc can figure out that both sides of the if produce\n            // identical code and merges them.if\n            // We don't make any assumptions about other c++ libraries.\n            // There is request with gcc to define it inline: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90295\n#ifndef __GLIBCXX__\n            ex.~exception_ptr();\n#endif\n            st = state::invalid;\n            return ret;\n        }\n        void move_it(any&& x) noexcept {\n#ifdef __GLIBCXX__\n            // Unfortunally gcc cannot fully optimize the regular\n            // implementation:\n            // https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95014\n            // Given what we know about the libstdc++ implementation\n            // (see the comment in take_exception), we can just\n            // memmove and zero x.  We use memmove to guarantee\n            // vaild results if &x == this.\n            memmove(static_cast<void*>(this), &x, sizeof(any));\n            x.st = state::invalid;\n#else\n            if (x.st < state::exception_min) {\n                st = x.st;\n                x.st = state::invalid;\n            } else {\n                new (&ex) std::exception_ptr(x.take_exception());\n            }\n#endif\n        }\n        any(any&& x) noexcept {\n            move_it(std::move(x));\n        }\n        any& operator=(any&& x) noexcept {\n            check_failure();\n            // If this is a self move assignment, check_failure\n            // guarantees that we don't have an exception and calling\n            // move_it is safe.\n            move_it(std::move(x));\n            return *this;\n        }\n        bool has_result() const noexcept {\n            return st == state::result || st == state::result_unavailable;\n        }\n        state st;\n        std::exception_ptr ex;\n    } _u;\n\n    future_state_base() noexcept = default;\n    future_state_base(state st) noexcept : _u(st) { }\n    future_state_base(std::exception_ptr&& ex) noexcept : _u(std::move(ex)) { }\n    future_state_base(future_state_base&& x) noexcept : _u(std::move(x._u)) { }\n\n    // We never need to destruct this polymorphicly, so we can make it\n    // protected instead of virtual.\nprotected:\n    struct current_exception_future_marker {};\n    future_state_base(current_exception_future_marker) noexcept;\n    struct nested_exception_marker {};\n    future_state_base(nested_exception_marker, future_state_base&& old) noexcept;\n    future_state_base(nested_exception_marker, future_state_base&& n, future_state_base&& old) noexcept;\n    ~future_state_base() noexcept = default;\n\n    void rethrow_exception() &&;\n    void rethrow_exception() const&;\n\npublic:\n\n    bool valid() const noexcept { return _u.valid(); }\n    bool available() const noexcept { return _u.available(); }\n    bool failed() const noexcept { return _u.failed(); }\n\n    void ignore() noexcept;\n\n    void set_exception(std::exception_ptr&& ex) noexcept {\n        assert(_u.st == state::future);\n        _u.set_exception(std::move(ex));\n    }\n    future_state_base& operator=(future_state_base&& x) noexcept = default;\n    void set_exception(future_state_base&& state) noexcept {\n        assert(_u.st == state::future);\n        *this = std::move(state);\n    }\n    std::exception_ptr get_exception() && noexcept {\n        assert(_u.st >= state::exception_min);\n        // Move ex out so future::~future() knows we've handled it\n        return _u.take_exception();\n    }\n    const std::exception_ptr& get_exception() const& noexcept {\n        assert(_u.st >= state::exception_min);\n        return _u.ex;\n    }\n    template <typename U>\n    friend struct future_state;\n    template <typename U>\n    friend future<U> current_exception_as_future() noexcept;\n    template <typename U>\n    friend class future;\n    template <typename T>\n    friend struct futurize;\n};\n\nnamespace internal {\nvoid report_failed_future(const std::exception_ptr& ex) noexcept;\nvoid report_failed_future(const future_state_base& state) noexcept;\nvoid report_failed_future(future_state_base::any&& state) noexcept;\n} // internal namespace\n\n\ninline void future_state_base::any::check_failure() noexcept {\n    if (failed()) {\n        internal::report_failed_future(std::move(*this));\n    }\n}\n\nstruct ready_future_marker {};\nstruct exception_future_marker {};\nstruct future_for_get_promise_marker {};\n\n/// \\cond internal\ntemplate <typename T>\nstruct future_state :  public future_state_base, private internal::uninitialized_wrapper<T> {\n    static constexpr bool copy_noexcept = std::is_nothrow_copy_constructible_v<T>;\n    static constexpr bool has_trivial_move_and_destroy = internal::is_trivially_move_constructible_and_destructible<T>::value;\n    static_assert(std::is_nothrow_move_constructible_v<T>,\n                  \"Types must be no-throw move constructible\");\n    static_assert(std::is_nothrow_destructible_v<T>,\n                  \"Types must be no-throw destructible\");\n    future_state() noexcept = default;\n    void move_it(future_state&& x) noexcept {\n        if constexpr (has_trivial_move_and_destroy) {\n#pragma GCC diagnostic push\n            // This function may copy uninitialized memory, such as when\n            // creating an uninitialized promise and calling get_future()\n            // on it. Gcc 12 started to catch some simple cases of this\n            // at compile time, so we need to tell it that it's fine.\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n            memmove(reinterpret_cast<char*>(&this->uninitialized_get()),\n                   &x.uninitialized_get(),\n                   internal::used_size<internal::maybe_wrap_ref<T>>::value);\n#pragma GCC diagnostic pop\n        } else if (_u.has_result()) {\n            this->uninitialized_set(std::move(x.uninitialized_get()));\n            std::destroy_at(&x.uninitialized_get());\n        }\n    }\n\n    [[gnu::always_inline]]\n    future_state(future_state&& x) noexcept : future_state_base(std::move(x)) {\n        move_it(std::move(x));\n    }\n\n    void clear() noexcept {\n        if (_u.has_result()) {\n            std::destroy_at(&this->uninitialized_get());\n        } else {\n            _u.check_failure();\n        }\n    }\n    __attribute__((always_inline))\n    ~future_state() noexcept {\n        clear();\n    }\n    future_state& operator=(future_state&& x) noexcept {\n        clear();\n        future_state_base::operator=(std::move(x));\n        // If &x == this, _u.st is now state::invalid and so it is\n        // safe to call move_it.\n        move_it(std::move(x));\n        return *this;\n    }\n    template <typename... A>\n    future_state(ready_future_marker, A&&... a) noexcept : future_state_base(state::result) {\n      try {\n        this->uninitialized_set(std::forward<A>(a)...);\n      } catch (...) {\n        new (this) future_state(current_exception_future_marker());\n      }\n    }\n    template <typename... A>\n    void set(A&&... a) noexcept {\n        assert(_u.st == state::future);\n        new (this) future_state(ready_future_marker(), std::forward<A>(a)...);\n    }\n    future_state(exception_future_marker, std::exception_ptr&& ex) noexcept : future_state_base(std::move(ex)) { }\n    future_state(exception_future_marker, future_state_base&& state) noexcept : future_state_base(std::move(state)) { }\n    future_state(current_exception_future_marker m) noexcept : future_state_base(m) { }\n    future_state(nested_exception_marker m, future_state_base&& old) noexcept : future_state_base(m, std::move(old)) { }\n    future_state(nested_exception_marker m, future_state_base&& n, future_state_base&& old) noexcept : future_state_base(m, std::move(n), std::move(old)) { }\n    T&& get_value() && noexcept {\n        assert(_u.st == state::result);\n        return static_cast<T&&>(this->uninitialized_get());\n    }\n    T&& take_value() && noexcept {\n        assert(_u.st == state::result);\n        _u.st = state::result_unavailable;\n        return static_cast<T&&>(this->uninitialized_get());\n    }\n    template<std::copy_constructible U = T>\n    const U& get_value() const& noexcept(copy_noexcept) {\n        assert(_u.st == state::result);\n        return this->uninitialized_get();\n    }\n    T&& take() && {\n        assert(available());\n        if (_u.st >= state::exception_min) {\n            std::move(*this).rethrow_exception();\n        }\n        _u.st = state::result_unavailable;\n        return static_cast<T&&>(this->uninitialized_get());\n    }\n    T&& get() && {\n        assert(available());\n        if (_u.st >= state::exception_min) {\n            std::move(*this).rethrow_exception();\n        }\n        return static_cast<T&&>(this->uninitialized_get());\n    }\n    const T& get() const& {\n        assert(available());\n        if (_u.st >= state::exception_min) {\n            rethrow_exception();\n        }\n        return this->uninitialized_get();\n    }\n    using get0_return_type = typename internal::get0_return_type<T>::type;\n    static get0_return_type get0(T&& x) {\n        return internal::get0_return_type<T>::get0(std::move(x));\n    }\n\n    get0_return_type get0() {\n        return std::move(*this).get();\n    }\n};\n\ntemplate <typename T = void>\nclass continuation_base : public task {\nprotected:\n    using future_state = seastar::future_state<internal::future_stored_type_t<T>>;\n    future_state _state;\n    using future_type = future<T>;\n    using promise_type = promise<T>;\npublic:\n    continuation_base() noexcept = default;\n    void set_state(future_state&& state) noexcept {\n        _state = std::move(state);\n    }\n    // This override of waiting_task() is needed here because there are cases\n    // when backtrace is obtained from the destructor of this class and objects\n    // of derived classes are already destroyed at that time. If we didn't\n    // have this override we would get a \"pure virtual function call\" exception.\n    virtual task* waiting_task() noexcept override { return nullptr; }\n    friend class internal::promise_base_with_type<T>;\n    friend class promise<T>;\n    friend class future<T>;\n};\n\n// Given a future type, find the corresponding continuation_base.\ntemplate <typename Future>\nstruct continuation_base_from_future;\n\ntemplate <typename... T>\nstruct continuation_base_from_future<future<T...>> {\n    using type = continuation_base<T...>;\n};\n\ntemplate <typename Future>\nusing continuation_base_from_future_t = typename continuation_base_from_future<Future>::type;\n\ntemplate <typename Promise, typename T = void>\nclass continuation_base_with_promise : public continuation_base<T> {\n    friend class internal::promise_base_with_type<T>;\nprotected:\n    continuation_base_with_promise(Promise&& pr) noexcept : _pr(std::move(pr)) {\n        task::make_backtrace();\n    }\n    virtual task* waiting_task() noexcept override;\n    Promise _pr;\n};\n\ntemplate <typename Promise, typename Func, typename Wrapper, typename T = void>\nstruct continuation final : continuation_base_with_promise<Promise, T> {\n    // Func is the original function passed to then/then_wrapped. The\n    // Wrapper is a helper function that implements the specific logic\n    // needed by then/then_wrapped. We call the wrapper passing it the\n    // original function, promise and state.\n    // Note that if Func's move constructor throws, this will call\n    // std::unexpected. We could try to require Func to be nothrow\n    // move constructible, but that will cause a lot of churn. Since\n    // we can't support a failure to create a continuation, calling\n    // std::unexpected as close to the failure as possible is the best\n    // we can do.\n    continuation(Promise&& pr, Func&& func, Wrapper&& wrapper) noexcept\n        : continuation_base_with_promise<Promise, T>(std::move(pr))\n        , _func(std::move(func))\n        , _wrapper(std::move(wrapper)) {}\n    virtual void run_and_dispose() noexcept override {\n        try {\n            _wrapper(std::move(this->_pr), std::move(_func), std::move(this->_state));\n        } catch (...) {\n            this->_pr.set_to_current_exception();\n        }\n        delete this;\n    }\n    Func _func;\n    [[no_unique_address]] Wrapper _wrapper;\n};\n\nnamespace internal {\n\ntemplate <typename T = void>\nfuture<T> make_exception_future(future_state_base&& state) noexcept;\n\ntemplate <typename T = void>\nvoid set_callback(future<T>&& fut, continuation_base<T>* callback) noexcept;\n\nclass future_base;\n\nclass promise_base {\nprotected:\n    enum class urgent { no, yes };\n    future_base* _future = nullptr;\n\n    // This points to the future_state that is currently being\n    // used. See comment above the future_state struct definition for\n    // details.\n    future_state_base* _state;\n\n    task* _task = nullptr;\n#ifdef SEASTAR_DEBUG_PROMISE\n    int _task_shard = -1;\n\n    void set_task(task* task) noexcept {\n        _task = task;\n        _task_shard = this_shard_id();\n    }\n    void assert_task_shard() const noexcept;\n#else\n    void set_task(task* task) noexcept {\n        _task = task;\n    }\n    void assert_task_shard() const noexcept { }\n#endif\n\n    promise_base(const promise_base&) = delete;\n    promise_base(future_state_base* state) noexcept : _state(state) {}\n    promise_base(future_base* future, future_state_base* state) noexcept;\n    void move_it(promise_base&& x) noexcept;\n    promise_base(promise_base&& x) noexcept;\n\n    void clear() noexcept;\n\n    // We never need to destruct this polymorphicly, so we can make it\n    // protected instead of virtual\n    ~promise_base() noexcept {\n        clear();\n    }\n\n    void operator=(const promise_base&) = delete;\n    promise_base& operator=(promise_base&& x) noexcept;\n\n    template<urgent Urgent>\n    void make_ready() noexcept;\n\n    template<typename T>\n    void set_exception_impl(T&& val) noexcept {\n        if (_state) {\n            _state->set_exception(std::move(val));\n            make_ready<urgent::no>();\n        } else {\n            // We get here if promise::get_future is called and the\n            // returned future is destroyed without creating a\n            // continuation.\n            // In older versions of seastar we would store a local\n            // copy of ex and warn in the promise destructor.\n            // Since there isn't any way for the user to clear\n            // the exception, we issue the warning from here.\n            internal::report_failed_future(val);\n        }\n    }\n\n    void set_exception(future_state_base&& state) noexcept {\n        set_exception_impl(std::move(state));\n    }\n\n    void set_exception(std::exception_ptr&& ex) noexcept {\n        set_exception_impl(std::move(ex));\n    }\n\n    void set_exception(const std::exception_ptr& ex) noexcept {\n        set_exception(std::exception_ptr(ex));\n    }\n\n    template<typename Exception>\n    requires (!std::same_as<std::remove_reference_t<Exception>, std::exception_ptr>)\n    void set_exception(Exception&& e) noexcept {\n        set_exception(std::make_exception_ptr(std::forward<Exception>(e)));\n    }\n\n    friend class future_base;\n    template <typename U> friend class seastar::future;\n\npublic:\n    /// Set this promise to the current exception.\n    ///\n    /// This is equivalent to set_exception(std::current_exception()),\n    /// but expands to less code.\n    void set_to_current_exception() noexcept;\n\n    /// Returns the task which is waiting for this promise to resolve, or nullptr.\n    task* waiting_task() const noexcept { return _task; }\n};\n\n/// \\brief A promise with type but no local data.\n///\n/// This is a promise without any local data. We use this for when the\n/// future is created first, so we know the promise always has an\n/// external place to point to. We cannot just use promise_base\n/// because we need to know the type that is being stored.\ntemplate <typename T>\nclass promise_base_with_type : protected internal::promise_base {\nprotected:\n    using future_state = seastar::future_state<future_stored_type_t<T>>;\n    future_state* get_state() noexcept {\n        return static_cast<future_state*>(_state);\n    }\n    static constexpr bool copy_noexcept = future_state::copy_noexcept;\npublic:\n    promise_base_with_type(future_state_base* state) noexcept : promise_base(state) { }\n    promise_base_with_type(future<T>* future) noexcept : promise_base(future, &future->_state) { }\n    promise_base_with_type(promise_base_with_type&& x) noexcept = default;\n    promise_base_with_type(const promise_base_with_type&) = delete;\n    promise_base_with_type& operator=(promise_base_with_type&& x) noexcept = default;\n    void operator=(const promise_base_with_type&) = delete;\n\n    void set_urgent_state(future_state&& state) noexcept {\n        auto* ptr = get_state();\n        // The state can be null if the corresponding future has been\n        // destroyed without producing a continuation.\n        if (ptr) {\n            assert(ptr->_u.st == future_state_base::state::future);\n            new (ptr) future_state(std::move(state));\n            make_ready<urgent::yes>();\n        }\n    }\n\n    template <typename... A>\n    void set_value(A&&... a) noexcept {\n        if (auto *s = get_state()) {\n            s->set(std::forward<A>(a)...);\n            make_ready<urgent::no>();\n        }\n    }\n\n    /// Set this promise to the current exception.\n    ///\n    /// This is equivalent to set_exception(std::current_exception()),\n    /// but expands to less code.\n    void set_to_current_exception() noexcept {\n        internal::promise_base::set_to_current_exception();\n    }\n\n    /// Returns the task which is waiting for this promise to resolve, or nullptr.\n    using internal::promise_base::waiting_task;\n\nprivate:\n\n    template <typename U>\n    friend class seastar::future;\n\n    friend future_state;\n};\n}\n/// \\endcond\n\n/// \\brief promise - allows a future value to be made available at a later time.\n///\n/// \\tparam T A type to be carried as the result of the associated future. Use void (default) for no result.\ntemplate <typename T>\nclass promise : private internal::promise_base_with_type<T> {\n    using future_state = typename internal::promise_base_with_type<T>::future_state;\n    future_state _local_state;\n\npublic:\n    /// \\brief Constructs an empty \\c promise.\n    ///\n    /// Creates promise with no associated future yet (see get_future()).\n    promise() noexcept : internal::promise_base_with_type<T>(&_local_state) {}\n\n    /// \\brief Moves a \\c promise object.\n    void move_it(promise&& x) noexcept;\n    promise(promise&& x) noexcept : internal::promise_base_with_type<T>(std::move(x)) {\n        move_it(std::move(x));\n    }\n    promise(const promise&) = delete;\n    promise& operator=(promise&& x) noexcept {\n        internal::promise_base_with_type<T>::operator=(std::move(x));\n        // If this is a self-move, _state is now nullptr and it is\n        // safe to call move_it.\n        move_it(std::move(x));\n        return *this;\n    }\n    void operator=(const promise&) = delete;\n\n    /// Set this promise to the current exception.\n    ///\n    /// This is equivalent to set_exception(std::current_exception()),\n    /// but expands to less code.\n    void set_to_current_exception() noexcept {\n        internal::promise_base::set_to_current_exception();\n    }\n\n    /// Returns the task which is waiting for this promise to resolve, or nullptr.\n    using internal::promise_base::waiting_task;\n\n    /// \\brief Gets the promise's associated future.\n    ///\n    /// The future and promise will be remember each other, even if either or\n    /// both are moved.  When \\c set_value() or \\c set_exception() are called\n    /// on the promise, the future will be become ready, and if a continuation\n    /// was attached to the future, it will run.\n    future<T> get_future() noexcept;\n\n    /// \\brief Sets the promises value\n    ///\n    /// Forwards the arguments and makes them available to the associated\n    /// future.  May be called either before or after \\c get_future().\n    /// Must be called at most once, setting value on a promise with a value\n    /// is undefined behavior. Must not be called after \\c set_exception().\n    ///\n    /// The arguments can have either the types the promise is\n    /// templated with, or a corresponding std::tuple. That is, given\n    /// a promise<int, double>, both calls are valid:\n    ///\n    /// pr.set_value(42, 43.0);\n    /// pr.set_value(std::tuple<int, double>(42, 43.0))\n    template <typename... A>\n    void set_value(A&&... a) noexcept {\n        internal::promise_base_with_type<T>::set_value(std::forward<A>(a)...);\n    }\n\n    /// \\brief Marks the promise as failed\n    ///\n    /// Forwards the exception argument to the future and makes it\n    /// available.  May be called either before or after \\c get_future().\n    /// Just like \\c set_value(), must not be called more than once, must\n    /// not be called after set_value() itself either.\n    void set_exception(std::exception_ptr&& ex) noexcept {\n        internal::promise_base::set_exception(std::move(ex));\n    }\n\n    void set_exception(const std::exception_ptr& ex) noexcept {\n        internal::promise_base::set_exception(ex);\n    }\n\n    /// \\brief Marks the promise as failed\n    ///\n    /// Forwards the exception argument to the future and makes it\n    /// available.  May be called either before or after \\c get_future().\n    template<typename Exception>\n    requires (!std::same_as<std::remove_reference_t<Exception>, std::exception_ptr>)\n    void set_exception(Exception&& e) noexcept {\n        internal::promise_base::set_exception(std::forward<Exception>(e));\n    }\n\n    using internal::promise_base_with_type<T>::set_urgent_state;\n\n    template <typename U>\n    friend class future;\n};\n\n/// @}\n\n/// \\addtogroup future-util\n/// @{\n\n\n/// \\brief Check whether a type is a future\n///\n/// This is a type trait evaluating to \\c true if the given type is a\n/// future.\n///\ntemplate <typename... T> struct is_future : std::false_type {};\n\n/// \\cond internal\n/// \\addtogroup future-util\ntemplate <typename... T> struct is_future<future<T...>> : std::true_type {};\n\n/// \\endcond\n\n\n/// \\brief Converts a type to a future type, if it isn't already.\n///\n/// \\return Result in member type 'type'.\ntemplate <typename T>\nstruct futurize;\n\ntemplate <typename T>\nconcept Future = is_future<T>::value;\n\ntemplate <typename Func, typename... T>\nconcept CanInvoke = std::invocable<Func, T...>;\n\ntemplate <typename Func, typename... T>\nconcept CanApplyTuple\n    = sizeof...(T) == 1\n        && requires (Func func, std::tuple<T...> wrapped_val) {\n        { std::apply(func, std::get<0>(std::move(wrapped_val))) };\n    };\n\ntemplate <typename Func, typename... T>\nconcept InvokeReturnsAnyFuture = Future<std::invoke_result_t<Func, T...>>;\n\n/// \\endcond\n\n// Converts a type to a future type, if it isn't already.\ntemplate <typename T>\nusing futurize_t = typename futurize<T>::type;\n\n/// @}\n\ntemplate<typename Func, typename... Args>\nauto futurize_invoke(Func&& func, Args&&... args) noexcept;\n\ntemplate<typename Func, typename... Args>\nauto futurize_apply(Func&& func, std::tuple<Args...>&& args) noexcept;\n\n/// \\addtogroup future-module\n/// @{\nnamespace internal {\nclass future_base {\nprotected:\n    promise_base* _promise;\n    future_base() noexcept : _promise(nullptr) {}\n    future_base(promise_base* promise, future_state_base* state) noexcept : _promise(promise) {\n        _promise->_future = this;\n        _promise->_state = state;\n    }\n\n    void move_it(future_base&& x, future_state_base* state) noexcept {\n        _promise = x._promise;\n        if (auto* p = _promise) {\n            x.detach_promise();\n            p->_future = this;\n            p->_state = state;\n        }\n    }\n\n    future_base(future_base&& x, future_state_base* state) noexcept {\n        move_it(std::move(x), state);\n    }\n\n    void clear() noexcept {\n        if (_promise) {\n            detach_promise();\n        }\n    }\n\n    ~future_base() noexcept {\n        clear();\n    }\n\n    promise_base* detach_promise() noexcept {\n        _promise->_state = nullptr;\n        _promise->_future = nullptr;\n        return std::exchange(_promise, nullptr);\n    }\n\n    void schedule(task* tws, future_state_base* state) noexcept {\n        promise_base* p = detach_promise();\n        p->_state = state;\n        p->set_task(tws);\n    }\n\n    void do_wait() noexcept;\n\n    void set_coroutine(task& coroutine) noexcept;\n\n    friend class promise_base;\n};\n\ntemplate <typename Func, typename... T>\nstruct future_result  {\n    using type = std::invoke_result_t<Func, T...>;\n    using future_type = futurize_t<type>;\n    using func_type = future_type (T&&...);\n};\n\ntemplate <typename Func>\nstruct future_result<Func, void> {\n    using type = std::invoke_result_t<Func>;\n    using future_type = futurize_t<type>;\n    using func_type = future_type ();\n};\n\ntemplate <typename Func, typename T>\nusing future_result_t = typename future_result<Func, T>::type;\n\ntemplate <typename Func, typename T>\nauto future_invoke(Func&& func, T&& v) {\n    if constexpr (std::is_same_v<T, monostate>) {\n        return std::invoke(std::forward<Func>(func));\n    } else {\n        return std::invoke(std::forward<Func>(func), std::forward<T>(v));\n    }\n}\n\ntemplate <typename Func, typename... T>\nstruct result_of_apply {\n    // no \"type\" member if not a function call signature or not a tuple\n};\n\ntemplate <typename Func, typename... T>\nstruct result_of_apply<Func, std::tuple<T...>> : std::invoke_result<Func, T...> {\n    // Let std::invoke_result_t determine the result if the input is a tuple\n};\n\ntemplate <typename Func, typename... T>\nusing result_of_apply_t = typename result_of_apply<Func, T...>::type;\n\n// To simply implementation, at various top-level (user-callable) entry-points\n// like then(), finally(), handle_exception(), etc, we convert lvalue callables\n// into rvalues, using a recursive call back to that entry point, so that all\n// the downstream implementation functions can deal with rvalues only. Inside\n// those functions we then assert on this trait to ensure we do only have rvalues\n// since we can't match rvalue references directly (because that syntax was\n// co-opted for universal references).\ntemplate <typename Func>\nconstexpr bool expect_only_rvalue_refs = !std::is_lvalue_reference_v<Func>;\n}\n\ntemplate <typename Promise, typename T>\ntask* continuation_base_with_promise<Promise, T>::waiting_task() noexcept {\n    return _pr.waiting_task();\n}\n\n/// \\brief A representation of a possibly not-yet-computed value.\n///\n/// A \\c future represents a value that has not yet been computed\n/// (an asynchronous computation).  It can be in one of several\n/// states:\n///    - unavailable: the computation has not been completed yet\n///    - value: the computation has been completed successfully and a\n///      value is available.\n///    - failed: the computation completed with an exception.\n///\n/// methods in \\c future allow querying the state and, most importantly,\n/// scheduling a \\c continuation to be executed when the future becomes\n/// available.  Only one such continuation may be scheduled.\n///\n/// A \\ref future should not be discarded before it is waited upon and\n/// its result is extracted. Discarding a \\ref future means that the\n/// computed value becomes inaccessible, but more importantly, any\n/// exceptions raised from the computation will disappear unchecked as\n/// well. Another very important consequence is potentially unbounded\n/// resource consumption due to the launcher of the deserted\n/// continuation not being able track the amount of in-progress\n/// continuations, nor their individual resource consumption.\n/// To prevent accidental discarding of futures, \\ref future is\n/// declared `[[nodiscard]]` if the compiler supports it. Also, when a\n/// discarded \\ref future resolves with an error a warning is logged\n/// (at runtime).\n/// That said there can be legitimate cases where a \\ref future is\n/// discarded. The most prominent example is launching a new\n/// [fiber](\\ref fiber-module), or in other words, moving a continuation\n/// chain to the background (off the current [fiber](\\ref fiber-module)).\n/// Even if a \\ref future is discarded purposefully, it is still strongly\n/// advisable to wait on it indirectly (via a \\ref gate or\n/// \\ref semaphore), control their concurrency, their resource consumption\n/// and handle any errors raised from them.\n///\n/// \\tparam T A type to be carried as the result of the future, or void\n///           for no result. An available future<void> only\n///           contains a success/failure indication (and in the case of a\n///           failure, an exception).\ntemplate <typename T>\nclass [[nodiscard]] future : private internal::future_base {\n    using future_state = seastar::future_state<internal::future_stored_type_t<T>>;\n    future_state _state;\n    static constexpr bool copy_noexcept = future_state::copy_noexcept;\n\nprivate:\n    // This constructor creates a future that is not ready but has no\n    // associated promise yet. The use case is to have a less flexible\n    // but more efficient future/promise pair where we know that\n    // promise::set_value cannot possibly be called without a matching\n    // future and so that promise doesn't need to store a\n    // future_state.\n    future(future_for_get_promise_marker) noexcept { }\n\n    future(promise<T>* pr) noexcept : future_base(pr, &_state), _state(std::move(pr->_local_state)) { }\n    template <typename... A>\n    future(ready_future_marker m, A&&... a) noexcept : _state(m, std::forward<A>(a)...) { }\n    future(future_state_base::current_exception_future_marker m) noexcept : _state(m) {}\n    future(future_state_base::nested_exception_marker m, future_state_base&& old) noexcept : _state(m, std::move(old)) {}\n    future(future_state_base::nested_exception_marker m, future_state_base&& n, future_state_base&& old) noexcept : _state(m, std::move(n), std::move(old)) {}\n    future(exception_future_marker m, std::exception_ptr&& ex) noexcept : _state(m, std::move(ex)) { }\n    future(exception_future_marker m, future_state_base&& state) noexcept : _state(m, std::move(state)) { }\n    [[gnu::always_inline]]\n    explicit future(future_state&& state) noexcept\n            : _state(std::move(state)) {\n    }\n    internal::promise_base_with_type<T> get_promise() noexcept {\n        assert(!_promise);\n        return internal::promise_base_with_type<T>(this);\n    }\n    internal::promise_base_with_type<T>* detach_promise() noexcept {\n        return static_cast<internal::promise_base_with_type<T>*>(future_base::detach_promise());\n    }\n    void schedule(continuation_base<T>* tws) noexcept {\n        future_base::schedule(tws, &tws->_state);\n    }\n    template <typename Pr, typename Func, typename Wrapper>\n    void schedule(Pr&& pr, Func&& func, Wrapper&& wrapper, std::source_location sl) noexcept {\n        static_assert(internal::expect_only_rvalue_refs<Func>);\n        // If this new throws a std::bad_alloc there is nothing that\n        // can be done about it. The corresponding future is not ready\n        // and we cannot break the chain. Since this function is\n        // noexcept, it will call std::terminate if new throws.\n        memory::scoped_critical_alloc_section _;\n        auto tws = new continuation<Pr, Func, Wrapper, T>(std::move(pr), std::move(func), std::move(wrapper));\n        tws->update_resume_point(sl);\n        // In a debug build we schedule ready futures, but not in\n        // other build modes.\n#ifdef SEASTAR_DEBUG\n        if (_state.available()) {\n            tws->set_state(get_available_state_ref());\n            ::seastar::schedule(tws);\n            return;\n        }\n#endif\n        schedule(tws);\n        _state._u.st = future_state_base::state::invalid;\n    }\n\n    [[gnu::always_inline]]\n    future_state&& get_available_state_ref() noexcept {\n        if (_promise) {\n            detach_promise();\n        }\n        return std::move(_state);\n    }\n\n    future<T> rethrow_with_nested(future_state_base&& n) noexcept {\n        return future<T>(future_state_base::nested_exception_marker(), std::move(n), std::move(_state));\n    }\n\n    future<T> rethrow_with_nested() noexcept {\n        return future<T>(future_state_base::nested_exception_marker(), std::move(_state));\n    }\n\n    template<typename... U>\n    friend class shared_future;\npublic:\n    /// \\brief The data type carried by the future.\n    using value_type = internal::future_stored_type_t<T>;\n    using tuple_type = internal::future_tuple_type_t<value_type>;\n    /// \\brief The data type carried by the future.\n    using promise_type = promise<T>;\n    /// \\brief Moves the future into a new object.\n    [[gnu::always_inline]]\n    future(future&& x) noexcept : future_base(std::move(x), &_state), _state(std::move(x._state)) { }\n    future(const future&) = delete;\n    future& operator=(future&& x) noexcept {\n        clear();\n        move_it(std::move(x), &_state);\n        _state = std::move(x._state);\n        return *this;\n    }\n    void operator=(const future&) = delete;\n    /// \\brief gets the value returned by the computation\n    ///\n    /// Requires that the future be available.  If the value\n    /// was computed successfully, it is returned (as an\n    /// \\c std::tuple).  Otherwise, an exception is thrown.\n    ///\n    /// If get() is called in a \\ref seastar::thread context,\n    /// then it need not be available; instead, the thread will\n    /// be paused until the future becomes available.\n    [[gnu::always_inline]]\n    value_type&& get() {\n        wait();\n        return get_available_state_ref().take();\n    }\n\n    [[gnu::always_inline]]\n    std::exception_ptr get_exception() noexcept {\n        return get_available_state_ref().get_exception();\n    }\n\n    using get0_return_type = typename future_state::get0_return_type;\n\n    /// Wait for the future to be available (in a seastar::thread)\n    ///\n    /// When called from a seastar::thread, this function blocks the\n    /// thread until the future is available. Other threads and\n    /// continuations continue to execute; only the thread is blocked.\n    void wait() noexcept {\n        if (_state.available()) {\n            return;\n        }\n        do_wait();\n    }\n\n    /// \\brief Checks whether the future is available.\n    ///\n    /// \\return \\c true if the future has a value, or has failed.\n    [[gnu::always_inline]]\n    bool available() const noexcept {\n        return _state.available();\n    }\n\n    /// \\brief Checks whether the future has failed.\n    ///\n    /// \\return \\c true if the future is available and has failed.\n    [[gnu::always_inline]]\n    bool failed() const noexcept {\n        return _state.failed();\n    }\n\n    /// \\brief Schedule a block of code to run when the future is ready.\n    ///\n    /// Schedules a function (often a lambda) to run when the future becomes\n    /// available.  The function is called with the result of this future's\n    /// computation as parameters.  The return value of the function becomes\n    /// the return value of then(), itself as a future; this allows then()\n    /// calls to be chained.\n    ///\n    /// If the future failed, the function is not called, and the exception\n    /// is propagated into the return value of then().\n    ///\n    /// The passed function is moved (if an rvalue) or copied (if an lvalue) into\n    /// the continuation, so their lifetime is automatically extended until the\n    /// continuation runs (but not further - if the function itself may suspend\n    /// you will need to ensure its lifetime is sufficiently long).\n    ///\n    /// \\param func - function to be called when the future becomes available,\n    ///               unless it has failed.\n    /// \\return a \\c future representing the return value of \\c func, applied\n    ///         to the eventual value of this future.\n    template <typename Func, typename Result = typename internal::future_result<Func, T>::future_type>\n    requires std::invocable<Func, T>\n                 || (std::same_as<void, T> && std::invocable<Func>)\n    Result\n    then(Func&& func, std::source_location sl = std::source_location::current()) noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return then(func_to_rvalue(func), sl);\n      } else {\n#ifndef SEASTAR_TYPE_ERASE_MORE\n        return then_impl(std::move(func), sl);\n#else\n        using func_type = typename internal::future_result<Func, T>::func_type;\n        noncopyable_function<func_type> ncf;\n        {\n            memory::scoped_critical_alloc_section _;\n            ncf = noncopyable_function<func_type>([func = std::forward<Func>(func)](auto&&... args) mutable {\n                return futurize_invoke(std::move(func), std::forward<decltype(args)>(args)...);\n            });\n        }\n        return then_impl(std::move(ncf), sl);\n#endif\n      }\n    }\n\n    /// \\brief Schedule a block of code to run when the future is ready, unpacking tuples.\n    ///\n    /// Schedules a function (often a lambda) to run when the future becomes\n    /// available.  The function is called with the result of this future's\n    /// computation as parameters.  The return value of the function becomes\n    /// the return value of then(), itself as a future; this allows then()\n    /// calls to be chained.\n    ///\n    /// This member function is only available when the payload is std::tuple;\n    /// The tuple elements are passed as individual arguments to `func`, which\n    /// must have the same arity as the tuple.\n    ///\n    /// If the future failed, the function is not called, and the exception\n    /// is propagated into the return value of then().\n    ///\n    /// \\param func - function to be called when the future becomes available,\n    ///               unless it has failed.\n    /// \\return a \\c future representing the return value of \\c func, applied\n    ///         to the eventual value of this future.\n    template <typename Func, typename Result = futurize_t<internal::result_of_apply_t<Func, T>>>\n    requires ::seastar::CanApplyTuple<Func, T>\n    Result\n    then_unpack(Func&& func, std::source_location sl = std::source_location::current()) noexcept {\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return then_unpack(func_to_rvalue(func));\n      } else {\n        return then([func = std::forward<Func>(func)] (T&& tuple) mutable {\n            // sizeof...(tuple) is required to be 1\n            return std::apply(func, std::move(tuple));\n        }, sl);\n      }\n    }\n\nprivate:\n\n    // Keep this simple so that Named Return Value Optimization is used.\n    template <typename Func, typename Result>\n    Result then_impl_nrvo(Func&& func, std::source_location sl) noexcept {\n        using futurator = futurize<internal::future_result_t<Func, T>>;\n        typename futurator::type fut(future_for_get_promise_marker{});\n        using pr_type = decltype(fut.get_promise());\n        schedule(fut.get_promise(), std::move(func), [](pr_type&& pr, Func&& func, future_state&& state) {\n            if (state.failed()) {\n                pr.set_exception(static_cast<future_state_base&&>(std::move(state)));\n            } else {\n                futurator::satisfy_with_result_of(std::move(pr), [&func, &state] {\n                    // clang thinks that \"state\" is not used, below, for future<>.\n                    // Make it think it is used to avoid an unused-lambda-capture warning.\n                    (void)state;\n                    return internal::future_invoke(std::move(func), std::move(state).get_value());\n                });\n            }\n        }, sl);\n        return fut;\n    }\n\n    template <typename Func, typename Result = futurize_t<internal::future_result_t<Func, T>>>\n    Result\n    then_impl(Func&& func, std::source_location sl) noexcept {\n        static_assert(internal::expect_only_rvalue_refs<Func>);\n#ifndef SEASTAR_DEBUG\n        using futurator = futurize<internal::future_result_t<Func, T>>;\n        if (failed()) {\n            return futurator::make_exception_future(static_cast<future_state_base&&>(get_available_state_ref()));\n        } else if (available()) {\n            return futurator::invoke(std::forward<Func>(func), get_available_state_ref().take_value());\n        }\n#endif\n        return then_impl_nrvo<Func, Result>(std::forward<Func>(func), sl);\n    }\n\npublic:\n    /// \\brief Schedule a block of code to run when the future is ready, allowing\n    ///        for exception handling.\n    ///\n    /// Schedules a function (often a lambda) to run when the future becomes\n    /// available.  The function is called with the this future as a parameter;\n    /// it will be in an available state.  The return value of the function becomes\n    /// the return value of then_wrapped(), itself as a future; this allows\n    /// then_wrapped() calls to be chained.\n    ///\n    /// Unlike then(), the function will be called for both value and exceptional\n    /// futures.\n    ///\n    /// \\param func - function to be called when the future becomes available,\n    /// \\return a \\c future representing the return value of \\c func, applied\n    ///         to the eventual value of this future.\n    template <std::invocable<future> Func, typename FuncResult = std::invoke_result_t<Func, future>>\n    futurize_t<FuncResult>\n    then_wrapped(Func&& func, std::source_location sl = std::source_location::current()) & noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return then_wrapped(func_to_rvalue(func), sl);\n      } else {\n        return then_wrapped_maybe_erase<false, FuncResult>(std::forward<Func>(func), sl);\n      }\n    }\n\n    template <std::invocable<future&&> Func, typename FuncResult = std::invoke_result_t<Func, future&&>>\n    futurize_t<FuncResult>\n    then_wrapped(Func&& func, std::source_location sl = std::source_location::current()) && noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return std::move(*this).then_wrapped(func_to_rvalue(func), sl);\n      } else {\n        return then_wrapped_maybe_erase<true, FuncResult>(std::forward<Func>(func), sl);\n      }\n    }\n\nprivate:\n\n    template <bool AsSelf, typename FuncResult, typename Func>\n    futurize_t<FuncResult>\n    then_wrapped_maybe_erase(Func&& func, std::source_location sl) noexcept {\n        static_assert(internal::expect_only_rvalue_refs<Func>);\n#ifndef SEASTAR_TYPE_ERASE_MORE\n        return then_wrapped_common<AsSelf, FuncResult>(std::forward<Func>(func), sl);\n#else\n        using futurator = futurize<FuncResult>;\n        using WrapFuncResult = typename futurator::type;\n        noncopyable_function<WrapFuncResult (future&&)> ncf;\n        {\n            memory::scoped_critical_alloc_section _;\n            ncf = noncopyable_function<WrapFuncResult(future &&)>([func = std::forward<Func>(func)](future&& f) mutable {\n                return futurator::invoke(std::move(func), std::move(f));\n            });\n        }\n        return then_wrapped_common<AsSelf, WrapFuncResult>(std::move(ncf), sl);\n#endif\n    }\n\n    // Keep this simple so that Named Return Value Optimization is used.\n    template <typename FuncResult, typename Func>\n    futurize_t<FuncResult>\n    then_wrapped_nrvo(Func&& func, std::source_location sl) noexcept {\n        using futurator = futurize<FuncResult>;\n        typename futurator::type fut(future_for_get_promise_marker{});\n        using pr_type = decltype(fut.get_promise());\n        schedule(fut.get_promise(), std::move(func), [](pr_type&& pr, Func&& func, future_state&& state) {\n            futurator::satisfy_with_result_of(std::move(pr), [&func, &state] {\n                return std::move(func)(future(std::move(state)));\n            });\n        }, sl);\n        return fut;\n    }\n\n\n    template <bool AsSelf, typename FuncResult, typename Func>\n    futurize_t<FuncResult>\n    then_wrapped_common(Func&& func, std::source_location sl) noexcept {\n#ifndef SEASTAR_DEBUG\n        using futurator = futurize<FuncResult>;\n        if (available()) {\n            if constexpr (AsSelf) {\n                if (_promise) {\n                    detach_promise();\n                }\n                return futurator::invoke(std::forward<Func>(func), std::move(*this));\n            } else {\n                return futurator::invoke(std::forward<Func>(func), future(get_available_state_ref()));\n            }\n        }\n#endif\n        return then_wrapped_nrvo<FuncResult, Func>(std::forward<Func>(func), sl);\n    }\n\n    void forward_to(internal::promise_base_with_type<T>&& pr) noexcept {\n        if (_state.available()) {\n            pr.set_urgent_state(std::move(_state));\n        } else {\n            *detach_promise() = std::move(pr);\n        }\n    }\n\npublic:\n    /// \\brief Satisfy some \\ref promise object with this future as a result.\n    ///\n    /// Arranges so that when this future is resolve, it will be used to\n    /// satisfy an unrelated promise.  This is similar to scheduling a\n    /// continuation that moves the result of this future into the promise\n    /// (using promise::set_value() or promise::set_exception(), except\n    /// that it is more efficient.\n    ///\n    /// \\param pr a promise that will be fulfilled with the results of this\n    /// future.\n    void forward_to(promise<T>&& pr) noexcept {\n        if (_state.available()) {\n            pr.set_urgent_state(std::move(_state));\n        } else if (&pr._local_state != pr._state) {\n            // The only case when _state points to _local_state is\n            // when get_future was never called. Given that pr will\n            // soon be destroyed, we know get_future will never be\n            // called and we can just ignore this request.\n            *detach_promise() = std::move(pr);\n        }\n    }\n\n\n\n    /**\n     * Finally continuation for statements that require waiting for the result.\n     * I.e. you need to \"finally\" call a function that returns a possibly\n     * unavailable future. The returned future will be \"waited for\", any\n     * exception generated will be propagated, but the return value is ignored.\n     * I.e. the original return value (the future upon which you are making this\n     * call) will be preserved.\n     *\n     * If the original return value or the callback return value is an\n     * exceptional future it will be propagated.\n     *\n     * If both of them are exceptional - the seastar::nested_exception exception\n     * with the callback exception on top and the original future exception\n     * nested will be propagated.\n     *\n     * See then() for lifetime and call semantics.\n     */\n    template <std::invocable Func>\n    future<T> finally(Func&& func, std::source_location sl = std::source_location::current()) noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return finally(func_to_rvalue(func), sl);\n      } else {\n        return then_wrapped(finally_body<Func, is_future<std::invoke_result_t<Func>>::value>(std::forward<Func>(func)), sl);\n      }\n    }\n\n\n    template <typename Func, bool FuncReturnsFuture>\n    struct finally_body;\n\n    template <typename Func>\n    struct finally_body<Func, true> {\n        Func _func;\n\n        finally_body(Func&& func) noexcept : _func(std::forward<Func>(func))\n        {\n            static_assert(internal::expect_only_rvalue_refs<Func>);\n        }\n\n        future<T> operator()(future<T>&& result) && noexcept {\n            return futurize_invoke(std::move(_func)).then_wrapped([result = std::move(result)](auto&& f_res) mutable {\n                if (!f_res.failed()) {\n                    return std::move(result);\n                } else {\n                    return result.rethrow_with_nested(std::move(f_res._state));\n                }\n            });\n        }\n    };\n\n    template <typename Func>\n    struct finally_body<Func, false> {\n        Func _func;\n\n        finally_body(Func&& func) noexcept : _func(std::forward<Func>(func))\n        { }\n\n        future<T> operator()(future<T>&& result) && noexcept {\n            try {\n                std::move(_func)();\n                return std::move(result);\n            } catch (...) {\n                return result.rethrow_with_nested();\n            }\n        };\n    };\n\n    /// \\brief Terminate the program if this future fails.\n    ///\n    /// Terminates the entire program is this future resolves\n    /// to an exception.  Use with caution.\n    future<> or_terminate(std::source_location sl = std::source_location::current()) noexcept {\n        return then_wrapped([] (auto&& f) {\n            try {\n                f.get();\n            } catch (...) {\n                engine_exit(std::current_exception());\n            }\n        }, sl);\n    }\n\n    /// \\brief Discards the value carried by this future.\n    ///\n    /// Converts the future into a no-value \\c future<>, by\n    /// ignoring any result.  Exceptions are propagated unchanged.\n    future<> discard_result(std::source_location sl = std::source_location::current()) noexcept {\n        // We need the generic variadic lambda, below, because then() behaves differently\n        // when value_type is when_all_succeed_tuple\n        return then([] (auto&&...) {}, sl);\n    }\n\n    /// \\brief Handle the exception carried by this future.\n    ///\n    /// When the future resolves, if it resolves with an exception,\n    /// handle_exception(func) replaces the exception with the value\n    /// returned by func. The exception is passed (as a std::exception_ptr)\n    /// as a parameter to func; func may return the replacement value\n    /// immediately (T or std::tuple<T...>) or in the future (future<T...>)\n    /// and is even allowed to return (or throw) its own exception.\n    ///\n    /// The idiom fut.discard_result().handle_exception(...) can be used\n    /// to handle an exception (if there is one) without caring about the\n    /// successful value; Because handle_exception() is used here on a\n    /// future<>, the handler function does not need to return anything.\n    template <typename Func>\n    requires std::is_invocable_r_v<future<T> ,Func, std::exception_ptr>\n                    || (std::tuple_size_v<tuple_type> == 0 && std::is_invocable_r_v<void, Func, std::exception_ptr>)\n                    || (std::tuple_size_v<tuple_type> == 1 && std::is_invocable_r_v<T, Func, std::exception_ptr>)\n                    || (std::tuple_size_v<tuple_type> > 1 && std::is_invocable_r_v<tuple_type ,Func, std::exception_ptr>)\n    future<T> handle_exception(Func&& func, std::source_location sl = std::source_location::current()) noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return handle_exception(func_to_rvalue(func), sl);\n      } else {\n        return then_wrapped([func = std::forward<Func>(func)]\n                             (auto&& fut) mutable -> future<T> {\n            if (!fut.failed()) {\n                return make_ready_future<T>(fut.get());\n            } else {\n                return futurize_invoke(func, fut.get_exception());\n            }\n        }, sl);\n      }\n    }\n\n    /// \\brief Handle the exception of a certain type carried by this future.\n    ///\n    /// When the future resolves, if it resolves with an exception of a type that\n    /// provided callback receives as a parameter, \\c handle_exception_type(func) replaces\n    /// the exception with the value returned by func. The exception is passed (by\n    /// reference) as a parameter to func; func may return the replacement value\n    /// immediately (T or std::tuple<T...>) or in the future (future<T...>)\n    /// and is even allowed to return (or throw) its own exception.\n    /// If exception, that future holds, does not match func parameter type\n    /// it is propagated as is.\n    template <typename Func>\n    future<T> handle_exception_type(Func&& func, std::source_location sl = std::source_location::current()) noexcept {\n      // Avoid having to special-case lvalue-references downstream by converting\n      // them to an rvalue reference here.\n      if constexpr (std::is_lvalue_reference_v<Func>) {\n        return handle_exception_type(func_to_rvalue(func), sl);\n      } else {\n        using trait = function_traits<Func>;\n        static_assert(trait::arity == 1, \"func can take only one parameter\");\n        using ex_type = typename trait::template arg<0>::type;\n        return then_wrapped([func = std::forward<Func>(func)]\n                             (auto&& fut) mutable -> future<T> {\n            try {\n                return make_ready_future<T>(fut.get());\n            } catch(ex_type& ex) {\n                return futurize_invoke(func, ex);\n            }\n        }, sl);\n      }\n    }\n\n    /// \\brief Ignore any result hold by this future\n    ///\n    /// Ignore any result (value or exception) hold by this future.\n    /// Use with caution since usually ignoring exception is not what\n    /// you want\n    void ignore_ready_future() noexcept {\n        _state.ignore();\n    }\n\n    using future_base::set_coroutine;\n\nprivate:\n    void set_task(task& t) noexcept {\n        assert(_promise);\n        _promise->set_task(&t);\n    }\n\n    void set_callback(continuation_base<T>* callback) noexcept {\n        if (_state.available()) {\n            callback->set_state(get_available_state_ref());\n            ::seastar::schedule(callback);\n        } else {\n            assert(_promise);\n            schedule(callback);\n        }\n\n    }\n\n    /// \\cond internal\n    template <typename U>\n    friend class future;\n    template <typename U>\n    friend class promise;\n    template <typename U>\n    friend struct futurize;\n    template <typename U>\n    friend class internal::promise_base_with_type;\n    template <typename U, typename... A>\n    friend future<U> make_ready_future(A&&... value) noexcept;\n    template <typename U>\n    friend future<U> make_exception_future(std::exception_ptr&& ex) noexcept;\n    template <typename U, typename Exception>\n    friend future<U> make_exception_future(Exception&& ex) noexcept;\n    template <typename U>\n    friend future<U> internal::make_exception_future(future_state_base&& state) noexcept;\n    template <typename U>\n    friend future<U> current_exception_as_future() noexcept;\n    template <typename U>\n    friend void internal::set_callback(future<U>&&, continuation_base<U>*) noexcept;\n    /// \\endcond\n};\n\n\nnamespace internal {\ntemplate <typename T>\nstruct futurize_base {\n    /// If \\c T is a future, \\c T; otherwise \\c future<T>\n    using type = future<T>;\n    /// The promise type associated with \\c type.\n    using promise_type = promise<T>;\n    using promise_base_with_type = internal::promise_base_with_type<T>;\n\n    /// Convert a value or a future to a future\n    static inline type convert(T&& value) { return make_ready_future<T>(std::move(value)); }\n    static inline type convert(type&& value) { return std::move(value); }\n\n    /// Makes an exceptional future of type \\ref type.\n    template <typename Arg>\n    static inline type make_exception_future(Arg&& arg) noexcept;\n};\n\ntemplate <>\nstruct futurize_base<void> {\n    using type = future<>;\n    using promise_type = promise<>;\n    using promise_base_with_type = internal::promise_base_with_type<>;\n\n    static inline type convert(type&& value) {\n        return std::move(value);\n    }\n    template <typename Arg>\n    static inline type make_exception_future(Arg&& arg) noexcept;\n};\n\ntemplate <typename T>\nstruct futurize_base<future<T>> : public futurize_base<T> {};\n\ntemplate <>\nstruct futurize_base<future<>> : public futurize_base<void> {};\n}\n\ntemplate <typename T>\nstruct futurize : public internal::futurize_base<T> {\n    using base = internal::futurize_base<T>;\n    using type = typename base::type;\n    using promise_type = typename base::promise_type;\n    using promise_base_with_type = typename base::promise_base_with_type;\n    /// The value tuple type associated with \\c type\n    using value_type = typename type::value_type;\n    using tuple_type = typename type::tuple_type;\n    using base::convert;\n    using base::make_exception_future;\n\n    /// Apply a function to an argument list (expressed as a tuple)\n    /// and return the result, as a future (if it wasn't already).\n    template<typename Func, typename... FuncArgs>\n    static inline type apply(Func&& func, std::tuple<FuncArgs...>&& args) noexcept;\n\n    /// Invoke a function to an argument list\n    /// and return the result, as a future (if it wasn't already).\n    template<typename Func, typename... FuncArgs>\n    static inline type invoke(Func&& func, FuncArgs&&... args) noexcept;\n\n    template<typename Func>\n    static inline type invoke(Func&& func, internal::monostate) noexcept {\n        return invoke(std::forward<Func>(func));\n    }\n\n    static type current_exception_as_future() noexcept {\n        return type(future_state_base::current_exception_future_marker());\n    }\n\n    /// Convert the tuple representation into a future\n    static type from_tuple(tuple_type&& value) {\n        return type(ready_future_marker(), std::move(value));\n    }\n    /// Convert the tuple representation into a future\n    static type from_tuple(const tuple_type& value) {\n        return type(ready_future_marker(), value);\n    }\n\n    /// Convert the tuple representation into a future\n    static type from_tuple(value_type&& value) {\n        return type(ready_future_marker(), std::move(value));\n    }\n    /// Convert the tuple representation into a future\n    static type from_tuple(const value_type& value) {\n        return type(ready_future_marker(), value);\n    }\nprivate:\n    /// Forwards the result of, or exception thrown by, func() to the\n    /// promise. This avoids creating a future if func() doesn't\n    /// return one.\n    template<std::invocable Func>\n    static void satisfy_with_result_of(promise_base_with_type&&, Func&& func);\n\n    template <typename U>\n    friend class future;\n};\n\ninline internal::promise_base::promise_base(future_base* future, future_state_base* state) noexcept\n    : _future(future), _state(state) {\n    _future->_promise = this;\n}\n\ntemplate <typename T>\ninline\nfuture<T>\npromise<T>::get_future() noexcept {\n    assert(!this->_future && this->_state && !this->_task);\n    return future<T>(this);\n}\n\ntemplate <typename T>\ninline\nvoid promise<T>::move_it(promise&& x) noexcept {\n    if (this->_state == &x._local_state) {\n        this->_state = &_local_state;\n        new (&_local_state) future_state(std::move(x._local_state));\n    }\n}\n\ntemplate <typename T, typename... A>\ninline\nfuture<T> make_ready_future(A&&... value) noexcept {\n    return future<T>(ready_future_marker(), std::forward<A>(value)...);\n}\n\ntemplate <typename T>\ninline\nfuture<T> make_exception_future(std::exception_ptr&& ex) noexcept {\n    return future<T>(exception_future_marker(), std::move(ex));\n}\n\ntemplate <typename T>\ninline\nfuture<T> internal::make_exception_future(future_state_base&& state) noexcept {\n    return future<T>(exception_future_marker(), std::move(state));\n}\n\ntemplate <typename T>\nfuture<T> current_exception_as_future() noexcept {\n    return future<T>(future_state_base::current_exception_future_marker());\n}\n\nvoid log_exception_trace(log_level level) noexcept;\n\n/// \\brief Creates a \\ref future in an available, failed state.\n///\n/// Creates a \\ref future object that is already resolved in a failed\n/// state.  This no I/O needs to be performed to perform a computation\n/// (for example, because the connection is closed and we cannot read\n/// from it).\ntemplate <typename T, typename Exception>\ninline\nfuture<T> make_exception_future(Exception&& ex) noexcept {\n    log_exception_trace(log_level::trace);\n    return make_exception_future<T>(std::make_exception_ptr(std::forward<Exception>(ex)));\n}\n\ntemplate <typename T, typename Exception>\nfuture<T> make_exception_future_with_backtrace(Exception&& ex) noexcept {\n    return make_exception_future<T>(make_backtraced_exception_ptr<Exception>(std::forward<Exception>(ex)));\n}\n\n/// @}\n\n/// \\cond internal\n\ntemplate<typename T>\ntemplate<typename Func, typename... FuncArgs>\ntypename futurize<T>::type futurize<T>::apply(Func&& func, std::tuple<FuncArgs...>&& args) noexcept {\n    try {\n        using ret_t = decltype(std::apply(std::forward<Func>(func), std::move(args)));\n        if constexpr (std::is_void_v<ret_t>) {\n            std::apply(std::forward<Func>(func), std::move(args));\n            return make_ready_future<>();\n        } else if constexpr (is_future<ret_t>::value){\n            return std::apply(std::forward<Func>(func), std::move(args));\n        } else {\n            return convert(std::apply(std::forward<Func>(func), std::move(args)));\n        }\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate<typename T>\ntemplate<std::invocable Func>\nvoid futurize<T>::satisfy_with_result_of(promise_base_with_type&& pr, Func&& func) {\n    using ret_t = decltype(func());\n    if constexpr (std::is_void_v<ret_t>) {\n        func();\n        pr.set_value();\n    } else if constexpr (is_future<ret_t>::value) {\n        func().forward_to(std::move(pr));\n    } else {\n        pr.set_value(func());\n    }\n}\n\ntemplate<typename T>\ntemplate<typename Func, typename... FuncArgs>\ntypename futurize<T>::type futurize<T>::invoke(Func&& func, FuncArgs&&... args) noexcept {\n    try {\n        using ret_t = std::invoke_result_t<Func, FuncArgs&&...>;\n        if constexpr (std::is_void_v<ret_t>) {\n            std::invoke(std::forward<Func>(func), std::forward<FuncArgs>(args)...);\n            return make_ready_future<>();\n        } else if constexpr (is_future<ret_t>::value) {\n            return std::invoke(std::forward<Func>(func), std::forward<FuncArgs>(args)...);\n        } else {\n            return convert(std::invoke(std::forward<Func>(func), std::forward<FuncArgs>(args)...));\n        }\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename T>\ntemplate <typename Arg>\ninline\nfuture<T>\ninternal::futurize_base<T>::make_exception_future(Arg&& arg) noexcept {\n    using ::seastar::make_exception_future;\n    using ::seastar::internal::make_exception_future;\n    return make_exception_future<T>(std::forward<Arg>(arg));\n}\n\ntemplate <typename Arg>\ninline\nfuture<>\ninternal::futurize_base<void>::make_exception_future(Arg&& arg) noexcept {\n    using ::seastar::make_exception_future;\n    using ::seastar::internal::make_exception_future;\n    return make_exception_future<>(std::forward<Arg>(arg));\n}\n\ntemplate<typename Func, typename... Args>\nauto futurize_invoke(Func&& func, Args&&... args) noexcept {\n    using futurator = futurize<std::invoke_result_t<Func, Args&&...>>;\n    return futurator::invoke(std::forward<Func>(func), std::forward<Args>(args)...);\n}\n\ntemplate<typename Func, typename... Args>\nauto futurize_apply(Func&& func, std::tuple<Args...>&& args) noexcept {\n    using futurator = futurize<std::invoke_result_t<Func, Args&&...>>;\n    return futurator::apply(std::forward<Func>(func), std::move(args));\n}\n\nnamespace internal {\n\ntemplate <typename T>\ninline\nvoid set_callback(future<T>&& fut, continuation_base<T>* callback) noexcept {\n    return std::move(fut).set_callback(callback);\n}\n\n}\n\n\n/// \\endcond\n\n}\n"
  },
  {
    "path": "include/seastar/core/gate.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <boost/intrusive/list.hpp>\n\n#include <seastar/core/future.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <cassert>\n#include <exception>\n#include <optional>\n#include <utility>\n\n#ifdef SEASTAR_DEBUG\n#define SEASTAR_GATE_HOLDER_DEBUG\n#endif\n\nnamespace seastar {\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// Exception thrown when a \\ref gate object has been closed\n/// by the \\ref gate::close() method.\nclass gate_closed_exception : public std::exception {\npublic:\n    virtual const char* what() const noexcept override {\n        return \"gate closed\";\n    }\n};\n\n/// Exception thrown when a \\ref named_gate object has been closed\n/// by the \\ref gate::close() method.\nclass named_gate_closed_exception : public gate_closed_exception {\n    static constexpr const char* _default_what = \"named gate closed\";\n    sstring _what;\npublic:\n    named_gate_closed_exception(const sstring& name) noexcept : gate_closed_exception() {\n        if (!name.empty()) {\n            try {\n                _what = fmt::format(\"{} gate closed\", name);\n            } catch (...) {\n                // Ignore\n            }\n        }\n    }\n    virtual const char* what() const noexcept override {\n        return _what.empty() ? _default_what : _what.c_str();\n    }\n};\n\n/// Facility to stop new requests, and to tell when existing requests are done.\n///\n/// When stopping a service that serves asynchronous requests, we are faced with\n/// two problems: preventing new requests from coming in, and knowing when existing\n/// requests have completed.  The \\c gate class provides a solution.\nclass gate {\n    size_t _count = 0;\n    std::optional<promise<>> _stopped;\n\n#ifdef SEASTAR_GATE_HOLDER_DEBUG\n    void assert_not_held_when_moved() const noexcept;\n    void assert_not_held_when_destroyed() const noexcept;\n#else   // SEASTAR_GATE_HOLDER_DEBUG\n    void assert_not_held_when_moved() const noexcept {}\n    void assert_not_held_when_destroyed() const noexcept {}\n#endif  // SEASTAR_GATE_HOLDER_DEBUG\n\npublic:\n    // Implemented to force noexcept due to boost:intrusive::list\n    gate() noexcept {};\n    gate(const gate&) = delete;\n    gate(gate&& x) noexcept\n        : _count(std::exchange(x._count, 0)), _stopped(std::exchange(x._stopped, std::nullopt)) {\n        x.assert_not_held_when_moved();\n    }\n    gate& operator=(gate&& x) noexcept {\n        if (this != &x) {\n            SEASTAR_ASSERT(!_count && \"gate reassigned with outstanding requests\");\n            x.assert_not_held_when_moved();\n            _count = std::exchange(x._count, 0);\n            _stopped = std::exchange(x._stopped, std::nullopt);\n        }\n        return *this;\n    }\n    virtual ~gate() {\n        SEASTAR_ASSERT(!_count && \"gate destroyed with outstanding requests\");\n        assert_not_held_when_destroyed();\n    }\n    /// Tries to register an in-progress request.\n    ///\n    /// If the gate is not closed, the request is registered and the function returns `true`,\n    /// Otherwise the function just returns `false` and has no other effect.\n    bool try_enter() noexcept {\n        bool opened = !_stopped;\n        if (opened) {\n            ++_count;\n        }\n        return opened;\n    }\n    /// Registers an in-progress request.\n    ///\n    /// If the gate is not closed, the request is registered.  Otherwise,\n    /// a \\ref gate_closed_exception is thrown.\n    void enter() {\n        if (!try_enter()) {\n            throw_closed_exception();\n        }\n    }\n    /// Unregisters an in-progress request.\n    ///\n    /// If the gate is closed, and there are no more in-progress requests,\n    /// the `_stopped` promise will be fulfilled.\n    void leave() noexcept {\n        --_count;\n        if (!_count && _stopped) {\n            _stopped->set_value();\n        }\n    }\n    /// Potentially stop an in-progress request.\n    ///\n    /// If the gate is already closed, a \\ref gate_closed_exception is thrown.\n    /// By using \\ref enter() and \\ref leave(), the program can ensure that\n    /// no further requests are serviced. However, long-running requests may\n    /// continue to run. The check() method allows such a long operation to\n    /// voluntarily stop itself after the gate is closed, by making calls to\n    /// check() in appropriate places. check() with throw an exception and\n    /// bail out of the long-running code if the gate is closed.\n    void check() const {\n        if (_stopped) {\n            throw_closed_exception();\n        }\n    }\n    /// Closes the gate.\n    ///\n    /// Future calls to \\ref enter() will fail with an exception, and when\n    /// all current requests call \\ref leave(), the returned future will be\n    /// made ready.\n    future<> close() noexcept {\n        SEASTAR_ASSERT(!_stopped && \"seastar::gate::close() cannot be called more than once\");\n        _stopped = std::make_optional(promise<>());\n        if (!_count) {\n            _stopped->set_value();\n        }\n        return _stopped->get_future();\n    }\n\n    /// Returns a current number of registered in-progress requests.\n    size_t get_count() const noexcept {\n        return _count;\n    }\n\n    /// Returns whether the gate is closed.\n    bool is_closed() const noexcept {\n        return bool(_stopped);\n    }\n\n    /// Facility to hold a gate opened using RAII.\n    ///\n    /// A \\ref gate::holder is usually obtained using \\ref gate::hold.\n    ///\n    /// The \\c gate is entered when the \\ref gate::holder is constructed,\n    /// And the \\c gate is left when the \\ref gate::holder is destroyed.\n    ///\n    /// Copying the \\ref gate::holder reenters the \\c gate to keep an extra reference on it.\n    /// Moving the \\ref gate::holder is supported and has no effect on the \\c gate itself.\n    class holder {\n        gate* _g;\n#ifdef SEASTAR_GATE_HOLDER_DEBUG\n        using member_hook_t = boost::intrusive::list_member_hook<boost::intrusive::link_mode<boost::intrusive::auto_unlink>>;\n        member_hook_t _link;\n\n        void debug_hold_gate() noexcept {\n            if (_g) {\n                _g->_holders.push_back(*this);\n            }\n        }\n\n        void debug_release_gate() noexcept {\n            _link.unlink();\n        }\n#else   // SEASTAR_GATE_HOLDER_DEBUG\n        void debug_hold_gate() noexcept {}\n        void debug_release_gate() noexcept {}\n#endif  // SEASTAR_GATE_HOLDER_DEBUG\n\n        // release the holder from the gate without leaving it.\n        // used for release and move.\n        gate* release_gate() noexcept {\n            auto* g = std::exchange(_g, nullptr);\n            if (g) {\n                debug_release_gate();\n            }\n            return g;\n        }\n\n        friend class gate;\n    public:\n        /// Construct a default \\ref holder, referencing no \\ref gate.\n        /// Never throws.\n        holder() noexcept : _g(nullptr) { }\n\n        /// Construct a \\ref holder by entering the \\c gate.\n        /// May throw \\ref gate_closed_exception if the gate is already closed.\n        explicit holder(gate& g) : _g(&g) {\n            _g->enter();\n            debug_hold_gate();\n        }\n\n        /// Construct a \\ref holder by copying another \\c holder.\n        /// Copying a holder never throws: The original holder has already entered the gate,\n        /// so even if later the gate was \\ref close \"close()d\", the copy of the holder is also allowed to enter too.\n        /// Note that the fiber waiting for the close(), which until now was waiting for the one holder to leave,\n        /// will now wait for both copies to leave.\n        holder(const holder& x) noexcept : _g(x._g) {\n            if (_g) {\n                _g->_count++;\n                debug_hold_gate();\n            }\n        }\n\n        /// Construct a \\ref holder by moving another \\c holder.\n        /// The referenced \\ref gate is unaffected, and so the\n        /// move-constructor must never throw.\n        holder(holder&& x) noexcept : _g(std::move(x).release_gate()) {\n            debug_hold_gate();\n        }\n\n        /// Destroy a \\ref holder and leave the referenced \\ref gate.\n        ~holder() {\n            release();\n        }\n\n        /// Copy-assign another \\ref holder.\n        /// \\ref leave \"Leave()\" the current \\ref gate before assigning the other one, if they are different.\n        /// Copying a holder never throws: The original holder has already entered the gate,\n        /// so even if later the gate was \\ref close \"close()d\", the copy of the holder is also allowed to enter too.\n        /// Note that the fiber waiting for the close(), which until now was waiting for the one holder to leave,\n        /// will now wait for both copies to leave.\n        holder& operator=(const holder& x) noexcept {\n            if (x._g != _g) {\n                release();\n                _g = x._g;\n                if (_g) {\n                    _g->_count++;\n                    debug_hold_gate();\n                }\n            }\n            return *this;\n        }\n\n        /// Move-assign another \\ref holder.\n        /// The other \\ref gate is unaffected,\n        /// and so the move-assign operator must always succeed.\n        /// Leave the current \\ref gate before assigning the other one.\n        holder& operator=(holder&& x) noexcept {\n            if (&x != this) {\n                release();\n                _g = std::move(x).release_gate();\n                debug_hold_gate();\n            }\n            return *this;\n        }\n\n        /// Leave the held \\c gate\n        void release() noexcept {\n            if (auto g = release_gate()) {\n                g->leave();\n            }\n        }\n    };\n\n    /// Get a RAII-based gate::holder object that \\ref enter \"enter()s\"\n    /// the gate when constructed and \\ref leave \"leave()s\" it when destroyed.\n    holder hold() {\n        return holder(*this);\n    }\n\n    /// Try getting an optional RAII-based gate::holder object that \\ref enter \"enter()s\"\n    /// the gate when constructed and \\ref leave \"leave()s\" it when destroyed.\n    /// Returns std::nullopt iff the gate is closed.\n    std::optional<holder> try_hold() noexcept {\n        return is_closed() ? std::nullopt : std::make_optional<holder>(*this);\n    }\n\nprotected:\n    virtual std::exception_ptr make_closed_exception() const {\n        return std::make_exception_ptr(gate_closed_exception{});\n    }\n\n    template <typename Func>\n    friend auto try_with_gate(gate&, Func&&) noexcept;\nprivate:\n#ifdef SEASTAR_GATE_HOLDER_DEBUG\n    using holders_list_t = boost::intrusive::list<holder,\n        boost::intrusive::member_hook<holder, holder::member_hook_t, &holder::_link>,\n        boost::intrusive::constant_time_size<false>>;\n\n    holders_list_t _holders;\n#endif  // SEASTAR_GATE_HOLDER_DEBUG\n\n    [[noreturn]] void throw_closed_exception() const {\n        std::rethrow_exception(make_closed_exception());\n    }\n};\n\n#ifdef SEASTAR_GATE_HOLDER_DEBUG\ninline void gate::assert_not_held_when_moved() const noexcept {\n    SEASTAR_ASSERT(_holders.empty() && \"gate moved with outstanding holders\");\n}\ninline void gate::assert_not_held_when_destroyed() const noexcept {\n    SEASTAR_ASSERT(_holders.empty() && \"gate destroyed with outstanding holders\");\n}\n#endif  // SEASTAR_GATE_HOLDER_DEBUG\n\nclass named_gate : public gate {\n    sstring _name;\n\npublic:\n    named_gate() = default;\n    explicit named_gate(sstring name) noexcept\n        : gate()\n        , _name(std::move(name))\n    {}\n    named_gate(named_gate&&) = default;\n    named_gate& operator=(named_gate&&) = default;\n\nprotected:\n    virtual std::exception_ptr make_closed_exception() const override {\n        return std::make_exception_ptr(named_gate_closed_exception(_name));\n    }\n};\n\nnamespace internal {\n\ntemplate <typename Func>\ninline\nauto\ninvoke_func_with_gate(gate::holder&& gh, Func&& func) noexcept {\n    return futurize_invoke(std::forward<Func>(func)).finally([gh = std::forward<gate::holder>(gh)] {});\n}\n\n} // namespace internal\n\n/// Executes the function \\c func making sure the gate \\c g is properly entered\n/// and later on, properly left.\n///\n/// \\param func function to be executed\n/// \\param g the gate. Caller must make sure that it outlives this function.\n/// \\returns whatever \\c func returns\n///\n/// \\relates gate\ntemplate <typename Func>\ninline\nauto\nwith_gate(gate& g, Func&& func) {\n    return internal::invoke_func_with_gate(g.hold(), std::forward<Func>(func));\n}\n\n/// Executes the function \\c func if the gate \\c g can be entered\n/// and later on, properly left.\n///\n/// \\param func function to be executed\n/// \\param g the gate. Caller must make sure that it outlives this function.\n///\n/// If the gate is already closed, an exception future holding\n/// \\ref gate_closed_exception is returned, otherwise\n/// \\returns whatever \\c func returns.\n///\n/// \\relates gate\ntemplate <typename Func>\ninline\nauto\ntry_with_gate(gate& g, Func&& func) noexcept {\n    if (g.is_closed()) {\n        using futurator = futurize<std::invoke_result_t<Func>>;\n        return futurator::make_exception_future(g.make_closed_exception());\n    }\n    return internal::invoke_func_with_gate(g.hold(), std::forward<Func>(func));\n}\n/// @}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/idle_cpu_handler.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/util/noncopyable_function.hh>\n\n/// \\file\n\nnamespace seastar {\n\n\n/// Indicates the outcome of a user callback installed to take advantage of\n/// idle CPU cycles.\nenum class idle_cpu_handler_result {\n    no_more_work,                       //!< The user callback has no more work to perform\n    interrupted_by_higher_priority_task //!< A call to the work_waiting_on_reactor parameter to idle_cpu_handler returned `true`\n};\n\n/// Signature of a callback provided by the reactor to a user callback installed to take\n/// advantage of idle cpu cycles, used to periodically check if the CPU is still idle.\n///\n/// \\return true if the reactor has new work to do\nusing work_waiting_on_reactor = const noncopyable_function<bool()>&;\n\n/// Signature of a callback provided by the user, that the reactor calls when it has idle cycles.\n///\n/// The `poll` parameter is a work_waiting_on_reactor function that should be periodically called\n/// to check if the idle callback should return with idle_cpu_handler_result::interrupted_by_higher_priority_task\nusing idle_cpu_handler = noncopyable_function<idle_cpu_handler_result(work_waiting_on_reactor poll)>;\n\n/// Set a handler that will be called when there is no task to execute on cpu.\n/// Handler should do a low priority work.\n///\n/// Handler's return value determines whether handler did any actual work. If no work was done then reactor will go\n/// into sleep.\n///\n/// Handler's argument is a function that returns true if a task which should be executed on cpu appears or false\n/// otherwise. This function should be used by a handler to return early if a task appears.\nvoid set_idle_cpu_handler(idle_cpu_handler&& handler);\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/internal/api-level.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n#pragma once\n\n// For IDEs that don't see SEASTAR_API_LEVEL, generate a nice default\n#ifndef SEASTAR_API_LEVEL\n#define SEASTAR_API_LEVEL 9\n#endif\n\n#if SEASTAR_API_LEVEL == 8\n#define SEASTAR_INCLUDE_API_V8 inline\n#else\n#define SEASTAR_INCLUDE_API_V8\n#endif\n\n// Declare them here so we don't have to use the macros everywhere\nnamespace seastar {\n    SEASTAR_INCLUDE_API_V8 namespace api_v8 {\n        inline namespace and_newer {\n        }\n    }\n}\n\n// Helpers for ignoring deprecation warnings in code that has to deal with\n// deprecated APIs, e.g., constructors/etc for structs with deprecated fields.\n#define SEASTAR_INTERNAL_BEGIN_IGNORE_DEPRECATIONS \\\n    _Pragma(\"GCC diagnostic push\") \\\n    _Pragma(\"GCC diagnostic ignored \\\"-Wdeprecated-declarations\\\"\")\n\n#define SEASTAR_INTERNAL_END_IGNORE_DEPRECATIONS \\\n    _Pragma(\"GCC diagnostic pop\")\n"
  },
  {
    "path": "include/seastar/core/internal/buffer_allocator.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar {\n\ntemplate <typename CharType>\nclass temporary_buffer;\n\nnamespace internal {\n\n// Internal interface for allocating buffers for reads. Used to decouple\n// allocation strategies (where to allocate from, and what sizes) from the\n// point where allocation happens, to make it as late as possible.\nclass buffer_allocator {\npublic:\n    virtual ~buffer_allocator() = default;\n    virtual temporary_buffer<char> allocate_buffer() = 0;\n};\n\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/internal/current_task.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2025 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar::internal {\n\n// If a task wants to resume a different task instead of returning control to the reactor,\n// it should set _current_task to the resumed task.\n// In particular, this is mandatory if the task is going to die before control is returned\n// to the reactor -- otherwise _current_task will be left dangling.\nvoid set_current_task(task* t);\n\n} // namespace seastar::internal\n"
  },
  {
    "path": "include/seastar/core/internal/estimated_histogram.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB.\n */\n\n#pragma once\n\n#include <cmath>\n#include <algorithm>\n#include <vector>\n#include <chrono>\n#include <string>\n#include <seastar/core/metrics_types.hh>\n#include <seastar/core/bitops.hh>\n#include <limits>\n#include <array>\n\nnamespace seastar::metrics::internal {\n/**\n * This is a pseudo-exponential implementation of an estimated histogram.\n *\n * An exponential-histogram with coefficient 'coef', is a histogram where for bucket 'i'\n * the lower limit is coef^i and the higher limit is coef^(i+1).\n *\n * A pseudo-exponential is similar but the bucket limits are an approximation.\n *\n * The approximate_exponential_histogram is an efficient pseudo-exponential implementation.\n *\n * The histogram is defined by a Min and Max value limits, and a Precision (all should be power of 2\n * and will be explained).\n *\n * When adding a value to a histogram:\n * All values lower than Min will be included in the first bucket (the assumption is that it's\n * not suppose to happen but it is ok if it does).\n *\n * All values higher than Max will be included in the last bucket that serves as the\n * infinity bucket (the assumption is that it can happen but it is rare).\n *\n * Note the difference between the first and last buckets.\n * The first bucket is just like a regular bucket but has a second roll to collect unexpected low values.\n * The last bucket, also known as the infinity bucket, collect all values that passes the defined Max,\n * it only collect those values.\n *\n * Buckets Distribution (limits)\n * =============================\n * The buckets limits in the histogram are defined similar to a floating-point representation.\n *\n * Buckets limits have an exponent part and a linear part.\n *\n * The exponential part is a power of 2. Each power-of-2 range [2^n..2^n+1)\n * is split linearly to 'Precision' number of buckets.\n *\n * The total number of buckets is:\n *   NUM_BUCKETS = log2(Max/Min)*Precision +1\n *\n * For example, if the Min value is 128, the Max is 1024 and the Precision is 4, the number of buckets is 13.\n *\n * Anything below 160 will be in the bucket 0, anything above 1024 will be in bucket 13.\n * Note that the first bucket will include all values below Min.\n *\n * the range [128, 1024) will be split into log2(1024/128) = 3 ranges:\n * 128, 256, 512, 1024\n * Or more mathematically: [128, 256), [256, 512), [512,1024)\n *\n * Each range is split into 4 (The Precision).\n * 128            | 256            | 512            | 1024\n * 128 160 192 224| 256 320 384 448| 512 640 768 896|\n *\n * To get the exponential part of an index you divide by the Precision.\n * The linear part of the index is Modulus the precision.\n *\n * Calculating the bucket lower limit of bucket i:\n * The exponential part: exp_part = 2^floor(i/Precision)* Min\n *    with the above example 2^floor(i/4)*128\n * The linear part: (i%Precision) * (exp_part/Precision)\n *    With the example: (i%4) * (exp_part/4)\n *\n * So the lower limit of bucket 6:\n *    2^floor(6/4)* 128  = 256\n *    (6%4) * 256/4 = 128\n *    lower-limit   = 384\n *\n * How to find a bucket index for a value\n * =======================================\n * The bucket index consist of two parts:\n * higher bits (exponential part) are based on log2(value/min)\n *\n * lower bits (linear part) are based on the 'n' MSB (ignoring the leading 1) where n=log2(Precision).\n * Continuing with the example where the number of precision bits: PRECISION_BITS = log2(4) = 2\n *\n * for example: 330 (101001010)\n * The number of precision_bits: PRECISION_BITS = log2(4) = 2\n * higher bits: log2(330/128) = 1\n * MSB: 01 (the highest two bits following the leading 1)\n * So the index: 101 = 5\n *\n * About the Min, Max and Precision\n * ================================\n * For Min, Max and Precision, choose numbers that are a power of 2.\n *\n * Limitation: You must set the MIN value to be higher or equal to the Precision.\n *\n * \\param Min - The lowest expected value. A value below that will be set to the Min value.\n * \\param Max - The highest number to consider. A value above that is considered infinite.\n * \\param Precision - The number of buckets in each range (where a range is the i^2..i^(2+1) )\n */\ntemplate<uint64_t Min, uint64_t Max, size_t Precision>\nclass approximate_exponential_histogram {\n    static constexpr unsigned NUM_EXP_RANGES = log2floor(Max/Min);\n    static constexpr size_t NUM_BUCKETS = NUM_EXP_RANGES * Precision + 1;\n    static constexpr unsigned PRECISION_BITS = log2floor(Precision);\n    static constexpr unsigned BASESHIFT = log2floor(Min);\n    static constexpr uint64_t LOWER_BITS_MASK = Precision - 1;\n    static constexpr size_t MIN_ID = log2ceil(Min) * Precision + 1; // id0 (x,1]\n    std::array<uint64_t, NUM_BUCKETS> _buckets;\npublic:\n    approximate_exponential_histogram() {\n        clear();\n    }\n\n    /*!\n     * \\brief Returns the bucket lower limit given the bucket id.\n     * The first and last bucket will always return the MIN and MAX respectively.\n     *\n     */\n    uint64_t get_bucket_lower_limit(uint16_t bucket_id) const {\n        if (bucket_id == NUM_BUCKETS - 1) {\n            return Max;\n        }\n        int16_t exp_rang = (bucket_id >> PRECISION_BITS);\n        return (Min << exp_rang) +  ((bucket_id & LOWER_BITS_MASK) << (exp_rang + BASESHIFT - PRECISION_BITS));\n    }\n\n    /*!\n     * \\brief Returns the bucket upper limit given the bucket id.\n     * The last bucket (Infinity bucket) will return UMAX_INT.\n     *\n     */\n    uint64_t get_bucket_upper_limit(uint16_t bucket_id) const {\n        if (bucket_id == NUM_BUCKETS - 1) {\n            return std::numeric_limits<uint64_t>::max();\n        }\n        return get_bucket_lower_limit(bucket_id + 1);\n    }\n\n    /*!\n     * \\brief Find the bucket index for a given value\n     * The position of a value that is lower or equal to Min will always be 0.\n     * The position of a value that is higher or equal to MAX will always be NUM_BUCKETS - 1.\n     */\n    uint16_t find_bucket_index(uint64_t val) const {\n        if (val >= Max) {\n            return NUM_BUCKETS - 1;\n        }\n        if (val <= Min) {\n            return 0;\n        }\n        uint16_t range = log2floor(val);\n        val >>= range - PRECISION_BITS; // leave the top most N+1 bits where N is the resolution.\n        return ((range - BASESHIFT) << PRECISION_BITS) + (val & LOWER_BITS_MASK);\n    }\n\n    /*!\n     * \\brief clear the current values.\n     */\n    void clear() {\n        std::fill(_buckets.begin(), _buckets.end(), 0);\n    }\n\n    /*!\n     * \\brief Add an item to the histogram\n     * Increments the count of the bucket holding that value\n     */\n    void add(uint64_t n) {\n        _buckets.at(find_bucket_index(n))++;\n    }\n\n    /*!\n     * \\brief returns the smallest value that could have been added to this histogram\n     * This method looks for the first non-empty bucket and returns its lower limit.\n     * Note that for non-empty histogram the lowest potentail value is Min.\n     *\n     * It will return 0 if the histogram is empty.\n     */\n    uint64_t min() const {\n        for (size_t i = 0; i < NUM_BUCKETS; i ++) {\n            if (_buckets[i] > 0) {\n                return get_bucket_lower_limit(i);\n            }\n        }\n        return 0;\n    }\n\n    /*!\n     * \\brief returns the largest value that could have been added to this histogram.\n     * This method looks for the first non empty bucket and return its upper limit.\n     * If the histogram overflowed, it will returns UINT64_MAX.\n     *\n     * It will return 0 if the histogram is empty.\n     */\n    uint64_t max() const {\n        for (int i = NUM_BUCKETS - 1; i >= 0; i--) {\n            if (_buckets[i] > 0) {\n                return get_bucket_upper_limit(i);\n            }\n        }\n        return 0;\n    }\n\n    /*!\n     * \\brief merge a histogram to the current one.\n     */\n    approximate_exponential_histogram& merge(const approximate_exponential_histogram& b) {\n        for (size_t i = 0; i < NUM_BUCKETS; i++) {\n            _buckets[i] += b.get(i);\n        }\n        return *this;\n    }\n\n    template<uint64_t A, uint64_t B, size_t C>\n    friend approximate_exponential_histogram<A, B, C> merge(approximate_exponential_histogram<A, B, C> a, const approximate_exponential_histogram<A, B, C>& b);\n\n    /*\n     * \\brief returns the count in the given bucket\n     */\n    uint64_t get(size_t bucket) const {\n        return _buckets[bucket];\n    }\n\n    /*!\n     * \\brief get a histogram quantile\n     *\n     * This method will returns the estimated value at a given quantile.\n     * If there are N values in the histogram.\n     * It would look for the bucket that the total number of elements in the buckets\n     * before it are less than N * quantile and return that bucket lower limit.\n     *\n     * For example, quantile(0.5) will find the bucket that that sum of all buckets values\n     * below it is less than half and will return that bucket lower limit.\n     * In this example, this is a median estimation.\n     *\n     * It will return 0 if the histogram is empty.\n     *\n     */\n    uint64_t quantile(float quantile) const {\n        if (quantile < 0 || quantile > 1.0) {\n            throw std::runtime_error(\"Invalid quantile value \" + std::to_string(quantile) + \". Value should be between 0 and 1\");\n        }\n        auto c = count();\n\n        if (!c) {\n            return 0; // no data\n        }\n\n        auto pcount = uint64_t(std::floor(c * quantile));\n        uint64_t elements = 0;\n        for (size_t i = 0; i < NUM_BUCKETS - 2; i++) {\n            if (_buckets[i]) {\n                elements += _buckets[i];\n                if (elements >= pcount) {\n                    return get_bucket_lower_limit(i);\n                }\n            }\n        }\n        return Max; // overflowed value is in the requested quantile\n    }\n\n    /*!\n     * \\brief returns the mean histogram value (average of bucket offsets, weighted by count)\n     * It will return 0 if the histogram is empty.\n     */\n    uint64_t mean() const {\n        uint64_t elements = 0;\n        double sum = 0;\n        for (size_t i = 0; i < NUM_BUCKETS - 1; i++) {\n            elements += _buckets[i];\n            sum += _buckets[i] * get_bucket_lower_limit(i);\n        }\n        return (elements) ?  sum / elements : 0;\n    }\n\n    /*!\n     * \\brief returns the number of buckets;\n     */\n    size_t size() const {\n        return NUM_BUCKETS;\n    }\n\n    /*!\n     * \\brief returns the total number of values inserted\n     */\n    uint64_t count() const {\n        uint64_t sum = 0L;\n        for (size_t i = 0; i < NUM_BUCKETS; i++) {\n            sum += _buckets[i];\n        }\n        return sum;\n    }\n\n    /*!\n     * \\brief multiple all the buckets content in the histogram by a constant\n     */\n    approximate_exponential_histogram& operator*=(double v) {\n        for (size_t i = 0; i < NUM_BUCKETS; i++) {\n            _buckets[i] *= v;\n        }\n        return *this;\n    }\n\n    uint64_t& operator[](size_t b) noexcept {\n        return _buckets[b];\n    }\n    /*!\n     * \\brief get_schema the schema of a native histogram\n     *\n     * Native histograms (also known as sparse histograms), are an experimental Prometheus feature.\n     *\n     * schema defines the bucket schema. Currently, valid numbers are -4 <= n <= 8.\n     * They are all for base-2 bucket schemas, where 1 is a bucket boundary in each case, and\n     * then each power of two is divided into 2^n logarithmic buckets.\n     * Or in other words, each bucket boundary is the previous boundary times 2^(2^-n).\n     */\n    int32_t get_schema() const noexcept {\n        return PRECISION_BITS;\n    }\n\n    /*!\n     * \\brief return the histogram min as a native histogram ID\n     *\n     * Native histograms (also known as sparse histograms), are an experimental Prometheus feature.\n     *\n     * Native histograms always starts from 1, while approximate_exponential_histogram have a min first bucket\n     * This function returns approximate_exponential_histogram min value as the bucket id of native histogram.\n     */\n    int32_t min_as_native_histogram_id() const noexcept {\n        return MIN_ID;\n    }\n\n    seastar::metrics::histogram to_metrics_histogram() const noexcept {\n        seastar::metrics::histogram res;\n        res.buckets.resize(size() - 1);\n        uint64_t cummulative_count = 0;\n        res.sample_sum = 0;\n        res.native_histogram = seastar::metrics::native_histogram_info{get_schema(), min_as_native_histogram_id()};\n\n        for (size_t i = 0; i < NUM_BUCKETS - 1; i++) {\n            auto& v = res.buckets[i];\n            v.upper_bound = get_bucket_lower_limit(i + 1);\n            cummulative_count += get(i);\n            v.count = cummulative_count;\n            res.sample_sum += get(i) * v.upper_bound;\n        }\n        // The count serves as the infinite bucket\n        res.sample_count = cummulative_count + get(NUM_BUCKETS - 1);\n        res.sample_sum += get(NUM_BUCKETS - 1) * get_bucket_lower_limit(NUM_BUCKETS - 1);\n        return res;\n    }\n};\n\ntemplate<uint64_t Min, uint64_t Max, size_t NumBuckets>\ninline approximate_exponential_histogram<Min, Max, NumBuckets> base_estimated_histogram_merge(approximate_exponential_histogram<Min, Max, NumBuckets> a, const approximate_exponential_histogram<Min, Max, NumBuckets>& b) {\n    return a.merge(b);\n}\n\n/*!\n * \\brief estimated histogram for duration values\n * time_estimated_histogram is used for short task timing.\n * It covers the range of 0.5ms to 33s with a precision of 4.\n *\n * 512us, 640us, 768us, 896us, 1024us, 1280us, 1536us, 1792us...16s, 20s, 25s, 29s, 33s (33554432us)\n */\nclass time_estimated_histogram : public approximate_exponential_histogram<512, 33554432, 4> {\npublic:\n    time_estimated_histogram& merge(const time_estimated_histogram& b) {\n        approximate_exponential_histogram<512, 33554432, 4>::merge(b);\n        return *this;\n    }\n\n    void add_micro(uint64_t n) {\n        approximate_exponential_histogram<512, 33554432, 4>::add(n);\n    }\n\n    template<typename T>\n    void add(const T& latency) {\n        add_micro(std::chrono::duration_cast<std::chrono::microseconds>(latency).count());\n    }\n};\n\ninline time_estimated_histogram time_estimated_histogram_merge(time_estimated_histogram a, const time_estimated_histogram& b) {\n    return a.merge(b);\n}\n}\n"
  },
  {
    "path": "include/seastar/core/internal/io_desc.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <exception>\n\nnamespace seastar {\n\nclass kernel_completion {\nprotected:\n     kernel_completion() = default;\n     kernel_completion(kernel_completion&&) = default;\n     kernel_completion& operator=(kernel_completion&&) = default;\n     ~kernel_completion() = default;\npublic:\n    virtual void complete_with(ssize_t res) = 0;\n};\n}\n"
  },
  {
    "path": "include/seastar/core/internal/io_intent.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/util/assert.hh>\n#include <utility>\n#include <boost/intrusive/list.hpp>\n#include <boost/intrusive/slist.hpp>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\nclass io_intent;\n\nnamespace internal {\n\n/*\n * The tracker of cancellable sub-queue of requests.\n *\n * This queue is stuffed with requests that sit in the\n * same IO queue for dispatching (there can be other requests\n * as well) and ties them together for cancellation.\n * This IO queue is the fair_queue's priority_class's one.\n * Beware, if requests from different IO queues happen\n * in the same cancellable queue the whole thing blows up.\n */\nclass cancellable_queue {\npublic:\n    class link {\n        friend class cancellable_queue;\n\n        union {\n            cancellable_queue* _ref;\n            bi::slist_member_hook<> _hook;\n        };\n\n    public:\n        link() noexcept : _ref(nullptr) {}\n        ~link() { SEASTAR_ASSERT(_ref == nullptr); }\n\n        void enqueue(cancellable_queue& cq) noexcept {\n            cq.push_back(*this);\n        }\n\n        void maybe_dequeue() noexcept {\n            if (_ref != nullptr) {\n                _ref->pop_front();\n            }\n        }\n    };\n\nprivate:\n    static_assert(sizeof(link) == sizeof(void*), \"cancellable_queue::link size is too big\");\n    using list_of_links_t = bi::slist<link,\n        bi::constant_time_size<false>,\n        bi::cache_last<true>,\n        bi::member_hook<link, bi::slist_member_hook<>, &link::_hook>>;\n\n    link* _first;\n    list_of_links_t _rest;\n\n    void push_back(link& il) noexcept;\n    void pop_front() noexcept;\n\npublic:\n    cancellable_queue() noexcept : _first(nullptr) {}\n    cancellable_queue(const cancellable_queue&) = delete;\n    cancellable_queue(cancellable_queue&& o) noexcept;\n    cancellable_queue& operator=(cancellable_queue&& o) noexcept;\n    ~cancellable_queue();\n};\n\n/*\n * A \"safe\" reference on a intent. Safe here means that the original\n * intent can be destroyed at any time and this reference will be\n * updated not to point at it any longer.\n * The retrieve() method brings the original intent back or throws\n * and exception if the intent was cancelled.\n */\nclass intent_reference : public bi::list_base_hook<bi::link_mode<bi::auto_unlink>> {\n    friend class seastar::io_intent;\n    using container_type = bi::list<intent_reference, bi::constant_time_size<false>>;\n    static constexpr uintptr_t _cancelled_intent = 1;\n    io_intent* _intent;\n\n    void on_cancel() noexcept { _intent = reinterpret_cast<io_intent*>(_cancelled_intent); }\n    bool is_cancelled() const noexcept { return _intent == reinterpret_cast<io_intent*>(_cancelled_intent); }\n\npublic:\n    intent_reference(io_intent* intent) noexcept;\n\n    intent_reference(intent_reference&& o) noexcept : _intent(std::exchange(o._intent, nullptr)) {\n        container_type::node_algorithms::swap_nodes(o.this_ptr(), this_ptr());\n    }\n\n    intent_reference& operator=(intent_reference&& o) noexcept {\n        if (this != &o) {\n            _intent = std::exchange(o._intent, nullptr);\n            unlink();\n            container_type::node_algorithms::swap_nodes(o.this_ptr(), this_ptr());\n        }\n        return *this;\n    }\n\n    io_intent* retrieve() const;\n};\n\n} // namespace internal\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/internal/io_request.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/on_internal_error.hh>\n#include <seastar/util/assert.hh>\n#include <cstdint>\n#include <vector>\n#include <sys/types.h>\n#include <sys/socket.h>\n\nnamespace seastar {\nextern logger io_log;\n\nnamespace internal {\n\nclass io_request {\npublic:\n    enum class operation : char { read, readv, write, writev, fdatasync, recv, recvmsg, send, sendmsg, accept, connect, poll_add, poll_remove, cancel };\nprivate:\n    // the upper layers give us void pointers, but storing void pointers here is just\n    // dangerous. The constructors seem to be happy to convert other pointers to void*,\n    // even if they are marked as explicit, and then you end up losing approximately 3 hours\n    // and 15 minutes (hypothetically, of course), trying to chase the weirdest bug.\n    // Let's store a char* for safety, and cast it back to void* in the accessor.\n    struct read_op {\n        operation op;\n        bool nowait_works;\n        int fd;\n        uint64_t pos;\n        char* addr;\n        size_t size;\n    };\n    struct readv_op {\n        operation op;\n        bool nowait_works;\n        int fd;\n        uint64_t pos;\n        ::iovec* iovec;\n        size_t iov_len;\n    };\n    struct recv_op {\n        operation op;\n        int fd;\n        char* addr;\n        size_t size;\n        int flags;\n    };\n    struct recvmsg_op {\n        operation op;\n        int fd;\n        ::msghdr* msghdr;\n        int flags;\n    };\n    using send_op = recv_op;\n    using sendmsg_op = recvmsg_op;\n    using write_op = read_op;\n    using writev_op = readv_op;\n    struct fdatasync_op {\n        operation op;\n        int fd;\n    };\n    struct accept_op {\n        operation op;\n        int fd;\n        ::sockaddr* sockaddr;\n        socklen_t* socklen_ptr;\n        int flags;\n    };\n    struct connect_op {\n        operation op;\n        int fd;\n        ::sockaddr* sockaddr;\n        socklen_t socklen;\n    };\n    struct poll_add_op {\n        operation op;\n        int fd;\n        int events;\n    };\n    struct poll_remove_op {\n        operation op;\n        int fd;\n        char* addr;\n    };\n    struct cancel_op {\n        operation op;\n        int fd;\n        char* addr;\n    };\n\n    union {\n        read_op _read;\n        readv_op _readv;\n        recv_op _recv;\n        recvmsg_op _recvmsg;\n        send_op _send;\n        sendmsg_op _sendmsg;\n        write_op _write;\n        writev_op _writev;\n        fdatasync_op _fdatasync;\n        accept_op _accept;\n        connect_op _connect;\n        poll_add_op _poll_add;\n        poll_remove_op _poll_remove;\n        cancel_op _cancel;\n    };\n\npublic:\n    static io_request make_read(int fd, uint64_t pos, void* address, size_t size, bool nowait_works) {\n        io_request req;\n        req._read = {\n          .op = operation::read,\n          .nowait_works = nowait_works,\n          .fd = fd,\n          .pos = pos,\n          .addr = reinterpret_cast<char*>(address),\n          .size = size,\n        };\n        return req;\n    }\n\n    static io_request make_readv(int fd, uint64_t pos, std::vector<iovec>& iov, bool nowait_works) {\n        io_request req;\n        req._readv = {\n          .op = operation::readv,\n          .nowait_works = nowait_works,\n          .fd = fd,\n          .pos = pos,\n          .iovec = iov.data(),\n          .iov_len = iov.size(),\n        };\n        return req;\n    }\n\n    static io_request make_recv(int fd, void* address, size_t size, int flags) {\n        io_request req;\n        req._recv = {\n          .op = operation::recv,\n          .fd = fd,\n          .addr = reinterpret_cast<char*>(address),\n          .size = size,\n          .flags = flags,\n        };\n        return req;\n    }\n\n    static io_request make_recvmsg(int fd, ::msghdr* msg, int flags) {\n        io_request req;\n        req._recvmsg = {\n          .op = operation::recvmsg,\n          .fd = fd,\n          .msghdr = msg,\n          .flags = flags,\n        };\n        return req;\n    }\n\n    static io_request make_send(int fd, const void* address, size_t size, int flags) {\n        io_request req;\n        req._send = {\n          .op = operation::send,\n          .fd = fd,\n          .addr = const_cast<char*>(reinterpret_cast<const char*>(address)),\n          .size = size,\n          .flags = flags,\n        };\n        return req;\n    }\n\n    static io_request make_sendmsg(int fd, ::msghdr* msg, int flags) {\n        io_request req;\n        req._sendmsg = {\n          .op = operation::sendmsg,\n          .fd = fd,\n          .msghdr = msg,\n          .flags = flags,\n        };\n        return req;\n    }\n\n    static io_request make_write(int fd, uint64_t pos, const void* address, size_t size, bool nowait_works) {\n        io_request req;\n        req._write = {\n          .op = operation::write,\n          .nowait_works = nowait_works,\n          .fd = fd,\n          .pos = pos,\n          .addr = const_cast<char*>(reinterpret_cast<const char*>(address)),\n          .size = size,\n        };\n        return req;\n    }\n\n    static io_request make_writev(int fd, uint64_t pos, std::vector<iovec>& iov, bool nowait_works) {\n        io_request req;\n        req._writev = {\n          .op = operation::writev,\n          .nowait_works = nowait_works,\n          .fd = fd,\n          .pos = pos,\n          .iovec = iov.data(),\n          .iov_len = iov.size(),\n        };\n        return req;\n    }\n\n    static io_request make_fdatasync(int fd) {\n        io_request req;\n        req._fdatasync = {\n          .op = operation::fdatasync,\n          .fd = fd,\n        };\n        return req;\n    }\n\n    static io_request make_accept(int fd, struct sockaddr* addr, socklen_t* addrlen, int flags) {\n        io_request req;\n        req._accept = {\n          .op = operation::accept,\n          .fd = fd,\n          .sockaddr = addr,\n          .socklen_ptr = addrlen,\n          .flags = flags,\n        };\n        return req;\n    }\n\n    static io_request make_connect(int fd, struct sockaddr* addr, socklen_t addrlen) {\n        io_request req;\n        req._connect = {\n          .op = operation::connect,\n          .fd = fd,\n          .sockaddr = addr,\n          .socklen = addrlen,\n        };\n        return req;\n    }\n\n    static io_request make_poll_add(int fd, int events) {\n        io_request req;\n        req._poll_add = {\n          .op = operation::poll_add,\n          .fd = fd,\n          .events = events,\n        };\n        return req;\n    }\n\n    static io_request make_poll_remove(int fd, void *addr) {\n        io_request req;\n        req._poll_remove = {\n          .op = operation::poll_remove,\n          .fd = fd,\n          .addr = reinterpret_cast<char*>(addr),\n        };\n        return req;\n    }\n\n    static io_request make_cancel(int fd, void *addr) {\n        io_request req;\n        req._cancel = {\n          .op = operation::cancel,\n          .fd = fd,\n          .addr = reinterpret_cast<char*>(addr),\n        };\n        return req;\n    }\n\n    bool is_read() const {\n        switch (opcode()) {\n        case operation::read:\n        case operation::readv:\n        case operation::recvmsg:\n        case operation::recv:\n            return true;\n        default:\n            return false;\n        }\n    }\n\n    bool is_write() const {\n        switch (opcode()) {\n        case operation::write:\n        case operation::writev:\n        case operation::send:\n        case operation::sendmsg:\n            return true;\n        default:\n            return false;\n        }\n    }\n\n    sstring opname() const;\n\n    // All operation variants are tagged unions with an operation as\n    // the first member, which we read through the _read union member\n    // (chosen arbitrarily) which is allowed by the common-initial-subsequence rule.\n    operation opcode() const {\n        return _read.op;\n    }\n\n    template <operation Op>\n    auto& as() const {\n        if constexpr (Op == operation::read) {\n            return _read;\n        }\n        if constexpr (Op == operation::readv) {\n            return _readv;\n        }\n        if constexpr (Op == operation::recv) {\n            return _recv;\n        }\n        if constexpr (Op == operation::recvmsg) {\n            return _recvmsg;\n        }\n        if constexpr (Op == operation::send) {\n            return _send;\n        }\n        if constexpr (Op == operation::sendmsg) {\n            return _sendmsg;\n        }\n        if constexpr (Op == operation::write) {\n            return _write;\n        }\n        if constexpr (Op == operation::writev) {\n            return _writev;\n        }\n        if constexpr (Op == operation::fdatasync) {\n            return _fdatasync;\n        }\n        if constexpr (Op == operation::accept) {\n            return _accept;\n        }\n        if constexpr (Op == operation::connect) {\n            return _connect;\n        }\n        if constexpr (Op == operation::poll_add) {\n            return _poll_add;\n        }\n        if constexpr (Op == operation::poll_remove) {\n            return _poll_remove;\n        }\n        if constexpr (Op == operation::cancel) {\n            return _cancel;\n        }\n    }\n\n    struct part;\n    std::vector<part> split(size_t max_length);\n\nprivate:\n    io_request sub_req_buffer(size_t pos, size_t len) const {\n        io_request sub_req;\n        // read_op and write_op share the same layout, so we don't handle\n        // them separately\n        auto& op = _read;\n        auto& sub_op = sub_req._read;\n        sub_op = {\n          .op = op.op,\n          .nowait_works = op.nowait_works,\n          .fd = op.fd,\n          .pos = op.pos + pos,\n          .addr = op.addr + pos,\n          .size = len,\n        };\n        return sub_req;\n    }\n    std::vector<part> split_buffer(size_t max_length);\n\n    io_request sub_req_iovec(size_t pos, std::vector<iovec>& iov) const {\n        io_request sub_req;\n        // readv_op and writev_op share the same layout, so we don't handle\n        // them separately\n        auto& op = _readv;\n        auto& sub_op = sub_req._readv;\n        sub_op = {\n          .op = op.op,\n          .nowait_works = op.nowait_works,\n          .fd = op.fd,\n          .pos = op.pos + pos,\n          .iovec = iov.data(),\n          .iov_len = iov.size(),\n        };\n        return sub_req;\n    }\n    std::vector<part> split_iovec(size_t max_length);\n};\n\nstruct io_request::part {\n    io_request req;\n    size_t size;\n    std::vector<::iovec> iovecs;\n};\n\n// Helper pair of IO direction and length\nstruct io_direction_and_length {\n    size_t _directed_length; // bit 0 is R/W flag\n\npublic:\n    size_t length() const noexcept { return _directed_length >> 1; }\n    int rw_idx() const noexcept { return _directed_length & 0x1; }\n    static constexpr int read_idx = 1;\n    static constexpr int write_idx = 0;\n\n    io_direction_and_length(int idx, size_t val) noexcept\n            : _directed_length((val << 1) | idx)\n    {\n        SEASTAR_ASSERT(idx == read_idx || idx == write_idx);\n    }\n};\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/internal/io_sink.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/internal/io_request.hh>\n\n#include <concepts>\n#include <cstddef>\n#include <type_traits>\n#include <utility>\n\nnamespace seastar {\n\nclass io_completion;\n\nnamespace internal {\n\nclass io_sink;\n\nclass pending_io_request : private internal::io_request {\n    friend class io_sink;\n    io_completion* _completion;\n\npublic:\n    pending_io_request(internal::io_request req, io_completion* desc) noexcept\n            : io_request(std::move(req))\n            , _completion(desc)\n    { }\n};\n\nclass io_sink {\n    chunked_fifo<pending_io_request> _pending_io;\npublic:\n    void submit(io_completion* desc, internal::io_request req) noexcept;\n\n    template <typename Fn>\n    // Fn should return whether the request was consumed and\n    // draining should try to drain more\n    requires std::is_invocable_r<bool, Fn, internal::io_request&, io_completion*>::value\n    size_t drain(Fn&& consume) {\n        size_t drained = 0;\n\n        for (auto& req : _pending_io) {\n            if (!consume(req, req._completion)) {\n                break;\n            }\n            drained++;\n        }\n\n        _pending_io.pop_front_n(drained);\n        return drained;\n    }\n};\n\n} // namespace internal\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/internal/linux-aio.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <endian.h>\n#include <sys/time.h>\n#include <sys/uio.h>\n#include <cstdint>\n\nnamespace seastar {\n\nnamespace internal {\n\nnamespace linux_abi {\n\nusing aio_context_t = unsigned long;\n\nenum class iocb_cmd : uint16_t {\n    PREAD = 0,\n    PWRITE = 1,\n    FSYNC = 2,\n    FDSYNC = 3,\n    POLL = 5,\n    NOOP = 6,\n    PREADV = 7,\n    PWRITEV = 8,\n};\n\nstruct io_event {\n    uint64_t data;\n    uint64_t obj;\n    int64_t res;\n    int64_t res2;\n};\n\nconstexpr int IOCB_FLAG_RESFD = 1;\n\nstruct iocb {\n        uint64_t   aio_data;\n\n#if __BYTE_ORDER == __LITTLE_ENDIAN\n        uint32_t   aio_key;\n        int32_t aio_rw_flags;\n#elif __BYTE_ORDER == __BIG_ENDIAN\n        int32_t aio_rw_flags;\n        uint32_t   aio_key;\n#else\n#error bad byteorder\n#endif\n\n        iocb_cmd   aio_lio_opcode;\n        int16_t   aio_reqprio;\n        uint32_t   aio_fildes;\n\n        uint64_t   aio_buf;\n        uint64_t   aio_nbytes;\n        int64_t   aio_offset;\n\n        uint64_t   aio_reserved2;\n\n        uint32_t   aio_flags;\n\n        uint32_t   aio_resfd;\n};\n\nstruct aio_sigset {\n    const sigset_t *sigmask;\n    size_t sigsetsize;\n};\n\n}\n\nlinux_abi::iocb make_read_iocb(int fd, uint64_t offset, void* buffer, size_t len);\nlinux_abi::iocb make_write_iocb(int fd, uint64_t offset, const void* buffer, size_t len);\nlinux_abi::iocb make_readv_iocb(int fd, uint64_t offset, const ::iovec* iov, size_t niov);\nlinux_abi::iocb make_writev_iocb(int fd, uint64_t offset, const ::iovec* iov, size_t niov);\nlinux_abi::iocb make_poll_iocb(int fd, uint32_t events);\n\nvoid set_user_data(linux_abi::iocb& iocb, void* data);\n\nvoid set_eventfd_notification(linux_abi::iocb& iocb, int eventfd);\n\nlinux_abi::iocb* get_iocb(const linux_abi::io_event& ioev);\n\nint io_setup(int nr_events, linux_abi::aio_context_t* io_context);\nint io_destroy(linux_abi::aio_context_t io_context) noexcept;\nint io_submit(linux_abi::aio_context_t io_context, long nr, linux_abi::iocb** iocbs);\nint io_cancel(linux_abi::aio_context_t io_context, linux_abi::iocb* iocb, linux_abi::io_event* result);\nint io_getevents(linux_abi::aio_context_t io_context, long min_nr, long nr, linux_abi::io_event* events, const ::timespec* timeout,\n        bool force_syscall = false);\nint io_pgetevents(linux_abi::aio_context_t io_context, long min_nr, long nr, linux_abi::io_event* events, const ::timespec* timeout, const sigset_t* sigmask,\n        bool force_syscall = false);\n\nvoid setup_aio_context(size_t nr, linux_abi::aio_context_t* io_context);\n\ninline\nlinux_abi::iocb\nmake_read_iocb(int fd, uint64_t offset, void* buffer, size_t len) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::PREAD;\n    iocb.aio_fildes = fd;\n    iocb.aio_offset = offset;\n    iocb.aio_buf = reinterpret_cast<uintptr_t>(buffer);\n    iocb.aio_nbytes = len;\n    return iocb;\n}\n\ninline\nlinux_abi::iocb\nmake_write_iocb(int fd, uint64_t offset, const void* buffer, size_t len) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::PWRITE;\n    iocb.aio_fildes = fd;\n    iocb.aio_offset = offset;\n    iocb.aio_buf = reinterpret_cast<uintptr_t>(buffer);\n    iocb.aio_nbytes = len;\n    return iocb;\n}\n\ninline\nlinux_abi::iocb\nmake_readv_iocb(int fd, uint64_t offset, const ::iovec* iov, size_t niov) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::PREADV;\n    iocb.aio_fildes = fd;\n    iocb.aio_offset = offset;\n    iocb.aio_buf = reinterpret_cast<uintptr_t>(iov);\n    iocb.aio_nbytes = niov;\n    return iocb;\n}\n\ninline\nlinux_abi::iocb\nmake_writev_iocb(int fd, uint64_t offset, const ::iovec* iov, size_t niov) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::PWRITEV;\n    iocb.aio_fildes = fd;\n    iocb.aio_offset = offset;\n    iocb.aio_buf = reinterpret_cast<uintptr_t>(iov);\n    iocb.aio_nbytes = niov;\n    return iocb;\n}\n\ninline\nlinux_abi::iocb\nmake_poll_iocb(int fd, uint32_t events) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::POLL;\n    iocb.aio_fildes = fd;\n    iocb.aio_buf = events;\n    return iocb;\n}\n\ninline\nlinux_abi::iocb\nmake_fdsync_iocb(int fd) {\n    linux_abi::iocb iocb{};\n    iocb.aio_lio_opcode = linux_abi::iocb_cmd::FDSYNC;\n    iocb.aio_fildes = fd;\n    return iocb;\n}\n\ninline\nvoid\nset_user_data(linux_abi::iocb& iocb, void* data) {\n    iocb.aio_data = reinterpret_cast<uintptr_t>(data);\n}\n\ntemplate <typename T>\ninline T* get_user_data(const linux_abi::iocb& iocb) noexcept {\n    return reinterpret_cast<T*>(uintptr_t(iocb.aio_data));\n}\n\ntemplate <typename T>\ninline T* get_user_data(const linux_abi::io_event& ev) noexcept {\n    return reinterpret_cast<T*>(uintptr_t(ev.data));\n}\n\ninline\nvoid\nset_eventfd_notification(linux_abi::iocb& iocb, int eventfd) {\n    iocb.aio_flags |= linux_abi::IOCB_FLAG_RESFD;\n    iocb.aio_resfd = eventfd;\n}\n\ninline\nlinux_abi::iocb*\nget_iocb(const linux_abi::io_event& ev) {\n    return reinterpret_cast<linux_abi::iocb*>(uintptr_t(ev.obj));\n}\n\ninline\nvoid\nset_nowait(linux_abi::iocb& iocb, bool nowait) {\n#ifdef RWF_NOWAIT\n        if (nowait) {\n            iocb.aio_rw_flags |= RWF_NOWAIT;\n        } else {\n            iocb.aio_rw_flags &= ~RWF_NOWAIT;\n        }\n#endif\n}\n\n} // internal namespace\n\n} // seastar namespace\n\n"
  },
  {
    "path": "include/seastar/core/internal/poll.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar {\n\nstruct pollfn {\n    virtual ~pollfn() {}\n    // Returns true if work was done (false = idle)\n    virtual bool poll() = 0;\n    // Checks if work needs to be done, but without actually doing any\n    // returns true if works needs to be done (false = idle)\n    virtual bool pure_poll() = 0;\n    // Tries to enter interrupt mode.\n    //\n    // If it returns true, then events from this poller will wake\n    // a sleeping idle loop, and exit_interrupt_mode() must be called\n    // to return to normal polling.\n    //\n    // If it returns false, the sleeping idle loop may not be entered.\n    virtual bool try_enter_interrupt_mode() = 0;\n    virtual void exit_interrupt_mode() = 0;\n};\n\n// The common case for poller -- do not make any difference between\n// poll() and pure_poll(), always/never agree to go to sleep and do\n// nothing on wakeup.\ntemplate <bool Passive>\nstruct simple_pollfn : public pollfn {\n    virtual bool pure_poll() override final {\n        return poll();\n    }\n    virtual bool try_enter_interrupt_mode() override final {\n        return Passive;\n    }\n    virtual void exit_interrupt_mode() override final {\n    }\n};\n\nnamespace internal {\n\ntemplate <typename Func>\nrequires std::is_invocable_r_v<bool, Func>\ninline\nstd::unique_ptr<seastar::pollfn> make_pollfn(Func&& func) {\n    struct the_pollfn : simple_pollfn<false> {\n        the_pollfn(Func&& func) : func(std::forward<Func>(func)) {}\n        Func func;\n        virtual bool poll() override final {\n            return func();\n        }\n    };\n    return std::make_unique<the_pollfn>(std::forward<Func>(func));\n}\n\nclass poller {\n    std::unique_ptr<pollfn> _pollfn;\n    class registration_task;\n    class deregistration_task;\n    registration_task* _registration_task = nullptr;\npublic:\n    template <typename Func>\n    requires std::is_invocable_r_v<bool, Func>\n    static poller simple(Func&& poll) {\n        return poller(make_pollfn(std::forward<Func>(poll)));\n    }\n    poller(std::unique_ptr<pollfn> fn)\n            : _pollfn(std::move(fn)) {\n        do_register();\n    }\n    ~poller();\n    poller(poller&& x) noexcept;\n    poller& operator=(poller&& x) noexcept;\n    void do_register() noexcept;\n    friend class reactor;\n};\n\n} // internal namespace\n\n}\n"
  },
  {
    "path": "include/seastar/core/internal/pollable_fd.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/util/bool_class.hh>\n#include <boost/intrusive_ptr.hpp>\n#include <cstdint>\n#include <span>\n#include <vector>\n#include <tuple>\n#include <sys/uio.h>\n\nnamespace seastar {\n\nclass reactor;\nclass pollable_fd;\nclass pollable_fd_state;\nclass socket_address;\n\nnamespace internal {\n\nclass buffer_allocator;\n\n}\n\nnamespace net {\n\nclass packet;\n\n}\n\nclass pollable_fd_state;\n\nusing pollable_fd_state_ptr = boost::intrusive_ptr<pollable_fd_state>;\n\nclass pollable_fd_state {\n    unsigned _refs = 0;\npublic:\n    struct speculation {\n        int events = 0;\n        explicit speculation(int epoll_events_guessed = 0) : events(epoll_events_guessed) {}\n    };\n    pollable_fd_state(const pollable_fd_state&) = delete;\n    void operator=(const pollable_fd_state&) = delete;\n    /// Set the speculation of specified I/O events\n    ///\n    /// We try to speculate. If an I/O is completed successfully without being\n    /// blocked and it didn't return the short read/write. We anticipate that\n    /// the next I/O will also be non-blocking and will not return EAGAIN.\n    /// But the speculation is invalidated once it is \"used\" by\n    /// \\c take_speculation()\n    void speculate_epoll(int events) { events_known |= events; }\n    /// Check whether we speculate specified I/O is possible on the fd,\n    /// invalidate the speculation if it matches with all specified \\c events.\n    ///\n    /// \\return true if the current speculation includes all specified events\n    bool take_speculation(int events) {\n        // invalidate the speculation set by the last speculate_epoll() call,\n        if (events_known & events) {\n            events_known &= ~events;\n            return true;\n        }\n        return false;\n    }\n    file_desc fd;\n    bool events_rw = false;   // single consumer for both read and write (accept())\n    unsigned shutdown_mask = 0;  // For udp, there is no shutdown indication from the kernel\n    int events_requested = 0; // wanted by pollin/pollout promises\n    int events_epoll = 0;     // installed in epoll\n    int events_known = 0;     // returned from epoll\n\n    friend class reactor;\n    friend class pollable_fd;\n    friend class reactor_backend_uring;\n\n    future<size_t> read_some(char* buffer, size_t size);\n    future<size_t> read_some(uint8_t* buffer, size_t size);\n    future<size_t> read_some(const std::vector<iovec>& iov);\n    future<temporary_buffer<char>> read_some(internal::buffer_allocator* ba);\n#if SEASTAR_API_LEVEL >= 9\n    future<size_t> write_some(std::span<iovec> iovs);\n    future<> write_all(std::span<iovec> iovs);\n#else\n    future<size_t> write_some(net::packet& p);\n    future<> write_all(net::packet& p);\n    future<> write_all(const char* buffer, size_t size);\n    future<> write_all(const uint8_t* buffer, size_t size);\n#endif\n    future<> readable();\n    future<> writeable();\n    future<> readable_or_writeable();\n    future<std::tuple<pollable_fd, socket_address>> accept();\n    future<> connect(socket_address& sa);\n    future<temporary_buffer<char>> recv_some(internal::buffer_allocator* ba);\n    future<size_t> sendmsg(struct msghdr *msg);\n    future<size_t> recvmsg(struct msghdr *msg);\n    future<size_t> sendto(socket_address addr, const void* buf, size_t len);\n    future<> poll_rdhup();\n    void shutdown(int how);\n\nprotected:\n    explicit pollable_fd_state(file_desc fd, speculation speculate = speculation())\n        : fd(std::move(fd)), events_known(speculate.events) {}\n    ~pollable_fd_state() = default;\nprivate:\n    void maybe_no_more_recv();\n    void maybe_no_more_send();\n    void forget(); // called on end-of-life\n\n    friend void intrusive_ptr_add_ref(pollable_fd_state* fd) {\n        ++fd->_refs;\n    }\n    friend void intrusive_ptr_release(pollable_fd_state* fd);\n    static pollable_fd_state_ptr make(file_desc, speculation);\n};\n\nclass pollable_fd {\npublic:\n    using speculation = pollable_fd_state::speculation;\n    pollable_fd() = default;\n    pollable_fd(file_desc fd, speculation speculate = speculation())\n        : _s(pollable_fd_state::make(std::move(fd), speculate))\n    {}\n\npublic:\n    future<size_t> read_some(char* buffer, size_t size) {\n        return _s->read_some(buffer, size);\n    }\n    future<size_t> read_some(uint8_t* buffer, size_t size) {\n        return _s->read_some(buffer, size);\n    }\n    future<size_t> read_some(const std::vector<iovec>& iov) {\n        return _s->read_some(iov);\n    }\n    future<temporary_buffer<char>> read_some(internal::buffer_allocator* ba) {\n        return _s->read_some(ba);\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<size_t> write_some(std::span<iovec> iov) {\n        return _s->write_some(iov);\n    }\n    future<> write_all(std::span<iovec> iov) {\n        return _s->write_all(iov);\n    }\n#else\n    future<size_t> write_some(net::packet& p) {\n        return _s->write_some(p);\n    }\n    future<> write_all(net::packet& p) {\n        return _s->write_all(p);\n    }\n    future<> write_all(const char* buffer, size_t size) {\n        return _s->write_all(buffer, size);\n    }\n    future<> write_all(const uint8_t* buffer, size_t size) {\n        return _s->write_all(buffer, size);\n    }\n#endif\n    future<> readable() {\n        return _s->readable();\n    }\n    future<> writeable() {\n        return _s->writeable();\n    }\n    future<> readable_or_writeable() {\n        return _s->readable_or_writeable();\n    }\n    future<std::tuple<pollable_fd, socket_address>> accept() {\n        return _s->accept();\n    }\n    future<> connect(socket_address& sa) {\n        return _s->connect(sa);\n    }\n    future<temporary_buffer<char>> recv_some(internal::buffer_allocator* ba) {\n        return _s->recv_some(ba);\n    }\n    future<size_t> sendmsg(struct msghdr *msg) {\n        return _s->sendmsg(msg);\n    }\n    future<size_t> recvmsg(struct msghdr *msg) {\n        return _s->recvmsg(msg);\n    }\n    future<size_t> sendto(socket_address addr, const void* buf, size_t len) {\n        return _s->sendto(addr, buf, len);\n    }\n    file_desc& get_file_desc() const { return _s->fd; }\n    using shutdown_kernel_only = bool_class<struct shutdown_kernel_only_tag>;\n    void shutdown(int how, shutdown_kernel_only kernel_only = shutdown_kernel_only::yes);\n    void close() { _s.reset(); }\n    explicit operator bool() const noexcept {\n        return bool(_s);\n    }\n    future<> poll_rdhup() {\n        return _s->poll_rdhup();\n    }\n    int get_fd() const { return _s->fd.get(); }\n\nprotected:\n    void maybe_no_more_recv() { return _s->maybe_no_more_recv(); }\n    void maybe_no_more_send() { return _s->maybe_no_more_send(); }\n\nprivate:\n    pollable_fd_state_ptr _s;\n};\n\nclass writeable_eventfd;\n\nclass readable_eventfd {\n    pollable_fd _fd;\npublic:\n    explicit readable_eventfd(size_t initial = 0) : _fd(try_create_eventfd(initial)) {}\n    readable_eventfd(readable_eventfd&&) = default;\n    writeable_eventfd write_side();\n    future<size_t> wait();\n    int get_write_fd() { return _fd.get_fd(); }\nprivate:\n    explicit readable_eventfd(file_desc&& fd) : _fd(std::move(fd)) {}\n    static file_desc try_create_eventfd(size_t initial);\n\n    friend class writeable_eventfd;\n};\n\nclass writeable_eventfd {\n    file_desc _fd;\npublic:\n    explicit writeable_eventfd(size_t initial = 0) : _fd(try_create_eventfd(initial)) {}\n    writeable_eventfd(writeable_eventfd&&) = default;\n    readable_eventfd read_side();\n    void signal(size_t nr);\n    int get_read_fd() { return _fd.get(); }\nprivate:\n    explicit writeable_eventfd(file_desc&& fd) : _fd(std::move(fd)) {}\n    static file_desc try_create_eventfd(size_t initial);\n\n    friend class readable_eventfd;\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/internal/run_in_background.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n\nnamespace seastar::internal {\n\nvoid run_in_background(future<> f);\n\ntemplate <typename Func>\nvoid run_in_background(Func&& func) {\n    run_in_background(futurize_invoke(std::forward<Func>(func)));\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/internal/stall_detector.hh",
    "content": "\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB\n */\n\n#pragma once\n\n#include <signal.h>\n#include <atomic>\n#include <limits>\n#include <chrono>\n#include <functional>\n#include <memory>\n#include <linux/perf_event.h>\n#include <seastar/core/posix.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/scheduling.hh>\n\nnamespace seastar {\n\nclass reactor;\nclass thread_cputime_clock;\nclass backtrace_buffer;\n\nnamespace internal {\n\nstruct cpu_stall_detector_config {\n    std::chrono::duration<double> threshold = std::chrono::seconds(2);\n    unsigned stall_detector_reports_per_minute = 1;\n    float slack = 0.3;  // fraction of threshold that we're allowed to overshoot\n    bool oneline = true; // print a simplified backtrace on a single line\n    std::function<void ()> report;  // alternative reporting function for tests\n};\n\n// Detects stalls in continuations that run for too long\nclass cpu_stall_detector {\nprotected:\n    std::atomic<uint64_t> _last_tasks_processed_seen{};\n    std::atomic<uint64_t> _stall_detector_missed_ticks = { 0 };\n    unsigned _reported = 0;\n    unsigned _total_reported = 0;\n    unsigned _max_reports_per_minute;\n    unsigned _shard_id;\n    unsigned _thread_id;\n    unsigned _report_at{};\n    sched_clock::time_point _minute_mark{};\n    sched_clock::time_point _rearm_timer_at{};\n    sched_clock::time_point _run_started_at{};\n    sched_clock::duration _threshold;\n    sched_clock::duration _slack;\n    cpu_stall_detector_config _config;\n    seastar::metrics::metric_groups _metrics;\n    friend reactor;\n    virtual bool is_spurious_signal() {\n        return false;\n    }\n    virtual void maybe_report_kernel_trace(backtrace_buffer& buf) {}\nprivate:\n    void maybe_report();\n    virtual void arm_timer() = 0;\n    void report_suppressions(sched_clock::time_point now);\n    void reset_suppression_state(sched_clock::time_point now);\npublic:\n    using clock_type = thread_cputime_clock;\npublic:\n    explicit cpu_stall_detector(cpu_stall_detector_config cfg = {});\n    virtual ~cpu_stall_detector() = default;\n    static int signal_number() { return SIGRTMIN + 1; }\n    void start_task_run(sched_clock::time_point now);\n    void end_task_run(sched_clock::time_point now);\n    void generate_trace();\n    void update_config(cpu_stall_detector_config cfg);\n    cpu_stall_detector_config get_config() const;\n    void on_signal();\n    virtual void start_sleep() = 0;\n    void end_sleep();\n};\n\nclass cpu_stall_detector_posix_timer : public cpu_stall_detector {\n    timer_t _timer;\npublic:\n    explicit cpu_stall_detector_posix_timer(cpu_stall_detector_config cfg = {});\n    virtual ~cpu_stall_detector_posix_timer() override;\nprivate:\n    virtual void arm_timer() override;\n    virtual void start_sleep() override;\n};\n\nclass cpu_stall_detector_linux_perf_event : public cpu_stall_detector {\n    file_desc _fd;\n    bool _enabled = false;\n    uint64_t _current_period = 0;\n    struct ::perf_event_mmap_page* _mmap;\n    char* _data_area;\n    size_t _data_area_mask;\n    // after the detector has been armed (i.e., _enabled is true), this\n    // is the moment at or after which the next signal is expected to occur\n    // and can be used for detecting spurious signals\n    sched_clock::time_point _next_signal_time{};\nprivate:\n    class data_area_reader {\n        cpu_stall_detector_linux_perf_event& _p;\n        const char* _data_area;\n        size_t _data_area_mask;\n        uint64_t _head;\n        uint64_t _tail;\n    public:\n        explicit data_area_reader(cpu_stall_detector_linux_perf_event& p)\n                : _p(p)\n                , _data_area(p._data_area)\n                , _data_area_mask(p._data_area_mask) {\n            _head = _p._mmap->data_head;\n            _tail = _p._mmap->data_tail;\n            std::atomic_thread_fence(std::memory_order_acquire); // required after reading data_head\n        }\n        ~data_area_reader() {\n            std::atomic_thread_fence(std::memory_order_release); // not documented, but probably required before writing data_tail\n            _p._mmap->data_tail = _tail;\n        }\n        uint64_t read_u64() {\n            uint64_t ret;\n            // We cannot wrap around if the 8-byte unit is aligned\n            std::copy_n(_data_area + (_tail & _data_area_mask), 8, reinterpret_cast<char*>(&ret));\n            _tail += 8;\n            return ret;\n        }\n        template <typename S>\n        S read_struct() {\n            static_assert(sizeof(S) % 8 == 0);\n            S ret;\n            char* p = reinterpret_cast<char*>(&ret);\n            for (size_t i = 0; i != sizeof(S); i += 8) {\n                uint64_t w = read_u64();\n                std::copy_n(reinterpret_cast<const char*>(&w), 8, p + i);\n            }\n            return ret;\n        }\n        void skip(uint64_t bytes_to_skip) {\n            _tail += bytes_to_skip;\n        }\n        // skip all the remaining data in the buffer, as-if calling read until\n        // have_data returns false (but much faster)\n        void skip_all() {\n            _tail = _head;\n        }\n        bool have_data() const {\n            return _head != _tail;\n        }\n    };\n\n    virtual void maybe_report_kernel_trace(backtrace_buffer& buf) override;\npublic:\n    static std::unique_ptr<cpu_stall_detector_linux_perf_event> try_make(cpu_stall_detector_config cfg = {});\n    explicit cpu_stall_detector_linux_perf_event(file_desc fd, cpu_stall_detector_config cfg = {});\n    ~cpu_stall_detector_linux_perf_event();\n    virtual void arm_timer() override;\n    virtual void start_sleep() override;\n    virtual bool is_spurious_signal() override;\n};\n\nstd::unique_ptr<cpu_stall_detector> make_cpu_stall_detector(cpu_stall_detector_config cfg = {});\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/internal/systemwide_memory_barrier.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Scylla DB\n */\n\n#pragma once\n\nnamespace seastar {\nnamespace internal {\n\n// cause all threads to invoke a full memory barrier\nvoid systemwide_memory_barrier();\n// attempt to invoke a systemwide memory barrier; return false\n// if doing so would cause lock contention in the kernel\nbool try_systemwide_memory_barrier();\n\n}\n}\n\n"
  },
  {
    "path": "include/seastar/core/internal/uname.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <optional>\n#include <string>\n#include <initializer_list>\n#include <iosfwd>\n\nnamespace seastar {\n\nnamespace internal {\n\n// Representation of a Linux kernel version number\nstruct uname_t {\n    int version;   // 4 in \"4.5\"\n    int patchlevel;   // 5 in \"4.5\"\n    std::optional<int> sublevel;   // 1 in \"4.5.1\"\n    std::optional<int> subsublevel;  // 33 in \"2.6.44.33\"\n    std::optional<int> distro_patch; // 957 in \"3.10.0-957.5.1.el7.x86_64\"\n    std::string distro_extra; // .5.1.el7.x86_64\n\n    bool same_as_or_descendant_of(const uname_t& x) const;\n    bool same_as_or_descendant_of(const char* x) const;\n    bool whitelisted(std::initializer_list<const char*>) const;\n\n    // 3 for \"4.5.0\", 5 for \"5.1.3-33.3.el7\"; \"el7\" doesn't count as a component\n    int component_count() const;\n\n    // The \"el7\" that wasn't counted in components()\n    bool has_distro_extra(std::string extra) const;\n    friend std::ostream& operator<<(std::ostream& os, const uname_t& u);\n};\n\nuname_t kernel_uname();\n\nuname_t parse_uname(const char* u);\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/io_intent.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/internal/io_intent.hh>\n#include <seastar/core/io_priority_class.hh>\n#include <boost/container/small_vector.hpp>\n\nnamespace seastar {\n\n/// \\example file_demo.cc\n/// A handle confirming the caller's intent to do the IO\n///\n/// When a pointer to an intent is passed to the \\ref io_queue\n/// \"io_queue\"::queue_request() method, the issued request is pinned\n/// to the intent and is only processed as long as the intent object\n/// is alive and the **cancel()** method is not called.\n///\n/// If no intent is provided, then the request is processed till its\n/// completion be it success or error\nclass io_intent {\n    struct intents_for_queue {\n        unsigned qid;\n        io_priority_class_id cid;\n        internal::cancellable_queue cq;\n\n        intents_for_queue(unsigned qid_, io_priority_class_id cid_) noexcept\n            : qid(qid_), cid(cid_), cq() {}\n\n        intents_for_queue(intents_for_queue&&) noexcept = default;\n        intents_for_queue& operator=(intents_for_queue&&) noexcept = default;\n    };\n\n    struct references {\n        internal::intent_reference::container_type list;\n\n        references(references&& o) noexcept : list(std::move(o.list)) {}\n        references() noexcept : list() {}\n        ~references() { clear(); }\n\n        void clear() {\n            list.clear_and_dispose([] (internal::intent_reference* r) { r->on_cancel(); });\n        }\n\n        void bind(internal::intent_reference& iref) noexcept {\n            list.push_back(iref);\n        }\n    };\n\n    boost::container::small_vector<intents_for_queue, 1> _intents;\n    references _refs;\n    friend internal::intent_reference::intent_reference(io_intent*) noexcept;\n\npublic:\n    io_intent() = default;\n    ~io_intent() = default;\n\n    io_intent(const io_intent&) = delete;\n    io_intent& operator=(const io_intent&) = delete;\n    io_intent& operator=(io_intent&&) = delete;\n    io_intent(io_intent&& o) noexcept : _intents(std::move(o._intents)), _refs(std::move(o._refs)) {\n        for (auto&& r : _refs.list) {\n            r._intent = this;\n        }\n    }\n\n    /// Explicitly cancels all the requests attached to this intent\n    /// so far. The respective futures are resolved into the \\ref\n    /// cancelled_error \"cancelled_error\"\n    void cancel() noexcept {\n        _refs.clear();\n        _intents.clear();\n    }\n\n    /// @private\n    internal::cancellable_queue& find_or_create_cancellable_queue(unsigned qid, io_priority_class_id cid) {\n        for (auto&& i : _intents) {\n            if (i.qid == qid && i.cid == cid) {\n                return i.cq;\n            }\n        }\n\n        _intents.emplace_back(qid, cid);\n        return _intents.back().cq;\n    }\n};\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/io_priority_class.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/future.hh>\n\n\nnamespace seastar {\n\nclass io_queue;\n\n\nusing io_priority_class_id = unsigned;\n\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/io_queue.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <boost/container/static_vector.hpp>\n#include <chrono>\n#include <memory>\n#include <vector>\n#include <sys/uio.h>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/fair_queue.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/internal/io_request.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/util/spinlock.hh>\n#include <seastar/util/shared_token_bucket.hh>\n\nstruct io_queue_for_tests;\n\nnamespace seastar {\n\nclass io_queue;\nclass io_throttler;\n\nnamespace internal {\nconst io_throttler& get_throttler(const io_queue& ioq, unsigned stream);\n}\n\nclass io_intent;\n\nnamespace internal {\nclass io_sink;\n}\n\nusing shard_id = unsigned;\nusing stream_id = unsigned;\n\nclass io_desc_read_write;\nclass queued_io_request;\nclass io_group;\n\nusing io_group_ptr = std::shared_ptr<io_group>;\nusing iovec_keeper = std::vector<::iovec>;\n\nnamespace internal {\nclass priority_class {\n    unsigned _id;\npublic:\n    explicit priority_class(const scheduling_group& sg) noexcept;\n    unsigned id() const noexcept { return _id; }\n};\n}\n\nclass io_queue {\npublic:\n    class priority_class_data;\n    using clock_type = std::chrono::steady_clock;\n\nprivate:\n    std::vector<std::unique_ptr<priority_class_data>> _priority_classes;\n    io_group_ptr _group;\n    const unsigned _id;\n    struct stream {\n        using capacity_t = fair_queue_entry::capacity_t;\n        fair_queue fq;\n        clock_type::time_point replenish;\n        io_throttler& out;\n        // _pending represents a reservation of tokens from the bucket.\n        //\n        // In the \"dispatch timeline\" defined by the growing bucket head of the group,\n        // tokens in the range [_pending.head - cap, _pending.head) belong\n        // to this queue.\n        //\n        // For example, if:\n        //    _group._token_bucket.head == 300\n        //    _pending.head == 700\n        //    _pending.cap == 500\n        // then the reservation is [200, 700), 100 tokens are ready to be dispatched by this queue,\n        // and another 400 tokens are going to be appear soon. (And after that, this queue\n        // will be able to make its next reservation).\n        struct pending {\n            capacity_t head = 0;\n            capacity_t cap = 0;\n        };\n        pending _pending;\n        stream(io_throttler& t, fair_queue::config cfg)\n            : fq(std::move(cfg))\n            , replenish(clock_type::now())\n            , out(t)\n        {}\n\n        // Shaves off the fulfilled frontal part from `_pending` (if any),\n        // and returns the fulfilled tokens in `ready_tokens`.\n        // Sets `our_turn_has_come` to the truth value of \"`_pending` is empty or\n        // there are no unfulfilled reservations (from other shards) earlier than `_pending`\".\n        //\n        // Assumes that `_group.maybe_replenish_capacity()` was called recently.\n        struct reap_result {\n            capacity_t ready_tokens;\n            bool our_turn_has_come;\n        };\n        enum class grab_result { ok, stop, again };\n\n        clock_type::time_point next_pending_aio() const noexcept;\n        reap_result reap_pending_capacity() noexcept;\n        grab_result grab_capacity(capacity_t cap, reap_result& available);\n\n        std::vector<seastar::metrics::impl::metric_definition_impl> metrics(const priority_class_data&);\n    };\n    boost::container::static_vector<stream, 2> _streams;\n    internal::io_sink& _sink;\n\n    friend struct ::io_queue_for_tests;\n    friend const io_throttler& internal::get_throttler(const io_queue& ioq, unsigned stream);\n\n    priority_class_data& find_or_create_class(internal::priority_class pc);\n    future<size_t> queue_request(internal::priority_class pc, internal::io_direction_and_length dnl, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept;\n    future<size_t> queue_one_request(internal::priority_class pc, internal::io_direction_and_length dnl, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept;\n\n    // The fields below are going away, they are just here so we can implement deprecated\n    // functions that used to be provided by the fair_queue and are going away (from both\n    // the fair_queue and the io_queue). Double-accounting for now will allow for easier\n    // decoupling and is temporary\n    size_t _queued_requests = 0;\n    size_t _requests_executing = 0;\n    uint64_t _requests_dispatched = 0;\n    uint64_t _requests_completed = 0;\n\n    // Flow monitor\n    uint64_t _prev_dispatched = 0;\n    uint64_t _prev_completed = 0;\n    double _flow_ratio = 1.0;\n\n    timer<lowres_clock> _averaging_decay_timer;\n\n    const std::chrono::milliseconds _stall_threshold_min;\n    std::chrono::milliseconds _stall_threshold;\n    std::optional<uint32_t> _physical_block_size;\n\n    void update_flow_ratio() noexcept;\n    void lower_stall_threshold() noexcept;\n\n    metrics::metric_groups _metric_groups;\npublic:\n\n    // We want to represent the fact that write requests are (maybe) more expensive\n    // than read requests. To avoid dealing with floating point math we will scale one\n    // read request to be counted by this amount.\n    //\n    // A write request that is 30% more expensive than a read will be accounted as\n    // (read_request_base_count * 130) / 100.\n    // It is also technically possible for reads to be the expensive ones, in which case\n    // writes will have an integer value lower than read_request_base_count.\n    static constexpr unsigned read_request_base_count = 128;\n    static constexpr unsigned block_size_shift = 9;\n\n    struct config {\n        unsigned id;\n        unsigned long req_count_rate = std::numeric_limits<unsigned long>::max();\n        unsigned long blocks_count_rate = std::numeric_limits<unsigned long>::max();\n        unsigned disk_req_write_to_read_multiplier = read_request_base_count;\n        unsigned disk_blocks_write_to_read_multiplier = read_request_base_count;\n        size_t disk_read_saturation_length = std::numeric_limits<size_t>::max();\n        size_t disk_write_saturation_length = std::numeric_limits<size_t>::max();\n        sstring mountpoint = \"undefined\";\n        bool duplex = false;\n        std::chrono::duration<double> rate_limit_duration = std::chrono::milliseconds(1);\n        size_t block_count_limit_min = 1;\n        unsigned averaging_decay_ticks = 100;\n        double flow_ratio_ema_factor = 0.95;\n        double flow_ratio_backpressure_threshold = 1.1;\n        std::chrono::milliseconds stall_threshold = std::chrono::milliseconds(100);\n        std::chrono::microseconds tau = std::chrono::milliseconds(5);\n        std::optional<uint32_t> physical_block_size; // Override for disks that lie about their physical block size\n    };\n\n    io_queue(io_group_ptr group, internal::io_sink& sink);\n    ~io_queue();\n\n    stream_id request_stream(internal::io_direction_and_length dnl) const noexcept;\n\n    future<size_t> submit_io_read(size_t len, internal::io_request req, io_intent* intent, iovec_keeper iovs = {}) noexcept;\n    future<size_t> submit_io_write(size_t len, internal::io_request req, io_intent* intent, iovec_keeper iovs = {}) noexcept;\n\n    void submit_request(io_desc_read_write* desc, internal::io_request req) noexcept;\n    void cancel_request(queued_io_request& req) noexcept;\n    void complete_cancelled_request(queued_io_request& req) noexcept;\n    void complete_request(io_desc_read_write& desc, std::chrono::duration<double> delay) noexcept;\n\n    // Dispatch requests that are pending in the I/O queue\n    void poll_io_queue();\n\n    clock_type::time_point next_pending_aio() const noexcept;\n    fair_queue_entry::capacity_t request_capacity(internal::io_direction_and_length dnl) const noexcept;\n\n    sstring mountpoint() const;\n    unsigned id() const noexcept { return _id; }\n\n    void update_shares_for_class(internal::priority_class pc, size_t new_shares);\n    void update_shares_for_class_group(unsigned index, size_t new_shares);\n    future<> update_bandwidth_for_class(internal::priority_class pc, uint64_t new_bandwidth);\n    future<> update_bandwidth_for_class_group(unsigned group_index, uint64_t new_bandwidth);\n    void rename_priority_class(internal::priority_class pc, sstring new_name);\n    void destroy_priority_class(internal::priority_class pc) noexcept;\n    void throttle_priority_class(const priority_class_data& pc) noexcept;\n    void unthrottle_priority_class(const priority_class_data& pc) noexcept;\n    void throttle_priority_class_group(unsigned group) noexcept;\n    void unthrottle_priority_class_group(unsigned group) noexcept;\n\n    struct request_limits {\n        size_t max_read;\n        size_t max_write;\n    };\n\n    request_limits get_request_limits() const noexcept;\n    const config& get_config() const noexcept;\n    std::chrono::duration<double> get_io_latency_goal() const noexcept;\n    std::optional<uint32_t> physical_block_size() const noexcept { return _physical_block_size; }\n\n    // @private -- temporary, to make posix_file_impl work\n    internal::io_sink& sink() noexcept { return _sink; }\n\nprivate:\n    static fair_queue::config make_fair_queue_config(const config& cfg, sstring label);\n    void register_stats(sstring name, priority_class_data& pc);\n};\n\n/// \\brief Outgoing throttler\n///\n/// This is a fair group. It's attached by one or mode fair queues. On machines having the\n/// big* amount of shards, queues use the group to borrow/lend the needed capacity for\n/// requests dispatching.\n///\n/// * Big means that when all shards sumbit requests alltogether the disk is unable to\n/// dispatch them efficiently. The inability can be of two kinds -- either disk cannot\n/// cope with the number of arriving requests, or the total size of the data withing\n/// the given time frame exceeds the disk throughput.\nclass io_throttler {\npublic:\n    using capacity_t = fair_queue_entry::capacity_t;\n    using clock_type = std::chrono::steady_clock;\n\n    /*\n     * tldr; The math\n     *\n     *    Bw, Br -- write/read bandwidth (bytes per second)\n     *    Ow, Or -- write/read iops (ops per second)\n     *\n     *    xx_max -- their maximum values (configured)\n     *\n     * Throttling formula:\n     *\n     *    Bw/Bw_max + Br/Br_max + Ow/Ow_max + Or/Or_max <= K\n     *\n     * where K is the scalar value <= 1.0 (also configured)\n     *\n     * Bandwidth is bytes time derivatite, iops is ops time derivative, i.e.\n     * Bx = d(bx)/dt, Ox = d(ox)/dt. Then the formula turns into\n     *\n     *   d(bw/Bw_max + br/Br_max + ow/Ow_max + or/Or_max)/dt <= K\n     *\n     * Fair queue tickets are {w, s} weight-size pairs that are\n     *\n     *   s = read_base_count * br, for reads\n     *       Br_max/Bw_max * read_base_count * bw, for writes\n     *\n     *   w = read_base_count, for reads\n     *       Or_max/Ow_max * read_base_count, for writes\n     *\n     * Thus the formula turns into\n     *\n     *   d(sum(w/W + s/S))/dr <= K\n     *\n     * where {w, s} is the ticket value if a request and sum summarizes the\n     * ticket values from all the requests seen so far, {W, S} is the ticket\n     * value that corresonds to a virtual summary of Or_max requests of\n     * Br_max size total.\n     */\n\n    /*\n     * The normalization results in a float of the 2^-30 seconds order of\n     * magnitude. Not to invent float point atomic arithmetics, the result\n     * is converted to an integer by multiplying by a factor that's large\n     * enough to turn these values into a non-zero integer.\n     *\n     * Also, the rates in bytes/sec when adjusted by io-queue according to\n     * multipliers become too large to be stored in 32-bit ticket value.\n     * Thus the rate resolution is applied. The t.bucket is configured with a\n     * time period for which the speeds from F (in above formula) are taken.\n     */\n\n    static constexpr float fixed_point_factor = float(1 << 24);\n    using rate_resolution = std::milli;\n    using token_bucket_t = internal::shared_token_bucket<capacity_t, rate_resolution, internal::capped_release::no>;\n\nprivate:\n\n    /*\n     * The dF/dt <= K limitation is managed by the modified token bucket\n     * algo where tokens are ticket.normalize(cost_capacity), the refill\n     * rate is K.\n     *\n     * The token bucket algo must have the limit on the number of tokens\n     * accumulated. Here it's configured so that it accumulates for the\n     * latency_goal duration.\n     *\n     * The replenish threshold is the minimal number of tokens to put back.\n     * It's reserved for future use to reduce the load on the replenish\n     * timestamp.\n     *\n     * The timestamp, in turn, is the time when the bucket was replenished\n     * last. Every time a shard tries to get tokens from bucket it first\n     * tries to convert the time that had passed since this timestamp\n     * into more tokens in the bucket.\n     */\n\n    token_bucket_t _token_bucket;\n    const capacity_t _per_tick_threshold;\n\npublic:\n\n    // Convert internal capacity value back into the real token\n    static double capacity_tokens(capacity_t cap) noexcept {\n        return (double)cap / fixed_point_factor / token_bucket_t::rate_cast(std::chrono::seconds(1)).count();\n    }\n\n    // Convert floating-point tokens into the token bucket capacity\n    static capacity_t tokens_capacity(double tokens) noexcept {\n        return tokens * token_bucket_t::rate_cast(std::chrono::seconds(1)).count() * fixed_point_factor;\n    }\n\n    auto capacity_duration(capacity_t cap) const noexcept {\n        return _token_bucket.duration_for(cap);\n    }\n\n    struct config {\n        sstring label = \"\";\n        /*\n         * There are two \"min\" values that can be configured. The former one\n         * is the minimal weight:size pair that the upper layer is going to\n         * submit. However, it can submit _larger_ values, and the fair queue\n         * must accept those as large as the latter pair (but it can accept\n         * even larger values, of course)\n         */\n        double min_tokens = 0.0;\n        double limit_min_tokens = 0.0;\n        std::chrono::duration<double> rate_limit_duration = std::chrono::milliseconds(1);\n    };\n\n    explicit io_throttler(config cfg, unsigned nr_queues);\n    io_throttler(io_throttler&&) = delete;\n\n    capacity_t maximum_capacity() const noexcept { return _token_bucket.limit(); }\n    capacity_t per_tick_grab_threshold() const noexcept { return _per_tick_threshold; }\n    capacity_t grab_capacity(capacity_t cap) noexcept;\n    clock_type::time_point replenished_ts() const noexcept { return _token_bucket.replenished_ts(); }\n    void refund_tokens(capacity_t) noexcept;\n    void replenish_capacity(clock_type::time_point now) noexcept;\n    void maybe_replenish_capacity(clock_type::time_point& local_ts) noexcept;\n\n    capacity_t capacity_deficiency(capacity_t from) const noexcept;\n\n    std::chrono::duration<double> rate_limit_duration() const noexcept {\n        std::chrono::duration<double, rate_resolution> dur((double)_token_bucket.limit() / _token_bucket.rate());\n        return std::chrono::duration_cast<std::chrono::duration<double>>(dur);\n    }\n\n    const token_bucket_t& token_bucket() const noexcept { return _token_bucket; }\n};\n\nclass io_group {\npublic:\n    explicit io_group(io_queue::config io_cfg, unsigned nr_queues);\n    ~io_group();\n    struct priority_class_data;\n    using priority_class_group_data = priority_class_data;\n\n    std::chrono::duration<double> io_latency_goal() const noexcept;\n\nprivate:\n    friend class io_queue;\n    friend struct ::io_queue_for_tests;\n    friend const io_throttler& internal::get_throttler(const io_queue& ioq, unsigned stream);\n\n    /*\n     * This value is used as a cut-off point for calculating the maximum request length.\n     * We look for max(2^i) such that capacity(2^i) < maximum capacity. If 2^i gets bigger\n     * than this value, we stop looking and the maximum request length is set to this value.\n     */\n    static constexpr unsigned request_length_limit = 16 << 20; // 16 MiB\n\n    const io_queue::config _config;\n    size_t _max_request_length[2] = {\n        request_length_limit, // write\n        request_length_limit  // read\n    };\n    boost::container::static_vector<io_throttler, 2> _fgs;\n    std::vector<std::unique_ptr<priority_class_group_data>> _priority_groups;\n    std::vector<std::unique_ptr<priority_class_data>> _priority_classes;\n    util::spinlock _lock;\n    const shard_id _allocated_on;\n\n    static io_throttler::config configure_throttler(const io_queue::config& qcfg) noexcept;\n    priority_class_data& find_or_create_class(internal::priority_class pc);\n    priority_class_data& find_or_create_class(internal::priority_class pc, std::optional<unsigned> group_index);\n    priority_class_group_data& find_or_create_class_group(unsigned group_index);\n    priority_class_group_data& find_or_create_class_group_locked(unsigned group_index);\n\n    inline size_t max_request_length(int dnl_idx) const noexcept {\n        return _max_request_length[dnl_idx];\n    }\n};\n\ninline const io_queue::config& io_queue::get_config() const noexcept {\n    return _group->_config;\n}\n\ninline sstring io_queue::mountpoint() const {\n    return get_config().mountpoint;\n}\n\nnamespace internal {\ndouble request_tokens(io_direction_and_length dnl, const io_queue::config& cfg) noexcept;\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/iostream-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n\n#pragma once\n\n#include <numeric>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/variant_utils.hh>\n\nnamespace seastar {\n\ninline future<temporary_buffer<char>> data_source_impl::skip(uint64_t n)\n{\n    return do_with(uint64_t(n), [this] (uint64_t& n) {\n        return repeat_until_value([&] {\n            return get().then([&] (temporary_buffer<char> buffer) -> std::optional<temporary_buffer<char>> {\n                if (buffer.empty()) {\n                    return buffer;\n                }\n                if (buffer.size() >= n) {\n                    buffer.trim_front(n);\n                    return buffer;\n                }\n                n -= buffer.size();\n                return { };\n            });\n        });\n    });\n}\n\ntemplate<typename CharType>\ninline\nfuture<> output_stream<CharType>::write(const char_type* buf) noexcept {\n    return write(buf, strlen(buf));\n}\n\ntemplate<typename CharType>\ntemplate<typename StringChar, typename SizeType, SizeType MaxSize, bool NulTerminate>\ninline\nfuture<> output_stream<CharType>::write(const basic_sstring<StringChar, SizeType, MaxSize, NulTerminate>& s) noexcept {\n    return write(reinterpret_cast<const CharType *>(s.c_str()), s.size());\n}\n\ntemplate<typename CharType>\ninline\nfuture<> output_stream<CharType>::write(const std::basic_string<CharType>& s) noexcept {\n    return write(s.c_str(), s.size());\n}\n\ntemplate<typename CharType>\nfuture<>\noutput_stream<CharType>::zero_copy_put(std::vector<temporary_buffer<CharType>> b) noexcept {\n    // if flush is scheduled, disable it, so it will not try to write in parallel\n    _flush = false;\n    if (_flushing) {\n        // flush in progress, wait for it to end before continuing\n        return _in_batch.value().get_future().then([this, b = std::move(b)] () mutable {\n            return _fd.put(std::move(b));\n        });\n    } else {\n        return _fd.put(std::move(b));\n    }\n}\n\n// Writes @p in chunks of _buffer_size length. The last chunk is buffered if smaller.\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::zero_copy_split_and_put(std::vector<temporary_buffer<CharType>> b, size_t len) noexcept {\n    return repeat([this, b = std::move(b), len] () mutable {\n        if (len < _buffer_size) {\n            _zc_bufs = std::move(b);\n            _zc_len = len;\n            return make_ready_future<stop_iteration>(stop_iteration::yes);\n        }\n        auto chunk = internal::detach_front(b, _buffer_size);\n        len -= _buffer_size;\n        return zero_copy_put(std::move(chunk)).then([] {\n            return stop_iteration::no;\n        });\n    });\n}\n\ntemplate<typename CharType>\nfuture<> output_stream<CharType>::write(std::span<temporary_buffer<CharType>> bufs) noexcept {\n    static_assert(std::is_same_v<CharType, char>, \"packet works on char\");\n  try {\n    size_t size = std::accumulate(bufs.begin(), bufs.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n    if (size != 0) {\n        if (_end) {\n            // Seal the filled prefix as a shared view into _buf, then\n            // advance _buf past it so the same allocation can be reused\n            // for future buffered writes after this zero-copy sequence.\n            _zc_bufs.emplace_back(_buf.share(0, _end));\n            _buf.trim_front(_end);\n            if (!_buf.size()) {\n                _buf = {};\n            }\n            _zc_len += _end;\n            _end = 0;\n        }\n\n        _zc_len += size;\n        _zc_bufs.insert(_zc_bufs.end(), std::make_move_iterator(bufs.begin()), std::make_move_iterator(bufs.end()));\n        if (_zc_len >= _buffer_size) {\n            if (_trim_to_size) {\n                return zero_copy_split_and_put(std::move(_zc_bufs), std::exchange(_zc_len, 0));\n            } else {\n                _zc_len = 0;\n                return zero_copy_put(std::move(_zc_bufs));\n            }\n        }\n    }\n    return make_ready_future<>();\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate<typename CharType>\nfuture<> output_stream<CharType>::write(temporary_buffer<CharType> p) noexcept {\n  try {\n    return write(std::span<temporary_buffer<CharType>>(&p, 1));\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\n#if SEASTAR_API_LEVEL < 9\ntemplate<typename CharType>\nfuture<> output_stream<CharType>::write(net::packet p) noexcept {\n    try {\n        std::vector<temporary_buffer<CharType>> bufs = std::move(p).release();\n        return write(std::span<temporary_buffer<CharType>>(bufs));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate<typename CharType>\nfuture<> output_stream<CharType>::write(scattered_message<CharType> msg) noexcept {\n    return write(std::move(msg).release());\n}\n#endif\n\ntemplate <typename CharType>\nfuture<temporary_buffer<CharType>>\ninput_stream<CharType>::read_exactly_part(size_t n) noexcept {\n    temporary_buffer<CharType> out(n);\n    size_t completed{0U};\n    while (completed < n) {\n        size_t avail = available();\n        if (avail) {\n            auto now = std::min(n - completed, avail);\n            std::copy_n(_buf.get(), now, out.get_write() + completed);\n            _buf.trim_front(now);\n            completed += now;\n            if (completed == n) {\n                break;\n            }\n        }\n\n        // _buf is now empty\n        temporary_buffer<CharType> buf = co_await _fd.get();\n        if (buf.size() == 0) {\n            _eof = true;\n            out.trim(completed);\n            break;\n        }\n        _buf = std::move(buf);\n    }\n    co_return out;\n}\n\ntemplate <typename CharType>\nfuture<temporary_buffer<CharType>>\ninput_stream<CharType>::read_exactly(size_t n) noexcept {\n    if (_buf.size() == n) {\n        // easy case: steal buffer, return to caller\n        return make_ready_future<tmp_buf>(std::move(_buf));\n    } else if (_buf.size() > n) {\n        // buffer large enough, share it with caller\n        auto front = _buf.share(0, n);\n        _buf.trim_front(n);\n        return make_ready_future<tmp_buf>(std::move(front));\n    } else if (_buf.size() == 0) {\n        // buffer is empty: grab one and retry\n        return _fd.get().then([this, n] (auto buf) mutable {\n            if (buf.size() == 0) {\n                _eof = true;\n                return make_ready_future<tmp_buf>(std::move(buf));\n            }\n            _buf = std::move(buf);\n            return this->read_exactly(n);\n        });\n    } else {\n        // buffer too small: start copy/read loop\n        return read_exactly_part(n);\n    }\n}\n\ntemplate <typename CharType>\ntemplate <typename Consumer>\nrequires InputStreamConsumer<Consumer, CharType> || ObsoleteInputStreamConsumer<Consumer, CharType>\nfuture<>\ninput_stream<CharType>::consume(Consumer&& consumer) noexcept(std::is_nothrow_move_constructible_v<Consumer>) {\n    return repeat([consumer = std::move(consumer), this] () mutable {\n        if (_buf.empty() && !_eof) {\n            return _fd.get().then([this] (tmp_buf buf) {\n                _buf = std::move(buf);\n                _eof = _buf.empty();\n                return make_ready_future<stop_iteration>(stop_iteration::no);\n            });\n        }\n        return consumer(std::move(_buf)).then([this] (consumption_result_type result) {\n            return seastar::visit(result.get(), [this] (const continue_consuming&) {\n               // If we're here, consumer consumed entire buffer and is ready for\n                // more now. So we do not return, and rather continue the loop.\n                //\n                // If we're at eof, we should stop.\n                return make_ready_future<stop_iteration>(stop_iteration(this->_eof));\n            }, [this] (stop_consuming<CharType>& stop) {\n                // consumer is done\n                this->_buf = std::move(stop.get_buffer());\n                return make_ready_future<stop_iteration>(stop_iteration::yes);\n            }, [this] (const skip_bytes& skip) {\n                return this->_fd.skip(skip.get_value()).then([this](tmp_buf buf) {\n                    if (!buf.empty()) {\n                        this->_buf = std::move(buf);\n                    }\n                    return make_ready_future<stop_iteration>(stop_iteration::no);\n                });\n            });\n        });\n    });\n}\n\ntemplate <typename CharType>\ntemplate <typename Consumer>\nrequires InputStreamConsumer<Consumer, CharType> || ObsoleteInputStreamConsumer<Consumer, CharType>\nfuture<>\ninput_stream<CharType>::consume(Consumer& consumer) noexcept(std::is_nothrow_move_constructible_v<Consumer>) {\n    return consume(std::ref(consumer));\n}\n\ntemplate <typename CharType>\nfuture<temporary_buffer<CharType>>\ninput_stream<CharType>::read_up_to(size_t n) noexcept {\n    using tmp_buf = temporary_buffer<CharType>;\n    if (_buf.empty()) {\n        if (_eof) {\n            return make_ready_future<tmp_buf>();\n        } else {\n            return _fd.get().then([this, n] (tmp_buf buf) {\n                _eof = buf.empty();\n                _buf = std::move(buf);\n                return read_up_to(n);\n            });\n        }\n    } else if (_buf.size() <= n) {\n        // easy case: steal buffer, return to caller\n        return make_ready_future<tmp_buf>(std::move(_buf));\n    } else {\n      try {\n        // buffer is larger than n, so share its head with a caller\n        auto front = _buf.share(0, n);\n        _buf.trim_front(n);\n        return make_ready_future<tmp_buf>(std::move(front));\n      } catch (...) {\n        return current_exception_as_future<tmp_buf>();\n      }\n    }\n}\n\ntemplate <typename CharType>\nfuture<temporary_buffer<CharType>>\ninput_stream<CharType>::read() noexcept {\n    using tmp_buf = temporary_buffer<CharType>;\n    if (_eof) {\n        return make_ready_future<tmp_buf>();\n    }\n    if (_buf.empty()) {\n        return _fd.get().then([this] (tmp_buf buf) {\n            _eof = buf.empty();\n            return make_ready_future<tmp_buf>(std::move(buf));\n        });\n    } else {\n        return make_ready_future<tmp_buf>(std::move(_buf));\n    }\n}\n\ntemplate <typename CharType>\nfuture<>\ninput_stream<CharType>::skip(uint64_t n) noexcept {\n    auto skip_buf = std::min(n, _buf.size());\n    _buf.trim_front(skip_buf);\n    n -= skip_buf;\n    if (!n) {\n        return make_ready_future<>();\n    }\n    return _fd.skip(n).then([this] (temporary_buffer<CharType> buffer) {\n        _buf = std::move(buffer);\n    });\n}\n\ntemplate <typename CharType>\ndata_source\ninput_stream<CharType>::detach() && {\n    if (_buf) {\n        throw std::logic_error(\"detach() called on a used input_stream\");\n    }\n\n    return std::move(_fd);\n}\n\n// Writes @buf in chunks of _buffer_size length. The last chunk is buffered if smaller.\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::split_and_put(temporary_buffer<CharType> buf) noexcept {\n    SEASTAR_ASSERT(_end == 0);\n\n    return repeat([this, buf = std::move(buf)] () mutable {\n        if (buf.size() < _buffer_size) {\n            if (!_buf || _buf.size() < buf.size()) {\n                // _buf is absent or a trim_front'd remnant whose remaining\n                // capacity is smaller than the tail we need to store. We\n                // allocate a fresh buffer and abandon the remnant. The unused\n                // bytes of the remnant's underlying allocation are not leaked\n                // (the allocation is freed once all shared references to it are\n                // dropped), but they are wasted and will never be written to.\n                // This is a deliberate trade-off: filling the remnant partially\n                // and then copying the rest into a new buffer would require an\n                // async put() here, complicating the code with no clear benefit.\n                _buf = _fd.allocate_buffer(_buffer_size);\n            }\n            std::copy(buf.get(), buf.get() + buf.size(), _buf.get_write());\n            _end = buf.size();\n            return make_ready_future<stop_iteration>(stop_iteration::yes);\n        }\n        auto chunk = buf.share(0, _buffer_size);\n        buf.trim_front(_buffer_size);\n        return put(std::move(chunk)).then([] {\n            return stop_iteration::no;\n        });\n    });\n}\n\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::write(const char_type* buf, size_t n) noexcept {\n    if (__builtin_expect(!_buf || n > _buf.size() - _end, false)) {\n        return slow_write(buf, n);\n    }\n    std::copy_n(buf, n, _buf.get_write() + _end);\n    _end += n;\n    return make_ready_future<>();\n}\n\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::slow_write(const char_type* buf, size_t n) noexcept {\n    try {\n        if (!_end && (n >= _buffer_size)) {\n            temporary_buffer<char> tmp = _fd.allocate_buffer(n);\n            std::copy(buf, buf + n, tmp.get_write());\n            if (!_zc_bufs.empty()) {\n                // No buffered data yet, but zero-copy data is pending.\n                // Append to _zc_bufs so ordering is preserved.\n                _zc_bufs.emplace_back(std::move(tmp));\n                _zc_len += n;\n                if (_zc_len >= _buffer_size) {\n                    if (_trim_to_size) {\n                        return zero_copy_split_and_put(std::move(_zc_bufs), std::exchange(_zc_len, 0));\n                    } else {\n                        _zc_len = 0;\n                        return zero_copy_put(std::move(_zc_bufs));\n                    }\n                }\n                return make_ready_future<>();\n            }\n            if (_trim_to_size) {\n                return split_and_put(std::move(tmp));\n            } else {\n                return put(std::move(tmp));\n            }\n        }\n\n        if (!_buf) {\n            _buf = _fd.allocate_buffer(_buffer_size);\n        }\n\n        auto now = std::min(n, _buf.size() - _end);\n        std::copy(buf, buf + now, _buf.get_write() + _end);\n        _end += now;\n        if (now == n) {\n            return make_ready_future<>();\n        }\n        temporary_buffer<char> next = _fd.allocate_buffer(std::max(n - now, _buffer_size));\n        std::copy(buf + now, buf + n, next.get_write());\n        // Buffer is full. Seal both _buf and next into _zc_bufs if zero-copy\n        // data is pending (to preserve ordering), or if _buf is a trim_front'd\n        // remnant (flushing it directly would produce an undersized non-last chunk).\n        if (!_zc_bufs.empty() || _buf.size() < _buffer_size) {\n            _zc_bufs.emplace_back(_buf.share(0, _end));\n            _buf.trim_front(_end);\n            if (!_buf.size()) {\n                _buf = {};\n            }\n            _zc_len += _end;\n            _end = 0;\n            next.trim(n - now);\n            _zc_len += n - now;\n            _zc_bufs.emplace_back(std::move(next));\n            if (_zc_len >= _buffer_size) {\n                if (_trim_to_size) {\n                    return zero_copy_split_and_put(std::move(_zc_bufs), std::exchange(_zc_len, 0));\n                } else {\n                    _zc_len = 0;\n                    return zero_copy_put(std::move(_zc_bufs));\n                }\n            }\n            return make_ready_future<>();\n        }\n\n\n        if (n - now >= _buffer_size) {\n            _end = 0;\n            return put(std::move(_buf)).then([this, next = std::move(next)]() mutable {\n                if (_trim_to_size) {\n                    return split_and_put(std::move(next));\n                } else {\n                    return put(std::move(next));\n                }\n            });\n        }\n\n        _end = n - now;\n        return put(std::exchange(_buf, std::move(next)));\n    } catch (...) {\n      return current_exception_as_future();\n    }\n}\n\nnamespace internal {\nvoid add_to_flush_poller(output_stream<char>& x) noexcept;\n}\n\ntemplate <typename CharType>\nfuture<> output_stream<CharType>::do_flush() noexcept {\n    if (_end) {\n        if (_zc_bufs.empty()) {\n            _buf.trim(_end);\n            _end = 0;\n            return _fd.put(std::move(_buf)).then([this] {\n                return _fd.flush();\n            });\n        } else {\n            // Fold buffered tail into the zero-copy vector and flush together.\n            _zc_bufs.emplace_back(_buf.share(0, _end));\n            _buf.trim_front(_end);\n            _zc_len += _end;\n            _end = 0;\n        }\n    }\n    if (!_zc_bufs.empty()) {\n        _zc_len = 0;\n        return _fd.put(std::move(_zc_bufs)).then([this] {\n            return _fd.flush();\n        });\n    } else {\n        return _fd.flush();\n    }\n}\n\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::flush() noexcept {\n    if (!_batch_flushes) {\n        return do_flush();\n    } else {\n        if (_ex) {\n            // flush is a good time to deliver outstanding errors\n            return make_exception_future<>(std::move(_ex));\n        } else {\n            _flush = true;\n            if (!_in_batch) {\n                internal::add_to_flush_poller(*this);\n                _in_batch = promise<>();\n            }\n        }\n    }\n    return make_ready_future<>();\n}\n\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::put(temporary_buffer<CharType> buf) noexcept {\n    // if flush is scheduled, disable it, so it will not try to write in parallel\n    _flush = false;\n    if (_flushing) {\n        // flush in progress, wait for it to end before continuing\n        return _in_batch.value().get_future().then([this, buf = std::move(buf)] () mutable {\n            return _fd.put(std::move(buf));\n        });\n    } else {\n        return _fd.put(std::move(buf));\n    }\n}\n\ntemplate <typename CharType>\nvoid\noutput_stream<CharType>::poll_flush() noexcept {\n    if (!_flush) {\n        // flush was canceled, do nothing\n        _flushing = false;\n        _in_batch.value().set_value();\n        _in_batch = std::nullopt;\n        return;\n    }\n\n    _flush = false;\n    _flushing = true; // make whoever wants to write into the fd to wait for flush to complete\n\n    // FIXME: future is discarded\n    (void)do_flush().then_wrapped([this] (future<> f) {\n        try {\n            f.get();\n        } catch (...) {\n            _ex = std::current_exception();\n            _fd.on_batch_flush_error();\n        }\n        // if flush() was called while flushing flush once more\n        poll_flush();\n    });\n}\n\ntemplate <typename CharType>\nfuture<>\noutput_stream<CharType>::close() noexcept {\n    return flush().finally([this] {\n        if (_in_batch) {\n            return _in_batch.value().get_future();\n        } else {\n            return make_ready_future();\n        }\n    }).then([this] {\n        // report final exception as close error\n        if (_ex) {\n            std::rethrow_exception(_ex);\n        }\n    }).finally([this] {\n        return _fd.close();\n    });\n}\n\ntemplate <typename CharType>\ndata_sink\noutput_stream<CharType>::detach() && {\n    if (_buf || !_zc_bufs.empty()) {\n        throw std::logic_error(\"detach() called on a used output_stream\");\n    }\n\n    return std::move(_fd);\n}\n\nnamespace internal {\n\n/// \\cond internal\ntemplate <typename CharType>\nstruct stream_copy_consumer {\nprivate:\n    output_stream<CharType>& _os;\n    using consumption_result_type = consumption_result<CharType>;\npublic:\n    stream_copy_consumer(output_stream<CharType>& os) : _os(os) {\n    }\n    future<consumption_result_type> operator()(temporary_buffer<CharType> data) {\n        if (data.empty()) {\n            return make_ready_future<consumption_result_type>(stop_consuming(std::move(data)));\n        }\n        return _os.write(data.get(), data.size()).then([] {\n            return make_ready_future<consumption_result_type>(continue_consuming());\n        });\n    }\n};\n/// \\endcond\n\n}\n\nextern template struct internal::stream_copy_consumer<char>;\n\ntemplate <typename CharType>\nfuture<> copy(input_stream<CharType>& in, output_stream<CharType>& out) {\n    return in.consume(internal::stream_copy_consumer<CharType>(out));\n}\n\nextern template future<> copy<char>(input_stream<char>&, output_stream<char>&);\n}\n"
  },
  {
    "path": "include/seastar/core/iostream.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n//\n// Buffered input and output streams\n//\n// Two abstract classes (data_source and data_sink) provide means\n// to acquire bulk data from, or push bulk data to, some provider.\n// These could be tied to a TCP connection, a disk file, or a memory\n// buffer.\n//\n// Two concrete classes (input_stream and output_stream) buffer data\n// from data_source and data_sink and provide easier means to process\n// it.\n//\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/scattered_message.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <boost/intrusive/slist.hpp>\n#include <ranges>\n#include <algorithm>\n#include <memory>\n#include <optional>\n#include <variant>\n#include <vector>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\n\nnamespace net { class packet; }\nnamespace testing {\nclass input_stream_test;\nclass output_stream_test;\n}\n\nclass data_source_impl {\npublic:\n    virtual ~data_source_impl() {}\n    virtual future<temporary_buffer<char>> get() = 0;\n    virtual future<temporary_buffer<char>> skip(uint64_t n);\n    virtual future<> close() { return make_ready_future<>(); }\n};\n\nclass data_source {\n    std::unique_ptr<data_source_impl> _dsi;\nprotected:\n    data_source_impl* impl() const { return _dsi.get(); }\npublic:\n    using tmp_buf = temporary_buffer<char>;\n\n    data_source() noexcept = default;\n    explicit data_source(std::unique_ptr<data_source_impl> dsi) noexcept : _dsi(std::move(dsi)) {}\n    data_source(data_source&& x) noexcept = default;\n    data_source& operator=(data_source&& x) noexcept = default;\n\n    future<tmp_buf> get() noexcept {\n        try {\n            return _dsi->get();\n        } catch (...) {\n            return current_exception_as_future<tmp_buf>();\n        }\n    }\n    future<tmp_buf> skip(uint64_t n) noexcept {\n        try {\n            return _dsi->skip(n);\n        } catch (...) {\n            return current_exception_as_future<tmp_buf>();\n        }\n    }\n    future<> close() noexcept {\n        try {\n            return _dsi->close();\n        } catch (...) {\n            return current_exception_as_future<>();\n        }\n    }\n};\n\nclass data_sink_impl {\npublic:\n    virtual ~data_sink_impl() {}\n    virtual temporary_buffer<char> allocate_buffer(size_t size) {\n        return temporary_buffer<char>(size);\n    }\n#if SEASTAR_API_LEVEL >= 9\n    // The caller assumes that the storage that backs this span can be released\n    // once this method returns, so implementations should move the buffers into\n    // stable storage on their own early, before the returned future resolves.\n    virtual future<> put(std::span<temporary_buffer<char>> data) = 0;\n#else\n    virtual future<> put(net::packet data) = 0;\n    virtual future<> put(std::vector<temporary_buffer<char>> data) {\n        net::packet p;\n        p.reserve(data.size());\n        for (auto& buf : data) {\n            p = net::packet(std::move(p), net::fragment{buf.get_write(), buf.size()}, buf.release());\n        }\n        return put(std::move(p));\n    }\n    virtual future<> put(temporary_buffer<char> buf) {\n        return put(net::packet(net::fragment{buf.get_write(), buf.size()}, buf.release()));\n    }\n#endif\n    virtual future<> flush() {\n        return make_ready_future<>();\n    }\n    virtual future<> close() = 0;\n\n    // The method should return the maximum buffer size that's acceptable by\n    // the sink. It's used when the output stream is constructed without any\n    // specific buffer size. In this case the stream accepts this value as its\n    // buffer size and doesn't put larger buffers (see trim_to_size).\n    virtual size_t buffer_size() const noexcept {\n        SEASTAR_ASSERT(false && \"Data sink must have the buffer_size() method overload\");\n        return 0;\n    }\n\n    // In order to support flushes batching (output_stream_options.batch_flushes)\n    // the sink mush handle flush errors that may happen in the background by\n    // overriding the on_batch_flush_error() method. If the sink doesn't do it,\n    // turning on batch_flushes would have no effect\n    virtual bool can_batch_flushes() const noexcept {\n        return false;\n    }\n\n    virtual void on_batch_flush_error() noexcept {\n        SEASTAR_ASSERT(false && \"Data sink must implement on_batch_flush_error() method\");\n    }\n\nprotected:\n#if SEASTAR_API_LEVEL >= 9\n    // A helper function that class that inhrerit from data_sink_impl\n    // can use to create a future chain holding buffers from the span\n    // to sequentially put them with the help of fn function\n    template <typename Fn>\n    requires std::is_invocable_r_v<future<>, Fn, temporary_buffer<char>&&>\n    static future<> fallback_put(std::span<temporary_buffer<char>> bufs, Fn fn) {\n        if (bufs.size() == 1) [[likely]] {\n            return fn(std::move(bufs.front()));\n        }\n\n        auto f = fn(std::move(bufs.front()));\n        for (auto&& buf : bufs.subspan(1)) {\n            f = std::move(f).then([fn, buf = std::move(buf)] () mutable {\n                return fn(std::move(buf));\n            });\n        }\n        return f;\n    }\n#else\n    // This is a helper function that classes that inherit from data_sink_impl\n    // can use to implement the put overload for net::packet.\n    // Unfortunately, we currently cannot define this function as\n    // 'virtual future<> put(net::packet)', because we would get infinite\n    // recursion between this function and\n    // 'virtual future<> put(temporary_buffer<char>)'.\n    future<> fallback_put(net::packet data) {\n        auto buffers = data.release();\n        for (temporary_buffer<char>& buf : buffers) {\n            co_await this->put(std::move(buf));\n        }\n    }\n#endif\n};\n\nclass data_sink {\n    std::unique_ptr<data_sink_impl> _dsi;\npublic:\n    data_sink() noexcept = default;\n    explicit data_sink(std::unique_ptr<data_sink_impl> dsi) noexcept : _dsi(std::move(dsi)) {}\n    data_sink(data_sink&& x) noexcept = default;\n    data_sink& operator=(data_sink&& x) noexcept = default;\n    temporary_buffer<char> allocate_buffer(size_t size) {\n        return _dsi->allocate_buffer(size);\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> data) noexcept {\n        try {\n            return _dsi->put(data);\n        } catch (...) {\n            return current_exception_as_future();\n        }\n    }\n    future<> put(std::vector<temporary_buffer<char>> data) noexcept {\n        return put(std::span<temporary_buffer<char>>(data));\n    }\n    future<> put(temporary_buffer<char> data) noexcept {\n        return put(std::span<temporary_buffer<char>>(&data, 1));\n    }\n#else\n    future<> put(std::vector<temporary_buffer<char>> data) noexcept {\n      try {\n        return _dsi->put(std::move(data));\n      } catch (...) {\n        return current_exception_as_future();\n      }\n    }\n    future<> put(temporary_buffer<char> data) noexcept {\n      try {\n        return _dsi->put(std::move(data));\n      } catch (...) {\n        return current_exception_as_future();\n      }\n    }\n    future<> put(net::packet p) noexcept {\n      try {\n        return _dsi->put(std::move(p));\n      } catch (...) {\n        return current_exception_as_future();\n      }\n    }\n#endif\n    future<> flush() noexcept {\n      try {\n        return _dsi->flush();\n      } catch (...) {\n        return current_exception_as_future();\n      }\n    }\n    future<> close() noexcept {\n        try {\n            return _dsi->close();\n        } catch (...) {\n            return current_exception_as_future();\n        }\n    }\n\n    size_t buffer_size() const noexcept { return _dsi->buffer_size(); }\n    bool can_batch_flushes() const noexcept { return _dsi->can_batch_flushes(); }\n    void on_batch_flush_error() noexcept { _dsi->on_batch_flush_error(); }\n};\n\nstruct continue_consuming {};\n\ntemplate <typename CharType>\nclass stop_consuming {\npublic:\n    using tmp_buf = temporary_buffer<CharType>;\n    explicit stop_consuming(tmp_buf buf) : _buf(std::move(buf)) {}\n\n    tmp_buf& get_buffer() { return _buf; }\n    const tmp_buf& get_buffer() const { return _buf; }\nprivate:\n    tmp_buf _buf;\n};\n\nclass skip_bytes {\npublic:\n    explicit skip_bytes(uint64_t v) : _value(v) {}\n    uint64_t get_value() const { return _value; }\nprivate:\n    uint64_t _value;\n};\n\ntemplate <typename CharType>\nclass consumption_result {\npublic:\n    using stop_consuming_type = stop_consuming<CharType>;\n    using consumption_variant = std::variant<continue_consuming, stop_consuming_type, skip_bytes>;\n    using tmp_buf = typename stop_consuming_type::tmp_buf;\n\n    /*[[deprecated]]*/ consumption_result(std::optional<tmp_buf> opt_buf) {\n        if (opt_buf) {\n            _result = stop_consuming_type{std::move(opt_buf.value())};\n        }\n    }\n\n    consumption_result(const continue_consuming&) {}\n    consumption_result(stop_consuming_type&& stop) : _result(std::move(stop)) {}\n    consumption_result(skip_bytes&& skip) : _result(std::move(skip)) {}\n\n    consumption_variant& get() { return _result; }\n    const consumption_variant& get() const { return _result; }\n\nprivate:\n    consumption_variant _result;\n};\n\n// Consumer concept, for consume() method\n\n// The consumer should operate on the data given to it, and\n// return a future \"consumption result\", which can be\n//  - continue_consuming, if the consumer has consumed all the input given\n// to it and is ready for more\n//  - stop_consuming, when the consumer is done (and in that case\n// the contained buffer is the unconsumed part of the last data buffer - this\n// can also happen to be empty).\n//  - skip_bytes, when the consumer has consumed all the input given to it\n// and wants to skip before processing the next chunk\n//\n// For backward compatibility reasons, we also support the deprecated return value\n// of type \"unconsumed remainder\" which can be\n//  - empty optional, if the consumer consumed all the input given to it\n// and is ready for more\n//  - non-empty optional, when the consumer is done (and in that case\n// the value is the unconsumed part of the last data buffer - this\n// can also happen to be empty).\n\ntemplate <typename Consumer, typename CharType>\nconcept InputStreamConsumer = requires (Consumer c) {\n    { c(temporary_buffer<CharType>{}) } -> std::same_as<future<consumption_result<CharType>>>;\n};\n\ntemplate <typename Consumer, typename CharType>\nconcept ObsoleteInputStreamConsumer = requires (Consumer c) {\n    { c(temporary_buffer<CharType>{}) } -> std::same_as<future<std::optional<temporary_buffer<CharType>>>>;\n};\n\n/// Buffers data from a data_source and provides a stream interface to the user.\n///\n/// \\note All methods must be called sequentially.  That is, no method may be\n/// invoked before the previous method's returned future is resolved.\ntemplate <typename CharType>\nclass input_stream final {\n    static_assert(sizeof(CharType) == 1, \"must buffer stream of bytes\");\n    data_source _fd;\n    temporary_buffer<CharType> _buf;\n    bool _eof = false;\nprivate:\n    using tmp_buf = temporary_buffer<CharType>;\n    size_t available() const noexcept { return _buf.size(); }\nprotected:\n    void reset() noexcept { _buf = {}; }\n    data_source* fd() noexcept { return &_fd; }\npublic:\n    using consumption_result_type = consumption_result<CharType>;\n    // unconsumed_remainder is mapped for compatibility only; new code should use consumption_result_type\n    using unconsumed_remainder = std::optional<tmp_buf>;\n    using char_type = CharType;\n    [[deprecated(\"Uninitialized input_stream is useless\")]]\n    input_stream() noexcept = default;\n    explicit input_stream(data_source fd) noexcept : _fd(std::move(fd)), _buf() {}\n    input_stream(input_stream&&) = default;\n    [[deprecated(\"Input stream cannot be move-assigned, consider move-constructing the target in place\")]]\n    input_stream& operator=(input_stream&&) = default;\n    /// Reads n bytes from the stream, or fewer if reached the end of stream.\n    ///\n    /// \\returns a future that waits until n bytes are available in the\n    /// stream and returns them. If the end of stream is reached before n\n    /// bytes were read, fewer than n bytes will be returned - so despite\n    /// the method's name, the caller must not assume the returned buffer\n    /// will always contain exactly n bytes.\n    ///\n    /// \\throws if an I/O error occurs during the read. As explained above,\n    /// prematurely reaching the end of stream is *not* an I/O error.\n    future<temporary_buffer<CharType>> read_exactly(size_t n) noexcept;\n    template <typename Consumer>\n    requires InputStreamConsumer<Consumer, CharType> || ObsoleteInputStreamConsumer<Consumer, CharType>\n    future<> consume(Consumer&& c) noexcept(std::is_nothrow_move_constructible_v<Consumer>);\n    template <typename Consumer>\n    requires InputStreamConsumer<Consumer, CharType> || ObsoleteInputStreamConsumer<Consumer, CharType>\n    future<> consume(Consumer& c) noexcept(std::is_nothrow_move_constructible_v<Consumer>);\n    /// Returns true if the end-of-file flag is set on the stream.\n    /// Note that the eof flag is only set after a previous attempt to read\n    /// from the stream noticed the end of the stream. In other words, it is\n    /// possible that eof() returns false but read() will return an empty\n    /// buffer. Checking eof() again after the read will return true.\n    bool eof() const noexcept { return _eof; }\n    /// Returns some data from the stream, or an empty buffer on end of\n    /// stream.\n    future<tmp_buf> read() noexcept;\n    /// Returns up to n bytes from the stream, or an empty buffer on end of\n    /// stream.\n    future<tmp_buf> read_up_to(size_t n) noexcept;\n    /// Detaches the \\c input_stream from the underlying data source.\n    ///\n    /// Waits for any background operations (for example, read-ahead) to\n    /// complete, so that the any resources the stream is using can be\n    /// safely destroyed.  An example is a \\ref file resource used by\n    /// the stream returned by make_file_input_stream().\n    ///\n    /// \\return a future that becomes ready when this stream no longer\n    ///         needs the data source.\n    future<> close() noexcept {\n        return _fd.close();\n    }\n    /// Ignores n next bytes from the stream.\n    future<> skip(uint64_t n) noexcept;\n\n    /// Detaches the underlying \\c data_source from the \\c input_stream.\n    ///\n    /// The intended usage is custom \\c data_source_impl implementations\n    /// wrapping an existing \\c input_stream, therefore it shouldn't be\n    /// called on an \\c input_stream that was already used.\n    /// After calling \\c detach() the \\c input_stream is in an unusable,\n    /// moved-from state.\n    ///\n    /// \\throws std::logic_error if called on a used stream\n    ///\n    /// \\returns the data_source\n    data_source detach() &&;\nprivate:\n    future<temporary_buffer<CharType>> read_exactly_part(size_t n) noexcept;\n    friend class testing::input_stream_test;\n};\n\nstruct output_stream_options {\n    bool trim_to_size = false; ///< Make sure that buffers put into sink haven't\n                               ///< grown larger than the configured size\n    bool batch_flushes = false; ///< Try to merge flushes with each other\n};\n\n/// Facilitates data buffering before it's handed over to data_sink.\n///\n/// When trim_to_size is true it's guaranteed that data sink will not receive\n/// chunks larger than the configured size, which could be the case when a\n/// single write call is made with data larger than the configured size.\n///\n/// The data sink will not receive empty chunks.\n///\n/// There are two sets of write() overloads that put data into the stream.\n/// Methods from the first set accumulate the given data into the inner buffer\n/// by copying it there. Methods from the second set take the ownership of the\n/// provided object and append it to the stream without copying the data.\n///\n/// The copying write-s are good for constructing the stream out of small pieces\n/// but are not constrained with that usage. Data of any size can be passed, and\n/// it will be correctly split and copied if needed. Below these write()s are\n/// documented to \"write ... into the buffer\".\n///\n/// The no-copy write-s are good for large blobs as they avoid memcpy-ing the\n/// bytes around and just pass the memory handler around. Below these write()s\n/// are documented to \"append ... as zero-copy buffer\".\n///\n/// \\note All methods must be called sequentially.  That is, no method\n/// may be invoked before the previous method's returned future is\n/// resolved.\n///\n/// \\note Bufferred and zero-copy write()-s can be interleaved with care.\n/// If the stream was written to with zero-copy buffers, it must be flushed\n/// before writing bufferred data into it. However, bufferred data can be\n/// followed by zero-copy buffers put into stream. Respectively, once flushed\n/// the stream can be written to with bufferred data again.\ntemplate <typename CharType>\nclass output_stream final {\n    static_assert(sizeof(CharType) == 1, \"must buffer stream of bytes\");\n    data_sink _fd;\n    // Buffered writes accumulate into _buf[0.._end). Once _buf is full it is\n    // put() to the sink and a fresh buffer is allocated.\n    //\n    // Zero-copy writes bypass _buf and go directly into _zc_bufs. If buffered\n    // data is pending when a zero-copy write arrives, the filled prefix of _buf\n    // is shared into _zc_bufs and _buf is trim_front()'d so its tail can still\n    // be reused for future buffered writes after the zero-copy sequence.\n    //\n    // _zc_bufs is flushed to the sink (via zero_copy_put or\n    // zero_copy_split_and_put) when _zc_len >= _buffer_size, or on flush().\n    // After a flush _buf, _zc_bufs, _end and _zc_len are all reset to empty/0.\n    temporary_buffer<CharType> _buf;\n    std::vector<temporary_buffer<CharType>> _zc_bufs; // zero copy buffers\n    size_t _buffer_size = 0;\n    size_t _end = 0;\n    size_t _zc_len = 0;\n    bool _trim_to_size = false;\n    bool _batch_flushes = false;\n    std::optional<promise<>> _in_batch;\n    bool _flush = false;\n    bool _flushing = false;\n    std::exception_ptr _ex;\n    bi::slist_member_hook<> _in_poller;\n\nprivate:\n    future<> split_and_put(temporary_buffer<CharType> buf) noexcept;\n    future<> put(temporary_buffer<CharType> buf) noexcept;\n    void poll_flush() noexcept;\n    future<> do_flush() noexcept;\n    future<> zero_copy_put(std::vector<temporary_buffer<CharType>> b) noexcept;\n    future<> zero_copy_split_and_put(std::vector<temporary_buffer<CharType>> b, size_t len) noexcept;\n    [[gnu::noinline]]\n    future<> slow_write(const CharType* buf, size_t n) noexcept;\npublic:\n    using char_type = CharType;\n    [[deprecated(\"Uninitialized output_stream is useless\")]]\n    output_stream() noexcept = default;\n    output_stream(data_sink fd, size_t size, output_stream_options opts = {}) noexcept\n        : _fd(std::move(fd)), _buffer_size(size), _trim_to_size(opts.trim_to_size), _batch_flushes(opts.batch_flushes && _fd.can_batch_flushes()) {}\n    output_stream(data_sink fd) noexcept\n        : _fd(std::move(fd)), _buffer_size(_fd.buffer_size()), _trim_to_size(true) {}\n    output_stream(output_stream&&) noexcept = default;\n    [[deprecated(\"Output stream cannot be move-assigned, consider move-constructing the target in place\")]]\n    output_stream& operator=(output_stream&&) noexcept = default;\n    ~output_stream() {\n        if (_batch_flushes) {\n            SEASTAR_ASSERT(!_in_batch && \"Was this stream properly closed?\");\n        } else {\n            SEASTAR_ASSERT(!_end && !_zc_len && \"Was this stream properly closed?\");\n        }\n    }\n    /// Writes n bytes from the memory pointed by buf into the buffer\n    future<> write(const char_type* buf, size_t n) noexcept;\n    /// Writes zero-terminated string into the buffer\n    future<> write(const char_type* buf) noexcept;\n    /// Writes the given string into the buffer\n    template <typename StringChar, typename SizeType, SizeType MaxSize, bool NulTerminate>\n    future<> write(const basic_sstring<StringChar, SizeType, MaxSize, NulTerminate>& s) noexcept;\n    /// Writes the given string into the buffer\n    future<> write(const std::basic_string<char_type>& s) noexcept;\n\n#if SEASTAR_API_LEVEL < 9\n    /// Appends the packet as zero-copy buffer\n    future<> write(net::packet p) noexcept;\n    /// Appends the scattered message as zero-copy buffer\n    future<> write(scattered_message<char_type> msg) noexcept;\n#endif\n\n    /// Appends the temporary buffer as zero-copy buffer\n    future<> write(temporary_buffer<char_type>) noexcept;\n    /// Appends a bunch of buffers as zero-copy\n    future<> write(std::span<temporary_buffer<char_type>>) noexcept;\n\n    future<> flush() noexcept;\n\n    /// Flushes the stream before closing it (and the underlying data sink) to\n    /// any further writes.  The resulting future must be waited on before\n    /// destroying this object.\n    future<> close() noexcept;\n\n    /// Detaches the underlying \\c data_sink from the \\c output_stream.\n    ///\n    /// The intended usage is custom \\c data_sink_impl implementations\n    /// wrapping an existing \\c output_stream, therefore it shouldn't be\n    /// called on an \\c output_stream that was already used.\n    /// After calling \\c detach() the \\c output_stream is in an unusable,\n    /// moved-from state.\n    ///\n    /// \\throws std::logic_error if called on a used stream\n    ///\n    /// \\returns the data_sink\n    data_sink detach() &&;\n\n    using batch_flush_list_t = bi::slist<output_stream,\n            bi::constant_time_size<false>, bi::cache_last<true>,\n            bi::member_hook<output_stream, bi::slist_member_hook<>, &output_stream::_in_poller>>;\nprivate:\n    friend class reactor;\n    friend class testing::output_stream_test;\n};\n\n/*!\n * \\brief copy all the content from the input stream to the output stream\n */\ntemplate <typename CharType>\nfuture<> copy(input_stream<CharType>&, output_stream<CharType>&);\n\n}\n\n#include \"iostream-impl.hh\"\n"
  },
  {
    "path": "include/seastar/core/layered_file.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/file.hh>\n\nnamespace seastar {\n\n/// \\addtogroup fileio-module\n/// @{\n\n/// Base class for layered file implementations.\n///\n/// A layered file implementation implements `file_impl` virtual\n/// functions such as dma_read() by forwarding them to another, existing\n/// file called the underlying file. This base class simplifies construction\n/// of layered files by performing standard tasks such as setting up the\n/// file alignment. Actual implementation of the I/O methods is left for the\n/// derived class.\nclass layered_file_impl : public file_impl {\nprotected:\n    file _underlying_file;\npublic:\n    /// Constructs a layered file. This sets up the underlying_file() method\n    /// and initializes alignment constants to be the same as the underlying file.\n    explicit layered_file_impl(file underlying_file) noexcept\n            : _underlying_file(std::move(underlying_file)) {\n        _memory_dma_alignment = _underlying_file.memory_dma_alignment();\n        _disk_read_dma_alignment = _underlying_file.disk_read_dma_alignment();\n        _disk_write_dma_alignment = _underlying_file.disk_write_dma_alignment();\n        _disk_overwrite_dma_alignment = _underlying_file.disk_overwrite_dma_alignment();\n    }\n\n    /// The underlying file which can be used to back I/O methods.\n    file& underlying_file() noexcept {\n        return _underlying_file;\n    }\n\n    /// The underlying file which can be used to back I/O methods.\n    const file& underlying_file() const noexcept {\n        return _underlying_file;\n    }\n};\n\n\n/// @}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/loop.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#pragma once\n\n#include <cassert>\n#include <cstddef>\n#include <iterator>\n#include <memory>\n#include <optional>\n#include <type_traits>\n#include <vector>\n#include <seastar/core/future.hh>\n#include <seastar/core/task.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/bool_class.hh>\n#include <seastar/core/semaphore.hh>\n\nnamespace seastar {\n\n\n/// \\addtogroup future-util\n/// @{\n\n// The AsyncAction concept represents an action which can complete later than\n// the actual function invocation. It is represented by a function which\n// returns a future which resolves when the action is done.\n\nstruct stop_iteration_tag { };\nusing stop_iteration = bool_class<stop_iteration_tag>;\n\nnamespace internal {\n\ntemplate <typename AsyncAction>\nclass repeater final : public continuation_base<stop_iteration> {\n    promise<> _promise;\n    AsyncAction _action;\npublic:\n    explicit repeater(AsyncAction&& action) : _action(std::move(action)) {}\n    future<> get_future() { return _promise.get_future(); }\n    task* waiting_task() noexcept override { return _promise.waiting_task(); }\n    virtual void run_and_dispose() noexcept override {\n        if (_state.failed()) {\n            _promise.set_exception(std::move(_state).get_exception());\n            delete this;\n            return;\n        } else {\n            if (_state.get() == stop_iteration::yes) {\n                _promise.set_value();\n                delete this;\n                return;\n            }\n            _state = {};\n        }\n        try {\n            do {\n                auto f = futurize_invoke(_action);\n                if (!f.available()) {\n                    internal::set_callback(std::move(f), this);\n                    return;\n                }\n                if (f.failed()) {\n                    _promise.set_exception(f.get_exception());\n                    delete this;\n                    return;\n                }\n                if (f.get() == stop_iteration::yes) {\n                    _promise.set_value();\n                    delete this;\n                    return;\n                }\n            } while (!need_preempt());\n        } catch (...) {\n            _promise.set_exception(std::current_exception());\n            delete this;\n            return;\n        }\n        _state.set(stop_iteration::no);\n        schedule(this);\n    }\n};\n\n} // namespace internal\n\n// Delete these overloads so that the actual implementation can use a\n// universal reference but still reject lvalue references.\ntemplate<typename AsyncAction>\nfuture<> repeat(const AsyncAction& action) noexcept = delete;\ntemplate<typename AsyncAction>\nfuture<> repeat(AsyncAction& action) noexcept = delete;\n\n/// Invokes given action until it fails or the function requests iteration to stop by returning\n/// \\c stop_iteration::yes.\n///\n/// \\param action a callable taking no arguments, returning a future<stop_iteration>.  Will\n///               be called again as soon as the future resolves, unless the\n///               future fails, action throws, or it resolves with \\c stop_iteration::yes.\n///               If \\c action is an r-value it can be moved in the middle of iteration.\n/// \\return a ready future if we stopped successfully, or a failed future if\n///         a call to to \\c action failed.\ntemplate<typename AsyncAction>\nrequires std::is_invocable_r_v<stop_iteration, AsyncAction> || std::is_invocable_r_v<future<stop_iteration>, AsyncAction>\ninline\nfuture<> repeat(AsyncAction&& action) noexcept {\n    using futurator = futurize<std::invoke_result_t<AsyncAction>>;\n    static_assert(std::is_same_v<future<stop_iteration>, typename futurator::type>, \"bad AsyncAction signature\");\n    for (;;) {\n        // Do not type-erase here in case this is a short repeat()\n        auto f = futurator::invoke(action);\n\n        if (!f.available() || f.failed() || need_preempt()) {\n            return [&] () noexcept {\n                memory::scoped_critical_alloc_section _;\n                auto repeater = new internal::repeater<AsyncAction>(std::move(action));\n                auto ret = repeater->get_future();\n                internal::set_callback(std::move(f), repeater);\n                return ret;\n            }();\n        }\n\n        if (f.get() == stop_iteration::yes) {\n            return make_ready_future<>();\n        }\n    }\n}\n\n/// \\cond internal\n\ntemplate <typename T>\nstruct repeat_until_value_type_helper;\n\n/// Type helper for repeat_until_value()\ntemplate <typename T>\nstruct repeat_until_value_type_helper<future<std::optional<T>>> {\n    /// The type of the value we are computing\n    using value_type = T;\n    /// Type used by \\c AsyncAction while looping\n    using optional_type = std::optional<T>;\n    /// Return type of repeat_until_value()\n    using future_type = future<value_type>;\n};\n\n/// Return value of repeat_until_value()\ntemplate <typename AsyncAction>\nusing repeat_until_value_return_type\n        = typename repeat_until_value_type_helper<typename futurize<std::invoke_result_t<AsyncAction>>::type>::future_type;\n\n/// \\endcond\n\nnamespace internal {\n\ntemplate <typename AsyncAction, typename T>\nclass repeat_until_value_state final : public continuation_base<std::optional<T>> {\n    promise<T> _promise;\n    AsyncAction _action;\npublic:\n    explicit repeat_until_value_state(AsyncAction action) : _action(std::move(action)) {}\n    repeat_until_value_state(std::optional<T> st, AsyncAction action) : repeat_until_value_state(std::move(action)) {\n        this->_state.set(std::move(st));\n    }\n    future<T> get_future() { return _promise.get_future(); }\n    task* waiting_task() noexcept override { return _promise.waiting_task(); }\n    virtual void run_and_dispose() noexcept override {\n        if (this->_state.failed()) {\n            _promise.set_exception(std::move(this->_state).get_exception());\n            delete this;\n            return;\n        } else {\n            auto v = std::move(this->_state).get();\n            if (v) {\n                _promise.set_value(std::move(*v));\n                delete this;\n                return;\n            }\n            this->_state = {};\n        }\n        try {\n            do {\n                auto f = futurize_invoke(_action);\n                if (!f.available()) {\n                    internal::set_callback(std::move(f), this);\n                    return;\n                }\n                if (f.failed()) {\n                    _promise.set_exception(f.get_exception());\n                    delete this;\n                    return;\n                }\n                auto ret = f.get();\n                if (ret) {\n                    _promise.set_value(std::move(*ret));\n                    delete this;\n                    return;\n                }\n            } while (!need_preempt());\n        } catch (...) {\n            _promise.set_exception(std::current_exception());\n            delete this;\n            return;\n        }\n        this->_state.set(std::nullopt);\n        schedule(this);\n    }\n};\n\n} // namespace internal\n\n/// Invokes given action until it fails or the function requests iteration to stop by returning\n/// an engaged \\c future<std::optional<T>> or std::optional<T>.  The value is extracted\n/// from the \\c optional, and returned, as a future, from repeat_until_value().\n///\n/// \\param action a callable taking no arguments, returning a future<std::optional<T>>\n///               or std::optional<T>.  Will be called again as soon as the future\n///               resolves, unless the future fails, action throws, or it resolves with\n///               an engaged \\c optional.  If \\c action is an r-value it can be moved\n///               in the middle of iteration.\n/// \\return a ready future if we stopped successfully, or a failed future if\n///         a call to to \\c action failed.  The \\c optional's value is returned.\ntemplate<typename AsyncAction>\nrequires requires (AsyncAction aa) {\n    bool(futurize_invoke(aa).get());\n    futurize_invoke(aa).get().value();\n}\nrepeat_until_value_return_type<AsyncAction>\nrepeat_until_value(AsyncAction action) noexcept {\n    using futurator = futurize<std::invoke_result_t<AsyncAction>>;\n    using type_helper = repeat_until_value_type_helper<typename futurator::type>;\n    // the \"T\" in the documentation\n    using value_type = typename type_helper::value_type;\n    using optional_type = typename type_helper::optional_type;\n    do {\n        auto f = futurator::invoke(action);\n\n        if (!f.available()) {\n          return [&] () noexcept {\n            memory::scoped_critical_alloc_section _;\n            auto state = new internal::repeat_until_value_state<AsyncAction, value_type>(std::move(action));\n            auto ret = state->get_future();\n            internal::set_callback(std::move(f), state);\n            return ret;\n          }();\n        }\n\n        if (f.failed()) {\n            return make_exception_future<value_type>(f.get_exception());\n        }\n\n        optional_type&& optional = std::move(f).get();\n        if (optional) {\n            return make_ready_future<value_type>(std::move(optional.value()));\n        }\n    } while (!need_preempt());\n\n    try {\n        auto state = new internal::repeat_until_value_state<AsyncAction, value_type>(std::nullopt, std::move(action));\n        auto f = state->get_future();\n        schedule(state);\n        return f;\n    } catch (...) {\n        return make_exception_future<value_type>(std::current_exception());\n    }\n}\n\nnamespace internal {\n\ntemplate <typename StopCondition, typename AsyncAction>\nclass do_until_state final : public continuation_base<> {\n    promise<> _promise;\n    StopCondition _stop;\n    AsyncAction _action;\npublic:\n    explicit do_until_state(StopCondition stop, AsyncAction action) : _stop(std::move(stop)), _action(std::move(action)) {}\n    future<> get_future() { return _promise.get_future(); }\n    task* waiting_task() noexcept override { return _promise.waiting_task(); }\n    virtual void run_and_dispose() noexcept override {\n        if (_state.available()) {\n            if (_state.failed()) {\n                _promise.set_urgent_state(std::move(_state));\n                delete this;\n                return;\n            }\n            _state = {}; // allow next cycle to overrun state\n        }\n        try {\n            do {\n                if (_stop()) {\n                    _promise.set_value();\n                    delete this;\n                    return;\n                }\n                auto f = _action();\n                if (!f.available()) {\n                    internal::set_callback(std::move(f), this);\n                    return;\n                }\n                if (f.failed()) {\n                    f.forward_to(std::move(_promise));\n                    delete this;\n                    return;\n                }\n            } while (!need_preempt());\n        } catch (...) {\n            _promise.set_exception(std::current_exception());\n            delete this;\n            return;\n        }\n        schedule(this);\n    }\n};\n\n} // namespace internal\n\n/// Invokes given action until it fails or given condition evaluates to true or fails.\n///\n/// \\param stop_cond a callable taking no arguments, returning a boolean that\n///                  evalutes to true when you don't want to call \\c action\n///                  any longer. If \\c stop_cond fails, the exception is propagated\n//                   in the returned future.\n/// \\param action a callable taking no arguments, returning a future<>.  Will\n///               be called again as soon as the future resolves, unless the\n///               future fails, or \\c stop_cond returns \\c true or fails.\n/// \\return a ready future if we stopped successfully, or a failed future if\n///         a call to to \\c action or a call to \\c stop_cond failed.\ntemplate<typename AsyncAction, typename StopCondition>\nrequires std::is_invocable_r_v<bool, StopCondition> && std::is_invocable_r_v<future<>, AsyncAction>\ninline\nfuture<> do_until(StopCondition stop_cond, AsyncAction action) noexcept {\n    using namespace internal;\n    for (;;) {\n      try {\n        if (stop_cond()) {\n            return make_ready_future<>();\n        }\n      } catch (...) {\n        return current_exception_as_future();\n      }\n        auto f = futurize_invoke(action);\n        if (f.failed()) {\n            return f;\n        }\n        if (!f.available() || need_preempt()) {\n            return [&] () noexcept {\n                memory::scoped_critical_alloc_section _;\n                auto task = new do_until_state<StopCondition, AsyncAction>(std::move(stop_cond), std::move(action));\n                auto ret = task->get_future();\n                internal::set_callback(std::move(f), task);\n                return ret;\n            }();\n        }\n    }\n}\n\n/// Invoke given action until it fails.\n///\n/// Calls \\c action repeatedly until it returns a failed future.\n///\n/// \\param action a callable taking no arguments, returning a \\c future<>\n///        that becomes ready when you wish it to be called again.\n/// \\return a future<> that will resolve to the first failure of \\c action\ntemplate<typename AsyncAction>\nrequires std::is_invocable_r_v<future<>, AsyncAction>\ninline\nfuture<> keep_doing(AsyncAction action) noexcept {\n    return repeat([action = std::move(action)] () mutable {\n        return action().then([] {\n            return stop_iteration::no;\n        });\n    });\n}\n\nnamespace internal {\ntemplate <typename Iterator, class Sentinel, typename AsyncAction>\nclass do_for_each_state final : public continuation_base<> {\n    Iterator _begin;\n    Sentinel _end;\n    AsyncAction _action;\n    promise<> _pr;\n\npublic:\n    do_for_each_state(Iterator begin, Sentinel end, AsyncAction action, future<>&& first_unavailable)\n        : _begin(std::move(begin)), _end(std::move(end)), _action(std::move(action)) {\n        internal::set_callback(std::move(first_unavailable), this);\n    }\n    virtual void run_and_dispose() noexcept override {\n        std::unique_ptr<do_for_each_state> zis(this);\n        if (_state.failed()) {\n            _pr.set_urgent_state(std::move(_state));\n            return;\n        }\n        while (_begin != _end) {\n            auto f = futurize_invoke(_action, *_begin++);\n            if (f.failed()) {\n                f.forward_to(std::move(_pr));\n                return;\n            }\n            if (!f.available() || need_preempt()) {\n                _state = {};\n                internal::set_callback(std::move(f), this);\n                zis.release();\n                return;\n            }\n        }\n        _pr.set_value();\n    }\n    task* waiting_task() noexcept override {\n        return _pr.waiting_task();\n    }\n    future<> get_future() {\n        return _pr.get_future();\n    }\n};\n\ntemplate<typename Iterator, typename Sentinel, typename AsyncAction>\ninline\nfuture<> do_for_each_impl(Iterator begin, Sentinel end, AsyncAction action) {\n    while (begin != end) {\n        auto f = futurize_invoke(action, *begin++);\n        if (f.failed()) {\n            return f;\n        }\n        if (!f.available() || need_preempt()) {\n            auto* s = new internal::do_for_each_state<Iterator, Sentinel, AsyncAction>{\n                std::move(begin), std::move(end), std::move(action), std::move(f)};\n            return s->get_future();\n        }\n    }\n    return make_ready_future<>();\n}\n} // namespace internal\n\n/// \\addtogroup future-util\n\n/// \\brief Call a function for each item in a range, sequentially (iterator version).\n///\n/// For each item in a range, call a function, waiting for the previous\n/// invocation to complete before calling the next one.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range\n/// \\param end an \\c InputIterator designating the endof the range\n/// \\param action a callable, taking a reference to objects from the range\n///               as a parameter, and returning a \\c future<> that resolves\n///               when it is acceptable to process the next item.\n/// \\return a ready future on success, or the first failed future if\n///         \\c action failed.\ntemplate<typename Iterator, typename Sentinel, typename AsyncAction>\nrequires (\n    requires (Iterator i, AsyncAction aa) {\n        { futurize_invoke(aa, *i) } -> std::same_as<future<>>;\n    } &&\n    (std::same_as<Sentinel, Iterator> || std::sentinel_for<Sentinel, Iterator>)\n)\ninline\nfuture<> do_for_each(Iterator begin, Sentinel end, AsyncAction action) noexcept {\n    try {\n        return internal::do_for_each_impl(std::move(begin), std::move(end), std::move(action));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\n/// \\brief Call a function for each item in a range, sequentially (range version).\n///\n/// For each item in a range, call a function, waiting for the previous\n/// invocation to complete before calling the next one.\n///\n/// \\param c an \\c Range object designating input range\n/// \\param action a callable, taking a reference to objects from the range\n///               as a parameter, and returning a \\c future<> that resolves\n///               when it is acceptable to process the next item.\n/// \\return a ready future on success, or the first failed future if\n///         \\c action failed.\ntemplate<typename Range, typename AsyncAction>\nrequires requires (Range c, AsyncAction aa) {\n    { futurize_invoke(aa, *std::begin(c)) } -> std::same_as<future<>>;\n    std::end(c);\n}\ninline\nfuture<> do_for_each(Range& c, AsyncAction action) noexcept {\n    try {\n        return internal::do_for_each_impl(std::begin(c), std::end(c), std::move(action));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\nnamespace internal {\n\ntemplate <typename T, typename = void>\nstruct has_iterator_category : std::false_type {};\n\ntemplate <typename T>\nstruct has_iterator_category<T, std::void_t<typename std::iterator_traits<T>::iterator_category >> : std::true_type {};\n\ntemplate <typename Iterator, typename Sentinel>\ninline\nsize_t\niterator_range_estimate_vector_capacity(Iterator begin, Sentinel end) {\n    if constexpr (std::forward_iterator<Iterator> &&\n                  std::forward_iterator<Sentinel>) {\n        return std::ranges::distance(begin, end);\n    } else if constexpr (std::random_access_iterator<Iterator> &&\n                         std::random_access_iterator<Sentinel>) {\n        return std::ranges::distance(begin, end);\n    } else {\n        // For InputIterators we can't estimate needed capacity\n        return 0;\n    }\n}\n\n} // namespace internal\n\n/// \\cond internal\n\nclass parallel_for_each_state final : private continuation_base<> {\n    std::vector<future<>> _incomplete;\n    promise<> _result;\n    std::exception_ptr _ex;\nprivate:\n    // Wait for one of the futures in _incomplete to complete, and then\n    // decide what to do: wait for another one, or deliver _result if all\n    // are complete.\n    void wait_for_one() noexcept;\n    virtual void run_and_dispose() noexcept override;\n    task* waiting_task() noexcept override { return _result.waiting_task(); }\npublic:\n    parallel_for_each_state(size_t n);\n    void add_future(future<>&& f);\n    future<> get_future();\n};\n\n/// \\endcond\n\n/// \\brief Run tasks in parallel (iterator version).\n///\n/// Given a range [\\c begin, \\c end) of objects, run \\c func on each \\c *i in\n/// the range, and return a future<> that resolves when all the functions\n/// complete.  \\c func should return a future<> that indicates when it is\n/// complete.  All invocations are performed in parallel. This allows the range\n/// to refer to stack objects, but means that unlike other loops this cannot\n/// check need_preempt and can only be used with small ranges.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range\n/// \\param end an \\c InputIterator designating the end of the range\n/// \\param func Function to invoke with each element in the range (returning\n///             a \\c future<>)\n/// \\return a \\c future<> that resolves when all the function invocations\n///         complete.  If one or more return an exception, the return value\n///         contains one of the exceptions.\n/// \\note parallel_for_each() schedules all invocations of \\c func on the\n///       current shard. If you want to run a function on all shards in parallel,\n///       have a look at \\ref smp::invoke_on_all() instead.\ntemplate <typename Iterator, typename Sentinel, typename Func>\nrequires (requires (Func f, Iterator i) { { f(*i) } -> std::same_as<future<>>; { i++ }; } && (std::same_as<Sentinel, Iterator> || std::sentinel_for<Sentinel, Iterator>))\n// We use a conjunction with std::same_as<Sentinel, Iterator> because std::sentinel_for requires Sentinel to be semiregular,\n// which implies that it requires Sentinel to be default-constructible, which is unnecessarily strict in below's context and could\n// break legacy code, for which it holds that Sentinel equals Iterator.\ninline\nfuture<>\nparallel_for_each(Iterator begin, Sentinel end, Func&& func) noexcept {\n    parallel_for_each_state* s = nullptr;\n    // Process all elements, giving each future the following treatment:\n    //   - available, not failed: do nothing\n    //   - available, failed: collect exception in ex\n    //   - not available: collect in s (allocating it if needed)\n    while (begin != end) {\n        auto f = futurize_invoke(std::forward<Func>(func), *begin);\n        ++begin;\n        memory::scoped_critical_alloc_section _;\n        if (!f.available() || f.failed()) {\n            if (!s) {\n                size_t n{0U};\n                if constexpr (internal::has_iterator_category<Iterator>::value) {\n                    // We need if-constexpr here because there exist iterators for which std::iterator_traits\n                    // does not have 'iterator_category' as member type\n                    n = (internal::iterator_range_estimate_vector_capacity(begin, end) + 1);\n                }\n                s = new parallel_for_each_state(n);\n            }\n            s->add_future(std::move(f));\n        }\n    }\n    // If any futures were not available, hand off to parallel_for_each_state::start().\n    // Otherwise we can return a result immediately.\n    if (s) {\n        // s->get_future() takes ownership of s (and chains it to one of the futures it contains)\n        // so this isn't a leak\n        return s->get_future();\n    }\n    return make_ready_future<>();\n}\n\n/// \\brief Run tasks in parallel (range version).\n///\n/// Given a \\c range of objects, invoke \\c func with each object\n/// in the range, and return a future<> that resolves when all\n/// the functions complete.  \\c func should return a future<> that indicates\n/// when it is complete.  All invocations are performed in parallel. This allows\n/// the range to refer to stack objects, but means that unlike other loops this\n/// cannot check need_preempt and can only be used with small ranges.\n///\n/// \\param range A range of objects to iterate run \\c func on\n/// \\param func  A callable, accepting reference to the range's\n///              \\c value_type, and returning a \\c future<>.\n/// \\return a \\c future<> that becomes ready when the entire range\n///         was processed.  If one or more of the invocations of\n///         \\c func returned an exceptional future, then the return\n///         value will contain one of those exceptions.\n/// \\note parallel_for_each() schedules all invocations of \\c func on the\n///       current shard. If you want to run a function on all shards in parallel,\n///       have a look at \\ref smp::invoke_on_all() instead.\n\nnamespace internal {\n\ntemplate <typename Range, typename Func>\ninline\nfuture<>\nparallel_for_each_impl(Range&& range, Func&& func) {\n    return parallel_for_each(std::begin(range), std::end(range),\n            std::forward<Func>(func));\n}\n\n} // namespace internal\n\ntemplate <typename Range, typename Func>\nrequires requires (Func f, Range r) {\n    { f(*std::begin(r)) } -> std::same_as<future<>>;\n    std::end(r);\n}\ninline\nfuture<>\nparallel_for_each(Range&& range, Func&& func) noexcept {\n    auto impl = internal::parallel_for_each_impl<Range, Func>;\n    return futurize_invoke(impl, std::forward<Range>(range), std::forward<Func>(func));\n}\n\n/// Run a maximum of \\c max_concurrent tasks in parallel (iterator version).\n///\n/// Given a range [\\c begin, \\c end) of objects, run \\c func on each \\c *i in\n/// the range, and return a future<> that resolves when all the functions\n/// complete.  \\c func should return a future<> that indicates when it is\n/// complete.  Up to \\c max_concurrent invocations are performed in parallel.\n/// This does not allow the range to refer to stack objects. The caller\n/// must ensure that the range outlives the call to max_concurrent_for_each\n/// so it can be iterated in the background.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range\n/// \\param end an \\c InputIterator designating the end of the range\n/// \\param max_concurrent maximum number of concurrent invocations of \\c func, must be greater than zero.\n/// \\param func Function to invoke with each element in the range (returning\n///             a \\c future<>)\n/// \\return a \\c future<> that resolves when all the function invocations\n///         complete.  If one or more return an exception, the return value\n///         contains one of the exceptions.\n/// \\note max_concurrent_for_each() schedules all invocations of \\c func on the\n///       current shard. If you want to run a function on all shards in parallel,\n///       have a look at \\ref smp::invoke_on_all() instead.\ntemplate <typename Iterator, typename Sentinel, typename Func>\nrequires (requires (Func f, Iterator i) { { f(*i) } -> std::same_as<future<>>; { ++i }; } && (std::same_as<Sentinel, Iterator> || std::sentinel_for<Sentinel, Iterator>) )\n// We use a conjunction with std::same_as<Sentinel, Iterator> because std::sentinel_for requires Sentinel to be semiregular,\n// which implies that it requires Sentinel to be default-constructible, which is unnecessarily strict in below's context and could\n// break legacy code, for which it holds that Sentinel equals Iterator.\ninline\nfuture<>\nmax_concurrent_for_each(Iterator begin, Sentinel end, size_t max_concurrent, Func&& func) noexcept {\n    struct state {\n        Iterator begin;\n        Sentinel end;\n        Func func;\n        size_t max_concurrent;\n        semaphore sem;\n        std::exception_ptr err;\n\n        state(Iterator begin_, Sentinel end_, size_t max_concurrent_, Func func_)\n            : begin(std::move(begin_))\n            , end(std::move(end_))\n            , func(std::move(func_))\n            , max_concurrent(max_concurrent_)\n            , sem(max_concurrent_)\n            , err()\n        { }\n    };\n\n    SEASTAR_ASSERT(max_concurrent > 0);\n\n    try {\n        return do_with(state(std::move(begin), std::move(end), max_concurrent, std::forward<Func>(func)), [] (state& s) {\n            return do_until([&s] { return s.begin == s.end; }, [&s] {\n                return s.sem.wait().then([&s] () mutable noexcept {\n                    // Possibly run in background and signal _sem when the task is done.\n                    // The background tasks are waited on using _sem.\n                    (void)futurize_invoke(s.func, *s.begin).then_wrapped([&s] (future<> fut) {\n                        if (fut.failed()) {\n                            auto e = fut.get_exception();;\n                            if (!s.err) {\n                                s.err = std::move(e);\n                            }\n                        }\n                        s.sem.signal();\n                    });\n                    ++s.begin;\n                });\n            }).then([&s] {\n                // Wait for any background task to finish\n                // and signal and semaphore\n                return s.sem.wait(s.max_concurrent);\n            }).then([&s] {\n                if (!s.err) {\n                    return make_ready_future<>();\n                }\n                return seastar::make_exception_future<>(std::move(s.err));\n            });\n        });\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\n/// Run a maximum of \\c max_concurrent tasks in parallel (range version).\n///\n/// Given a range of objects, run \\c func on each \\c *i in\n/// the range, and return a future<> that resolves when all the functions\n/// complete.  \\c func should return a future<> that indicates when it is\n/// complete.  Up to \\c max_concurrent invocations are performed in parallel.\n/// This does not allow the range to refer to stack objects. The caller\n/// must ensure that the range outlives the call to max_concurrent_for_each\n/// so it can be iterated in the background.\n///\n/// \\param range a \\c Range to be processed\n/// \\param max_concurrent maximum number of concurrent invocations of \\c func, must be greater than zero.\n/// \\param func Function to invoke with each element in the range (returning\n///             a \\c future<>)\n/// \\return a \\c future<> that resolves when all the function invocations\n///         complete.  If one or more return an exception, the return value\n///         contains one of the exceptions.\n/// \\note max_concurrent_for_each() schedules all invocations of \\c func on the\n///       current shard. If you want to run a function on all shards in parallel,\n///       have a look at \\ref smp::invoke_on_all() instead.\ntemplate <typename Range, typename Func>\nrequires requires (Func f, Range r) {\n    { f(*std::begin(r)) } -> std::same_as<future<>>;\n    std::end(r);\n}\ninline\nfuture<>\nmax_concurrent_for_each(Range&& range, size_t max_concurrent, Func&& func) noexcept {\n    try {\n        return max_concurrent_for_each(std::begin(range), std::end(range), max_concurrent, std::forward<Func>(func));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\n/// @}\n\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/lowres_clock.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/cacheline.hh>\n#include <seastar/core/timer.hh>\n\n#include <chrono>\n\nnamespace seastar {\n\n\n//\n// Forward declarations.\n//\n\nclass lowres_clock;\nclass lowres_system_clock;\n\n/// \\endcond\n\n//\n/// \\brief Low-resolution and efficient steady clock.\n///\n/// This is a monotonic clock with a granularity of ~task_quota. Time points from this clock do not correspond to system\n/// time.\n///\n/// The primary benefit of this clock is that invoking \\c now() is inexpensive compared to\n/// \\c std::chrono::steady_clock::now().\n///\n/// \\see \\c lowres_system_clock for a low-resolution clock which produces time points corresponding to system time.\n///\nclass lowres_clock final {\npublic:\n    using rep = std::chrono::steady_clock::rep;\n    using period = std::chrono::steady_clock::period;\n    using duration = std::chrono::steady_clock::duration;\n    using time_point = std::chrono::time_point<lowres_clock, duration>;\n    static constexpr bool is_steady = true;\nprivate:\n#ifdef SEASTAR_BUILD_SHARED_LIBS\n    static thread_local time_point _now;\n#else\n    // Use inline variable to prevent the compiler from introducing an initialization guard\n    inline static thread_local time_point _now;\n#endif\npublic:\n    ///\n    /// \\note Outside of a Seastar application, the result is undefined.\n    ///\n    static time_point now() noexcept {\n        return _now;\n    }\n\n    static void update() noexcept;\n};\n\n///\n/// \\brief Low-resolution and efficient system clock.\n///\n/// This clock has the same granularity as \\c lowres_clock, but it is not required to be monotonic and its time points\n/// correspond to system time.\n///\n/// The primary benefit of this clock is that invoking \\c now() is inexpensive compared to\n/// \\c std::chrono::system_clock::now().\n///\nclass lowres_system_clock final {\npublic:\n    using rep = std::chrono::system_clock::rep;\n    using period = std::chrono::system_clock::period;\n    using duration = std::chrono::system_clock::duration;\n    using time_point = std::chrono::time_point<lowres_system_clock, duration>;\n    static constexpr bool is_steady = false;\nprivate:\n#ifdef SEASTAR_BUILD_SHARED_LIBS\n    static thread_local time_point _now;\n#else\n    // Use inline variable to prevent the compiler from introducing an initialization guard\n    inline static thread_local time_point _now;\n#endif\n    friend class lowres_clock; // for updates\npublic:\n    ///\n    /// \\note Outside of a Seastar application, the result is undefined.\n    ///\n    static time_point now() noexcept {\n        return _now;\n    }\n\n    static std::time_t to_time_t(time_point t) noexcept {\n        return std::chrono::duration_cast<std::chrono::seconds>(t.time_since_epoch()).count();\n    }\n\n    static time_point from_time_t(std::time_t t) noexcept {\n        return time_point(std::chrono::duration_cast<duration>(std::chrono::seconds(t)));\n    }\n};\n\nextern template class timer<lowres_clock>;\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/make_task.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/task.hh>\n#include <seastar/core/future.hh>\n\nnamespace seastar {\n\n\ntemplate <typename Func>\nrequires std::invocable<Func>\nclass lambda_task final : public task {\n    Func _func;\n    using futurator = futurize<std::invoke_result_t<Func>>;\n    typename futurator::promise_type _result;\npublic:\n    lambda_task(scheduling_group sg, const Func& func) : task(sg), _func(func) {}\n    lambda_task(scheduling_group sg, Func&& func) : task(sg), _func(std::move(func)) {}\n    typename futurator::type get_future() noexcept { return _result.get_future(); }\n    virtual void run_and_dispose() noexcept override {\n        futurator::invoke(_func).forward_to(std::move(_result));\n        delete this;\n    }\n    virtual task* waiting_task() noexcept override {\n        return _result.waiting_task();\n    }\n};\n\ntemplate <typename Func>\nrequires std::invocable<Func>\ninline\nlambda_task<Func>*\nmake_task(Func&& func) noexcept {\n    return new lambda_task<Func>(current_scheduling_group(), std::forward<Func>(func));\n}\n\ntemplate <typename Func>\nrequires std::invocable<Func>\ninline\nlambda_task<Func>*\nmake_task(scheduling_group sg, Func&& func) noexcept {\n    return new lambda_task<Func>(sg, std::forward<Func>(func));\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/manual_clock.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/timer.hh>\n\n#include <atomic>\n#include <chrono>\n\nnamespace seastar {\n\nclass manual_clock {\npublic:\n    using rep = int64_t;\n    using period = std::chrono::nanoseconds::period;\n    using duration = std::chrono::duration<rep, period>;\n    using time_point = std::chrono::time_point<manual_clock, duration>;\nprivate:\n    static std::atomic<rep> _now;\n    static void expire_timers() noexcept;\npublic:\n    manual_clock() noexcept;\n    static time_point now() noexcept {\n        return time_point(duration(_now.load(std::memory_order_relaxed)));\n    }\n    static void advance(duration d) noexcept;\n};\n\nextern template class timer<manual_clock>;\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/map_reduce.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#pragma once\n\n#include <iterator>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/shared_ptr.hh>\n\nnamespace seastar {\n\n\n/// \\addtogroup future-util\n/// @{\n\n/// \\cond internal\n\ntemplate <typename T, typename Ptr, bool IsFuture>\nstruct reducer_with_get_traits;\n\ntemplate <typename T, typename Ptr>\nstruct reducer_with_get_traits<T, Ptr, false> {\n    using result_type = decltype(std::declval<T>().get());\n    using future_type = future<result_type>;\n    static future_type maybe_call_get(future<> f, Ptr r) {\n        return f.then([r = std::move(r)] () mutable {\n            return make_ready_future<result_type>(std::move(r->reducer).get());\n        });\n    }\n};\n\ntemplate <typename T, typename Ptr>\nstruct reducer_with_get_traits<T, Ptr, true> {\n    using future_type = decltype(std::declval<T>().get());\n    static future_type maybe_call_get(future<> f, Ptr r) {\n        return f.then([r = r.get()] {\n            return r->reducer.get();\n        }).then_wrapped([r] (future_type f) {\n            return f;\n        });\n    }\n};\n\ntemplate <typename T, typename Ptr = lw_shared_ptr<T>, typename V = void>\nstruct reducer_traits {\n    using future_type = future<>;\n    static future_type maybe_call_get(future<> f, Ptr r) {\n        return f.then([r = std::move(r)] {});\n    }\n};\n\ntemplate <typename T, typename Ptr>\nstruct reducer_traits<T, Ptr, decltype(std::declval<T>().get(), void())> : public reducer_with_get_traits<T, Ptr, is_future<std::invoke_result_t<decltype(&T::get),T>>::value> {};\n\n/// \\endcond\n\n/// Map a function over a range and reduce the result.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range\n/// \\param end an \\c InputIterator designating the end of the range\n/// \\param mapper is a callable which transforms values from the iterator range into a future<T>\n/// \\param r is an object which can be called with T as parameter and yields a future<>\n///     It may have a get() method which returns a value of type U which holds the result of reduction.\n/// \\return Th reduced value wrapped in a future.\n///     If the reducer has no get() method then this function returns future<>.\n/// \\note map-reduce() schedules all invocations of both \\c mapper and \\c reduce\n///       on the current shard. If you want to run a function on all shards in\n///       parallel, have a look at \\ref smp::invoke_on_all() instead, or combine\n///       map_reduce() with \\ref smp::submit_to().\n///       Sharded services have their own \\ref sharded::map_reduce() which\n///       map-reduces across all shards.\n\n// TODO: specialize for non-deferring reducer\ntemplate <typename Iterator, typename Mapper, typename Reducer>\nrequires requires (Iterator i, Mapper mapper, Reducer reduce) {\n     *i++;\n     { i != i } -> std::convertible_to<bool>;\n     mapper(*i);\n     reduce(futurize_invoke(mapper, *i).get());\n}\ninline\nauto\nmap_reduce(Iterator begin, Iterator end, Mapper&& mapper, Reducer&& r)\n    -> typename reducer_traits<Reducer>::future_type\n{\n    struct state {\n        Mapper mapper;\n        Reducer reducer;\n    };\n    auto s = make_lw_shared(state{std::forward<Mapper>(mapper), std::forward<Reducer>(r)});\n    future<> ret = make_ready_future<>();\n    while (begin != end) {\n        ret = futurize_invoke(s->mapper, *begin++).then_wrapped([ret = std::move(ret), s] (auto f) mutable {\n            return ret.then_wrapped([f = std::move(f), s] (auto rf) mutable {\n                if (rf.failed()) {\n                    f.ignore_ready_future();\n                    return rf;\n                } else {\n                    return futurize_invoke(s->reducer, std::move(f.get()));\n                }\n            });\n        });\n    }\n    return reducer_traits<Reducer, lw_shared_ptr<state>>::maybe_call_get(std::move(ret), s);\n}\n\n/// Asynchronous map/reduce transformation.\n///\n/// Given a range of objects, an asynchronous unary function\n/// operating on these objects, an initial value, and a\n/// binary function for reducing, map_reduce() will\n/// transform each object in the range, then invoke\n/// the the reducing function with the result.\n///\n/// Example:\n///\n/// Calculate the total size of several files:\n///\n/// \\code\n///  map_reduce(files.begin(), files.end(),\n///             std::mem_fn(file::size),\n///             size_t(0),\n///             std::plus<size_t>())\n/// \\endcode\n///\n/// Requirements:\n///    - Iterator: an InputIterator.\n///    - Mapper: unary function taking Iterator::value_type and producing a future<...>.\n///    - Initial: any value type\n///    - Reduce: a binary function taking two Initial values and returning an Initial\n///\n/// Return type:\n///    - future<Initial>\n///\n/// \\param begin beginning of object range to operate on\n/// \\param end end of object range to operate on\n/// \\param mapper map function to call on each object, returning a future\n/// \\param initial initial input value to reduce function\n/// \\param reduce binary function for merging two result values from \\c mapper\n///\n/// \\return equivalent to \\c reduce(reduce(initial, mapper(obj0)), mapper(obj1)) ...\n///\n/// \\note map-reduce() schedules all invocations of both \\c mapper and \\c reduce\n///       on the current shard. If you want to run a function on all shards in\n///       parallel, have a look at \\ref smp::invoke_on_all() instead, or combine\n///       map_reduce() with \\ref smp::submit_to().\n///       Sharded services have their own \\ref sharded::map_reduce() which\n///       map-reduces across all shards.\ntemplate <typename Iterator, typename Mapper, typename Initial, typename Reduce>\nrequires requires (Iterator i, Mapper mapper, Initial initial, Reduce reduce) {\n     *i++;\n     { i != i} -> std::convertible_to<bool>;\n     mapper(*i);\n     requires is_future<decltype(mapper(*i))>::value;\n     { reduce(std::move(initial), mapper(*i).get()) } -> std::convertible_to<Initial>;\n}\ninline\nfuture<Initial>\nmap_reduce(Iterator begin, Iterator end, Mapper&& mapper, Initial initial, Reduce reduce) {\n    struct state {\n        Mapper mapper;\n        Initial result;\n        Reduce reduce;\n        std::exception_ptr ex;\n    };\n    auto s = make_lw_shared(state{std::forward<Mapper>(mapper), std::move(initial), std::move(reduce), std::exception_ptr()});\n    future<> ret = make_ready_future<>();\n    while (begin != end) {\n        ret = futurize_invoke(s->mapper, *begin++).then_wrapped([s = s.get(), ret = std::move(ret)] (auto f) mutable {\n            if (s->ex) {\n                f.ignore_ready_future();  // We only report the first exception\n            } else if (f.failed()) {\n                s->ex = f.get_exception();\n            } else {\n                try {\n                    s->result = s->reduce(std::move(s->result), f.get());\n                } catch (...) {\n                    s->ex = std::current_exception();\n                }\n            }\n            return std::move(ret);\n        });\n    }\n    return ret.then([s] {\n        if (s->ex) {\n            return make_exception_future<Initial>(std::move(s->ex));\n        }\n        return make_ready_future<Initial>(std::move(s->result));\n    });\n}\n\n/// Asynchronous map/reduce transformation (range version).\n///\n/// Given a range of objects, an asynchronous unary function\n/// operating on these objects, an initial value, and a\n/// binary function for reducing, map_reduce() will\n/// transform each object in the range, then invoke\n/// the the reducing function with the result.\n///\n/// Example:\n///\n/// Calculate the total size of several files:\n///\n/// \\code\n///  std::vector<file> files = ...;\n///  map_reduce(files,\n///             std::mem_fn(file::size),\n///             size_t(0),\n///             std::plus<size_t>())\n/// \\endcode\n///\n/// Requirements:\n///    - Iterator: an InputIterator.\n///    - Mapper: unary function taking Iterator::value_type and producing a future<...>.\n///    - Initial: any value type\n///    - Reduce: a binary function taking two Initial values and returning an Initial\n///\n/// Return type:\n///    - future<Initial>\n///\n/// \\param range object range to operate on\n/// \\param mapper map function to call on each object, returning a future\n/// \\param initial initial input value to reduce function\n/// \\param reduce binary function for merging two result values from \\c mapper\n///\n/// \\return equivalent to \\c reduce(reduce(initial, mapper(obj0)), mapper(obj1)) ...\n///\n/// \\note map-reduce() schedules all invocations of both \\c mapper and \\c reduce\n///       on the current shard. If you want to run a function on all shards in\n///       parallel, have a look at \\ref smp::invoke_on_all() instead, or combine\n///       map_reduce() with \\ref smp::submit_to().\n///       Sharded services have their own \\ref sharded::map_reduce() which\n///       map-reduces across all shards.\ntemplate <typename Range, typename Mapper, typename Initial, typename Reduce>\nrequires requires (Range range, Mapper mapper, Initial initial, Reduce reduce) {\n     std::begin(range);\n     std::end(range);\n     mapper(*std::begin(range));\n     requires is_future<std::remove_reference_t<decltype(mapper(*std::begin(range)))>>::value;\n     { reduce(std::move(initial), mapper(*std::begin(range)).get()) } -> std::convertible_to<Initial>;\n}\ninline\nfuture<Initial>\nmap_reduce(Range&& range, Mapper&& mapper, Initial initial, Reduce reduce) {\n    return map_reduce(std::begin(range), std::end(range), std::forward<Mapper>(mapper),\n            std::move(initial), std::move(reduce));\n}\n\n/// Implements @Reducer concept. Calculates the result by\n/// adding elements to the accumulator.\ntemplate <typename Result, typename Addend = Result>\nclass adder {\nprivate:\n    Result _result;\npublic:\n    future<> operator()(const Addend& value) {\n        _result += value;\n        return make_ready_future<>();\n    }\n    Result get() && {\n        return std::move(_result);\n    }\n};\n\n/// @}\n\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/memory.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/resource.hh>\n#include <seastar/core/bitops.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/sampler.hh>\n#include <new>\n#include <cstdint>\n#include <functional>\n#include <optional>\n#include <string>\n#include <vector>\n\nnamespace seastar {\n\n/// \\defgroup memory-module Memory management\n///\n/// Functions and classes for managing memory.\n///\n/// Memory management in seastar consists of the following:\n///\n///   - Low-level memory management in the \\ref memory namespace.\n///   - Various smart pointers: \\ref shared_ptr, \\ref lw_shared_ptr,\n///     and \\ref foreign_ptr.\n///   - zero-copy support: \\ref temporary_buffer and \\ref deleter.\n\n/// Low-level memory management support\n///\n/// The \\c memory namespace provides functions and classes for interfacing\n/// with the seastar memory allocator.\n///\n/// The seastar memory allocator splits system memory into a pool per\n/// logical core (lcore).  Memory allocated one an lcore should be freed\n/// on the same lcore; failing to do so carries a severe performance\n/// penalty.  It is possible to share memory with another core, but this\n/// should be limited to avoid cache coherency traffic.\n/// You can obtain the memory layout of the current shard with\n/// \\ref get_memory_layout().\n///\n/// ## Critical allocation scopes\n///\n/// Seastar supports marking scopes as critical allocation scopes for the purpose\n/// of special treatment from various memory related utilities.\n/// See \\ref scoped_critical_alloc_section.\n///\n/// ## Diagnostics and debugging features\n///\n/// ### Allocation failure injector\n///\n/// Allows injecting allocation failures for testing resiliency against\n/// allocation failures, or exceptions in general. See:\n/// * \\ref alloc_failure_injector\n/// * \\ref with_allocation_failures()\n///\n/// ### Large allocation warning\n///\n/// Large allocations put great pressure on the allocator which might be unable\n/// to serve them even if there is enough memory available, due to memory\n/// fragmentation. This is especially relevant for long-running applications,\n/// the kind of applications that are typically built with seastar. This feature\n/// allows finding these large by logging a warning on large allocations, with\n/// the stacktrace of the. See:\n/// * \\ref set_large_allocation_warning_threshold()\n/// * \\ref get_large_allocation_warning_threshold()\n/// * \\ref scoped_large_allocation_warning_threshold\n/// * \\ref scoped_large_allocation_warning_disable\n///\n/// ### Heap profiling\n///\n/// Heap profiling allows finding out how memory is used by your application, by\n/// recording the stacktrace of a sampled subset (or all) allocations. See:\n/// * \\ref set_heap_profiling_sampling_rate()\n/// * \\ref sampled_memory_profile()\n/// * \\ref scoped_heap_profiling\n///\n/// ### Abort on allocation failure\n///\n/// Often, the best way to debug an allocation failure is a coredump. This\n/// feature allows dumping core on allocation failures, containing the stack of\n/// the failed allocation, by means of aborting. To enable this behavior, set\n/// `abort_on_seastar_bad_alloc` in `reactor_options` or pass the\n/// `--abort-on-seastar-bad-alloc` command line flag. Additionally, applications\n/// may enable or disable this functionality globally at runtime by calling\n/// `set_abort_on_allocation_failure()`.\n///\n/// ### Dump diagnostics report\n///\n/// Dump a diagnostic report of the state of the seastar allocator upon allocation\n/// failure. The report is dumped with the `seastar_memory` logger, with debug\n/// level.\n/// You can configure a report to be dumped with error level on certain allocation\n/// kinds, see:\n/// * set_dump_memory_diagnostics_on_alloc_failure_kind()\n/// * set_additional_diagnostics_producer()\n/// * generate_memory_diagnostics_report()\n///\n/// The diagnostics report dump can be configured with the command\n/// line/configuration file via the \\p dump-memory-diagnostics-on-alloc-failure-kind\n/// command-line flag/configuration item.\nnamespace memory {\n\n/// \\cond internal\n\n#ifdef SEASTAR_OVERRIDE_ALLOCATOR_PAGE_SIZE\n#define SEASTAR_INTERNAL_ALLOCATOR_PAGE_SIZE (SEASTAR_OVERRIDE_ALLOCATOR_PAGE_SIZE)\n#else\n#define SEASTAR_INTERNAL_ALLOCATOR_PAGE_SIZE 4096\n#endif\n\nconstexpr inline size_t page_size = SEASTAR_INTERNAL_ALLOCATOR_PAGE_SIZE;\nconstexpr inline size_t page_bits = log2ceil(page_size);\nconstexpr inline size_t huge_page_size =\n#if defined(__x86_64__) || defined(__i386__) || defined(__s390x__) || defined(__zarch__)\n    1 << 21; // 2M\n#elif defined(__aarch64__)\n    1 << 21; // 2M\n#elif defined(__PPC__)\n    1 << 24; // 16M\n#else\n#error \"Huge page size is not defined for this architecture\"\n#endif\n\n\nnamespace internal {\n\nstruct memory_range {\n    char* start;\n    char* end;\n    unsigned numa_node_id;\n};\n\nstruct numa_layout {\n    std::vector<memory_range> ranges;\n};\n\nnuma_layout merge(numa_layout one, numa_layout two);\n\nsize_t per_shard_memory(size_t total_memory, unsigned nr_shards);\n\nvoid global_setup(unsigned nr_shards);\n\n}\n\ninternal::numa_layout configure(std::vector<resource::memory> m, bool mbind,\n        bool transparent_hugepages,\n        std::optional<std::string> hugetlbfs_path = {});\n\nvoid configure_minimal();\n\nnamespace internal {\n\nextern thread_local constinit int abort_on_alloc_failure_suppressed;\n\n}\n\nclass disable_abort_on_alloc_failure_temporarily {\npublic:\n    disable_abort_on_alloc_failure_temporarily() {\n        ++internal::abort_on_alloc_failure_suppressed;\n    }\n    ~disable_abort_on_alloc_failure_temporarily() noexcept {\n        --internal::abort_on_alloc_failure_suppressed;\n    }\n};\n\n// Disables heap profiling as long as this object is alive.\n// Can be nested, in which case the profiling is re-enabled when all\n// the objects go out of scope.\nclass disable_backtrace_temporarily {\n    sampler::disable_sampling_temporarily _disable_sampling;\npublic:\n    disable_backtrace_temporarily();\n};\n\nenum class reclaiming_result {\n    reclaimed_nothing,\n    reclaimed_something\n};\n\n// Determines when reclaimer can be invoked\nenum class reclaimer_scope {\n    //\n    // Reclaimer is only invoked in its own fiber. That fiber will be\n    // given higher priority than regular application fibers.\n    //\n    async,\n\n    //\n    // Reclaimer may be invoked synchronously with allocation.\n    // It may also be invoked in async scope.\n    //\n    // Reclaimer may invoke allocation, though it is discouraged because\n    // the system may be low on memory and such allocations may fail.\n    // Reclaimers which allocate should be prepared for re-entry.\n    //\n    sync\n};\n\nclass reclaimer {\npublic:\n    struct request {\n        // The number of bytes which is needed to be released.\n        // The reclaimer can release a different amount.\n        // If less is released then the reclaimer may be invoked again.\n        size_t bytes_to_reclaim;\n    };\n    using reclaim_fn = std::function<reclaiming_result ()>;\nprivate:\n    std::function<reclaiming_result (request)> _reclaim;\n    reclaimer_scope _scope;\npublic:\n    // Installs new reclaimer which will be invoked when system is falling\n    // low on memory. 'scope' determines when reclaimer can be executed.\n    reclaimer(std::function<reclaiming_result ()> reclaim, reclaimer_scope scope = reclaimer_scope::async);\n    reclaimer(std::function<reclaiming_result (request)> reclaim, reclaimer_scope scope = reclaimer_scope::async);\n    ~reclaimer();\n    reclaiming_result do_reclaim(size_t bytes_to_reclaim) { return _reclaim(request{bytes_to_reclaim}); }\n    reclaimer_scope scope() const { return _scope; }\n};\n\nextern std::pmr::polymorphic_allocator<char>* malloc_allocator;\n\n// Call periodically to recycle objects that were freed\n// on cpu other than the one they were allocated on.\n//\n// Returns @true if any work was actually performed.\nbool drain_cross_cpu_freelist();\n\n\n// We don't want the memory code calling back into the rest of\n// the system, so allow the rest of the system to tell the memory\n// code how to initiate reclaim.\n//\n// When memory is low, calling \\c hook(fn) will result in fn being called\n// in a safe place wrt. allocations.\nvoid set_reclaim_hook(\n        std::function<void (std::function<void ()>)> hook);\n\n/// \\endcond\n\n\n/// \\brief Set the global state of the abort on allocation failure behavior.\n///\n/// If enabled, an allocation failure (i.e., the requested memory\n/// could not be allocated even after reclaim was attempted), will\n/// generally result in `abort()` being called. If disabled, the\n/// failure is reported to the caller, e.g., by throwing a\n/// `std::bad_alloc` for C++ allocations such as new, or returning\n/// a null pointer from malloc.\n///\n/// Note that even if the global state is set to enabled, the\n/// `disable_abort_on_alloc_failure_temporarily` class may override\n/// the behavior tepmorarily on a given shard. That is, abort only\n/// occurs if abort is globablly enabled on this shard _and_ there\n/// are no `disable_abort_on_alloc_failure_temporarily` objects\n/// currently alive on this shard.\nvoid set_abort_on_allocation_failure(bool enabled);\n\n/// \\brief Determine the abort on allocation failure mode.\n///\n/// Return true if the global abort on allocation failure behavior\n/// is enabled, or false otherwise. Always returns false if the\n/// default (system) allocator is being used.\nbool is_abort_on_allocation_failure();\n\nclass statistics;\n\n/// Capture a snapshot of memory allocation statistics for this lcore.\nstatistics stats();\n\n/// Memory allocation statistics.\nclass statistics {\n    uint64_t _mallocs;\n    uint64_t _frees;\n    uint64_t _cross_cpu_frees;\n    size_t _total_memory;\n    size_t _free_memory;\n    uint64_t _total_bytes_allocated;\n    uint64_t _reclaims;\n    uint64_t _large_allocs;\n    uint64_t _failed_allocs;\n\n    uint64_t _foreign_mallocs;\n    uint64_t _foreign_frees;\n    uint64_t _foreign_cross_frees;\nprivate:\n    statistics(uint64_t mallocs, uint64_t frees, uint64_t cross_cpu_frees,\n            uint64_t total_memory, uint64_t free_memory, uint64_t total_bytes_allocated, uint64_t reclaims,\n            uint64_t large_allocs, uint64_t failed_allocs,\n            uint64_t foreign_mallocs, uint64_t foreign_frees, uint64_t foreign_cross_frees)\n        : _mallocs(mallocs), _frees(frees), _cross_cpu_frees(cross_cpu_frees)\n        , _total_memory(total_memory), _free_memory(free_memory), _total_bytes_allocated(total_bytes_allocated), _reclaims(reclaims)\n        , _large_allocs(large_allocs), _failed_allocs(failed_allocs)\n        , _foreign_mallocs(foreign_mallocs), _foreign_frees(foreign_frees)\n        , _foreign_cross_frees(foreign_cross_frees) {}\npublic:\n    /// Total number of memory allocations calls since the system was started.\n    uint64_t mallocs() const { return _mallocs; }\n    /// Total number of memory deallocations calls since the system was started.\n    uint64_t frees() const { return _frees; }\n    /// Total number of memory deallocations that occured on a different lcore\n    /// than the one on which they were allocated.\n    uint64_t cross_cpu_frees() const { return _cross_cpu_frees; }\n    /// Total number of objects which were allocated but not freed.\n    size_t live_objects() const { return mallocs() - frees(); }\n    /// Total free memory (in bytes)\n    size_t free_memory() const { return _free_memory; }\n    /// Total allocated memory (in bytes)\n    size_t allocated_memory() const { return _total_memory - _free_memory; }\n    /// Total memory (in bytes)\n    size_t total_memory() const { return _total_memory; }\n    /// Total number of bytes allocated since the system was started.\n    uint64_t total_bytes_allocated() const { return _total_bytes_allocated; }\n    /// Number of reclaims performed due to low memory\n    uint64_t reclaims() const { return _reclaims; }\n    /// Number of allocations which violated the large allocation threshold\n    uint64_t large_allocations() const { return _large_allocs; }\n    /// Number of allocations which failed, i.e., where the required memory could not be obtained\n    /// even after reclaim was attempted\n    uint64_t failed_allocations() const { return _failed_allocs; }\n    /// Number of foreign allocations\n    uint64_t foreign_mallocs() const { return _foreign_mallocs; }\n    /// Number of foreign frees\n    uint64_t foreign_frees() const { return _foreign_frees; }\n    /// Number of foreign frees on reactor threads\n    uint64_t foreign_cross_frees() const { return _foreign_cross_frees; }\n    friend statistics stats();\n};\n\nstruct memory_layout {\n    uintptr_t start;\n    uintptr_t end;\n};\n\n// Discover virtual address range used by the allocator on current shard.\n// Supported only when seastar allocator is enabled.\nmemory::memory_layout get_memory_layout();\n\n/// Returns the size of free memory in bytes.\nsize_t free_memory();\n\n/// Returns the value of free memory low water mark in bytes.\n/// When free memory is below this value, reclaimers are invoked until it goes above again.\nsize_t min_free_memory();\n\n/// Sets the value of free memory low water mark in memory::page_size units.\nvoid set_min_free_pages(size_t pages);\n\n/// Enable the large allocation warning threshold.\n///\n/// Warn when allocation above a given threshold are performed.\n///\n/// \\param threshold size (in bytes) above which an allocation will be logged\nvoid set_large_allocation_warning_threshold(size_t threshold);\n\n/// Gets the current large allocation warning threshold.\nsize_t get_large_allocation_warning_threshold();\n\n/// Disable large allocation warnings.\nvoid disable_large_allocation_warning();\n\n/// Set a different large allocation warning threshold for a scope.\nclass scoped_large_allocation_warning_threshold {\n    size_t _old_threshold;\npublic:\n    explicit scoped_large_allocation_warning_threshold(size_t threshold)\n            : _old_threshold(get_large_allocation_warning_threshold()) {\n        set_large_allocation_warning_threshold(threshold);\n    }\n    scoped_large_allocation_warning_threshold(const scoped_large_allocation_warning_threshold&) = delete;\n    scoped_large_allocation_warning_threshold(scoped_large_allocation_warning_threshold&& x) = delete;\n    ~scoped_large_allocation_warning_threshold() {\n        if (_old_threshold) {\n            set_large_allocation_warning_threshold(_old_threshold);\n        }\n    }\n    void operator=(const scoped_large_allocation_warning_threshold&) const = delete;\n    void operator=(scoped_large_allocation_warning_threshold&&) = delete;\n};\n\n/// Disable large allocation warnings for a scope.\nclass scoped_large_allocation_warning_disable {\n    size_t _old_threshold;\npublic:\n    scoped_large_allocation_warning_disable()\n            : _old_threshold(get_large_allocation_warning_threshold()) {\n        disable_large_allocation_warning();\n    }\n    scoped_large_allocation_warning_disable(const scoped_large_allocation_warning_disable&) = delete;\n    scoped_large_allocation_warning_disable(scoped_large_allocation_warning_disable&& x) = delete;\n    ~scoped_large_allocation_warning_disable() {\n        if (_old_threshold) {\n            set_large_allocation_warning_threshold(_old_threshold);\n        }\n    }\n    void operator=(const scoped_large_allocation_warning_disable&) const = delete;\n    void operator=(scoped_large_allocation_warning_disable&&) = delete;\n};\n\n/// @brief Describes an allocation location in the code.\n///\n/// The location is identified by its backtrace. One allocation_site can\n/// represent many allocations at the same location. `count` and `size`\n/// represent the cumulative sum of all allocations at the location. Note the\n/// size represents an extrapolated size and not the sampled one, i.e.: when\n/// looking at the total size of all allocation sites it will approximate the\n/// total memory usage\nstruct allocation_site {\n    mutable size_t count = 0; /// number of live objects allocated at backtrace.\n    mutable size_t size = 0; /// amount of bytes in live objects allocated at backtrace.\n    simple_backtrace backtrace; /// call site for this allocation\n\n    // All allocation sites are linked to each other. This can be used for easy\n    // iteration across them in gdb scripts where it's difficult to work with\n    // the C++ data structures.\n    mutable const allocation_site* next = nullptr; // next allocation site in the chain\n    mutable const allocation_site* prev = nullptr; // previous allocation site in the chain\n\n    bool operator==(const allocation_site& o) const {\n        return backtrace == o.backtrace;\n    }\n\n    bool operator!=(const allocation_site& o) const {\n        return !(*this == o);\n    }\n};\n\n/// @brief If memory sampling is on returns the current sampled memory live set\n///\n/// If there is tracked allocations (because heap profiling was on earlier)\n/// these will still be returned if heap profiling is now off\n///\n/// @return a vector of \\ref allocation_site\nstd::vector<allocation_site> sampled_memory_profile();\n\n/// @brief Copies the current sampled set of allocation_sites into the\n/// array pointed to by the output parameter\n///\n/// Copies up to \\p size elements of the current sampled set of allocation\n/// sites into the output array, which must have length at least \\p size.\n///\n/// Returns amount of copied elements. This method does not allocate so it\n/// is a useful alternative to \\ref sampled_memory_profile() when one wants\n/// to avoid allocating (e.g.: under OOM conditions).\n///\n/// @param output array to copy the allocation sites to\n/// @param size the size of the array pointed to by \\p output\n/// @return number of \\ref allocation_site copied to the vector\nsize_t sampled_memory_profile(allocation_site* output, size_t size);\n\n/// @brief Enable sampled heap profiling by setting a sample rate\n///\n/// @param sample_rate the sample rate to use. Disable heap profiling by setting\n/// the sample rate to 0\n///\n/// In order to use heap profiling you have to define\n/// `SEASTAR_HEAPPROF`.\n///\n/// Use \\ref sampled_memory_profile for API access to profiling data\n///\n/// Note: Changing the sampling rate while previously sampled allocations are\n/// still alive can lead to inconsistent results of their reported size (i.e.:\n/// their size will be over or under reported). Undefined behavior or memory\n/// corruption will not occur.\n///\n/// For an example script that makes use of the heap profiling data\n/// see [scylla-gdb.py] (https://github.com/scylladb/scylla/blob/e1b22b6a4c56b4f1d0adf65d1a11db4bcb51fe7d/scylla-gdb.py#L1439)\n/// This script can generate either textual representation of the data,\n/// or a zoomable flame graph ([flame graph generation instructions](https://github.com/scylladb/scylla/wiki/Seastar-heap-profiler),\n/// [example flame graph](https://user-images.githubusercontent.com/1389273/72920437-f0cf8a80-3d51-11ea-92f0-f3dbeb698871.png)).\nvoid set_heap_profiling_sampling_rate(size_t sample_rate);\n\n/// @brief Returns the current heap profiling sampling rate (0 means off)\n/// @return the current heap profiling sampling rate\nsize_t get_heap_profiling_sample_rate();\n\n/// @brief Enable heap profiling for the duration of the scope.\n///\n/// Note: Nesting different sample rates is currently not supported.\n///\n/// For more information about heap profiling see\n/// \\ref set_heap_profiling_sampling_rate().\nclass scoped_heap_profiling {\npublic:\n    scoped_heap_profiling(size_t) noexcept;\n    ~scoped_heap_profiling();\n};\n\n\n}\n}\n\nnamespace std {\n\ntemplate<>\nstruct hash<seastar::memory::allocation_site> {\n    size_t operator()(const seastar::memory::allocation_site& bi) const {\n        return std::hash<seastar::simple_backtrace>()(bi.backtrace);\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/metrics.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n\n#include <concepts>\n#include <functional>\n#include <limits>\n#include <map>\n#include <type_traits>\n#include <variant>\n#include <fmt/format.h>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/metrics_types.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/bool_class.hh>\n\n/*! \\file metrics.hh\n *  \\brief header for metrics creation.\n *\n *  This header file contains the metrics creation method with their helper function.\n *  Include this file when need to create metrics.\n *  Typically this will be in your source file.\n *\n *  Code that is under the impl namespace should not be used directly.\n *\n */\n\nnamespace seastar {\n\n/*!\n * \\addtogroup metrics\n * @{\n *\n * \\namespace seastar::metrics\n * \\brief metrics creation and registration\n *\n * the metrics namespace holds the relevant method and classes to generate metrics.\n *\n * The metrics layer support registering metrics, that later will be\n * exported via different API protocols.\n *\n * To be able to support multiple protocols the following simplifications where made:\n * 1. The id of the metrics is based on the collectd id\n * 2. A metric could be a single value either a reference or a function\n *\n * To add metrics definition to class A do the following:\n * * Add a metrics_group memeber to A\n * * Add a a set_metrics() method that would be called in the constructor.\n *\n *\n * In A header file\n * \\code\n * #include \"core/metrics_registration.hh\"\n * class A {\n *   metric_groups _metrics\n *\n *   void setup_metrics();\n *\n * };\n * \\endcode\n *\n * In A source file:\n *\n * \\code\n * include \"core/metrics.hh\"\n *\n * void A::setup_metrics() {\n *   namespace sm = seastar::metrics;\n *   _metrics = sm::create_metric_group();\n *   _metrics->add_group(\"cache\", {sm::make_gauge(\"bytes\", \"used\", [this] { return _region.occupancy().used_space(); })});\n * }\n * \\endcode\n */\n\nnamespace metrics {\n\n\nclass double_registration : public std::runtime_error {\npublic:\n    double_registration(std::string what);\n};\n\n/*!\n * \\defgroup metrics_types metrics type definitions\n * The following are for the metric layer use, do not use them directly\n * Instead use the make_counter, make_gauge\n *\n */\nusing metric_type_def = sstring; /*!< Used to hold an inherit type (like bytes)*/\nusing metric_name_type = sstring; /*!<  The metric name'*/\nusing instance_id_type = sstring; /*!<  typically used for the shard id*/\nusing skip_when_empty = bool_class<class skip_when_empty_tag>;\n\n/*!\n * \\brief Human-readable description of a metric/group.\n *\n *\n * Uses a separate class to deal with type resolution\n *\n * Add this to metric creation:\n *\n * \\code\n * _metrics->add_group(\"groupname\", {\n *   sm::make_gauge(\"metric_name\", value, description(\"A documentation about the return value\"))\n * });\n * \\endcode\n *\n */\nclass description {\npublic:\n    description(sstring s = sstring()) : _s(std::move(s))\n    {}\n    const sstring& str() const {\n        return _s;\n    }\nprivate:\n    sstring _s;\n};\n\n/*!\n * \\brief Label a metrics\n *\n * Label are useful for adding information about a metric that\n * later you would need to aggregate by.\n * For example, if you have multiple queues on a shard.\n * Adding the queue id as a Label will allow you to use the same name\n * of the metrics with multiple id instances.\n *\n * label_instance holds an instance of label consist of a key and value.\n *\n * Typically you will not generate a label_instance yourself, but use a label\n * object for that.\n * @see label for more information\n *\n *\n */\nclass label_instance {\n    sstring _key;\n    sstring _value;\npublic:\n    /*!\n     * \\brief create a label_instance\n     * label instance consists of key and value.\n     * The key is an sstring.\n     * T - the value type can be any type that can be fmt::format'ed to string\n     * (ie. if it provides fmt::format<T> specialization).\n     *\n     * All primitive types are supported so all the following examples are valid:\n     * label_instance a(\"smp_queue\", 1)\n     * label_instance a(\"my_key\", \"my_value\")\n     * label_instance a(\"internal_id\", -1)\n     */\n    template<typename T>\n    label_instance(const sstring& key, T v) : _key(key), _value(fmt::to_string(v)){}\n\n    /*!\n     * \\brief returns the label key\n     */\n    const sstring key() const {\n        return _key;\n    }\n\n    /*!\n     * \\brief returns the label value\n     */\n    const sstring value() const {\n        return _value;\n    }\n    bool operator<(const label_instance&) const;\n    bool operator==(const label_instance&) const;\n    bool operator!=(const label_instance&) const;\n};\n\n\n/*!\n * \\brief Class that creates label instances\n *\n * A factory class to create label instance\n * Typically, the same Label name is used in multiple places.\n * label is a label factory, you create it once, and use it to create the label_instance.\n *\n * In the example we would like to label the smp_queue with with the queue owner\n *\n * seastar::metrics::label smp_owner(\"smp_owner\");\n *\n * now, when creating a new smp metric we can add a label to it:\n *\n * sm::make_queue_length(\"send_batch_queue_length\", _last_snt_batch, {smp_owner(cpuid)})\n *\n * where cpuid in this case is unsiged.\n */\nclass label {\n    sstring key;\npublic:\n    using instance = label_instance;\n    /*!\n     * \\brief creating a label\n     * key is the label name, it will be the key for all label_instance\n     * that will be created from this label.\n     */\n    explicit label(const sstring& key) : key(key) {\n    }\n\n    /*!\n     * \\brief creating a label instance\n     *\n     * Use the function operator to create a new label instance.\n     * T - the value type can be any type that can be lexical_cast to string\n     * (ie. if it support the redirection operator for stringstream).\n     *\n     * All primitive types are supported so if lab is a label, all the following examples are valid:\n     * lab(1)\n     * lab(\"my_value\")\n     * lab(-1)\n     */\n    template<typename T>\n    instance operator()(T value) const {\n        return label_instance(key, std::forward<T>(value));\n    }\n\n    /*!\n     * \\brief returns the label name\n     */\n    const sstring& name() const {\n        return key;\n    }\n};\n\n/*!\n * \\namespace impl\n * \\brief holds the implementation parts of the metrics layer, do not use directly.\n *\n * The metrics layer define a thin API for adding metrics.\n * Some of the implementation details need to be in the header file, they should not be use directly.\n */\nnamespace impl {\n\n// A escaped label value.\n//\n// This is \"just\" as a string, with the additional invariant that it has already been\n// escaped, and this type preserves that invariant.\nclass escaped_string {\n    sstring _value;\n\npublic:\n    escaped_string() = default;\n    escaped_string(const escaped_string&) = default;\n    escaped_string(escaped_string&&) = default;\n\n    // this constructor escapes the input if necessary\n    explicit escaped_string(sstring v);\n\n    const sstring& value() const {\n        return _value;\n    }\n\n    operator const sstring&() const {\n        return value();\n    }\n\n    bool operator==(const escaped_string& o) const = default;\n    auto operator<=>(const escaped_string& o) const = default;\n\n    escaped_string& operator=(const escaped_string&) = default;\n    escaped_string& operator=(escaped_string&&) = default;\n\n    // assignment from an unescaped string escapes it if necessary\n    escaped_string& operator=(const sstring&);\n};\n\nusing labels_type = std::map<sstring, escaped_string>;\n\n// The value binding data types\nenum class data_type : uint8_t {\n    COUNTER,\n    REAL_COUNTER,\n    GAUGE,\n    HISTOGRAM,\n    SUMMARY,\n};\n\ntemplate <bool callable, typename T>\nstruct real_counter_type_traits {\n    using type = T;\n};\n\ntemplate <typename T>\nstruct real_counter_type_traits<true, T> {\n    using type = std::invoke_result_t<T>;\n};\n\ntemplate <typename T>\nstruct counter_type_traits {\n    using real_traits = real_counter_type_traits<std::is_invocable_v<T>, T>;\n    static constexpr bool is_integral = std::is_integral_v<typename real_traits::type>;\n    static constexpr data_type type = is_integral ? data_type::COUNTER : data_type::REAL_COUNTER;\n};\n\n/*!\n * \\brief A helper class that used to return metrics value.\n *\n * Do not use directly @see metrics_creation\n */\nclass metric_value {\npublic:\n    std::variant<double, histogram> u;\n    data_type _type;\n    data_type type() const {\n        return _type;\n    }\n\n    double d() const {\n        return std::get<double>(u);\n    }\n\n    uint64_t ui() const {\n        auto d = std::get<double>(u);\n        if (d >= 0 && d <= double(std::numeric_limits<long>::max())) {\n            return lround(d);\n        } else {\n            // double value is out of range or NaN or Inf\n            ulong_conversion_error(d);\n            return 0;\n        }\n    }\n\n    int64_t i() const {\n        auto d = std::get<double>(u);\n        if (d >= double(std::numeric_limits<long>::min()) && d <= double(std::numeric_limits<long>::max())) {\n            return lround(d);\n        } else {\n            // double value is out of range or NaN or Inf\n            ulong_conversion_error(d);\n            return 0;\n        }\n    }\n\n    metric_value()\n            : _type(data_type::GAUGE) {\n    }\n\n    metric_value(histogram&& h, data_type t = data_type::HISTOGRAM) :\n        u(std::move(h)), _type(t) {\n    }\n    metric_value(const histogram& h, data_type t = data_type::HISTOGRAM) :\n        u(h), _type(t) {\n    }\n\n    metric_value(double d, data_type t)\n            : u(d), _type(t) {\n    }\n\n    metric_value& operator+=(const metric_value& c);\n\n    metric_value operator+(const metric_value& c) {\n        return metric_value(*this) += c;\n    }\n\n    const histogram& get_histogram() const {\n        return std::get<histogram>(u);\n    }\n\n    /*!\n     * \\brief return true if this metric was never used\n     *\n     * Histograms, Summaries and counters are ever growing by nature, so\n     * it is possible to check if they have been used or not.\n     */\n    bool is_empty() const noexcept {\n        return ((_type == data_type::HISTOGRAM || _type == data_type::SUMMARY) && get_histogram().sample_count == 0) ||\n                ((_type == data_type::COUNTER || _type == data_type::REAL_COUNTER) && d() == 0);\n    }\nprivate:\n    static void ulong_conversion_error(double d);\n};\n\nusing metric_function = std::function<metric_value()>;\n\nstruct metric_type {\n    data_type base_type;\n    metric_type_def type_name;\n};\n\nstruct metric_definition_impl {\n    metric_name_type name;\n    metric_type type;\n    metric_function f;\n    description d;\n    bool enabled = true;\n    skip_when_empty _skip_when_empty = skip_when_empty::no;\n    std::vector<std::string> aggregate_labels;\n    labels_type labels;\n    metric_definition_impl& operator ()(bool enabled);\n    metric_definition_impl& operator ()(const label_instance& label);\n    metric_definition_impl& operator ()(skip_when_empty skip) noexcept;\n    metric_definition_impl& aggregate(const std::vector<label>& labels) noexcept;\n    metric_definition_impl& set_skip_when_empty(bool skip=true) noexcept;\n    metric_definition_impl& set_type(const sstring& type_name);\n    metric_definition_impl(\n        metric_name_type name,\n        metric_type type,\n        metric_function f,\n        description d,\n        std::vector<label_instance> labels,\n        std::vector<label> aggregate_labels = {});\n};\n\nclass metric_groups_def {\npublic:\n    metric_groups_def() = default;\n    virtual ~metric_groups_def() = default;\n    metric_groups_def(const metric_groups_def&) = delete;\n    metric_groups_def(metric_groups_def&&) = default;\n    virtual metric_groups_def& add_metric(group_name_type name, const metric_definition& md) = 0;\n    virtual metric_groups_def& add_group(group_name_type name, const std::initializer_list<metric_definition>& l) = 0;\n    virtual metric_groups_def& add_group(group_name_type name, const std::vector<metric_definition>& l) = 0;\n};\n\nescaped_string shard();\n\ntemplate<std::invocable T>\nmetric_function make_function(T val, data_type dt) {\n    return [dt, val = std::move(val)] {\n        return metric_value(val(), dt);\n    };\n}\n\ntemplate<typename T>\nmetric_function make_function(T& val, data_type dt) {\n    return [dt, &val] {\n        return metric_value(val, dt);\n    };\n}\n}\n\nextern const bool metric_disabled;\n\nextern label shard_label;\n\n/*\n * The metrics definition are defined to be compatible with collectd metrics defintion.\n * Typically you should used gauge or derived.\n */\n\n\n/*!\n * \\brief Gauge are a general purpose metric.\n *\n * They can support floating point and can increase or decrease\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_gauge(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {}) {\n    return {name, {impl::data_type::GAUGE, \"gauge\"}, make_function(std::forward<T>(val), impl::data_type::GAUGE), d, labels};\n}\n\n/*!\n * \\brief Gauge are a general purpose metric.\n *\n * They can support floating point and can increase or decrease\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_gauge(metric_name_type name,\n        description d, T&& val) {\n    return {name, {impl::data_type::GAUGE, \"gauge\"}, make_function(std::forward<T>(val), impl::data_type::GAUGE), d, {}};\n}\n\n/*!\n * \\brief Gauge are a general purpose metric.\n *\n * They can support floating point and can increase or decrease\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_gauge(metric_name_type name,\n        description d, std::vector<label_instance> labels, T&& val) {\n    return {name, {impl::data_type::GAUGE, \"gauge\"}, make_function(std::forward<T>(val), impl::data_type::GAUGE), d, labels};\n}\n\n\n/*!\n * \\brief create a counter metric\n *\n * Counters are used when a rate is more interesting than the value, monitoring systems take\n * derivation from it to display.\n *\n * It's an integer or floating point value that can increase or decrease.\n *\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_counter(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {}) {\n    auto type = impl::counter_type_traits<std::remove_reference_t<T>>::type;\n    return {name, {type, \"counter\"}, make_function(std::forward<T>(val), type), d, labels};\n}\n\n/*!\n * \\brief create a counter metric\n *\n * Counters are used when a rate is more interesting than the value, monitoring systems take\n * derivation from it to display.\n *\n * It's an integer or floating point value that can increase or decrease.\n *\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_counter(metric_name_type name, description d, T&& val) {\n    return make_counter(std::move(name), std::forward<T>(val), std::move(d), {});\n}\n\n/*!\n * \\brief create a counter metric\n *\n * Counters are used when a rate is more interesting than the value, monitoring systems take\n * derivation from it to display.\n *\n * It's an integer or floating point value that can increase or decrease.\n *\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_counter(metric_name_type name, description d, std::vector<label_instance> labels, T&& val) {\n    return make_counter(std::move(name), std::forward<T>(val), std::move(d), std::move(labels));\n}\n\n\n/*!\n * \\brief create a histogram metric.\n *\n * Histograms are a list o buckets with upper values and counter for the number\n * of entries in each bucket.\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_histogram(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {}) {\n    return  {name, {impl::data_type::HISTOGRAM, \"histogram\"}, make_function(std::forward<T>(val), impl::data_type::HISTOGRAM), d, labels};\n}\n\n/*!\n * \\brief create a histogram metric.\n *\n * Histograms are a list o buckets with upper values and counter for the number\n * of entries in each bucket.\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_histogram(metric_name_type name,\n        description d, std::vector<label_instance> labels, T&& val) {\n    return  {name, {impl::data_type::HISTOGRAM, \"histogram\"}, make_function(std::forward<T>(val), impl::data_type::HISTOGRAM), d, labels};\n}\n\n\n/*!\n * \\brief create a histogram metric.\n *\n * Histograms are a list o buckets with upper values and counter for the number\n * of entries in each bucket.\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_histogram(metric_name_type name,\n        description d, T&& val) {\n    return  {name, {impl::data_type::HISTOGRAM, \"histogram\"}, make_function(std::forward<T>(val), impl::data_type::HISTOGRAM), d, {}};\n}\n\n/*!\n * \\brief create a summary metric.\n *\n * Summaries are a different kind of histograms. It reports in quantiles.\n * For example, the p99 and p95 latencies.\n */\ntemplate<typename T>\nimpl::metric_definition_impl make_summary(metric_name_type name,\n        description d, T&& val) {\n    return  {name, {impl::data_type::SUMMARY, \"summary\"}, make_function(std::forward<T>(val), impl::data_type::SUMMARY), d, {}};\n}\n\n\n/*!\n * \\brief create a total_bytes metric.\n *\n * total_bytes are used for an ever growing counters, like the total bytes\n * passed on a network.\n */\n\ntemplate<typename T>\nimpl::metric_definition_impl make_total_bytes(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {},\n        instance_id_type = impl::shard()) {\n    return make_counter(name, std::forward<T>(val), d, labels).set_type(\"total_bytes\");\n}\n\n/*!\n * \\brief create a current_bytes metric.\n *\n * current_bytes are used to report on current status in bytes.\n * For example the current free memory.\n */\n\ntemplate<typename T>\nimpl::metric_definition_impl make_current_bytes(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {},\n        instance_id_type = impl::shard()) {\n    return make_gauge(name, std::forward<T>(val), d, labels).set_type(\"bytes\");\n}\n\n\n/*!\n * \\brief create a queue_length metric.\n *\n * queue_length are used to report on queue length\n */\n\ntemplate<typename T>\nimpl::metric_definition_impl make_queue_length(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {},\n        instance_id_type = impl::shard()) {\n    return make_gauge(name, std::forward<T>(val), d, labels).set_type(\"queue_length\");\n}\n\n\n/*!\n * \\brief create a total operation metric.\n *\n * total_operations are used for ever growing operation counter.\n */\n\ntemplate<typename T>\nimpl::metric_definition_impl make_total_operations(metric_name_type name,\n        T&& val, description d = description(), std::vector<label_instance> labels = {},\n        instance_id_type = impl::shard()) {\n    return make_counter(name, std::forward<T>(val), d, labels).set_type(\"total_operations\");\n}\n\n/*! @} */\n}\n}\n\nnamespace fmt {\ntemplate<>\nstruct formatter<seastar::metrics::impl::escaped_string> : public fmt::formatter<seastar::sstring> {\n    template <typename FormatContext>\n    auto format(const seastar::metrics::impl::escaped_string& s, FormatContext& ctx) const {\n        return fmt::formatter<seastar::sstring>::format(s.value(), ctx);\n    }\n};\n}\n"
  },
  {
    "path": "include/seastar/core/metrics_api.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n\n#include <seastar/core/metrics.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/sharded.hh>\n#include <boost/functional/hash.hpp>\n\n#include <deque>\n\n/*!\n * \\file metrics_api.hh\n * \\brief header file for metric API layer (like prometheus or collectd)\n *\n *\n *\n */\nnamespace seastar {\nnamespace metrics {\nnamespace impl {\n\nusing internalized_labels_ref = lw_shared_ptr<const labels_type>;\n\n}\n}\n}\n\nnamespace std {\n\ntemplate<>\nstruct hash<seastar::metrics::impl::labels_type> {\n    using argument_type = seastar::metrics::impl::labels_type;\n    using result_type = ::std::size_t;\n    result_type operator()(argument_type const& s) const {\n        result_type h = 0;\n        for (auto&& i : s) {\n            boost::hash_combine(h, std::hash<seastar::sstring>{}(i.second));\n        }\n        return h;\n    }\n};\n\n}\n\nnamespace seastar {\nnamespace metrics {\n\nstruct relabel_config;\n\nstruct metric_family_config;\n/*!\n * \\brief result of metric relabeling\n *\n * The result of calling set_relabel_configs.\n *\n * metrics_relabeled_due_to_collision the number of metrics that caused conflict\n * and were relabeled to avoid name collision.\n *\n * Non zero value indicates there were name collisions.\n *\n */\nstruct metric_relabeling_result {\n    size_t metrics_relabeled_due_to_collision;\n};\n\nnamespace impl {\n\n/**\n * Metrics are collected in groups that belongs to some logical entity.\n * For example, different measurements of the cpu, will belong to group \"cpu\".\n *\n * Name is the metric name like used_objects or used_bytes\n *\n * Inherit type allows customizing one of the basic types (gauge, counter, derive).\n *\n * Instance_id is used to differentiate multiple instance of the metrics.\n * In the seastar environment it is typical to have a metric per shard.\n *\n */\n\nclass metric_id {\npublic:\n    metric_id() = default;\n    metric_id(group_name_type group, metric_name_type name,\n                    internalized_labels_ref labels)\n                    : _group(std::move(group)), _name(\n                                    std::move(name)), _labels(labels) {\n    }\n    metric_id(metric_id &&) = default;\n    metric_id(const metric_id &) = default;\n\n    metric_id & operator=(metric_id &&) = default;\n    metric_id & operator=(const metric_id &) = default;\n\n    const group_name_type & group_name() const {\n        return _group;\n    }\n    void group_name(const group_name_type & name) {\n        _group = name;\n    }\n    const instance_id_type & instance_id() const {\n        return _labels->at(shard_label.name());\n    }\n    const metric_name_type & name() const {\n        return _name;\n    }\n    const labels_type& labels() const {\n        return *_labels;\n    }\n    internalized_labels_ref internalized_labels() const {\n        return _labels;\n    }\n    void update_labels(internalized_labels_ref labels) {\n        _labels = labels;\n    }\n    sstring full_name() const;\n\n    bool operator<(const metric_id&) const;\n    bool operator==(const metric_id&) const;\nprivate:\n    auto as_tuple() const {\n        return std::tie(group_name(), instance_id(), name(), labels());\n    }\n    group_name_type _group;\n    metric_name_type _name;\n    internalized_labels_ref _labels;\n};\n}\n}\n}\n\nnamespace std {\n\ntemplate<>\nstruct hash<seastar::metrics::impl::metric_id>\n{\n    typedef seastar::metrics::impl::metric_id argument_type;\n    typedef ::std::size_t result_type;\n    result_type operator()(argument_type const& s) const\n    {\n        result_type const h1 ( std::hash<seastar::sstring>{}(s.group_name()) );\n        result_type const h2 ( std::hash<seastar::sstring>{}(s.instance_id()) );\n        return h1 ^ (h2 << 1); // or use boost::hash_combine\n    }\n};\n\n}\n\nnamespace seastar {\nnamespace metrics {\nnamespace impl {\n\n/*!\n * \\brief holds metadata information of a metric family\n *\n * Holds the information that is shared between all metrics\n * that belongs to the same metric_family\n */\nstruct metric_family_info {\n    data_type type;\n    metric_type_def inherit_type;\n    description d;\n    sstring name;\n    std::vector<std::string> aggregate_labels;\n};\n\n\n/*!\n * \\brief holds metric metadata\n */\nstruct metric_info {\n    metric_id id;\n    internalized_labels_ref original_labels;\n    bool enabled;\n    skip_when_empty should_skip_when_empty;\n};\n\nclass internalized_holder {\n    internalized_labels_ref _labels;\npublic:\n    explicit internalized_holder(labels_type labels) : _labels(make_lw_shared<labels_type>(std::move(labels))) {\n    }\n\n    explicit internalized_holder(internalized_labels_ref labels) : _labels(std::move(labels)) {\n    }\n\n    internalized_labels_ref labels_ref() const {\n        return _labels;\n    }\n\n    const labels_type& labels() const {\n        return *_labels;\n    }\n\n    size_t has_users() const {\n        // Getting the count wrong isn't a correctness issue but will just make internalization worse\n        return _labels.use_count() > 1;\n    }\n};\n\ninline bool operator<(const internalized_holder& lhs, const labels_type& rhs) {\n    return lhs.labels() < rhs;\n}\ninline bool operator<(const labels_type& lhs, const internalized_holder& rhs) {\n    return lhs < rhs.labels();\n}\ninline bool operator<(const internalized_holder& lhs, const internalized_holder& rhs) {\n    return lhs.labels() < rhs.labels();\n}\n\n\nclass impl;\n\nclass registered_metric final {\n    metric_info _info;\n    metric_function _f;\npublic:\n    registered_metric(metric_id id, metric_function f, bool enabled=true, skip_when_empty skip=skip_when_empty::no);\n    metric_value operator()() const {\n        return _f();\n    }\n\n    bool is_enabled() const {\n        return _info.enabled;\n    }\n\n    void set_enabled(bool b) {\n        _info.enabled = b;\n    }\n    void set_skip_when_empty(skip_when_empty skip) noexcept {\n        _info.should_skip_when_empty = skip;\n    }\n    const metric_id& get_id() const {\n        return _info.id;\n    }\n\n    const metric_info& info() const {\n        return _info;\n    }\n    metric_info& info() {\n        return _info;\n   }\n    const metric_function& get_function() const {\n        return _f;\n    }\n};\n\nusing register_ref = shared_ptr<registered_metric>;\nusing metric_instances = std::map<internalized_holder, register_ref, std::less<>>;\nusing metrics_registration = std::vector<register_ref>;\n\nclass metric_groups_impl : public metric_groups_def {\n    metrics_registration _registration;\n    shared_ptr<impl> _impl; // keep impl alive while metrics are registered\npublic:\n    metric_groups_impl();\n    ~metric_groups_impl();\n    metric_groups_impl(const metric_groups_impl&) = delete;\n    metric_groups_impl(metric_groups_impl&&) = default;\n    metric_groups_impl& add_metric(group_name_type name, const metric_definition& md);\n    metric_groups_impl& add_group(group_name_type name, const std::initializer_list<metric_definition>& l);\n    metric_groups_impl& add_group(group_name_type name, const std::vector<metric_definition>& l);\n};\n\nclass metric_family {\n    metric_instances _instances;\n    metric_family_info _info;\npublic:\n    using iterator = metric_instances::iterator;\n    using const_iterator = metric_instances::const_iterator;\n\n    metric_family() = default;\n    metric_family(const metric_family&) = default;\n    metric_family(const metric_instances& instances) : _instances(instances) {\n    }\n    metric_family(const metric_instances& instances, const metric_family_info& info) : _instances(instances), _info(info) {\n    }\n    metric_family(metric_instances&& instances, metric_family_info&& info) : _instances(std::move(instances)), _info(std::move(info)) {\n    }\n    metric_family(metric_instances&& instances) : _instances(std::move(instances)) {\n    }\n\n    register_ref& operator[](const internalized_labels_ref& l) {\n        return _instances[internalized_holder(l)];\n    }\n\n    const register_ref& at(const internalized_labels_ref& l) const {\n        return _instances.at(internalized_holder(l));\n    }\n\n    metric_family_info& info() {\n        return _info;\n    }\n\n    const metric_family_info& info() const {\n        return _info;\n    }\n\n    iterator find(const labels_type& l) {\n        return _instances.find(l);\n    }\n\n    const_iterator find(const labels_type& l) const {\n        return _instances.find(l);\n    }\n\n    iterator begin() {\n        return _instances.begin();\n    }\n\n    const_iterator begin() const {\n        return _instances.cbegin();\n    }\n\n    iterator end() {\n        return _instances.end();\n    }\n\n    bool empty() const {\n        return _instances.empty();\n    }\n\n    iterator erase(const_iterator position) {\n        return _instances.erase(position);\n    }\n\n    const_iterator end() const {\n        return _instances.cend();\n    }\n\n    uint32_t size() const {\n        return _instances.size();\n    }\n\n};\n\nusing value_map = std::map<sstring, metric_family>;\n\n/*!\n * \\brief Subset of the per series metadata that is shared via get_values to other shards.\n *\n * Allows omitting metadata that is already stored elsewhere or not needed by\n * the metrics scrap handlers.\n *\n * Not copyable to allow for safely sharing internalized data.\n */\nclass metric_series_metadata {\n    // prom backend only needs the label from here but scollectd needs group and\n    // metric name separately. metric_family_info only stores the merged and\n    // filtered name so we have to duplicate it here.\n    metric_id _id;\n    skip_when_empty _should_skip_when_empty;\npublic:\n    metric_series_metadata() = default;\n    metric_series_metadata(metric_id id, skip_when_empty should_skip_when_empty)\n        : _id(std::move(id)), _should_skip_when_empty(should_skip_when_empty) {\n    }\n\n    metric_series_metadata(const metric_series_metadata&) = delete;\n    metric_series_metadata& operator=(const metric_series_metadata&) = delete;\n\n    metric_series_metadata(metric_series_metadata&&) noexcept = default;\n    metric_series_metadata& operator=(metric_series_metadata&&) noexcept = default;\n\n    const labels_type& labels() const {\n      return _id.labels();\n    }\n\n    skip_when_empty should_skip_when_empty() const {\n        return _should_skip_when_empty;\n    }\n\n    group_name_type group_name() const {\n        return _id.group_name();\n    }\n\n    group_name_type name() const {\n        return _id.name();\n    }\n};\n\nusing metric_metadata_fifo = std::deque<metric_series_metadata>;\n\n/*!\n * \\brief holds a metric family metadata\n *\n * The meta data of a metric family compose of the\n * metadata of the family, and a vector of the metadata for\n * each of the metrics.\n *\n * The struct is used for two purposes. First, it allows iterating over all metric_families\n * and all metrics related to them. Second, it only contains enabled metrics,\n * making disabled metrics more efficient.\n * The struct is recreated when impl._value_map changes\n */\nstruct metric_family_metadata {\n    metric_family_info mf;\n    metric_metadata_fifo metrics;\n\n    metric_family_metadata() = default;\n    metric_family_metadata(metric_family_metadata &&) = default;\n    metric_family_metadata &operator=(metric_family_metadata &&) = default;\n    metric_family_metadata(metric_family_info mf, metric_metadata_fifo metrics)\n        : mf(std::move(mf)), metrics(std::move(metrics)) {}\n};\n\nstatic_assert(std::is_nothrow_move_assignable_v<metric_family_metadata>);\n\nusing value_vector = std::deque<metric_value>;\nusing metric_metadata = std::vector<metric_family_metadata>;\nusing metric_values = std::deque<value_vector>;\n\nstruct values_copy {\n    shared_ptr<metric_metadata> metadata;\n    metric_values values;\n};\n\nstruct config {\n    sstring hostname;\n};\n\nusing internalized_set = std::set<internalized_holder, std::less<>>;\n\nclass impl {\n    value_map _value_map;\n    config _config;\n    bool _dirty = true;\n    shared_ptr<metric_metadata> _metadata;\n    std::set<sstring> _labels;\n    std::vector<std::deque<metric_function>> _current_metrics;\n    std::vector<relabel_config> _relabel_configs;\n    std::vector<metric_family_config> _metric_family_configs;\n    internalized_set _internalized_labels;\npublic:\n    value_map& get_value_map() {\n        return _value_map;\n    }\n\n    const value_map& get_value_map() const {\n        return _value_map;\n    }\n\n    register_ref add_registration(const metric_id& id, const metric_type& type, metric_function f, const description& d, bool enabled, skip_when_empty skip, const std::vector<std::string>& aggregate_labels);\n    internalized_labels_ref internalize_labels(labels_type labels);\n    void remove_registration(const metric_id& id);\n    future<> stop() {\n        return make_ready_future<>();\n    }\n    const config& get_config() const {\n        return _config;\n    }\n    void set_config(const config& c) {\n        _config = c;\n    }\n\n    shared_ptr<metric_metadata> metadata();\n\n    std::vector<std::deque<metric_function>>& functions();\n\n    void update_metrics_if_needed();\n\n    void dirty() {\n        _dirty = true;\n    }\n\n    const std::set<sstring>& get_labels() const noexcept {\n        return _labels;\n    }\n\n    future<metric_relabeling_result> set_relabel_configs(const std::vector<relabel_config>& relabel_configs);\n\n    const std::vector<relabel_config>& get_relabel_configs() const noexcept {\n        return _relabel_configs;\n    }\n    const std::vector<metric_family_config>& get_metric_family_configs() const noexcept {\n        return _metric_family_configs;\n    }\n\n    void set_metric_family_configs(const std::vector<metric_family_config>& metrics_config);\n\n    void update_aggregate(metric_family_info& mf) const noexcept;\n\nprivate:\n    void gc_internalized_labels();\n    bool apply_relabeling(const relabel_config& rc, metric_info& info);\n};\n\nconst value_map& get_value_map();\nusing values_reference = shared_ptr<values_copy>;\n\nforeign_ptr<values_reference> get_values();\n\nshared_ptr<impl> get_local_impl();\n\nvoid unregister_metric(const metric_id & id);\n\n/*!\n * \\brief initialize metric group\n *\n * Create a metric_group_def.\n * No need to use it directly.\n */\nstd::unique_ptr<metric_groups_def> create_metric_groups();\n\n}\n\n/// Metrics configuration options.\nstruct options : public program_options::option_group {\n    /// \\brief The hostname used by the metrics.\n    ///\n    /// If not set, the local hostname will be used.\n    program_options::value<std::string> metrics_hostname;\n\n    options(program_options::option_group* parent_group);\n};\n\n/*!\n * \\brief set the metrics configuration\n */\nfuture<> configure(const options& opts);\n\n/*!\n * \\brief Perform relabeling and operation on metrics dynamically.\n *\n * The function would return true if the changes were applied with no conflict\n * or false, if there was a conflict in the registration.\n *\n  * The general logic follows Prometheus metrics_relabel_config configuration.\n * The relabel rules are applied one after the other.\n * You can add or change a label. you can enable or disable a metric,\n * in that case the metrics will not be reported at all.\n * You can turn on and off the skip_when_empty flag.\n *\n * Using the Prometheus convention, the metric name is __name__.\n * Names cannot be changed.\n *\n * Import notes:\n * - The relabeling always starts from the original set of labels the metric\n *   was created with.\n * - calling with an empty set will remove the relabel config and will\n *   return all metrics to their original labels\n * - To prevent a situation that calling this function would crash the system.\n *   in a situation where a conflicting metrics name are entered, an additional label\n *   will be added to the labels with a unique ID.\n *\n * A few examples:\n * To add a level label with a value 1, to the reactor_utilization metric:\n *  std::vector<sm::relabel_config> rl(1);\n    rl[0].source_labels = {\"__name__\"};\n    rl[0].target_label = \"level\";\n    rl[0].replacement = \"1\";\n    rl[0].expr = \"reactor_utilization\";\n   set_relabel_configs(rl);\n *\n * To report only the metrics with the level label equals 1\n *\n    std::vector<sm::relabel_config> rl(2);\n    rl[0].source_labels = {\"__name__\"};\n    rl[0].action = sm::relabel_config::relabel_action::drop;\n\n    rl[1].source_labels = {\"level\"};\n    rl[1].expr = \"1\";\n    rl[1].action = sm::relabel_config::relabel_action::keep;\n    set_relabel_configs(rl);\n\n */\nfuture<metric_relabeling_result> set_relabel_configs(const std::vector<relabel_config>& relabel_configs);\n/*\n * \\brief return the current relabel_configs\n * This function returns a vector of the current relabel configs\n */\nconst std::vector<relabel_config>& get_relabel_configs();\n\n/*\n * \\brief change the metrics family config\n *\n * Family config is a configuration that relates to all metrics with the same name but with different labels.\n * set_metric_family_configs allows changing that configuration during run time.\n * Specifically, change the label aggregation based on a metric name.\n *\n * The following is an example for setting the aggregate labels for the metric test_gauge_1\n * and all metrics matching the regex test_gauge1.*:\n *\n * std::vector<sm::metric_family_config> fc(2);\n * fc[0].name = \"test_gauge_1\";\n * fc[0].aggregate_labels = { \"lb\" };\n * fc[1].regex_name = \"test_gauge1.*\";\n * fc[1].aggregate_labels = { \"ll\", \"aa\" };\n * sm::set_metric_family_configs(fc);\n */\nvoid set_metric_family_configs(const std::vector<metric_family_config>& metrics_config);\n\n/*\n * \\brief return the current metric_family_config\n * This function returns a vector of the current metrics family config\n */\nconst std::vector<metric_family_config>& get_metric_family_configs();\n}\n}\n"
  },
  {
    "path": "include/seastar/core/metrics_registration.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n\n#include <memory>\n#include <vector>\n\n#include <seastar/core/sstring.hh>\n\n/*!\n * \\file metrics_registration.hh\n * \\brief holds the metric_groups definition needed by class that reports metrics\n *\n * If class A needs to report metrics,\n * typically you include metrics_registration.hh, in A header file and add to A:\n * * metric_groups _metrics as a member\n * * set_metrics() method that would be called in the constructor.\n * \\code\n * class A {\n *   metric_groups _metrics\n *\n *   void setup_metrics();\n *\n * };\n * \\endcode\n * To define the metrics, include in your source file metircs.hh\n * @see metrics.hh for the definition for adding a metric.\n */\n\nnamespace seastar {\n\nnamespace metrics {\n\nnamespace impl {\nclass metric_groups_def;\nstruct metric_definition_impl;\nclass metric_groups_impl;\n}\n\n\nusing group_name_type = sstring; /*!< A group of logically related metrics */\nclass metric_groups;\n\nclass metric_definition {\n    std::unique_ptr<impl::metric_definition_impl> _impl;\npublic:\n    metric_definition(const impl::metric_definition_impl& impl) noexcept;\n    metric_definition(metric_definition&& m) noexcept;\n    ~metric_definition();\n    friend metric_groups;\n    friend impl::metric_groups_impl;\n};\n\nclass metric_group_definition {\npublic:\n    group_name_type name;\n    std::initializer_list<metric_definition> metrics;\n    metric_group_definition(const group_name_type& name, std::initializer_list<metric_definition> l);\n    metric_group_definition(const metric_group_definition&) = delete;\n    ~metric_group_definition();\n};\n\n/*!\n * metric_groups\n * \\brief holds the metric definition.\n *\n * Add multiple metric groups definitions.\n * Initialization can be done in the constructor or with a call to add_group\n * @see metrics.hh for example and supported metrics\n */\nclass metric_groups {\n    std::unique_ptr<impl::metric_groups_def> _impl;\npublic:\n    metric_groups() noexcept;\n    metric_groups(metric_groups&&) = default;\n    virtual ~metric_groups();\n    metric_groups& operator=(metric_groups&&) = default;\n    /*!\n     * \\brief add metrics belong to the same group in the constructor.\n     *\n     * combine the constructor with the add_group functionality.\n     */\n    metric_groups(std::initializer_list<metric_group_definition> mg);\n\n    /*!\n     * \\brief Add metrics belonging to the same group.\n     *\n     * Use the metrics creation functions to add metrics.\n     *\n     * For example:\n     *  _metrics.add_group(\"my_group\", {\n     *      make_counter(\"my_counter_name1\", counter, description(\"my counter description\")),\n     *      make_counter(\"my_counter_name2\", counter, description(\"my second counter description\")),\n     *      make_gauge(\"my_gauge_name1\", gauge, description(\"my gauge description\")),\n     *  });\n     *\n     * Metric name should be unique inside the group.\n     * You can chain add_group calls like:\n     * _metrics.add_group(\"my group1\", {...}).add_group(\"my group2\", {...});\n     *\n     * This overload (with initializer_list) is needed because metric_definition\n     * has no copy constructor, so the other overload (with vector) cannot be\n     * invoked on a braced-init-list.\n     */\n    metric_groups& add_group(const group_name_type& name, const std::initializer_list<metric_definition>& l);\n\n    /*!\n     * \\brief Add metrics belonging to the same group.\n     *\n     * Use the metrics creation functions to add metrics.\n     *\n     * For example:\n     *  vector<metric_definition> v;\n     *  v.push_back(make_counter(\"my_counter_name1\", counter, description(\"my counter description\")));\n     *  v.push_back(make_counter(\"my_counter_name2\", counter, description(\"my second counter description\")));\n     *  v.push_back(make_gauge(\"my_gauge_name1\", gauge, description(\"my gauge description\")));\n     *  _metrics.add_group(\"my_group\", v);\n     *\n     * Metric name should be unique inside the group.\n     * You can chain add_group calls like:\n     * _metrics.add_group(\"my group1\", vec1).add_group(\"my group2\", vec2);\n     */\n    metric_groups& add_group(const group_name_type& name, const std::vector<metric_definition>& l);\n\n    /*!\n     * \\brief clear all metrics groups registrations.\n     */\n    void clear();\n};\n\n\n/*!\n * \\brief hold a single metric group\n * Initialization is done in the constructor or\n * with a call to add_group\n */\nclass metric_group : public metric_groups {\npublic:\n    metric_group() noexcept;\n    metric_group(const metric_group&) = delete;\n    metric_group(metric_group&&) = default;\n    virtual ~metric_group();\n    metric_group& operator=(metric_group&&) = default;\n\n    /*!\n     * \\brief add metrics belong to the same group in the constructor.\n     *\n     *\n     */\n    metric_group(const group_name_type& name, std::initializer_list<metric_definition> l);\n};\n\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/metrics_types.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <cstdint>\n#include <vector>\n#include <optional>\n\nnamespace seastar {\nnamespace metrics {\n\n\n/*!\n * \\brief Histogram bucket type\n *\n * A histogram bucket contains an upper bound and the number\n * of events in the buckets.\n */\nstruct histogram_bucket {\n    uint64_t count = 0; // number of events.\n    double upper_bound = 0;      // Inclusive.\n};\n/*!\n * \\brief native histogram specific information\n *\n * Native histograms (also known as sparse histograms) are an experimental Prometheus feature.\n */\nstruct native_histogram_info {\n    // schema defines the bucket schema. Currently, valid numbers are -4 <= n <= 8.\n    // They are all for base-2 bucket schemas, where 1 is a bucket boundary in each case, and\n    // then each power of two is divided into 2^n logarithmic buckets.\n    // Or in other words, each bucket boundary is the previous boundary times 2^(2^-n).\n    // In the future, more bucket schemas may be added using numbers < -4 or > 8.\n    int32_t schema;\n    // min_id is the first bucket id of a given schema.\n    int32_t min_id;\n};\n\n/*!\n * \\brief Histogram data type\n *\n * The histogram struct is a container for histogram values.\n * It is not a histogram implementation but it will be used by histogram\n * implementation to return its internal values.\n */\nstruct histogram {\n    uint64_t sample_count = 0;\n    double sample_sum = 0;\n    std::vector<histogram_bucket> buckets; // Ordered in increasing order of upper_bound, +Inf bucket is optional.\n\n    /*!\n     * \\brief Addition assigning a historgram\n     *\n     * The histogram must match the buckets upper bounds\n     * or an exception will be thrown\n     */\n    histogram& operator+=(const histogram& h);\n\n    /*!\n     * \\brief Addition historgrams\n     *\n     * Add two histograms and return the result as a new histogram\n     * The histogram must match the buckets upper bounds\n     * or an exception will be thrown\n     */\n    histogram operator+(const histogram& h) const;\n\n    /*!\n     * \\brief Addition historgrams\n     *\n     * Add two histograms and return the result as a new histogram\n     * The histogram must match the buckets upper bounds\n     * or an exception will be thrown\n     */\n    histogram operator+(histogram&& h) const;\n\n    // Native histograms are an experimental Prometheus feature.\n    std::optional<native_histogram_info> native_histogram;\n};\n\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/on_internal_error.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <exception>\n#include <string_view>\n\nnamespace seastar {\n\n\nclass logger;\n\n/// Controls whether on_internal_error() aborts or throws. The default\n/// is to throw.\n/// \\returns the current abort_on_internal_error state.\nbool set_abort_on_internal_error(bool do_abort) noexcept;\n\n/// Report an internal error\n///\n/// Depending on the value passed to set_abort_on_internal_error, this\n/// will either abort or throw a std::runtime_error.\n/// In both cases an error will be logged with \\p logger, containing\n/// \\p reason and the current backtrace.\n[[noreturn]] void on_internal_error(logger& logger, std::string_view reason);\n\n/// Report an internal error\n///\n/// Depending on the value passed to set_abort_on_internal_error, this\n/// will either abort or throw the passed-in \\p ex.\n/// In both cases an error will be logged with \\p logger, containing\n/// \\p ex and the current backtrace.\n/// This overload cannot attach a backtrace to the exception, so if the\n/// caller wishes to have one attached they have to do it themselves.\n[[noreturn]] void on_internal_error(logger& logger, std::exception_ptr ex);\n\n/// Report an internal error in a noexcept context\n///\n/// The error will be logged to \\logger and if set_abort_on_internal_error,\n/// was set to true, the program will be aborted. This overload can be used\n/// in noexcept contexts like destructors or noexcept functions.\nvoid on_internal_error_noexcept(logger& logger, std::string_view reason) noexcept;\n\n/// Report an internal error and abort unconditionally\n///\n/// The error will be logged to \\logger and the program will be aborted,\n/// regardless of the abort_on_internal_error setting.\n/// This overload can be used to replace SEASTAR_ASSERT().\n[[noreturn]] void on_fatal_internal_error(logger& logger, std::string_view reason) noexcept;\n\nnamespace internal {\nextern thread_local uint64_t internal_errors;\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/pipe.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/queue.hh>\n\n#include <seastar/util/std-compat.hh>\n\n/// \\defgroup fiber-module Fibers\n///\n/// \\brief Fibers of execution\n///\n/// Seastar continuations are normally short, but often chained to one\n/// another, so that one continuation does a bit of work and then schedules\n/// another continuation for later. Such chains can be long, and often even\n/// involve loopings - see for example \\ref repeat. We call such chains\n/// \"fibers\" of execution.\n///\n/// These fibers are not threads - each is just a string of continuations -\n/// but they share some common requirements with traditional threads.\n/// For example, we want to avoid one fiber getting starved while a second\n/// fiber continuously runs its continuations one after another.\n/// As another example, fibers may want to communicate - e.g., one fiber\n/// produces data that a second fiber consumes, and we wish to ensure that\n/// both fibers get a chance to run, and that if one stops prematurely,\n/// the other doesn't hang forever.\n///\n/// Consult the following table to see which APIs are useful for fiber tasks:\n///\n/// Task                                           | APIs\n/// -----------------------------------------------|-------------------\n/// Repeat a blocking task indefinitely            | \\ref keep_doing()\n/// Repeat a blocking task, then exit              | \\ref repeat(), \\ref do_until()\n/// Provide mutual exclusion between two tasks     | \\ref semaphore, \\ref shared_mutex\n/// Pass a stream of data between two fibers       | \\ref seastar::pipe\n/// Safely shut down a resource                    | \\ref seastar::gate\n/// Hold on to an object while a fiber is running  | \\ref do_with()\n///\n\n/// Seastar API namespace\nnamespace seastar {\n\n\n\n/// \\addtogroup fiber-module\n/// @{\nclass broken_pipe_exception : public std::exception {\npublic:\n    virtual const char* what() const noexcept {\n        return \"Broken pipe\";\n    }\n};\n\nclass unread_overflow_exception : public std::exception {\npublic:\n    virtual const char* what() const noexcept {\n        return \"pipe_reader::unread() overflow\";\n    }\n};\n\n/// \\cond internal\nnamespace internal {\ntemplate <typename T>\nclass pipe_buffer {\nprivate:\n    queue<std::optional<T>> _buf;\n    bool _read_open = true;\n    bool _write_open = true;\npublic:\n    pipe_buffer(size_t size) : _buf(size) {}\n    future<std::optional<T>> read() {\n        return _buf.pop_eventually();\n    }\n    future<> write(T&& data) {\n        return _buf.push_eventually(std::move(data));\n    }\n    bool readable() const {\n        return _write_open || !_buf.empty();\n    }\n    bool writeable() const {\n        return _read_open;\n    }\n    bool close_read() {\n        // If a writer blocking (on a full queue), need to stop it.\n        if (_buf.full()) {\n            _buf.abort(std::make_exception_ptr(broken_pipe_exception()));\n        }\n        _read_open = false;\n        return !_write_open;\n    }\n    bool close_write() {\n        // If the queue is empty, write the EOF (disengaged optional) to the\n        // queue to wake a blocked reader. If the queue is not empty, there is\n        // no need to write the EOF to the queue - the reader will return an\n        // EOF when it sees that _write_open == false.\n        if (_buf.empty()) {\n            _buf.push({});\n        }\n        _write_open = false;\n        return !_read_open;\n    }\n};\n} // namespace internal\n/// \\endcond\n\ntemplate <typename T>\nclass pipe;\n\n/// \\brief Read side of a \\ref seastar::pipe\n///\n/// The read side of a pipe, which allows only reading from the pipe.\n/// A pipe_reader object cannot be created separately, but only as part of a\n/// reader/writer pair through \\ref seastar::pipe.\ntemplate <typename T>\nclass pipe_reader {\nprivate:\n    internal::pipe_buffer<T> *_bufp;\n    std::optional<T> _unread;\n    pipe_reader(internal::pipe_buffer<T> *bufp) noexcept : _bufp(bufp) { }\n    friend class pipe<T>;\npublic:\n    /// \\brief Read next item from the pipe\n    ///\n    /// Returns a future value, which is fulfilled when the pipe's buffer\n    /// becomes non-empty, or the write side is closed. The value returned\n    /// is an optional<T>, which is disengaged to mark and end of file\n    /// (i.e., the write side was closed, and we've read everything it sent).\n    future<std::optional<T>> read() {\n        if (_unread) {\n            auto ret = std::move(*_unread);\n            _unread = {};\n            return make_ready_future<std::optional<T>>(std::move(ret));\n        }\n        if (_bufp->readable()) {\n            return _bufp->read();\n        } else {\n            return make_ready_future<std::optional<T>>();\n        }\n    }\n    /// \\brief Return an item to the front of the pipe\n    ///\n    /// Pushes the given item to the front of the pipe, so it will be\n    /// returned by the next read() call. The typical use case is to\n    /// unread() the last item returned by read().\n    /// More generally, it is legal to unread() any item, not just one\n    /// previously returned by read(), but note that the unread() is limited\n    /// to just one item - two calls to unread() without an intervening call\n    /// to read() will cause an exception.\n    void unread(T&& item) {\n        if (_unread) {\n            throw unread_overflow_exception();\n        }\n        _unread = std::move(item);\n    }\n    ~pipe_reader() {\n        if (_bufp && _bufp->close_read()) {\n            delete _bufp;\n        }\n    }\n    // Allow move, but not copy, of pipe_reader\n    pipe_reader(pipe_reader&& other) noexcept : _bufp(other._bufp) {\n        other._bufp = nullptr;\n    }\n    pipe_reader& operator=(pipe_reader&& other) noexcept {\n        std::swap(_bufp, other._bufp);\n        return *this;\n    }\n};\n\n/// \\brief Write side of a \\ref seastar::pipe\n///\n/// The write side of a pipe, which allows only writing to the pipe.\n/// A pipe_writer object cannot be created separately, but only as part of a\n/// reader/writer pair through \\ref seastar::pipe.\ntemplate <typename T>\nclass pipe_writer {\nprivate:\n    internal::pipe_buffer<T> *_bufp;\n    pipe_writer(internal::pipe_buffer<T> *bufp) noexcept : _bufp(bufp) { }\n    friend class pipe<T>;\npublic:\n    /// \\brief Write an item to the pipe\n    ///\n    /// Returns a future value, which is fulfilled when the data was written\n    /// to the buffer (when it become non-full). If the data could not be\n    /// written because the read side was closed, an exception\n    /// \\ref broken_pipe_exception is returned in the future.\n    future<> write(T&& data) {\n        if (_bufp->writeable()) {\n            return _bufp->write(std::move(data));\n        } else {\n            return make_exception_future<>(broken_pipe_exception());\n        }\n    }\n    ~pipe_writer() {\n        if (_bufp && _bufp->close_write()) {\n            delete _bufp;\n        }\n    }\n    // Allow move, but not copy, of pipe_writer\n    pipe_writer(pipe_writer&& other) noexcept : _bufp(other._bufp) {\n        other._bufp = nullptr;\n    }\n    pipe_writer& operator=(pipe_writer&& other) noexcept {\n        std::swap(_bufp, other._bufp);\n        return *this;\n    }\n};\n\n/// \\brief A fixed-size pipe for communicating between two fibers.\n///\n/// A pipe<T> is a mechanism to transfer data between two fibers, one\n/// producing data, and the other consuming it. The fixed-size buffer also\n/// ensures a balanced execution of the two fibers, because the producer\n/// fiber blocks when it writes to a full pipe, until the consumer fiber gets\n/// to run and read from the pipe.\n///\n/// A pipe<T> resembles a Unix pipe, in that it has a read side, a write side,\n/// and a fixed-sized buffer between them, and supports either end to be closed\n/// independently (and EOF or broken pipe when using the other side).\n/// A pipe<T> object holds the reader and write sides of the pipe as two\n/// separate objects. These objects can be moved into two different fibers.\n/// Importantly, if one of the pipe ends is destroyed (i.e., the continuations\n/// capturing it end), the other end of the pipe will stop blocking, so the\n/// other fiber will not hang.\n///\n/// The pipe's read and write interfaces are future-based blocking. I.e., the\n/// write() and read() methods return a future which is fulfilled when the\n/// operation is complete. The pipe is single-reader single-writer, meaning\n/// that until the future returned by read() is fulfilled, read() must not be\n/// called again (and same for write).\n///\n/// Note: The pipe reader and writer are movable, but *not* copyable. It is\n/// often convenient to wrap each end in a shared pointer, so it can be\n/// copied (e.g., used in an std::function which needs to be copyable) or\n/// easily captured into multiple continuations.\ntemplate <typename T>\nclass pipe {\npublic:\n    pipe_reader<T> reader;\n    pipe_writer<T> writer;\n    explicit pipe(size_t size) : pipe(new internal::pipe_buffer<T>(size)) { }\nprivate:\n    pipe(internal::pipe_buffer<T> *bufp) noexcept : reader(bufp), writer(bufp) { }\n};\n\n\n/// @}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/polymorphic_temporary_buffer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Elazar Leibovich\n */\n\n#pragma once\n\n#include <seastar/core/memory.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\n/// Creates a `temporary_buffer` allocated by a custom allocator\n///\n/// \\param allocator allocator to use when allocating the temporary_buffer\n/// \\param size      size of the temporary buffer\ntemplate <typename CharType>\ntemporary_buffer<CharType> make_temporary_buffer(std::pmr::polymorphic_allocator<CharType>* allocator, std::size_t size) {\n    if (allocator == memory::malloc_allocator) {\n        return temporary_buffer<CharType>(size);\n    }\n    CharType *buffer = allocator->allocate(size);\n    return temporary_buffer<CharType>(buffer, size,\n        make_deleter(deleter(), [allocator, buffer, size] () mutable { allocator->deallocate(buffer, size); }));\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/posix.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <sys/epoll.h>\n#include <sys/eventfd.h>\n#include <sys/ioctl.h>\n#include <sys/mman.h>\n#include <sys/socket.h>\n#include <sys/stat.h>\n#include <sys/timerfd.h>\n#include <sys/types.h>\n#include <sys/uio.h>\n#include <assert.h>\n#include <fcntl.h>\n#include <pthread.h>\n#include <signal.h>\n#include <spawn.h>\n#include <unistd.h>\n#include <utility>\n#include <system_error>\n#include <chrono>\n#include <cstring>\n#include <functional>\n#include <memory>\n#include <set>\n#include <optional>\n#include \"abort_on_ebadf.hh\"\n#include <seastar/core/sstring.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\n\n/// \\file\n/// \\defgroup posix-support POSIX Support\n///\n/// Mostly-internal APIs to provide C++ glue for the underlying POSIX platform;\n/// but can be used by the application when they don't block.\n///\n/// \\addtogroup posix-support\n/// @{\n\ninline void throw_system_error_on(bool condition, const char* what_arg = \"\");\n\ntemplate <typename T>\nrequires std::signed_integral<T>\ninline void throw_kernel_error(T r);\n\ntemplate <typename T>\ninline void throw_pthread_error(T r);\n\nstruct mmap_deleter {\n    size_t _size;\n    void operator()(void* ptr) const;\n};\n\nusing mmap_area = std::unique_ptr<char[], mmap_deleter>;\n\nmmap_area mmap_anonymous(void* addr, size_t length, int prot, int flags);\n\nclass file_desc {\n    int _fd;\npublic:\n    file_desc() = delete;\n    file_desc(const file_desc&) = delete;\n    file_desc(file_desc&& x) noexcept : _fd(x._fd) { x._fd = -1; }\n    ~file_desc() { if (_fd != -1) { ::close(_fd); } }\n    void operator=(const file_desc&) = delete;\n    file_desc& operator=(file_desc&& x) {\n        if (this != &x) {\n            std::swap(_fd, x._fd);\n            if (x._fd != -1) {\n                x.close();\n            }\n        }\n        return *this;\n    }\n    void close() {\n        SEASTAR_ASSERT(_fd != -1);\n        auto r = ::close(_fd);\n        throw_system_error_on(r == -1, \"close\");\n        _fd = -1;\n    }\n    int get() const { return _fd; }\n\n    sstring fdinfo() const noexcept;\n\n    static file_desc from_fd(int fd) {\n        return file_desc(fd);\n    }\n\n    static file_desc open(sstring name, int flags, mode_t mode = 0) {\n        int fd = ::open(name.c_str(), flags, mode);\n        throw_system_error_on(fd == -1, \"open\");\n        return file_desc(fd);\n    }\n    static file_desc socket(int family, int type, int protocol = 0) {\n        int fd = ::socket(family, type, protocol);\n        throw_system_error_on(fd == -1, \"socket\");\n        return file_desc(fd);\n    }\n    static file_desc eventfd(unsigned initval, int flags) {\n        int fd = ::eventfd(initval, flags);\n        throw_system_error_on(fd == -1, \"eventfd\");\n        return file_desc(fd);\n    }\n    static file_desc epoll_create(int flags = 0) {\n        int fd = ::epoll_create1(flags);\n        throw_system_error_on(fd == -1, \"epoll_create1\");\n        return file_desc(fd);\n    }\n    static file_desc timerfd_create(int clockid, int flags) {\n        int fd = ::timerfd_create(clockid, flags);\n        throw_system_error_on(fd == -1, \"timerfd_create\");\n        return file_desc(fd);\n    }\n    static file_desc temporary(sstring directory);\n    file_desc dup() const {\n        int fd = ::dup(get());\n        throw_system_error_on(fd == -1, \"dup\");\n        return file_desc(fd);\n    }\n    file_desc accept(socket_address& sa, int flags = 0) {\n        auto ret = ::accept4(_fd, &sa.as_posix_sockaddr(), &sa.addr_length, flags);\n        throw_system_error_on(ret == -1, \"accept4\");\n        return file_desc(ret);\n    }\n    static file_desc inotify_init(int flags);\n    // return nullopt if no connection is availbale to be accepted\n    std::optional<file_desc> try_accept(socket_address& sa, int flags = 0) {\n        auto ret = ::accept4(_fd, &sa.as_posix_sockaddr(), &sa.addr_length, flags);\n        if (ret == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(ret == -1, \"accept4\");\n        return file_desc(ret);\n    }\n    void shutdown(int how) {\n        auto ret = ::shutdown(_fd, how);\n        if (ret == -1 && errno != ENOTCONN) {\n            throw_system_error_on(ret == -1, \"shutdown\");\n        }\n    }\n    void truncate(size_t size) {\n        auto ret = ::ftruncate(_fd, size);\n        throw_system_error_on(ret, \"ftruncate\");\n    }\n    int ioctl(int request) {\n        return ioctl(request, 0);\n    }\n    int ioctl(int request, int value) {\n        int r = ::ioctl(_fd, request, value);\n        throw_system_error_on(r == -1, \"ioctl\");\n        return r;\n    }\n    int ioctl(int request, unsigned int value) {\n        int r = ::ioctl(_fd, request, value);\n        throw_system_error_on(r == -1, \"ioctl\");\n        return r;\n    }\n    template <class X>\n    int ioctl(int request, X& data) {\n        int r = ::ioctl(_fd, request, &data);\n        throw_system_error_on(r == -1, \"ioctl\");\n        return r;\n    }\n    template <class X>\n    int ioctl(int request, X&& data) {\n        int r = ::ioctl(_fd, request, &data);\n        throw_system_error_on(r == -1, \"ioctl\");\n        return r;\n    }\n    template <class X>\n    int setsockopt(int level, int optname, X&& data) {\n        int r = ::setsockopt(_fd, level, optname, &data, sizeof(data));\n        throw_system_error_on(r == -1, \"setsockopt\");\n        return r;\n    }\n    int setsockopt(int level, int optname, const char* data) {\n        int r = ::setsockopt(_fd, level, optname, data, strlen(data) + 1);\n        throw_system_error_on(r == -1, \"setsockopt\");\n        return r;\n    }\n    int setsockopt(int level, int optname, const void* data, socklen_t len) {\n        int r = ::setsockopt(_fd, level, optname, data, len);\n        throw_system_error_on(r == -1, \"setsockopt\");\n        return r;\n    }\n    template <typename Data>\n    Data getsockopt(int level, int optname) {\n        Data data;\n        socklen_t len = sizeof(data);\n        memset(&data, 0, len);\n        int r = ::getsockopt(_fd, level, optname, &data, &len);\n        throw_system_error_on(r == -1, \"getsockopt\");\n        return data;\n    }\n    int getsockopt(int level, int optname, char* data, socklen_t len) {\n        int r = ::getsockopt(_fd, level, optname, data, &len);\n        throw_system_error_on(r == -1, \"getsockopt\");\n        return r;\n    }\n    size_t size() {\n        struct stat buf;\n        auto r = ::fstat(_fd, &buf);\n        throw_system_error_on(r == -1, \"fstat\");\n        return buf.st_size;\n    }\n    std::optional<size_t> read(void* buffer, size_t len) {\n        auto r = ::read(_fd, buffer, len);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"read\");\n        return { size_t(r) };\n    }\n    std::optional<ssize_t> recv(void* buffer, size_t len, int flags) {\n        auto r = ::recv(_fd, buffer, len, flags);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"recv\");\n        return { ssize_t(r) };\n    }\n    std::optional<size_t> recvmsg(msghdr* mh, int flags) {\n        auto r = ::recvmsg(_fd, mh, flags);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"recvmsg\");\n        return { size_t(r) };\n    }\n    std::optional<size_t> send(const void* buffer, size_t len, int flags) {\n        auto r = ::send(_fd, buffer, len, flags);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"send\");\n        return { size_t(r) };\n    }\n    std::optional<size_t> sendto(socket_address& addr, const void* buf, size_t len, int flags) {\n        auto r = ::sendto(_fd, buf, len, flags, &addr.u.sa, addr.length());\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"sendto\");\n        return { size_t(r) };\n    }\n    std::optional<size_t> sendmsg(const msghdr* msg, int flags) {\n        auto r = ::sendmsg(_fd, msg, flags);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"sendmsg\");\n        return { size_t(r) };\n    }\n    void bind(sockaddr& sa, socklen_t sl) {\n        auto r = ::bind(_fd, &sa, sl);\n        throw_system_error_on(r == -1, \"bind\");\n    }\n    void connect(sockaddr& sa, socklen_t sl) {\n        auto r = ::connect(_fd, &sa, sl);\n        if (r == -1 && errno == EINPROGRESS) {\n            return;\n        }\n        throw_system_error_on(r == -1, \"connect\");\n    }\n    socket_address get_address() {\n        socket_address addr;\n        auto r = ::getsockname(_fd, &addr.u.sa, &addr.addr_length);\n        throw_system_error_on(r == -1, \"getsockname\");\n        return addr;\n    }\n    socket_address get_remote_address() {\n        socket_address addr;\n        auto r = ::getpeername(_fd, &addr.u.sa, &addr.addr_length);\n        throw_system_error_on(r == -1, \"getpeername\");\n        return addr;\n    }\n    void listen(int backlog) {\n        auto fd = ::listen(_fd, backlog);\n        throw_system_error_on(fd == -1, \"listen\");\n    }\n    std::optional<size_t> write(const void* buf, size_t len) {\n        auto r = ::write(_fd, buf, len);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"write\");\n        return { size_t(r) };\n    }\n    std::optional<size_t> writev(const iovec *iov, int iovcnt) {\n        auto r = ::writev(_fd, iov, iovcnt);\n        if (r == -1 && errno == EAGAIN) {\n            return {};\n        }\n        throw_system_error_on(r == -1, \"writev\");\n        return { size_t(r) };\n    }\n    size_t pread(void* buf, size_t len, off_t off) {\n        auto r = ::pread(_fd, buf, len, off);\n        throw_system_error_on(r == -1, \"pread\");\n        return size_t(r);\n    }\n    void timerfd_settime(int flags, const itimerspec& its) {\n        auto r = ::timerfd_settime(_fd, flags, &its, NULL);\n        throw_system_error_on(r == -1, \"timerfd_settime\");\n    }\n\n    mmap_area map(size_t size, unsigned prot, unsigned flags, size_t offset,\n            void* addr = nullptr) {\n        void *x = mmap(addr, size, prot, flags, _fd, offset);\n        throw_system_error_on(x == MAP_FAILED, \"mmap\");\n        return mmap_area(static_cast<char*>(x), mmap_deleter{size});\n    }\n\n    mmap_area map_shared_rw(size_t size, size_t offset) {\n        return map(size, PROT_READ | PROT_WRITE, MAP_SHARED, offset);\n    }\n\n    mmap_area map_shared_ro(size_t size, size_t offset) {\n        return map(size, PROT_READ, MAP_SHARED, offset);\n    }\n\n    mmap_area map_private_rw(size_t size, size_t offset) {\n        return map(size, PROT_READ | PROT_WRITE, MAP_PRIVATE, offset);\n    }\n\n    mmap_area map_private_ro(size_t size, size_t offset) {\n        return map(size, PROT_READ, MAP_PRIVATE, offset);\n    }\n\n    void spawn_actions_add_close(posix_spawn_file_actions_t* actions) {\n        auto r = ::posix_spawn_file_actions_addclose(actions, _fd);\n        throw_pthread_error(r);\n    }\n\n    void spawn_actions_add_dup2(posix_spawn_file_actions_t* actions, int newfd) {\n        auto r = ::posix_spawn_file_actions_adddup2(actions, _fd, newfd);\n        throw_pthread_error(r);\n    }\nprivate:\n    file_desc(int fd) : _fd(fd) {}\n };\n\n\nnamespace posix {\n\nconstexpr unsigned rcv_shutdown = 0x1;\nconstexpr unsigned snd_shutdown = 0x2;\ninline constexpr unsigned shutdown_mask(int how) { return how + 1; }\n\n/// Converts a duration value to a `timespec`\n///\n/// \\param d a duration value to convert to the POSIX `timespec` format\n/// \\return `d` as a `timespec` value\ntemplate <typename Rep, typename Period>\nstruct timespec\nto_timespec(std::chrono::duration<Rep, Period> d) {\n    auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(d).count();\n    struct timespec ts {};\n    ts.tv_sec = ns / 1000000000;\n    ts.tv_nsec = ns % 1000000000;\n    return ts;\n}\n\n/// Converts a relative start time and an interval to an `itimerspec`\n///\n/// \\param base First expiration of the timer, relative to the current time\n/// \\param interval period for re-arming the timer\n/// \\return `base` and `interval` converted to an `itimerspec`\ntemplate <typename Rep1, typename Period1, typename Rep2, typename Period2>\nstruct itimerspec\nto_relative_itimerspec(std::chrono::duration<Rep1, Period1> base, std::chrono::duration<Rep2, Period2> interval) {\n    struct itimerspec its {};\n    its.it_interval = to_timespec(interval);\n    its.it_value = to_timespec(base);\n    return its;\n}\n\n\n/// Converts a time_point and a duration to an `itimerspec`\n///\n/// \\param base  base time for the timer; must use the same clock as the timer\n/// \\param interval period for re-arming the timer\n/// \\return `base` and `interval` converted to an `itimerspec`\ntemplate <typename Clock, class Duration, class Rep, class Period>\nstruct itimerspec\nto_absolute_itimerspec(std::chrono::time_point<Clock, Duration> base, std::chrono::duration<Rep, Period> interval) {\n    return to_relative_itimerspec(base.time_since_epoch(), interval);\n}\n\n}\n\nclass posix_thread {\npublic:\n    class attr;\nprivate:\n    // must allocate, since this class is moveable\n    std::unique_ptr<std::function<void ()>> _func;\n    pthread_t _pthread;\n    bool _valid = true;\n    mmap_area _stack;\nprivate:\n    static void* start_routine(void* arg) noexcept;\npublic:\n    posix_thread(std::function<void ()> func);\n    posix_thread(attr a, std::function<void ()> func);\n    posix_thread(posix_thread&& x);\n    ~posix_thread();\n    void join();\npublic:\n    class attr {\n    public:\n        struct stack_size { size_t size = 0; };\n        attr() = default;\n        template <typename... A>\n        attr(A... a) {\n            set(std::forward<A>(a)...);\n        }\n        void set() {}\n        template <typename A, typename... Rest>\n        void set(A a, Rest... rest) {\n            set(std::forward<A>(a));\n            set(std::forward<Rest>(rest)...);\n        }\n        void set(stack_size ss) { _stack_size = ss; }\n        void set(cpu_set_t affinity) { _affinity = affinity; }\n    private:\n        stack_size _stack_size;\n        std::optional<cpu_set_t> _affinity;\n        friend class posix_thread;\n    };\n};\n\n\ninline\nvoid throw_system_error_on(bool condition, const char* what_arg) {\n    if (condition) {\n        if ((errno == EBADF || errno == ENOTSOCK) && is_abort_on_ebadf_enabled()) {\n            abort();\n        }\n        throw std::system_error(errno, std::system_category(), what_arg);\n    }\n}\n\ntemplate <typename T>\nrequires std::signed_integral<T>\ninline\nvoid throw_kernel_error(T r) {\n    if (r < 0) {\n        auto ec = -r;\n        if ((ec == EBADF || ec == ENOTSOCK) && is_abort_on_ebadf_enabled()) {\n            abort();\n        }\n        throw std::system_error(-r, std::system_category());\n    }\n}\n\ntemplate <typename T>\ninline\nvoid throw_pthread_error(T r) {\n    if (r != 0) {\n        throw std::system_error(r, std::system_category());\n    }\n}\n\ninline\nsigset_t make_sigset_mask(int signo) {\n    sigset_t set;\n    sigemptyset(&set);\n    sigaddset(&set, signo);\n    return set;\n}\n\ninline\nsigset_t make_full_sigset_mask() {\n    sigset_t set;\n    sigfillset(&set);\n    return set;\n}\n\ninline\nsigset_t make_empty_sigset_mask() {\n    sigset_t set;\n    sigemptyset(&set);\n    return set;\n}\n\ninline\nvoid pin_this_thread(unsigned cpu_id) {\n    cpu_set_t cs;\n    CPU_ZERO(&cs);\n    CPU_SET(cpu_id, &cs);\n    auto r = pthread_setaffinity_np(pthread_self(), sizeof(cs), &cs);\n    SEASTAR_ASSERT(r == 0);\n    (void)r;\n}\n\nstd::set<unsigned> get_current_cpuset();\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/preempt.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n#include <atomic>\n\nnamespace seastar {\n\nnamespace internal {\n\nstruct preemption_monitor {\n    // We preempt when head != tail\n    // This happens to match the Linux aio completion ring, so we can have the\n    // kernel preempt a task by queuing a completion event to an io_context.\n    std::atomic<uint32_t> head;\n    std::atomic<uint32_t> tail;\n};\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nconst preemption_monitor*& get_need_preempt_var();\n#else\ninline const preemption_monitor*& get_need_preempt_var() {\n    static preemption_monitor bootstrap_preemption_monitor;\n    static thread_local const preemption_monitor* g_need_preempt = &bootstrap_preemption_monitor;\n    return g_need_preempt;\n}\n#endif\n\nvoid set_need_preempt_var(const preemption_monitor* pm);\n\ninline\nbool\nmonitor_need_preempt() noexcept {\n    // prevent compiler from eliminating loads in a loop\n    std::atomic_signal_fence(std::memory_order_seq_cst);\n    auto np = internal::get_need_preempt_var();\n    // We aren't reading anything from the ring, so we don't need\n    // any barriers.\n    auto head = np->head.load(std::memory_order_relaxed);\n    auto tail = np->tail.load(std::memory_order_relaxed);\n    // Possible optimization: read head and tail in a single 64-bit load,\n    // and find a funky way to compare the two 32-bit halves.\n    return __builtin_expect(head != tail, false);\n}\n\n}\n\ninline bool need_preempt() noexcept {\n#ifndef SEASTAR_DEBUG\n    return internal::monitor_need_preempt();\n#else\n    return true;\n#endif\n}\n\nnamespace internal {\n\n\n\n// Same as need_preempt(), but for the scheduler's use. Outside debug\n// mode they have the same meaning - the task quota expired and we need\n// to check for I/O.\ninline\nbool\nscheduler_need_preempt() {\n#ifndef SEASTAR_DEBUG\n    return need_preempt();\n#else\n    // Within the scheduler, preempting all the time (as need_preempt()\n    // does in debug mode) reduces performance drastically since we check\n    // for I/O after every task. Since we don't care about latency in debug\n    // mode, run some random-but-bounded number of tasks instead. Latency\n    // will be high if those tasks are slow, but this is debug mode anyway.\n    // We still check if preemption was requested to allow lowres_clock\n    // updates.\n    static thread_local unsigned counter = 0;\n    return ++counter % 64 == 0 || monitor_need_preempt();;\n#endif\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/prefetch.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <algorithm>\n#include <atomic>\n#include <utility>\n#include <functional>\n#include <seastar/core/align.hh>\n#include <seastar/core/cacheline.hh>\n\nnamespace seastar {\n\n\ntemplate <size_t N, int RW, int LOC>\nstruct prefetcher;\n\ntemplate<int RW, int LOC>\nstruct prefetcher<0, RW, LOC> {\n    prefetcher(uintptr_t ptr) {}\n};\n\ntemplate <size_t N, int RW, int LOC>\nstruct prefetcher {\n    prefetcher(uintptr_t ptr) {\n        __builtin_prefetch(reinterpret_cast<void*>(ptr), RW, LOC);\n        std::atomic_signal_fence(std::memory_order_seq_cst);\n        prefetcher<N-cache_line_size, RW, LOC>(ptr + cache_line_size);\n    }\n};\n\n// LOC is a locality from __buitin_prefetch() gcc documentation:\n// \"The value locality must be a compile-time constant integer between zero and three. A value of\n//  zero means that the data has no temporal locality, so it need not be left in the cache after\n//  the access. A value of three means that the data has a high degree of temporal locality and\n//  should be left in all levels of cache possible. Values of one and two mean, respectively, a\n//  low or moderate degree of temporal locality. The default is three.\"\ntemplate<typename T, int LOC = 3>\nvoid prefetch(T* ptr) {\n    prefetcher<align_up(sizeof(T), cache_line_size), 0, LOC>(reinterpret_cast<uintptr_t>(ptr));\n}\n\ntemplate<typename Iterator, int LOC = 3>\nvoid prefetch(Iterator begin, Iterator end) {\n    std::for_each(begin, end, [] (auto v) { prefetch<decltype(*v), LOC>(v); });\n}\n\ntemplate<size_t C, typename T, int LOC = 3>\nvoid prefetch_n(T** pptr) {\n    std::invoke([&] <size_t... x> (std::index_sequence<x...>) {\n        (..., prefetch<T, LOC>(*(pptr + x)));\n    }, std::make_index_sequence<C>{});\n}\n\ntemplate<size_t L, int LOC = 3>\nvoid prefetch(void* ptr) {\n    prefetcher<L*cache_line_size, 0, LOC>(reinterpret_cast<uintptr_t>(ptr));\n}\n\ntemplate<size_t L, typename Iterator, int LOC = 3>\nvoid prefetch_n(Iterator begin, Iterator end) {\n    std::for_each(begin, end, [] (auto v) { prefetch<L, LOC>(v); });\n}\n\ntemplate<size_t L, size_t C, typename T, int LOC = 3>\nvoid prefetch_n(T** pptr) {\n    std::invoke([&] <size_t... x> (std::index_sequence<x...>) {\n        (..., prefetch<L, LOC>(*(pptr + x)));\n    }, std::make_index_sequence<C>{});\n}\n\ntemplate<typename T, int LOC = 3>\nvoid prefetchw(T* ptr) {\n    prefetcher<align_up(sizeof(T), cache_line_size), 1, LOC>(reinterpret_cast<uintptr_t>(ptr));\n}\n\ntemplate<typename Iterator, int LOC = 3>\nvoid prefetchw_n(Iterator begin, Iterator end) {\n    std::for_each(begin, end, [] (auto v) { prefetchw<decltype(*v), LOC>(v); });\n}\n\ntemplate<size_t C, typename T, int LOC = 3>\nvoid prefetchw_n(T** pptr) {\n    std::invoke([&] <size_t... x> (std::index_sequence<x...>) {\n        (..., prefetchw<T, LOC>(*(pptr + x)));\n    }, std::make_index_sequence<C>{});\n}\n\ntemplate<size_t L, int LOC = 3>\nvoid prefetchw(void* ptr) {\n    prefetcher<L*cache_line_size, 1, LOC>(reinterpret_cast<uintptr_t>(ptr));\n}\n\ntemplate<size_t L, typename Iterator, int LOC = 3>\nvoid prefetchw_n(Iterator begin, Iterator end) {\n   std::for_each(begin, end, [] (auto v) { prefetchw<L, LOC>(v); });\n}\n\ntemplate<size_t L, size_t C, typename T, int LOC = 3>\nvoid prefetchw_n(T** pptr) {\n    std::invoke([&] <size_t... x> (std::index_sequence<x...>) {\n        (..., prefetchw<L, LOC>(*(pptr + x)));\n    }, std::make_index_sequence<C>{});\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/print.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/format.hh>\n#include <seastar/core/sstring.hh>\n#include <fmt/ostream.h>\n#include <iostream>\n#include <iomanip>\n#include <chrono>\n\ninline\nstd::ostream&\noperator<<(std::ostream&& os, const void* ptr) {\n    return os << ptr; // selects non-rvalue version\n}\n\nnamespace seastar {\ntemplate <typename Iterator>\n[[deprecated(\"use fmt::join()\")]]\nstd::string\nformat_separated(Iterator b, Iterator e, const char* sep = \", \") {\n    std::string ret;\n    if (b == e) {\n        return ret;\n    }\n    ret += *b++;\n    while (b != e) {\n        ret += sep;\n        ret += *b++;\n    }\n    return ret;\n}\n\ntemplate <typename TimePoint>\nstruct usecfmt_wrapper {\n    TimePoint val;\n};\n\ntemplate <typename TimePoint>\ninline\nusecfmt_wrapper<TimePoint>\nusecfmt(TimePoint tp) {\n    return { tp };\n};\n\ntemplate <typename Clock, typename Rep, typename Period>\nstd::ostream&\noperator<<(std::ostream& os, usecfmt_wrapper<std::chrono::time_point<Clock, std::chrono::duration<Rep, Period>>> tp) {\n    auto usec = std::chrono::duration_cast<std::chrono::microseconds>(tp.val.time_since_epoch()).count();\n    std::ostream tmp(os.rdbuf());\n    tmp << std::setw(12) << (usec / 1000000) << \".\" << std::setw(6) << std::setfill('0') << (usec % 1000000);\n    return os;\n}\n\ntemplate <typename... A>\nvoid\nlog(A&&... a) {\n    std::cout << usecfmt(std::chrono::high_resolution_clock::now()) << \" \";\n    print(std::forward<A>(a)...);\n}\n\n\n// temporary, use fmt::print() instead\ntemplate <typename... A>\n[[deprecated(\"use std::format() or fmt::print()\")]]\nstd::ostream&\nfmt_print(std::ostream& os, const char* format, A&&... a) {\n    fmt::print(os, format, std::forward<A>(a)...);\n    return os;\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/prometheus.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/http/httpd.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/util/std-compat.hh>\n#include <optional>\n\nstruct prometheus_test_fixture;\n\nnamespace seastar {\n\nnamespace prometheus {\n\n\n/*!\n * Holds prometheus related configuration\n */\nstruct config {\n\n    [[deprecated(\"metric_help is deprecated and no longer used, to be removed in 2027\")]]\n    sstring metric_help;\n\n    [[deprecated(\"hostname is deprecated and unused, use label instead, to be removed in 2027\")]]\n    sstring hostname;\n\n    SEASTAR_INTERNAL_BEGIN_IGNORE_DEPRECATIONS // prevent warnings about deprecated fields in implicitly-defined special member functions\n    config() = default;\n    config(const config&) = default;\n    config(config&&) = default;\n    ~config() = default;\n    SEASTAR_INTERNAL_END_IGNORE_DEPRECATIONS\n\n    std::optional<metrics::label_instance> label; //!< A label that will be added to all metrics, we advice not to use it and set it on the prometheus server\n    sstring prefix = \"seastar\"; //!< a prefix that will be added to metric names\n    bool allow_protobuf = false; // protobuf support is experimental and off by default\n};\n\nfuture<> start(httpd::http_server_control& http_server, config ctx);\n\n/// \\defgroup add_prometheus_routes adds a /metrics endpoint that returns prometheus metrics\n///    both in txt format and in protobuf according to the prometheus spec\n/// @{\nfuture<> add_prometheus_routes(sharded<httpd::http_server>& server, config ctx);\nfuture<> add_prometheus_routes(httpd::http_server& server, config ctx);\n/// @}\n\nnamespace details {\nusing filter_t = std::function<bool(const metrics::impl::labels_type&)>;\nusing family_filter_t = std::function<bool(std::string_view)>;\n\nstruct name_filter {\n    sstring name;\n    bool is_prefix;\n};\n\n// Creates a family filter from multiple name filters.\n// Returns true if the family name matches any of the filters.\n// If filters is empty, returns a filter that matches all families.\n// If prefix is provided, filter names starting with \"{prefix}_\" will have the prefix stripped,\n// allowing users to query with either \"foo\" or \"seastar_foo\" when prefix is \"seastar\".\nfamily_filter_t make_family_filter(std::vector<name_filter> filters, std::string_view prefix = \"\");\n\nstruct write_body_args {\n    filter_t filter;\n    family_filter_t family_filter;\n    bool use_protobuf_format;\n    bool show_help;\n    bool enable_aggregation;\n};\n\nclass test_access {\n    future<> write_body(config cfg, write_body_args args, output_stream<char>&& s);\n\n    friend struct metrics_perf_fixture;\n    friend struct ::prometheus_test_fixture;\n};\n}\n}\n}\n"
  },
  {
    "path": "include/seastar/core/queue.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/future.hh>\n#include <seastar/util/assert.hh>\n#include <optional>\n#include <queue>\n\nnamespace seastar {\n\n/// Asynchronous single-producer single-consumer queue with limited capacity.\n/// There can be at most one producer-side and at most one consumer-side operation active at any time.\n/// Operations returning a future are considered to be active until the future resolves.\n///\n/// Note: queue requires the data type T to be nothrow move constructible as it's\n/// returned as future<T> by \\ref pop_eventually and seastar futurized data type\n/// are required to be nothrow move-constructible.\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\nclass queue {\n    std::queue<T, circular_buffer<T>> _q;\n    size_t _max;\n    std::optional<promise<>> _not_empty;\n    std::optional<promise<>> _not_full;\n    std::exception_ptr _ex = nullptr;\nprivate:\n    void notify_not_empty() noexcept;\n    void notify_not_full() noexcept;\npublic:\n    explicit queue(size_t size);\n\n    /// \\brief Push an item.\n    ///\n    /// Returns false if the queue was full and the item was not pushed.\n    bool push(T&& a);\n\n    /// \\brief Pop an item.\n    ///\n    /// Popping from an empty queue will result in undefined behavior.\n    T pop() noexcept;\n\n    /// \\brief access the front element in the queue\n    ///\n    /// Accessing the front of an empty or aborted queue will result in undefined\n    /// behaviour.\n    T& front() noexcept;\n\n    /// Consumes items from the queue, passing them to \\c func, until \\c func\n    /// returns false or the queue it empty\n    ///\n    /// Returns false if func returned false.\n    template <typename Func>\n    bool consume(Func&& func);\n\n    /// Returns true when the queue is empty.\n    bool empty() const noexcept;\n\n    /// Returns true when the queue is full.\n    bool full() const noexcept;\n\n    /// Returns a future<> that becomes available when pop() or consume()\n    /// can be called.\n    /// A consumer-side operation. Cannot be called concurrently with other consumer-side operations.\n    future<> not_empty() noexcept;\n\n    /// Returns a future<> that becomes available when push() can be called.\n    /// A producer-side operation. Cannot be called concurrently with other producer-side operations.\n    future<> not_full() noexcept;\n\n    /// Pops element now or when there is some. Returns a future that becomes\n    /// available when some element is available.\n    /// If the queue is, or already was, abort()ed, the future resolves with\n    /// the exception provided to abort().\n    /// A consumer-side operation. Cannot be called concurrently with other consumer-side operations.\n    future<T> pop_eventually() noexcept;\n\n    /// Pushes the element now or when there is room. Returns a future<> which\n    /// resolves when data was pushed.\n    /// If the queue is, or already was, abort()ed, the future resolves with\n    /// the exception provided to abort().\n    /// A producer-side operation. Cannot be called concurrently with other producer-side operations.\n    future<> push_eventually(T&& data) noexcept;\n\n    /// Returns the number of items currently in the queue.\n    size_t size() const noexcept {\n        // std::queue::size() has no reason to throw\n        return _q.size();\n    }\n\n    /// Returns the size limit imposed on the queue during its construction\n    /// or by a call to set_max_size(). If the queue contains max_size()\n    /// items (or more), further items cannot be pushed until some are popped.\n    size_t max_size() const noexcept { return _max; }\n\n    /// Set the maximum size to a new value. If the queue's max size is reduced,\n    /// items already in the queue will not be expunged and the queue will be temporarily\n    /// bigger than its max_size.\n    void set_max_size(size_t max) noexcept {\n        _max = max;\n        if (!full()) {\n            notify_not_full();\n        }\n    }\n\n    /// Destroy any items in the queue, and pass the provided exception to any\n    /// waiting readers or writers - or to any later read or write attempts.\n    void abort(std::exception_ptr ex) noexcept {\n        // std::queue::empty() and pop() doesn't throw\n        // since it just calls seastar::circular_buffer::pop_front\n        // that is specified as noexcept.\n        while (!_q.empty()) {\n            _q.pop();\n        }\n        _ex = ex;\n        if (_not_full) {\n            _not_full->set_exception(ex);\n            _not_full= std::nullopt;\n        }\n        if (_not_empty) {\n            _not_empty->set_exception(std::move(ex));\n            _not_empty = std::nullopt;\n        }\n    }\n\n    /// \\brief Check if there is an active consumer\n    ///\n    /// Returns true if another fiber waits for an item to be pushed into the queue\n    bool has_blocked_consumer() const noexcept {\n        return bool(_not_empty);\n    }\n};\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nqueue<T>::queue(size_t size)\n    : _max(size) {\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nvoid queue<T>::notify_not_empty() noexcept {\n    if (_not_empty) {\n        _not_empty->set_value();\n        _not_empty = std::optional<promise<>>();\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nvoid queue<T>::notify_not_full() noexcept {\n    if (_not_full) {\n        _not_full->set_value();\n        _not_full = std::optional<promise<>>();\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nbool queue<T>::push(T&& data) {\n    if (_q.size() < _max) {\n        _q.push(std::move(data));\n        notify_not_empty();\n        return true;\n    } else {\n        return false;\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nT& queue<T>::front() noexcept {\n    // std::queue::front() has no reason to throw\n    return _q.front();\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nT queue<T>::pop() noexcept {\n    if (_q.size() == _max) {\n        notify_not_full();\n    }\n    // popping the front element must not throw\n    // as T is required to be nothrow_move_constructible\n    // and std::queue::pop won't throw since it uses\n    // seastar::circular_beffer::pop_front.\n    SEASTAR_ASSERT(!_q.empty());\n    T data = std::move(_q.front());\n    _q.pop();\n    return data;\n}\n\ntemplate <typename T>\n// seastar allows only nothrow_move_constructible types\n// to be returned as future<T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nfuture<T> queue<T>::pop_eventually() noexcept {\n    if (_ex) {\n        return make_exception_future<T>(_ex);\n    }\n    if (empty()) {\n        return not_empty().then([this] {\n            if (_ex) {\n                return make_exception_future<T>(_ex);\n            } else {\n                return make_ready_future<T>(pop());\n            }\n        });\n    } else {\n        return make_ready_future<T>(pop());\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nfuture<> queue<T>::push_eventually(T&& data) noexcept {\n    if (_ex) {\n        return make_exception_future<>(_ex);\n    }\n    if (full()) {\n        return not_full().then([this, data = std::move(data)] () mutable {\n            _q.push(std::move(data));\n            notify_not_empty();\n        });\n    } else {\n      try {\n        _q.push(std::move(data));\n        notify_not_empty();\n        return make_ready_future<>();\n      } catch (...) {\n        return current_exception_as_future();\n      }\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ntemplate <typename Func>\ninline\nbool queue<T>::consume(Func&& func) {\n    if (_ex) {\n        std::rethrow_exception(_ex);\n    }\n    bool running = true;\n    while (!_q.empty() && running) {\n        running = func(std::move(_q.front()));\n        _q.pop();\n    }\n    if (!full()) {\n        notify_not_full();\n    }\n    return running;\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nbool queue<T>::empty() const noexcept {\n    // std::queue::empty() has no reason to throw\n    return _q.empty();\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nbool queue<T>::full() const noexcept {\n    // std::queue::size() has no reason to throw\n    return _q.size() >= _max;\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nfuture<> queue<T>::not_empty() noexcept {\n    if (_ex) {\n        return make_exception_future<>(_ex);\n    }\n    if (!empty()) {\n        return make_ready_future<>();\n    } else {\n        _not_empty = promise<>();\n        return _not_empty->get_future();\n    }\n}\n\ntemplate <typename T>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nfuture<> queue<T>::not_full() noexcept {\n    if (_ex) {\n        return make_exception_future<>(_ex);\n    }\n    if (!full()) {\n        return make_ready_future<>();\n    } else {\n        _not_full = promise<>();\n        return _not_full->get_future();\n    }\n}\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/ragel.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/eclipse.hh>\n#include <algorithm>\n#include <memory>\n#include <cassert>\n#include <optional>\n#include <seastar/core/future.hh>\n\nnamespace seastar {\n\n// Support classes for Ragel parsers\n\n// Builds an sstring that can be scattered across multiple packets.\n//\n// Use a sstring_build::guard variable to designate each scattered\n// char array, and call mark_start() and mark_end() at the start\n// and end points, respectively.  sstring_builder will collect data\n// from intervening segments, if needed.\n//\n// After mark_end() has been called, use the get() method to obtain\n// the built string.\n//\n// FIXME: switch to string_view.\n//\nclass sstring_builder {\n    sstring _value;\n    const char* _start = nullptr;\npublic:\n    class guard;\npublic:\n    sstring get() && {\n        return std::move(_value);\n    }\n    void reset() {\n        _value = {};\n        _start = nullptr;\n    }\n    friend class guard;\n};\n\nclass sstring_builder::guard {\n    sstring_builder& _builder;\n    const char* _block_end;\npublic:\n    guard(sstring_builder& builder, const char* block_start, const char* block_end)\n        : _builder(builder), _block_end(block_end) {\n        if (!_builder._value.empty()) {\n            mark_start(block_start);\n        }\n    }\n    ~guard() {\n        if (_builder._start) {\n            mark_end(_block_end);\n        }\n    }\n    void mark_start(const char* p) {\n        _builder._start = p;\n    }\n    void mark_end(const char* p) {\n        if (_builder._value.empty()) {\n            // avoid an allocation in the common case\n            _builder._value = sstring(_builder._start, p);\n        } else {\n            _builder._value += sstring(_builder._start, p);\n        }\n        _builder._start = nullptr;\n    }\n};\n\n\n// CRTP\ntemplate <typename ConcreteParser>\nclass ragel_parser_base {\nprotected:\n    int _fsm_cs;\n    std::unique_ptr<int[]> _fsm_stack = nullptr;\n    int _fsm_stack_size = 0;\n    int _fsm_top;\n    int _fsm_act;\n    char* _fsm_ts;\n    char* _fsm_te;\n    sstring_builder _builder;\nprotected:\n    void init_base() {\n        _builder.reset();\n    }\n    void prepush() {\n        if (_fsm_top == _fsm_stack_size) {\n            auto old = _fsm_stack_size;\n            _fsm_stack_size = std::max(_fsm_stack_size * 2, 16);\n            SEASTAR_ASSERT(_fsm_stack_size > old);\n            std::unique_ptr<int[]> new_stack{new int[_fsm_stack_size]};\n            std::copy(_fsm_stack.get(), _fsm_stack.get() + _fsm_top, new_stack.get());\n            std::swap(_fsm_stack, new_stack);\n        }\n    }\n    void postpop() {}\n    sstring get_str() {\n        return std::move(_builder).get();\n    }\npublic:\n    using unconsumed_remainder = std::optional<temporary_buffer<char>>;\n    future<unconsumed_remainder> operator()(temporary_buffer<char> buf) {\n        char* p = buf.get_write();\n        char* pe = p + buf.size();\n        char* eof = buf.empty() ? pe : nullptr;\n        char* parsed = static_cast<ConcreteParser*>(this)->parse(p, pe, eof);\n        if (parsed) {\n            buf.trim_front(parsed - p);\n            return make_ready_future<unconsumed_remainder>(std::move(buf));\n        }\n        return make_ready_future<unconsumed_remainder>();\n    }\n};\n\ninline void trim_trailing_spaces_and_tabs(sstring& str) {\n    auto data = str.data();\n    size_t i;\n    for (i = str.size(); i > 0; --i) {\n        auto c = data[i-1];\n        if (!(c == ' ' || c == '\\t')) {\n            break;\n        }\n    }\n    str.resize(i);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/reactor.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/cacheline.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/circular_buffer_fixed_capacity.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/enum.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/idle_cpu_handler.hh>\n#include <seastar/core/internal/io_desc.hh>\n#include <seastar/core/internal/io_request.hh>\n#include <seastar/core/internal/io_sink.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/make_task.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/internal/estimated_histogram.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/reactor_config.hh>\n#include <seastar/core/scattered_message.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/scheduling_specific.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/shared_mutex.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/thread_cputime_clock.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/net/api.hh>\n#include <seastar/util/eclipse.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/std-compat.hh>\n#include \"internal/pollable_fd.hh\"\n\n#include <atomic>\n#include <cassert>\n#include <chrono>\n#include <cstring>\n#include <memory>\n#include <string_view>\n#include <unordered_map>\n#include <vector>\n#include <unistd.h>\n#include <sys/epoll.h>\n#include <sys/types.h>\n#include <sys/socket.h>\n#include <netinet/ip.h>\n\n\nstruct statfs;\nstruct _Unwind_Exception;\n\nnamespace seastar {\n\nusing shard_id = unsigned;\n\nnamespace alien {\nclass instance;\n}\nclass reactor;\n\n}\n\nnamespace std {\n\ntemplate <>\nstruct hash<::sockaddr_in> {\n    size_t operator()(::sockaddr_in a) const {\n        return a.sin_port ^ a.sin_addr.s_addr;\n    }\n};\n\n}\n\nbool operator==(const ::sockaddr_in a, const ::sockaddr_in b);\n\nnamespace seastar {\n\nclass thread_pool;\nclass smp;\n\nclass reactor_backend_selector;\n\nclass reactor_backend;\nstruct pollfn;\n\nnamespace internal {\n\nclass reactor_stall_sampler;\nclass cpu_stall_detector;\nclass buffer_allocator;\nclass priority_class;\nclass poller;\n\nsize_t scheduling_group_count();\n\nvoid increase_thrown_exceptions_counter() noexcept;\nvoid increase_internal_errors_counter() noexcept;\n\ntemplate <typename Func>\nvoid at_destroy(Func&& func);\n\nvoid at_exit(noncopyable_function<future<> ()> func);\nvoid set_current_task(task* t);\n\npollable_fd posix_listen(socket_address sa, listen_options opts = {});\nfuture<> posix_connect(pollable_fd pfd, socket_address sa, socket_address local);\n\n}\n\nclass io_queue;\nclass io_intent;\n\nclass io_completion : public kernel_completion {\npublic:\n    virtual void complete_with(ssize_t res) final override;\n\n    virtual void complete(size_t res) noexcept = 0;\n    virtual void set_exception(std::exception_ptr eptr) noexcept = 0;\n};\n\nclass reactor {\nprivate:\n    struct sched_entity;\n    using scheduler_list = circular_buffer_fixed_capacity<sched_entity*, 1 << log2ceil(max_scheduling_groups())>;\n    using pollfn = seastar::pollfn;\n\n    class signal_pollfn;\n    class batch_flush_pollfn;\n    class smp_pollfn;\n    class drain_cross_cpu_freelist_pollfn;\n    class lowres_timer_pollfn;\n    class manual_timer_pollfn;\n    class epoll_pollfn;\n    class reap_kernel_completions_pollfn;\n    class kernel_submit_work_pollfn;\n    class io_queue_submission_pollfn;\n    class syscall_pollfn;\n    class execution_stage_pollfn;\n    template <typename Func>\n    friend void internal::at_destroy(Func&&);\n    friend void internal::at_exit(noncopyable_function<future<> ()> func);\n    friend class manual_clock;\n    friend class internal::reactor_stall_sampler;\n    friend class preempt_io_context;\n    friend struct hrtimer_aio_completion;\n    friend class reactor_backend_epoll;\n    friend class reactor_backend_aio;\n    friend class reactor_backend_uring;\n    friend class reactor_backend_selector;\n    friend struct reactor_options;\n    friend class aio_storage_context;\npublic:\n    using poller = internal::poller;\n    using idle_cpu_handler_result = seastar::idle_cpu_handler_result;\n    using work_waiting_on_reactor = seastar::work_waiting_on_reactor;\n    using idle_cpu_handler = seastar::idle_cpu_handler;\n\n    struct io_stats {\n        uint64_t aio_reads = 0;\n        uint64_t aio_read_bytes = 0;\n        uint64_t aio_writes = 0;\n        uint64_t aio_write_bytes = 0;\n        uint64_t aio_outsizes = 0;\n        uint64_t aio_errors = 0;\n        uint64_t aio_retries = 0;\n        uint64_t fstream_reads = 0;\n        uint64_t fstream_read_bytes = 0;\n        uint64_t fstream_reads_blocked = 0;\n        uint64_t fstream_read_bytes_blocked = 0;\n        uint64_t fstream_read_aheads_discarded = 0;\n        uint64_t fstream_read_ahead_discarded_bytes = 0;\n        uint64_t fsyncs = 0;\n\n    private:\n        friend class file_data_source_impl;\n        friend void io_completion::complete_with(ssize_t);\n        friend class io_queue;\n        friend class posix_file_impl;\n        static io_stats& local() noexcept;\n    };\n    /// Scheduling statistics.\n    struct sched_stats {\n        /// Total number of tasks processed by this shard's reactor until this point.\n        /// Note that tasks can be tiny, running for a few nanoseconds, or can take an\n        /// entire task quota.\n        uint64_t tasks_processed = 0;\n    };\n\n    /// Obtains an alien::instance object that can be used to send messages\n    /// to Seastar shards from non-Seastar threads.\n    alien::instance& alien() { return _alien; }\n\nprivate:\n    std::shared_ptr<seastar::smp> _smp;\n    alien::instance& _alien;\n    reactor_config _cfg;\n    file_desc _notify_eventfd;\n    file_desc _task_quota_timer;\n    std::unique_ptr<reactor_backend> _backend;\n    sigset_t _active_sigmask; // holds sigmask while sleeping with sig disabled\n    std::vector<pollfn*> _pollers;\n\n    static constexpr unsigned max_aio_per_queue = 128;\n    static constexpr unsigned max_queues = 8;\n    static constexpr unsigned max_aio = max_aio_per_queue * max_queues;\n\n    // Each mountpouint is controlled by its own io_queue, but ...\n    std::unordered_map<dev_t, seastar::shared_ptr<io_queue>> _io_queues;\n    // ... when dispatched all requests get into this single sink\n    internal::io_sink _io_sink;\n    unsigned _num_io_groups = 0;\n\n    std::vector<noncopyable_function<future<> ()>> _exit_funcs;\n    const unsigned _id = 0;\n    bool _stopping = false;\n    bool _stopped = false;\n    bool _finished_running_tasks = false;\n    condition_variable _stop_requested;\n    std::optional<future<std::unique_ptr<network_stack>>> _network_stack_ready;\n    int _return = 0;\n    promise<> _start_promise;\n    internal::preemption_monitor _preemption_monitor{};\n    uint64_t _global_tasks_processed = 0;\n    uint64_t _polls = 0;\n    metrics::internal::time_estimated_histogram _stalls_histogram;\n    std::unique_ptr<internal::cpu_stall_detector> _cpu_stall_detector;\n\n    timer<>::set_t _timers;\n    timer<>::set_t::timer_list_t _expired_timers;\n    timer<lowres_clock>::set_t _lowres_timers;\n    timer<lowres_clock>::set_t::timer_list_t _expired_lowres_timers;\n    timer<manual_clock>::set_t _manual_timers;\n    timer<manual_clock>::set_t::timer_list_t _expired_manual_timers;\n    io_stats _io_stats;\n    uint64_t _cxx_exceptions = 0;\n    uint64_t _abandoned_failed_futures = 0;\n\n    struct task_queue_group;\n\n    struct sched_entity {\n    protected:\n        explicit sched_entity(task_queue_group* p, float shares);\n        ~sched_entity();\n    public:\n        int64_t _vruntime = 0;\n        float _shares;\n        int64_t _reciprocal_shares_times_2_power_32;\n        bool _active = false;\n        task_queue_group* _parent = nullptr;\n\n        sched_clock::time_point _ts; // to help calculating wait/starve-times\n        sched_clock::duration _runtime = {};\n        sched_clock::duration _waittime = {};\n        sched_clock::duration _starvetime = {};\n        sched_clock::duration _time_spent_on_task_quota_violations = {};\n\n        void set_shares(float shares) noexcept;\n\n        virtual bool run_tasks() = 0;\n        void wakeup();\n        int64_t to_vruntime(sched_clock::duration runtime) const;\n    };\n\n    struct task_queue final : public sched_entity {\n        explicit task_queue(task_queue_group* p, unsigned id, sstring name, sstring shortname, float shares);\n        const uint8_t _id;\n        uint64_t _tasks_processed = 0;\n        circular_buffer<task*> _q;\n        sstring _name;\n        // the shortened version of scheduling_gruop's name, only the first 4\n        // chars are used.\n        static constexpr size_t shortname_size = 4;\n        sstring _shortname;\n        struct indirect_compare;\n        seastar::metrics::metric_groups _metrics;\n        virtual bool run_tasks() override;\n        void rename(sstring new_name, sstring new_shortname);\n    private:\n        void register_stats();\n        internal::log_buf::inserter_iterator do_dump(seastar::internal::log_buf::inserter_iterator it) const;\n    };\n\n    struct task_queue_group final : public sched_entity {\n        explicit task_queue_group(task_queue_group* p, float shares);\n\n        int64_t _last_vruntime = 0;\n        scheduler_list _active;\n        scheduler_list _activating;;\n        unsigned _nr_children = 0;\n\n        virtual bool run_tasks() override;\n        void run_some_tasks();\n\n        bool active() const noexcept;\n        void activate(sched_entity*);\n    private:\n        void insert_active_entity(sched_entity*);\n        sched_entity* pop_active_entity(sched_clock::time_point now);\n        void insert_activating_entities();\n        void account_runtime(reactor&, sched_entity&, sched_clock::duration runtime);\n    };\n\n    task_queue_group _cpu_sched;\n    std::vector<std::unique_ptr<task_queue_group>> _supergroups;\n    std::array<std::unique_ptr<task_queue>, max_scheduling_groups()> _task_queues;\n    internal::scheduling_group_specific_thread_local_data _scheduling_group_specific_data;\n    shared_mutex _scheduling_group_keys_mutex;\n    task_queue* _at_destroy_tasks;\n    task* _current_task = nullptr;\n    /// Handler that will be called when there is no task to execute on cpu.\n    /// It represents a low priority work.\n    ///\n    /// Handler's return value determines whether handler did any actual work. If no work was done then reactor will go\n    /// into sleep.\n    ///\n    /// Handler's argument is a function that returns true if a task which should be executed on cpu appears or false\n    /// otherwise. This function should be used by a handler to return early if a task appears.\n    idle_cpu_handler _idle_cpu_handler{ [] (work_waiting_on_reactor) {return idle_cpu_handler_result::no_more_work;} };\n    std::unique_ptr<network_stack> _network_stack;\n    lowres_clock::time_point _lowres_next_timeout = lowres_clock::time_point::max();\n    std::optional<pollable_fd> _aio_eventfd;\n    static constexpr unsigned loads_size = 5;\n    timer<lowres_clock> _load_timer;\n    circular_buffer<double> _loads;\n    double _load = 0;\n    // Next two fields are required to enforce the monotonicity of total_steal_time()\n    // see that method for details.\n\n    // Last measured accumulated steal time, i.e., the simple difference of accumulated\n    // awake time and consumed thread CPU time.\n    mutable sched_clock::duration _last_true_steal{0};\n    // Accumulated steal time forced to be monotinic by rejecting any updates that would\n    // decrease it. See total_steal_time() for details.\n    mutable sched_clock::duration _last_mono_steal{0};\n    sched_clock::duration _total_idle{0};\n    sched_clock::duration _total_sleep{0};\n    sched_clock::time_point _start_time = now();\n    output_stream<char>::batch_flush_list_t _flush_batching;\n    std::atomic<bool> _sleeping alignas(seastar::cache_line_size){0};\n    gate _background_gate;\n\n    inline auto& get_sg_data(const scheduling_group& sg) {\n        return _scheduling_group_specific_data.per_scheduling_group_data[sg._id];\n    }\n\nprivate:\n    static std::chrono::nanoseconds calculate_poll_time();\n    static void block_notifier(int);\n    bool flush_pending_aio();\n    steady_clock_type::time_point next_pending_aio() const noexcept;\n    bool reap_kernel_completions();\n    bool flush_tcp_batches();\n    void update_lowres_clocks() noexcept;\n    bool do_expire_lowres_timers() noexcept;\n    bool do_check_lowres_timers() const noexcept;\n    void expire_manual_timers() noexcept;\n    void start_aio_eventfd_loop();\n    void stop_aio_eventfd_loop();\n\n    /**\n     * Returns TRUE if all pollers allow blocking.\n     *\n     * @return FALSE if at least one of the blockers requires a non-blocking\n     *         execution.\n     */\n    bool poll_once();\n    bool pure_poll_once();\npublic:\n    /// Register a user-defined signal handler\n    [[deprecated(\"Use seastar::handle_signal(signo, handler, once); instead\")]]\n    void handle_signal(int signo, noncopyable_function<void ()>&& handler);\n    void wakeup();\n    /// @private\n    bool stopped() const noexcept { return _stopped; }\n    /// @private\n    uint64_t polls() const noexcept { return _polls; }\n\nprivate:\n    class signals {\n    public:\n        signals();\n        ~signals();\n\n        bool poll_signal();\n        bool pure_poll_signal() const;\n        void handle_signal(int signo, noncopyable_function<void ()>&& handler);\n        void handle_signal_once(int signo, noncopyable_function<void ()>&& handler);\n        static void action(int signo, siginfo_t* siginfo, void* ignore);\n        static void failed_to_handle(int signo);\n    private:\n        struct signal_handler {\n            signal_handler(int signo, noncopyable_function<void ()>&& handler);\n            noncopyable_function<void ()> _handler;\n        };\n        std::atomic<uint64_t> _pending_signals;\n        std::unordered_map<int, signal_handler> _signal_handlers;\n    };\n\n    signals _signals;\n    std::unique_ptr<thread_pool> _thread_pool;\n    friend class internal::cpu_stall_detector;\n\n    friend void handle_signal(int signo, noncopyable_function<void ()>&& handler, bool once);\n\n    uint64_t pending_task_count() const;\n    bool have_more_tasks() const;\n    bool posix_reuseport_detect();\n    future<> rename_scheduling_group_specific_data(scheduling_group sg);\n    future<seastar::scheduling_group> init_scheduling_group(sstring name, sstring shortname, float shares, scheduling_supergroup p);\n    future<> init_scheduling_group(scheduling_group sg, sstring name, sstring shortname, float shares, scheduling_supergroup p);\n    future<> init_scheduling_group_specific_data(scheduling_group sg);\n    future<> init_new_scheduling_group_key(scheduling_group_key key, scheduling_group_key_config cfg);\n    future<> destroy_scheduling_group(scheduling_group sg) noexcept;\n    uint64_t tasks_processed() const;\n    void request_preemption();\n    void start_handling_signal();\n    void reset_preemption_monitor();\n    void service_highres_timer() noexcept;\n    void enable_timer(steady_clock_type::time_point when) noexcept;\n\n    future<std::tuple<pollable_fd, socket_address>>\n    do_accept(pollable_fd_state& listen_fd);\n    future<> do_connect(pollable_fd_state& pfd, socket_address& sa);\n\n    future<size_t>\n    do_read(pollable_fd_state& fd, void* buffer, size_t size);\n    future<size_t>\n    do_recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov);\n    future<temporary_buffer<char>>\n    do_read_some(pollable_fd_state& fd, internal::buffer_allocator* ba);\n\n#if SEASTAR_API_LEVEL < 9\n    future<size_t>\n    do_send(pollable_fd_state& fd, const void* buffer, size_t size);\n#endif\n\n    future<size_t>\n    do_sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len);\n\n    future<temporary_buffer<char>>\n    do_recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba);\n\n    void configure(const reactor_options& opts);\n    int do_run();\n    // Waits for all background tasks on all shards\n    static future<> drain();\n\npublic:\n    explicit reactor(std::shared_ptr<seastar::smp> smp, alien::instance& alien, unsigned id, reactor_backend_selector rbs, reactor_config cfg);\n    reactor(const reactor&) = delete;\n    ~reactor();\n    void operator=(const reactor&) = delete;\n\n    static sched_clock::time_point now() noexcept {\n        return sched_clock::now();\n    }\n    sched_clock::duration uptime() const {\n        return now() - _start_time;\n    }\n\n    io_queue& get_io_queue(dev_t devid = 0) {\n        auto queue = _io_queues.find(devid);\n        if (queue == _io_queues.end()) {\n            return *_io_queues.at(0);\n        } else {\n            return *(queue->second);\n        }\n    }\n\n    io_queue* try_get_io_queue(dev_t devid) noexcept {\n        auto queue = _io_queues.find(devid);\n        return queue != _io_queues.end() ? queue->second.get() : nullptr;\n    }\n\n    std::string_view get_backend_name() const;\n\nprivate:\n    future<> update_bandwidth_for_queues(internal::priority_class pc, uint64_t bandwidth);\n    future<> update_bandwidth_for_queues(unsigned group_index, uint64_t bandwidth);\n    void rename_queues(internal::priority_class pc, sstring new_name);\n    void update_shares_for_queues(internal::priority_class pc, uint32_t shares);\n    void update_group_shares_for_queues(unsigned, uint32_t shares);\n\npublic:\n    server_socket listen(socket_address sa, listen_options opts = {});\n\n    future<connected_socket> connect(socket_address sa);\n    future<connected_socket> connect(socket_address, socket_address, transport proto = transport::TCP);\n\n    [[deprecated(\"Internal reactor function, consider using seastar::listen()\")]]\n    pollable_fd posix_listen(socket_address sa, listen_options opts = {}) {\n        return internal::posix_listen(sa, opts);\n    }\n\n    // FIXME: reuseport currently leads to heavy load imbalance.\n    // Until we fix that, just disable it unconditionally.\n    bool posix_reuseport_available() const { return false; }\n    bool posix_sock_need_nonblock() const;\n    bool have_aio_fdatasync() const;\n\n    [[deprecated(\"Internal reactor function, consider using net sockets\")]]\n    pollable_fd make_pollable_fd(socket_address sa, int proto);\n\n    [[deprecated(\"Internal reactor function, consider using seastar::connect)\")]]\n    future<> posix_connect(pollable_fd pfd, socket_address sa, socket_address local) {\n        return internal::posix_connect(std::move(pfd), sa, local);\n    }\n\n#if SEASTAR_API_LEVEL < 9\n    future<> send_all(pollable_fd_state& fd, const void* buffer, size_t size);\n#endif\n\n    future<file> open_file_dma(std::string_view name, open_flags flags, file_open_options options = {}) noexcept;\n    future<file> open_directory(std::string_view name) noexcept;\n    future<> make_directory(std::string_view name, file_permissions permissions = file_permissions::default_dir_permissions) noexcept;\n    future<> touch_directory(std::string_view name, file_permissions permissions = file_permissions::default_dir_permissions) noexcept;\n    future<std::optional<directory_entry_type>>  file_type(std::string_view name, follow_symlink = follow_symlink::yes) noexcept;\n    future<stat_data> file_stat(std::string_view pathname, follow_symlink) noexcept;\n    future<stat_data> file_stat(file& directory, std::string_view pathname, follow_symlink) noexcept;\n    future<> chown(std::string_view filepath, uid_t owner, gid_t group);\n    future<std::optional<struct group_details>> getgrnam(std::string_view name);\n    future<uint64_t> file_size(std::string_view pathname) noexcept;\n    future<bool> file_accessible(std::string_view pathname, access_flags flags) noexcept;\n    future<bool> file_exists(std::string_view pathname) noexcept {\n        return file_accessible(pathname, access_flags::exists);\n    }\n    future<fs_type> file_system_at(std::string_view pathname) noexcept;\n    future<std::filesystem::space_info> file_system_space(std::string_view pathname) noexcept;\n    future<struct statvfs> statvfs(std::string_view pathname) noexcept;\n    future<> remove_file(std::string_view pathname) noexcept;\n    future<> rename_file(std::string_view old_pathname, std::string_view new_pathname) noexcept;\n    future<> link_file(std::string_view oldpath, std::string_view newpath) noexcept;\n    future<> chmod(std::string_view name, file_permissions permissions) noexcept;\n\n    [[deprecated(\"Use file::list_directory API instead\")]]\n    future<size_t> read_directory(int fd, char* buffer, size_t buffer_size);\n\n    future<int> inotify_add_watch(int fd, std::string_view path, uint32_t flags);\n\n    future<std::tuple<file_desc, file_desc>> make_pipe();\n    future<std::tuple<pid_t, file_desc, file_desc, file_desc>>\n    spawn(std::string_view pathname,\n          std::vector<sstring> argv,\n          std::vector<sstring> env = {});\n    future<int> waitpid(pid_t pid);\n    void kill(pid_t pid, int sig);\n\n    int run() noexcept;\n    void exit(int ret);\n    future<> when_started() { return _start_promise.get_future(); }\n    // The function waits for timeout period for reactor stop notification\n    // which happens on termination signals or call for exit().\n    template <typename Rep, typename Period>\n    future<> wait_for_stop(std::chrono::duration<Rep, Period> timeout) {\n        return _stop_requested.wait(timeout, [this] { return _stopping; });\n    }\n\n    /// Deprecated. Use following sequence instead:\n    ///\n    /// ```\n    ///    return seastar::app_template::run([] {\n    ///         return seastar::async([] {\n    ///              // Since the function runs in a thread, it can wait for futures using\n    ///              // future::get().\n    ///              auto deferred_task = defer(/* the function you want to run at exit */);\n    ///         });\n    ///    });\n    /// ```\n    [[deprecated(\"Use seastar::app_template::run(), seastar::async(), and seastar::defer() for orderly shutdown\")]]\n    void at_exit(noncopyable_function<future<> ()> func);\n\n    /// Deprecated. Use following sequence instead:\n    ///\n    /// ```\n    ///    return seastar::app_template::run([] {\n    ///         return seastar::async([] {\n    ///              auto deferred_task = defer(/* the function you want to run at exit */);\n    ///         });\n    ///    });\n    /// ```\n    template <typename Func>\n    [[deprecated(\"Use seastar::app_template::run(), seastar::async(), and seastar::defer() for orderly shutdown\")]]\n    void at_destroy(Func&& func) {\n        do_at_destroy(std::forward<Func>(func));\n    }\nprivate:\n    void do_at_exit(noncopyable_function<future<> ()> func);\ntemplate <typename Func>\n    void do_at_destroy(Func&& func) {\n        _at_destroy_tasks->_q.push_back(make_task(default_scheduling_group(), std::forward<Func>(func)));\n    }\npublic:\n    task* current_task() const { return _current_task; }\n\n    void add_task(task* t) noexcept;\n    void add_urgent_task(task* t) noexcept;\n\n    /// Pass a future to \"run\" in the background.\n    ///\n    /// The reactor will wait for the future to complete before stopping. This can be\n    /// use to ensure that a future completes before reactor shutdown.\n    /// If the future returns exceptionally, a warning is logged but the error is otherwise\n    /// ignored.\n    void run_in_background(future<> f);\n\n    /// Run func in the background.\n    ///\n    /// Executes a possibly-async func. If the function is synchronous, it runs immediately.\n    ///\n    /// If the function is async, it is executed and runs in the background (as if cast to void).\n    /// The reactor will wait for the returned to complete before stopping. This\n    /// can be used to ensure that a future completes before reactor shutdown.\n    ///\n    /// In both the sync and async cases if the function throws or the returned\n    /// future is exceptional, a warning is logged but the error is otherwise\n    /// ignored.\n    template <typename Func>\n    void run_in_background(Func&& func) {\n        run_in_background(futurize_invoke(std::forward<Func>(func)));\n    }\n\n    /// Set a handler that will be called when there is no task to execute on cpu.\n    /// Handler should do a low priority work.\n    ///\n    /// Handler's return value determines whether handler did any actual work. If no work was done then reactor will go\n    /// into sleep.\n    ///\n    /// Handler's argument is a function that returns true if a task which should be executed on cpu appears or false\n    /// otherwise. This function should be used by a handler to return early if a task appears.\n    void set_idle_cpu_handler(idle_cpu_handler&& handler) {\n        _idle_cpu_handler = std::move(handler);\n    }\n    void force_poll();\n\n    void add_high_priority_task(task*) noexcept;\n\n    network_stack& net() { return *_network_stack; }\n\n    [[deprecated(\"Use this_shard_id\")]]\n    shard_id cpu_id() const;\n\n    /// \\returns Returns the `smp` instance which owns this reactor.\n    const seastar::smp& smp() const noexcept;\n\n    steady_clock_type::duration total_idle_time() const;\n    steady_clock_type::duration total_busy_time() const;\n    steady_clock_type::duration total_awake_time() const;\n    std::chrono::nanoseconds total_cpu_time() const;\n    std::chrono::nanoseconds total_steal_time() const;\n\n    const io_stats& get_io_stats() const { return _io_stats; }\n    /// Returns statistics related to scheduling. The statistics are\n    /// local to this shard.\n    ///\n    /// See \\ref sched_stats for a description of individual statistics.\n    /// \\return An object containing a snapshot of the statistics at this point in time.\n    sched_stats get_sched_stats() const;\n    uint64_t cxx_exceptions() const { return _cxx_exceptions; }\n    uint64_t abandoned_failed_futures() const { return _abandoned_failed_futures; }\nprivate:\n    /**\n     * Add a new \"poller\" - a non-blocking function returning a boolean, that\n     * will be called every iteration of a main loop.\n     * If it returns FALSE then reactor's main loop is forbidden to block in the\n     * current iteration.\n     *\n     * @param fn a new \"poller\" function to register\n     */\n    void register_poller(pollfn* p);\n    void unregister_poller(pollfn* p);\n    void replace_poller(pollfn* old, pollfn* neww);\n    void register_metrics();\n\n#if SEASTAR_API_LEVEL < 9\n    future<> send_all_part(pollable_fd_state& fd, const void* buffer, size_t size, size_t completed);\n#endif\n\n    [[deprecated(\"Use file::flush() instead\")]]\n    future<> fdatasync(int fd) noexcept;\n\n    void add_timer(timer<steady_clock_type>*) noexcept;\n    bool queue_timer(timer<steady_clock_type>*) noexcept;\n    void del_timer(timer<steady_clock_type>*) noexcept;\n    void add_timer(timer<lowres_clock>*) noexcept;\n    bool queue_timer(timer<lowres_clock>*) noexcept;\n    void del_timer(timer<lowres_clock>*) noexcept;\n    void add_timer(timer<manual_clock>*) noexcept;\n    bool queue_timer(timer<manual_clock>*) noexcept;\n    void del_timer(timer<manual_clock>*) noexcept;\n\n    bool pollers_enter_interrupt_mode();\n    void pollers_exit_interrupt_mode();\n    void wait_and_process_events();\n\n    future<> run_exit_tasks();\n    void stop();\n    friend class pollable_fd_state;\n    friend class posix_file_impl;\n    friend class blockdev_file_impl;\n    friend class timer<>;\n    friend class timer<lowres_clock>;\n    friend class timer<manual_clock>;\n    friend class smp;\n    friend class internal::poller;\n    friend class scheduling_group;\n    friend class scheduling_supergroup;\n    friend size_t internal::scheduling_group_count();\n    friend void internal::add_to_flush_poller(output_stream<char>& os) noexcept;\n    friend void seastar::internal::increase_thrown_exceptions_counter() noexcept;\n    friend void seastar::internal::increase_internal_errors_counter() noexcept;\n    friend void internal::report_failed_future(const std::exception_ptr& eptr) noexcept;\n    metrics::metric_groups _metric_groups;\n    friend future<scheduling_group> create_scheduling_group(sstring name, sstring shortname, float shares, scheduling_supergroup) noexcept;\n    friend future<scheduling_supergroup> create_scheduling_supergroup(float shares) noexcept;\n    friend future<> destroy_scheduling_supergroup(scheduling_supergroup sg) noexcept;\n    friend future<> seastar::destroy_scheduling_group(scheduling_group) noexcept;\n    friend future<> seastar::rename_scheduling_group(scheduling_group sg, sstring new_name, sstring new_shortname) noexcept;\n    friend future<scheduling_group_key> scheduling_group_key_create(scheduling_group_key_config cfg) noexcept;\n    friend void internal::set_current_task(task* t);\n    friend scheduling_supergroup internal::scheduling_supergroup_for(scheduling_group sg) noexcept;\n\n    future<struct statfs> fstatfs(int fd) noexcept;\n    friend future<shared_ptr<file_impl>> make_file_impl(int fd, file_open_options options, int flags, struct stat st) noexcept;\npublic:\n    future<> readable(pollable_fd_state& fd);\n    future<> writeable(pollable_fd_state& fd);\n    future<> readable_or_writeable(pollable_fd_state& fd);\n    future<> poll_rdhup(pollable_fd_state& fd);\n    /// Sets the \"Strict DMA\" flag.\n    ///\n    /// When true (default), file I/O operations must use DMA.  This is\n    /// the most performant option, but does not work on some file systems\n    /// such as tmpfs or aufs (used in some Docker setups).\n    ///\n    /// When false, file I/O operations can fall back to buffered I/O if\n    /// DMA is not available.  This can result in dramatic reducation in\n    /// performance and an increase in memory consumption.\n    void set_strict_dma(bool value);\n    void set_bypass_fsync(bool value);\n    void update_blocked_reactor_notify_ms(std::chrono::milliseconds ms);\n    std::chrono::milliseconds get_blocked_reactor_notify_ms() const;\n\n    class test {\n    public:\n        static void with_allow_abandoned_failed_futures(unsigned count, noncopyable_function<void ()> func);\n\n        /// Sets the stall reporting function which is called when a stall\n        /// is detected (and not suppressed). Setting the function also\n        /// resets the supression state.\n        static void set_stall_detector_report_function(std::function<void ()> report);\n        static std::function<void ()> get_stall_detector_report_function();\n        static bool linux_aio_nowait();\n\n        struct long_task_queue_state {\n            bool abort_on_too_long_task_queue;\n            unsigned max_task_backlog;\n        };\n        static long_task_queue_state get_long_task_queue_state() noexcept;\n        static future<> restore_long_task_queue_state(const long_task_queue_state& state) noexcept;\n        static void set_abort_on_too_long_task_queue(bool value) noexcept;\n        static void set_max_task_backlog(unsigned value) noexcept;\n    };\n};\n\nextern __thread reactor* local_engine;\n\ninline reactor& engine() {\n    return *local_engine;\n}\n\ninline bool engine_is_ready() {\n    return local_engine != nullptr;\n}\n\ninline int hrtimer_signal() {\n    // We don't want to use SIGALRM, because the boost unit test library\n    // also plays with it.\n    return SIGRTMIN;\n}\n\ninline auto reactor::io_stats::local() noexcept -> io_stats& {\n    return engine()._io_stats;\n}\n\nextern logger seastar_logger;\n\nnamespace internal {\n\ntemplate <typename Func>\nvoid at_destroy(Func&& func) {\n    engine().do_at_destroy(std::forward<Func>(func));\n}\n\n} // namespace internal\n\n}\n"
  },
  {
    "path": "include/seastar/core/reactor_config.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/util/program-options.hh>\n#include <seastar/util/memory_diagnostics.hh>\n#include <seastar/core/scheduling.hh>\n\nnamespace seastar {\n\n/// \\cond internal\nstruct reactor_config {\n    sched_clock::duration task_quota;\n    std::chrono::nanoseconds max_poll_time;\n    bool handle_sigint = true;\n    bool auto_handle_sigint_sigterm = true;\n    unsigned max_networking_aio_io_control_blocks = 10000;\n    bool force_io_getevents_syscall = false;\n    bool kernel_page_cache = false;\n    bool have_aio_fsync = false;\n    unsigned max_task_backlog = 1000;\n    bool strict_o_direct = true;\n    bool bypass_fsync = false;\n    bool no_poll_aio = false;\n    bool aio_nowait_works = false;\n    bool abort_on_too_long_task_queue = false;\n};\n/// \\endcond\n\nclass reactor_backend_selector;\nclass network_stack_factory;\n\n/// Configuration for the reactor.\nstruct reactor_options : public program_options::option_group {\n    /// \\brief Select network stack to use.\n    ///\n    /// Each network stack has it corresponding\n    /// \\ref program_options::option_group to further tune it. The available\n    /// stacks are:\n    /// * Posix stack (default) - no tunable options;\n    /// * Native stack  - \\ref net::native_stack_options;\n    program_options::selection_value<network_stack_factory> network_stack;\n    /// Poll continuously (100% cpu use).\n    program_options::value<> poll_mode;\n    /// \\brief Idle polling time in microseconds.\n    ///\n    /// Reduce for overprovisioned environments or laptops.\n    program_options::value<unsigned> idle_poll_time_us;\n    /// \\brief Busy-poll for disk I/O.\n    ///\n    /// Reduces latency and increases throughput.\n    program_options::value<bool> poll_aio;\n    /// \\brief Max time (ms) between polls.\n    ///\n    /// Default: 0.5.\n    program_options::value<double> task_quota_ms;\n    /// \\brief Max time (ms) IO operations must take.\n    ///\n    /// Default: 1.5 * task_quota_ms value\n    program_options::value<double> io_latency_goal_ms;\n    /// \\bried Dispatch rate to completion rate ratio threshold\n    ///\n    /// Describes the worst ratio at which seastar reactor is allowed to delay\n    /// IO requests completion. If exceeded, the scheduler will consider it's\n    /// disk that's the reason for completion slow-down and will scale down\n    ///\n    /// Default: 1.1\n    program_options::value<double> io_flow_ratio_threshold;\n    /// \\brief If an IO request is executed longer than that, this is printed to\n    /// logs with extra debugging\n    ///\n    /// Default: infinite (detection is OFF)\n    program_options::value<unsigned> io_completion_notify_ms;\n    /// \\brief Maximum number of task backlog to allow.\n    ///\n    /// When the number of tasks grow above this, we stop polling (e.g. I/O)\n    /// until it goes back below the limit.\n    /// Default: 1000.\n    program_options::value<unsigned> max_task_backlog;\n    /// \\brief Threshold in milliseconds over which the reactor is considered\n    /// blocked if no progress is made.\n    ///\n    /// Default: 25.\n    program_options::value<unsigned> blocked_reactor_notify_ms;\n    /// \\brief Maximum number of backtraces reported by stall detector per minute.\n    ///\n    /// Default: 5.\n    program_options::value<unsigned> blocked_reactor_reports_per_minute;\n    /// \\brief Print a simplified backtrace on a single line.\n    ///\n    /// Default: \\p true.\n    program_options::value<bool> blocked_reactor_report_format_oneline;\n    /// \\brief Allow using buffered I/O if DMA is not available (reduces performance).\n    program_options::value<> relaxed_dma;\n    /// \\brief Use the Linux NOWAIT AIO feature, which reduces reactor stalls due\n    /// to aio (autodetected).\n    program_options::value<bool> linux_aio_nowait;\n    /// \\brief Bypass fsync(), may result in data loss.\n    ///\n    /// Use for testing on consumer drives.\n    /// Default: \\p false.\n    program_options::value<bool> unsafe_bypass_fsync;\n    /// \\brief Use the kernel page cache.\n    ///\n    /// This disables DMA (O_DIRECT). Useful for short-lived functional tests\n    /// with a small data set.\n    /// Default: \\p false.\n    program_options::value<bool> kernel_page_cache;\n    /// \\brief Run in an overprovisioned environment (such as docker or a laptop).\n    ///\n    /// Equivalent to:\n    /// * \\ref idle_poll_time_us = 0\n    /// * \\ref smp_options::thread_affinity = 0\n    /// * \\ref poll_aio = 0\n    program_options::value<> overprovisioned;\n    /// \\brief Abort when seastar allocator cannot allocate memory.\n    program_options::value<> abort_on_seastar_bad_alloc;\n    /// \\brief Abort when a task queue becomes too long.\n    program_options::value<bool> abort_on_too_long_task_queue;\n    /// \\brief Force \\p io_getevents(2) to issue a system call, instead of\n    /// bypassing the kernel when possible.\n    ///\n    /// This makes strace output more useful, but slows down the application.\n    /// Default: \\p false.\n    program_options::value<bool> force_aio_syscalls;\n    /// \\brief Dump diagnostics of the seastar allocator state on allocation\n    /// failure.\n    ///\n    /// See \\ref memory::alloc_failure_kind for allowed values. The diagnostics\n    /// will be written to the \\p seastar_memory logger, with error level.\n    /// Default: \\ref memory::alloc_failure_kind::critical.\n    /// \\note Even if the \\p seastar_memory logger is set to debug or trace\n    /// level, the diagnostics will be logged irrespective of this setting.\n    program_options::value<memory::alloc_failure_kind> dump_memory_diagnostics_on_alloc_failure_kind;\n    /// \\brief Internal reactor implementation.\n    ///\n    /// Available backends:\n    /// * \\p linux-aio\n    /// * \\p epoll\n    /// * \\p io_uring\n    ///\n    /// Default: \\p linux-aio (if available).\n    program_options::selection_value<reactor_backend_selector> reactor_backend;\n    /// \\brief Use Linux aio for fsync() calls.\n    ///\n    /// This reduces latency. Requires Linux 4.18 or later.\n    program_options::value<bool> aio_fsync;\n    /// \\brief Maximum number of I/O control blocks (IOCBs) to allocate per shard.\n    ///\n    /// This translates to the number of sockets supported per shard. Requires\n    /// tuning \\p /proc/sys/fs/aio-max-nr. Only valid for the \\p linux-aio\n    /// reactor backend (see \\ref reactor_backend).\n    ///\n    /// Default: 10000.\n    program_options::value<unsigned> max_networking_io_control_blocks;\n    /// \\brief Leave this many I/O control blocks (IOCBs) as reserve.\n    ///\n    /// This is to allows leaving a (small) reserve aside so other applications\n    /// also using IOCBs can run alongside the seastar application.\n    /// The reserve takes precedence over \\ref max_networking_io_control_blocks.\n    ///\n    /// Default: 0\n    ///\n    /// \\see max_networking_io_control_blocks\n    program_options::value<unsigned> reserve_io_control_blocks;\n    /// \\brief Enable seastar heap profiling.\n    ///\n    /// Allocations will be sampled every N bytes on average. Zero means off.\n    ///\n    /// Default: 0\n    ///\n    /// \\note Unused when seastar was compiled without heap profiling support.\n    program_options::value<unsigned> heapprof;\n    /// Ignore SIGINT (for gdb).\n    program_options::value<> no_handle_interrupt;\n\n    /// \\cond internal\n    std::string _argv0;\n    bool _auto_handle_sigint_sigterm = true;\n    /// \\endcond\n\npublic:\n    /// \\cond internal\n    reactor_options(program_options::option_group* parent_group);\n    /// \\endcond\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/relabel_config.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2022 ScyllaDB\n */\n\n#pragma once\n\n#include <regex>\n\nnamespace seastar {\nnamespace metrics {\n\n\n/*!\n * \\brief a wrapper class around regex with the original expr\n *\n * regex does not contain the original expression, this wrapper class\n * acts both as a string and as a regex.\n */\nclass relabel_config_regex {\n    std::string _regex_str;\n    std::regex _regex;\npublic:\n    relabel_config_regex() = default;\n    relabel_config_regex(const std::string& expr) : _regex_str(expr), _regex(std::regex(expr)) {}\n    relabel_config_regex(const char* expr) : _regex_str(expr), _regex(std::regex(expr)) {}\n    const std::string& str() const noexcept {\n        return _regex_str;\n    }\n    const std::regex& regex() const noexcept {\n        return _regex;\n    }\n\n    relabel_config_regex& operator=(const char* expr) {\n        std::string str(expr);\n        return operator=(str);\n    }\n\n    relabel_config_regex& operator=(const std::string& expr) {\n        _regex_str = expr;\n        _regex = std::regex(_regex_str);\n        return *this;\n    }\n    bool empty() const noexcept {\n        return _regex_str.empty();\n    }\n\n    bool match(const std::string& str)  const noexcept {\n        return !empty() && std::regex_match(str, _regex);\n    }\n};\n\n/*!\n * \\brief a relabel_config allows changing metrics labels dynamically\n *\n * The logic is similar to Prometheus configuration\n * This is how Prometheus entry looks like:\n *  - source_labels: [version]\n      regex: '([0-9]+\\.[0-9]+)(\\.?[0-9]*).*'\n      replacement: '$1$2'\n      target_label: svr\n * relabel_action values:\n *   skip_when_empty - when set supported metrics (histogram, summary and counters)\n *                     will not be reported if they were never used.\n *   report_when_empty - revert the skip_when_empty flag\n *   replace - replace the value of the target_label\n *   keep - enable the metrics\n *   drop - disable the metrics\n *   drop_label  - remove the target label\n *\n * source_labels - a list of source labels, the labels are concatenated\n *                 with the separator and and the combine value is match to the regex.\n * target_label  - the labels to perform the action on when replacing a value or when dropping a label.\n * replacement   - the string to use when replacing a label value, regex group can be used.\n * expr          - a regular expression in a string format. Action would be taken if the regex\n *                 match the concatenated labels.\n * action        - The action to perform when there is a match.\n * separator     - separator to use when concatenating the labels.\n *\n */\nstruct relabel_config {\n    enum class relabel_action {skip_when_empty, report_when_empty, replace, keep, drop, drop_label};\n    std::vector<std::string> source_labels;\n    std::string target_label;\n    std::string replacement = \"${1}\";\n    relabel_config_regex expr = \"(.*)\";\n    relabel_action action = relabel_action::replace;\n    std::string separator = \";\";\n};\n\n/*!\n * \\brief a helper function to translate a string to relabel_config::relabel_action enum values\n */\nrelabel_config::relabel_action relabel_config_action(const std::string& action);\n\n/*!\n * \\brief metric_family_config allow changing metrics family configuration\n *\n * Allow changing metrics family configuration\n * Supports changing the aggregation labels.\n * The metrics family can be identified by a name or by regex; name-matching is\n * more efficient and should be used for a single metrics family.\n *\n * name - optional exact metric name\n * regex_name - if set, all the metrics name that match the regular expression\n * aggregate_labels - The labels to aggregate the metrics by.\n *\n */\nstruct metric_family_config {\n    std::string name;\n    relabel_config_regex regex_name = \"\";\n    std::vector<std::string> aggregate_labels;\n};\n\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/report_exception.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <exception>\n#include <string_view>\n\nnamespace seastar {\n\nvoid report_exception(std::string_view message, std::exception_ptr) noexcept;\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/resource.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/util/spinlock.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <cassert>\n#include <cstdlib>\n#include <memory>\n#include <optional>\n#include <string>\n#include <vector>\n#include <set>\n#include <sched.h>\n#include <unordered_map>\n\nstruct hwloc_topology;\nusing hwloc_topology_t = hwloc_topology*;\n\nnamespace seastar {\n\nclass io_queue;\nclass io_group;\n\nnamespace resource {\n\nusing std::optional;\n\nusing cpuset = std::set<unsigned>;\n\n/// \\cond internal\nstd::optional<cpuset> parse_cpuset(std::string value);\n/// \\endcond\n\nnamespace hwloc::internal {\n\n#ifdef SEASTAR_HAVE_HWLOC\nclass topology_holder {\n    hwloc_topology_t _topology;\n\npublic:\n    topology_holder() noexcept\n        : _topology(nullptr)\n    { }\n\n    topology_holder(topology_holder&& o) noexcept;\n\n    ~topology_holder();\n\n    topology_holder& operator=(topology_holder&& o) noexcept;\n\n    operator bool() const noexcept {\n        return _topology != nullptr;\n    }\n\n    void init_and_load();\n    hwloc_topology_t get();\n};\n\n#else // SEASTAR_HAVE_HWLOC\n\nstruct topology_holder {};\n\n#endif // SEASTAR_HAVE_HWLOC\n\n} // namespace hwloc::internal\n\n\nstruct configuration {\n    optional<size_t> total_memory;\n    optional<size_t> reserve_memory;  // if total_memory not specified\n    size_t reserve_additional_memory_per_shard;\n    size_t cpus;\n    cpuset cpu_set;\n    bool assign_orphan_cpus = false;\n    std::vector<unsigned> io_queues;\n    unsigned num_io_groups;\n    hwloc::internal::topology_holder topology;\n    bool overcommit = false;\n};\n\nstruct memory {\n    size_t bytes;\n    unsigned nodeid;\n\n};\n\nstruct io_queue_topology {\n    std::vector<seastar::shared_ptr<io_queue>> queues;\n    std::vector<unsigned> shard_to_group;\n    std::vector<unsigned> shards_in_group;\n    std::vector<std::shared_ptr<io_group>> groups;\n\n    util::spinlock lock;\n\n    io_queue_topology();\n    io_queue_topology(const io_queue_topology&) = delete;\n    io_queue_topology(io_queue_topology&&);\n    ~io_queue_topology();\n};\n\nstruct cpu {\n    unsigned cpu_id;\n    std::vector<memory> mem;\n};\n\nstruct resources {\n    std::vector<cpu> cpus;\n    std::unordered_map<unsigned, io_queue_topology> ioq_topology;\n    std::unordered_map<unsigned /* numa node id */, cpuset> numa_node_id_to_cpuset;\n};\n\nresources allocate(configuration& c);\nunsigned nr_processing_units(configuration& c);\n\n\nstd::optional<resource::cpuset> parse_cpuset(std::string value);\n\n\n}\n}\n"
  },
  {
    "path": "include/seastar/core/rwlock.hh",
    "content": "/*\n* This file is open source software, licensed to you under the terms\n* of the Apache License, Version 2.0 (the \"License\"). See the NOTICE file\n* distributed with this work for additional information regarding copyright\n* ownership. You may not use this file except in compliance with the License.\n*\n* You may obtain a copy of the License at\n*\n* http://www.apache.org/licenses/LICENSE-2.0\n*\n* Unless required by applicable law or agreed to in writing,\n* software distributed under the License is distributed on an\n* \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n* KIND, either express or implied. See the License for the\n* specific language governing permissions and limitations\n* under the License.\n*/\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cstddef>\n#include <seastar/core/semaphore.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\n/// \\cond internal\n// lock / unlock semantics for rwlock, so it can be used with with_lock()\ntemplate<typename Clock>\nclass basic_rwlock;\n\ntemplate<typename Clock = typename timer<>::clock>\nclass rwlock_for_read {\npublic:\n    future<> lock() {\n        return static_cast<basic_rwlock<Clock>*>(this)->read_lock();\n    }\n    void unlock() {\n        static_cast<basic_rwlock<Clock>*>(this)->read_unlock();\n    }\n    friend class basic_rwlock<Clock>;\n};\n\ntemplate<typename Clock = typename timer<>::clock>\nclass rwlock_for_write {\npublic:\n    future<> lock() {\n        return static_cast<basic_rwlock<Clock>*>(this)->write_lock();\n    }\n    void unlock() {\n        static_cast<basic_rwlock<Clock>*>(this)->write_unlock();\n    }\n    friend class basic_rwlock<Clock>;\n};\n/// \\endcond\n\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// Implements a read-write lock mechanism. Beware: this is not a cross-CPU\n/// lock, due to seastar's sharded architecture.\n/// Instead, it can be used to achieve rwlock semantics between two (or more)\n/// fibers running in the same CPU that may use the same resource.\n/// Acquiring the write lock will effectively cause all readers not to be executed\n/// until the write part is done.\ntemplate<typename Clock = typename timer<>::clock>\nclass basic_rwlock : private rwlock_for_read<Clock>, rwlock_for_write<Clock> {\n    using semaphore_type = basic_semaphore<semaphore_default_exception_factory, Clock>;\n\n    static constexpr size_t max_ops = semaphore_type::max_counter();\n\n    semaphore_type _sem;\npublic:\n    basic_rwlock()\n            : _sem(max_ops) {\n    }\n\n    /// Cast this rwlock into read lock object with lock semantics appropriate to be used\n    /// by \"with_lock\". The resulting object will have lock / unlock calls that, when called,\n    /// will acquire / release the lock in read mode.\n    rwlock_for_read<Clock>& for_read() {\n        return *this;\n    }\n\n    /// Cast this rwlock into write lock object with lock semantics appropriate to be used\n    /// by \"with_lock\". The resulting object will have lock / unlock calls that, when called,\n    /// will acquire / release the lock in write mode.\n    rwlock_for_write<Clock>& for_write() {\n        return *this;\n    }\n\n    /// Acquires this lock in read mode. Many readers are allowed, but when\n    /// this future returns, and until \\ref read_unlock is called, all fibers\n    /// waiting on \\ref write_lock are guaranteed not to execute.\n    future<> read_lock(typename semaphore_type::time_point timeout = semaphore_type::time_point::max()) {\n        return _sem.wait(timeout);\n    }\n\n    future<> read_lock(abort_source& as) {\n        return _sem.wait(as);\n    }\n\n    /// Releases the lock, which must have been taken in read mode. After this\n    /// is called, one of the fibers waiting on \\ref write_lock will be allowed\n    /// to proceed.\n    void read_unlock() {\n        SEASTAR_ASSERT(_sem.current() < max_ops);\n        _sem.signal();\n    }\n\n    /// Acquires this lock in write mode. Only one writer is allowed. When\n    /// this future returns, and until \\ref write_unlock is called, all other\n    /// fibers waiting on either \\ref read_lock or \\ref write_lock are guaranteed\n    /// not to execute.\n    future<> write_lock(typename semaphore_type::time_point timeout = semaphore_type::time_point::max()) {\n        return _sem.wait(timeout, max_ops);\n    }\n\n    future<> write_lock(abort_source& as) {\n        return _sem.wait(as, max_ops);\n    }\n\n    /// Releases the lock, which must have been taken in write mode. After this\n    /// is called, one of the other fibers waiting on \\ref write_lock or the fibers\n    /// waiting on \\ref read_lock will be allowed to proceed.\n    void write_unlock() {\n        SEASTAR_ASSERT(_sem.current() == 0);\n        _sem.signal(max_ops);\n    }\n\n    /// Tries to acquire the lock in read mode iff this can be done without waiting.\n    bool try_read_lock() {\n        return _sem.try_wait();\n    }\n\n    /// Tries to acquire the lock in write mode iff this can be done without waiting.\n    bool try_write_lock() {\n        return _sem.try_wait(max_ops);\n    }\n\n    using holder = semaphore_units<semaphore_default_exception_factory, Clock>;\n\n    /// hold_read_lock() waits for a read lock and returns an object which,\n    /// when destroyed, releases the lock. This makes it easy to ensure that\n    /// the lock is eventually undone, at any circumstance (even including\n    /// exceptions). The release() method can be used on the returned object\n    /// to release its ownership of the lock and avoid the automatic unlock.\n    /// Note that both hold_read_lock() and hold_write_lock() return an object\n    /// of the same type, rwlock::holder.\n    ///\n    /// hold_read_lock() may throw an exception (or, in other implementations,\n    /// return an exceptional future) when it failed to obtain the lock -\n    /// e.g., on allocation failure.\n    future<holder> hold_read_lock(typename semaphore_type::time_point timeout = semaphore_type::time_point::max()) {\n        return get_units(_sem, 1, timeout);\n    }\n\n    future<holder> hold_read_lock(abort_source& as) {\n        return get_units(_sem, 1, as);\n    }\n\n    /// try_hold_read_lock() synchronously tries to get a read lock and, if successful, returns an\n    /// optional object which, when destroyed, releases the lock.\n    std::optional<holder> try_hold_read_lock() noexcept {\n        return try_get_units(_sem, 1);\n    }\n\n    /// hold_write_lock() waits for a write lock and returns an object which,\n    /// when destroyed, releases the lock. This makes it easy to ensure that\n    /// the lock is eventually undone, at any circumstance (even including\n    /// exceptions). The release() method can be used on the returned object\n    /// to release its ownership of the lock and avoid the automatic unlock.\n    /// Note that both hold_read_lock() and hold_write_lock() return an object\n    /// of the same type, rwlock::holder.\n    ///\n    /// hold_read_lock() may throw an exception (or, in other implementations,\n    /// return an exceptional future) when it failed to obtain the lock -\n    /// e.g., on allocation failure.\n    future<holder> hold_write_lock(typename semaphore_type::time_point timeout = semaphore_type::time_point::max()) {\n        return get_units(_sem, max_ops, timeout);\n    }\n\n    future<holder> hold_write_lock(abort_source& as) {\n        return get_units(_sem, max_ops, as);\n    }\n\n    /// try_hold_write_lock() synchronously tries to get a write lock and, if successful, returns an\n    /// optional object which, when destroyed, releases the lock.\n    std::optional<holder> try_hold_write_lock() noexcept {\n        return try_get_units(_sem, max_ops);\n    }\n\n    /// Checks if any read or write locks are currently held.\n    bool locked() const {\n        return _sem.available_units() != max_ops;\n    }\n\n    friend class rwlock_for_read<Clock>;\n    friend class rwlock_for_write<Clock>;\n};\n\nusing rwlock = basic_rwlock<>;\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/scattered_message.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/deleter.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\ntemplate <typename CharType>\nclass\n[[deprecated(\"Use output_stream::write(span<temporary_buffer>) from API level 9\")]]\nscattered_message {\nprivate:\n    using fragment = net::fragment;\n    using packet = net::packet;\n    using char_type = CharType;\n    packet _p;\npublic:\n    scattered_message() {}\n    scattered_message(scattered_message&&) = default;\n    scattered_message(const scattered_message&) = delete;\n\n    void append_static(const char_type* buf, size_t size) {\n        if (size) {\n            _p = packet(std::move(_p), fragment{(char_type*)buf, size}, deleter());\n        }\n    }\n\n    template <size_t N>\n    void append_static(const char_type(&s)[N]) {\n        append_static(s, N - 1);\n    }\n\n    void append_static(const char_type* s) {\n        append_static(s, strlen(s));\n    }\n\n    template <typename size_type, size_type max_size>\n    void append_static(const basic_sstring<char_type, size_type, max_size>& s) {\n        append_static(s.begin(), s.size());\n    }\n\n    void append_static(const std::string_view& s) {\n        append_static(s.data(), s.size());\n    }\n\n    void append(std::string_view v) {\n        if (v.size()) {\n            _p = packet(std::move(_p), temporary_buffer<char>::copy_of(v));\n        }\n    }\n\n    void append(temporary_buffer<CharType> buff) {\n        if (buff.size()) {\n            _p = packet(std::move(_p), std::move(buff));\n        }\n    }\n\n    template <typename size_type, size_type max_size>\n    void append(basic_sstring<char_type, size_type, max_size> s) {\n        if (s.size()) {\n            _p = packet(std::move(_p), std::move(s).release());\n        }\n    }\n\n    template <typename size_type, size_type max_size, typename Callback>\n    void append(const basic_sstring<char_type, size_type, max_size>& s, Callback callback) {\n        if (s.size()) {\n            _p = packet(std::move(_p), fragment{s.begin(), s.size()}, make_deleter(std::move(callback)));\n        }\n    }\n\n    void reserve(int n_frags) {\n        _p.reserve(n_frags);\n    }\n\n    packet release() && {\n        return std::move(_p);\n    }\n\n    template <typename Callback>\n    void on_delete(Callback callback) {\n        _p = packet(std::move(_p), make_deleter(std::move(callback)));\n    }\n\n    operator bool() const {\n        return _p.len();\n    }\n\n    size_t size() {\n        return _p.len();\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/scheduling.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylla DB Ltd\n */\n\n#pragma once\n\n#include <chrono>\n#include <concepts>\n#include <functional>\n#include <typeindex>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/function_traits.hh>\n\n/// \\file\n\nnamespace seastar {\n\nconstexpr unsigned max_scheduling_groups() { return SEASTAR_SCHEDULING_GROUPS_COUNT; }\n\ntemplate <typename T = void>\nclass future;\n\nclass reactor;\n\nclass scheduling_group;\nclass scheduling_group_key;\nclass scheduling_supergroup;\n\nusing sched_clock = std::chrono::steady_clock;\n\nnamespace internal {\n\n// Returns an index between 0 and max_scheduling_groups()\nunsigned scheduling_group_index(scheduling_group sg) noexcept;\nscheduling_group scheduling_group_from_index(unsigned index) noexcept;\n\nunsigned long scheduling_group_key_id(scheduling_group_key) noexcept;\n\ntemplate<typename T>\nT* scheduling_group_get_specific_ptr(scheduling_group sg, scheduling_group_key key) noexcept;\n\nscheduling_supergroup scheduling_supergroup_for(scheduling_group sg) noexcept;\n\n}\n\n\n/// Creates a scheduling supergroup with a specified number of shares.\n///\n/// The operation is global and affects all shards. The returned scheduling\n/// supergroup can then be used in any shard.\n///\n/// \\param shares number of shares of the CPU time allotted to the group;\n///              Use numbers in the 1-1000 range (but can go above).\n/// \\return a scheduling supergroup that can be used on any shard\nfuture<scheduling_supergroup> create_scheduling_supergroup(float shares) noexcept;\n\n/// Destroys a scheduling supergroup.\n///\n/// Destroys a \\ref scheduling_supergroup previously created with create_scheduling_supergroup().\n/// The destroyed group must not be populated with sub-groups use and must not be used later.\n///\n/// The operation is global and affects all shards.\n///\n/// \\param sg The scheduling supergroup to be destroyed\n/// \\return a future that is ready when the scheduling supergroup has been torn down\nfuture<> destroy_scheduling_supergroup(scheduling_supergroup sg) noexcept;\n\n/// Creates a scheduling group with a specified number of shares.\n///\n/// The operation is global and affects all shards. The returned scheduling\n/// group can then be used in any shard.\n///\n/// \\param name A name that identifiers the group; will be used as a label\n///             in the group's metrics\n/// \\param shares number of shares of the CPU time allotted to the group;\n///              Use numbers in the 1-1000 range (but can go above).\n/// \\return a scheduling group that can be used on any shard\nfuture<scheduling_group> create_scheduling_group(sstring name, float shares) noexcept;\n\n/// Creates a scheduling group with a specified number of shares.\n///\n/// The operation is global and affects all shards. The returned scheduling\n/// group can then be used in any shard.\n///\n/// \\param name A name that identifies the group; will be used as a label\n///             in the group's metrics\n/// \\param shortname A name that identifies the group; will be printed in the\n///                  logging message aside of the shard id. please note, the\n///                  \\c shortname will be truncated to 4 characters.\n/// \\param shares number of shares of the CPU time allotted to the group;\n///              Use numbers in the 1-1000 range (but can go above).\n/// \\return a scheduling group that can be used on any shard\nfuture<scheduling_group> create_scheduling_group(sstring name, sstring shortname, float shares) noexcept;\n\n/// Creates a scheduling group with a specified number of shares.\n///\n/// The operation is global and affects all shards. The returned scheduling\n/// group can then be used in any shard.\n///\n/// \\param name A name that identifies the group; will be used as a label\n///             in the group's metrics\n/// \\param shortname A name that identifies the group; will be printed in the\n///                  logging message aside of the shard id. please note, the\n///                  \\c shortname will be truncated to 4 characters.\n/// \\param shares number of shares of the CPU time allotted to the group;\n///              Use numbers in the 1-1000 range (but can go above).\n/// \\param parent the supergroup to create the group in\n/// \\return a scheduling group that can be used on any shard\nfuture<scheduling_group> create_scheduling_group(sstring name, sstring shortname, float shares, scheduling_supergroup parent) noexcept;\n\n/// Destroys a scheduling group.\n///\n/// Destroys a \\ref scheduling_group previously created with create_scheduling_group().\n/// The destroyed group must not be currently in use and must not be used later.\n///\n/// The operation is global and affects all shards.\n///\n/// \\param sg The scheduling group to be destroyed\n/// \\return a future that is ready when the scheduling group has been torn down\nfuture<> destroy_scheduling_group(scheduling_group sg) noexcept;\n\n/// Rename scheduling group.\n///\n/// Renames a \\ref scheduling_group previously created with create_scheduling_group().\n///\n/// The operation is global and affects all shards.\n/// The operation affects the exported statistics labels.\n///\n/// \\param sg The scheduling group to be renamed\n/// \\param new_name The new name for the scheduling group.\n/// \\return a future that is ready when the scheduling group has been renamed\nfuture<> rename_scheduling_group(scheduling_group sg, sstring new_name) noexcept;\n/// Rename scheduling group.\n///\n/// Renames a \\ref scheduling_group previously created with create_scheduling_group().\n///\n/// The operation is global and affects all shards.\n/// The operation affects the exported statistics labels.\n///\n/// \\param sg The scheduling group to be renamed\n/// \\param new_name The new name for the scheduling group.\n/// \\param new_shortname The new shortname for the scheduling group.\n/// \\return a future that is ready when the scheduling group has been renamed\nfuture<> rename_scheduling_group(scheduling_group sg, sstring new_name, sstring new_shortname) noexcept;\n\n\n/**\n * Represents a configuration for a specific scheduling group value,\n * it contains all that is needed to maintain a scheduling group specific\n * value when it needs to be created, due to, for example, a new\n * \\ref scheduling_group being created.\n *\n * @note is is recomended to use @ref make_scheduling_group_key_config in order to\n * create and configure this syructure. The only reason that one might want to not use\n * this method is because of a need for specific intervention in the construction or\n * destruction of the value. Even then, it is recommended to first create the configuration\n * with @ref make_scheduling_group_key_config and only the change it.\n *\n */\nstruct scheduling_group_key_config {\n    /**\n     * Constructs a default configuration\n     */\n    scheduling_group_key_config() :\n        scheduling_group_key_config(typeid(void)) {}\n    /**\n     * Creates a configuration that is made for a specific type.\n     * It does not contain the right alignment and allocation sizes\n     * neither the correct construction or destruction logic, but only\n     * the indication for the intended type which is used in debug mode\n     * to make sure that the correct type is reffered to when accessing\n     * the value.\n     * @param type_info - the type information class (create with typeid(T)).\n     */\n    scheduling_group_key_config(const std::type_info& type_info) :\n            type_index(type_info) {}\n    /// The allocation size for the value (usually: sizeof(T))\n    size_t allocation_size;\n    /// The required alignment of the value (usually: alignof(T))\n    size_t alignment;\n    /// Holds the type information for debug mode runtime validation\n    std::type_index type_index;\n    /// A function that will be called for each newly allocated value\n    std::function<void (void*)> constructor;\n    /// A function that will be called for each value after the scheduling group is renamed.\n    std::function<void (void*)> rename;\n    /// A function that will be called for each element that is about\n    /// to be dealocated.\n    std::function<void (void*)> destructor;\n\n};\n\n\n/**\n * A class that is intended to encapsulate the scheduling group specific\n * key and \"hide\" it implementation concerns and details.\n *\n * @note this object can be copied accross shards and scheduling groups.\n */\nclass scheduling_group_key {\npublic:\n    /// The only user allowed operation on a key is copying.\n    scheduling_group_key(const scheduling_group_key&) noexcept = default;\n    scheduling_group_key(scheduling_group_key&&) noexcept = default;\nprivate:\n    scheduling_group_key(unsigned long id) noexcept :\n        _id(id) {}\n    unsigned long _id;\n    unsigned long id() const noexcept {\n        return _id;\n    }\n    friend future<scheduling_group_key> scheduling_group_key_create(scheduling_group_key_config cfg) noexcept;\n    template<typename T>\n    friend T* internal::scheduling_group_get_specific_ptr(scheduling_group sg, scheduling_group_key key) noexcept;\n    template<typename T>\n    friend T& scheduling_group_get_specific(scheduling_group_key key) noexcept;\n\n    friend unsigned long internal::scheduling_group_key_id(scheduling_group_key key) noexcept;\n};\n\nnamespace internal {\n\ninline unsigned long scheduling_group_key_id(scheduling_group_key key) noexcept {\n    return key.id();\n}\n\n/**\n * @brief A function in the spirit of Cpp17 apply, but specifically for constructors.\n * This function is used in order to preserve support in Cpp14.\n\n * @tparam ConstructorType - the constructor type or in other words the type to be constructed\n * @tparam Tuple - T params tuple type (should be deduced)\n * @tparam size_t...Idx - a sequence of indexes in order to access the typpels members in compile time.\n * (should be deduced)\n *\n * @param pre_alocated_mem - a pointer to the pre allocated memory chunk that will hold the\n * the initialized object.\n * @param args - A tupple that holds the prarameters for the constructor\n * @param idx_seq - An index sequence that will be used to access the members of the tuple in compile\n * time.\n *\n * @note this function was not intended to be called by users and it is only a utility function\n * for suporting \\ref make_scheduling_group_key_config\n */\ntemplate<typename ConstructorType, typename Tuple, size_t...Idx>\nvoid apply_constructor(void* pre_alocated_mem, Tuple args, std::index_sequence<Idx...>) {\n    new (pre_alocated_mem) ConstructorType(std::get<Idx>(args)...);\n}\n}\n\n/**\n * A template function that builds a scheduling group specific value configuration.\n * This configuration is used by the infrastructure to allocate memory for the values\n * and initialize or deinitialize them when they are created or destroyed.\n *\n * If the type T has a member function T::rename()\n * then it will be called after the scheduling group is renamed.\n *\n * @tparam T - the type for the newly created value.\n * @tparam ...ConstructorArgs - the types for the constructor parameters (should be deduced)\n * @param args - The parameters for the constructor.\n * @return a fully initialized \\ref scheduling_group_key_config object.\n */\ntemplate <typename T, typename... ConstructorArgs>\nscheduling_group_key_config\nmake_scheduling_group_key_config(ConstructorArgs... args) {\n    scheduling_group_key_config sgkc(typeid(T));\n    sgkc.allocation_size = sizeof(T);\n    sgkc.alignment = alignof(T);\n    sgkc.constructor = [args = std::make_tuple(args...)] (void* p) {\n        internal::apply_constructor<T>(p, args, std::make_index_sequence<sizeof...(ConstructorArgs)>());\n    };\n    sgkc.destructor = [] (void* p) {\n        static_cast<T*>(p)->~T();\n    };\n    if constexpr (requires(T key) { key.rename(); }) {\n        sgkc.rename = [] (void* p) {\n            static_cast<T*>(p)->rename();\n        };\n    }\n    return sgkc;\n}\n\n/**\n * Returns a future that holds a scheduling key and resolves when this key can be used\n * to access the scheduling group specific value it represents.\n * @param cfg - A \\ref scheduling_group_key_config object (by recomendation: initialized with\n * \\ref make_scheduling_group_key_config )\n * @return A future containing \\ref scheduling_group_key for the newly created specific value.\n */\nfuture<scheduling_group_key> scheduling_group_key_create(scheduling_group_key_config cfg) noexcept;\n\n/**\n * Returnes a reference to the given scheduling group specific value\n * @tparam T - the type of the scheduling specific type (cannot be deduced)\n * @param sg - the scheduling group which it's specific value to retrieve\n * @param key - the key of the value to retrieve.\n * @return A reference to the scheduling specific value.\n */\ntemplate<typename T>\nT& scheduling_group_get_specific(scheduling_group sg, scheduling_group_key key);\n\nclass scheduling_supergroup {\n    unsigned _id;\nprivate:\n    explicit scheduling_supergroup(unsigned index) noexcept : _id(index + 1) {}\npublic:\n    bool is_root() const noexcept { return _id == 0; }\n    unsigned index() const noexcept { return _id - 1; } // assumes that it's not root\n    /// Creates a `scheduling_supergroup` object denoting the root supergroup\n    scheduling_supergroup() noexcept : _id(0) {}\n    bool operator==(scheduling_supergroup x) const noexcept { return _id == x._id; }\n    bool operator!=(scheduling_supergroup x) const noexcept { return _id != x._id; }\n\n    friend future<scheduling_supergroup> create_scheduling_supergroup(float shares) noexcept;\n    friend scheduling_supergroup internal::scheduling_supergroup_for(scheduling_group sg) noexcept;\n    friend class reactor;\n\n    /// Adjusts the number of shares allotted to the supergroup.\n    ///\n    /// Dynamically adjust the number of shares allotted to the group, increasing or\n    /// decreasing the amount of CPU bandwidth it gets. The adjustment is local to\n    /// the shard.\n    ///\n    /// This can be used to reduce a background job's interference with a foreground\n    /// load: the shares can be started at a low value, increased when the background\n    /// job's backlog increases, and reduced again when the backlog decreases.\n    ///\n    /// \\param shares number of shares allotted to the group. Use numbers\n    ///               in the 1-1000 range.\n    void set_shares(float shares) noexcept;\n\n    /// Returns the number of shares the supergroup has\n    ///\n    /// Similarly to the \\ref set_shares, the returned value is only relevant to\n    /// the calling shard\n    float get_shares() const noexcept;\n\n    /// \\brief Updates the current IO bandwidth for a given scheduling supergroup\n    ///\n    /// The bandwidth applied is NOT shard-local, instead it is applied so that\n    /// all shards cannot consume more bytes-per-second altogether\n    ///\n    /// Must not be called on the root supergroup\n    ///\n    /// \\param bandwidth the new bandwidth value in bytes/second\n    /// \\return a future that is ready when the bandwidth update is applied\n    future<> update_io_bandwidth(uint64_t bandwidth) const;\n};\n\n/// \\brief Identifies function calls that are accounted as a group\n///\n/// A `scheduling_group` is a tag that can be used to mark a function call.\n/// Executions of such tagged calls are accounted as a group.\nclass scheduling_group {\n    unsigned _id;\nprivate:\n    explicit scheduling_group(unsigned id) noexcept : _id(id) {}\npublic:\n    /// Creates a `scheduling_group` object denoting the default group\n    constexpr scheduling_group() noexcept : _id(0) {} // must be constexpr for current_scheduling_group_holder\n    bool active() const noexcept;\n    const sstring& name() const noexcept;\n    const sstring& short_name() const noexcept;\n    bool operator==(scheduling_group x) const noexcept { return _id == x._id; }\n    bool operator!=(scheduling_group x) const noexcept { return _id != x._id; }\n    bool is_main() const noexcept { return _id == 0; }\n    bool is_at_exit() const noexcept { return _id == 1; }\n    template<typename T>\n    /**\n     * Returnes a reference to this scheduling group specific value\n     * @tparam T - the type of the scheduling specific type (cannot be deduced)\n     * @param key - the key of the value to retrieve.\n     * @return A reference to this scheduling specific value.\n     */\n    T& get_specific(scheduling_group_key key) noexcept {\n        return *internal::scheduling_group_get_specific_ptr<T>(*this, key);\n    }\n    /// Adjusts the number of shares allotted to the group.\n    ///\n    /// Dynamically adjust the number of shares allotted to the group, increasing or\n    /// decreasing the amount of CPU bandwidth it gets. The adjustment is local to\n    /// the shard.\n    ///\n    /// This can be used to reduce a background job's interference with a foreground\n    /// load: the shares can be started at a low value, increased when the background\n    /// job's backlog increases, and reduced again when the backlog decreases.\n    ///\n    /// \\param shares number of shares allotted to the group. Use numbers\n    ///               in the 1-1000 range.\n    void set_shares(float shares) noexcept;\n\n    /// Returns the number of shares the group has\n    ///\n    /// Similarly to the \\ref set_shares, the returned value is only relevant to\n    /// the calling shard\n    float get_shares() const noexcept;\n\n    /// \\brief Updates the current IO bandwidth for a given scheduling group\n    ///\n    /// The bandwidth applied is NOT shard-local, instead it is applied so that\n    /// all shards cannot consume more bytes-per-second altogether\n    ///\n    /// \\param bandwidth the new bandwidth value in bytes/second\n    /// \\return a future that is ready when the bandwidth update is applied\n    future<> update_io_bandwidth(uint64_t bandwidth) const;\n\n    friend class reactor;\n    friend unsigned internal::scheduling_group_index(scheduling_group sg) noexcept;\n    friend scheduling_group internal::scheduling_group_from_index(unsigned index) noexcept;\n\n    template<typename SpecificValType, typename Mapper, typename Reducer, typename Initial>\n    requires requires(SpecificValType specific_val, Mapper mapper, Reducer reducer, Initial initial) {\n        {reducer(initial, mapper(specific_val))} -> std::convertible_to<Initial>;\n    }\n    friend future<typename function_traits<Reducer>::return_type>\n    map_reduce_scheduling_group_specific(Mapper mapper, Reducer reducer, Initial initial_val, scheduling_group_key key);\n\n    template<typename SpecificValType, typename Reducer, typename Initial>\n    requires requires(SpecificValType specific_val, Reducer reducer, Initial initial) {\n        {reducer(initial, specific_val)} -> std::convertible_to<Initial>;\n    }\n    friend future<typename function_traits<Reducer>::return_type>\n        reduce_scheduling_group_specific(Reducer reducer, Initial initial_val, scheduling_group_key key);\n\n\n};\n\n/// \\cond internal\nnamespace internal {\n\ninline\nunsigned\nscheduling_group_index(scheduling_group sg) noexcept {\n    return sg._id;\n}\n\ninline\nscheduling_group\nscheduling_group_from_index(unsigned index) noexcept {\n    return scheduling_group(index);\n}\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nscheduling_group*\ncurrent_scheduling_group_ptr() noexcept;\n#else\ninline\nscheduling_group*\ncurrent_scheduling_group_ptr() noexcept {\n    // Slow unless constructor is constexpr\n    static thread_local scheduling_group sg;\n    return &sg;\n}\n#endif\n}\n/// \\endcond\n\n/// Returns the current scheduling group\ninline\nscheduling_group\ncurrent_scheduling_group() noexcept {\n    return *internal::current_scheduling_group_ptr();\n}\n\ninline\nscheduling_group\ndefault_scheduling_group() noexcept {\n    return scheduling_group();\n}\n\n\ninline\nbool\nscheduling_group::active() const noexcept {\n    return *this == current_scheduling_group();\n}\n\n}\n\nnamespace std {\n\ntemplate <>\nstruct hash<seastar::scheduling_group> {\n    size_t operator()(seastar::scheduling_group sg) const noexcept {\n        return seastar::internal::scheduling_group_index(sg);\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/scheduling_specific.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Scylla DB Ltd\n */\n\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/util/assert.hh>\n#include <array>\n#include <cstdlib>\n#include <map>\n#include <typeindex>\n#include <vector>\n#include <ranges>\n\n#pragma once\n\nnamespace seastar {\n\nnamespace internal {\n\nstruct scheduling_group_specific_thread_local_data {\n    using val_ptr = std::unique_ptr<void, void (*)(void*) noexcept>;\n    using cfg_ptr = lw_shared_ptr<scheduling_group_key_config>;\n\n    struct specific_val {\n        val_ptr valp;\n        cfg_ptr cfg;\n\n        static void free(void* ptr) noexcept {\n            std::free(ptr);\n        }\n\n        specific_val() : valp(nullptr, &free), cfg(nullptr) {}\n\n        specific_val(val_ptr&& valp_, const cfg_ptr& cfg_) : valp(std::move(valp_)), cfg(cfg_) {\n            if (valp && cfg->constructor) {\n                cfg->constructor(valp.get());\n            }\n        }\n\n        ~specific_val() {\n            if (valp && cfg->destructor) {\n                cfg->destructor(valp.get());\n            }\n        }\n\n        specific_val(const specific_val& other) = delete;\n        specific_val& operator=(const specific_val& other) = delete;\n\n        specific_val(specific_val&& other) : valp(std::move(other.valp)), cfg(std::move(other.cfg)) {}\n\n        specific_val& operator=(specific_val&& other) {\n            if (this != &other) {\n                valp = std::move(other.valp);\n                cfg = std::move(other.cfg);\n            }\n            return *this;\n        }\n\n        void* get() { return valp.get(); }\n\n        void rename() {\n            if (valp && cfg->rename) {\n                cfg->rename(valp.get());\n            }\n        }\n    };\n\n    struct per_scheduling_group {\n        bool queue_is_initialized = false;\n        /**\n         * This array holds pointers to the scheduling group specific\n         * data. The pointer is not use as is but is cast to a reference\n         * to the appropriate type that is actually pointed to.\n         */\n        std::vector<specific_val> specific_vals;\n\n        void rename() {\n            for (auto& v : specific_vals) {\n                v.rename();\n            }\n        }\n    };\n    std::array<per_scheduling_group, max_scheduling_groups()> per_scheduling_group_data;\n    std::map<unsigned long, cfg_ptr> scheduling_group_key_configs;\n};\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nscheduling_group_specific_thread_local_data** get_scheduling_group_specific_thread_local_data_ptr() noexcept;\n#else\ninline\nscheduling_group_specific_thread_local_data** get_scheduling_group_specific_thread_local_data_ptr() noexcept {\n    static thread_local scheduling_group_specific_thread_local_data* data;\n    return &data;\n}\n#endif\ninline\nscheduling_group_specific_thread_local_data& get_scheduling_group_specific_thread_local_data() noexcept {\n    return **get_scheduling_group_specific_thread_local_data_ptr();\n}\n\n[[noreturn]] void no_such_scheduling_group(scheduling_group sg);\n\n/**\n * Returns a pointer to the given scheduling group specific data.\n * @param sg - The scheduling group which it's data needs to be accessed\n * @param key - The scheduling group key that for the data to access\n * @return A pointer of type T* to the data, if sg is valid initialized.\n *\n * @note The parameter T has to be given since there is no way to deduce it.\n */\ntemplate<typename T>\nT* scheduling_group_get_specific_ptr(scheduling_group sg, scheduling_group_key key) noexcept {\n    auto& data = internal::get_scheduling_group_specific_thread_local_data();\n#ifdef SEASTAR_DEBUG\n    SEASTAR_ASSERT(std::type_index(typeid(T)) == data.scheduling_group_key_configs[key.id()]->type_index);\n#endif\n    auto sg_id = internal::scheduling_group_index(sg);\n    if (__builtin_expect(sg_id < data.per_scheduling_group_data.size() &&\n            data.per_scheduling_group_data[sg_id].queue_is_initialized, true)) {\n        return reinterpret_cast<T*>(data.per_scheduling_group_data[sg_id].specific_vals[key.id()].get());\n    }\n    return nullptr;\n}\n\n}\n\n\n/**\n * Returns a reference to the given scheduling group specific data.\n * @param sg - The scheduling group which it's data needs to be accessed\n * @param key - The scheduling group key that for the data to access\n * @return A reference of type T& to the data.\n *\n * @note The parameter T has to be given since there is no way to deduce it.\n *       May throw std::invalid_argument if sg does not exist or is uninitialized.\n */\ntemplate<typename T>\nT& scheduling_group_get_specific(scheduling_group sg, scheduling_group_key key) {\n    T* p = internal::scheduling_group_get_specific_ptr<T>(sg, std::move(key));\n    if (!p) {\n        internal::no_such_scheduling_group(sg);\n    }\n    return *p;\n}\n\n/**\n * Returns a reference to the current specific data.\n * @param key - The scheduling group key that for the data to access\n * @return A reference of type T& to the data.\n *\n * @note The parameter T has to be given since there is no way to deduce it.\n */\ntemplate<typename T>\nT& scheduling_group_get_specific(scheduling_group_key key) noexcept {\n    // Unlike internal::scheduling_group_get_specific_ptr, this can\n    // return a reference to an element whose queue_is_initialized is\n    // false.\n    auto& data = internal::get_scheduling_group_specific_thread_local_data();\n    SEASTAR_ASSERT(std::type_index(typeid(T)) == data.scheduling_group_key_configs[key.id()]->type_index);\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    return *reinterpret_cast<T*>(data.per_scheduling_group_data[sg_id].specific_vals[key.id()].get());\n}\n\n/**\n * A map reduce over all values of a specific scheduling group data.\n * @param mapper -  A functor SomeType(SpecificValType&) or SomeType(SpecificValType) that maps\n * the specific data to a value of any type.\n * @param reducer - A functor of of type ConvetibleToInitial(Initial, MapperReurnType) that reduces\n * a value of type Initial and of the mapper return type to a value of type convertible to Initial.\n * @param initial_val - the initial value to pass in the first call to the reducer.\n * @param key - the key to the specific data that the mapper should act upon.\n * @return A future that resolves when the result of the map reduce is ready.\n * @note The type of SpecificValType must be given because there is no way to deduce it in a *consistent*\n * manner.\n * @note Theoretically the parameter type of Mapper can be deduced to be the type (function_traits<Mapper>::arg<0>)\n * but then there is a danger when the Mapper accepts a parameter type T where SpecificValType is convertible to\n * SpecificValType.\n */\ntemplate<typename SpecificValType, typename Mapper, typename Reducer, typename Initial>\nrequires requires(SpecificValType specific_val, Mapper mapper, Reducer reducer, Initial initial) {\n    {reducer(initial, mapper(specific_val))} -> std::convertible_to<Initial>;\n}\nfuture<typename function_traits<Reducer>::return_type>\nmap_reduce_scheduling_group_specific(Mapper mapper, Reducer reducer,\n        Initial initial_val, scheduling_group_key key) {\n    using per_scheduling_group = internal::scheduling_group_specific_thread_local_data::per_scheduling_group;\n    auto& data = internal::get_scheduling_group_specific_thread_local_data();\n    auto wrapped_mapper = [key, mapper] (per_scheduling_group& psg) {\n        auto id = internal::scheduling_group_key_id(key);\n        return make_ready_future<typename function_traits<Mapper>::return_type>\n            (mapper(*reinterpret_cast<SpecificValType*>(psg.specific_vals[id].get())));\n    };\n\n    return map_reduce(\n            data.per_scheduling_group_data\n            | std::views::filter(std::mem_fn(&per_scheduling_group::queue_is_initialized)),\n            wrapped_mapper, std::move(initial_val), reducer);\n}\n\n/**\n * A reduce over all values of a specific scheduling group data.\n * @param reducer - A functor of of type ConvetibleToInitial(Initial, SpecificValType) that reduces\n * a value of type Initial and of the sg specific data type to a value of type convertible to Initial.\n * @param initial_val - the initial value to pass in the first call to the reducer.\n * @param key - the key to the specific data that the mapper should act upon.\n * @return A future that resolves when the result of the reduce is ready.\n * * @note The type of SpecificValType must be given because there is no way to deduce it in a *consistent*\n * manner.\n * @note Theoretically the parameter type of Reducer can be deduced to be the type (function_traits<Reducer>::arg<0>)\n * but then there is a danger when the Reducer accepts a parameter type T where SpecificValType is convertible to\n * SpecificValType.\n */\ntemplate<typename SpecificValType, typename Reducer, typename Initial>\nrequires requires(SpecificValType specific_val, Reducer reducer, Initial initial) {\n    {reducer(initial, specific_val)} -> std::convertible_to<Initial>;\n}\nfuture<typename function_traits<Reducer>::return_type>\nreduce_scheduling_group_specific(Reducer reducer, Initial initial_val, scheduling_group_key key) {\n    using per_scheduling_group = internal::scheduling_group_specific_thread_local_data::per_scheduling_group;\n    auto& data = internal::get_scheduling_group_specific_thread_local_data();\n\n    auto mapper = [key] (per_scheduling_group& psg) {\n        auto id = internal::scheduling_group_key_id(key);\n        return make_ready_future<SpecificValType>(*reinterpret_cast<SpecificValType*>(psg.specific_vals[id].get()));\n    };\n\n    return map_reduce(\n            data.per_scheduling_group_data\n            | std::views::filter(std::mem_fn(&per_scheduling_group::queue_is_initialized)),\n            mapper, std::move(initial_val), reducer);\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/scollectd.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <concepts>\n#include <type_traits>\n#include <utility>\n#include <functional>\n#include <array>\n#include <iterator>\n#include <stdint.h>\n#include <memory>\n#include <string>\n#include <tuple>\n#include <chrono>\n\n#include <seastar/core/future.hh>\n#include <seastar/net/byteorder.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/program-options.hh>\n#include <seastar/core/metrics_api.hh>\n\nnamespace seastar {\n\n/**\n * Implementation of rudimentary collectd data gathering.\n *\n * Usage is hopefully straight forward. Though, feel free to read\n * https://collectd.org/wiki/index.php/Naming_schema\n * for an explanation on the naming model.\n *\n * Typically, you'll add values something like:\n *\n * \t\tscollectd::type_instance_id typ(\"<pluginname>\", \"<instance_name>\", \"<type_name>\", \"<instance_name>\");\n *\t\tscollectd::add_polled_metric(typ, [<metric var> | scollectd::make_typed(<data_type>, <metric_var>) [, ...]);\n *\n * Where\n * \t`<pluginname>` would be the overall 'module', e.g. \"cpu\"\n *  `<instance_name>` -> optional distinguisher between plugin instances. For cpu, the built-in\n *  scollectd::per_cpu_plugin_instance constant is a good choice, i.e. 0->N cpu.\n *  If there are no instances (e.g. only one), empty constant is appropriate (none)\n *  `<type_name>` is the 'type' of metric collected, for ex. \"usage\" (cpu/0/usage)\n *  `<type_instance>` is a distinguisher for metric parts of the type, e.g. \"idle\", \"user\", \"kernel\"\n *  -> cpu/0/usage/idle | cpu/0/usage/user | cpu/0/usage/kernel\n *\n *  Each type instance can bind an arbitrary number of values, ech representing some aspect in turn of the instance.\n *  The structure and interpretation is up to the producer/consumer\n *\n * There is a single \"scollectd\" instance per cpu, and values should be bound locally\n * to this cpu. Polling is done at a frequency set in the seastar config (def once per s),\n * and all registered values will be sent via UDP packages to the destination host(s)\n *\n * Note that the tuple { plugin, plugin_instance, type, type_instance } is considered a\n * unique ID for a value registration, so using the same tuple twice will remove the previously\n * registered values.\n *\n * Values can be unregistered at any time, though they must be so on the same thread/cpu\n * as they we're registered. The \"registration\" achor type provides RAII style value unregistration\n * semantics.\n *\n */\n\nnamespace scollectd {\n\nextern seastar::logger logger;\n\nusing data_type = seastar::metrics::impl::data_type;\n\nenum class known_type {\n    // from types.db. Defined collectd types (type_id) selection.\n    // This enum omits the very application specific types, such\n    // as mysql_* etc, since if you really are re-writing mysql\n    // in seastar, you probably know how to look the type up manually...\n\n    absolute,\n    backends,\n    bitrate,\n    blocked_clients,\n    bytes,\n    cache_eviction,\n    cache_operation,\n    cache_ratio,\n    cache_result,\n    cache_size,\n    capacity,\n    changes_since_last_save,\n    charge,\n    clock_last_meas,\n    clock_last_update,\n    clock_mode,\n    clock_reachability,\n    clock_skew_ppm,\n    clock_state,\n    clock_stratum,\n    compression,\n    compression_ratio,\n    connections,\n    conntrack,\n    contextswitch,\n    count,\n    counter,\n    cpu,\n    cpufreq,\n    current,\n    current_connections,\n    current_sessions,\n    delay,\n    derive,\n    df,\n    df_complex,\n    df_inodes,\n    disk_io_time,\n    disk_latency,\n    disk_merged,\n    disk_octets,\n    disk_ops,\n    disk_ops_complex,\n    disk_time,\n    dns_answer,\n    dns_notify,\n    dns_octets,\n    dns_opcode,\n    dns_qtype,\n    dns_qtype_cached,\n    dns_query,\n    dns_question,\n    dns_rcode,\n    dns_reject,\n    dns_request,\n    dns_resolver,\n    dns_response,\n    dns_transfer,\n    dns_update,\n    dns_zops,\n    drbd_resource,\n    duration,\n    email_check,\n    email_count,\n    email_size,\n    entropy,\n    evicted_keys,\n    expired_keys,\n    fanspeed,\n    file_handles,\n    file_size,\n    files,\n    flow,\n    fork_rate,\n    frequency,\n    frequency_error,\n    frequency_offset,\n    fscache_stat,\n    gauge,\n    hash_collisions,\n    http_request_methods,\n    http_requests,\n    http_response_codes,\n    humidity,\n    if_collisions,\n    if_dropped,\n    if_errors,\n    if_multicast,\n    if_octets,\n    if_packets,\n    if_rx_errors,\n    if_rx_octets,\n    if_tx_errors,\n    if_tx_octets,\n    invocations,\n    io_octets,\n    io_packets,\n    ipt_bytes,\n    ipt_packets,\n    irq,\n    latency,\n    links,\n    load,\n    md_disks,\n    memory,\n    memory_lua,\n    memory_throttle_count,\n    multimeter,\n    mutex_operations,\n    objects,\n    operations,\n    packets,\n    pending_operations,\n    percent,\n    percent_bytes,\n    percent_inodes,\n    ping,\n    ping_droprate,\n    ping_stddev,\n    players,\n    power,\n    pressure,\n    protocol_counter,\n    pubsub,\n    queue_length,\n    records,\n    requests,\n    response_code,\n    response_time,\n    root_delay,\n    root_dispersion,\n    route_etx,\n    route_metric,\n    routes,\n    segments,\n    serial_octets,\n    signal_noise,\n    signal_power,\n    signal_quality,\n    snr,\n    spl,\n    swap,\n    swap_io,\n    tcp_connections,\n    temperature,\n    threads,\n    time_dispersion,\n    time_offset,\n    time_offset_ntp,\n    time_offset_rms,\n    time_ref,\n    timeleft,\n    total_bytes,\n    total_connections,\n    total_objects,\n    total_operations,\n    total_requests,\n    total_sessions,\n    total_threads,\n    total_time_in_ms,\n    total_values,\n    uptime,\n    users,\n    vcl,\n    vcpu,\n    virt_cpu_total,\n    virt_vcpu,\n    vmpage_action,\n    vmpage_faults,\n    vmpage_io,\n    vmpage_number,\n    volatile_changes,\n    voltage,\n    voltage_threshold,\n    vs_memory,\n    vs_processes,\n    vs_threads,\n};\n\n// don't use directly. use make_typed.\ntemplate<typename T>\nstruct typed {\n    typed(data_type t, T && v)\n    : type(t), value(std::forward<T>(v)) {\n    }\n    data_type type;\n    T value;\n};\n\ntemplate<typename T>\ninline typed<T> make_typed(data_type type, T&& t) {\n    return typed<T>(type, std::forward<T>(t));\n}\n\nusing plugin_id = seastar::metrics::group_name_type;\nusing plugin_instance_id = seastar::metrics::instance_id_type;\nusing type_id = seastar::metrics::metric_type_def;\nusing type_instance = seastar::metrics::metric_name_type;\n\ntype_id type_id_for(known_type);\n\nusing description = seastar::metrics::description;\n\nconstexpr inline unsigned max_collectd_field_text_len = 63;\n\nclass type_instance_id {\n    static thread_local unsigned _next_truncated_idx;\n\n    /// truncate a given field to the maximum allowed length\n    void truncate(sstring& field, const char* field_desc);\npublic:\n    type_instance_id() = default;\n    type_instance_id(plugin_id p, plugin_instance_id pi, type_id t,\n                    scollectd::type_instance ti = std::string())\n                    : _plugin(std::move(p)), _plugin_instance(std::move(pi)), _type(\n                                    std::move(t)), _type_instance(std::move(ti)) {\n        // truncate strings to the maximum allowed length\n        truncate(_plugin, \"plugin\");\n        truncate(_plugin_instance, \"plugin_instance\");\n        truncate(_type, \"type\");\n        truncate(_type_instance, \"type_instance\");\n    }\n    type_instance_id(const seastar::metrics::impl::metric_id &id, const type_id& inherit_type) : _plugin(id.group_name()),\n            _plugin_instance(id.instance_id()), _type(inherit_type),\n            _type_instance(id.name()) {\n    }\n    type_instance_id(type_instance_id &&) = default;\n    type_instance_id(const type_instance_id &) = default;\n\n    type_instance_id & operator=(type_instance_id &&) = default;\n    type_instance_id & operator=(const type_instance_id &) = default;\n\n    const plugin_id & plugin() const {\n        return _plugin;\n    }\n    const plugin_instance_id & plugin_instance() const {\n        return _plugin_instance;\n    }\n    const type_id & type() const {\n        return _type;\n    }\n    const scollectd::type_instance & type_instance() const {\n        return _type_instance;\n    }\n    bool operator<(const type_instance_id&) const;\n    bool operator==(const type_instance_id&) const;\nprivate:\n    plugin_id _plugin;\n    plugin_instance_id _plugin_instance;\n    type_id _type;\n    scollectd::type_instance _type_instance;\n};\n\nextern const plugin_instance_id per_cpu_plugin_instance;\n\n// Scollectd configuration options.\nstruct options : public program_options::option_group {\n    /// \\brief Enable collectd daemon.\n    ///\n    /// Default: \\p false.\n    program_options::value<bool> collectd;\n    /// \\brief Address to send/broadcast metrics to.\n    ///\n    /// Default: \\p 239.192.74.66:25826.\n    program_options::value<std::string> collectd_address;\n    /// \\brief Poll period (ms).\n    ///\n    /// Frequency of sending counter metrics (0 disables).\n    /// Default: \\p 1000.\n    program_options::value<unsigned> collectd_poll_period;\n    /// \\deprecated use \\ref metrics::options::metrics_hostname instead\n    program_options::value<std::string> collectd_hostname;\n\n    /// \\cond internal\n    options(program_options::option_group* parent_group);\n    /// \\endcond\n};\n\nvoid configure(const options&);\nvoid remove_polled_metric(const type_instance_id &);\n\nclass plugin_instance_metrics;\n\n/**\n * Anchor for polled registration.\n * Iff the registered type is in some way none-persistent,\n * use this as receiver of the reg and ensure it dies before the\n * added value(s).\n *\n * Use:\n * uint64_t v = 0;\n * registration r = add_polled_metric(v);\n * ++r;\n * <scope end, above dies>\n */\nstruct registration {\n    registration() = default;\n    registration(const type_instance_id& id);\n    registration(type_instance_id&& id);\n    registration(const registration&) = delete;\n    registration(registration&&) = default;\n    ~registration();\n    registration & operator=(const registration&) = delete;\n    registration & operator=(registration&&) = default;\n\n    void unregister() {\n        remove_polled_metric(_id);\n        _id = type_instance_id();\n    }\nprivate:\n    friend class plugin_instance_metrics;\n    type_instance_id _id;\n    shared_ptr<seastar::metrics::impl::impl> _impl;\n};\n\n/**\n * Helper type to make generating vectors of registration objects\n * easier, since it constructs from an initializer list of\n * type_instance_id:s, avoiding early conversion to registration objs,\n * which in case of init lists, are copy semantics, not move...\n */\nclass registrations\n    : public std::vector<registration>\n{\npublic:\n    typedef std::vector<registration> vector_type;\n\n    registrations()\n    {}\n    registrations(vector_type&& v) : vector_type(std::move(v))\n    {}\n    registrations(const std::initializer_list<type_instance_id>& l)\n        : vector_type(l.begin(),l.end())\n    {}\n    registrations& operator=(vector_type&& v) {\n        vector_type::operator=(std::move(v));\n        return *this;\n    }\n    registrations& operator=(const std::initializer_list<type_instance_id>& l) {\n        return registrations::operator=(registrations(l));\n    }\n};\n\nclass value_list;\n\nstruct typed_value {\n    /**\n     * Wraps N values of a given type (type_id).\n     * Used to group types into a plugin_instance_metrics\n     */\n    template<typename... Args>\n    typed_value(const type_id& tid, const scollectd::type_instance& ti, description, Args&&... args);\n\n    template<typename... Args>\n    typed_value(const type_id& tid, const scollectd::type_instance& ti, Args&&... args)\n        : typed_value(tid, ti, description(), std::forward<Args>(args)...)\n    {}\n\n    const scollectd::type_instance& type_instance() const {\n        return _type_instance;\n    }\n    const shared_ptr<value_list>& values() const {\n        return _values;\n    }\n    const type_id & type() const {\n        return _type_id;\n    }\nprivate:\n    type_id _type_id;\n    scollectd::type_instance _type_instance;\n    shared_ptr<value_list> _values;\n};\n\nclass plugin_instance_metrics {\npublic:\n    template<typename... TypedValues>\n    plugin_instance_metrics(const plugin_id& p, const plugin_instance_id& pi, TypedValues&&... values)\n        : _plugin_id(p)\n        , _plugin_instance(pi)\n        , _registrations({ add_impl(values)... })\n    {}\n    std::vector<type_instance_id> bound_ids() const;\n    void add(const typed_value&);\nprivate:\n    type_instance_id add_impl(const typed_value&);\n\n    plugin_id _plugin_id;\n    plugin_instance_id _plugin_instance;\n    registrations _registrations;\n};\n\n/**\n * Simplified wrapper for the common case of per-cpu plugin instances\n * (i.e. distributed objects)\n */\nclass percpu_plugin_instance_metrics : public plugin_instance_metrics {\npublic:\n    template<typename... TypedValues>\n    percpu_plugin_instance_metrics(const plugin_id& p, TypedValues&&... values)\n        : plugin_instance_metrics(p, per_cpu_plugin_instance, std::forward<TypedValues>(values)...)\n    {}\n};\n\n/**\n * Template wrapper for type_id values, deriving type_id string\n * from the known_types enum, for auto-completetion joy.\n */\ntemplate<known_type Type>\nstruct typed_value_impl: public typed_value {\n    template<typename ... Args>\n    typed_value_impl(const scollectd::type_instance& ti, Args&& ... args)\n        : typed_value(type_id_for(Type), ti, std::forward<Args>(args)...)\n    {}\n\n    template<typename ... Args>\n    typed_value_impl(scollectd::type_instance ti, description d, Args&& ... args)\n        : typed_value(type_id_for(Type), std::move(ti), std::move(d), std::forward<Args>(args)...)\n    {}\n    template<typename ... Args>\n    typed_value_impl(description d, Args&& ... args)\n        : typed_value(type_id_for(Type), scollectd::type_instance(), std::move(d), std::forward<Args>(args)...)\n    {}\n};\n\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n * Some typedefs for common used types. Feel free to add.\n */\ntypedef typed_value_impl<known_type::total_bytes> total_bytes;\ntypedef typed_value_impl<known_type::total_connections> total_connections;\ntypedef typed_value_impl<known_type::total_objects> total_objects;\ntypedef typed_value_impl<known_type::total_operations> total_operations;\ntypedef typed_value_impl<known_type::total_requests> total_requests;\ntypedef typed_value_impl<known_type::total_sessions> total_sessions;\ntypedef typed_value_impl<known_type::total_threads> total_threads;\ntypedef typed_value_impl<known_type::total_time_in_ms> total_time_in_ms;\ntypedef typed_value_impl<known_type::total_values> total_values;\ntypedef typed_value_impl<known_type::queue_length> queue_length;\ntypedef typed_value_impl<known_type::counter> counter;\ntypedef typed_value_impl<known_type::count> count;\ntypedef typed_value_impl<known_type::gauge> gauge;\n\n// lots of template junk to build typed value list tuples\n// for registered values.\ntemplate<typename T>\nstruct data_type_for;\n\n\ntemplate<std::unsigned_integral T>\nstruct data_type_for<T> : public std::integral_constant<data_type, data_type::COUNTER> {\n};\n\ntemplate<std::floating_point T>\nstruct data_type_for<T> : public std::integral_constant<data_type, data_type::GAUGE> {\n};\n\ntemplate<std::invocable Func>\nstruct data_type_for<Func> : public data_type_for<std::invoke_result_t<Func>> {\n};\n\ntemplate<typename T>\nstruct data_type_for<typed<T>> : public data_type_for<T> {\n};\n\ntemplate<typename T>\nclass value {\npublic:\n    template<typename W>\n    struct wrap {\n        wrap(const W & v)\n        : _v(v) {\n        }\n        const W & operator()() const {\n            return _v;\n        }\n        const W & _v;\n    };\n\n    typedef std::remove_reference_t<T> value_type;\n    typedef std::conditional_t<\n            std::invocable<T>,\n            value_type, wrap<value_type> > stored_type;\n\n    value(const value_type & t)\n    : value<T>(data_type_for<value_type>::value, t) {\n    }\n    value(data_type type, const value_type & t)\n    : _type(type), _t(t) {\n    }\n    uint64_t operator()() const {\n        auto v = _t();\n        if (_type == data_type::GAUGE) {\n            return convert(double(v));\n        } else {\n            uint64_t u = v;\n            return convert(u);\n        }\n    }\n    operator uint64_t() const {\n        return (*this)();\n    }\n    operator data_type() const {\n        return _type;\n    }\n    data_type type() const {\n        return _type;\n    }\nprivate:\n    // not super quick value -> protocol endian 64-bit values.\n    template<typename _Iter>\n    void bpack(_Iter s, _Iter e, uint64_t v) const {\n        while (s != e) {\n            *s++ = (v & 0xff);\n            v >>= 8;\n        }\n    }\n    template<std::integral V>\n    uint64_t convert(\n            V v) const {\n        uint64_t i = v;\n        // network byte order\n        return ntohq(i);\n    }\n    template<std::floating_point V>\n    uint64_t convert(\n            V t) const {\n        union {\n            uint64_t i;\n            double v;\n        } v;\n        union {\n            uint64_t i;\n            uint8_t b[8];\n        } u;\n        v.v = t;\n        // intel byte order. could also obviously be faster.\n        // could be ignored if we just assume we're le (for now),\n        // but this is ok me thinks.\n        bpack(std::begin(u.b), std::end(u.b), v.i);\n        return u.i;\n    }\n    ;\n\n    const data_type _type;\n    const stored_type _t;\n};\n\ntemplate<typename T>\nclass value<typed<T>> : public value<T> {\npublic:\n    value(const typed<T> & args)\n: value<T>(args.type, args.value) {\n    }\n};\n\nclass value_list {\n    bool _enabled = true;\npublic:\n    value_list(description d) : _description(std::move(d))\n    {}\n    value_list(value_list&&) = default;\n    virtual ~value_list() {}\n\n    virtual size_t size() const = 0;\n\n    virtual void types(data_type *) const = 0;\n    virtual void values(net::packed<uint64_t> *) const = 0;\n\n    const description& desc() const {\n        return _description;\n    }\n\n    bool empty() const {\n        return size() == 0;\n    }\n\n    bool is_enabled() const {\n        return _enabled;\n    }\n\n    void set_enabled(bool b) {\n        _enabled = b;\n    }\nprivate:\n    description _description;\n};\n\ntemplate<typename ... Args>\nclass values_impl: public value_list {\npublic:\n    static const size_t num_values = sizeof...(Args);\n\n    values_impl(description d, Args&& ...args)\n        : value_list(std::move(d))\n        , _values(std::forward<Args>(args)...)\n    {}\n\n    values_impl(values_impl<Args...>&& a) = default;\n    values_impl(const values_impl<Args...>& a) = default;\n\n    size_t size() const override {\n        return num_values;\n    }\n    void types(data_type * p) const override {\n        unpack(_values, [p](Args... args) {\n            std::initializer_list<data_type> tmp = { args...  };\n            std::copy(tmp.begin(), tmp.end(), p);\n        });\n    }\n    void values(net::packed<uint64_t> * p) const override {\n        unpack(_values, [p](Args... args) {\n            std::initializer_list<uint64_t> tmp = { args...  };\n            std::copy(tmp.begin(), tmp.end(), p);\n        });\n    }\nprivate:\n    template<typename _Op>\n    void unpack(const std::tuple<Args...>& t, _Op&& op) const {\n        do_unpack(t, std::index_sequence_for<Args...> {}, std::forward<_Op>(op));\n    }\n\n    template<size_t ...S, typename _Op>\n    void do_unpack(const std::tuple<Args...>& t, const std::index_sequence<S...> &, _Op&& op) const {\n        op(std::get<S>(t)...);\n    }\n\n    std::tuple < Args... > _values;\n};\n\nvoid add_polled(const type_instance_id &, const shared_ptr<value_list> &, bool enabled = true);\n\ntypedef std::function<void()> notify_function;\ntemplate<typename... _Args>\nstatic auto make_type_instance(description d, _Args && ... args) -> values_impl < decltype(value<_Args>(std::forward<_Args>(args)))... >\n{\n    return values_impl<decltype(value<_Args>(std::forward<_Args>(args)))...>(\n                    std::move(d), value<_Args>(std::forward<_Args>(args))...);\n}\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n */\ntemplate<typename ... _Args>\n[[deprecated(\"Use the metrics layer\")]] static type_instance_id add_polled_metric(const plugin_id & plugin,\n        const plugin_instance_id & plugin_instance, const type_id & type,\n        const scollectd::type_instance & type_instance, _Args&& ... args) {\n    return add_polled_metric(plugin, plugin_instance, type, type_instance, description(),\n            std::forward<_Args>(args)...);\n}\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n */\ntemplate<typename ... _Args>\n[[deprecated(\"Use the metrics layer\")]] static type_instance_id add_polled_metric(const plugin_id & plugin,\n        const plugin_instance_id & plugin_instance, const type_id & type,\n        const scollectd::type_instance & type_instance, description d, _Args&& ... args) {\n    return add_polled_metric(\n            type_instance_id(plugin, plugin_instance, type, type_instance), std::move(d),\n            std::forward<_Args>(args)...);\n}\ntemplate<typename ... _Args>\nstatic future<> send_explicit_metric(const plugin_id & plugin,\n        const plugin_instance_id & plugin_instance, const type_id & type,\n        const scollectd::type_instance & type_instance, _Args&& ... args) {\n    return send_explicit_metric(\n            type_instance_id(plugin, plugin_instance, type, type_instance),\n            std::forward<_Args>(args)...);\n}\ntemplate<typename ... _Args>\nstatic notify_function create_explicit_metric(const plugin_id & plugin,\n        const plugin_instance_id & plugin_instance, const type_id & type,\n        const scollectd::type_instance & type_instance, _Args&& ... args) {\n    return create_explicit_metric(\n            type_instance_id(plugin, plugin_instance, type, type_instance),\n            std::forward<_Args>(args)...);\n}\n\nseastar::metrics::impl::metric_id to_metrics_id(const type_instance_id & id);\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n */\ntemplate<typename Arg>\n[[deprecated(\"Use the metrics layer\")]] static type_instance_id add_polled_metric(const type_instance_id & id, description d,\n        Arg&& arg, bool enabled = true) {\n    seastar::metrics::impl::get_local_impl()->add_registration(to_metrics_id(id), arg.type, seastar::metrics::impl::make_function(arg.value, arg.type), d, enabled);\n    return id;\n}\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n */\ntemplate<typename Arg>\n[[deprecated(\"Use the metrics layer\")]] static type_instance_id add_polled_metric(const type_instance_id & id,\n        Arg&& arg) {\n    return std::move(add_polled_metric(id, description(), std::forward<Arg>(arg)));\n}\n\n/*!\n * \\deprecated metrics registration should be done using the metrics layer\n *\n */\ntemplate<typename Args>\n[[deprecated(\"Use the metrics layer\")]] static type_instance_id add_disabled_polled_metric(const type_instance_id & id, description d,\n        Args&& arg) {\n    return add_polled_metric(id, d, std::forward<Args>(arg), false);\n}\n\ntemplate<typename Args>\nstatic type_instance_id add_disabled_polled_metric(const type_instance_id & id,\n        Args&& args) {\n    return add_disabled_polled_metric(id, description(), std::forward<Args>(args));\n}\n\ntemplate<typename ... Args>\nstatic type_instance_id add_disabled_polled_metric(const type_instance_id & id,\n        Args&& ... args) {\n    return add_disabled_polled_metric(id, description(), std::forward<Args>(args)...);\n}\n\n// \"Explicit\" metric sends. Sends a single value list as a message.\n// Obviously not super efficient either. But maybe someone needs it sometime.\ntemplate<typename ... _Args>\nstatic future<> send_explicit_metric(const type_instance_id & id,\n        _Args&& ... args) {\n    return send_metric(id, make_type_instance(std::forward<_Args>(args)...));\n}\ntemplate<typename ... _Args>\nstatic notify_function create_explicit_metric(const type_instance_id & id,\n        _Args&& ... args) {\n    auto list = make_type_instance(std::forward<_Args>(args)...);\n    return [id, list=std::move(list)]() {\n        send_metric(id, list);\n    };\n}\n\ntemplate<typename... Args>\ntyped_value::typed_value(const type_id& tid, const scollectd::type_instance& ti, description d, Args&&... args)\n    : _type_id(tid)\n    , _type_instance(ti)\n    , _values(::seastar::make_shared<decltype(make_type_instance(std::move(d), std::forward<Args>(args)...))>(make_type_instance(std::move(d), std::forward<Args>(args)...)))\n{}\n\n// Send a message packet (string)\nfuture<> send_notification(const type_instance_id & id, const sstring & msg);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/scollectd_api.hh",
    "content": "/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/core/scollectd.hh>\n#include <seastar/core/metrics_api.hh>\n#include <vector>\n\nnamespace seastar {\n\nnamespace scollectd {\n\n\nusing collectd_value  = seastar::metrics::impl::metric_value;\n\nstd::vector<collectd_value> get_collectd_value(\n        const scollectd::type_instance_id& id);\n\nstd::vector<scollectd::type_instance_id> get_collectd_ids();\n\nsstring get_collectd_description_str(const scollectd::type_instance_id&);\n\nbool is_enabled(const scollectd::type_instance_id& id);\n/**\n * Enable or disable collectd metrics on local instance\n * @param id - the metric to enable or disable\n * @param enable - should the collectd metrics be enable or disable\n */\nvoid enable(const scollectd::type_instance_id& id, bool enable);\n\n\nmetrics::impl::value_map get_value_map();\n\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/seastar.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n/// \\mainpage\n///\n/// Seastar is a high performance C++ application framework for high\n/// concurrency server applications.\n///\n/// A good place to start is the [Tutorial](tutorial.html) or [Multi-page version](split/).\n///\n/// Please see:\n///   - \\ref future-module Documentation on futures and promises, which are\n///          the seastar building blocks.\n///   - \\ref future-util Utililty functions for working with futures\n///   - \\ref memory-module Memory management\n///   - \\ref networking-module TCP/IP networking\n///   - \\ref fileio-module File Input/Output\n///   - \\ref smp-module Multicore support\n///   - \\ref fiber-module Utilities for managing loosely coupled chains of\n///          continuations, also known as fibers\n///   - \\ref thread-module Support for traditional threaded execution\n///   - \\ref rpc Build high-level communication protocols\n///   - \\ref websocket (experimental) Implement a WebSocket-based server\n///   - \\ref fsnotifier (experimental) Implement a filesystem modification notifier.\n///\n/// View the [Seastar compatibility statement](./md_compatibility.html) for\n/// information about library evolution.\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/file-types.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/util/bool_class.hh>\n#include <seastar/util/std-compat.hh>\n#include \"./internal/api-level.hh\"\n#include <cstdint>\n#include <filesystem>\n#include <optional>\n#include <string_view>\n\nnamespace seastar {\n\n\n// iostream.hh\ntemplate <class CharType> class input_stream;\ntemplate <class CharType> class output_stream;\n\nclass server_socket;\nclass socket;\nclass connected_socket;\nclass socket_address;\nstruct listen_options;\nenum class transport;\n\n// file.hh\nclass file;\nstruct file_open_options;\nstruct stat_data;\n\nnamespace net {\n\nusing udp_channel = class datagram_channel;\n\n}\n\nnamespace experimental {\n// process.hh\nclass process;\nstruct spawn_parameters;\n}\n\n// Networking API\n\n/// \\defgroup networking-module Networking\n///\n/// Seastar provides a simple networking API, backed by two\n/// TCP/IP stacks: the POSIX stack, utilizing the kernel's\n/// BSD socket APIs, and the native stack, implement fully\n/// within seastar and able to drive network cards directly.\n/// The native stack supports zero-copy on both transmit\n/// and receive, and is implemented using seastar's high\n/// performance, lockless sharded design.  The network stack\n/// can be selected with the \\c \\--network-stack command-line\n/// parameter.\n\n/// \\addtogroup networking-module\n/// @{\n\n/// Listen for connections on a given port\n///\n/// Starts listening on a given address for incoming connections.\n///\n/// \\param sa socket address to listen on\n///\n/// \\return \\ref server_socket object ready to accept connections.\n///\n/// \\see listen(socket_address sa, listen_options opts)\nserver_socket listen(socket_address sa);\n\n/// Listen for connections on a given port\n///\n/// Starts listening on a given address for incoming connections.\n///\n/// \\param sa socket address to listen on\n/// \\param opts options controlling the listen operation\n///\n/// \\return \\ref server_socket object ready to accept connections.\n///\n/// \\see listen(socket_address sa)\nserver_socket listen(socket_address sa, listen_options opts);\n\n/// Establishes a connection to a given address\n///\n/// Attempts to connect to the given address.\n///\n/// \\param sa socket address to connect to\n///\n/// \\return a \\ref connected_socket object, or an exception\nfuture<connected_socket> connect(socket_address sa);\n\n/// Establishes a connection to a given address\n///\n/// Attempts to connect to the given address with a defined local endpoint\n///\n/// \\param sa socket address to connect to\n/// \\param local socket address for local endpoint\n/// \\param proto transport protocol (TCP or SCTP)\n///\n/// \\return a \\ref connected_socket object, or an exception\nfuture<connected_socket> connect(socket_address sa, socket_address local, transport proto);\n\n\n/// Creates a socket object suitable for establishing stream-oriented connections\n///\n/// \\return a \\ref socket object that can be used for establishing connections\nsocket make_socket();\n\n/// Creates a udp_channel object suitable for sending UDP packets\n///\n/// The channel is not bound to a local address, and thus can only be used\n/// for sending.\n///\n/// \\return a \\ref net::udp_channel object that can be used for UDP transfers.\n[[deprecated(\"Use `make_unbound_datagram_channel` instead\")]]\nnet::udp_channel make_udp_channel();\n\n\n/// Creates a udp_channel object suitable for sending and receiving UDP packets\n///\n/// \\param local local address to bind to\n///\n/// \\return a \\ref net::udp_channel object that can be used for UDP transfers.\n[[deprecated(\"Use `make_bound_datagram_channel` instead\")]]\nnet::udp_channel make_udp_channel(const socket_address& local);\n\n/// Creates a datagram_channel object suitable for sending datagrams to\n/// destinations that belong to the provided address family.\n/// Supported address families: AF_INET, AF_INET6 and AF_UNIX.\n///\n/// Setting family to AF_INET or AF_INET6 creates a datagram_channel that uses\n/// UDP protocol. AF_UNIX creates a datagram_channel that uses UNIX domain\n/// sockets.\n///\n/// The channel is not bound to a local address, and thus can only be used\n/// for sending.\n///\n/// \\param family address family in which the \\ref datagram_channel will operate\n///\n/// \\return a \\ref net::datagram_channel object for sending datagrams in a\n/// specified address family.\nnet::datagram_channel make_unbound_datagram_channel(sa_family_t family);\n\n/// Creates a datagram_channel object suitable for sending and receiving\n/// datagrams to/from destinations that belong to the provided address family.\n/// Supported address families: AF_INET, AF_INET6 and AF_UNIX.\n///\n/// \\param local local address to bind to\n///\n/// \\return a \\ref net::datagram_channel object for sending/receiving datagrams\n/// in a specified address family.\nnet::datagram_channel make_bound_datagram_channel(const socket_address& local);\n\n/// @}\n\n/// \\defgroup fileio-module File Input/Output\n///\n/// Seastar provides a file API to deal with persistent storage.\n/// Unlike most file APIs, seastar offers unbuffered file I/O\n/// (similar to, and based on, \\c O_DIRECT).  Unbuffered I/O means\n/// that the application is required to do its own caching, but\n/// delivers better performance if this caching is done correctly.\n///\n/// For random I/O or sequential unbuffered I/O, the \\ref file\n/// class provides a set of methods for reading, writing, discarding,\n/// or otherwise manipulating a file.  For buffered sequential I/O,\n/// see \\ref make_file_input_stream() and \\ref make_file_output_stream().\n\n/// \\addtogroup fileio-module\n/// @{\n\n/// Opens or creates a file.  The \"dma\" in the name refers to the fact\n/// that data transfers are unbuffered and uncached.\n///\n/// \\param name  the name of the file to open or create\n/// \\param flags various flags controlling the open process\n/// \\return a \\ref file object, as a future\n///\n/// \\note\n/// The file name is not guaranteed to be stable on disk, unless the\n/// containing directory is sync'ed.\n///\n/// \\relates file\nfuture<file> open_file_dma(std::string_view name, open_flags flags) noexcept;\n\n/// Opens or creates a file.  The \"dma\" in the name refers to the fact\n/// that data transfers are unbuffered and uncached.\n///\n/// \\param name  the name of the file to open or create\n/// \\param flags various flags controlling the open process\n/// \\param options options for opening the file\n/// \\return a \\ref file object, as a future\n///\n/// \\note\n/// The file name is not guaranteed to be stable on disk, unless the\n/// containing directory is sync'ed.\n///\n/// \\relates file\nfuture<file> open_file_dma(std::string_view name, open_flags flags, file_open_options options) noexcept;\n\n/// Checks if a given directory supports direct io\n///\n/// Seastar bypasses the Operating System caches and issues direct io to the\n/// underlying block devices. Projects using seastar should check if the directory\n/// lies in a filesystem that support such operations. This function can be used\n/// to do that.\n///\n/// It will return if direct io can be used, or throw an std::system_error\n/// exception, with the EINVAL error code.\n///\n/// A std::system_error with the respective error code is also thrown if \\c path is\n/// not a directory.\n///\n/// \\param path the directory we need to verify.\nfuture<> check_direct_io_support(std::string_view path) noexcept;\n\n/// Opens a directory.\n///\n/// \\param name name of the directory to open\n///\n/// \\return a \\ref file object representing a directory.  The only\n///    legal operations are \\ref file::list_directory(),\n///    \\ref file::flush(), and \\ref file::close().\n///\n/// \\relates file\nfuture<file> open_directory(std::string_view name) noexcept;\n\n/// Creates a new directory.\n///\n/// \\param name name of the directory to create\n/// \\param permissions optional file permissions of the directory to create.\n///\n/// \\note\n/// The directory is not guaranteed to be stable on disk, unless the\n/// containing directory is sync'ed.\nfuture<> make_directory(std::string_view name, file_permissions permissions = file_permissions::default_dir_permissions) noexcept;\n\n/// Ensures a directory exists\n///\n/// Checks whether a directory exists, and if not, creates it.  Only\n/// the last component of the directory name is created.\n///\n/// \\param name name of the directory to potentially create\n/// \\param permissions optional file permissions of the directory to create.\n///\n/// \\note\n/// The directory is not guaranteed to be stable on disk, unless the\n/// containing directory is sync'ed.\n/// If the directory exists, the provided permissions are not applied.\nfuture<> touch_directory(std::string_view name, file_permissions permissions = file_permissions::default_dir_permissions) noexcept;\n\n/// Recursively ensures a directory exists\n///\n/// Checks whether each component of a directory exists, and if not, creates it.\n///\n/// \\param name name of the directory to potentially create\n/// \\param permissions optional file permissions of the directory to create.\n///\n/// \\note\n/// This function fsyncs each component created, and is therefore guaranteed to be stable on disk.\n/// The provided permissions are applied only on the last component in the path, if it needs to be created,\n/// if intermediate directories do not exist, they are created with the default_dir_permissions.\n/// If any directory exists, the provided permissions are not applied.\nfuture<> recursive_touch_directory(std::string_view name, file_permissions permissions = file_permissions::default_dir_permissions) noexcept;\n\n/// Synchronizes a directory to disk\n///\n/// Makes sure the modifications in a directory are synchronized in disk.\n/// This is useful, for instance, after creating or removing a file inside the\n/// directory.\n///\n/// \\param name name of the directory to potentially create\nfuture<> sync_directory(std::string_view name) noexcept;\n\n\n/// Removes (unlinks) a file or an empty directory\n///\n/// \\param name name of the file or the directory to remove\n///\n/// \\note\n/// The removal is not guaranteed to be stable on disk, unless the\n/// containing directory is sync'ed.\nfuture<> remove_file(std::string_view name) noexcept;\n\n/// Renames (moves) a file.\n///\n/// \\param old_name existing file name\n/// \\param new_name new file name\n///\n/// \\note\n/// The rename is not guaranteed to be stable on disk, unless the\n/// both containing directories are sync'ed.\nfuture<> rename_file(std::string_view old_name, std::string_view new_name) noexcept;\n\nstruct follow_symlink_tag { };\nusing follow_symlink = bool_class<follow_symlink_tag>;\n\n/// Return stat information about a file.\n///\n/// \\param name name of the file to return its stat information\n/// \\param fs a follow_symlink flag to follow symbolic links.\n///\n/// \\return stat_data of the file identified by name.\n/// If name identifies a symbolic link then stat_data is returned either for the target of the link,\n/// with follow_symlink::yes, or for the link itself, with follow_symlink::no.\nfuture<stat_data> file_stat(std::string_view name, follow_symlink fs = follow_symlink::yes) noexcept;\n\n/// Return stat information about a file in a directory.\n///\n/// \\param directory a open directory\n/// \\param name name of the file to return its stat information\n/// \\param fs a follow_symlink flag to follow symbolic links.\n///\n/// \\return stat_data of the file identified by name.\n///\n/// If the pathname given in \\c name is relative, then it is interpreted relative to the open \\c directory.\n/// If pathname given in \\c name is absolute, then \\c directory is ignored.\n///\n/// If name identifies a symbolic link then stat_data is returned either for the target of the link,\n/// with follow_symlink::yes, or for the link itself, with follow_symlink::no.\nfuture<stat_data> file_stat(file& directory, std::string_view name, follow_symlink fs = follow_symlink::yes) noexcept;\n\n/// Wrapper around getgrnam_r.\n/// If the provided group name does not exist in the group database, this call will return an empty optional.\n/// If the provided group name exists in the group database, the optional returned will contain the struct group_details information.\n/// When an unexpected error is thrown by the getgrnam_r libc call, this function throws std::system_error with std::error_code.\n/// \\param groupname name of the group\n///\n/// \\return optional struct group_details of the group identified by name. struct group_details has details of the group from the group database.\nfuture<std::optional<struct group_details>> getgrnam(std::string_view name);\n\n/// Change the owner and group of file. This is a wrapper around chown syscall.\n/// The function throws std::system_error, when the chown syscall fails.\n/// \\param filepath\n/// \\param owner\n/// \\param group\nfuture<> chown(std::string_view filepath, uid_t owner, gid_t group);\n/// Return the size of a file.\n///\n/// \\param name name of the file to return the size\n///\n/// Note that file_size of a symlink is NOT the size of the symlink -\n/// which is the length of the pathname it contains -\n/// but rather the size of the file to which it points.\nfuture<uint64_t> file_size(std::string_view name) noexcept;\n\n/// Check file access.\n///\n/// \\param name name of the file to check\n/// \\param flags bit pattern containing type of access to check (read/write/execute or exists).\n///\n/// If only access_flags::exists is queried, returns true if the file exists, or false otherwise.\n/// Throws a std::filesystem::filesystem_error exception if any error other than ENOENT is encountered.\n///\n/// If any of the access_flags (read/write/execute) is set, returns true if the file exists and is\n/// accessible with the requested flags, or false if the file exists and is not accessible\n/// as queried.\n/// Throws a std::filesystem::filesystem_error exception if any error other than EACCES is encountered.\n/// Note that if any path component leading to the file is not searchable, the file is considered inaccessible\n/// with the requested mode and false will be returned.\nfuture<bool> file_accessible(std::string_view name, access_flags flags) noexcept;\n\n/// check if a file exists.\n///\n/// \\param name name of the file to check\nfuture<bool> file_exists(std::string_view name) noexcept;\n\n/// Determine the type of a file (regular file, directory, etc.)\n///\n/// \\param name name of the file for which type information is requested\n/// \\param follow a follow_symlink flag that determines whether a trailing symbolic link should be followed or not\n///\n/// \\return a engaged optional with the file type if lookup was successful; a disengaged optional\n///      if the file (or one of its parent directories) does not exist; an exceptional future on\n///      other errors.\nfuture<std::optional<directory_entry_type>> file_type(std::string_view name, follow_symlink follow = follow_symlink::yes) noexcept;\n\n\n/// Creates a hard link for a file\n///\n/// \\param oldpath existing file name\n/// \\param newpath name of link\n///\nfuture<> link_file(std::string_view oldpath, std::string_view newpath) noexcept;\n\n/// Changes the permissions mode of a file or directory\n///\n/// \\param name name of the file ot directory to change\n/// \\param permissions permissions to set\n///\nfuture<> chmod(std::string_view name, file_permissions permissions) noexcept;\n\n/// Return information about the filesystem where a file is located.\n///\n/// \\param name name of the file to inspect\nfuture<fs_type> file_system_at(std::string_view name) noexcept;\n\n/// Return space available to unprivileged users in filesystem where a file is located, in bytes.\n///\n/// \\param name name of the file to inspect\nfuture<uint64_t> fs_avail(std::string_view name) noexcept;\n\n/// Return free space in filesystem where a file is located, in bytes.\n///\n/// \\param name name of the file to inspect\nfuture<uint64_t> fs_free(std::string_view name) noexcept;\n/// @}\n\n/// Return filesystem-wide space_info where a file is located.\n///\n/// \\param name name of the file in the filesystem to inspect\nfuture<std::filesystem::space_info> file_system_space(std::string_view name) noexcept;\n\nnamespace experimental {\n/// \\defgroup interprocess-module Interprocess Communication\n///\n/// Seastar provides a set of APIs for interprocess communicate\n\n/// \\addtogroup interprocess-module\n/// @{\n\n/// Create a pipe using \\c pipe2\n///\n/// \\return a tuple of \\c file_desc, the first one for reading from the pipe, the second\n/// for writing to it.\nfuture<std::tuple<file_desc, file_desc>> make_pipe();\n\n/// Spawn a subprocess\n///\n/// \\param pathname the path to the executable\n/// \\param params parameters for spawning the subprocess\n///\n/// \\return a process representing the spawned subprocess\n/// \\note\n/// the subprocess is spawned with \\c posix_spawn() system call, so the\n/// pathname should be relative or absolute path of the executable.\nfuture<process> spawn_process(const std::filesystem::path& pathname,\n                              spawn_parameters params);\n/// Spawn a subprocess\n///\n/// \\param pathname the path to the executable\n///\n/// \\return a process representing the spawned subprocess\n/// \\note\n/// the this overload does not specify a \\c params parameters for spawning the\n/// subprocess. Instead, it uses the pathname for the \\c argv[0] in the params.\nfuture<process> spawn_process(const std::filesystem::path& pathname);\n/// @}\n}\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/semaphore.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include \"future.hh\"\n#include <seastar/core/future.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/abortable_fifo.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <seastar/core/abort_on_expiry.hh>\n#include <seastar/util/assert.hh>\n#include <exception>\n#include <optional>\n#include <stdexcept>\n#include <utility>\n\nnamespace seastar {\n\nnamespace internal {\n// Test if a class T has member function broken()\ntemplate <typename T>\nclass has_broken {\n    template <typename U> constexpr static bool check(decltype(&U::broken)) { return true; }\n    template <typename U> constexpr static bool check(...) { return false; }\n\npublic:\n    constexpr static bool value = check<T>(nullptr);\n};\n// Test if a class T has member function aborted()\ntemplate <typename T>\nclass has_aborted {\n    template <typename U> constexpr static bool check(decltype(&U::aborted)) { return true; }\n    template <typename U> constexpr static bool check(...) { return false; }\n\npublic:\n    constexpr static bool value = check<T>(nullptr);\n};\n}\n\n/// \\addtogroup fiber-module\n/// @{\n/// Exception thrown when a semaphore is broken by\n/// \\ref semaphore::broken().\nclass broken_semaphore : public std::exception {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept;\n};\n\n/// Exception thrown when a semaphore wait operation\n/// times out.\n///\n/// \\see semaphore::wait(typename timer<>::duration timeout, size_t nr)\nclass semaphore_timed_out : public timed_out_error {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept;\n};\n\n/// Exception thrown when a semaphore wait operation\n/// was aborted.\n///\n/// \\see semaphore::wait(abort_source&, size_t nr)\nclass semaphore_aborted : public abort_requested_exception {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept;\n};\n\n/// Exception Factory for standard semaphore\n///\n/// constructs standard semaphore exceptions\n/// \\see semaphore_timed_out and broken_semaphore\nstruct semaphore_default_exception_factory {\n    static semaphore_timed_out timeout() noexcept;\n    static broken_semaphore broken() noexcept;\n    static semaphore_aborted aborted() noexcept;\n};\n\nclass named_semaphore_timed_out : public semaphore_timed_out {\n    sstring _msg;\npublic:\n    named_semaphore_timed_out(std::string_view msg) noexcept;\n    virtual const char* what() const noexcept;\n};\n\nclass broken_named_semaphore : public broken_semaphore {\n    sstring _msg;\npublic:\n    broken_named_semaphore(std::string_view msg) noexcept;\n    virtual const char* what() const noexcept;\n};\n\nclass named_semaphore_aborted : public semaphore_aborted {\n    sstring _msg;\npublic:\n    named_semaphore_aborted(std::string_view msg) noexcept;\n    virtual const char* what() const noexcept;\n};\n\n// A factory of semaphore exceptions that contain additional context: the semaphore name\n// auto sem = named_semaphore(0, named_semaphore_exception_factory{\"file_opening_limit_semaphore\"});\nstruct named_semaphore_exception_factory {\n    sstring name;\n    named_semaphore_timed_out timeout() const noexcept;\n    broken_named_semaphore broken() const noexcept;\n    named_semaphore_aborted aborted() const noexcept;\n};\n\n/// \\brief Counted resource guard.\n///\n/// This is a standard computer science semaphore, adapted\n/// for futures.  You can deposit units into a counter,\n/// or take them away.  Taking units from the counter may wait\n/// if not enough units are available.\n///\n/// To support exceptional conditions, a \\ref broken() method\n/// is provided, which causes all current waiters to stop waiting,\n/// with an exceptional future returned.  This allows causing all\n/// fibers that are blocked on a semaphore to continue.  This is\n/// similar to POSIX's `pthread_cancel()`, with \\ref wait() acting\n/// as a cancellation point.\n///\n/// \\tparam ExceptionFactory template parameter allows modifying a semaphore to throw\n/// customized exceptions on timeout/broken/aborted. It has to provide three functions:\n/// ExceptionFactory::timeout() - that returns a \\ref semaphore_timed_out exception by default,\n/// ExceptionFactory::broken() - that returns a \\ref broken_semaphore exception by default, and\n/// ExceptionFactory::aborted() - that returns a \\ref semaphore_aborted exception by default.\ntemplate<typename ExceptionFactory, typename Clock = typename timer<>::clock>\nclass basic_semaphore : private ExceptionFactory {\npublic:\n    using duration = typename timer<Clock>::duration;\n    using clock = typename timer<Clock>::clock;\n    using time_point = typename timer<Clock>::time_point;\n    using exception_factory = ExceptionFactory;\nprivate:\n    ssize_t _count;\n    std::exception_ptr _ex;\n    struct entry {\n        promise<> pr;\n        size_t nr;\n        std::optional<abort_on_expiry<clock>> timer;\n        entry(promise<>&& pr_, size_t nr_) noexcept : pr(std::move(pr_)), nr(nr_) {}\n    };\n    std::exception_ptr get_timeout_exception() {\n        try {\n            return std::make_exception_ptr(this->timeout());\n        } catch (...) {\n            return std::make_exception_ptr(semaphore_timed_out());\n        }\n    }\n    std::exception_ptr get_aborted_exception() {\n        if constexpr (internal::has_aborted<exception_factory>::value) {\n            try {\n                return std::make_exception_ptr(this->aborted());\n            } catch (...) {\n                return std::make_exception_ptr(semaphore_aborted());\n            }\n        } else {\n            return std::make_exception_ptr(semaphore_aborted());\n        }\n    }\n    struct expiry_handler {\n        basic_semaphore& sem;\n        void operator()(entry& e, const std::optional<std::exception_ptr>& ex) noexcept {\n            if (e.timer) {\n                e.pr.set_exception(sem.get_timeout_exception());\n            } else if (ex) {\n                e.pr.set_exception(*ex);\n            } else if (sem._ex) {\n                e.pr.set_exception(sem._ex);\n            } else {\n                e.pr.set_exception(sem.get_aborted_exception());\n            }\n        }\n    };\n    internal::abortable_fifo<entry, expiry_handler> _wait_list;\n\n#ifdef SEASTAR_SEMAPHORE_DEBUG\n    struct used_flag {\n        // set to true from the wait path\n        // prevents the semaphore from being moved or move-reassigned when _used\n        bool _used = false;\n\n        used_flag() = default;\n        used_flag(used_flag&& o) noexcept {\n            SEASTAR_ASSERT(!_used && \"semaphore cannot be moved after it has been used\");\n        }\n        used_flag& operator=(used_flag&& o) noexcept {\n            if (this != &o) {\n                SEASTAR_ASSERT(!_used && !o._used && \"semaphore cannot be moved after it has been used\");\n            }\n            return *this;\n        }\n        void use() noexcept {\n            _used = true;\n        }\n    };\n#else\n    struct used_flag {\n        void use() noexcept {}\n    };\n#endif\n\n    [[no_unique_address]] used_flag _used;\n\n    bool has_available_units(size_t nr) const noexcept {\n        return _count >= 0 && (static_cast<size_t>(_count) >= nr);\n    }\n    bool may_proceed(size_t nr) const noexcept {\n        return has_available_units(nr) && _wait_list.empty();\n    }\npublic:\n    /// Returns the maximum number of units the semaphore counter can hold\n    static constexpr size_t max_counter() noexcept {\n        return std::numeric_limits<decltype(_count)>::max();\n    }\n\n    /// Constructs a semaphore object with a specific number of units\n    /// in its internal counter. E.g., starting it at 1 is suitable for use as\n    /// an unlocked mutex.\n    ///\n    /// \\param count number of initial units present in the counter.\n    basic_semaphore(size_t count) noexcept(std::is_nothrow_default_constructible_v<exception_factory>)\n        : exception_factory()\n        , _count(count),\n        _wait_list(expiry_handler{*this})\n    {}\n    basic_semaphore(size_t count, exception_factory&& factory) noexcept(std::is_nothrow_move_constructible_v<exception_factory>)\n        : exception_factory(std::move(factory))\n        , _count(count)\n        , _wait_list(expiry_handler{*this})\n    {\n        static_assert(std::is_nothrow_move_constructible_v<expiry_handler>);\n    }\n\n    /// Move-constructs a semaphore object from a moved-from semaphore object,\n    /// inheriting the number of units from the moved-from semaphore.\n    /// The moved-from semaphore must be unused.\n    ///\n    /// \\param other the moved-from semaphore object.\n    basic_semaphore(basic_semaphore&& other) noexcept(std::is_nothrow_move_constructible_v<exception_factory>)\n        : exception_factory(other)\n        , _count(other._count)\n        , _ex(std::exchange(other._ex, std::exception_ptr()))\n        , _wait_list(expiry_handler{*this})\n        , _used(std::move(other._used))\n    {\n        // semaphore cannot be moved with non-empty waiting list\n        SEASTAR_ASSERT(other._wait_list.empty());\n    }\n\n    /// Move-assigns a semaphore object from a moved-from semaphore object,\n    /// inheriting the number of units from the moved-from semaphore.\n    /// The number of units of the assigned semaphore is overwritten by the\n    /// the moved-from number of units.\n    /// Both the moved-to and moved-from semaphores must be unused.\n    ///\n    /// \\param other the moved-from semaphore object.\n    basic_semaphore& operator=(basic_semaphore&& other) noexcept(std::is_nothrow_move_assignable_v<exception_factory>) {\n        // semaphore cannot be moved with non-empty waiting list\n        SEASTAR_ASSERT(_wait_list.empty());\n        SEASTAR_ASSERT(other._wait_list.empty());\n        if (this != &other) {\n            exception_factory::operator=(other);\n            _count = other._count;\n            _ex = std::exchange(other._ex, std::exception_ptr());\n            _used = std::move(other._used);\n        }\n        return *this;\n    }\n\n    /// Waits until at least a specific number of units are available in the\n    /// counter, and reduces the counter by that amount of units.\n    ///\n    /// \\note Waits are serviced in FIFO order, though if several are awakened\n    ///       at once, they may be reordered by the scheduler.\n    ///\n    /// \\param nr Amount of units to wait for (default 1).\n    /// \\return a future that becomes ready when sufficient units are available\n    ///         to satisfy the request.  If the semaphore was \\ref broken(), may\n    ///         contain an exception.\n    future<> wait(size_t nr = 1) noexcept {\n        return wait(time_point::max(), nr);\n    }\n    /// Waits until at least a specific number of units are available in the\n    /// counter, and reduces the counter by that amount of units.  If the request\n    /// cannot be satisfied in time, the request is aborted.\n    ///\n    /// \\note Waits are serviced in FIFO order, though if several are awakened\n    ///       at once, they may be reordered by the scheduler.\n    ///\n    /// \\param timeout expiration time.\n    /// \\param nr Amount of units to wait for (default 1).\n    /// \\return a future that becomes ready when sufficient units are available\n    ///         to satisfy the request.  On timeout, the future contains a\n    ///         \\ref semaphore_timed_out exception.  If the semaphore was\n    ///         \\ref broken(), may contain a \\ref broken_semaphore exception.\n    future<> wait(time_point timeout, size_t nr = 1) noexcept {\n        _used.use();\n        if (may_proceed(nr)) {\n            _count -= nr;\n            return make_ready_future<>();\n        }\n        if (_ex) {\n            return make_exception_future(_ex);\n        }\n        if (Clock::now() >= timeout) [[unlikely]] {\n            return make_exception_future(get_timeout_exception());\n        }\n        try {\n            entry& e = _wait_list.emplace_back(promise<>(), nr);\n            auto f = e.pr.get_future();\n            if (timeout != time_point::max()) {\n                e.timer.emplace(timeout);\n                abort_source& as = e.timer->abort_source();\n                _wait_list.make_back_abortable(as);\n            }\n            return f;\n        } catch (...) {\n            return make_exception_future(std::current_exception());\n        }\n    }\n\n    /// Waits until at least a specific number of units are available in the\n    /// counter, and reduces the counter by that amount of units.  The request\n    /// can be aborted using an \\ref abort_source.\n    ///\n    /// \\note Waits are serviced in FIFO order, though if several are awakened\n    ///       at once, they may be reordered by the scheduler.\n    ///\n    /// \\param as abort source.\n    /// \\param nr Amount of units to wait for (default 1).\n    /// \\return a future that becomes ready when sufficient units are available\n    ///         to satisfy the request.  On abort, the future contains a\n    ///         \\ref semaphore_aborted exception.  If the semaphore was\n    ///         \\ref broken(), may contain a \\ref broken_semaphore exception.\n    future<> wait(abort_source& as, size_t nr = 1) noexcept {\n        _used.use();\n        if (may_proceed(nr)) {\n            _count -= nr;\n            return make_ready_future<>();\n        }\n        if (_ex) {\n            return make_exception_future(_ex);\n        }\n        if (as.abort_requested()) [[unlikely]] {\n            return make_exception_future(get_aborted_exception());\n        }\n        try {\n            entry& e = _wait_list.emplace_back(promise<>(), nr);\n            // taking future here since make_back_abortable may expire the entry\n            auto f = e.pr.get_future();\n            _wait_list.make_back_abortable(as);\n            return f;\n        } catch (...) {\n            return make_exception_future(std::current_exception());\n        }\n    }\n\n    /// Waits until at least a specific number of units are available in the\n    /// counter, and reduces the counter by that amount of units.  If the request\n    /// cannot be satisfied in time, the request is aborted.\n    ///\n    /// \\note Waits are serviced in FIFO order, though if several are awakened\n    ///       at once, they may be reordered by the scheduler.\n    ///\n    /// \\param timeout how long to wait.\n    /// \\param nr Amount of units to wait for (default 1).\n    /// \\return a future that becomes ready when sufficient units are available\n    ///         to satisfy the request.  On timeout, the future contains a\n    ///         \\ref semaphore_timed_out exception.  If the semaphore was\n    ///         \\ref broken(), may contain a \\ref broken_semaphore exception.\n    future<> wait(duration timeout, size_t nr = 1) noexcept {\n        return wait(clock::now() + timeout, nr);\n    }\n    /// Deposits a specified number of units into the counter.\n    ///\n    /// The counter is incremented by the specified number of units.\n    /// If the new counter value is sufficient to satisfy the request\n    /// of one or more waiters, their futures (in FIFO order) become\n    /// ready, and the value of the counter is reduced according to\n    /// the amount requested.\n    ///\n    /// \\param nr Number of units to deposit (default 1).\n    void signal(size_t nr = 1) noexcept {\n        if (_ex) {\n            return;\n        }\n        _count += nr;\n        while (!_wait_list.empty() && has_available_units(_wait_list.front().nr)) {\n            auto& x = _wait_list.front();\n            _count -= x.nr;\n            x.pr.set_value();\n            _wait_list.pop_front();\n        }\n    }\n\n    /// Consume the specific number of units without blocking\n    //\n    /// Consume the specific number of units now, regardless of how many units are available\n    /// in the counter, and reduces the counter by that amount of units. This operation may\n    /// cause the counter to go negative.\n    ///\n    /// \\param nr Amount of units to consume (default 1).\n    void consume(size_t nr = 1) noexcept {\n        _used.use();\n        if (_ex) {\n            return;\n        }\n        _count -= nr;\n    }\n\n    /// Attempts to reduce the counter value by a specified number of units.\n    ///\n    /// If sufficient units are available in the counter, and if no\n    /// other fiber is waiting, then the counter is reduced.  Otherwise,\n    /// nothing happens.  This is useful for \"opportunistic\" waits where\n    /// useful work can happen if the counter happens to be ready, but\n    /// when it is not worthwhile to wait.\n    ///\n    /// \\param nr number of units to reduce the counter by (default 1).\n    /// \\return `true` if the counter had sufficient units, and was decremented.\n    bool try_wait(size_t nr = 1) noexcept {\n        _used.use();\n        if (may_proceed(nr)) {\n            _count -= nr;\n            return true;\n        } else {\n            return false;\n        }\n    }\n    /// Returns the number of units available in the counter.\n    ///\n    /// Does not take into account any waiters.\n    size_t current() const noexcept { return std::max(_count, ssize_t(0)); }\n\n    /// Returns the number of available units.\n    ///\n    /// Takes into account units consumed using \\ref consume() and therefore\n    /// may return a negative value.\n    ssize_t available_units() const noexcept { return _count; }\n\n    /// Returns the current number of waiters\n    size_t waiters() const noexcept { return _wait_list.size(); }\n\n    /// Signal to waiters that an error occurred.  \\ref wait() will see\n    /// an exceptional future<> containing a \\ref broken_semaphore exception.\n    /// The future is made available immediately.\n    void broken() noexcept {\n        std::exception_ptr ep;\n        if constexpr (internal::has_broken<exception_factory>::value) {\n            try {\n                ep = std::make_exception_ptr(exception_factory::broken());\n            } catch (...) {\n                ep = std::make_exception_ptr(broken_semaphore());\n            }\n        } else {\n            ep = std::make_exception_ptr(broken_semaphore());\n        }\n        broken(std::move(ep));\n    }\n\n    /// Signal to waiters that an error occurred.  \\ref wait() will see\n    /// an exceptional future<> containing the provided exception parameter.\n    /// The future is made available immediately.\n    template <typename Exception>\n    void broken(const Exception& ex) noexcept {\n        broken(std::make_exception_ptr(ex));\n    }\n\n    /// Signal to waiters that an error occurred.  \\ref wait() will see\n    /// an exceptional future<> containing the provided exception parameter.\n    /// The future is made available immediately.\n    void broken(std::exception_ptr ex) noexcept;\n\n    /// Reserve memory for waiters so that wait() will not throw.\n    void ensure_space_for_waiters(size_t n) {\n        _wait_list.reserve(n);\n    }\n};\n\ntemplate<typename ExceptionFactory, typename Clock>\ninline\nvoid\nbasic_semaphore<ExceptionFactory, Clock>::broken(std::exception_ptr xp) noexcept {\n    static_assert(std::is_nothrow_copy_constructible_v<std::exception_ptr>);\n    _ex = xp;\n    _count = 0;\n    while (!_wait_list.empty()) {\n        auto& x = _wait_list.front();\n        x.pr.set_exception(xp);\n        _wait_list.pop_front();\n    }\n}\n\n\ntemplate<typename ExceptionFactory = semaphore_default_exception_factory, typename Clock = typename timer<>::clock>\nclass semaphore_units {\n    basic_semaphore<ExceptionFactory, Clock>* _sem;\n    size_t _n;\n\n    semaphore_units(basic_semaphore<ExceptionFactory, Clock>* sem, size_t n) noexcept : _sem(sem), _n(n) {}\npublic:\n    semaphore_units() noexcept : semaphore_units(nullptr, 0) {}\n    semaphore_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t n) noexcept : semaphore_units(&sem, n) {}\n    semaphore_units(semaphore_units&& o) noexcept : _sem(o._sem), _n(std::exchange(o._n, 0)) {\n    }\n    semaphore_units& operator=(semaphore_units&& o) noexcept {\n        if (this != &o) {\n            return_all();\n            _sem = o._sem;\n            _n = std::exchange(o._n, 0);\n        }\n        return *this;\n    }\n    semaphore_units(const semaphore_units&) = delete;\n    ~semaphore_units() noexcept {\n        return_all();\n    }\n    /// Return ownership of some units to the semaphore. The semaphore will be signaled by the number of units returned.\n    ///\n    /// \\param units number of units to subtract.\n    ///\n    /// \\note throws exception if \\c units is more than those protected by the semaphore\n    ///\n    /// \\return the number of remaining units\n    size_t return_units(size_t units) {\n        if (units > _n) {\n            throw std::invalid_argument(\"Cannot take more units than those protected by the semaphore\");\n        }\n        _n -= units;\n        _sem->signal(units);\n        return _n;\n    }\n    /// Return ownership of all units. The semaphore will be signaled by the number of units returned.\n    void return_all() noexcept {\n        if (_n) {\n            _sem->signal(_n);\n            _n = 0;\n        }\n    }\n    /// Releases ownership of the units. The semaphore will not be signalled.\n    ///\n    /// \\return the number of units held\n    size_t release() noexcept {\n        return std::exchange(_n, 0);\n    }\n    /// Splits this instance into a \\ref semaphore_units object holding the specified amount of units.\n    /// This object will continue holding the remaining units.\n    ///\n    /// \\param units number of units to subtract.\n    ///\n    /// \\note throws exception if \\c units is more than those protected by the semaphore\n    ///\n    /// \\return semaphore_units holding the specified number of units\n    semaphore_units split(size_t units) {\n        if (units > _n) {\n            throw std::invalid_argument(\"Cannot take more units than those protected by the semaphore\");\n        }\n        _n -= units;\n        return semaphore_units(_sem, units);\n    }\n    /// The inverse of split(), in which the units held by the specified \\ref semaphore_units\n    /// object are merged into the current one. The function assumes (and asserts) that both\n    /// are associated with the same \\ref semaphore.\n    ///\n    /// \\return the updated semaphore_units object\n    void adopt(semaphore_units&& other) noexcept {\n        SEASTAR_ASSERT(other._sem == _sem);\n        _n += other.release();\n    }\n\n    /// Returns the number of units held\n    size_t count() const noexcept {\n        return _n;\n    }\n\n    /// Returns true iff there any units held\n    explicit operator bool() const noexcept {\n        return _n != 0;\n    }\n};\n\n/// \\brief Take units from semaphore temporarily\n///\n/// Takes units from the semaphore and returns them when the \\ref semaphore_units object goes out of scope.\n/// This provides a safe way to temporarily take units from a semaphore and ensure\n/// that they are eventually returned under all circumstances (exceptions, premature scope exits, etc).\n///\n/// Unlike with_semaphore(), the scope of unit holding is not limited to the scope of a single async lambda.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to take\n/// \\return a \\ref future<> holding \\ref semaphore_units object. When the object goes out of scope\n///         the units are returned to the semaphore.\n///\n/// \\note The caller must guarantee that \\c sem is valid as long as\n///      \\ref seaphore_units object is alive.\n///\n/// \\related semaphore\ntemplate<typename ExceptionFactory, typename Clock = typename timer<>::clock>\nfuture<semaphore_units<ExceptionFactory, Clock>>\nget_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units) noexcept {\n    return sem.wait(units).then([&sem, units] {\n        return semaphore_units<ExceptionFactory, Clock>{ sem, units };\n    });\n}\n\n/// \\brief Take units from semaphore temporarily with time bound on wait\n///\n/// Like \\ref get_units(basic_semaphore<ExceptionFactory>&, size_t) but when\n/// timeout is reached before units are granted, returns an exceptional future holding \\ref semaphore_timed_out.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to take\n/// \\return a \\ref future<> holding \\ref semaphore_units object. When the object goes out of scope\n///         the units are returned to the semaphore.\n///         If a timeout is reached before units are granted, returns an exceptional future holding \\ref semaphore_timed_out.\n///\n/// \\note The caller must guarantee that \\c sem is valid as long as\n///      \\ref seaphore_units object is alive.\n///\n/// \\related semaphore\ntemplate<typename ExceptionFactory, typename Clock = typename timer<>::clock>\nfuture<semaphore_units<ExceptionFactory, Clock>>\nget_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units, typename basic_semaphore<ExceptionFactory, Clock>::time_point timeout) noexcept {\n    return sem.wait(timeout, units).then([&sem, units] {\n        return semaphore_units<ExceptionFactory, Clock>{ sem, units };\n    });\n}\n\n/// \\brief Take units from semaphore temporarily with time bound on wait\n///\n/// Like \\ref get_units(basic_semaphore<ExceptionFactory>&, size_t, basic_semaphore<ExceptionFactory>::time_point) but\n/// allow the timeout to be specified as a duration.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to take\n/// \\param timeout a duration specifying when to timeout the current request\n/// \\return a \\ref future<> holding \\ref semaphore_units object. When the object goes out of scope\n///         the units are returned to the semaphore.\n///         If a timeout is reached before units are granted, returns an exceptional future holding \\ref semaphore_timed_out.\n///\n/// \\note The caller must guarantee that \\c sem is valid as long as\n///      \\ref seaphore_units object is alive.\n///\n/// \\related semaphore\ntemplate<typename ExceptionFactory, typename Clock>\nfuture<semaphore_units<ExceptionFactory, Clock>>\nget_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units, typename basic_semaphore<ExceptionFactory, Clock>::duration timeout) noexcept {\n    return sem.wait(timeout, units).then([&sem, units] {\n        return semaphore_units<ExceptionFactory, Clock>{ sem, units };\n    });\n}\n\n/// \\brief Take units from semaphore temporarily with an \\ref abort_source\n///\n/// Like \\ref get_units(basic_semaphore<ExceptionFactory>&, size_t, basic_semaphore<ExceptionFactory>::time_point) but\n/// allow the function to be aborted using an \\ref abort_source.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to take\n/// \\param as abort source.\n/// \\return a \\ref future<> holding \\ref semaphore_units object. When the object goes out of scope\n///         the units are returned to the semaphore.\n///         If get_units is aborted before units are granted, returns an exceptional future holding \\ref semaphore_aborted.\n///\n/// \\note The caller must guarantee that \\c sem is valid as long as\n///      \\ref seaphore_units object is alive.\n///\n/// \\related semaphore\ntemplate<typename ExceptionFactory, typename Clock>\nfuture<semaphore_units<ExceptionFactory, Clock>>\nget_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units, abort_source& as) noexcept {\n    return sem.wait(as, units).then([&sem, units] {\n        return semaphore_units<ExceptionFactory, Clock>{ sem, units };\n    });\n}\n\n/// \\brief Try to take units from semaphore temporarily\n///\n/// Takes units from the semaphore, if available, and returns them when the \\ref semaphore_units object goes out of scope.\n/// This provides a safe way to temporarily take units from a semaphore and ensure\n/// that they are eventually returned under all circumstances (exceptions, premature scope exits, etc).\n///\n/// Unlike with_semaphore(), the scope of unit holding is not limited to the scope of a single async lambda.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to take\n/// \\return an optional \\ref semaphore_units object. If engaged, when the object goes out of scope or is reset\n///         the units are returned to the semaphore.\n///\n/// \\note The caller must guarantee that \\c sem is valid as long as\n///      \\ref seaphore_units object is alive.\n///\n/// \\related semaphore\ntemplate<typename ExceptionFactory, typename Clock = typename timer<>::clock>\nstd::optional<semaphore_units<ExceptionFactory, Clock>>\ntry_get_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units) noexcept {\n    if (!sem.try_wait(units)) {\n        return std::nullopt;\n    }\n    return std::make_optional<semaphore_units<ExceptionFactory, Clock>>(sem, units);\n}\n\n/// \\brief Consume units from semaphore temporarily\n///\n/// Consume units from the semaphore and returns them when the \\ref semaphore_units object goes out of scope.\n/// This provides a safe way to temporarily take units from a semaphore and ensure\n/// that they are eventually returned under all circumstances (exceptions, premature scope exits, etc).\n///\n/// Unlike get_units(), this calls the non-blocking consume() API.\n///\n/// Unlike with_semaphore(), the scope of unit holding is not limited to the scope of a single async lambda.\n///\n/// \\param sem The semaphore to take units from\n/// \\param units  Number of units to consume\ntemplate<typename ExceptionFactory, typename Clock = typename timer<>::clock>\nsemaphore_units<ExceptionFactory, Clock>\nconsume_units(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units) noexcept {\n    sem.consume(units);\n    return semaphore_units<ExceptionFactory, Clock>{ sem, units };\n}\n\n/// \\brief Runs a function protected by a semaphore\n///\n/// Acquires a \\ref semaphore, runs a function, and releases\n/// the semaphore, returning the the return value of the function,\n/// as a \\ref future.\n///\n/// \\param sem The semaphore to be held while the \\c func is\n///            running.\n/// \\param units  Number of units to acquire from \\c sem (as\n///               with semaphore::wait())\n/// \\param func   The function to run; signature \\c void() or\n///               \\c future<>().\n/// \\return a \\ref future<> holding the function's return value\n///         or exception thrown; or a \\ref future<> containing\n///         an exception from one of the semaphore::broken()\n///         variants.\n///\n/// \\note The caller must guarantee that \\c sem is valid until\n///       the future returned by with_semaphore() resolves.\n///\n/// \\related semaphore\ntemplate <typename ExceptionFactory, typename Func, typename Clock = typename timer<>::clock>\ninline\nfuturize_t<std::invoke_result_t<Func>>\nwith_semaphore(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units, Func&& func) noexcept {\n    return get_units(sem, units).then([func = std::forward<Func>(func)] (auto units) mutable {\n        return futurize_invoke(std::forward<Func>(func)).finally([units = std::move(units)] {});\n    });\n}\n\n/// \\brief Runs a function protected by a semaphore with time bound on wait\n///\n/// If possible, acquires a \\ref semaphore, runs a function, and releases\n/// the semaphore, returning the the return value of the function,\n/// as a \\ref future.\n///\n/// If the semaphore can't be acquired within the specified timeout, returns\n/// a semaphore_timed_out exception\n///\n/// \\param sem The semaphore to be held while the \\c func is\n///            running.\n/// \\param units  Number of units to acquire from \\c sem (as\n///               with semaphore::wait())\n/// \\param timeout a duration specifying when to timeout the current request\n/// \\param func   The function to run; signature \\c void() or\n///               \\c future<>().\n/// \\return a \\ref future<> holding the function's return value\n///         or exception thrown; or a \\ref future<> containing\n///         an exception from one of the semaphore::broken()\n///         variants.\n///\n/// \\note The caller must guarantee that \\c sem is valid until\n///       the future returned by with_semaphore() resolves.\n///\n/// \\related semaphore\ntemplate <typename ExceptionFactory, typename Clock, typename Func>\ninline\nfuturize_t<std::invoke_result_t<Func>>\nwith_semaphore(basic_semaphore<ExceptionFactory, Clock>& sem, size_t units, typename basic_semaphore<ExceptionFactory, Clock>::duration timeout, Func&& func) noexcept {\n    return get_units(sem, units, timeout).then([func = std::forward<Func>(func)] (auto units) mutable {\n        return futurize_invoke(std::forward<Func>(func)).finally([units = std::move(units)] {});\n    });\n}\n\n/// default basic_semaphore specialization that throws semaphore specific exceptions\n/// on error conditions.\nusing semaphore = basic_semaphore<semaphore_default_exception_factory>;\nusing named_semaphore = basic_semaphore<named_semaphore_exception_factory>;\n\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/shard_id.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2023-present ScyllaDB\n */\n\n#pragma once\n\n\n/// \\file\n\nnamespace seastar {\n\n\nusing shard_id = unsigned;\n\n\nnamespace internal {\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nshard_id* this_shard_id_ptr() noexcept;\n#else\ninline shard_id* this_shard_id_ptr() noexcept {\n    static thread_local shard_id g_this_shard_id;\n    return &g_this_shard_id;\n}\n#endif\n\n} // namespace internal\n\n\n/// Returns shard_id of the of the current shard.\ninline shard_id this_shard_id() noexcept {\n    return *internal::this_shard_id_ptr();\n}\n\n\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/sharded.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/smp.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/core/internal/run_in_background.hh>\n#include <seastar/core/on_internal_error.hh>\n#include <seastar/util/is_smart_ptr.hh>\n#include <seastar/util/tuple_utils.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/log.hh>\n\n#include <concepts>\n#include <functional>\n#include <ranges>\n#include <type_traits>\n\n/// \\defgroup smp-module Multicore\n///\n/// \\brief Support for exploiting multiple cores on a server.\n///\n/// Seastar supports multicore servers by using *sharding*.  Each logical\n/// core (lcore) runs a separate event loop, with its own memory allocator,\n/// TCP/IP stack, and other services.  Shards communicate by explicit message\n/// passing, rather than using locks and condition variables as with traditional\n/// threaded programming.\n\nnamespace seastar {\n\n\ntemplate <typename Func, typename... Param>\nclass sharded_parameter;\n\ntemplate <typename Service>\nclass sharded;\n\n\nnamespace internal {\n\ntemplate <typename Func, typename... Param>\nauto unwrap_sharded_arg(sharded_parameter<Func, Param...> sp);\n\nusing on_each_shard_func = std::function<future<> (unsigned shard)>;\n\nfuture<> sharded_parallel_for_each(unsigned nr_shards, on_each_shard_func on_each_shard) noexcept(std::is_nothrow_move_constructible_v<on_each_shard_func>);\n\ntemplate <typename Service>\nclass either_sharded_or_local {\n    sharded<Service>& _sharded;\npublic:\n    either_sharded_or_local(sharded<Service>& s) : _sharded(s) {}\n    operator sharded<Service>& ();\n    operator Service& ();\n};\n\ntemplate <typename T>\nstruct sharded_unwrap {\n    using evaluated_type = T;\n    using type = T;\n};\n\ntemplate <typename T>\nstruct sharded_unwrap<std::reference_wrapper<sharded<T>>> {\n    using evaluated_type = T&;\n    using type = either_sharded_or_local<T>;\n};\n\ntemplate <typename Func, typename... Param>\nstruct sharded_unwrap<sharded_parameter<Func, Param...>> {\n    using type = std::invoke_result_t<Func, Param...>;\n};\n\ntemplate <typename T>\nusing sharded_unwrap_evaluated_t = typename sharded_unwrap<T>::evaluated_type;\n\ntemplate <typename T>\nusing sharded_unwrap_t = typename sharded_unwrap<T>::type;\n\ntemplate<typename R>\nconcept unsigned_range = std::ranges::range<R>\n    && std::is_unsigned_v<std::ranges::range_value_t<R>>;\n\n} // internal\n\n\n/// \\addtogroup smp-module\n/// @{\n\n\ntemplate <typename T>\nclass sharded;\n\n/// If a sharded service inherits from this class, sharded::stop() will wait\n/// until all references to this service on each shard are released before\n/// returning. It is still service's own responsibility to track its references\n/// in asynchronous code by calling shared_from_this() and keeping returned smart\n/// pointer as long as object is in use.\ntemplate<typename T>\nclass async_sharded_service : public enable_shared_from_this<T> {\n    promise<> _freed;\nprotected:\n    async_sharded_service() noexcept = default;\n    virtual ~async_sharded_service() {\n        _freed.set_value();\n    }\n    future<> freed() noexcept {\n        return _freed.get_future();\n    }\n    template <typename Service> friend class sharded;\n};\n\n\n/// \\brief Provide a sharded service with access to its peers\n///\n/// If a service class inherits from this, it will gain a \\code container()\n/// \\endcode method that provides access to the \\ref sharded object, with which\n/// it can call its peers.\ntemplate <typename Service>\nclass peering_sharded_service {\n    sharded<Service>* _container = nullptr;\nprivate:\n    template <typename T> friend class sharded;\n    void set_container(sharded<Service>* container) noexcept { _container = container; }\npublic:\n    peering_sharded_service() noexcept = default;\n    peering_sharded_service(peering_sharded_service<Service>&&) noexcept = default;\n    peering_sharded_service(const peering_sharded_service<Service>&) = delete;\n    peering_sharded_service& operator=(const peering_sharded_service<Service>&) = delete;\n    sharded<Service>& container() noexcept { return *_container; }\n    const sharded<Service>& container() const noexcept { return *_container; }\n};\n\n\n/// Exception thrown when a \\ref sharded object does not exist\nclass no_sharded_instance_exception : public std::exception {\n    sstring _msg;\npublic:\n    no_sharded_instance_exception() : _msg(\"sharded instance does not exist\") {}\n    explicit no_sharded_instance_exception(sstring type_info)\n        : _msg(\"sharded instance does not exist: \" + type_info) {}\n    virtual const char* what() const noexcept override {\n        return _msg.c_str();\n    }\n};\n\n/// Template helper to distribute a service across all logical cores.\n///\n/// The \\c sharded template manages a sharded service, by creating\n/// a copy of the service on each logical core, providing mechanisms to communicate\n/// with each shard's copy, and a way to stop the service.\n///\n/// \\tparam Service a class to be instantiated on each core.  Must expose\n///         a \\c stop() method that returns a \\c future<>, to be called when\n///         the service is stopped.\ntemplate <typename Service>\nclass sharded {\n    struct entry {\n        shared_ptr<Service> service;\n\n        future<> track_deletion() noexcept {\n            // do not wait for instance to be deleted if it is not going to notify us\n            if constexpr (std::is_base_of_v<async_sharded_service<Service>, Service>) {\n                if (service) {\n                    return service->freed();\n                }\n            }\n            return make_ready_future<>();\n        }\n    };\n    std::vector<entry> _instances;\nprivate:\n    using invoke_on_multiple_func_type = std::function<future<> (Service&)>;\nprivate:\n    template <typename U, bool async>\n    friend struct shared_ptr_make_helper;\n\n    template <typename T>\n    requires std::is_base_of_v<peering_sharded_service<T>, T>\n    void set_container(T& service) noexcept {\n        service.set_container(this);\n    }\n\n    template <typename T>\n    requires (!std::is_base_of_v<peering_sharded_service<T>, T>)\n    void set_container(T&) noexcept {\n    }\n\n    future<>\n    sharded_parallel_for_each(internal::on_each_shard_func func) noexcept(std::is_nothrow_move_constructible_v<internal::on_each_shard_func>) {\n        return internal::sharded_parallel_for_each(_instances.size(), std::move(func));\n    }\npublic:\n    /// Constructs an empty \\c sharded object.  No instances of the service are\n    /// created.\n    sharded() noexcept {}\n    sharded(const sharded& other) = delete;\n    sharded& operator=(const sharded& other) = delete;\n    /// Sharded object with T that inherits from peering_sharded_service\n    /// cannot be moved safely, so disable move operations.\n    sharded(sharded&& other) = delete;\n    sharded& operator=(sharded&& other) = delete;\n    /// Destroyes a \\c sharded object.  Must not be in a started state.\n    ~sharded();\n\n    /// Starts \\c Service by constructing an instance on every logical core\n    /// with a copy of \\c args passed to the constructor.\n    ///\n    /// \\param args Arguments to be forwarded to \\c Service constructor\n    /// \\return a \\ref seastar::future<> that becomes ready when all instances have been\n    ///         constructed.\n    template <typename... Args>\n    future<> start(Args&&... args) noexcept;\n\n    /// Starts \\c Service by constructing an instance on a single logical core\n    /// with a copy of \\c args passed to the constructor.\n    ///\n    /// \\param args Arguments to be forwarded to \\c Service constructor\n    /// \\return a \\ref seastar::future<> that becomes ready when the instance has been\n    ///         constructed.\n    template <typename... Args>\n    future<> start_single(Args&&... args) noexcept;\n\n    /// Stops all started instances and destroys them.\n    ///\n    /// For every started instance, its \\c stop() method is called, and then\n    /// it is destroyed.\n    future<> stop() noexcept;\n\n    /// Invoke a type-erased function on all instances of `Service`.\n    /// The return value becomes ready when all instances have processed\n    /// the message.\n    ///\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func Function to be invoked on all shards\n    /// \\return Future that becomes ready once all calls have completed\n    future<> invoke_on_all(smp_submit_to_options options, std::function<future<> (Service&)> func) noexcept;\n\n    /// Invoke a type-erased function on all instances of `Service`.\n    /// The return value becomes ready when all instances have processed\n    /// the message.\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    future<> invoke_on_all(std::function<future<> (Service&)> func) noexcept;\n\n    /// Invoke a function on all instances of `Service`.\n    /// The return value becomes ready when all instances have processed\n    /// the message. The function can be a member pointer to function,\n    /// a free function, or a functor. The first argument of the function\n    /// will be a reference to the local service on the shard.\n    ///\n    /// For a non-static pointer-to-member-function, the first argument\n    /// becomes `this`, not the first declared parameter.\n    ///\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func invocable accepting a `Service&` as the first parameter\n    ///        to be invoked on all shards\n    /// \\return Future that becomes ready once all calls have completed\n    template <typename Func, typename... Args>\n    requires std::invocable<Func, Service&, internal::sharded_unwrap_t<Args>...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n    future<> invoke_on_all(smp_submit_to_options options, Func func, Args... args) noexcept;\n\n    /// Invoke a function on all instances of `Service`.\n    /// The return value becomes ready when all instances have processed\n    /// the message.\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    template <typename Func, typename... Args>\n    requires std::invocable<Func, Service&, internal::sharded_unwrap_t<Args>...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n    future<> invoke_on_all(Func func, Args... args) noexcept;\n\n    /// Invoke a callable on all instances of  \\c Service except the instance\n    /// which is allocated on current shard.\n    ///\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func a callable with the signature `void (Service&)`\n    ///             or `future<> (Service&)`, to be called on each core\n    ///             with the local instance as an argument.\n    /// \\return a `future<>` that becomes ready when all cores but the current one have\n    ///         processed the message.\n    template <typename Func, typename... Args>\n    requires std::invocable<Func, Service&, Args...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, Args...>>, future<>>\n    future<> invoke_on_others(smp_submit_to_options options, Func func, Args... args) noexcept;\n\n    /// Invoke a callable on all instances of  \\c Service except the instance\n    /// which is allocated on current shard.\n    ///\n    /// \\param func a callable with the signature `void (Service&)`\n    ///             or `future<> (Service&)`, to be called on each core\n    ///             with the local instance as an argument.\n    /// \\return a `future<>` that becomes ready when all cores but the current one have\n    ///         processed the message.\n    ///\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    template <typename Func, typename... Args>\n    requires std::invocable<Func, Service&, Args...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, Args...>>, future<>>\n    future<> invoke_on_others(Func func, Args... args) noexcept;\n\n    /// Invoke a callable on a specific instance of `Service`.\n    ///\n    /// \\param id shard id to call\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func a callable with signature `Value (Service&, Args...)` or\n    ///        `future<Value> (Service&, Args...)` (for some `Value` type), or a pointer\n    ///        to a member function of Service\n    /// \\param args parameters to the callable; will be copied or moved. To pass by reference,\n    ///              use std::ref().\n    ///\n    /// \\return result of calling `func(instance)` on the designated instance\n    template <typename Func, typename... Args, typename Ret = futurize_t<std::invoke_result_t<Func, Service&, Args...>>>\n    requires std::invocable<Func, Service&, Args&&...>\n    Ret\n    invoke_on(unsigned id, smp_submit_to_options options, Func&& func, Args&&... args);\n\n    /// Invoke a callable on a specific instance of `Service`.\n    ///\n    /// \\param id shard id to call\n    /// \\param func a callable with signature `Value (Service&)` or\n    ///        `future<Value> (Service&)` (for some `Value` type), or a pointer\n    ///        to a member function of Service\n    /// \\param args parameters to the callable\n    /// \\return result of calling `func(instance)` on the designated instance\n    template <typename Func, typename... Args, typename Ret = futurize_t<std::invoke_result_t<Func, Service&, Args&&...>>>\n    requires std::invocable<Func, Service&, Args&&...>\n    Ret\n    invoke_on(unsigned id, Func&& func, Args&&... args);\n\n    /// Invoke a callable on a range of instances of `Service`.\n    ///\n    /// \\param range std::ranges::range of unsigned integers\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func a callable with signature `Value (Service&, Args...)` or\n    ///        `future<Value> (Service&, Args...)` (for some `Value` type), or a pointer\n    ///        to a member function of Service\n    /// \\param args parameters to the callable; will be copied or moved. To pass by reference,\n    ///              use std::ref().\n    /// \\return Future that becomes ready once all calls have completed\n    template <typename R, typename Func, typename... Args>\n    requires std::invocable<Func, Service&, Args...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n        && internal::unsigned_range<R>\n    future<>\n    invoke_on(R range, smp_submit_to_options options, Func func, Args... args) noexcept;\n\n    /// Invoke a callable on a range of instances of `Service`.\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    ///\n    /// \\param range std::ranges::range of unsigned integers\n    /// \\param func a callable with signature `Value (Service&, Args...)` or\n    ///        `future<Value> (Service&, Args...)` (for some `Value` type), or a pointer\n    ///        to a member function of Service\n    /// \\param args parameters to the callable; will be copied or moved. To pass by reference,\n    ///              use std::ref().\n    /// \\return Future that becomes ready once all calls have completed\n    template <typename R, typename Func, typename... Args>\n    requires std::invocable<Func, Service&, Args...>\n        && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n        && internal::unsigned_range<R>\n    future<>\n    invoke_on(R range, Func func, Args... args) noexcept;\n\n    /// Invoke a callable on all instances of `Service` and reduce the results using\n    /// `Reducer`.\n    ///\n    /// \\see map_reduce(Iterator begin, Iterator end, Mapper&& mapper, Reducer&& r)\n    template <typename Reducer, typename Func, typename... Args>\n    auto map_reduce(Reducer&& r, Func&& func, Args&&... args) -> typename reducer_traits<Reducer>::future_type\n    {\n        auto rng = std::views::iota(size_t(0), _instances.size());\n        return ::seastar::map_reduce(rng.begin(), rng.end(),\n            [this, func = std::forward<Func>(func), args = std::make_tuple(std::forward<Args>(args)...)] (unsigned c) mutable {\n                return smp::submit_to(c, [this, &func, args] () mutable {\n                    return std::apply([this, &func] (Args&&... args) mutable {\n                        auto inst = get_local_service();\n                        return std::invoke(func, *inst, std::forward<Args>(args)...);\n                    }, std::move(args));\n                });\n            }, std::forward<Reducer>(r));\n    }\n\n    /// The const version of \\ref map_reduce(Reducer&& r, Func&& func)\n    template <typename Reducer, typename Func, typename... Args>\n    auto map_reduce(Reducer&& r, Func&& func, Args&&... args) const -> typename reducer_traits<Reducer>::future_type\n    {\n        auto rng = std::views::iota(size_t(0), _instances.size());\n        return ::seastar::map_reduce(rng.begin(), rng.end(),\n            [this, func = std::forward<Func>(func), args = std::make_tuple(std::forward<Args>(args)...)] (unsigned c) {\n                return smp::submit_to(c, [this, &func, args] () {\n                    return std::apply([this, &func] (Args&&... args) {\n                        auto inst = get_local_service();\n                        return std::invoke(func, *inst, std::forward<Args>(args)...);\n                    }, std::move(args));\n                });\n            }, std::forward<Reducer>(r));\n    }\n\n    /// Applies a map function to all shards, then reduces the output by calling a reducer function.\n    ///\n    /// \\param map callable with the signature `Value (Service&)` or\n    ///               `future<Value> (Service&)` (for some `Value` type).\n    ///               used as the second input to \\c reduce\n    /// \\param initial initial value used as the first input to \\c reduce.\n    /// \\param reduce binary function used to left-fold the return values of \\c map\n    ///               into \\c initial .\n    ///\n    /// Each \\c map invocation runs on the shard associated with the service.\n    ///\n    /// \\tparam  Mapper unary function taking `Service&` and producing some result.\n    /// \\tparam  Initial any value type\n    /// \\tparam  Reduce a binary function taking two Initial values and returning an Initial\n    /// \\return  Result of invoking `map` with each instance in parallel, reduced by calling\n    ///          `reduce()` on each adjacent pair of results.\n    template <typename Mapper, typename Initial, typename Reduce>\n    future<Initial>\n    map_reduce0(Mapper map, Initial initial, Reduce reduce) {\n        auto wrapped_map = [this, map] (unsigned c) {\n            return smp::submit_to(c, [this, map] {\n                auto inst = get_local_service();\n                return std::invoke(map, *inst);\n            });\n        };\n        return ::seastar::map_reduce(smp::all_cpus().begin(), smp::all_cpus().end(),\n                            std::move(wrapped_map),\n                            std::move(initial),\n                            std::move(reduce));\n    }\n\n    /// The const version of \\ref map_reduce0(Mapper map, Initial initial, Reduce reduce)\n    template <typename Mapper, typename Initial, typename Reduce>\n    future<Initial>\n    map_reduce0(Mapper map, Initial initial, Reduce reduce) const {\n        auto wrapped_map = [this, map] (unsigned c) {\n            return smp::submit_to(c, [this, map] {\n                auto inst = get_local_service();\n                return std::invoke(map, *inst);\n            });\n        };\n        return ::seastar::map_reduce(smp::all_cpus().begin(), smp::all_cpus().end(),\n                            std::move(wrapped_map),\n                            std::move(initial),\n                            std::move(reduce));\n    }\n\n    /// Applies a map function to all shards, and return a vector of the result.\n    ///\n    /// \\param mapper callable with the signature `Value (Service&)` or\n    ///               `future<Value> (Service&)` (for some `Value` type).\n    ///\n    /// Each \\c map invocation runs on the shard associated with the service.\n    ///\n    /// \\tparam  Mapper unary function taking `Service&` and producing some result.\n    /// \\return  Result vector of invoking `map` with each instance in parallel\n    template <typename Mapper, typename Future = futurize_t<std::invoke_result_t<Mapper,Service&>>, typename return_type = decltype(internal::untuple(std::declval<typename Future::tuple_type>()))>\n    future<std::vector<return_type>> map(Mapper mapper) {\n        return do_with(std::vector<return_type>(), std::move(mapper),\n                [this] (std::vector<return_type>& vec, Mapper& mapper) mutable {\n            vec.resize(_instances.size());\n            return parallel_for_each(std::views::iota(0u, _instances.size()), [this, &vec, &mapper] (unsigned c) {\n                return smp::submit_to(c, [this, &mapper] {\n                    auto inst = get_local_service();\n                    return mapper(*inst);\n                }).then([&vec, c] (auto&& res) {\n                    vec[c] = std::move(res);\n                });\n            }).then([&vec] {\n                return make_ready_future<std::vector<return_type>>(std::move(vec));\n            });\n        });\n    }\n\n    /// Gets a reference to the local instance.\n    const Service& local() const noexcept;\n\n    /// Gets a reference to the local instance.\n    Service& local() noexcept;\n\n    /// Gets a shared pointer to the local instance.\n    shared_ptr<Service> local_shared() noexcept;\n\n    /// Checks whether the local instance has been initialized.\n    bool local_is_initialized() const noexcept;\nprivate:\n    template <typename... Args>\n    shared_ptr<Service> create_local_service(Args&&... args) {\n        auto s = ::seastar::make_shared<Service>(std::forward<Args>(args)...);\n        set_container(*s);\n        return s;\n    }\n\n    /// Performs the same check as `local_is_initialized`, but throws an exception\n    /// if the local instance is not initialized.\n    void check_local() const;\n\n    shared_ptr<Service> get_local_service() {\n        check_local();\n        return _instances[this_shard_id()].service;\n    }\n\n    shared_ptr<const Service> get_local_service() const {\n        check_local();\n        return _instances[this_shard_id()].service;\n    }\n};\n\n\n/// \\brief Helper to pass a parameter to a `sharded<>` object that depends\n/// on the shard. It is evaluated on the shard, just before being\n/// passed to the local instance. It is useful when passing\n/// parameters to sharded::start().\ntemplate <typename Func, typename... Params>\nclass sharded_parameter {\n    Func _func;\n    std::tuple<Params...> _params;\npublic:\n    /// Creates a sharded parameter, which evaluates differently based on\n    /// the shard it is executed on.\n    ///\n    /// \\param func      Function to be executed\n    /// \\param params    optional parameters to be passed to the function. Can\n    ///                  be std::ref(sharded<whatever>), in which case the local\n    ///                  instance will be passed. Anything else\n    ///                  will be passed by value unchanged.\n    explicit sharded_parameter(Func func, Params... params)\n            requires std::invocable<Func, internal::sharded_unwrap_evaluated_t<Params>...>\n            : _func(std::move(func)), _params(std::make_tuple(std::move(params)...)) {\n    }\nprivate:\n    auto evaluate() const;\n\n    template <typename Func_, typename... Param_>\n    friend auto internal::unwrap_sharded_arg(sharded_parameter<Func_, Param_...> sp);\n};\n\n/// \\example sharded_parameter_demo.cc\n///\n/// Example use of \\ref sharded_parameter.\n/// @}\n\ntemplate <typename Service>\nsharded<Service>::~sharded() {\n\tSEASTAR_ASSERT(_instances.empty());\n}\n\nnamespace internal {\n\ntemplate <typename T>\nT&&\nunwrap_sharded_arg(T&& arg) {\n    return std::forward<T>(arg);\n}\n\ntemplate <typename Service>\neither_sharded_or_local<Service>\nunwrap_sharded_arg(std::reference_wrapper<sharded<Service>> arg) {\n    return either_sharded_or_local<Service>(arg);\n}\n\ntemplate <typename Func, typename... Param>\nauto\nunwrap_sharded_arg(sharded_parameter<Func, Param...> sp) {\n    return sp.evaluate();\n}\n\ntemplate <typename Service>\neither_sharded_or_local<Service>::operator sharded<Service>& () { return _sharded; }\n\ntemplate <typename Service>\neither_sharded_or_local<Service>::operator Service& () { return _sharded.local(); }\n\n}\n\ntemplate <typename Func, typename... Param>\nauto\nsharded_parameter<Func, Param...>::evaluate() const {\n    auto unwrap_params_and_invoke = [this] (const auto&... params) {\n        return std::invoke(_func, internal::unwrap_sharded_arg(params)...);\n    };\n    return std::apply(unwrap_params_and_invoke, _params);\n}\n\ntemplate <typename Service>\ntemplate <typename... Args>\nfuture<>\nsharded<Service>::start(Args&&... args) noexcept {\n  try {\n    _instances.resize(smp::count);\n    return sharded_parallel_for_each(\n        [this, args = std::make_tuple(std::forward<Args>(args)...)] (unsigned c) mutable {\n            return smp::submit_to(c, [this, args] () mutable {\n                _instances[this_shard_id()].service = std::apply([this] (Args... args) {\n                    return create_local_service(internal::unwrap_sharded_arg(std::forward<Args>(args))...);\n                }, args);\n            });\n    }).then_wrapped([this] (future<> f) {\n        try {\n            f.get();\n            return make_ready_future<>();\n        } catch (...) {\n            return this->stop().then([e = std::current_exception()] () mutable {\n                std::rethrow_exception(e);\n            });\n        }\n    });\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate <typename Service>\ntemplate <typename... Args>\nfuture<>\nsharded<Service>::start_single(Args&&... args) noexcept {\n  try {\n    SEASTAR_ASSERT(_instances.empty());\n    _instances.resize(1);\n    return smp::submit_to(0, [this, args = std::make_tuple(std::forward<Args>(args)...)] () mutable {\n        _instances[0].service = std::apply([this] (Args... args) {\n            return create_local_service(internal::unwrap_sharded_arg(std::forward<Args>(args))...);\n        }, args);\n    }).then_wrapped([this] (future<> f) {\n        try {\n            f.get();\n            return make_ready_future<>();\n        } catch (...) {\n            return this->stop().then([e = std::current_exception()] () mutable {\n                std::rethrow_exception(e);\n            });\n        }\n    });\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\nnamespace internal {\n\n// Helper check if Service::stop exists\n\ntemplate <typename Service>\nconcept sharded_has_stop = requires {\n    // If a member names \"stop\" exists, try to call it, even if it doesn't\n    // have the correct signature. This is so that we don't ignore a function\n    // named stop() just because the signature is incorrect, and instead\n    // force the user to resolve the ambiguity.\n    { &Service::stop };\n};\n\ntemplate <bool stop_exists>\nstruct sharded_call_stop {\n    template <typename Service>\n    static future<> call(Service& instance);\n};\n\ntemplate <>\ntemplate <typename Service>\nfuture<> sharded_call_stop<true>::call(Service& instance) {\n    return instance.stop();\n}\n\ntemplate <>\ntemplate <typename Service>\nfuture<> sharded_call_stop<false>::call(Service&) {\n    return make_ready_future<>();\n}\n\ntemplate <typename Service>\nfuture<>\nstop_sharded_instance(Service& instance) {\n    constexpr bool has_stop = internal::sharded_has_stop<Service>;\n    return internal::sharded_call_stop<has_stop>::call(instance);\n}\n\n}\n\ntemplate <typename Service>\nfuture<>\nsharded<Service>::stop() noexcept {\n  try {\n    return sharded_parallel_for_each([this] (unsigned c) mutable {\n        return smp::submit_to(c, [this] () mutable {\n            auto inst = _instances[this_shard_id()].service;\n            if (!inst) {\n                return make_ready_future<>();\n            }\n            return internal::stop_sharded_instance(*inst);\n        });\n    }).then_wrapped([this] (future<> fut) {\n        return sharded_parallel_for_each([this] (unsigned c) {\n            return smp::submit_to(c, [this] {\n                auto fut = _instances[this_shard_id()].track_deletion();\n                _instances[this_shard_id()].service = nullptr;\n                return fut;\n            });\n        }).finally([this, fut = std::move(fut)] () mutable {\n            _instances.clear();\n            _instances = std::vector<sharded<Service>::entry>();\n            return std::move(fut);\n        });\n    });\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate <typename Service>\nfuture<>\nsharded<Service>::invoke_on_all(smp_submit_to_options options, std::function<future<> (Service&)> func) noexcept {\n  try {\n    return sharded_parallel_for_each([this, options, func = std::move(func)] (unsigned c) {\n        return smp::submit_to(c, options, [this, func] {\n            return func(*get_local_service());\n        });\n    });\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate <typename Service>\nfuture<>\nsharded<Service>::invoke_on_all(std::function<future<> (Service&)> func) noexcept {\n    try {\n        return invoke_on_all(smp_submit_to_options{}, std::move(func));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args>\nrequires std::invocable<Func, Service&, internal::sharded_unwrap_t<Args>...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\nfuture<>\nsharded<Service>::invoke_on_all(smp_submit_to_options options, Func func, Args... args) noexcept {\n  try {\n    return invoke_on_all(options, invoke_on_multiple_func_type([func = std::move(func), args = std::tuple(std::move(args)...)] (Service& service) mutable {\n        return std::apply([&service, &func] (Args&&... args) mutable {\n            return futurize_apply(func, std::tuple_cat(std::forward_as_tuple(service), std::tuple(internal::unwrap_sharded_arg(std::forward<Args>(args))...)));\n        }, std::move(args));\n    }));\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args>\nrequires std::invocable<Func, Service&, internal::sharded_unwrap_t<Args>...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\nfuture<>\nsharded<Service>::invoke_on_all(Func func, Args... args) noexcept {\n    try {\n        return invoke_on_all(smp_submit_to_options{}, std::move(func), std::move(args)...);\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args>\nrequires std::invocable<Func, Service&, Args...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, Args...>>, future<>>\nfuture<>\nsharded<Service>::invoke_on_others(smp_submit_to_options options, Func func, Args... args) noexcept {\n  try {\n    return invoke_on_all(options, [orig = this_shard_id(), func = std::move(func), args = std::tuple(std::move(args)...)] (Service& s) mutable -> future<> {\n        return this_shard_id() == orig ? make_ready_future<>() : futurize_apply(func, std::tuple_cat(std::forward_as_tuple(s), args));;\n    });\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args>\nrequires std::invocable<Func, Service&, Args...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, Args...>>, future<>>\nfuture<>\nsharded<Service>::invoke_on_others(Func func, Args... args) noexcept {\n    try {\n        return invoke_on_others(smp_submit_to_options{}, std::move(func), std::move(args)...);\n    } catch (...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args, typename Ret>\nrequires std::invocable<Func, Service&, Args&&...>\nRet\nsharded<Service>::invoke_on(unsigned id, smp_submit_to_options options, Func&& func, Args&&... args) {\n    return smp::submit_to(id, options, [this, func = std::forward<Func>(func), args = std::tuple(std::move(args)...)] () mutable {\n        auto inst = get_local_service();\n        return std::apply(std::forward<Func>(func), std::tuple_cat(std::forward_as_tuple(*inst), std::move(args)));\n    });\n}\n\ntemplate <typename Service>\ntemplate <typename Func, typename... Args, typename Ret>\nrequires std::invocable<Func, Service&, Args&&...>\nRet\nsharded<Service>::invoke_on(unsigned id, Func&& func, Args&&... args) {\n    return invoke_on(id, smp_submit_to_options(), std::forward<Func>(func), std::forward<Args>(args)...);\n}\n\ntemplate <typename Service>\ntemplate <typename R, typename Func, typename... Args>\nrequires std::invocable<Func, Service&, Args...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n    && internal::unsigned_range<R>\nfuture<>\nsharded<Service>::invoke_on(R range, smp_submit_to_options options, Func func, Args... args) noexcept {\n    try {\n        auto func_futurized = invoke_on_multiple_func_type([func = std::move(func), args = std::tuple(std::move(args)...)] (Service& service) mutable {\n            // Avoid false-positive unused-lambda-capture warning on Clang\n            (void)args;\n            return futurize_apply(func, std::tuple_cat(std::forward_as_tuple(service), std::tuple(internal::unwrap_sharded_arg(std::forward<Args>(args))...)));\n        });\n        return parallel_for_each(range, [this, options, func = std::move(func_futurized)] (unsigned s) {\n            if (s > smp::count - 1) {\n                throw std::invalid_argument(format(\"Invalid shard id in range: {}. Must be in range [0,{})\", s, smp::count));\n            }\n            return smp::submit_to(s, options, [this, func] {\n                return func(*get_local_service());\n            });\n        });\n    } catch(...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename Service>\ntemplate <typename R, typename Func, typename... Args>\nrequires std::invocable<Func, Service&, Args...>\n    && std::is_same_v<futurize_t<std::invoke_result_t<Func, Service&, internal::sharded_unwrap_t<Args>...>>, future<>>\n    && internal::unsigned_range<R>\nfuture<>\nsharded<Service>::invoke_on(R range, Func func, Args... args) noexcept {\n    try {\n        return invoke_on(std::forward<R>(range), smp_submit_to_options{}, std::move(func), std::move(args)...);\n    } catch(...) {\n        return current_exception_as_future();\n    }\n}\n\ntemplate <typename Service>\nconst Service& sharded<Service>::local() const noexcept {\n    SEASTAR_ASSERT(local_is_initialized());\n    return *_instances[this_shard_id()].service;\n}\n\ntemplate <typename Service>\nService& sharded<Service>::local() noexcept {\n    SEASTAR_ASSERT(local_is_initialized());\n    return *_instances[this_shard_id()].service;\n}\n\ntemplate <typename Service>\nshared_ptr<Service> sharded<Service>::local_shared() noexcept {\n    SEASTAR_ASSERT(local_is_initialized());\n    return _instances[this_shard_id()].service;\n}\n\ntemplate <typename Service>\nbool sharded<Service>::local_is_initialized() const noexcept {\n    return _instances.size() > this_shard_id() &&\n           _instances[this_shard_id()].service;\n}\n\ntemplate <typename Service>\nvoid sharded<Service>::check_local() const {\n    if (!local_is_initialized()) {\n        throw no_sharded_instance_exception(pretty_type_name(typeid(Service)));\n    }\n}\n\n/// \\addtogroup smp-module\n/// @{\n/// Smart pointer wrapper which makes it safe to move across CPUs.\n///\n/// \\c foreign_ptr<> is a smart pointer wrapper which, unlike\n/// \\ref shared_ptr and \\ref lw_shared_ptr, is safe to move to a\n/// different core.\n///\n/// As seastar avoids locking, any but the most trivial objects must\n/// be destroyed on the same core they were created on, so that,\n/// for example, their destructors can unlink references to the\n/// object from various containers.  In addition, for performance\n/// reasons, the shared pointer types do not use atomic operations\n/// to manage their reference counts.  As a result they cannot be\n/// used on multiple cores in parallel.\n///\n/// \\c foreign_ptr<> provides a solution to that problem.\n/// \\c foreign_ptr<> wraps smart pointers -- \\ref seastar::shared_ptr<>,\n/// or similar, and remembers on what core this happened.\n/// When the \\c foreign_ptr<> object is destroyed, it sends a message to\n/// the original core so that the wrapped object can be safely destroyed.\n///\n/// \\c foreign_ptr<> is a move-only object; it cannot be copied.\n///\ntemplate <typename PtrType>\nrequires (!std::is_pointer_v<PtrType>)\nclass foreign_ptr {\nprivate:\n    PtrType _value;\n    unsigned _cpu;\nprivate:\n    void destroy(PtrType p, unsigned cpu) noexcept {\n        // `destroy()` is called from the destructor and other\n        // synchronous methods (like `reset()`), that have no way to\n        // wait for this future.\n        auto f = destroy_on(std::move(p), cpu);\n        if (!f.available() || f.failed()) {\n            internal::run_in_background(std::move(f));\n        }\n    }\n\n    static future<> destroy_on(PtrType p, unsigned cpu) noexcept {\n        if (p) {\n            if (cpu != this_shard_id()) {\n                return smp::submit_to(cpu, [v = std::move(p)] () mutable {\n                    // Destroy the contained pointer. We do this explicitly\n                    // in the current shard, because the lambda is destroyed\n                    // in the shard that submitted the task.\n                    v = {};\n                });\n            } else {\n                p = {};\n            }\n        }\n        return make_ready_future<>();\n    }\n\n    void check_shard() const {\n#ifdef SEASTAR_DEBUG_SHARED_PTR\n        if (_cpu != this_shard_id()) [[unlikely]] {\n            on_fatal_internal_error(seastar_logger, \"foreign_ptr accessed on non-owner cpu\");\n        }\n#endif\n    }\npublic:\n    using element_type = typename std::pointer_traits<PtrType>::element_type;\n    using pointer = element_type*;\n\n    /// Constructs a null \\c foreign_ptr<>.\n    foreign_ptr() noexcept(std::is_nothrow_default_constructible_v<PtrType>)\n        : _value(PtrType())\n        , _cpu(this_shard_id()) {\n    }\n    /// Constructs a null \\c foreign_ptr<>.\n    foreign_ptr(std::nullptr_t) noexcept(std::is_nothrow_default_constructible_v<foreign_ptr>) : foreign_ptr() {}\n    /// Wraps a pointer object and remembers the current core.\n    foreign_ptr(PtrType value) noexcept(std::is_nothrow_move_constructible_v<PtrType>)\n        : _value(std::move(value))\n        , _cpu(this_shard_id()) {\n    }\n    // The type is intentionally non-copyable because copies\n    // are expensive because each copy requires across-CPU call.\n    foreign_ptr(const foreign_ptr&) = delete;\n    /// Moves a \\c foreign_ptr<> to another object.\n    foreign_ptr(foreign_ptr&& other) noexcept(std::is_nothrow_move_constructible_v<PtrType>) = default;\n    /// Destroys the wrapped object on its original cpu.\n    ~foreign_ptr() {\n        destroy(std::move(_value), _cpu);\n    }\n    /// Creates a copy of this foreign ptr. Only works if the stored ptr is copyable.\n    future<foreign_ptr> copy() const noexcept {\n        return smp::submit_to(_cpu, [this] () mutable {\n            auto v = _value;\n            return make_foreign(std::move(v));\n        });\n    }\n    /// Accesses the wrapped object.\n    element_type& operator*() const noexcept(noexcept(*_value)) { return *_value; }\n    /// Accesses the wrapped object.\n    element_type* operator->() const noexcept(noexcept(&*_value)) { return &*_value; }\n    /// Access the raw pointer to the wrapped object.\n    pointer get() const  noexcept(noexcept(&*_value)) { return &*_value; }\n    /// Return the owner-shard of this pointer.\n    ///\n    /// The owner shard of the pointer can change as a result of\n    /// move-assigment or a call to reset().\n    unsigned get_owner_shard() const noexcept { return _cpu; }\n    /// Checks whether the wrapped pointer is non-null.\n    operator bool() const noexcept(noexcept(static_cast<bool>(_value))) { return static_cast<bool>(_value); }\n    /// Move-assigns a \\c foreign_ptr<>.\n    foreign_ptr& operator=(foreign_ptr&& other) noexcept(std::is_nothrow_move_constructible_v<PtrType>) {\n         destroy(std::move(_value), _cpu);\n        _value = std::move(other._value);\n        _cpu = other._cpu;\n        return *this;\n    }\n    /// Return a reference to the wrapped pointer.\n    ///\n    /// Warning: This method must be called on the\n    /// owner shard to avoid accidents.\n    const PtrType& unwrap_on_owner_shard() const noexcept {\n        check_shard();\n        return _value;\n    }\n    PtrType& unwrap_on_owner_shard() noexcept {\n        check_shard();\n        return _value;\n    }\n    /// Releases the owned pointer\n    ///\n    /// Warning: the caller is now responsible for destroying the\n    /// pointer on its owner shard. This method is best called on the\n    /// owner shard to avoid accidents.\n    PtrType release() noexcept(std::is_nothrow_default_constructible_v<PtrType>) {\n        check_shard();\n        return std::exchange(_value, {});\n    }\n    /// Replace the managed pointer with new_ptr.\n    ///\n    /// The previous managed pointer is destroyed on its owner shard.\n    void reset(PtrType new_ptr) noexcept(std::is_nothrow_move_constructible_v<PtrType>) {\n        auto old_ptr = std::move(_value);\n        auto old_cpu = _cpu;\n\n        _value = std::move(new_ptr);\n        _cpu = this_shard_id();\n\n        destroy(std::move(old_ptr), old_cpu);\n    }\n    /// Replace the managed pointer with a null value.\n    ///\n    /// The previous managed pointer is destroyed on its owner shard.\n    void reset(std::nullptr_t = nullptr) noexcept(std::is_nothrow_default_constructible_v<PtrType>) {\n        reset(PtrType());\n    }\n\n    /// Destroy the managed pointer.\n    ///\n    /// \\returns a future that is resolved when managed pointer is destroyed on its owner shard.\n    future<> destroy() noexcept {\n        return destroy_on(std::move(_value), _cpu);\n    }\n};\n\n/// Wraps a raw or smart pointer object in a \\ref foreign_ptr<>.\n///\n/// \\relates foreign_ptr\ntemplate <typename T>\nforeign_ptr<T> make_foreign(T ptr) {\n    return foreign_ptr<T>(std::move(ptr));\n}\n\n/// @}\n\ntemplate<typename T>\nstruct is_smart_ptr<foreign_ptr<T>> : std::true_type {};\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/shared_future.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/abortable_fifo.hh>\n#include <seastar/core/abort_on_expiry.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <exception>\n#include <optional>\n\nnamespace seastar {\n\n/// \\addtogroup future-module\n/// @{\n\n/// Changes the clock used by shared_future<> and shared_promise<> when passed as the first template parameter.\ntemplate<typename Clock>\nstruct with_clock {};\n\n/// \\cond internal\n\ntemplate <typename... T>\nstruct future_option_traits;\n\ntemplate <typename Clock, typename T>\nstruct future_option_traits<with_clock<Clock>, T> {\n    using clock_type = Clock;\n\n    using future_type = future<T>;\n    using promise_type = promise<T>;\n};\n\ntemplate <typename Clock>\nstruct future_option_traits<with_clock<Clock>> {\n    using clock_type = Clock;\n\n    using future_type = future<>;\n    using promise_type = promise<>;\n};\n\ntemplate <typename T>\nstruct future_option_traits<T> : public future_option_traits<with_clock<lowres_clock>, T> {\n};\n\ntemplate <>\nstruct future_option_traits<> : future_option_traits<void> {\n};\n\n/// \\endcond\n\n/// \\brief Like \\ref future except the result can be waited for by many fibers.\n///\n/// Represents a value which may not yet be ready. A fiber can wait for the value using\n/// the \\ref future obtained by calling \\ref get_future() or casting to \\ref future type.\n/// Multiple fibers are allowed to obtain a \\ref future for the result using the same\n/// instance of \\ref shared_future.\n///\n/// All futures obtained from shared_future should end up in the same state. However,\n/// if the value's copy constructor throws, some of the futures may end up in a failed state\n/// with an exception thrown from the copy constructor and end up with a state\n/// different than other futures.\n///\n/// The scope of shared_future instance doesn't have to include scopes of the futures\n/// obtained from that instance. In that sense the returned futures are independent.\n///\n/// shared_future can be copied at any time and all copies will resolve with the same value.\n///\n/// shared_future can be in a disengaged state when it's default-constructed or moved-from.\n/// When it's in such a state we say it's invalid and obtaining futures must not be attempted.\n///\n/// The types in the parameter pack T must all be copy-constructible.\n///\n/// When the first type in the parameter pack is \\ref with_clock then it has the effect\n/// of changing the clock used for timeouts by this instance. This type is omitted from\n/// the parameter of the future<> objects.\n///\n/// Example:\n///\n///    future<int> f;\n///    shared_future<with_clock<manual_clock>, int> sf(std::move(f));\n///    future<int> f2 = sf;\n///\ntemplate<typename... T>\nclass shared_future {\n    template <typename... U> friend class shared_promise;\n    using options = future_option_traits<T...>;\npublic:\n    using clock = typename options::clock_type;\n    using time_point = typename clock::time_point;\n    using future_type = typename future_option_traits<T...>::future_type;\n    using promise_type = typename future_option_traits<T...>::promise_type;\n    using value_tuple_type = typename future_type::tuple_type;\nprivate:\n    /// \\cond internal\n    class shared_state final : public enable_lw_shared_from_this<shared_state>, public task {\n        future_type _original_future;\n        // Ensures that shared_state is alive until run_and_dispose() runs (if the task was scheduled)\n        lw_shared_ptr<shared_state> _keepaliver;\n        struct entry {\n            promise_type pr;\n            std::optional<abort_on_expiry<clock>> timer;\n        };\n\n        struct entry_expiry {\n            void operator()(entry& e) noexcept {\n                if (e.timer) {\n                    e.pr.set_exception(std::make_exception_ptr(timed_out_error()));\n                } else {\n                    e.pr.set_exception(std::make_exception_ptr(abort_requested_exception()));\n                }\n            };\n        };\n\n        internal::abortable_fifo<entry, entry_expiry> _peers;\n\n    public:\n        ~shared_state() {\n            // Don't warn if the shared future is exceptional. Any\n            // warnings will be reported by the futures returned by\n            // get_future.\n            if (_original_future.failed()) {\n                _original_future.ignore_ready_future();\n            }\n        }\n        explicit shared_state(future_type f) noexcept\n                : task(default_scheduling_group()) // SG is set later, when the task is scheduled\n                , _original_future(std::move(f)) {\n        }\n        void run_and_dispose() noexcept override {\n            auto& state = _original_future._state;\n            if (_original_future.failed()) {\n                while (_peers) {\n                    _peers.front().pr.set_exception(state.get_exception());\n                    _peers.pop_front();\n                }\n            } else {\n                while (_peers) {\n                    auto& p = _peers.front().pr;\n                    try {\n                        p.set_value(state.get_value());\n                    } catch (...) {\n                        p.set_exception(std::current_exception());\n                    }\n                    _peers.pop_front();\n                }\n            }\n            // _peer is now empty, but let's also make sure it releases any\n            // memory it might hold in reserve.\n            _peers = {};\n            _keepaliver.release();\n        }\n\n        task* waiting_task() noexcept override {\n            return nullptr;\n        }\n\n        future_type get_future(time_point timeout = time_point::max()) noexcept {\n            // Note that some functions called below may throw,\n            // like pushing to _peers or copying _original_future's ready value.\n            // We'd rather terminate than propagate these errors similar to\n            // .then()'s failure to allocate a continuation as the caller cannot\n            // distinguish between an error returned by the original future to\n            // failing to perform `get_future` itself.\n            memory::scoped_critical_alloc_section _;\n            if (!_original_future.available()) {\n                entry& e = _peers.emplace_back();\n\n                auto f = e.pr.get_future();\n                if (timeout != time_point::max()) {\n                    e.timer.emplace(timeout);\n                    abort_source& as = e.timer->abort_source();\n                   _peers.make_back_abortable(as);\n                }\n                if (!_keepaliver) {\n                    this->set_scheduling_group(current_scheduling_group());\n                    _original_future.set_task(*this);\n                    _keepaliver = this->shared_from_this();\n                }\n                return f;\n            } else if (_original_future.failed()) {\n                return future_type(exception_future_marker(), std::exception_ptr(_original_future._state.get_exception()));\n            } else {\n                return future_type(ready_future_marker(), _original_future._state.get_value());\n            }\n        }\n\n        future_type get_future(abort_source& as) noexcept {\n            // Note that some functions called below may throw,\n            // like pushing to _peers or copying _original_future's ready value.\n            // We'd rather terminate than propagate these errors similar to\n            // .then()'s failure to allocate a continuation as the caller cannot\n            // distinguish between an error returned by the original future to\n            // failing to perform `get_future` itself.\n            memory::scoped_critical_alloc_section _;\n            if (!_original_future.available()) {\n                entry& e = _peers.emplace_back();\n\n                auto f = e.pr.get_future();\n                _peers.make_back_abortable(as);\n                if (!_keepaliver) {\n                    this->set_scheduling_group(current_scheduling_group());\n                    _original_future.set_task(*this);\n                    _keepaliver = this->shared_from_this();\n                }\n                return f;\n            } else if (_original_future.failed()) {\n                return future_type(exception_future_marker(), std::exception_ptr(_original_future._state.get_exception()));\n            } else {\n                return future_type(ready_future_marker(), _original_future._state.get_value());\n            }\n        }\n\n        bool available() const noexcept {\n            return _original_future.available();\n        }\n\n        bool failed() const noexcept {\n            return _original_future.failed();\n        }\n\n        // Used only in tests (see shared_future_tester in futures_test.cc)\n        bool has_scheduled_task() const noexcept {\n            return _keepaliver != nullptr;\n        }\n    };\n    /// \\endcond\n    lw_shared_ptr<shared_state> _state;\npublic:\n    /// \\brief Forwards the result of future \\c f into this shared_future.\n    shared_future(future_type f)\n        : _state(make_lw_shared<shared_state>(std::move(f))) { }\n\n    shared_future() = default; // noexcept, based on the respective lw_shared_ptr constructor\n    shared_future(const shared_future&) = default; // noexcept, based on the respective lw_shared_ptr constructor\n    shared_future& operator=(const shared_future&) = default; // noexcept, based on respective constructor\n    shared_future(shared_future&&) = default; // noexcept, based on the respective lw_shared_ptr constructor\n    shared_future& operator=(shared_future&&) = default; // noexcept, based on the respective constructor\n\n    /// \\brief Creates a new \\c future which will resolve with the result of this shared_future\n    ///\n    /// \\param timeout When engaged, the returned future will resolve with \\ref timed_out_error\n    /// if this shared_future doesn't resolve before timeout is reached.\n    ///\n    /// This object must be in a valid state.\n    future_type get_future(time_point timeout = time_point::max()) const noexcept {\n        return _state->get_future(timeout);\n    }\n\n    /// \\brief Creates a new \\c future which will resolve with the result of this shared_future\n    ///\n    /// \\param as abort source. The returned future will resolve with \\ref abort_requested_exception\n    /// if this shared_future doesn't resolve before aborted.\n    ///\n    /// This object must be in a valid state.\n    future_type get_future(abort_source& as) const noexcept {\n        return _state->get_future(as);\n    }\n\n    /// \\brief Returns true if the future is available (ready or failed)\n    ///\n    /// \\note This object must be in a valid state.\n    bool available() const noexcept {\n        return _state->available();\n    }\n\n    /// \\brief Returns true if the future is failed\n    ///\n    /// \\note This object must be in a valid state.\n    bool failed() const noexcept {\n        return _state->failed();\n    }\n\n    /// \\brief Equivalent to \\ref get_future()\n    operator future_type() const noexcept {\n        return get_future();\n    }\n\n    /// \\brief Returns true if the instance is in valid state\n    bool valid() const noexcept {\n        return bool(_state);\n    }\n\n    /// \\cond internal\n    friend class shared_future_tester;\n    /// \\endcond\n};\n\n/// \\brief Like \\ref promise except that its counterpart is \\ref shared_future instead of \\ref future\n///\n/// When the shared_promise is made ready, every waiter is also made ready.\n///\n/// Like the shared_future, the types in the parameter pack T must all be copy-constructible.\ntemplate <typename... T>\nclass shared_promise {\npublic:\n    using shared_future_type = shared_future<T...>;\n    using future_type = typename shared_future_type::future_type;\n    using promise_type = typename shared_future_type::promise_type;\n    using clock = typename shared_future_type::clock;\n    using time_point = typename shared_future_type::time_point;\n    using value_tuple_type = typename shared_future_type::value_tuple_type;\nprivate:\n    promise_type _promise;\n    shared_future_type _shared_future;\n    static constexpr bool copy_noexcept = future_type::copy_noexcept;\npublic:\n    shared_promise(const shared_promise&) = delete;\n    shared_promise(shared_promise&&) = default; // noexcept, based on the respective promise and shared_future constructors\n    shared_promise& operator=(shared_promise&&) = default; // noexcept, based on the respective promise and shared_future constructors\n    shared_promise() : _promise(), _shared_future(_promise.get_future()) {\n    }\n\n    /// \\brief Gets new future associated with this promise.\n    /// If the promise is not resolved before timeout the returned future will resolve with \\ref timed_out_error.\n    /// This instance doesn't have to be kept alive until the returned future resolves.\n    future_type get_shared_future(time_point timeout = time_point::max()) const noexcept {\n        return _shared_future.get_future(timeout);\n    }\n\n    /// \\brief Gets new future associated with this promise.\n    /// If the promise is not resolved before abort source is triggered the returned future will\n    /// resolve with \\ref abort_requests_exception.\n    /// This instance doesn't have to be kept alive until the returned future resolves.\n    future_type get_shared_future(abort_source& as) const noexcept {\n        return _shared_future.get_future(as);\n    }\n\n    /// \\brief Sets the shared_promise's value (as tuple; by copying), same as normal promise\n    void set_value(const value_tuple_type& result) noexcept(copy_noexcept) {\n        _promise.set_value(result);\n    }\n\n    /// \\brief Sets the shared_promise's value (as tuple; by moving), same as normal promise\n    void set_value(value_tuple_type&& result) noexcept {\n        _promise.set_value(std::move(result));\n    }\n\n    /// \\brief Sets the shared_promise's value (variadic), same as normal promise\n    template <typename... A>\n    void set_value(A&&... a) noexcept {\n        _promise.set_value(std::forward<A>(a)...);\n    }\n\n    /// \\brief Marks the shared_promise as failed, same as normal promise\n    void set_exception(std::exception_ptr ex) noexcept {\n        _promise.set_exception(std::move(ex));\n    }\n\n    /// \\brief Marks the shared_promise as failed, same as normal promise\n    template<typename Exception>\n    void set_exception(Exception&& e) noexcept {\n        set_exception(make_exception_ptr(std::forward<Exception>(e)));\n    }\n\n    /// \\brief Returns true if the underlying future is available (ready or failed)\n    bool available() const noexcept {\n        return _shared_future.available();\n    }\n\n    /// \\brief Returns true if the underlying future is  failed\n    bool failed() const noexcept {\n        return _shared_future.failed();\n    }\n};\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/shared_mutex.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/util/assert.hh>\n#include <mutex>\n#include <utility>\n#include <shared_mutex>\n\nnamespace seastar {\n\n\n/// \\addtogroup fiber-module\n/// @{\n\n/// \\brief Shared/exclusive mutual exclusion.\n///\n/// Similar to \\c std::shared_mutex, this class provides protection\n/// for a shared resource, with two levels of access protection: shared\n/// and exclusive.  Shared access allows multiple tasks to access the\n/// shared resource concurrently, while exclusive access allows just\n/// one task to access the resource at a time.\n///\n/// Note that many seastar tasks do not require protection at all,\n/// since the seastar scheduler is not preemptive; however tasks that do\n/// (by waiting on a future) may require explicit locking.\n///\n/// The \\ref with_shared(shared_mutex&, Func&&) and\n/// \\ref with_lock(shared_mutex&, Func&&) provide exception-safe\n/// wrappers for use with \\c shared_mutex.\n///\n/// \\see semaphore simpler mutual exclusion\nclass shared_mutex {\n    unsigned _readers = 0;\n    bool _writer = false;\n    struct waiter {\n        waiter(promise<>&& pr, bool for_write) : pr(std::move(pr)), for_write(for_write) {}\n        promise<> pr;\n        bool for_write;\n    };\n    chunked_fifo<waiter> _waiters;\npublic:\n    shared_mutex() = default;\n    shared_mutex(shared_mutex&&) = default;\n    shared_mutex& operator=(shared_mutex&&) = default;\n    shared_mutex(const shared_mutex&) = delete;\n    void operator=(const shared_mutex&) = delete;\n    /// Lock the \\c shared_mutex for shared access\n    ///\n    /// \\return a future that becomes ready when no exclusive access\n    ///         is granted to anyone.\n    future<> lock_shared() noexcept {\n        if (try_lock_shared()) {\n            return make_ready_future<>();\n        }\n        try {\n            _waiters.emplace_back(promise<>(), false);\n            return _waiters.back().pr.get_future();\n        } catch (...) {\n            return current_exception_as_future();\n        }\n    }\n    /// Try to lock the \\c shared_mutex for shared access\n    ///\n    /// \\return true iff could acquire the lock for shared access.\n    bool try_lock_shared() noexcept {\n        if (!_writer && _waiters.empty()) {\n            ++_readers;\n            return true;\n        }\n        return false;\n    }\n    /// Unlocks a \\c shared_mutex after a previous call to \\ref lock_shared().\n    void unlock_shared() noexcept {\n        SEASTAR_ASSERT(_readers > 0);\n        --_readers;\n        wake();\n    }\n    /// Lock the \\c shared_mutex for exclusive access\n    ///\n    /// \\return a future that becomes ready when no access, shared or exclusive\n    ///         is granted to anyone.\n    future<> lock() noexcept {\n        if (try_lock()) {\n            return make_ready_future<>();\n        }\n        try {\n            _waiters.emplace_back(promise<>(), true);\n            return _waiters.back().pr.get_future();\n        } catch (...) {\n            return current_exception_as_future();\n        }\n    }\n    /// Try to lock the \\c shared_mutex for exclusive access\n    ///\n    /// \\return true iff could acquire the lock for exclusive access.\n    bool try_lock() noexcept {\n        if (!_readers && !_writer) {\n            _writer = true;\n            return true;\n        }\n        return false;\n    }\n    /// Unlocks a \\c shared_mutex after a previous call to \\ref lock().\n    void unlock() noexcept {\n        SEASTAR_ASSERT(_writer);\n        _writer = false;\n        wake();\n    }\nprivate:\n    void wake() noexcept {\n        while (!_waiters.empty()) {\n            auto& w = _waiters.front();\n            // note: _writer == false in wake()\n            if (w.for_write) {\n                if (!_readers) {\n                    _writer = true;\n                    w.pr.set_value();\n                    _waiters.pop_front();\n                }\n                break;\n            } else { // for read\n                ++_readers;\n                w.pr.set_value();\n                _waiters.pop_front();\n            }\n        }\n    }\n};\n\n/// Executes a function while holding shared access to a resource.\n///\n/// Executes a function while holding shared access to a resource.  When\n/// the function returns, the mutex is automatically unlocked.\n///\n/// \\param sm a \\ref shared_mutex guarding access to the shared resource\n/// \\param func callable object to invoke while the mutex is held for shared access\n/// \\return whatever \\c func returns, as a future\n///\n/// \\relates shared_mutex\ntemplate <std::invocable Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    inline\n    futurize_t<std::invoke_result_t<Func>>\nwith_shared(shared_mutex& sm, Func&& func) noexcept {\n    return sm.lock_shared().then([&sm, func = std::forward<Func>(func)] () mutable {\n        return futurize_invoke(func).finally([&sm] {\n            sm.unlock_shared();\n        });\n    });\n}\n\ntemplate <std::invocable Func>\n    requires (!std::is_nothrow_move_constructible_v<Func>)\n    inline\n    futurize_t<std::invoke_result_t<Func>>\nwith_shared(shared_mutex& sm, Func&& func) noexcept {\n    // FIXME: use a coroutine when c++17 support is dropped\n    try {\n        return do_with(std::forward<Func>(func), [&sm] (Func& func) {\n            return sm.lock_shared().then([&func] {\n                return func();\n            }).finally([&sm] {\n                sm.unlock_shared();\n            });\n        });\n    } catch (...) {\n        return futurize<std::invoke_result_t<Func>>::current_exception_as_future();\n    }\n}\n\n/// Executes a function while holding exclusive access to a resource.\n///\n/// Executes a function while holding exclusive access to a resource.  When\n/// the function returns, the mutex is automatically unlocked.\n///\n/// \\param sm a \\ref shared_mutex guarding access to the shared resource\n/// \\param func callable object to invoke while the mutex is held for shared access\n/// \\return whatever \\c func returns, as a future\n///\n/// \\relates shared_mutex\ntemplate <std::invocable Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    inline\n    futurize_t<std::invoke_result_t<Func>>\nwith_lock(shared_mutex& sm, Func&& func) noexcept {\n    return sm.lock().then([&sm, func = std::forward<Func>(func)] () mutable {\n        return futurize_invoke(func).finally([&sm] {\n            sm.unlock();\n        });\n    });\n}\n\n\ntemplate <std::invocable Func>\n    requires (!std::is_nothrow_move_constructible_v<Func>)\n    inline\n    futurize_t<std::invoke_result_t<Func>>\nwith_lock(shared_mutex& sm, Func&& func) noexcept {\n    // FIXME: use a coroutine when c++17 support is dropped\n    try {\n        return do_with(std::forward<Func>(func), [&sm] (Func& func) {\n            return sm.lock().then([&func] {\n                return func();\n            }).finally([&sm] {\n                sm.unlock();\n            });\n        });\n    } catch (...) {\n        return futurize<std::invoke_result_t<Func>>::current_exception_as_future();\n    }\n}\n\nnamespace internal {\n\n/// Seastar analogue of the `BasicLockable` named requirement, see:\n///   [https://eel.is/c++draft/thread.req.lockable.basic#def:Cpp17BasicLockable]\ntemplate <typename T>\nconcept FutureBasicLockable = requires (T& t) {\n    { t.lock() } -> std::same_as<future<>>;\n    { t.unlock() } noexcept -> std::same_as<void>;\n};\n\n/// Seastar analogue of the `SharedLockable` named requirement, see:\n///   [https://eel.is/c++draft/thread.req.lockable.shared#def:Cpp17SharedLockable]\n/// It's called `basic` because the concept doesn't require the type\n/// to define `try_lock_shared()` method to satisfy the constraint, similarly\n/// to the `BasicLockable` named requirement.\ntemplate <typename T>\nconcept FutureBasicSharedLockable = requires (T& t) {\n    { t.lock_shared() } -> std::same_as<future<>>;\n    { t.unlock_shared() } noexcept -> std::same_as<void>;\n};\n\n} // namespace internal\n\n/// \\brief Construct a RAII-based shared lock corresponding to a given object.\n///\n/// Since constructors cannot be exectued asynchronously, use this function\n/// to shared lock the passed object and construct an std::shared_lock\n/// that will unlock it when the lock is being destroyed.\n///\n/// The caller is responsible for keeping the passed object\n/// alive at least until the returned lock is destroyed.\ntemplate <internal::FutureBasicSharedLockable T>\nfuture<std::shared_lock<T>> get_shared_lock(T& t) {\n    co_await t.lock_shared();\n\n    try {\n        co_return std::shared_lock<T>{t, std::adopt_lock_t{}};\n    } catch (...) {\n        t.unlock_shared();\n        throw;\n    }\n}\n\n/// \\brief Construct a RAII-based unique lock corresponding to a given object.\n///\n/// Since constructors cannot be exectued asynchronously, use this function\n/// to lock the passed object and construct an std::unique_lock\n/// that will unlock it when the lock is being destroyed.\n///\n/// The caller is responsible for keeping the passed object\n/// alive at least until the returned lock is destroyed.\ntemplate <internal::FutureBasicLockable T>\nfuture<std::unique_lock<T>> get_unique_lock(T& t) {\n    co_await t.lock();\n\n    try {\n        co_return std::unique_lock<T>{t, std::adopt_lock_t{}};\n    } catch (...) {\n        t.unlock();\n        throw;\n    }\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/shared_ptr.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/shared_ptr_debug_helper.hh>\n#include <seastar/util/is_smart_ptr.hh>\n#include <seastar/util/indirect.hh>\n#include <boost/intrusive/parent_from_member.hpp>\n#include <fmt/core.h>\n#include <concepts>\n#include <ostream>\n#include <type_traits>\n#include <utility>\n\n#if defined(__GNUC__) && !defined(__clang__) && (__GNUC__ >= 12)\n// to silence the false alarm from GCC 12, see\n// https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105204\n#define SEASTAR_IGNORE_USE_AFTER_FREE\n#endif\n\nnamespace seastar {\n\n// This header defines two shared pointer facilities, lw_shared_ptr<> and\n// shared_ptr<>, both modeled after std::shared_ptr<>.\n//\n// Unlike std::shared_ptr<>, neither of these implementations are thread\n// safe, and two pointers sharing the same object must not be used in\n// different threads.\n//\n// lw_shared_ptr<> is the more lightweight variant, with a lw_shared_ptr<>\n// occupying just one machine word, and adding just one word to the shared\n// object.  However, it does not support polymorphism.\n//\n// shared_ptr<> is more expensive, with a pointer occupying two machine\n// words, and with two words of overhead in the shared object.  In return,\n// it does support polymorphism.\n//\n// Both variants support shared_from_this() via enable_shared_from_this<>\n// and lw_enable_shared_from_this<>().\n//\n\n\n#ifndef SEASTAR_DEBUG_SHARED_PTR\nusing shared_ptr_counter_type = long;\n#else\nusing shared_ptr_counter_type = debug_shared_ptr_counter_type;\n#endif\n\ntemplate <typename T>\nclass lw_shared_ptr;\n\ntemplate <typename T>\nclass shared_ptr;\n\ntemplate <typename T>\nclass enable_lw_shared_from_this;\n\ntemplate <typename T>\nclass enable_shared_from_this;\n\ntemplate <typename T, typename... A>\nlw_shared_ptr<T> make_lw_shared(A&&... a);\n\ntemplate <typename T>\nlw_shared_ptr<T> make_lw_shared(T&& a);\n\ntemplate <typename T>\nlw_shared_ptr<T> make_lw_shared(T& a);\n\ntemplate <typename T, typename... A>\nshared_ptr<T> make_shared(A&&... a);\n\ntemplate <typename T>\nshared_ptr<T> make_shared(T&& a);\n\ntemplate <typename T, typename U>\nshared_ptr<T> static_pointer_cast(const shared_ptr<U>& p);\n\ntemplate <typename T, typename U>\nshared_ptr<T> dynamic_pointer_cast(const shared_ptr<U>& p);\n\ntemplate <typename T, typename U>\nshared_ptr<T> const_pointer_cast(const shared_ptr<U>& p);\n\nstruct lw_shared_ptr_counter_base {\n    shared_ptr_counter_type _count = 0;\n};\n\n\nnamespace internal {\n\ntemplate <class T, class U>\nstruct lw_shared_ptr_accessors;\n\ntemplate <class T>\nstruct lw_shared_ptr_accessors_esft;\n\ntemplate <class T>\nstruct lw_shared_ptr_accessors_no_esft;\n\n}\n\n\n// We want to support two use cases for shared_ptr<T>:\n//\n//   1. T is any type (primitive or class type)\n//\n//   2. T is a class type that inherits from enable_shared_from_this<T>.\n//\n// In the first case, we must wrap T in an object containing the counter,\n// since T may be a primitive type and cannot be a base class.\n//\n// In the second case, we want T to reach the counter through its\n// enable_shared_from_this<> base class, so that we can implement\n// shared_from_this().\n//\n// To implement those two conflicting requirements (T alongside its counter;\n// T inherits from an object containing the counter) we use std::conditional<>\n// and some accessor functions to select between two implementations.\n\n\n// CRTP from this to enable shared_from_this:\ntemplate <typename T>\nclass enable_lw_shared_from_this : private lw_shared_ptr_counter_base {\n    using ctor = T;\nprotected:\n    enable_lw_shared_from_this() noexcept {}\n    enable_lw_shared_from_this(enable_lw_shared_from_this&&) noexcept {}\n    enable_lw_shared_from_this(const enable_lw_shared_from_this&) noexcept {}\n    enable_lw_shared_from_this& operator=(const enable_lw_shared_from_this&) noexcept { return *this; }\n    enable_lw_shared_from_this& operator=(enable_lw_shared_from_this&&) noexcept { return *this; }\npublic:\n    lw_shared_ptr<T> shared_from_this() noexcept;\n    lw_shared_ptr<const T> shared_from_this() const noexcept;\n    long use_count() const noexcept { return _count; }\n\n    template <typename X>\n    friend class lw_shared_ptr;\n    template <typename X>\n    friend struct internal::lw_shared_ptr_accessors_esft;\n    template <typename X, class Y>\n    friend struct internal::lw_shared_ptr_accessors;\n};\n\ntemplate <typename T>\nstruct lw_shared_ptr_no_esft : private lw_shared_ptr_counter_base {\n    T _value;\n\n    lw_shared_ptr_no_esft() = default;\n    lw_shared_ptr_no_esft(const T& x) : _value(x) {}\n    lw_shared_ptr_no_esft(T&& x) : _value(std::move(x)) {}\n    template <typename... A>\n    lw_shared_ptr_no_esft(A&&... a) : _value(std::forward<A>(a)...) {}\n\n    template <typename X>\n    friend class lw_shared_ptr;\n    template <typename X>\n    friend struct internal::lw_shared_ptr_accessors_no_esft;\n    template <typename X, class Y>\n    friend struct internal::lw_shared_ptr_accessors;\n};\n\n\n/// Extension point: the user may override this to change how \\ref lw_shared_ptr objects are destroyed,\n/// primarily so that incomplete classes can be used.\n///\n/// Customizing the deleter requires that \\c T be derived from \\c enable_lw_shared_from_this<T>.\n/// The specialization must be visible for all uses of \\c lw_shared_ptr<T>.\n///\n/// To customize, the template must have a `static void dispose(T*)` operator that disposes of\n/// the object.\ntemplate <typename T>\nstruct lw_shared_ptr_deleter;  // No generic implementation\n\nnamespace internal {\n\ntemplate <typename T>\nstruct lw_shared_ptr_accessors_esft {\n    using concrete_type = std::remove_const_t<T>;\n    static T* to_value(lw_shared_ptr_counter_base* counter) {\n        return static_cast<T*>(counter);\n    }\n    static void dispose(lw_shared_ptr_counter_base* counter) {\n        dispose(static_cast<T*>(counter));\n    }\n    static void dispose(T* value_ptr) {\n        delete value_ptr;\n    }\n    static void instantiate_to_value(lw_shared_ptr_counter_base*) {\n        // since to_value() is defined above, we don't need to do anything special\n        // to force-instantiate it\n    }\n};\n\ntemplate <typename T>\nstruct lw_shared_ptr_accessors_no_esft {\n    using concrete_type = lw_shared_ptr_no_esft<T>;\n    static T* to_value(lw_shared_ptr_counter_base* counter) {\n        return &static_cast<concrete_type*>(counter)->_value;\n    }\n    static void dispose(lw_shared_ptr_counter_base* counter) {\n        delete static_cast<concrete_type*>(counter);\n    }\n    static void dispose(T* value_ptr) {\n        delete boost::intrusive::get_parent_from_member(value_ptr, &concrete_type::_value);\n    }\n    static void instantiate_to_value(lw_shared_ptr_counter_base*) {\n        // since to_value() is defined above, we don't need to do anything special\n        // to force-instantiate it\n    }\n};\n\n// Generic case: lw_shared_ptr_deleter<T> is not specialized, select\n// implementation based on whether T inherits from enable_lw_shared_from_this<T>.\ntemplate <typename T, typename U = void>\nstruct lw_shared_ptr_accessors : std::conditional_t<\n         std::is_base_of_v<enable_lw_shared_from_this<T>, T>,\n         lw_shared_ptr_accessors_esft<T>,\n         lw_shared_ptr_accessors_no_esft<T>> {\n};\n\n// void_t is C++17, use this temporarily\ntemplate <typename... T>\nusing void_t = void;\n\n// Overload when lw_shared_ptr_deleter<T> specialized\ntemplate <typename T>\nstruct lw_shared_ptr_accessors<T, void_t<decltype(lw_shared_ptr_deleter<T>{})>> {\n    using concrete_type = T;\n    static T* to_value(lw_shared_ptr_counter_base* counter);\n    static void dispose(lw_shared_ptr_counter_base* counter) {\n        lw_shared_ptr_deleter<T>::dispose(to_value(counter));\n    }\n    static void instantiate_to_value(lw_shared_ptr_counter_base* p) {\n        // instantiate to_value(); must be defined by shared_ptr_incomplete.hh\n        to_value(p);\n    }\n};\n\n}\n\ntemplate <typename T>\nclass lw_shared_ptr {\n    template <typename U>\n    using accessors = internal::lw_shared_ptr_accessors<std::remove_const_t<U>>;\n\n    mutable lw_shared_ptr_counter_base* _p = nullptr;\nprivate:\n    lw_shared_ptr(lw_shared_ptr_counter_base* p) noexcept : _p(p) {\n        if (_p) {\n            ++_p->_count;\n        }\n    }\n    template <typename... A>\n    static lw_shared_ptr make(A&&... a) {\n        auto p = new typename accessors<T>::concrete_type(std::forward<A>(a)...);\n        accessors<T>::instantiate_to_value(p);\n        return lw_shared_ptr(p);\n    }\npublic:\n    using element_type = T;\n\n    // Destroys the object pointed to by p and disposes of its storage.\n    // The pointer to the object must have been obtained through release().\n    static void dispose(T* p) noexcept {\n        accessors<T>::dispose(const_cast<std::remove_const_t<T>*>(p));\n    }\n\n    // A functor which calls dispose().\n    class disposer {\n    public:\n        void operator()(T* p) const noexcept {\n            dispose(p);\n        }\n    };\n\n    lw_shared_ptr() noexcept = default;\n    lw_shared_ptr(std::nullptr_t) noexcept : lw_shared_ptr() {}\n    lw_shared_ptr(const lw_shared_ptr& x) noexcept : _p(x._p) {\n        if (_p) {\n#pragma GCC diagnostic push\n#ifdef SEASTAR_IGNORE_USE_AFTER_FREE\n#pragma GCC diagnostic ignored \"-Wuse-after-free\"\n#endif\n            ++_p->_count;\n#pragma GCC diagnostic pop\n        }\n    }\n    lw_shared_ptr(lw_shared_ptr&& x) noexcept  : _p(x._p) {\n        x._p = nullptr;\n    }\n    [[gnu::always_inline]]\n    ~lw_shared_ptr() {\n#pragma GCC diagnostic push\n#ifdef SEASTAR_IGNORE_USE_AFTER_FREE\n#pragma GCC diagnostic ignored \"-Wuse-after-free\"\n#endif\n        if (_p && !--_p->_count) {\n            accessors<T>::dispose(_p);\n        }\n#pragma GCC diagnostic pop\n    }\n    lw_shared_ptr& operator=(const lw_shared_ptr& x) noexcept {\n        if (_p != x._p) {\n            this->~lw_shared_ptr();\n            new (this) lw_shared_ptr(x);\n        }\n        return *this;\n    }\n    lw_shared_ptr& operator=(lw_shared_ptr&& x) noexcept {\n        if (_p != x._p) {\n            this->~lw_shared_ptr();\n            new (this) lw_shared_ptr(std::move(x));\n        }\n        return *this;\n    }\n    lw_shared_ptr& operator=(std::nullptr_t) noexcept {\n        return *this = lw_shared_ptr();\n    }\n\n    T& operator*() const noexcept { return *accessors<T>::to_value(_p); }\n    T* operator->() const noexcept { return accessors<T>::to_value(_p); }\n    T* get() const noexcept {\n        if (_p) {\n            return accessors<T>::to_value(_p);\n        } else {\n            return nullptr;\n        }\n    }\n\n    // Releases ownership of the object without destroying it.\n    // If this was the last owner then returns an engaged unique_ptr\n    // which is now the sole owner of the object.\n    // Returns a disengaged pointer if there are still some owners.\n    //\n    // Note that in case the raw pointer is extracted from the unique_ptr\n    // using unique_ptr::release(), it must be still destroyed using\n    // lw_shared_ptr::disposer or lw_shared_ptr::dispose().\n    std::unique_ptr<T, disposer> release() noexcept {\n        auto p = std::exchange(_p, nullptr);\n        if (--p->_count) {\n            return nullptr;\n        } else {\n            return std::unique_ptr<T, disposer>(accessors<T>::to_value(p));\n        }\n    }\n\n    long int use_count() const noexcept {\n        if (_p) {\n            return _p->_count;\n        } else {\n            return 0;\n        }\n    }\n\n    operator lw_shared_ptr<const T>() const noexcept {\n        return lw_shared_ptr<const T>(_p);\n    }\n\n    explicit operator bool() const noexcept {\n        return _p;\n    }\n\n    bool owned() const noexcept {\n        return _p->_count == 1;\n    }\n\n    bool operator==(const lw_shared_ptr<const T>& x) const {\n        return _p == x._p;\n    }\n\n    bool operator!=(const lw_shared_ptr<const T>& x) const {\n        return !operator==(x);\n    }\n\n    bool operator==(const lw_shared_ptr<std::remove_const_t<T>>& x) const {\n        return _p == x._p;\n    }\n\n    bool operator!=(const lw_shared_ptr<std::remove_const_t<T>>& x) const {\n        return !operator==(x);\n    }\n\n    bool operator<(const lw_shared_ptr<const T>& x) const {\n        return _p < x._p;\n    }\n\n    bool operator<(const lw_shared_ptr<std::remove_const_t<T>>& x) const {\n        return _p < x._p;\n    }\n\n    template <typename U>\n    friend class lw_shared_ptr;\n\n    template <typename X, typename... A>\n    friend lw_shared_ptr<X> make_lw_shared(A&&...);\n\n    template <typename U>\n    friend lw_shared_ptr<U> make_lw_shared(U&&);\n\n    template <typename U>\n    friend lw_shared_ptr<U> make_lw_shared(U&);\n\n    template <typename U>\n    friend class enable_lw_shared_from_this;\n};\n\ntemplate <typename T, typename... A>\ninline\nlw_shared_ptr<T> make_lw_shared(A&&... a) {\n    return lw_shared_ptr<T>::make(std::forward<A>(a)...);\n}\n\ntemplate <typename T>\ninline\nlw_shared_ptr<T> make_lw_shared(T&& a) {\n    return lw_shared_ptr<T>::make(std::move(a));\n}\n\ntemplate <typename T>\ninline\nlw_shared_ptr<T> make_lw_shared(T& a) {\n    return lw_shared_ptr<T>::make(a);\n}\n\ntemplate <typename T>\ninline\nlw_shared_ptr<T>\nenable_lw_shared_from_this<T>::shared_from_this() noexcept {\n    return lw_shared_ptr<T>(this);\n}\n\ntemplate <typename T>\ninline\nlw_shared_ptr<const T>\nenable_lw_shared_from_this<T>::shared_from_this() const noexcept {\n    return lw_shared_ptr<const T>(const_cast<enable_lw_shared_from_this*>(this));\n}\n\ntemplate <typename T>\ninline\nstd::ostream& operator<<(std::ostream& out, const lw_shared_ptr<T>& p) {\n    if (!p) {\n        return out << \"null\";\n    }\n    return out << *p;\n}\n\n// Polymorphic shared pointer class\n\nstruct shared_ptr_count_base {\n    // destructor is responsible for fully-typed deletion\n    virtual ~shared_ptr_count_base() {}\n    shared_ptr_counter_type count = 0;\n};\n\ntemplate <typename T>\nstruct shared_ptr_count_for : shared_ptr_count_base {\n    T data;\n    template <typename... A>\n    shared_ptr_count_for(A&&... a) : data(std::forward<A>(a)...) {}\n};\n\ntemplate <typename T>\nclass enable_shared_from_this : private shared_ptr_count_base {\npublic:\n    shared_ptr<T> shared_from_this() noexcept;\n    shared_ptr<const T> shared_from_this() const noexcept;\n    long use_count() const noexcept { return count; }\n\n    template <typename U>\n    friend class shared_ptr;\n\n    template <typename U, bool esft>\n    friend struct shared_ptr_make_helper;\n};\n\ntemplate <typename T>\nclass shared_ptr {\n    mutable shared_ptr_count_base* _b = nullptr;\n    mutable T* _p = nullptr;\nprivate:\n    explicit shared_ptr(shared_ptr_count_for<T>* b) noexcept : _b(b), _p(&b->data) {\n        ++_b->count;\n    }\n    shared_ptr(shared_ptr_count_base* b, T* p) noexcept : _b(b), _p(p) {\n        if (_b) {\n            ++_b->count;\n        }\n    }\n    explicit shared_ptr(enable_shared_from_this<std::remove_const_t<T>>* p) noexcept : _b(p), _p(static_cast<T*>(p)) {\n        if (_b) {\n            ++_b->count;\n        }\n    }\npublic:\n    using element_type = T;\n\n    shared_ptr() noexcept = default;\n    shared_ptr(std::nullptr_t) noexcept : shared_ptr() {}\n    shared_ptr(const shared_ptr& x) noexcept\n            : _b(x._b)\n            , _p(x._p) {\n        if (_b) {\n            ++_b->count;\n        }\n    }\n    shared_ptr(shared_ptr&& x) noexcept\n            : _b(x._b)\n            , _p(x._p) {\n        x._b = nullptr;\n        x._p = nullptr;\n    }\n    template <std::derived_from<T> U>\n    shared_ptr(const shared_ptr<U>& x) noexcept\n            : _b(x._b)\n            , _p(x._p) {\n        if (_b) {\n            ++_b->count;\n        }\n    }\n    template <std::derived_from<T> U>\n    shared_ptr(shared_ptr<U>&& x) noexcept\n            : _b(x._b)\n            , _p(x._p) {\n        x._b = nullptr;\n        x._p = nullptr;\n    }\n    ~shared_ptr() {\n#pragma GCC diagnostic push\n#ifdef SEASTAR_IGNORE_USE_AFTER_FREE\n#pragma GCC diagnostic ignored \"-Wuse-after-free\"\n#endif\n        if (_b && !--_b->count) {\n            delete _b;\n        }\n#pragma GCC diagnostic pop\n    }\n    shared_ptr& operator=(const shared_ptr& x) noexcept {\n        if (this != &x) {\n            this->~shared_ptr();\n            new (this) shared_ptr(x);\n        }\n        return *this;\n    }\n    shared_ptr& operator=(shared_ptr&& x) noexcept {\n        if (this != &x) {\n            this->~shared_ptr();\n            new (this) shared_ptr(std::move(x));\n        }\n        return *this;\n    }\n    shared_ptr& operator=(std::nullptr_t) noexcept {\n        return *this = shared_ptr();\n    }\n    template <std::derived_from<T> U>\n    shared_ptr& operator=(const shared_ptr<U>& x) noexcept {\n        if (*this != x) {\n            this->~shared_ptr();\n            new (this) shared_ptr(x);\n        }\n        return *this;\n    }\n    template <std::derived_from<T> U>\n    shared_ptr& operator=(shared_ptr<U>&& x) noexcept {\n        if (*this != x) {\n            this->~shared_ptr();\n            new (this) shared_ptr(std::move(x));\n        }\n        return *this;\n    }\n    explicit operator bool() const noexcept {\n        return _p;\n    }\n    T& operator*() const noexcept {\n        return *_p;\n    }\n    T* operator->() const noexcept {\n        return _p;\n    }\n    T* get() const noexcept {\n        return _p;\n    }\n    long use_count() const noexcept {\n        if (_b) {\n            return _b->count;\n        } else {\n            return 0;\n        }\n    }\n\n    template <bool esft>\n    struct make_helper;\n\n    template <typename U, typename... A>\n    friend shared_ptr<U> make_shared(A&&... a);\n\n    template <typename U>\n    friend shared_ptr<U> make_shared(U&& a);\n\n    template <typename V, typename U>\n    friend shared_ptr<V> static_pointer_cast(const shared_ptr<U>& p);\n\n    template <typename V, typename U>\n    friend shared_ptr<V> dynamic_pointer_cast(const shared_ptr<U>& p);\n\n    template <typename V, typename U>\n    friend shared_ptr<V> const_pointer_cast(const shared_ptr<U>& p);\n\n    template <bool esft, typename... A>\n    static shared_ptr make(A&&... a);\n\n    template <typename U>\n    friend class enable_shared_from_this;\n\n    template <typename U, bool esft>\n    friend struct shared_ptr_make_helper;\n\n    template <typename U>\n    friend class shared_ptr;\n};\n\ntemplate <typename U, bool esft>\nstruct shared_ptr_make_helper;\n\ntemplate <typename T>\nstruct shared_ptr_make_helper<T, false> {\n    template <typename... A>\n    static shared_ptr<T> make(A&&... a) {\n        return shared_ptr<T>(new shared_ptr_count_for<T>(std::forward<A>(a)...));\n    }\n};\n\ntemplate <typename T>\nstruct shared_ptr_make_helper<T, true> {\n    template <typename... A>\n    static shared_ptr<T> make(A&&... a) {\n        auto p = new T(std::forward<A>(a)...);\n        return shared_ptr<T>(p, p);\n    }\n};\n\ntemplate <typename T, typename... A>\ninline\nshared_ptr<T>\nmake_shared(A&&... a) {\n    using helper = shared_ptr_make_helper<T, std::is_base_of_v<shared_ptr_count_base, T>>;\n    return helper::make(std::forward<A>(a)...);\n}\n\ntemplate <typename T>\ninline\nshared_ptr<T>\nmake_shared(T&& a) {\n    using helper = shared_ptr_make_helper<T, std::is_base_of_v<shared_ptr_count_base, T>>;\n    return helper::make(std::forward<T>(a));\n}\n\ntemplate <typename T, typename U>\ninline\nshared_ptr<T>\nstatic_pointer_cast(const shared_ptr<U>& p) {\n    return shared_ptr<T>(p._b, static_cast<T*>(p._p));\n}\n\ntemplate <typename T, typename U>\ninline\nshared_ptr<T>\ndynamic_pointer_cast(const shared_ptr<U>& p) {\n    auto q = dynamic_cast<T*>(p._p);\n    return shared_ptr<T>(q ? p._b : nullptr, q);\n}\n\ntemplate <typename T, typename U>\ninline\nshared_ptr<T>\nconst_pointer_cast(const shared_ptr<U>& p) {\n    return shared_ptr<T>(p._b, const_cast<T*>(p._p));\n}\n\ntemplate <typename T>\ninline\nshared_ptr<T>\nenable_shared_from_this<T>::shared_from_this() noexcept {\n    auto unconst = reinterpret_cast<enable_shared_from_this<std::remove_const_t<T>>*>(this);\n    return shared_ptr<T>(unconst);\n}\n\ntemplate <typename T>\ninline\nshared_ptr<const T>\nenable_shared_from_this<T>::shared_from_this() const noexcept {\n    auto esft = const_cast<enable_shared_from_this*>(this);\n    auto unconst = reinterpret_cast<enable_shared_from_this<std::remove_const_t<T>>*>(esft);\n    return shared_ptr<const T>(unconst);\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator==(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() == y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator==(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() == nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator==(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr == y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator==(const lw_shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() == nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator==(std::nullptr_t, const lw_shared_ptr<T>& y) {\n    return nullptr == y.get();\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator!=(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() != y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator!=(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() != nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator!=(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr != y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator!=(const lw_shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() != nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator!=(std::nullptr_t, const lw_shared_ptr<T>& y) {\n    return nullptr != y.get();\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator<(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() < y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator<(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() < nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator<(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr < y.get();\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator<=(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() <= y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator<=(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() <= nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator<=(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr <= y.get();\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator>(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() > y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator>(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() > nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator>(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr > y.get();\n}\n\ntemplate <typename T, typename U>\ninline\nbool\noperator>=(const shared_ptr<T>& x, const shared_ptr<U>& y) {\n    return x.get() >= y.get();\n}\n\ntemplate <typename T>\ninline\nbool\noperator>=(const shared_ptr<T>& x, std::nullptr_t) {\n    return x.get() >= nullptr;\n}\n\ntemplate <typename T>\ninline\nbool\noperator>=(std::nullptr_t, const shared_ptr<T>& y) {\n    return nullptr >= y.get();\n}\n\ntemplate <typename T>\ninline\nstd::ostream& operator<<(std::ostream& out, const shared_ptr<T>& p) {\n    if (!p) {\n        return out << \"null\";\n    }\n    return out << *p;\n}\n\ntemplate<typename T>\nusing shared_ptr_equal_by_value = indirect_equal_to<shared_ptr<T>>;\n\ntemplate<typename T>\nusing shared_ptr_value_hash = indirect_hash<shared_ptr<T>>;\n\n}\n\nnamespace std {\n\ntemplate <typename T>\nstruct hash<seastar::lw_shared_ptr<T>> : private hash<T*> {\n    size_t operator()(const seastar::lw_shared_ptr<T>& p) const {\n        return hash<T*>::operator()(p.get());\n    }\n};\n\ntemplate <typename T>\nstruct hash<seastar::shared_ptr<T>> : private hash<T*> {\n    size_t operator()(const seastar::shared_ptr<T>& p) const {\n        return hash<T*>::operator()(p.get());\n    }\n};\n\n}\n\nnamespace fmt {\n\ntemplate<typename T>\nconst void* ptr(const seastar::lw_shared_ptr<T>& p) {\n    return p.get();\n}\n\ntemplate<typename T>\nconst void* ptr(const seastar::shared_ptr<T>& p) {\n    return p.get();\n}\n\ntemplate <typename T>\nstruct formatter<seastar::shared_ptr<T>> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::shared_ptr<T>& p, fmt::format_context& ctx) const {\n        if (!p) {\n            return fmt::format_to(ctx.out(), \"null\");\n        }\n        return fmt::format_to(ctx.out(), \"{}\", *p);\n    }\n};\n\ntemplate <typename T>\nstruct formatter<seastar::lw_shared_ptr<T>> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::lw_shared_ptr<T>& p, fmt::format_context& ctx) const {\n        if (!p) {\n            return fmt::format_to(ctx.out(), \"null\");\n        }\n        return fmt::format_to(ctx.out(), \"{}\", *p);\n    }\n};\n}\n\nnamespace seastar {\n\ntemplate<typename T>\nstruct is_smart_ptr<shared_ptr<T>> : std::true_type {};\n\ntemplate<typename T>\nstruct is_smart_ptr<lw_shared_ptr<T>> : std::true_type {};\n\n}\n"
  },
  {
    "path": "include/seastar/core/shared_ptr_debug_helper.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#ifdef SEASTAR_DEBUG_SHARED_PTR\n\n#include <thread>\n#include <cassert>\n\n#include <seastar/core/on_internal_error.hh>\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\n// A counter that is only comfortable being incremented on the cpu\n// it was created on.  Useful for verifying that a shared_ptr\n// or lw_shared_ptr isn't misued across cores.\nclass debug_shared_ptr_counter_type {\n    long _counter = 0;\n    std::thread::id _cpu = std::this_thread::get_id();\npublic:\n    debug_shared_ptr_counter_type(long x) noexcept : _counter(x) {}\n    operator long() const {\n        check();\n        return _counter;\n    }\n    debug_shared_ptr_counter_type& operator++() {\n        check();\n        ++_counter;\n        return *this;\n    }\n    long operator++(int) {\n        check();\n        return _counter++;\n    }\n    debug_shared_ptr_counter_type& operator--() {\n        check();\n        --_counter;\n        return *this;\n    }\n    long operator--(int) {\n        check();\n        return _counter--;\n    }\nprivate:\n    void check() const {\n        if (__builtin_expect(_cpu != std::this_thread::get_id(), false)) {\n            on_fatal_internal_error(seastar_logger, \"shared_ptr accessed on non-owner cpu\");\n        }\n    }\n};\n\n}\n\n#endif\n\n"
  },
  {
    "path": "include/seastar/core/shared_ptr_incomplete.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/shared_ptr.hh>\n\n\n/// \\file\n/// \\brief Include this header files when using \\c lw_shared_ptr<some_incomplete_type>, at the point\n/// where \\c some_incomplete_type is defined.\n\nnamespace seastar {\n\nnamespace internal {\n\n// Overload when lw_shared_ptr_deleter<T> specialized\ntemplate <typename T>\nT*\nlw_shared_ptr_accessors<T, void_t<decltype(lw_shared_ptr_deleter<T>{})>>::to_value(lw_shared_ptr_counter_base* counter) {\n    return static_cast<T*>(counter);\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/signal.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n/// \\file\n\n// Seastar interface for POSIX signals.\n#include <seastar/util/noncopyable_function.hh>\n\nnamespace seastar {\n\n\n/// \\brief Sets a signal handler for the specified signal.\n///\n/// \\param signo Signal number.\n/// \\param handler Function to handle the signal.\n/// \\param once Should the handler be invoked only once.\nvoid handle_signal(int signo, noncopyable_function<void ()>&& handler, bool once = false);\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/simple-stream.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/util/variant_utils.hh>\n#include <algorithm>\n#include <concepts>\n#include <cstddef>\n#include <stdexcept>\n#include <type_traits>\n\nnamespace seastar {\n\n\nclass measuring_output_stream {\n    size_t _size = 0;\npublic:\n    void write(const char*, size_t size) {\n        _size += size;\n    }\n\n    size_t size() const {\n        return _size;\n    }\n};\n\ntemplate<typename>\nclass memory_output_stream;\n\nclass simple_memory_input_stream;\n\ntemplate<typename Iterator>\nclass fragmented_memory_input_stream;\n\ntemplate<typename Iterator>\nclass memory_input_stream;\n\nclass simple_memory_output_stream {\n    char* _p = nullptr;\n    size_t _size = 0;\npublic:\n    using has_with_stream = std::false_type;\n    simple_memory_output_stream() {}\n    simple_memory_output_stream(char* p, size_t size, size_t start = 0) : _p(p + start), _size(size) {}\n    char* begin() { return _p; }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"serialization buffer overflow\");\n        }\n        _p += size;\n        _size -= size;\n    }\n\n    [[gnu::always_inline]]\n    simple_memory_output_stream write_substream(size_t size) {\n       if (size > _size) {\n           throw std::out_of_range(\"serialization buffer overflow\");\n       }\n       simple_memory_output_stream substream(_p, size);\n       skip(size);\n       return substream;\n    }\n\n    [[gnu::always_inline]]\n    void write(const char* p, size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"serialization buffer overflow\");\n        }\n        std::copy_n(p, size, _p);\n        skip(size);\n    }\n\n    [[gnu::always_inline]]\n    void fill(char c, size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"serialization buffer overflow\");\n        }\n        std::fill_n(_p, size, c);\n        skip(size);\n    }\n\n    [[gnu::always_inline]]\n    size_t size() const {\n        return _size;\n    }\n\n    // simple_memory_output_stream is a write cursor that keeps a mutable view of some\n    // underlying buffer and provides write interface. to_input_stream() converts it\n    // to a read cursor that points to the same part of the buffer but provides\n    // read interface.\n    simple_memory_input_stream to_input_stream() const;\n};\n\ntemplate<typename Iterator>\nclass fragmented_memory_output_stream {\n    using simple = simple_memory_output_stream ;\n\n    Iterator _it;\n    simple _current;\n    size_t _size = 0;\n\n    friend class memory_input_stream<Iterator>;\nprivate:\n    template<typename Func>\n    //requires requires(Func f, view bv) { { f(bv) } -> void; }\n    [[gnu::always_inline]]\n    void for_each_fragment(size_t size, Func&& func) {\n        if (size > _size) {\n            throw std::out_of_range(\"serialization buffer overflow\");\n        }\n        _size -= size;\n        // Fast path, avoids memcpy for constant sizes.\n        if (__builtin_constant_p(size)) {\n            if (size <= _current.size()) [[likely]] {\n                func(_current.write_substream(size));\n                return;\n            }\n        }\n        for_each_fragment_slowpath(size, std::forward<Func>(func));\n    }\n    void for_each_fragment_slowpath(size_t size, std::invocable<simple_memory_output_stream> auto&& func) {\n        while (size) {\n            if (!_current.size()) {\n                _current = simple(reinterpret_cast<char*>((*_it).get_write()), (*_it).size());\n                _it++;\n            }\n            auto this_size = std::min(_current.size(), size);\n            func(_current.write_substream(this_size));\n            size -= this_size;\n        }\n    }\n    fragmented_memory_output_stream(Iterator it, simple_memory_output_stream bv, size_t size)\n        : _it(it), _current(bv), _size(size) { }\npublic:\n    using has_with_stream = std::false_type;\n    using iterator_type = Iterator;\n\n    fragmented_memory_output_stream() = default;\n\n    fragmented_memory_output_stream(Iterator it, size_t size)\n        : _it(it), _size(size) {\n    }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        for_each_fragment(size, [] (auto) { });\n    }\n    [[gnu::always_inline]]\n    memory_output_stream<Iterator> write_substream(size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"serialization buffer overflow\");\n        }\n        if (_current.size() >= size) {\n            _size -= size;\n            return _current.write_substream(size);\n        }\n        fragmented_memory_output_stream substream(_it, _current, size);\n        skip(size);\n        return substream;\n    }\n    [[gnu::always_inline]]\n    void write(const char* p, size_t size) {\n        for_each_fragment(size, [&p] (auto bv) {\n            std::copy_n(p, bv.size(), bv.begin());\n            p += bv.size();\n        });\n    }\n    [[gnu::always_inline]]\n    void fill(char c, size_t size) {\n        for_each_fragment(size, [c] (simple fragment) {\n            std::fill_n(fragment.begin(), fragment.size(), c);\n        });\n    }\n    size_t size() const {\n        return _size;\n    }\n\n    // fragmented_memory_input_stream is a write cursor that keeps a mutable view of some\n    // underlying fragmented buffer and provides write interface. to_input_stream() converts\n    // it to a read cursor that points to the same part of the buffer but provides read interface.\n    fragmented_memory_input_stream<Iterator> to_input_stream() const;\n};\n\ntemplate<typename Iterator>\nclass memory_output_stream {\npublic:\n    using simple = simple_memory_output_stream;\n    using fragmented = fragmented_memory_output_stream<Iterator>;\n\nprivate:\n    const bool _is_simple;\n    using fragmented_type = fragmented;\n    union {\n        simple _simple;\n        fragmented_type _fragmented;\n    };\npublic:\n    template<typename StreamVisitor>\n    [[gnu::always_inline]]\n    decltype(auto) with_stream(StreamVisitor&& visitor) {\n        if (__builtin_expect(_is_simple, true)) {\n            return visitor(_simple);\n        }\n        return visitor(_fragmented);\n    }\n\n    template<typename StreamVisitor>\n    [[gnu::always_inline]]\n    decltype(auto) with_stream(StreamVisitor&& visitor) const {\n        if (__builtin_expect(_is_simple, true)) {\n            return visitor(_simple);\n        }\n        return visitor(_fragmented);\n    }\npublic:\n    using has_with_stream = std::true_type;\n    using iterator_type = Iterator;\n    memory_output_stream()\n            : _is_simple(true), _simple() {}\n    memory_output_stream(simple stream)\n            : _is_simple(true), _simple(std::move(stream)) {}\n    memory_output_stream(fragmented stream)\n            : _is_simple(false), _fragmented(std::move(stream)) {}\n\n    [[gnu::always_inline]]\n    memory_output_stream(const memory_output_stream& other) noexcept : _is_simple(other._is_simple) {\n        // Making this copy constructor noexcept makes copy assignment simpler.\n        // Besides, performance of memory_output_stream relies on the fact that both\n        // fragmented and simple input stream are PODs and the branch below\n        // is optimized away, so throwable copy constructors aren't something\n        // we want.\n        static_assert(std::is_nothrow_copy_constructible_v<fragmented>,\n                      \"seastar::memory_output_stream::fragmented should be copy constructible\");\n        static_assert(std::is_nothrow_copy_constructible_v<simple>,\n                      \"seastar::memory_output_stream::simple should be copy constructible\");\n        if (_is_simple) {\n            new (&_simple) simple(other._simple);\n        } else {\n            new (&_fragmented) fragmented_type(other._fragmented);\n        }\n    }\n\n    [[gnu::always_inline]]\n    memory_output_stream(memory_output_stream&& other) noexcept : _is_simple(other._is_simple) {\n        if (_is_simple) {\n            new (&_simple) simple(std::move(other._simple));\n        } else {\n            new (&_fragmented) fragmented_type(std::move(other._fragmented));\n        }\n    }\n\n    [[gnu::always_inline]]\n    memory_output_stream& operator=(const memory_output_stream& other) noexcept {\n        // Copy constructor being noexcept makes copy assignment simpler.\n        static_assert(std::is_nothrow_copy_constructible_v<memory_output_stream>,\n                      \"memory_output_stream copy constructor shouldn't throw\");\n        if (this != &other) {\n            this->~memory_output_stream();\n            new (this) memory_output_stream(other);\n        }\n        return *this;\n    }\n\n    [[gnu::always_inline]]\n    memory_output_stream& operator=(memory_output_stream&& other) noexcept {\n        if (this != &other) {\n            this->~memory_output_stream();\n            new (this) memory_output_stream(std::move(other));\n        }\n        return *this;\n    }\n\n    [[gnu::always_inline]]\n    ~memory_output_stream() {\n        if (_is_simple) {\n            _simple.~simple();\n        } else {\n            _fragmented.~fragmented_type();\n        }\n    }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        with_stream([size] (auto& stream) {\n            stream.skip(size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    memory_output_stream write_substream(size_t size) {\n        return with_stream([size] (auto& stream) -> memory_output_stream {\n            return stream.write_substream(size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    void write(const char* p, size_t size) {\n        with_stream([p, size] (auto& stream) {\n            stream.write(p, size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    void fill(char c, size_t size) {\n        with_stream([c, size] (auto& stream) {\n            stream.fill(c, size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    size_t size() const {\n        return with_stream([] (auto& stream) {\n            return stream.size();\n        });\n    }\n\n    memory_input_stream<Iterator> to_input_stream() const;\n};\n\nclass simple_memory_input_stream {\n    using simple = simple_memory_input_stream;\n\n    const char* _p = nullptr;\n    size_t _size = 0;\npublic:\n    using has_with_stream = std::false_type;\n    simple_memory_input_stream() = default;\n    simple_memory_input_stream(const char* p, size_t size) : _p(p), _size(size) {}\n\n    const char* begin() const { return _p; }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"deserialization buffer underflow\");\n        }\n        _p += size;\n        _size -= size;\n    }\n\n    [[gnu::always_inline]]\n    simple read_substream(size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"deserialization buffer underflow\");\n        }\n        simple substream(_p, size);\n        skip(size);\n        return substream;\n    }\n\n    [[gnu::always_inline]]\n    void read(char* p, size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"deserialization buffer underflow\");\n        }\n        std::copy_n(_p, size, p);\n        skip(size);\n    }\n\n    template<typename Output>\n    [[gnu::always_inline]]\n    void copy_to(Output& out) const {\n        out.write(_p, _size);\n    }\n\n    [[gnu::always_inline]]\n    size_t size() const {\n        return _size;\n    }\n};\n\ntemplate<typename Iterator>\nclass fragmented_memory_input_stream {\n    using simple = simple_memory_input_stream;\n    using fragmented = fragmented_memory_input_stream;\n\n    Iterator _it;\n    simple _current;\n    size_t _size;\nprivate:\n    template<typename Func>\n    //requires requires(Func f, view bv) { { f(bv) } -> void; }\n    [[gnu::always_inline]]\n    void for_each_fragment(size_t size, Func&& func) {\n        if (size > _size) {\n            throw std::out_of_range(\"deserialization buffer underflow\");\n        }\n        _size -= size;\n        // Fast path, avoids memcpy for constant sizes.\n        if (__builtin_constant_p(size)) {\n            if (size <= _current.size()) [[likely]] {\n                func(_current.read_substream(size));\n                return;\n            }\n        }\n        for_each_fragment_slowpath(size, std::forward<Func>(func));\n    }\n    void for_each_fragment_slowpath(size_t size, std::invocable<simple_memory_input_stream> auto&& func) {\n        while (size) {\n            if (!_current.size()) {\n                _current = simple(reinterpret_cast<const char*>((*_it).begin()), (*_it).size());\n                _it++;\n            }\n            auto this_size = std::min(_current.size(), size);\n            func(_current.read_substream(this_size));\n            size -= this_size;\n        }\n    }\n    fragmented_memory_input_stream(Iterator it, simple bv, size_t size)\n        : _it(it), _current(bv), _size(size) { }\n    friend class fragmented_memory_output_stream<Iterator>;\npublic:\n    using has_with_stream = std::false_type;\n    using iterator_type = Iterator;\n    fragmented_memory_input_stream(Iterator it, size_t size)\n        : _it(it), _size(size) {\n    }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        for_each_fragment(size, [] (auto) { });\n    }\n    [[gnu::always_inline]]\n    fragmented read_substream(size_t size) {\n        if (size > _size) {\n            throw std::out_of_range(\"deserialization buffer underflow\");\n        }\n        fragmented substream(_it, _current, size);\n        skip(size);\n        return substream;\n    }\n    [[gnu::always_inline]]\n    void read(char* p, size_t size) {\n        for_each_fragment(size, [&p] (auto bv) {\n            p = std::copy_n(bv.begin(), bv.size(), p);\n        });\n    }\n    template<typename Output>\n    [[gnu::always_inline]]\n    void copy_to(Output& out) {\n        for_each_fragment(_size, [&out] (auto bv) {\n            bv.copy_to(out);\n        });\n    }\n    size_t size() const {\n        return _size;\n    }\n\n    const char* first_fragment_data() const { return _current.begin(); }\n    size_t first_fragment_size() const { return _current.size(); }\n    Iterator fragment_iterator() const { return _it; }\n};\n\n/*\ntemplate<typename Visitor>\nconcept bool StreamVisitor() {\n    return requires(Visitor visitor, simple& simple, fragmented& fragmented) {\n        visitor(simple);\n        visitor(fragmented);\n    };\n}\n*/\n// memory_input_stream performs type erasure optimized for cases where\n// simple is used.\n// By using a lot of [[gnu::always_inline]] attributes this class attempts to\n// make the compiler generate code with simple functions inlined\n// directly in the user of the intput_stream.\ntemplate<typename Iterator>\nclass memory_input_stream {\npublic:\n    using simple = simple_memory_input_stream;\n    using fragmented = fragmented_memory_input_stream<Iterator>;\nprivate:\n    const bool _is_simple;\n    using fragmented_type = fragmented;\n    union {\n        simple _simple;\n        fragmented_type _fragmented;\n    };\npublic:\n    template<typename StreamVisitor>\n    [[gnu::always_inline]]\n    decltype(auto) with_stream(StreamVisitor&& visitor) {\n        if (__builtin_expect(_is_simple, true)) {\n            return visitor(_simple);\n        }\n        return visitor(_fragmented);\n    }\n\n    template<typename StreamVisitor>\n    [[gnu::always_inline]]\n    decltype(auto) with_stream(StreamVisitor&& visitor) const {\n        if (__builtin_expect(_is_simple, true)) {\n            return visitor(_simple);\n        }\n        return visitor(_fragmented);\n    }\npublic:\n    using has_with_stream = std::true_type;\n    using iterator_type = Iterator;\n    memory_input_stream(simple stream)\n            : _is_simple(true), _simple(std::move(stream)) {}\n    memory_input_stream(fragmented stream)\n            : _is_simple(false), _fragmented(std::move(stream)) {}\n\n    [[gnu::always_inline]]\n    memory_input_stream(const memory_input_stream& other) noexcept : _is_simple(other._is_simple) {\n        // Making this copy constructor noexcept makes copy assignment simpler.\n        // Besides, performance of memory_input_stream relies on the fact that both\n        // fragmented and simple input stream are PODs and the branch below\n        // is optimized away, so throwable copy constructors aren't something\n        // we want.\n        static_assert(std::is_nothrow_copy_constructible_v<fragmented>,\n                      \"seastar::memory_input_stream::fragmented should be copy constructible\");\n        static_assert(std::is_nothrow_copy_constructible_v<simple>,\n                      \"seastar::memory_input_stream::simple should be copy constructible\");\n        if (_is_simple) {\n            new (&_simple) simple(other._simple);\n        } else {\n            new (&_fragmented) fragmented_type(other._fragmented);\n        }\n    }\n\n    [[gnu::always_inline]]\n    memory_input_stream(memory_input_stream&& other) noexcept : _is_simple(other._is_simple) {\n        if (_is_simple) {\n            new (&_simple) simple(std::move(other._simple));\n        } else {\n            new (&_fragmented) fragmented_type(std::move(other._fragmented));\n        }\n    }\n\n    [[gnu::always_inline]]\n    memory_input_stream& operator=(const memory_input_stream& other) noexcept {\n        // Copy constructor being noexcept makes copy assignment simpler.\n        static_assert(std::is_nothrow_copy_constructible_v<memory_input_stream>,\n                      \"memory_input_stream copy constructor shouldn't throw\");\n        if (this != &other) {\n            this->~memory_input_stream();\n            new (this) memory_input_stream(other);\n        }\n        return *this;\n    }\n\n    [[gnu::always_inline]]\n    memory_input_stream& operator=(memory_input_stream&& other) noexcept {\n        if (this != &other) {\n            this->~memory_input_stream();\n            new (this) memory_input_stream(std::move(other));\n        }\n        return *this;\n    }\n\n    [[gnu::always_inline]]\n    ~memory_input_stream() {\n        if (_is_simple) {\n            _simple.~simple_memory_input_stream();\n        } else {\n            _fragmented.~fragmented_type();\n        }\n    }\n\n    [[gnu::always_inline]]\n    void skip(size_t size) {\n        with_stream([size] (auto& stream) {\n            stream.skip(size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    memory_input_stream read_substream(size_t size) {\n        return with_stream([size] (auto& stream) -> memory_input_stream {\n            return stream.read_substream(size);\n        });\n    }\n\n    [[gnu::always_inline]]\n    void read(char* p, size_t size) {\n        with_stream([p, size] (auto& stream) {\n            stream.read(p, size);\n        });\n    }\n\n    template<typename Output>\n    [[gnu::always_inline]]\n    void copy_to(Output& out) {\n        with_stream([&out] (auto& stream) {\n            stream.copy_to(out);\n        });\n    }\n\n    [[gnu::always_inline]]\n    size_t size() const {\n        return with_stream([] (auto& stream) {\n            return stream.size();\n        });\n    }\n\n    template<typename Stream, typename StreamVisitor>\n    friend decltype(auto) with_serialized_stream(Stream& stream, StreamVisitor&& visitor);\n};\n\n\ninline simple_memory_input_stream simple_memory_output_stream::to_input_stream() const {\n    return simple_memory_input_stream(_p, _size);\n}\n\ntemplate<typename Iterator>\ninline fragmented_memory_input_stream<Iterator> fragmented_memory_output_stream<Iterator>::to_input_stream() const {\n    return fragmented_memory_input_stream<Iterator>(_it, _current.to_input_stream(), _size);\n}\n\ntemplate<typename Iterator>\ninline memory_input_stream<Iterator> memory_output_stream<Iterator>::to_input_stream() const {\n    return with_stream(make_visitor(\n        [] (const simple_memory_output_stream& ostream) -> memory_input_stream<Iterator> {\n            return ostream.to_input_stream();\n        },\n        [] (const fragmented_memory_output_stream<Iterator>& ostream) -> memory_input_stream<Iterator> {\n            return ostream.to_input_stream();\n        }\n    ));\n}\n\n// The purpose of the with_serialized_stream() is to minimize number of dynamic\n// dispatches. For example, a lot of IDL-generated code looks like this:\n// auto some_value() const {\n//     return seastar::with_serialized_stream(v, [] (auto& v) {\n//         auto in = v;\n//         ser::skip(in, boost::type<type1>());\n//         ser::skip(in, boost::type<type2>());\n//         return deserialize(in, boost::type<type3>());\n//     });\n// }\n// Using with_stream() there is at most one dynamic dispatch per such\n// function, instead of one per each skip() and deserialize() call.\n\ntemplate<typename Stream, typename StreamVisitor>\nrequires Stream::has_with_stream::value\n[[gnu::always_inline]]\n inline decltype(auto)\n with_serialized_stream(Stream& stream, StreamVisitor&& visitor) {\n    return stream.with_stream(std::forward<StreamVisitor>(visitor));\n}\n\ntemplate<typename Stream, typename StreamVisitor>\nrequires (!Stream::has_with_stream::value)\n[[gnu::always_inline]]\n inline decltype(auto)\n with_serialized_stream(Stream& stream, StreamVisitor&& visitor) {\n    return visitor(stream);\n}\n\nusing simple_input_stream = simple_memory_input_stream;\nusing simple_output_stream = simple_memory_output_stream;\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/slab.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#pragma once\n\n#include <boost/intrusive/unordered_set.hpp>\n#include <boost/intrusive/list.hpp>\n#include <stdlib.h>\n#include <stdio.h>\n#include <stdint.h>\n#include <assert.h>\n#include <vector>\n#include <algorithm>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nstatic constexpr uint16_t SLAB_MAGIC_NUMBER = 0x51AB; // meant to be 'SLAB' :-)\n\n/*\n * Item requirements\n * - Extend it to slab_item_base.\n * - First parameter of constructor must be uint32_t _slab_page_index.\n * - Implement get_slab_page_index() to return _slab_page_index.\n * - Implement is_unlocked() to check if Item can be evicted.\n */\n\n/*\n * slab_page_desc is 1:1 mapped to slab page.\n * footprint: 80b for each slab page.\n */\nstruct slab_page_desc {\nprivate:\n    boost::intrusive::list_member_hook<> _lru_link;\n    boost::intrusive::list_member_hook<> _free_pages_link;\n    void *_slab_page;\n    std::vector<uintptr_t> _free_objects;\n    uint32_t _refcnt;\n    uint32_t _index; // index into slab page vector\n    uint16_t _magic;\n    uint8_t _slab_class_id;\npublic:\n    slab_page_desc(void *slab_page, size_t objects, size_t object_size, uint8_t slab_class_id, uint32_t index)\n        : _slab_page(slab_page)\n        , _refcnt(0U)\n        , _index(index)\n        , _magic(SLAB_MAGIC_NUMBER)\n        , _slab_class_id(slab_class_id)\n    {\n        auto object = reinterpret_cast<uintptr_t>(slab_page);\n        _free_objects.reserve(objects - 1);\n        for (auto i = 1u; i < objects; i++) {\n            object += object_size;\n            _free_objects.push_back(object);\n        }\n    }\n\n    bool empty() const {\n        return _free_objects.empty();\n    }\n\n    size_t size() const {\n        return _free_objects.size();\n    }\n\n    uint32_t& refcnt() {\n        return _refcnt;\n    }\n\n    uint32_t index() const {\n        return _index;\n    }\n\n    uint16_t magic() const {\n        return _magic;\n    }\n\n    uint8_t slab_class_id() const {\n        return _slab_class_id;\n    }\n\n    void* slab_page() const {\n        return _slab_page;\n    }\n\n    std::vector<uintptr_t>& free_objects() {\n        return _free_objects;\n    }\n\n    void* allocate_object() {\n        SEASTAR_ASSERT(!_free_objects.empty());\n        auto object = reinterpret_cast<void*>(_free_objects.back());\n        _free_objects.pop_back();\n        return object;\n    }\n\n    void free_object(void *object) {\n        _free_objects.push_back(reinterpret_cast<uintptr_t>(object));\n    }\n\n    template<typename Item>\n    friend class slab_class;\n    template<typename Item>\n    friend class slab_allocator;\n};\n\nclass slab_item_base {\n    boost::intrusive::list_member_hook<> _lru_link;\n\n    template<typename Item>\n    friend class slab_class;\n};\n\ntemplate<typename Item>\nclass slab_class {\nprivate:\n    boost::intrusive::list<slab_page_desc,\n        boost::intrusive::member_hook<slab_page_desc, boost::intrusive::list_member_hook<>,\n        &slab_page_desc::_free_pages_link>> _free_slab_pages;\n    boost::intrusive::list<slab_item_base,\n        boost::intrusive::member_hook<slab_item_base, boost::intrusive::list_member_hook<>,\n        &slab_item_base::_lru_link>> _lru;\n    size_t _size; // size of objects\n    uint8_t _slab_class_id;\nprivate:\n    template<typename... Args>\n    inline\n    Item* create_item(void *object, uint32_t slab_page_index, Args&&... args) {\n        Item *new_item = new(object) Item(slab_page_index, std::forward<Args>(args)...);\n        _lru.push_front(reinterpret_cast<slab_item_base&>(*new_item));\n        return new_item;\n    }\n\n    inline\n    std::pair<void *, uint32_t> evict_lru_item(std::function<void (Item& item_ref)>& erase_func) {\n        if (_lru.empty()) {\n            return { nullptr, 0U };\n        }\n\n        Item& victim = reinterpret_cast<Item&>(_lru.back());\n        uint32_t index = victim.get_slab_page_index();\n        SEASTAR_ASSERT(victim.is_unlocked());\n        _lru.erase(_lru.iterator_to(reinterpret_cast<slab_item_base&>(victim)));\n        // WARNING: You need to make sure that erase_func will not release victim back to slab.\n        erase_func(victim);\n\n        return { reinterpret_cast<void*>(&victim), index };\n    }\npublic:\n    slab_class(size_t size, uint8_t slab_class_id)\n        : _size(size)\n        , _slab_class_id(slab_class_id)\n    {\n    }\n    slab_class(slab_class&&) = default;\n    ~slab_class() {\n        _free_slab_pages.clear();\n        _lru.clear();\n    }\n\n    size_t size() const {\n        return _size;\n    }\n\n    bool empty() const {\n        return _free_slab_pages.empty();\n    }\n\n    bool has_no_slab_pages() const {\n        return _lru.empty();\n    }\n\n    template<typename... Args>\n    Item *create(Args&&... args) {\n        SEASTAR_ASSERT(!_free_slab_pages.empty());\n        auto& desc = _free_slab_pages.back();\n        auto object = desc.allocate_object();\n        if (desc.empty()) {\n            // if empty, remove desc from the list of slab pages with free objects.\n            _free_slab_pages.erase(_free_slab_pages.iterator_to(desc));\n        }\n\n        return create_item(object, desc.index(), std::forward<Args>(args)...);\n    }\n\n    template<typename... Args>\n    Item *create_from_new_page(uint64_t max_object_size, uint32_t slab_page_index,\n                               std::function<void (slab_page_desc& desc)> insert_slab_page_desc,\n                               Args&&... args) {\n        // allocate slab page.\n        constexpr size_t alignment = std::alignment_of_v<Item>;\n        void *slab_page = aligned_alloc(alignment, max_object_size);\n        if (!slab_page) {\n            throw std::bad_alloc{};\n        }\n        // allocate descriptor to slab page.\n        slab_page_desc *desc = nullptr;\n        SEASTAR_ASSERT(_size % alignment == 0);\n        auto objects = max_object_size / _size;\n        try {\n            desc = new slab_page_desc(slab_page, objects, _size, _slab_class_id, slab_page_index);\n        } catch (const std::bad_alloc& e) {\n            ::free(slab_page);\n            throw std::bad_alloc{};\n        }\n\n        // Only add the page to the free list if it can hold more than one object.\n        // If it holds only one, it's already full since we are about to return it.\n        if (objects > 1) {\n            _free_slab_pages.push_front(*desc);\n        }\n        insert_slab_page_desc(*desc);\n\n        // first object from the allocated slab page is returned.\n        return create_item(slab_page, slab_page_index, std::forward<Args>(args)...);\n    }\n\n    template<typename... Args>\n    Item *create_from_lru(std::function<void (Item& item_ref)>& erase_func, Args&&... args) {\n        auto ret = evict_lru_item(erase_func);\n        if (!ret.first) {\n            throw std::bad_alloc{};\n        }\n        return create_item(ret.first, ret.second, std::forward<Args>(args)...);\n    }\n\n    void free_item(Item *item, slab_page_desc& desc) {\n        void *object = item;\n        _lru.erase(_lru.iterator_to(reinterpret_cast<slab_item_base&>(*item)));\n        desc.free_object(object);\n        if (desc.size() == 1) {\n            // push back desc into the list of slab pages with free objects.\n            _free_slab_pages.push_back(desc);\n        }\n    }\n\n    void touch_item(Item *item) {\n        auto& item_ref = reinterpret_cast<slab_item_base&>(*item);\n        _lru.erase(_lru.iterator_to(item_ref));\n        _lru.push_front(item_ref);\n    }\n\n    void remove_item_from_lru(Item *item) {\n        auto& item_ref = reinterpret_cast<slab_item_base&>(*item);\n        _lru.erase(_lru.iterator_to(item_ref));\n    }\n\n    void insert_item_into_lru(Item *item) {\n        auto& item_ref = reinterpret_cast<slab_item_base&>(*item);\n        _lru.push_front(item_ref);\n    }\n\n    void remove_desc_from_free_list(slab_page_desc& desc) {\n        SEASTAR_ASSERT(desc.slab_class_id() == _slab_class_id);\n        _free_slab_pages.erase(_free_slab_pages.iterator_to(desc));\n    }\n};\n\ntemplate<typename Item>\nclass slab_allocator {\nprivate:\n    std::vector<size_t> _slab_class_sizes;\n    std::vector<slab_class<Item>> _slab_classes;\n    seastar::metrics::metric_groups _metrics;\n    // erase_func() is used to remove the item from the cache using slab.\n    std::function<void (Item& item_ref)> _erase_func;\n    std::vector<slab_page_desc*> _slab_pages_vector;\n    boost::intrusive::list<slab_page_desc,\n        boost::intrusive::member_hook<slab_page_desc, boost::intrusive::list_member_hook<>,\n        &slab_page_desc::_lru_link>> _slab_page_desc_lru;\n    uint64_t _max_object_size;\n    uint64_t _available_slab_pages;\n    struct collectd_stats {\n        uint64_t allocs;\n        uint64_t frees;\n    } _stats;\n    memory::reclaimer *_reclaimer = nullptr;\n    bool _reclaimed = false;\nprivate:\n    memory::reclaiming_result evict_lru_slab_page() {\n        if (_slab_page_desc_lru.empty()) {\n            // NOTE: Nothing to evict. If this happens, it implies that all\n            // slab pages in the slab are being used at the same time.\n            // That being said, this event is very unlikely to happen.\n            return memory::reclaiming_result::reclaimed_nothing;\n        }\n        // get descriptor of the least-recently-used slab page and related info.\n        auto& desc = _slab_page_desc_lru.back();\n        SEASTAR_ASSERT(desc.refcnt() == 0);\n        uint8_t slab_class_id = desc.slab_class_id();\n        auto slab_class = get_slab_class(slab_class_id);\n        void *slab_page = desc.slab_page();\n\n        auto& free_objects = desc.free_objects();\n        if (!desc.empty()) {\n            // if not empty, remove desc from the list of slab pages with free objects.\n            slab_class->remove_desc_from_free_list(desc);\n            // and sort the array of free objects for binary search later on.\n            std::sort(free_objects.begin(), free_objects.end());\n        }\n        // remove desc from the list of slab page descriptors.\n        _slab_page_desc_lru.erase(_slab_page_desc_lru.iterator_to(desc));\n        // remove desc from the slab page vector.\n        _slab_pages_vector[desc.index()] = nullptr;\n\n        // Iterate through objects in the slab page and if the object is an allocated\n        // item, the item should be removed from LRU and then erased.\n        uintptr_t object = reinterpret_cast<uintptr_t>(slab_page);\n        auto object_size = slab_class->size();\n        auto objects = _max_object_size / object_size;\n        for (auto i = 0u; i < objects; i++, object += object_size) {\n            if (!desc.empty()) {\n                // if binary_search returns true, it means that object at the current\n                // offset isn't an item.\n                if (std::binary_search(free_objects.begin(), free_objects.end(), object)) {\n                    continue;\n                }\n            }\n            Item* item = reinterpret_cast<Item*>(object);\n            SEASTAR_ASSERT(item->is_unlocked());\n            slab_class->remove_item_from_lru(item);\n            _erase_func(*item);\n            _stats.frees++;\n        }\n#ifdef SEASTAR_DEBUG\n        printf(\"lru slab page eviction succeeded! desc_empty?=%d\\n\", desc.empty());\n#endif\n        ::free(slab_page); // free slab page object\n        delete &desc; // free its descriptor\n        return memory::reclaiming_result::reclaimed_something;\n    }\n\n    /*\n     * Reclaim the least recently used slab page that is unused.\n     */\n    memory::reclaiming_result reclaim() {\n        // once reclaimer was called, slab pages should no longer be allocated, as the\n        // memory used by slab is supposed to be calibrated.\n        _reclaimed = true;\n        // FIXME: Should reclaim() only evict a single slab page at a time?\n        return evict_lru_slab_page();\n    }\n\n    void initialize_slab_allocator(double growth_factor, uint64_t limit) {\n        constexpr size_t alignment = std::alignment_of_v<Item>;\n        constexpr size_t initial_size = 96;\n        size_t size = initial_size; // initial object size\n        uint8_t slab_class_id = 0U;\n\n        while (_max_object_size / size > 1) {\n            size = align_up(size, alignment);\n            _slab_class_sizes.push_back(size);\n            _slab_classes.emplace_back(size, slab_class_id);\n            size *= growth_factor;\n            SEASTAR_ASSERT(slab_class_id < std::numeric_limits<uint8_t>::max());\n            slab_class_id++;\n        }\n        _slab_class_sizes.push_back(_max_object_size);\n        _slab_classes.emplace_back(_max_object_size, slab_class_id);\n\n        // If slab limit is zero, enable reclaimer.\n        if (!limit) {\n            _reclaimer = new memory::reclaimer([this] { return reclaim(); });\n        } else {\n            _slab_pages_vector.reserve(_available_slab_pages);\n        }\n    }\n\n    slab_class<Item>* get_slab_class(const size_t size) {\n        // given a size, find slab class with binary search.\n        auto i = std::lower_bound(_slab_class_sizes.begin(), _slab_class_sizes.end(), size);\n        if (i == _slab_class_sizes.end()) {\n            return nullptr;\n        }\n        auto dist = std::distance(_slab_class_sizes.begin(), i);\n        return &_slab_classes[dist];\n    }\n\n    slab_class<Item>* get_slab_class(const uint8_t slab_class_id) {\n        SEASTAR_ASSERT(slab_class_id >= 0 && slab_class_id < _slab_classes.size());\n        return &_slab_classes[slab_class_id];\n    }\n\n    void register_metrics() {\n        namespace sm = seastar::metrics;\n        _metrics.add_group(\"slab\", {\n            sm::make_counter(\"malloc_total_operations\", sm::description(\"Total number of slab malloc operations\"), _stats.allocs),\n            sm::make_counter(\"free_total_operations\", sm::description(\"Total number of slab free operations\"), _stats.frees),\n            sm::make_gauge(\"malloc_objects\", sm::description(\"Number of slab created objects currently in memory\"), [this] {\n                return _stats.allocs - _stats.frees;\n            })\n        });\n    }\n\n    inline slab_page_desc& get_slab_page_desc(Item *item)\n    {\n        auto desc = _slab_pages_vector[item->get_slab_page_index()];\n        SEASTAR_ASSERT(desc != nullptr);\n        SEASTAR_ASSERT(desc->magic() == SLAB_MAGIC_NUMBER);\n        return *desc;\n    }\n\n    inline bool can_allocate_page(slab_class<Item>& sc) {\n        return (_reclaimer && !_reclaimed) || (_available_slab_pages > 0);\n    }\npublic:\n    slab_allocator(double growth_factor, uint64_t limit, uint64_t max_object_size)\n        : _max_object_size(max_object_size)\n        , _available_slab_pages(limit / max_object_size)\n    {\n        initialize_slab_allocator(growth_factor, limit);\n        register_metrics();\n    }\n\n    slab_allocator(double growth_factor, uint64_t limit, uint64_t max_object_size,\n                   std::function<void (Item& item_ref)> erase_func)\n        : _erase_func(std::move(erase_func))\n        , _max_object_size(max_object_size)\n        , _available_slab_pages(limit / max_object_size)\n    {\n        initialize_slab_allocator(growth_factor, limit);\n        register_metrics();\n    }\n\n    ~slab_allocator()\n    {\n        _slab_classes.clear();\n        _slab_page_desc_lru.clear();\n        for (auto desc : _slab_pages_vector) {\n            if (!desc) {\n                continue;\n            }\n            ::free(desc->slab_page());\n            delete desc;\n        }\n        delete _reclaimer;\n    }\n\n    /**\n     * Create an item from a given slab class based on requested size.\n     */\n    template<typename... Args>\n    Item* create(const size_t size, Args&&... args) {\n        auto slab_class = get_slab_class(size);\n        if (!slab_class) {\n            throw std::bad_alloc{};\n        }\n\n        Item *item = nullptr;\n        if (!slab_class->empty()) {\n            item = slab_class->create(std::forward<Args>(args)...);\n            _stats.allocs++;\n        } else {\n            if (can_allocate_page(*slab_class)) {\n                auto index_to_insert = _slab_pages_vector.size();\n                item = slab_class->create_from_new_page(_max_object_size, index_to_insert,\n                    [this](slab_page_desc& desc) {\n                        if (_reclaimer) {\n                            // insert desc into the LRU list of slab page descriptors.\n                            _slab_page_desc_lru.push_front(desc);\n                        }\n                        // insert desc into the slab page vector.\n                        _slab_pages_vector.push_back(&desc);\n                    },\n                    std::forward<Args>(args)...);\n                if (_available_slab_pages > 0) {\n                    _available_slab_pages--;\n                }\n                _stats.allocs++;\n            } else if (_erase_func) {\n                item = slab_class->create_from_lru(_erase_func, std::forward<Args>(args)...);\n            }\n        }\n        return item;\n    }\n\n    void lock_item(Item *item) {\n        auto& desc = get_slab_page_desc(item);\n        if (_reclaimer) {\n            auto& refcnt = desc.refcnt();\n\n            if (++refcnt == 1) {\n                // remove slab page descriptor from list of slab page descriptors.\n                _slab_page_desc_lru.erase(_slab_page_desc_lru.iterator_to(desc));\n            }\n        }\n        // remove item from the lru of its slab class.\n        auto slab_class = get_slab_class(desc.slab_class_id());\n        slab_class->remove_item_from_lru(item);\n    }\n\n    void unlock_item(Item *item) {\n        auto& desc = get_slab_page_desc(item);\n        if (_reclaimer) {\n            auto& refcnt = desc.refcnt();\n\n            if (--refcnt == 0) {\n                // insert slab page descriptor back into list of slab page descriptors.\n                _slab_page_desc_lru.push_front(desc);\n            }\n        }\n        // insert item into the lru of its slab class.\n        auto slab_class = get_slab_class(desc.slab_class_id());\n        slab_class->insert_item_into_lru(item);\n    }\n\n    /**\n     * Free an item back to its original slab class.\n     */\n    void free(Item *item) {\n        if (item) {\n            auto& desc = get_slab_page_desc(item);\n            auto slab_class = get_slab_class(desc.slab_class_id());\n            slab_class->free_item(item, desc);\n            _stats.frees++;\n        }\n    }\n\n    /**\n     * Update item position in the LRU of its slab class.\n     */\n    void touch(Item *item) {\n        if (item) {\n            auto& desc = get_slab_page_desc(item);\n            auto slab_class = get_slab_class(desc.slab_class_id());\n            slab_class->touch_item(item);\n        }\n    }\n\n    /**\n     * Helper function: Print all available slab classes and their respective properties.\n     */\n    void print_slab_classes() {\n        auto class_id = 0;\n        for (auto& slab_class : _slab_classes) {\n            size_t size = slab_class.size();\n            printf(\"slab[%3d]\\tsize: %10lu\\tper-slab-page: %5lu\\n\", class_id, size, _max_object_size / size);\n            class_id++;\n        }\n    }\n\n    /**\n     * Helper function: Useful for getting a slab class' chunk size from a size parameter.\n     */\n    size_t class_size(const size_t size) {\n        auto slab_class = get_slab_class(size);\n        return (slab_class) ? slab_class->size() : 0;\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/sleep.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <chrono>\n\n#include <seastar/core/abort_source.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/core/timer.hh>\n\nnamespace seastar {\n\n\n/// \\file\n\n/// Returns a future which completes after a specified time has elapsed.\n///\n/// \\param dur minimum amount of time before the returned future becomes\n///            ready.\n/// \\return A \\ref future which becomes ready when the sleep duration elapses.\ntemplate <typename Clock = steady_clock_type, typename Rep, typename Period>\nfuture<> sleep(std::chrono::duration<Rep, Period> dur) {\n    struct sleeper {\n        promise<> done;\n        timer<Clock> tmr;\n        sleeper(std::chrono::duration<Rep, Period> dur)\n            : tmr([this] { done.set_value(); })\n        {\n            tmr.arm(dur);\n        }\n    };\n    auto s = std::make_unique<sleeper>(dur);\n    future<> fut = s->done.get_future();\n    return fut.finally([s = std::move(s)] {});\n}\n\n/// exception that is thrown when application is in process of been stopped\nclass sleep_aborted : public abort_requested_exception {\npublic:\n    /// Reports the exception reason.\n    virtual const char* what() const noexcept {\n        return \"Sleep is aborted\";\n    }\n};\n\n/// Returns a future which completes after a specified time has elapsed\n/// or throws \\ref sleep_aborted exception if application is aborted\n///\n/// \\param dur minimum amount of time before the returned future becomes\n///            ready.\n/// \\return A \\ref future which becomes ready when the sleep duration elapses.\ntemplate <typename Clock = steady_clock_type>\nfuture<> sleep_abortable(typename Clock::duration dur);\n\nextern template future<> sleep_abortable<steady_clock_type>(typename steady_clock_type::duration);\nextern template future<> sleep_abortable<lowres_clock>(typename lowres_clock::duration);\n\n/// Returns a future which completes after a specified time has elapsed\n/// or throws \\ref sleep_aborted exception if the sleep is aborted.\n///\n/// \\param dur minimum amount of time before the returned future becomes\n///            ready.\n/// \\param as the \\ref abort_source that eventually notifies that the sleep\n///            should be aborted.\n/// \\return A \\ref future which becomes ready when the sleep duration elapses.\ntemplate <typename Clock = steady_clock_type>\nfuture<> sleep_abortable(typename Clock::duration dur, abort_source& as);\n\nextern template future<> sleep_abortable<steady_clock_type>(typename steady_clock_type::duration, abort_source&);\nextern template future<> sleep_abortable<lowres_clock>(typename lowres_clock::duration, abort_source&);\nextern template future<> sleep_abortable<manual_clock>(typename manual_clock::duration, abort_source&);\n\n}\n"
  },
  {
    "path": "include/seastar/core/smp.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/reactor_config.hh>\n#include <seastar/core/resource.hh>\n#include <seastar/core/shard_id.hh>\n\n#include <boost/lockfree/spsc_queue.hpp>\n#include <deque>\n#include <optional>\n#include <thread>\n#include <ranges>\n#include <span>\n#include <barrier>\n\n/// \\file\n\nnamespace seastar {\n\nclass reactor_backend_selector;\n\n\nclass smp_service_group;\n\nnamespace alien {\n\nclass instance;\n\n}\n\nnamespace internal {\n\nunsigned smp_service_group_id(smp_service_group ssg) noexcept;\n\nclass memory_prefaulter;\n\n}\n\nnamespace memory::internal {\n\nstruct numa_layout;\n\n}\n\n\n/// Configuration for smp_service_group objects.\n///\n/// \\see create_smp_service_group()\nstruct smp_service_group_config {\n    /// The maximum number of non-local requests that execute on a shard concurrently\n    ///\n    /// Will be adjusted upwards to allow at least one request per non-local shard.\n    unsigned max_nonlocal_requests = 0;\n    /// An optional name for this smp group\n    ///\n    /// If this optional is engaged, timeout exception messages of the group's\n    /// semaphores will indicate the group's name.\n    std::optional<sstring> group_name;\n};\n\n/// A resource controller for cross-shard calls.\n///\n/// An smp_service_group allows you to limit the concurrency of\n/// smp::submit_to() and similar calls. While it's easy to limit\n/// the caller's concurrency (for example, by using a semaphore),\n/// the concurrency at the remote end can be multiplied by a factor\n/// of smp::count-1, which can be large.\n///\n/// The class is called a service _group_ because it can be used\n/// to group similar calls that share resource usage characteristics,\n/// need not be isolated from each other, but do need to be isolated\n/// from other groups. Calls in a group should not nest; doing so\n/// can result in ABA deadlocks.\n///\n/// Nested submit_to() calls must form a directed acyclic graph\n/// when considering their smp_service_groups as nodes. For example,\n/// if a call using ssg1 then invokes another call using ssg2, the\n/// internal call may not call again via either ssg1 or ssg2, or it\n/// may form a cycle (and risking an ABBA deadlock). Create a\n/// new smp_service_group_instead.\nclass smp_service_group {\n    unsigned _id;\n#ifdef SEASTAR_DEBUG\n    unsigned _version = 0;\n#endif\nprivate:\n    explicit smp_service_group(unsigned id) noexcept : _id(id) {}\n\n    friend unsigned internal::smp_service_group_id(smp_service_group ssg) noexcept;\n    friend smp_service_group default_smp_service_group() noexcept;\n    friend future<smp_service_group> create_smp_service_group(smp_service_group_config ssgc) noexcept;\n    friend future<> destroy_smp_service_group(smp_service_group) noexcept;\n};\n\n\ninline\nunsigned\ninternal::smp_service_group_id(smp_service_group ssg) noexcept {\n    return ssg._id;\n}\n\n/// Returns the default smp_service_group. This smp_service_group\n/// does not impose any limits on concurrency in the target shard.\n/// This makes is deadlock-safe, but can consume unbounded resources,\n/// and should therefore only be used when initiator concurrency is\n/// very low (e.g. administrative tasks).\nsmp_service_group default_smp_service_group() noexcept;\n\n/// Creates an smp_service_group with the specified configuration.\n///\n/// The smp_service_group is global, and after this call completes,\n/// the returned value can be used on any shard.\nfuture<smp_service_group> create_smp_service_group(smp_service_group_config ssgc) noexcept;\n\n/// Destroy an smp_service_group.\n///\n/// Frees all resources used by an smp_service_group. It must not\n/// be used again once this function is called.\nfuture<> destroy_smp_service_group(smp_service_group ssg) noexcept;\n\ninline\nsmp_service_group default_smp_service_group() noexcept {\n    return smp_service_group(0);\n}\n\nusing smp_timeout_clock = lowres_clock;\nusing smp_service_group_semaphore = basic_semaphore<named_semaphore_exception_factory, smp_timeout_clock>;\nusing smp_service_group_semaphore_units = semaphore_units<named_semaphore_exception_factory, smp_timeout_clock>;\n\n\nstatic constexpr smp_timeout_clock::time_point smp_no_timeout = smp_timeout_clock::time_point::max();\n\n/// Options controlling the behaviour of \\ref smp::submit_to().\nstruct smp_submit_to_options {\n    /// Controls resource allocation.\n    smp_service_group service_group = default_smp_service_group();\n    /// The timeout is relevant only to the time the call spends waiting to be\n    /// processed by the remote shard, and *not* to the time it takes to be\n    /// executed there.\n    smp_timeout_clock::time_point timeout = smp_no_timeout;\n\n    smp_submit_to_options(smp_service_group service_group = default_smp_service_group(), smp_timeout_clock::time_point timeout = smp_no_timeout) noexcept\n        : service_group(service_group)\n        , timeout(timeout) {\n    }\n};\n\nvoid init_default_smp_service_group(shard_id cpu);\n\nsmp_service_group_semaphore& get_smp_service_groups_semaphore(unsigned ssg_id, shard_id t) noexcept;\n\nclass smp_message_queue {\n    static constexpr size_t queue_length = 128;\n    static constexpr size_t batch_size = 16;\n    static constexpr size_t prefetch_cnt = 2;\n    struct work_item;\n    struct lf_queue_remote {\n        reactor* remote;\n    };\n    using lf_queue_base = boost::lockfree::spsc_queue<work_item*,\n                            boost::lockfree::capacity<queue_length>>;\n    // use inheritence to control placement order\n    struct lf_queue : lf_queue_remote, lf_queue_base {\n        lf_queue(reactor* remote) : lf_queue_remote{remote} {}\n        void maybe_wakeup();\n        ~lf_queue();\n    };\n    lf_queue _pending;\n    lf_queue _completed;\n    struct alignas(seastar::cache_line_size) {\n        size_t _sent = 0;\n        size_t _compl = 0;\n        size_t _last_snt_batch = 0;\n        size_t _last_cmpl_batch = 0;\n        size_t _current_queue_length = 0;\n    };\n    // keep this between two structures with statistics\n    // this makes sure that they have at least one cache line\n    // between them, so hw prefetcher will not accidentally prefetch\n    // cache line used by another cpu.\n    metrics::metric_groups _metrics;\n    struct alignas(seastar::cache_line_size) {\n        size_t _received = 0;\n        size_t _last_rcv_batch = 0;\n    };\n    struct work_item : public task {\n        explicit work_item(smp_service_group ssg) : task(current_scheduling_group()), ssg(ssg) {}\n        smp_service_group ssg;\n        virtual ~work_item() {}\n        virtual void fail_with(std::exception_ptr) = 0;\n        void process();\n        virtual void complete() = 0;\n    };\n    template <typename Func>\n    struct async_work_item : work_item {\n        smp_message_queue& _queue;\n        Func _func;\n        using futurator = futurize<std::invoke_result_t<Func>>;\n        using future_type = typename futurator::type;\n        using value_type = typename future_type::value_type;\n        std::optional<value_type> _result;\n        std::exception_ptr _ex; // if !_result\n        typename futurator::promise_type _promise; // used on local side\n        async_work_item(smp_message_queue& queue, smp_service_group ssg, Func&& func) : work_item(ssg), _queue(queue), _func(std::move(func)) {}\n        virtual void fail_with(std::exception_ptr ex) override {\n            _promise.set_exception(std::move(ex));\n        }\n        virtual task* waiting_task() noexcept override {\n            // FIXME: waiting_tasking across shards is not implemented. Unsynchronized task access is unsafe.\n            return nullptr;\n        }\n        virtual void run_and_dispose() noexcept override {\n            // _queue.respond() below forwards the continuation chain back to the\n            // calling shard.\n            (void)futurator::invoke(this->_func).then_wrapped([this] (auto f) {\n                if (f.failed()) {\n                    _ex = f.get_exception();\n                } else {\n                    _result = f.get();\n                }\n                _queue.respond(this);\n            });\n            // We don't delete the task here as the creator of the work item will\n            // delete it on the origin shard.\n        }\n        virtual void complete() override {\n            if (_result) {\n                _promise.set_value(std::move(*_result));\n            } else {\n                // FIXME: _ex was allocated on another cpu\n                _promise.set_exception(std::move(_ex));\n            }\n        }\n        future_type get_future() { return _promise.get_future(); }\n    };\n    union tx_side {\n        tx_side() {}\n        ~tx_side() {}\n        void init() { new (&a) aa; }\n        struct aa {\n            std::deque<work_item*> pending_fifo;\n        } a;\n    } _tx;\n    std::vector<work_item*> _completed_fifo;\npublic:\n    smp_message_queue(reactor* from, reactor* to);\n    ~smp_message_queue();\n    template <typename Func>\n    futurize_t<std::invoke_result_t<Func>> submit(shard_id t, smp_submit_to_options options, Func&& func) noexcept {\n        memory::scoped_critical_alloc_section _;\n        auto wi = std::make_unique<async_work_item<Func>>(*this, options.service_group, std::forward<Func>(func));\n        auto fut = wi->get_future();\n        submit_item(t, options.timeout, std::move(wi));\n        return fut;\n    }\n    void start(unsigned cpuid);\n    template<size_t PrefetchCnt, typename Func>\n    size_t process_queue(lf_queue& q, Func process);\n    size_t process_incoming();\n    size_t process_completions(shard_id t);\n    void stop();\nprivate:\n    void work();\n    void submit_item(shard_id t, smp_timeout_clock::time_point timeout, std::unique_ptr<work_item> wi);\n    void respond(work_item* wi);\n    void move_pending();\n    void flush_request_batch();\n    void flush_response_batch();\n    bool has_unflushed_responses() const;\n    bool pure_poll_rx() const;\n    bool pure_poll_tx() const;\n\n    friend class smp;\n};\n\nclass smp_message_queue;\nstruct reactor_options;\nstruct smp_options;\n\nclass smp : public std::enable_shared_from_this<smp> {\n    alien::instance& _alien;\n    std::vector<posix_thread> _threads;\n    std::vector<std::function<void ()>> _thread_loops; // for dpdk\n    std::optional<std::barrier<>> _all_event_loops_done;\n    std::unique_ptr<internal::memory_prefaulter> _prefaulter;\n    struct qs_deleter {\n      void operator()(smp_message_queue** qs) const;\n    };\n    std::unique_ptr<smp_message_queue*[], qs_deleter> _qs_owner;\n    static thread_local smp_message_queue**_qs;\n    static thread_local std::thread::id _tmain;\n    bool _using_dpdk = false;\n    std::vector<unsigned> _shard_to_numa_node_mapping;\n\nprivate:\n    void setup_prefaulter(const seastar::resource::resources& res, seastar::memory::internal::numa_layout layout);\npublic:\n    explicit smp(alien::instance& alien);\n    ~smp();\n    void configure(const smp_options& smp_opts, const reactor_options& reactor_opts);\n    void cleanup() noexcept;\n    void cleanup_cpu();\n    void arrive_at_event_loop_end();\n    void join_all();\n    static bool main_thread() { return std::this_thread::get_id() == _tmain; }\n\n    /// \\returns A integer span of size smp::count, with nth integer being the ID of nth shard's NUMA node.\n    std::span<const unsigned> shard_to_numa_node_mapping() const noexcept;\n\n    /// Runs a function on a remote core.\n    ///\n    /// \\param t designates the core to run the function on (may be a remote\n    ///          core or the local core).\n    /// \\param options an \\ref smp_submit_to_options that contains options for this call.\n    /// \\param func a callable to run on core \\c t.\n    ///          If \\c func is a temporary object, its lifetime will be\n    ///          extended by moving. This movement and the eventual\n    ///          destruction of func are both done in the _calling_ core.\n    ///          If \\c func is a reference, the caller must guarantee that\n    ///          it will survive the call.\n    /// \\return whatever \\c func returns, as a future<> (if \\c func does not return a future,\n    ///         submit_to() will wrap it in a future<>).\n    template <typename Func>\n    static futurize_t<std::invoke_result_t<Func>> submit_to(unsigned t, smp_submit_to_options options, Func&& func) noexcept {\n        using ret_type = std::invoke_result_t<Func>;\n        if (t == this_shard_id()) {\n            try {\n                if (!is_future<ret_type>::value) {\n                    // Non-deferring function, so don't worry about func lifetime\n                    return futurize<ret_type>::invoke(std::forward<Func>(func));\n                } else if (std::is_lvalue_reference_v<Func>) {\n                    // func is an lvalue, so caller worries about its lifetime\n                    return futurize<ret_type>::invoke(func);\n                } else {\n                    // Deferring call on rvalue function, make sure to preserve it across call\n                    auto w = std::make_unique<std::decay_t<Func>>(std::move(func));\n                    auto ret = futurize<ret_type>::invoke(*w);\n                    return ret.finally([w = std::move(w)] {});\n                }\n            } catch (...) {\n                // Consistently return a failed future rather than throwing, to simplify callers\n                return futurize<std::invoke_result_t<Func>>::make_exception_future(std::current_exception());\n            }\n        } else {\n            return _qs[t][this_shard_id()].submit(t, options, std::forward<Func>(func));\n        }\n    }\n    /// Runs a function on a remote core.\n    ///\n    /// Uses default_smp_service_group() to control resource allocation.\n    ///\n    /// \\param t designates the core to run the function on (may be a remote\n    ///          core or the local core).\n    /// \\param func a callable to run on core \\c t.\n    ///          If \\c func is a temporary object, its lifetime will be\n    ///          extended by moving. This movement and the eventual\n    ///          destruction of func are both done in the _calling_ core.\n    ///          If \\c func is a reference, the caller must guarantee that\n    ///          it will survive the call.\n    /// \\return whatever \\c func returns, as a future<> (if \\c func does not return a future,\n    ///         submit_to() will wrap it in a future<>).\n    template <typename Func>\n    static futurize_t<std::invoke_result_t<Func>> submit_to(unsigned t, Func&& func) noexcept {\n        return submit_to(t, default_smp_service_group(), std::forward<Func>(func));\n    }\n    static bool poll_queues();\n    static bool pure_poll_queues();\n    static std::ranges::range auto all_cpus() noexcept {\n        return std::views::iota(0u, count);\n    }\nprivate:\n    template <typename Func>\n    requires std::is_nothrow_copy_constructible_v<Func>\n    static futurize_t<std::invoke_result_t<Func>> copy_and_submit_to(unsigned t, smp_submit_to_options options, const Func& func) noexcept {\n        return submit_to(t, options, Func(func));\n    }\n\n    template <typename Func>\n    requires (!std::is_nothrow_copy_constructible_v<Func>)\n    static futurize_t<std::invoke_result_t<Func>> copy_and_submit_to(unsigned t, smp_submit_to_options options, const Func& func) noexcept {\n        try {\n            return submit_to(t, options, Func(func));\n        } catch (...) {\n            return current_exception_as_future();\n        }\n    }\npublic:\n    /// Invokes func on all shards.\n    ///\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func the function to be invoked on each shard. May return void or\n    ///         future<>. Each async invocation will work with a separate copy\n    ///         of \\c func.\n    /// \\returns a future that resolves when all async invocations finish.\n    template<typename Func>\n     requires std::is_nothrow_move_constructible_v<Func>\n    static future<> invoke_on_all(smp_submit_to_options options, Func&& func) noexcept {\n        static_assert(std::is_same_v<future<>, typename futurize<std::invoke_result_t<Func>>::type>, \"bad Func signature\");\n        static_assert(std::is_nothrow_move_constructible_v<Func>);\n        return parallel_for_each(all_cpus(), [options, &func] (unsigned id) {\n            return smp::copy_and_submit_to(id, options, func);\n        });\n    }\n    /// Invokes func on all shards.\n    ///\n    /// \\param func the function to be invoked on each shard. May return void or\n    ///         future<>. Each async invocation will work with a separate copy\n    ///         of \\c func.\n    /// \\returns a future that resolves when all async invocations finish.\n    ///\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    template<typename Func>\n    static future<> invoke_on_all(Func&& func) noexcept {\n        return invoke_on_all(smp_submit_to_options{}, std::forward<Func>(func));\n    }\n    /// Invokes func on all other shards.\n    ///\n    /// \\param cpu_id the cpu on which **not** to run the function.\n    /// \\param options the options to forward to the \\ref smp::submit_to()\n    ///         called behind the scenes.\n    /// \\param func the function to be invoked on each shard. May return void or\n    ///         future<>. Each async invocation will work with a separate copy\n    ///         of \\c func.\n    /// \\returns a future that resolves when all async invocations finish.\n    template<typename Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    static future<> invoke_on_others(unsigned cpu_id, smp_submit_to_options options, Func func) noexcept {\n        static_assert(std::is_same_v<future<>, typename futurize<std::invoke_result_t<Func>>::type>, \"bad Func signature\");\n        static_assert(std::is_nothrow_move_constructible_v<Func>);\n        return parallel_for_each(all_cpus(), [cpu_id, options, func = std::move(func)] (unsigned id) {\n            return id != cpu_id ? smp::copy_and_submit_to(id, options, func) : make_ready_future<>();\n        });\n    }\n    /// Invokes func on all other shards.\n    ///\n    /// \\param cpu_id the cpu on which **not** to run the function.\n    /// \\param func the function to be invoked on each shard. May return void or\n    ///         future<>. Each async invocation will work with a separate copy\n    ///         of \\c func.\n    /// \\returns a future that resolves when all async invocations finish.\n    ///\n    /// Passes the default \\ref smp_submit_to_options to the\n    /// \\ref smp::submit_to() called behind the scenes.\n    template<typename Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    static future<> invoke_on_others(unsigned cpu_id, Func func) noexcept {\n        return invoke_on_others(cpu_id, smp_submit_to_options{}, std::move(func));\n    }\n    /// Invokes func on all shards but the current one\n    ///\n    /// \\param func the function to be invoked on each shard. May return void or\n    ///         future<>. Each async invocation will work with a separate copy\n    ///         of \\c func.\n    /// \\returns a future that resolves when all async invocations finish.\n    template<typename Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    static future<> invoke_on_others(Func func) noexcept {\n        return invoke_on_others(this_shard_id(), std::move(func));\n    }\nprivate:\n    void start_all_queues();\n    void pin(unsigned cpu_id);\n    void allocate_reactor(unsigned id, reactor_backend_selector rbs, reactor_config cfg);\n    void create_thread(std::function<void ()> thread_loop);\n    unsigned adjust_max_networking_aio_io_control_blocks(unsigned network_iocbs, unsigned reserve_iocbs);\n    static void log_aiocbs(log_level level, unsigned storage, unsigned preempt, unsigned network, unsigned reserve);\npublic:\n    static unsigned count;\n};\n\n\n}\n"
  },
  {
    "path": "include/seastar/core/smp_options.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2022 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/resource.hh>\n#include <seastar/util/program-options.hh>\n#include <string>\n\n/// \\file\n\nnamespace seastar {\n\n\nenum class memory_allocator {\n    /// Seastar's own allocator, optimized for its shard-per core design.\n    /// Strongly recommended for most seastar apps.\n    seastar,\n    /// The standard allocator from libc.\n    /// Useful when writing seastar-based tool apps that want to\n    /// minimize their footprint on the system they run on.\n    standard,\n};\n\n/// Configuration for the multicore aspect of seastar.\nstruct smp_options : public program_options::option_group {\n    /// Number of threads (default: one per CPU).\n    program_options::value<unsigned> smp;\n    /// CPUs to use (in cpuset(7) format; default: all)).\n    program_options::value<resource::cpuset> cpuset;\n    /// Memory to use, in bytes (ex: 4G) (default: all).\n    program_options::value<std::string> memory;\n    /// Memory reserved to the OS and other processes (if \\ref memory not specified),\n    /// and is not used by seastar.\n    program_options::value<std::string> reserve_memory;\n    /// Path to accessible hugetlbfs mount (typically /dev/hugepages/something).\n    program_options::value<std::string> hugepages;\n    /// Lock all memory (prevents swapping).\n    program_options::value<bool> lock_memory;\n    /// Pin threads to their cpus (disable for overprovisioning).\n    ///\n    /// Default: \\p true.\n    program_options::value<bool> thread_affinity;\n    /// \\brief Number of IO groups.\n    ///\n    /// Each IO group will be responsible for a fraction of the IO requests.\n    /// Defaults to the number of NUMA nodes\n    /// \\note Unused when seastar is compiled without \\p HWLOC support.\n    program_options::value<unsigned> num_io_groups;\n    /// Path to a YAML file describing the characteristics of the I/O Subsystem.\n    program_options::value<std::string> io_properties_file;\n    /// A YAML string describing the characteristics of the I/O Subsystem.\n    program_options::value<std::string> io_properties;\n    /// Enable mbind.\n    ///\n    /// Default: \\p true.\n    program_options::value<bool> mbind;\n    /// Enable workaround for glibc/gcc c++ exception scalablity problem.\n    ///\n    /// Default: \\p true.\n    /// \\note Unused when seastar is compiled without the exception scaling support.\n    program_options::value<bool> enable_glibc_exception_scaling_workaround;\n    /// If some CPUs are found not to have any local NUMA nodes, allow assigning\n    /// them to remote ones.\n    /// \\note Unused when seastar is compiled without \\p HWLOC support.\n    program_options::value<bool> allow_cpus_in_remote_numa_nodes;\n\n    /// Memory allocator to use.\n    ///\n    /// The following options only have effect if the \\ref memory_allocator::seastar is used:\n    /// * \\ref smp_options::memory\n    /// * \\ref smp_options::reserve_memory\n    /// * \\ref smp_options::hugepages\n    /// * \\ref smp_options::mbind\n    /// * \\ref reactor_options::heapprof\n    /// * \\ref reactor_options::abort_on_seastar_bad_alloc\n    /// * \\ref reactor_options::dump_memory_diagnostics_on_alloc_failure_kind\n    /// \\note Unused when seastar was compiled without the custom allocator support.\n    /// The options above won't be applied in this case either.\n    seastar::memory_allocator memory_allocator = memory_allocator::seastar;\n\n    /// \\cond internal\n    /// Additional memory reserved to the OS and other processes for each shard (added to the default value or the value specified by \\ref reserve_memory),\n    /// and is not used by seastar.\n    size_t reserve_additional_memory_per_shard = 0;\n    /// \\endcond\npublic:\n    smp_options(program_options::option_group* parent_group);\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/sstring.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <stdint.h>\n#include <algorithm>\n#if __has_include(<compare>)\n#include <compare>\n#endif\n#include <string>\n#include <vector>\n#include <unordered_map>\n#include <concepts>\n#include <cstring>\n#include <initializer_list>\n#include <istream>\n#include <ostream>\n#include <functional>\n#include <type_traits>\n#include <fmt/format.h>\n#if FMT_VERSION >= 110000\n#include <fmt/ranges.h>\n#endif\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/core/temporary_buffer.hh>\n\nnamespace seastar {\n\n\ntemplate <typename char_type, typename Size, Size max_size, bool NulTerminate = true>\nclass basic_sstring;\n\n#ifdef SEASTAR_SSTRING\n// Older std::string used atomic reference counting and had no small-buffer-optimization.\n// At some point the new std::string ABI improved -- no reference counting plus the small\n// buffer optimization. However, aliasing seastar::sstring to std::string still ends up\n// with a small performance degradation. (FIXME?)\nusing sstring = basic_sstring<char, uint32_t, 15>;\n#else\nusing sstring = std::string;\n#endif\n\n\nnamespace internal {\n[[noreturn]] void throw_bad_alloc();\n[[noreturn]] void throw_sstring_overflow();\n[[noreturn]] void throw_sstring_out_of_range();\n}\n\ntemplate <typename char_type, typename Size, Size max_size, bool NulTerminate>\nclass basic_sstring {\n    static_assert(\n            (std::is_same_v<char_type, char>\n             || std::is_same_v<char_type, signed char>\n             || std::is_same_v<char_type, unsigned char>),\n            \"basic_sstring only supports single byte char types\");\n    union contents {\n        struct external_type {\n            char_type* str;\n            Size size;\n            int8_t pad;\n        } external;\n        struct internal_type {\n            char_type str[max_size];\n            int8_t size;\n        } internal;\n        static_assert(sizeof(external_type) <= sizeof(internal_type), \"max_size too small\");\n        static_assert(max_size <= 127, \"max_size too large\");\n    } u;\n    bool is_internal() const noexcept {\n        return u.internal.size >= 0;\n    }\n    bool is_external() const noexcept {\n        return !is_internal();\n    }\n    const char_type* str() const noexcept {\n        return is_internal() ? u.internal.str : u.external.str;\n    }\n    char_type* str() noexcept {\n        return is_internal() ? u.internal.str : u.external.str;\n    }\n\npublic:\n    using value_type = char_type;\n    using traits_type = std::char_traits<char_type>;\n    using allocator_type = std::allocator<char_type>;\n    using reference = char_type&;\n    using const_reference = const char_type&;\n    using pointer = char_type*;\n    using const_pointer = const char_type*;\n    using iterator = char_type*;\n    using const_iterator = const char_type*;\n    // FIXME: add reverse_iterator and friend\n    using difference_type = ssize_t;  // std::make_signed_t<Size> can be too small\n    using size_type = Size;\n    static constexpr size_t npos = static_cast<size_t>(-1);\n    static constexpr unsigned padding() { return unsigned(NulTerminate); }\npublic:\n    struct initialized_later {};\n\n    basic_sstring() noexcept {\n        u.internal.size = 0;\n        if (NulTerminate) {\n            u.internal.str[0] = '\\0';\n        }\n    }\n    basic_sstring(const basic_sstring& x) {\n        if (x.is_internal()) {\n            u.internal = x.u.internal;\n        } else {\n            u.internal.size = -1;\n            u.external.str = reinterpret_cast<char_type*>(std::malloc(x.u.external.size + padding()));\n            if (!u.external.str) {\n                internal::throw_bad_alloc();\n            }\n            std::copy(x.u.external.str, x.u.external.str + x.u.external.size + padding(), u.external.str);\n            u.external.size = x.u.external.size;\n        }\n    }\n    basic_sstring(basic_sstring&& x) noexcept {\n#pragma GCC diagnostic push\n        // Is a small-string construction is followed by this move constructor, then the trailing bytes\n        // of x.u are not initialized, but copied. gcc complains, but it is both legitimate to copy\n        // these bytes, and more efficient than a variable-size copy\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n        u = x.u;\n#pragma GCC diagnostic pop\n        x.u.internal.size = 0;\n        x.u.internal.str[0] = '\\0';\n    }\n    basic_sstring(initialized_later, size_t size) {\n        if (size_type(size) != size) {\n            internal::throw_sstring_overflow();\n        }\n        if (size + padding() <= sizeof(u.internal.str)) {\n            if (NulTerminate) {\n                u.internal.str[size] = '\\0';\n            }\n            u.internal.size = size;\n        } else {\n            u.internal.size = -1;\n            u.external.str = reinterpret_cast<char_type*>(std::malloc(size + padding()));\n            if (!u.external.str) {\n                internal::throw_bad_alloc();\n            }\n            u.external.size = size;\n            if (NulTerminate) {\n                u.external.str[size] = '\\0';\n            }\n        }\n    }\n    basic_sstring(const char_type* x, size_t size) {\n        if (size_type(size) != size) {\n            internal::throw_sstring_overflow();\n        }\n        if (size + padding() <= sizeof(u.internal.str)) {\n            std::copy(x, x + size, u.internal.str);\n            if (NulTerminate) {\n                u.internal.str[size] = '\\0';\n            }\n            u.internal.size = size;\n        } else {\n            u.internal.size = -1;\n            u.external.str = reinterpret_cast<char_type*>(std::malloc(size + padding()));\n            if (!u.external.str) {\n                internal::throw_bad_alloc();\n            }\n            u.external.size = size;\n            std::copy(x, x + size, u.external.str);\n            if (NulTerminate) {\n                u.external.str[size] = '\\0';\n            }\n        }\n    }\n\n    basic_sstring(size_t size, char_type x) : basic_sstring(initialized_later(), size) {\n        memset(begin(), x, size);\n    }\n\n    basic_sstring(const char* x) : basic_sstring(reinterpret_cast<const char_type*>(x), std::strlen(x)) {}\n    basic_sstring(std::basic_string<char_type>& x) : basic_sstring(x.c_str(), x.size()) {}\n    basic_sstring(std::initializer_list<char_type> x) : basic_sstring(x.begin(), x.end() - x.begin()) {}\n    basic_sstring(const char_type* b, const char_type* e) : basic_sstring(b, e - b) {}\n    basic_sstring(const std::basic_string<char_type>& s)\n        : basic_sstring(s.data(), s.size()) {}\n    template <typename InputIterator>\n    basic_sstring(InputIterator first, InputIterator last)\n            : basic_sstring(initialized_later(), std::distance(first, last)) {\n        std::copy(first, last, begin());\n    }\n    explicit basic_sstring(std::basic_string_view<char_type, traits_type> v)\n            : basic_sstring(v.data(), v.size()) {\n    }\n    ~basic_sstring() noexcept {\n        if (is_external()) {\n            std::free(u.external.str);\n        }\n    }\n    basic_sstring& operator=(const basic_sstring& x) {\n        basic_sstring tmp(x);\n        swap(tmp);\n        return *this;\n    }\n    basic_sstring& operator=(basic_sstring&& x) noexcept {\n        if (this != &x) {\n            this->~basic_sstring();\n            new (this) basic_sstring(std::move(x));\n        }\n        return *this;\n    }\n    operator std::basic_string<char_type>() const {\n        return { str(), size() };\n    }\n\n    size_t size() const noexcept {\n        return is_internal() ? u.internal.size : u.external.size;\n    }\n\n    size_t length() const noexcept {\n        return size();\n    }\n\n    size_t find(char_type t, size_t pos = 0) const noexcept {\n        const char_type* it = str() + pos;\n        const char_type* end = str() + size();\n        while (it < end) {\n            if (*it == t) {\n                return it - str();\n            }\n            it++;\n        }\n        return npos;\n    }\n\n    size_t find(const char_type* c_str, size_t pos, size_t len2) const noexcept {\n        SEASTAR_ASSERT(c_str != nullptr || len2 == 0);\n        if (pos > size()) {\n            return npos;\n        }\n\n        if (len2 == 0) {\n            return pos;\n        }\n\n        const char_type* it = str() + pos;\n        const char_type* end = str() + size();\n        size_t len1 = end - it;\n        if (len1 < len2) {\n            return npos;\n        }\n\n        char_type f2 = *c_str;\n        while (true) {\n            len1 = end - it;\n            if (len1 < len2) {\n                return npos;\n            }\n\n            // find the first byte of pattern string matching in source string\n            it = traits_type::find(it, len1 - len2 + 1, f2);\n            if (it == nullptr) {\n                return npos;\n            }\n\n            if (traits_type::compare(it, c_str, len2) == 0) {\n                return it - str();\n            }\n\n            ++it;\n        }\n    }\n\n    constexpr size_t find(const char_type* s, size_t pos = 0) const noexcept {\n        return find(s, pos, traits_type::length(s));\n    }\n\n    size_t find(const basic_sstring& s, size_t pos = 0) const noexcept {\n        return find(s.str(), pos, s.size());\n    }\n\n    size_t find(const std::convertible_to<std::basic_string_view<char_type, traits_type>> auto& sv_like, size_type pos = 0) const noexcept {\n        std::basic_string_view<char_type, traits_type> sv = sv_like;\n        return find(sv.data(), pos, sv.size());\n    }\n\n    /**\n     * find_last_of find the last occurrence of c in the string.\n     * When pos is specified, the search only includes characters\n     * at or before position pos.\n     *\n     */\n    size_t find_last_of (char_type c, size_t pos = npos) const noexcept {\n        const char_type* str_start = str();\n        if (size()) {\n            if (pos >= size()) {\n                pos = size() - 1;\n            }\n            const char_type* p = str_start + pos + 1;\n            do {\n                p--;\n                if (*p == c) {\n                    return (p - str_start);\n                }\n            } while (p != str_start);\n        }\n        return npos;\n    }\n\n    /**\n     *  Append a C substring.\n     *  @param s  The C string to append.\n     *  @param n  The number of characters to append.\n     *  @return  Reference to this string.\n     */\n    basic_sstring& append (const char_type* s, size_t n) {\n        basic_sstring ret(initialized_later(), size() + n);\n        std::copy(begin(), end(), ret.begin());\n        std::copy(s, s + n, ret.begin() + size());\n        *this = std::move(ret);\n        return *this;\n    }\n\n    /**\n     *  Resize string and use the specified @c op to modify the content and the length\n     *  @param n  new size\n     *  @param op the function object used for setting the new content of the string\n     */\n    template <class Operation>\n    requires std::is_invocable_r_v<size_t, Operation, char_type*, size_t>\n    void resize_and_overwrite(size_t n, Operation op) {\n        if (n > size()) {\n            *this = basic_sstring(initialized_later(), n);\n        }\n        size_t r = std::move(op)(data(), n);\n        SEASTAR_ASSERT(r <= n);\n        resize(r);\n    }\n\n    /**\n     *  Resize string.\n     *  @param n  new size.\n     *  @param c  if n greater than current size character to fill newly allocated space with.\n     */\n    void resize(size_t n, const char_type c  = '\\0') {\n        if (n > size()) {\n            *this += basic_sstring(n - size(), c);\n        } else if (n < size()) {\n            if (is_internal()) {\n                u.internal.size = n;\n                if (NulTerminate) {\n                    u.internal.str[n] = '\\0';\n                }\n            } else if (n + padding() <= sizeof(u.internal.str)) {\n                *this = basic_sstring(u.external.str, n);\n            } else {\n                u.external.size = n;\n                if (NulTerminate) {\n                    u.external.str[n] = '\\0';\n                }\n            }\n        }\n    }\n\n    /**\n     *  Replace characters with a value of a C style substring.\n     *\n     */\n    basic_sstring& replace(size_type pos, size_type n1, const char_type* s,\n             size_type n2) {\n        if (pos > size()) {\n            internal::throw_sstring_out_of_range();\n        }\n\n        if (n1 > size() - pos) {\n            n1 = size() - pos;\n        }\n\n        if (n1 == n2) {\n            if (n2) {\n                std::copy(s, s + n2, begin() + pos);\n            }\n            return *this;\n        }\n        basic_sstring ret(initialized_later(), size() + n2 - n1);\n        char_type* p= ret.begin();\n        std::copy(begin(), begin() + pos, p);\n        p += pos;\n        if (n2) {\n            std::copy(s, s + n2, p);\n        }\n        p += n2;\n        std::copy(begin() + pos + n1, end(), p);\n        *this = std::move(ret);\n        return *this;\n    }\n\n    template <class InputIterator>\n    basic_sstring& replace (const_iterator i1, const_iterator i2,\n            InputIterator first, InputIterator last) {\n        if (i1 < begin() || i1 > end() || i2 < begin()) {\n            internal::throw_sstring_out_of_range();\n        }\n        if (i2 > end()) {\n            i2 = end();\n        }\n\n        if (i2 - i1 == last - first) {\n            //in place replacement\n            std::copy(first, last, const_cast<char_type*>(i1));\n            return *this;\n        }\n        basic_sstring ret(initialized_later(), size() + (last - first) - (i2 - i1));\n        char_type* p = ret.begin();\n        p = std::copy(cbegin(), i1, p);\n        p = std::copy(first, last, p);\n        std::copy(i2, cend(), p);\n        *this = std::move(ret);\n        return *this;\n    }\n\n    iterator erase(iterator first, iterator last) {\n        size_t pos = first - begin();\n        replace(pos, last - first, nullptr, 0);\n        return begin() + pos;\n    }\n\n    /**\n     * Inserts additional characters into the string right before\n     * the character indicated by p.\n     */\n    template <class InputIterator>\n    void insert(const_iterator p, InputIterator beg, InputIterator end) {\n        replace(p, p, beg, end);\n    }\n\n\n    /**\n     *  Returns a read/write reference to the data at the first\n     *  element of the string.\n     *  This function shall not be called on empty strings.\n     */\n    reference\n    front() noexcept {\n        SEASTAR_ASSERT(!empty());\n        return *str();\n    }\n\n    /**\n     *  Returns a  read-only (constant) reference to the data at the first\n     *  element of the string.\n     *  This function shall not be called on empty strings.\n     */\n    const_reference\n    front() const noexcept {\n        SEASTAR_ASSERT(!empty());\n        return *str();\n    }\n\n    /**\n     *  Returns a read/write reference to the data at the last\n     *  element of the string.\n     *  This function shall not be called on empty strings.\n     */\n    reference\n    back() noexcept {\n        return operator[](size() - 1);\n    }\n\n    /**\n     *  Returns a  read-only (constant) reference to the data at the last\n     *  element of the string.\n     *  This function shall not be called on empty strings.\n     */\n    const_reference\n    back() const noexcept {\n        return operator[](size() - 1);\n    }\n\n    basic_sstring substr(size_t from, size_t len = npos)  const {\n        if (from > size()) {\n            internal::throw_sstring_out_of_range();\n        }\n        if (len > size() - from) {\n            len = size() - from;\n        }\n        if (len == 0) {\n            return \"\";\n        }\n        return { str() + from , len };\n    }\n\n    const char_type& at(size_t pos) const {\n        if (pos >= size()) {\n            internal::throw_sstring_out_of_range();\n        }\n        return *(str() + pos);\n    }\n\n    char_type& at(size_t pos) {\n        if (pos >= size()) {\n            internal::throw_sstring_out_of_range();\n        }\n        return *(str() + pos);\n    }\n\n    bool empty() const noexcept {\n        return u.internal.size == 0;\n    }\n\n    // Deprecated March 2020.\n    [[deprecated(\"Use = {}\")]]\n    void reset() noexcept {\n        if (is_external()) {\n            std::free(u.external.str);\n        }\n        u.internal.size = 0;\n        if (NulTerminate) {\n            u.internal.str[0] = '\\0';\n        }\n    }\n    temporary_buffer<char_type> release() && {\n        if (is_external()) {\n            auto ptr = u.external.str;\n            auto size = u.external.size;\n            u.external.str = nullptr;\n            u.external.size = 0;\n            return temporary_buffer<char_type>(ptr, size, make_free_deleter(ptr));\n        } else {\n            auto buf = temporary_buffer<char_type>(u.internal.size);\n            std::copy(u.internal.str, u.internal.str + u.internal.size, buf.get_write());\n            u.internal.size = 0;\n            if (NulTerminate) {\n                u.internal.str[0] = '\\0';\n            }\n            return buf;\n        }\n    }\n    int compare(std::basic_string_view<char_type, traits_type> x) const noexcept {\n        auto n = traits_type::compare(begin(), x.data(), std::min(size(), x.size()));\n        if (n != 0) {\n            return n;\n        }\n        if (size() < x.size()) {\n            return -1;\n        } else if (size() > x.size()) {\n            return 1;\n        } else {\n            return 0;\n        }\n    }\n\n    int compare(size_t pos, size_t sz, std::basic_string_view<char_type, traits_type> x) const {\n        if (pos > size()) {\n            internal::throw_sstring_out_of_range();\n        }\n\n        sz = std::min(size() - pos, sz);\n        auto n = traits_type::compare(begin() + pos, x.data(), std::min(sz, x.size()));\n        if (n != 0) {\n            return n;\n        }\n        if (sz < x.size()) {\n            return -1;\n        } else if (sz > x.size()) {\n            return 1;\n        } else {\n            return 0;\n        }\n    }\n\n    constexpr bool starts_with(std::basic_string_view<char_type, traits_type> sv) const noexcept {\n        return size() >= sv.size() && compare(0, sv.size(), sv) == 0;\n    }\n\n    constexpr bool starts_with(char_type c) const noexcept {\n        return !empty() && traits_type::eq(front(), c);\n    }\n\n    constexpr bool starts_with(const char_type* s) const noexcept {\n        return starts_with(std::basic_string_view<char_type, traits_type>(s));\n    }\n\n    constexpr bool ends_with(std::basic_string_view<char_type, traits_type> sv) const noexcept {\n        return size() >= sv.size() && compare(size() - sv.size(), npos, sv) == 0;\n    }\n\n    constexpr bool ends_with(char_type c) const noexcept {\n        return !empty() && traits_type::eq(back(), c);\n    }\n\n    constexpr bool ends_with(const char_type* s) const noexcept {\n        return ends_with(std::basic_string_view<char_type, traits_type>(s));\n    }\n\n    constexpr bool contains(std::basic_string_view<char_type, traits_type> sv) const noexcept {\n        return find(sv) != npos;\n    }\n\n    constexpr bool contains(char_type c) const noexcept {\n        return find(c) != npos;\n    }\n\n    constexpr bool contains(const char_type* s) const noexcept {\n        return find(s) != npos;\n    }\n\n    void swap(basic_sstring& x) noexcept {\n        contents tmp;\n        tmp = x.u;\n        x.u = u;\n        u = tmp;\n    }\n    char_type* data() noexcept {\n        return str();\n    }\n    const char_type* data() const noexcept {\n        return str();\n    }\n    const char_type* c_str() const noexcept {\n        return str();\n    }\n    const char_type* begin() const noexcept { return str(); }\n    const char_type* end() const noexcept { return str() + size(); }\n    const char_type* cbegin() const noexcept { return str(); }\n    const char_type* cend() const noexcept { return str() + size(); }\n    char_type* begin() noexcept { return str(); }\n    char_type* end() noexcept { return str() + size(); }\n    bool operator==(const basic_sstring& x) const noexcept {\n        return size() == x.size() && std::equal(begin(), end(), x.begin());\n    }\n    bool operator!=(const basic_sstring& x) const noexcept {\n        return !operator==(x);\n    }\n    constexpr bool operator==(const std::basic_string<char_type> x) const noexcept {\n        return compare(x) == 0;\n    }\n    constexpr bool operator==(const char_type* x) const noexcept {\n        return compare(x) == 0;\n    }\n#if __cpp_lib_three_way_comparison\n    constexpr std::strong_ordering operator<=>(const auto& x) const noexcept {\n        return compare(x) <=> 0;\n    }\n#else\n    bool operator<(const basic_sstring& x) const noexcept {\n        return compare(x) < 0;\n    }\n#endif\n    basic_sstring operator+(const basic_sstring& x) const {\n        basic_sstring ret(initialized_later(), size() + x.size());\n        std::copy(begin(), end(), ret.begin());\n        std::copy(x.begin(), x.end(), ret.begin() + size());\n        return ret;\n    }\n    basic_sstring& operator+=(const basic_sstring& x) {\n        return *this = *this + x;\n    }\n    char_type& operator[](size_type pos) noexcept {\n        return str()[pos];\n    }\n    const char_type& operator[](size_type pos) const noexcept {\n        return str()[pos];\n    }\n\n    operator std::basic_string_view<char_type>() const noexcept {\n        // we assume that std::basic_string_view<char_type>(str(), size())\n        // won't throw, although it is not specified as noexcept in\n        // https://en.cppreference.com/w/cpp/string/basic_string_view/basic_string_view\n        // at this time (C++20).\n        //\n        // This is similar to std::string operator std::basic_string_view:\n        // https://en.cppreference.com/w/cpp/string/basic_string/operator_basic_string_view\n        // that is specified as noexcept too.\n        static_assert(noexcept(std::basic_string_view<char_type>(str(), size())));\n        return std::basic_string_view<char_type>(str(), size());\n    }\n};\n\nnamespace internal {\ntemplate <class T> struct is_sstring : std::false_type {};\ntemplate <typename char_type, typename Size, Size max_size, bool NulTerminate>\nstruct is_sstring<basic_sstring<char_type, Size, max_size, NulTerminate>> : std::true_type {};\n}\n\ntemplate <typename string_type = sstring>\nstring_type uninitialized_string(size_t size) {\n    if constexpr (internal::is_sstring<string_type>::value) {\n        return string_type(typename string_type::initialized_later(), size);\n    } else {\n        string_type ret;\n#ifdef __cpp_lib_string_resize_and_overwrite\n        ret.resize_and_overwrite(size, [](typename string_type::value_type*, typename string_type::size_type n) { return n; });\n#else\n        ret.resize(size);\n#endif\n        return ret;\n    }\n}\n\ntemplate <typename char_type, typename size_type, size_type Max, size_type N, bool NulTerminate>\ninline\nbasic_sstring<char_type, size_type, Max, NulTerminate>\noperator+(const char(&s)[N], const basic_sstring<char_type, size_type, Max, NulTerminate>& t) {\n    using sstring = basic_sstring<char_type, size_type, Max, NulTerminate>;\n    // don't copy the terminating NUL character\n    sstring ret(typename sstring::initialized_later(), N-1 + t.size());\n    auto p = std::copy(std::begin(s), std::end(s)-1, ret.begin());\n    std::copy(t.begin(), t.end(), p);\n    return ret;\n}\n\ntemplate <typename T>\nstatic inline\nsize_t constexpr str_len(const T& s) {\n    return std::string_view(s).size();\n}\n\ntemplate <typename char_type, typename size_type, size_type max_size>\ninline\nvoid swap(basic_sstring<char_type, size_type, max_size>& x,\n          basic_sstring<char_type, size_type, max_size>& y) noexcept\n{\n    return x.swap(y);\n}\n\ntemplate <typename char_type, typename size_type, size_type max_size, bool NulTerminate, typename char_traits>\ninline\nstd::basic_ostream<char_type, char_traits>&\noperator<<(std::basic_ostream<char_type, char_traits>& os,\n        const basic_sstring<char_type, size_type, max_size, NulTerminate>& s) {\n    return os.write(s.begin(), s.size());\n}\n\ntemplate <typename char_type, typename size_type, size_type max_size, bool NulTerminate, typename char_traits>\ninline\nstd::basic_istream<char_type, char_traits>&\noperator>>(std::basic_istream<char_type, char_traits>& is,\n        basic_sstring<char_type, size_type, max_size, NulTerminate>& s) {\n    std::string tmp;\n    is >> tmp;\n    s = tmp;\n    return is;\n}\n\n}\n\nnamespace std {\n\ntemplate <typename char_type, typename size_type, size_type max_size, bool NulTerminate>\nstruct hash<seastar::basic_sstring<char_type, size_type, max_size, NulTerminate>> {\n    size_t operator()(const seastar::basic_sstring<char_type, size_type, max_size, NulTerminate>& s) const {\n        return std::hash<std::basic_string_view<char_type>>()(s);\n    }\n};\n\n}\n\nnamespace seastar {\n\ntemplate <typename T>\nstatic inline\nvoid copy_str_to(char*& dst, const T& s) {\n    std::string_view v(s);\n    dst = std::copy(v.begin(), v.end(), dst);\n}\n\ntemplate <typename String = sstring, typename... Args>\nstatic String make_sstring(Args&&... args)\n{\n    String ret = uninitialized_string<String>((str_len(args) + ...));\n    auto dst = ret.data();\n    (copy_str_to(dst, args), ...);\n    return ret;\n}\n\nnamespace internal {\ntemplate <typename string_type, typename T>\nstring_type to_sstring(T value) {\n    auto size = fmt::formatted_size(\"{}\", value);\n    auto formatted = uninitialized_string<string_type>(size);\n    fmt::format_to(formatted.data(), \"{}\", value);\n    return formatted;\n}\n\ntemplate <typename string_type>\nstring_type to_sstring(const char* value) {\n    return string_type(value);\n}\n\ntemplate <typename string_type>\nstring_type to_sstring(sstring value) {\n    return value;\n}\n\ntemplate <typename string_type>\nstring_type to_sstring(const temporary_buffer<char>& buf) {\n    return string_type(buf.get(), buf.size());\n}\n}\n\ntemplate <typename string_type = sstring, typename T>\nstring_type to_sstring(T value) {\n    return internal::to_sstring<string_type>(value);\n}\n}\n\n#ifdef SEASTAR_DEPRECATED_OSTREAM_FORMATTERS\n\nnamespace std {\n\ntemplate <typename T>\n[[deprecated(\"Use {fmt} instead\")]]\ninline\nstd::ostream& operator<<(std::ostream& os, const std::vector<T>& v) {\n    bool first = true;\n    os << \"{\";\n    for (auto&& elem : v) {\n        if (!first) {\n            os << \", \";\n        } else {\n            first = false;\n        }\n        os << elem;\n    }\n    os << \"}\";\n    return os;\n}\n\ntemplate <typename Key, typename T, typename Hash, typename KeyEqual, typename Allocator>\n[[deprecated(\"Use {fmt} instead\")]]\nstd::ostream& operator<<(std::ostream& os, const std::unordered_map<Key, T, Hash, KeyEqual, Allocator>& v) {\n    bool first = true;\n    os << \"{\";\n    for (auto&& elem : v) {\n        if (!first) {\n            os << \", \";\n        } else {\n            first = false;\n        }\n        os << \"{\" << elem.first << \" -> \" << elem.second << \"}\";\n    }\n    os << \"}\";\n    return os;\n}\n}\n\n#endif\n\n#if FMT_VERSION >= 110000\n\ntemplate <typename char_type, typename Size, Size max_size, bool NulTerminate>\nstruct fmt::range_format_kind<seastar::basic_sstring<char_type, Size, max_size, NulTerminate>, char_type> : std::integral_constant<fmt::range_format, fmt::range_format::disabled>\n{};\n\n#endif\n\n\n// Due to https://github.com/llvm/llvm-project/issues/68849, we inherit\n// from formatter<string_view> publicly rather than privately\n\ntemplate <typename char_type, typename Size, Size max_size, bool NulTerminate>\nstruct fmt::formatter<seastar::basic_sstring<char_type, Size, max_size, NulTerminate>>\n    : public fmt::formatter<basic_string_view<char_type>> {\n    using format_as_t = basic_string_view<char_type>;\n    using base = fmt::formatter<format_as_t>;\n    template <typename FormatContext>\n    auto format(const ::seastar::basic_sstring<char_type, Size, max_size, NulTerminate>& s, FormatContext& ctx) const {\n        return base::format(format_as_t{s.c_str(), s.size()}, ctx);\n    }\n};\n\n"
  },
  {
    "path": "include/seastar/core/stall_sampler.hh",
    "content": "/*\n * Copyright (C) 2018 ScyllaDB\n */\n\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/util/noncopyable_function.hh>\n\n#include <iosfwd>\n\n// Instrumentation to detect context switches during reactor execution\n// and associated stall time, intended for use in tests\n\nnamespace seastar {\n\nnamespace internal {\n\nstruct stall_report {\n    uint64_t kernel_stalls;\n    sched_clock::duration run_wall_time;  // excludes sleeps\n    sched_clock::duration stall_time;\n};\n\n/// Run the unit-under-test (uut) function until completion, and report on any\n/// reactor stalls it generated.\nfuture<stall_report> report_reactor_stalls(noncopyable_function<future<> ()> uut);\n\nstd::ostream& operator<<(std::ostream& os, const stall_report& sr);\n\n}\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/stream.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <exception>\n#include <cassert>\n\nnamespace seastar {\n\n\n// A stream/subscription pair is similar to a promise/future pair,\n// but apply to a sequence of values instead of a single value.\n//\n// A stream<> is the producer side.  It may call produce() as long\n// as the future<> returned from the previous invocation is ready.\n// To signify no more data is available, call close().\n//\n// A subscription<> is the consumer side.  It is created by a call\n// to stream::listen().  Calling subscription::start(),\n// which registers the data processing callback, starts processing\n// events.  It may register for end-of-stream notifications by\n// chaining the when_done() future, which also delivers error\n// events (as exceptions).\n//\n// The consumer can pause generation of new data by returning\n// a non-ready future; when the future becomes ready, the producer\n// will resume processing.\n\ntemplate <typename... T>\nclass stream;\n\ntemplate <typename... T>\nclass subscription;\n\ntemplate <typename... T>\nclass stream {\npublic:\n    using next_fn = noncopyable_function<future<> (T...)>;\n\nprivate:\n    promise<> _done;\n    promise<> _ready;\n    next_fn _next;\n\n    /// \\brief Start receiving events from the stream.\n    ///\n    /// \\param next Callback to call for each event\n    void start(next_fn next) {\n        _next = std::move(next);\n        _ready.set_value();\n    }\n\npublic:\n    stream() = default;\n    stream(const stream&) = delete;\n    stream(stream&&) = delete;\n    void operator=(const stream&) = delete;\n    void operator=(stream&&) = delete;\n\n    // Returns a subscription that reads value from this\n    // stream.\n    subscription<T...> listen() {\n        return subscription<T...>(this);\n    }\n\n    // Returns a subscription that reads value from this\n    // stream, and also sets up the listen function.\n    subscription<T...> listen(next_fn next) {\n        start(std::move(next));\n        return subscription<T...>(this);\n    }\n\n    // Becomes ready when the listener is ready to accept\n    // values.  Call only once, when beginning to produce\n    // values.\n    future<> started() {\n        return _ready.get_future();\n    }\n\n    // Produce a value.  Call only after started(), and after\n    // a previous produce() is ready.\n    future<> produce(T... data);\n\n    // End the stream.   Call only after started(), and after\n    // a previous produce() is ready.  No functions may be called\n    // after this.\n    void close() {\n        _done.set_value();\n    }\n\n    // Signal an error.   Call only after started(), and after\n    // a previous produce() is ready.  No functions may be called\n    // after this.\n    template <typename E>\n    void set_exception(E ex) {\n        _done.set_exception(ex);\n    }\n\n    friend class subscription<T...>;\n};\n\ntemplate <typename... T>\nclass subscription {\n    stream<T...>* _stream;\n    future<> _done;\n    explicit subscription(stream<T...>* s) : _stream(s), _done(s->_done.get_future()) {\n    }\n\npublic:\n    using next_fn = typename stream<T...>::next_fn;\n    subscription(subscription&& x) : _stream(x._stream), _done(std::move(x._done)) {\n        x._stream = nullptr;\n    }\n\n    /// \\brief Start receiving events from the stream.\n    ///\n    /// \\param next Callback to call for each event\n    void start(next_fn next) {\n        return _stream->start(std::move(next));\n    }\n\n    // Becomes ready when the stream is empty, or when an error\n    // happens (in that case, an exception is held).\n    future<> done() {\n        return std::move(_done);\n    }\n\n    friend class stream<T...>;\n};\n\ntemplate <typename... T>\ninline\nfuture<>\nstream<T...>::produce(T... data) {\n    auto ret = futurize_invoke(_next, std::move(data)...);\n    if (ret.available() && !ret.failed()) {\n        // Native network stack depends on stream::produce() returning\n        // a ready future to push packets along without dropping.  As\n        // a temporary workaround, special case a ready, unfailed future\n        // and return it immediately, so that then_wrapped(), below,\n        // doesn't convert a ready future to an unready one.\n        return ret;\n    }\n    return ret.then_wrapped([this] (auto&& f) {\n        try {\n            f.get();\n        } catch (...) {\n            _done.set_exception(std::current_exception());\n            // FIXME: tell the producer to stop producing\n            throw;\n        }\n    });\n}\n}\n"
  },
  {
    "path": "include/seastar/core/task.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/scheduling.hh>\n#include <seastar/util/backtrace.hh>\n\n#include <utility>\n#include <source_location>\n\nnamespace seastar {\n\nclass task {\n    std::source_location _resume_point = {};\n\nprotected:\n    scheduling_group _sg;\n\nprivate:\n#ifdef SEASTAR_TASK_BACKTRACE\n    shared_backtrace _bt;\n#endif\nprotected:\n    // Task destruction is performed by run_and_dispose() via a concrete type,\n    // so no need for a virtual destructor here. Derived classes that implement\n    // run_and_dispose() should be declared final to avoid losing concrete type\n    // information via inheritance.\n    ~task() = default;\n\n    scheduling_group set_scheduling_group(scheduling_group new_sg) noexcept{\n        return std::exchange(_sg, new_sg);\n    }\npublic:\n    explicit task(scheduling_group sg = current_scheduling_group()) noexcept : _sg(sg) {}\n    virtual void run_and_dispose() noexcept = 0;\n    /// Returns the next task which is waiting for this task to complete execution, or nullptr.\n    virtual task* waiting_task() noexcept = 0;\n    void update_resume_point(std::source_location sl) { _resume_point = sl; }\n    auto get_resume_point() const { return _resume_point; }\n    scheduling_group group() const { return _sg; }\n#ifdef SEASTAR_TASK_BACKTRACE\n    void make_backtrace() noexcept;\n    shared_backtrace get_backtrace() const { return _bt; }\n#else\n    void make_backtrace() noexcept {}\n    shared_backtrace get_backtrace() const { return {}; }\n#endif\n};\n\n\nvoid schedule(task* t) noexcept;\nvoid schedule_checked(task* t) noexcept;\nvoid schedule_urgent(task* t) noexcept;\n\n}\n"
  },
  {
    "path": "include/seastar/core/temporary_buffer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <algorithm>\n#include <cstddef>\n#include <cstring>\n#include <string_view>\n#include <malloc.h>\n#include <seastar/core/deleter.hh>\n#include <seastar/util/eclipse.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\n/// \\addtogroup memory-module\n/// @{\n\n/// Temporary, self-managed byte buffer.\n///\n/// A \\c temporary_buffer is similar to an \\c std::string or a \\c std::unique_ptr<char[]>,\n/// but provides more flexible memory management.  A \\c temporary_buffer can own the memory\n/// it points to, or it can be shared with another \\c temporary_buffer, or point at a substring\n/// of a buffer.  It uses a \\ref deleter to manage its memory.\n///\n/// A \\c temporary_buffer should not be held indefinitely.  It can be held while a request\n/// is processed, or for a similar duration, but not longer, as it can tie up more memory\n/// that its size indicates.\n///\n/// A buffer can be shared: two \\c temporary_buffer objects will point to the same data,\n/// or a subset of it.  See the \\ref temporary_buffer::share() method.\n///\n/// Unless you created a \\c temporary_buffer yourself, do not modify its contents, as they\n/// may be shared with another user that does not expect the data to change.\n///\n/// Use cases for a \\c temporary_buffer include:\n///  - passing a substring of a tcp packet for the user to consume (zero-copy\n///    tcp input)\n///  - passing a refcounted blob held in memory to tcp, ensuring that when the TCP ACK\n///    is received, the blob is released (by decrementing its reference count) (zero-copy\n///    tcp output)\n///\n/// \\tparam CharType underlying character type (must be a variant of \\c char).\ntemplate <typename CharType>\nclass temporary_buffer {\n    static_assert(sizeof(CharType) == 1, \"must buffer stream of bytes\");\n    CharType* _buffer;\n    size_t _size;\n    deleter _deleter;\npublic:\n    /// Creates a \\c temporary_buffer of a specified size.  The buffer is not shared\n    /// with anyone, and is not initialized.\n    ///\n    /// \\param size buffer size, in bytes\n    explicit temporary_buffer(size_t size)\n        : _buffer(static_cast<CharType*>(malloc(size * sizeof(CharType)))), _size(size)\n        , _deleter(make_free_deleter(_buffer)) {\n        if (size && !_buffer) {\n            throw std::bad_alloc();\n        }\n    }\n    //explicit temporary_buffer(CharType* borrow, size_t size) : _buffer(borrow), _size(size) {}\n    /// Creates an empty \\c temporary_buffer that does not point at anything.\n    temporary_buffer() noexcept\n        : _buffer(nullptr)\n        , _size(0) {}\n    temporary_buffer(const temporary_buffer&) = delete;\n\n    /// Moves a \\c temporary_buffer.\n    temporary_buffer(temporary_buffer&& x) noexcept : _buffer(x._buffer), _size(x._size), _deleter(std::move(x._deleter)) {\n        x._buffer = nullptr;\n        x._size = 0;\n    }\n\n    /// Creates a \\c temporary_buffer with a specific deleter.\n    ///\n    /// \\param buf beginning of the buffer held by this \\c temporary_buffer\n    /// \\param size size of the buffer\n    /// \\param d deleter controlling destruction of the  buffer.  The deleter\n    ///          will be destroyed when there are no longer any users for the buffer.\n    temporary_buffer(CharType* buf, size_t size, deleter d) noexcept\n        : _buffer(buf), _size(size), _deleter(std::move(d)) {}\n    /// Creates a `temporary_buffer` containing a copy of the provided data\n    ///\n    /// \\param src  data buffer to be copied\n    /// \\param size size of data buffer in `src`\n    temporary_buffer(const CharType* src, size_t size) : temporary_buffer(size) {\n        std::copy_n(src, size, _buffer);\n    }\n    void operator=(const temporary_buffer&) = delete;\n    /// Moves a \\c temporary_buffer.\n    temporary_buffer& operator=(temporary_buffer&& x) noexcept {\n        if (this != &x) {\n            _buffer = x._buffer;\n            _size = x._size;\n            _deleter = std::move(x._deleter);\n            x._buffer = nullptr;\n            x._size = 0;\n        }\n        return *this;\n    }\n    /// Gets a pointer to the beginning of the buffer.\n    const CharType* get() const noexcept { return _buffer; }\n    /// Gets a writable pointer to the beginning of the buffer.  Use only\n    /// when you are certain no user expects the buffer data not to change.\n    CharType* get_write() noexcept { return _buffer; }\n    /// Gets the buffer size.\n    size_t size() const noexcept { return _size; }\n    /// Gets a pointer to the beginning of the buffer.\n    const CharType* begin() const noexcept { return _buffer; }\n    /// Gets a pointer to the end of the buffer.\n    const CharType* end() const noexcept { return _buffer + _size; }\n    /// Returns the buffer, but with a reduced size.  The original\n    /// buffer is consumed by this call and can no longer be used.\n    ///\n    /// \\param size New size; must be smaller than current size.\n    /// \\return the same buffer, with a prefix removed.\n    temporary_buffer prefix(size_t size) && noexcept {\n        auto ret = std::move(*this);\n        ret._size = size;\n        return ret;\n    }\n    /// Reads a character from a specific position in the buffer.\n    ///\n    /// \\param pos position to read character from; must be less than size.\n    CharType operator[](size_t pos) const noexcept {\n        return _buffer[pos];\n    }\n    /// Checks whether the buffer is empty.\n    bool empty() const noexcept { return !size(); }\n    /// Checks whether the buffer is not empty.\n    explicit operator bool() const noexcept { return size(); }\n    /// Create a new \\c temporary_buffer object referring to the same\n    /// underlying data.  The underlying \\ref deleter will not be destroyed\n    /// until both the original and the clone have been destroyed.\n    ///\n    /// \\return a clone of the buffer object.\n    temporary_buffer share() {\n        return temporary_buffer(_buffer, _size, _deleter.share());\n    }\n    /// Create a new \\c temporary_buffer object referring to a substring of the\n    /// same underlying data.  The underlying \\ref deleter will not be destroyed\n    /// until both the original and the clone have been destroyed.\n    ///\n    /// \\param pos Position of the first character to share.\n    /// \\param len Length of substring to share.\n    /// \\return a clone of the buffer object, referring to a substring.\n    temporary_buffer share(size_t pos, size_t len) {\n        auto ret = share();\n        ret._buffer += pos;\n        ret._size = len;\n        return ret;\n    }\n    /// Clone the current \\c temporary_buffer object into a new one.\n    /// This creates a temporary buffer with the same length and data but not\n    /// pointing to the memory of the original object.\n    temporary_buffer clone() const {\n        return {_buffer, _size};\n    }\n    /// Remove a prefix from the buffer.  The underlying data\n    /// is not modified.\n    ///\n    /// \\param pos Position of first character to retain.\n    void trim_front(size_t pos) noexcept {\n        _buffer += pos;\n        _size -= pos;\n    }\n    /// Remove a suffix from the buffer.  The underlying data\n    /// is not modified.\n    ///\n    /// \\param pos Position of first character to drop.\n    void trim(size_t pos) noexcept {\n        _size = pos;\n    }\n    /// Stops automatic memory management.  When the \\c temporary_buffer\n    /// object is destroyed, the underlying \\ref deleter will not be called.\n    /// Instead, it is the caller's responsibility to destroy the deleter object\n    /// when the data is no longer needed.\n    ///\n    /// \\return \\ref deleter object managing the data's lifetime.\n    deleter release() noexcept {\n        return std::move(_deleter);\n    }\n    /// Creates a \\c temporary_buffer object with a specified size, with\n    /// memory aligned to a specific boundary.\n    ///\n    /// \\param alignment Required alignment; must be a power of two and a multiple of sizeof(void *).\n    /// \\param size Required size; must be a multiple of alignment.\n    /// \\return a new \\c temporary_buffer object.\n    static temporary_buffer aligned(size_t alignment, size_t size) {\n        void *ptr = nullptr;\n        auto ret = ::posix_memalign(&ptr, alignment, size * sizeof(CharType));\n        auto buf = static_cast<CharType*>(ptr);\n        if (ret) {\n            throw std::bad_alloc();\n        }\n        return temporary_buffer(buf, size, make_free_deleter(buf));\n    }\n\n    static temporary_buffer copy_of(std::string_view view) {\n        void* ptr = ::malloc(view.size());\n        if (!ptr) {\n            throw std::bad_alloc();\n        }\n        auto buf = static_cast<CharType*>(ptr);\n        std::memcpy(buf, view.data(), view.size());\n        return temporary_buffer(buf, view.size(), make_free_deleter(buf));\n    }\n\n    /// Compare contents of this buffer with another buffer for equality\n    ///\n    /// \\param o buffer to compare with\n    /// \\return true if and only if contents are the same\n    bool operator==(const temporary_buffer& o) const noexcept {\n        return size() == o.size() && std::equal(begin(), end(), o.begin());\n    }\n\n    /// Compare contents of this buffer with another buffer for inequality\n    ///\n    /// \\param o buffer to compare with\n    /// \\return true if and only if contents are not the same\n    bool operator!=(const temporary_buffer& o) const noexcept {\n        return !(*this == o);\n    }\n};\n\n/// @}\n\nnamespace internal {\n\n// Takes a vector of buffers and detaches some buffers from its front so\n// that the total size of the detached part equals the given value. The\n// detached buffers are returned, the original vector is updated to hold\n// the remainder.\n//\n// Buffers content is not copied. If it happens that one buffer must be\n// split, it is shared between the returned and the original vectors.\n\ntemplate <typename CharType>\ninline std::vector<temporary_buffer<CharType>> detach_front(std::vector<temporary_buffer<CharType>>& bufs, size_t length) {\n    std::vector<temporary_buffer<CharType>> ret;\n    auto it = bufs.begin();\n    while (it != bufs.end()) {\n        if (it->size() <= length) {\n            length -= it->size();\n            ret.emplace_back(std::move(*it));\n            it++;\n            continue;\n        } else {\n            if (length > 0) {\n                auto b = it->share();\n                b.trim(length);\n                it->trim_front(length);\n                ret.emplace_back(std::move(b));\n                length = 0;\n            }\n            break;\n        }\n    }\n    bufs.erase(bufs.begin(), it);\n    SEASTAR_ASSERT(length == 0);\n    return ret;\n}\n\n} // internal namespace\n\n}\n"
  },
  {
    "path": "include/seastar/core/thread.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/thread_impl.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/scheduling.hh>\n#include <memory>\n#include <type_traits>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <ucontext.h>\n#include <boost/intrusive/list.hpp>\n\n/// \\defgroup thread-module Seastar threads\n///\n/// Seastar threads provide an execution environment where blocking\n/// is tolerated; you can issue I/O, and wait for it in the same function,\n/// rather then establishing a callback to be called with \\ref future<>::then().\n///\n/// Seastar threads are not the same as operating system threads:\n///   - seastar threads are cooperative; they are never preempted except\n///     at blocking points (see below)\n///   - seastar threads always run on the same core they were launched on\n///\n/// Like other seastar code, seastar threads may not issue blocking system calls.\n///\n/// A seastar thread blocking point is any function that returns a \\ref future.\n/// you block by calling \\ref future<>::get(); this waits for the future to become\n/// available, and in the meanwhile, other seastar threads and seastar non-threaded\n/// code may execute.\n///\n/// Example:\n/// \\code\n///    seastar::thread th([] {\n///       sleep(5s).get();  // blocking point\n///    });\n/// \\endcode\n///\n/// An easy way to launch a thread and carry out some computation, and return a\n/// result from this execution is by using the \\ref seastar::async() function.\n/// The result is returned as a future, so that non-threaded code can wait for\n/// the thread to terminate and yield a result.\n\n/// Seastar API namespace\nnamespace seastar {\n\n/// \\addtogroup thread-module\n/// @{\nclass thread;\nclass thread_attributes;\n\n/// Class that holds attributes controling the behavior of a thread.\nclass thread_attributes {\npublic:\n    std::optional<seastar::scheduling_group> sched_group;\n    // For stack_size 0, a default value will be used (128KiB when writing this comment)\n    size_t stack_size = 0;\n};\n\n/// \\cond internal\nextern thread_local jmp_buf_link g_unthreaded_context;\n\n// Internal class holding thread state.  We can't hold this in\n// \\c thread itself because \\c thread is movable, and we want pointers\n// to this state to be captured.\nclass thread_context final : private task {\n    struct stack_deleter {\n        void operator()(char *ptr) const noexcept;\n        int valgrind_id;\n        stack_deleter(int valgrind_id);\n    };\n    using stack_holder = std::unique_ptr<char[], stack_deleter>;\n\n    stack_holder _stack;\n    noncopyable_function<void ()> _func;\n    jmp_buf_link _context;\n    promise<> _done;\n    bool _joined = false;\n\n    boost::intrusive::list_member_hook<> _all_link;\n    using all_thread_list = boost::intrusive::list<thread_context,\n        boost::intrusive::member_hook<thread_context, boost::intrusive::list_member_hook<>,\n        &thread_context::_all_link>,\n        boost::intrusive::constant_time_size<false>>;\n\n    static thread_local all_thread_list _all_threads;\nprivate:\n    static void s_main(int lo, int hi); // all parameters MUST be 'int' for makecontext\n    void setup(size_t stack_size);\n    void main();\n    stack_holder make_stack(size_t stack_size);\n    virtual void run_and_dispose() noexcept override; // from task class\npublic:\n    thread_context(thread_attributes attr, noncopyable_function<void ()> func);\n    ~thread_context();\n    void switch_in();\n    void switch_out();\n    bool should_yield() const;\n    void reschedule();\n    void yield();\n    task* waiting_task() noexcept override { return _done.waiting_task(); }\n    friend class thread;\n    friend void thread_impl::switch_in(thread_context*);\n    friend void thread_impl::switch_out(thread_context*);\n};\n\n/// \\endcond\n\n/// \\brief thread - stateful thread of execution\n///\n/// Threads allow using seastar APIs in a blocking manner,\n/// by calling future::get() on a non-ready future.  When\n/// this happens, the thread is put to sleep until the future\n/// becomes ready.\nclass thread {\n    std::unique_ptr<thread_context> _context;\n    static thread_local thread* _current;\npublic:\n    /// \\brief Constructs a \\c thread object that does not represent a thread\n    /// of execution.\n    thread() = default;\n    /// \\brief Constructs a \\c thread object that represents a thread of execution\n    ///\n    /// \\param func Callable object to execute in thread.  The callable is\n    ///             called immediately.\n    template <typename Func>\n    thread(Func func);\n    /// \\brief Constructs a \\c thread object that represents a thread of execution\n    ///\n    /// \\param attr Attributes describing the new thread.\n    /// \\param func Callable object to execute in thread.  The callable is\n    ///             called immediately.\n    template <typename Func>\n    thread(thread_attributes attr, Func func);\n    /// \\brief Moves a thread object.\n    thread(thread&& x) noexcept = default;\n    /// \\brief Move-assigns a thread object.\n    thread& operator=(thread&& x) noexcept = default;\n    /// \\brief Destroys a \\c thread object.\n    ///\n    /// The thread must not represent a running thread of execution (see join()).\n    ~thread() { SEASTAR_ASSERT(!_context || _context->_joined); }\n    /// \\brief Waits for thread execution to terminate.\n    ///\n    /// Waits for thread execution to terminate, and marks the thread object as not\n    /// representing a running thread of execution.\n    future<> join();\n    /// \\brief Voluntarily defer execution of current thread.\n    ///\n    /// Gives other threads/fibers a chance to run on current CPU.\n    /// The current thread will resume execution promptly.\n    static void yield();\n    /// \\brief Checks whether this thread ought to call yield() now.\n    ///\n    /// Useful where we cannot call yield() immediately because we\n    /// Need to take some cleanup action first.\n    static bool should_yield() {\n        return need_preempt();\n    }\n\n    /// \\brief Yield if this thread ought to call yield() now.\n    ///\n    /// Useful where a code does long running computation and does\n    /// not want to hog cpu for more then its share\n    static void maybe_yield() {\n        if (should_yield()) [[unlikely]] {\n            yield();\n        }\n    }\n\n    static bool running_in_thread() {\n        return thread_impl::get() != nullptr;\n    }\n};\n\ntemplate <typename Func>\ninline\nthread::thread(thread_attributes attr, Func func)\n        : _context(std::make_unique<thread_context>(std::move(attr), std::move(func))) {\n}\n\ntemplate <typename Func>\ninline\nthread::thread(Func func)\n        : thread(thread_attributes(), std::move(func)) {\n}\n\ninline\nfuture<>\nthread::join() {\n    _context->_joined = true;\n    return _context->_done.get_future();\n}\n\n/// Executes a callable in a seastar thread.\n///\n/// Runs a block of code in a threaded context,\n/// which allows it to block (using \\ref future::get()).  The\n/// result of the callable is returned as a future.\n///\n/// \\param attr a \\ref thread_attributes instance\n/// \\param func a callable to be executed in a thread\n/// \\param args a parameter pack to be forwarded to \\c func.\n/// \\return whatever \\c func returns, as a future.\n///\n/// Example:\n/// \\code\n///    future<int> compute_sum(int a, int b) {\n///        thread_attributes attr = {};\n///        attr.sched_group = some_scheduling_group_ptr;\n///        return seastar::async(attr, [a, b] {\n///            // some blocking code:\n///            sleep(1s).get();\n///            return a + b;\n///        });\n///    }\n/// \\endcode\ntemplate <typename Func, typename... Args>\ninline\nfuturize_t<std::invoke_result_t<Func, Args...>>\nasync(thread_attributes attr, Func&& func, Args&&... args) noexcept {\n    using return_type = std::invoke_result_t<Func, Args...>;\n    struct work {\n        thread_attributes attr;\n        Func func;\n        std::tuple<Args...> args;\n        promise<return_type> pr{};\n        thread th{};\n    };\n\n    try {\n        auto wp = std::make_unique<work>(work{std::move(attr), std::forward<Func>(func), std::forward_as_tuple(std::forward<Args>(args)...)});\n        auto& w = *wp;\n        auto ret = w.pr.get_future();\n        w.th = thread(std::move(w.attr), [&w] {\n            futurize<return_type>::apply(std::move(w.func), std::move(w.args)).forward_to(std::move(w.pr));\n        });\n        return w.th.join().then([ret = std::move(ret)] () mutable {\n            return std::move(ret);\n        }).finally([wp = std::move(wp)] {});\n    } catch (...) {\n        return futurize<return_type>::make_exception_future(std::current_exception());\n    }\n}\n\n/// Executes a callable in a seastar thread.\n///\n/// Runs a block of code in a threaded context,\n/// which allows it to block (using \\ref future::get()).  The\n/// result of the callable is returned as a future.\n///\n/// \\param func a callable to be executed in a thread\n/// \\param args a parameter pack to be forwarded to \\c func.\n/// \\return whatever \\c func returns, as a future.\ntemplate <typename Func, typename... Args>\ninline\nfuturize_t<std::invoke_result_t<Func, Args...>>\nasync(Func&& func, Args&&... args) noexcept {\n    return async(thread_attributes{}, std::forward<Func>(func), std::forward<Args>(args)...);\n}\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/core/thread_cputime_clock.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB\n */\n\n#pragma once\n\n#include <chrono>\n#include <time.h>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nclass thread_cputime_clock {\npublic:\n    using rep = int64_t;\n    using period = std::chrono::nanoseconds::period;\n    using duration = std::chrono::duration<rep, period>;\n    using time_point = std::chrono::time_point<thread_cputime_clock, duration>;\npublic:\n    static time_point now() {\n        using namespace std::chrono_literals;\n\n        struct timespec tp;\n        [[gnu::unused]] auto ret = clock_gettime(CLOCK_THREAD_CPUTIME_ID, &tp);\n        SEASTAR_ASSERT(ret == 0);\n        return time_point(tp.tv_nsec * 1ns + tp.tv_sec * 1s);\n    }\n};\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/thread_impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n#include <seastar/core/preempt.hh>\n#include <seastar/util/std-compat.hh>\n#include <setjmp.h>\n#include <ucontext.h>\n#include <chrono>\n\nnamespace seastar {\n/// Clock used for scheduling threads\nusing thread_clock = std::chrono::steady_clock;\n\n/// \\cond internal\nclass thread_context;\nclass scheduling_group;\n\nstruct jmp_buf_link {\n#ifdef SEASTAR_ASAN_ENABLED\n    ucontext_t context;\n    void* fake_stack = nullptr;\n    const void* stack_bottom;\n    size_t stack_size;\n#else\n    jmp_buf jmpbuf;\n#endif\n    jmp_buf_link* link;\n    thread_context* thread;\npublic:\n    void initial_switch_in(ucontext_t* initial_context, const void* stack_bottom, size_t stack_size);\n    void switch_in();\n    void switch_out();\n    void initial_switch_in_completed();\n    void final_switch_out();\n};\n\nextern thread_local jmp_buf_link* g_current_context;\n\nnamespace thread_impl {\n\ninline thread_context* get() {\n    return g_current_context->thread;\n}\n\ninline bool should_yield() {\n    if (need_preempt()) {\n        return true;\n    } else {\n        return false;\n    }\n}\n\nvoid yield();\nvoid switch_in(thread_context* to);\nvoid switch_out(thread_context* from);\nvoid init();\n\n}\n}\n/// \\endcond\n\n\n"
  },
  {
    "path": "include/seastar/core/timed_out_error.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <fmt/core.h>\n#include <exception>\n\nnamespace seastar {\n\nclass timed_out_error : public std::exception {\npublic:\n    virtual const char* what() const noexcept {\n        return \"timedout\";\n    }\n};\n\nstruct default_timeout_exception_factory {\n    static auto timeout() {\n        return timed_out_error();\n    }\n};\n\n} // namespace seastar\n\n#if FMT_VERSION < 100000\n// fmt v10 introduced formatter for std::exception\ntemplate <>\nstruct fmt::formatter<seastar::timed_out_error> : fmt::formatter<string_view> {\n    auto format(const seastar::timed_out_error& e, fmt::format_context& ctx) const {\n        return fmt::format_to(ctx.out(), \"{}\", e.what());\n    }\n};\n#endif\n"
  },
  {
    "path": "include/seastar/core/timer-set.hh",
    "content": "/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n/*\n * Imported from OSv:\n *\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n * This work is open source software, licensed under the terms of the\n * BSD license as described in the LICENSE file in the top-level directory.\n */\n\n#pragma once\n\n#include <seastar/core/bitset-iter.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/util/assert.hh>\n#include <boost/intrusive/list.hpp>\n#include <exception>\n#include <array>\n#include <bitset>\n#include <limits>\n\nnamespace seastar {\n\nnamespace internal {\nvoid log_timer_callback_exception(std::exception_ptr) noexcept;\n}\n\n/**\n * A data structure designed for holding and expiring timers. It's\n * optimized for timer non-delivery by deferring sorting cost until\n * expiry time. The optimization is based on the observation that in\n * many workloads timers are cancelled or rescheduled before they\n * expire. That's especially the case for TCP timers.\n *\n * The template type \"Timer\" should have a method named\n * get_timeout() which returns Timer::time_point which denotes\n * timer's expiration.\n */\ntemplate<typename Timer, boost::intrusive::list_member_hook<> Timer::*link>\nclass timer_set {\npublic:\n    using time_point = typename Timer::time_point;\n    using timer_list_t = boost::intrusive::list<Timer, boost::intrusive::member_hook<Timer, boost::intrusive::list_member_hook<>, link>>;\nprivate:\n    using duration = typename Timer::duration;\n    using timestamp_t = typename Timer::duration::rep;\n\n    static constexpr timestamp_t max_timestamp = std::numeric_limits<timestamp_t>::max();\n    static constexpr int timestamp_bits = std::numeric_limits<timestamp_t>::digits;\n\n    // The last bucket is reserved for active timers with timeout <= _last.\n    static constexpr int n_buckets = timestamp_bits + 1;\n\n    std::array<timer_list_t, n_buckets> _buckets;\n    timestamp_t _last;\n    timestamp_t _next;\n\n    std::bitset<n_buckets> _non_empty_buckets;\nprivate:\n    static timestamp_t get_timestamp(time_point _time_point) noexcept\n    {\n        return _time_point.time_since_epoch().count();\n    }\n\n    static timestamp_t get_timestamp(Timer& timer) noexcept\n    {\n        return get_timestamp(timer.get_timeout());\n    }\n\n    int get_index(timestamp_t timestamp) const noexcept\n    {\n        if (timestamp <= _last) {\n            return n_buckets - 1;\n        }\n\n        auto index = bitsets::count_leading_zeros(timestamp ^ _last);\n        SEASTAR_ASSERT(index < n_buckets - 1);\n        return index;\n    }\n\n    int get_index(Timer& timer) const noexcept\n    {\n        return get_index(get_timestamp(timer));\n    }\n\n    int get_last_non_empty_bucket() const noexcept\n    {\n        return bitsets::get_last_set(_non_empty_buckets);\n    }\n\npublic:\n    timer_set() noexcept\n        : _last(0)\n        , _next(max_timestamp)\n        , _non_empty_buckets(0)\n    {\n    }\n\n    ~timer_set() {\n        for (auto&& list : _buckets) {\n            while (!list.empty()) {\n                auto& timer = *list.begin();\n                timer.cancel();\n            }\n        }\n    }\n\n    /**\n     * Adds timer to the active set.\n     *\n     * The value returned by timer.get_timeout() is used as timer's expiry. The result\n     * of timer.get_timeout() must not change while the timer is in the active set.\n     *\n     * Preconditions:\n     *  - this timer must not be currently in the active set or in the expired set.\n     *\n     * Postconditions:\n     *  - this timer will be added to the active set until it is expired\n     *    by a call to expire() or removed by a call to remove().\n     *\n     * Returns true if and only if this timer's timeout is less than get_next_timeout().\n     * When this function returns true the caller should reschedule expire() to be\n     * called at timer.get_timeout() to ensure timers are expired in a timely manner.\n     */\n    bool insert(Timer& timer) noexcept\n    {\n        auto timestamp = get_timestamp(timer);\n        auto index = get_index(timestamp);\n\n        _buckets[index].push_back(timer);\n        _non_empty_buckets[index] = true;\n\n        if (timestamp < _next) {\n            _next = timestamp;\n            return true;\n        }\n        return false;\n    }\n\n    /**\n     * Removes timer from the active set.\n     *\n     * Preconditions:\n     *  - timer must be currently in the active set. Note: it must not be in\n     *    the expired set.\n     *\n     * Postconditions:\n     *  - timer is no longer in the active set.\n     *  - this object will no longer hold any references to this timer.\n     */\n    void remove(Timer& timer) noexcept\n    {\n        auto index = get_index(timer);\n        auto& list = _buckets[index];\n        list.erase(list.iterator_to(timer));\n        if (list.empty()) {\n            _non_empty_buckets[index] = false;\n        }\n    }\n\n    /**\n     * Removes timer from the active set or the expired list, if the timer is expired\n     */\n    void remove(Timer& timer, timer_list_t& expired) noexcept\n    {\n        if (timer._expired) {\n            expired.erase(expired.iterator_to(timer));\n            timer._expired = false;\n        } else {\n            remove(timer);\n        }\n    }\n\n    /**\n     * Expires active timers.\n     *\n     * The time points passed to this function must be monotonically increasing.\n     * Use get_next_timeout() to query for the next time point.\n     *\n     * Preconditions:\n     *  - the time_point passed to this function must not be lesser than\n     *    the previous one passed to this function.\n     *\n     * Postconditons:\n     *  - all timers from the active set with Timer::get_timeout() <= now are moved\n     *    to the expired set.\n     */\n    timer_list_t expire(time_point now) noexcept\n    {\n        timer_list_t exp;\n        auto timestamp = get_timestamp(now);\n\n        if (timestamp < _last) {\n            abort();\n        }\n\n        auto index = get_index(timestamp);\n\n        for (int i : bitsets::for_each_set(_non_empty_buckets, index + 1)) {\n            exp.splice(exp.end(), _buckets[i]);\n            _non_empty_buckets[i] = false;\n        }\n\n        _last = timestamp;\n        _next = max_timestamp;\n\n        auto& list = _buckets[index];\n        while (!list.empty()) {\n            auto& timer = *list.begin();\n            list.pop_front();\n            if (timer.get_timeout() <= now) {\n                exp.push_back(timer);\n            } else {\n                insert(timer);\n            }\n        }\n\n        _non_empty_buckets[index] = !list.empty();\n\n        if (_next == max_timestamp && _non_empty_buckets.any()) {\n            for (auto& timer : _buckets[get_last_non_empty_bucket()]) {\n                _next = std::min(_next, get_timestamp(timer));\n            }\n        }\n        return exp;\n    }\n\n    void complete(timer_list_t& expired_timers) noexcept {\n        expired_timers = expire(this->now());\n        for (auto& t : expired_timers) {\n            t._expired = true;\n        }\n        const auto prev_sg = current_scheduling_group();\n        while (!expired_timers.empty()) {\n            auto t = &*expired_timers.begin();\n            expired_timers.pop_front();\n            t->_queued = false;\n            if (t->_armed) {\n                t->_armed = false;\n                if (t->_period) {\n                    t->readd_periodic();\n                }\n                try {\n                    *internal::current_scheduling_group_ptr() = t->_sg;\n                    t->_callback();\n                } catch (...) {\n                    internal::log_timer_callback_exception(std::current_exception());\n                }\n            }\n        }\n        // complete_timers() can be called from the context of run_tasks()\n        // as well so we need to restore the previous scheduling group (set by run_tasks()).\n        *internal::current_scheduling_group_ptr() = prev_sg;\n    }\n\n    /**\n     * Returns a time point at which expire() should be called\n     * in order to ensure timers are expired in a timely manner.\n     *\n     * Returned values are monotonically increasing.\n     */\n    time_point get_next_timeout() const noexcept\n    {\n        return time_point(duration(std::max(_last, _next)));\n    }\n\n    /**\n     * Clears both active and expired timer sets.\n     */\n    void clear() noexcept\n    {\n        for (int i : bitsets::for_each_set(_non_empty_buckets)) {\n            _buckets[i].clear();\n        }\n    }\n\n    size_t size() const noexcept\n    {\n        size_t res = 0;\n        for (int i : bitsets::for_each_set(_non_empty_buckets)) {\n            res += _buckets[i].size();\n        }\n        return res;\n    }\n\n    /**\n     * Returns true if and only if there are no timers in the active set.\n     */\n    bool empty() const noexcept\n    {\n        return _non_empty_buckets.none();\n    }\n\n    time_point now() noexcept {\n        return Timer::clock::now();\n    }\n};\n}\n"
  },
  {
    "path": "include/seastar/core/timer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/timer-set.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <boost/intrusive/list.hpp>\n#include <chrono>\n#include <optional>\n\n/// \\file\n\n/// \\defgroup timers Timers\n///\n/// Seastar provides timers that can be defined to run a callback at a certain\n/// time point in the future; timers are provided for \\ref lowres_clock (10ms\n/// resolution, efficient), for std::chrono::steady_clock (accurate but less\n/// efficient) and for \\ref manual_clock (for testing purposes).\n///\n/// Timers are optimized for cancellation; that is, adding a timer and cancelling\n/// it is very efficient. This means that attaching a timer per object for\n/// a timeout that rarely happens is reasonable; one does not have to maintain\n/// a single timer and a sorted list for this use case.\n///\n/// Timer callbacks should be short and execute quickly. If involved processing\n/// is required, a timer can launch a continuation.\n\nnamespace seastar {\n\n\nusing steady_clock_type = std::chrono::steady_clock;\n\n\n/// \\addtogroup timers\n/// @{\n\n/// Timer - run a callback at a certain time point in the future.\n///\n/// Timer callbacks should execute quickly. If more involved computation\n/// is required, the timer should launch it as a fiber (or signal an\n/// existing fiber to continue execution). Fibers launched from a timer\n/// callback are executed under the scheduling group that was current\n/// when the timer was created (see current_scheduling_group()), or the\n/// scheduling that was given explicitly by the caller when the callback\n/// was specified.\n///\n/// Expiration of a `timer<std::chrono::steady_clock>` is independent of\n/// task_quota, so it has relatively high accuracy, but as a result this\n/// is a relatively expensive timer. It is recommended to use `timer<lowres_clock>`\n/// instead, which has very coarse resolution (~10ms) but is quite efficient.\n/// It is suitable for most user timeouts.\n///\n/// \\tparam Clock type of clock used to denote time points; can be\n///        std::chrono::steady_clock_type (default), lowres_clock (more efficient\n///        but with less resolution) and manual_clock_type (fine-grained control\n///        for testing.\ntemplate <typename Clock = steady_clock_type>\nclass timer {\npublic:\n    typedef typename Clock::time_point time_point;\n    typedef typename Clock::duration duration;\n    typedef Clock clock;\nprivate:\n    using callback_t = noncopyable_function<void()>;\n    boost::intrusive::list_member_hook<> _link;\n    scheduling_group _sg;\n    callback_t _callback;\n    time_point _expiry;\n    std::optional<duration> _period;\n    bool _armed = false;\n    bool _queued = false;\n    bool _expired = false;\n    void readd_periodic() noexcept;\n    void arm_state(time_point until, std::optional<duration> period) noexcept {\n        SEASTAR_ASSERT(!_armed);\n        _period = period;\n        _armed = true;\n        _expired = false;\n        _expiry = until;\n        _queued = true;\n    }\npublic:\n    /// Constructs a timer with no callback set and no expiration time.\n    timer() noexcept {};  // implementation is required (instead of = default) for noexcept due to a bug in gcc 9.3.1,\n                          // since boost::intrusive::list_member_hook default constructor is not specified as noexcept.\n    /// Constructs a timer from another timer that is moved from.\n    ///\n    /// \\note care should be taken when moving a timer whose callback captures `this`,\n    ///       since the object pointed to by `this` may have been moved as well.\n    timer(timer&& t) noexcept : _sg(t._sg), _callback(std::move(t._callback)), _expiry(std::move(t._expiry)), _period(std::move(t._period)),\n            _armed(t._armed), _queued(t._queued), _expired(t._expired) {\n        _link.swap_nodes(t._link);\n        t._queued = false;\n        t._armed = false;\n    }\n    /// Constructs a timer with a callback. The timer is not armed.\n    ///\n    /// \\param sg Scheduling group to run the callback under.\n    /// \\param callback function (with signature `void ()`) to execute after the timer is armed and expired.\n    timer(scheduling_group sg, noncopyable_function<void ()>&& callback) noexcept : _sg(sg), _callback{std::move(callback)} {\n    }\n    /// Constructs a timer with a callback. The timer is not armed.\n    ///\n    /// \\param callback function (with signature `void ()`) to execute after the timer is armed and expired.\n    explicit timer(noncopyable_function<void ()>&& callback) noexcept : timer(current_scheduling_group(), std::move(callback)) {\n    }\n    /// Destroys the timer. The timer is cancelled if armed.\n    ~timer();\n    /// Sets the callback function to be called when the timer expires.\n    ///\n    /// \\param sg the scheduling group under which the callback will be executed.\n    /// \\param callback the callback to be executed when the timer expires.\n    void set_callback(scheduling_group sg, noncopyable_function<void ()>&& callback) noexcept {\n        _sg = sg;\n        _callback = std::move(callback);\n    }\n    /// Sets the callback function to be called when the timer expires.\n    ///\n    /// \\param callback the callback to be executed when the timer expires.\n    void set_callback(noncopyable_function<void ()>&& callback) noexcept {\n        set_callback(current_scheduling_group(), std::move(callback));\n    }\n    /// Sets the timer expiration time.\n    ///\n    /// It is illegal to arm a timer that has already been armed (and\n    /// not disarmed by expiration or cancel()). In the current\n    /// implementation, this will result in an assertion failure. See\n    /// rearm().\n    ///\n    /// \\param until the time when the timer expires\n    /// \\param period optional automatic rearm duration; if given the timer\n    ///        will automatically rearm itself when it expires, using the period\n    ///        to calculate the next expiration time.\n    void arm(time_point until, std::optional<duration> period = {}) noexcept;\n    /// Sets the timer expiration time. If the timer was already armed, it is\n    /// canceled first.\n    ///\n    /// \\param until the time when the timer expires\n    /// \\param period optional automatic rearm duration; if given the timer\n    ///        will automatically rearm itself when it expires, using the period\n    ///        to calculate the next expiration time.\n    void rearm(time_point until, std::optional<duration> period = {}) noexcept {\n        if (_armed) {\n            cancel();\n        }\n        arm(until, period);\n    }\n    /// Sets the timer expiration time.\n    ///\n    /// It is illegal to arm a timer that has already been armed (and\n    /// not disarmed by expiration or cancel()). In the current\n    /// implementation, this will result in an assertion failure. See\n    /// rearm().\n    ///\n    /// \\param delta the time when the timer expires, relative to now\n    void arm(duration delta) noexcept {\n        return arm(Clock::now() + delta);\n    }\n    /// Sets the timer expiration time, with automatic rearming\n    ///\n    /// \\param delta the time when the timer expires, relative to now. The timer\n    ///        will also rearm automatically using the same delta time.\n    void arm_periodic(duration delta) noexcept {\n        arm(Clock::now() + delta, {delta});\n    }\n    /// Sets the timer expiration time, with automatic rearming.\n    /// If the timer was already armed, it is canceled first.\n    ///\n    /// \\param delta the time when the timer expires, relative to now. The timer\n    ///        will also rearm automatically using the same delta time.\n    void rearm_periodic(duration delta) noexcept {\n        if (_armed) {\n            cancel();\n        }\n        arm_periodic(delta);\n    }\n    /// Returns whether the timer is armed\n    ///\n    /// \\return `true` if the timer is armed and has not expired yet.\n    bool armed() const noexcept { return _armed; }\n    /// Cancels an armed timer.\n    ///\n    /// If the timer was armed, it is disarmed. If the timer was not\n    /// armed, does nothing.\n    ///\n    /// \\return `true` if the timer was armed before the call.\n    bool cancel() noexcept;\n    /// Gets the expiration time of an armed timer.\n    ///\n    /// \\return the time at which the timer is scheduled to expire (undefined if the\n    ///       timer is not armed).\n    time_point get_timeout() const noexcept {\n        return _expiry;\n    }\n\n    friend class timer_set<timer, &timer::_link>;\n    using set_t = timer_set<timer, &timer::_link>;\n};\n\nextern template class timer<steady_clock_type>;\n\n\n/// @}\n}\n\n"
  },
  {
    "path": "include/seastar/core/transfer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n// Helper functions for copying or moving multiple objects in an exception\n// safe manner, then destroying the sources.\n//\n// To transfer, call transfer_pass1(allocator, &from, &to) on all object pairs,\n// (this copies the object from @from to @to).  If no exceptions are encountered,\n// call transfer_pass2(allocator, &from, &to).  This destroys the object at the\n// origin.  If exceptions were encountered, simply destroy all copied objects.\n//\n// As an optimization, if the objects are moveable without throwing (noexcept)\n// transfer_pass1() simply moves the objects and destroys the source, and\n// transfer_pass2() does nothing.\n\n#include <memory>\n#include <type_traits>\n#include <utility>\n\nnamespace seastar {\n\ntemplate <typename T, typename Alloc>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nvoid\ntransfer_pass1(Alloc& a, T* from, T* to) {\n    std::allocator_traits<Alloc>::construct(a, to, std::move(*from));\n    std::allocator_traits<Alloc>::destroy(a, from);\n}\n\ntemplate <typename T, typename Alloc>\nrequires std::is_nothrow_move_constructible_v<T>\ninline\nvoid\ntransfer_pass2(Alloc&, T*, T*) {\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ntransfer_pass1(Alloc& a, T* from, T* to) {\n    std::allocator_traits<Alloc>::construct(a, to, *from);\n}\n\ntemplate <typename T, typename Alloc>\ninline\nvoid\ntransfer_pass2(Alloc& a, T* from, T*) {\n    std::allocator_traits<Alloc>::destroy(a, from);\n}\n}\n\n"
  },
  {
    "path": "include/seastar/core/unaligned.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n// The following unaligned_cast<T*>(p) is a portable replacement for\n// reinterpret_cast<T*>(p) which should be used every time address p\n// is not guaranteed to be properly aligned to alignof(T).\n//\n// On architectures like x86 and ARM, where unaligned access is allowed,\n// unaligned_cast will behave the same as reinterpret_cast and will generate\n// the same code.\n//\n// Certain architectures (e.g., MIPS) make it extremely slow or outright\n// forbidden to use ordinary machine instructions on a primitive type at an\n// unaligned addresses - e.g., access a uint32_t at an address which is not\n// a multiple of 4. Gcc's \"undefined behavior sanitizer\" (enabled in our debug\n// build) also catches such unaligned accesses and reports them as errors,\n// even when running on x86.\n//\n// Therefore, reinterpret_cast<int32_t*> on an address which is not guaranteed\n// to be a multiple of 4 may generate extremely slow code or runtime errors,\n// and must be avoided. The compiler needs to be told about the unaligned\n// access, so it can generate reasonably-efficient code for the access\n// (in MIPS, this means generating two instructions \"lwl\" and \"lwr\", instead\n// of the one instruction \"lw\" which faults on unaligned/ access). The way to\n// tell the compiler this is with __attribute__((packed)). This will also\n// cause the sanitizer not to generate runtime alignment checks for this\n// access.\n\n#include <type_traits>\n\nnamespace seastar {\n\ntemplate <typename T>\nstruct unaligned {\n    // This is made to support only simple types, so it is fine to\n    // require them to be trivially copy constructible.\n    static_assert(std::is_trivially_copy_constructible_v<T>);\n    T raw;\n    unaligned() noexcept = default;\n    unaligned(T x) noexcept : raw(x) {}\n    unaligned& operator=(const T& x) noexcept { raw = x; return *this; }\n    operator T() const noexcept { return raw; }\n} __attribute__((packed));\n\n\ntemplate <typename T, typename F>\n[[deprecated(\"violates strict aliasing rules. See issue #165.\")]]\ninline auto unaligned_cast(F* p) noexcept {\n    return reinterpret_cast<unaligned<std::remove_pointer_t<T>>*>(p);\n}\n\ntemplate <typename T, typename F>\n[[deprecated(\"violates strict aliasing rules. See issue #165.\")]]\ninline auto unaligned_cast(const F* p) noexcept {\n    return reinterpret_cast<const unaligned<std::remove_pointer_t<T>>*>(p);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/core/units.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cstddef>\n\nnamespace seastar {\n\n\nconstexpr size_t KB = 1 << 10;\nconstexpr size_t MB = 1 << 20;\nconstexpr size_t GB = 1 << 30;\n\nconstexpr size_t operator\"\"_KiB(unsigned long long n) { return n << 10; }\nconstexpr size_t operator\"\"_MiB(unsigned long long n) { return n << 20; }\nconstexpr size_t operator\"\"_GiB(unsigned long long n) { return n << 30; }\nconstexpr size_t operator\"\"_TiB(unsigned long long n) { return n << 40; }\n\n}\n"
  },
  {
    "path": "include/seastar/core/vector-data-sink.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\n\nclass\n#if SEASTAR_API_LEVEL >= 9\n[[deprecated(\"Use util::memory_data_sink\")]]\n#endif\nvector_data_sink final : public data_sink_impl {\npublic:\n    using vector_type = std::vector<net::packet>;\nprivate:\n    vector_type& _v;\npublic:\n    vector_data_sink(vector_type& v) : _v(v) {}\n\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        net::packet p = net::packet(bufs);\n        _v.push_back(std::move(p));\n        return make_ready_future<>();\n    }\n#else\n    virtual future<> put(net::packet p) override {\n        _v.push_back(std::move(p));\n        return make_ready_future<>();\n    }\n#endif\n\n    virtual future<> close() override {\n        // TODO: close on local side\n        return make_ready_future<>();\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/core/weak_ptr.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <utility>\n#include <boost/intrusive/list.hpp>\n\nnamespace seastar {\n\n/// A non-owning reference to an object.\n///\n/// weak_ptr allows one to keep a non-owning reference to an object. When the\n/// object is destroyed, it notifies all weak_ptr instances pointing to it.\n/// A weak_ptr instance pointing to a destroyed object is equivalent to a\n/// `nullptr`.\n///\n/// The referenced object must inherit from weakly_referencable.\n/// weak_ptr instances can only be obtained by calling weak_from_this() on\n/// the to-be-referenced object.\n///\n/// \\see weakly_referencable\ntemplate<typename T>\nclass weak_ptr {\n    template<typename U>\n    friend class weakly_referencable;\n    template <typename U>\n    friend class weak_ptr;\nprivate:\n    using hook_type = boost::intrusive::list_member_hook<boost::intrusive::link_mode<boost::intrusive::auto_unlink>>;\n    hook_type _hook;\n    T* _ptr = nullptr;\n    weak_ptr(T* p) noexcept : _ptr(p) {}\n    void clear() noexcept {\n        _hook = {};\n        _ptr = nullptr;\n    }\n    void swap(weak_ptr&& o) noexcept {\n        _hook.swap_nodes(o._hook);\n        std::swap(_ptr, o._ptr);\n    }\n\npublic:\n    template <typename U>\n    requires std::convertible_to<U*, T*>\n    weak_ptr(weak_ptr<U>&& o)\n    {\n        if (o._ptr) {\n            _ptr = std::exchange(o._ptr, nullptr);\n            _hook.swap_nodes(o._hook);\n        }\n    }\n\n    // Note: The default constructor's body is implemented as no-op\n    // rather than `noexcept = default` due to a bug with gcc 9.3.1\n    // that deletes the constructor since boost::intrusive::list_member_hook\n    // is not default_nothrow_constructible.\n    weak_ptr() noexcept {}\n    weak_ptr(std::nullptr_t) noexcept : weak_ptr() {}\n    weak_ptr(weak_ptr&& o) noexcept\n    {\n        swap(std::move(o));\n    }\n    weak_ptr(const weak_ptr& o) noexcept {\n        if (o._ptr) {\n            swap(o._ptr->weak_from_this());\n        }\n    }\n    weak_ptr& operator=(weak_ptr&& o) noexcept {\n        if (this != &o) {\n            clear();\n            swap(std::move(o));\n        }\n        return *this;\n    }\n    weak_ptr& operator=(const weak_ptr& o) noexcept {\n        if (this != &o) {\n            clear();\n            if (o._ptr) {\n                swap(o._ptr->weak_from_this());\n            }\n        }\n        return *this;\n    }\n    explicit operator bool() const noexcept { return _ptr != nullptr; }\n    T* operator->() const noexcept { return _ptr; }\n    T& operator*() const noexcept { return *_ptr; }\n    T* get() const noexcept { return _ptr; }\n    bool operator==(const weak_ptr& o) const noexcept { return _ptr == o._ptr; }\n    bool operator!=(const weak_ptr& o) const noexcept { return _ptr != o._ptr; }\n};\n\n/// Allows obtaining a non-owning reference (weak_ptr) to the object.\n///\n/// A live weak_ptr object doesn't prevent the referenced object form being destroyed.\n///\n/// The underlying pointer held by weak_ptr is valid as long as the referenced object is alive.\n/// When the object dies, all weak_ptr objects associated with it are emptied.\n///\n/// A weak reference is obtained like this:\n///\n///     class X : public weakly_referencable<X> {};\n///     auto x = std::make_unique<X>();\n///     weak_ptr<X> ptr = x->weak_from_this();\n///\n/// The user of weak_ptr can check if it still holds a valid pointer like this:\n///\n///     if (ptr) ptr->do_something();\n///\ntemplate<typename T>\nclass weakly_referencable {\n    boost::intrusive::list<weak_ptr<T>,\n            boost::intrusive::member_hook<weak_ptr<T>, typename weak_ptr<T>::hook_type, &weak_ptr<T>::_hook>,\n            boost::intrusive::constant_time_size<false>> _ptr_list;\npublic:\n    // Note: The default constructor's body is implemented as no-op\n    // rather than `noexcept = default` due to a bug with gcc 9.3.1\n    // that deletes the constructor since boost::intrusive::member_hook\n    // is not default_nothrow_constructible.\n    weakly_referencable() noexcept {}\n    weakly_referencable(weakly_referencable&&) = delete; // pointer to this is captured and passed to weak_ptr\n    weakly_referencable(const weakly_referencable&) = delete;\n    ~weakly_referencable() noexcept {\n        _ptr_list.clear_and_dispose([] (weak_ptr<T>* wp) noexcept {\n            wp->_ptr = nullptr;\n        });\n    }\n    weak_ptr<T> weak_from_this() noexcept {\n        weak_ptr<T> ptr(static_cast<T*>(this));\n        _ptr_list.push_back(ptr);\n        return ptr;\n    }\n\n    weak_ptr<const T> weak_from_this() const noexcept {\n        return const_cast<weakly_referencable*>(this)->weak_from_this();\n    }\n};\n\n}\n\n"
  },
  {
    "path": "include/seastar/core/when_all.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/tuple_utils.hh>\n#include <seastar/util/critical_alloc_section.hh>\n#include <cstddef>\n#include <concepts>\n#include <exception>\n#include <tuple>\n#include <type_traits>\n#include <utility>\n#include <vector>\n\nnamespace seastar {\n\n/// \\addtogroup future-util\n/// @{\n\nnamespace internal {\n\ntemplate<typename... Futures>\nstruct identity_futures_tuple {\n    using future_type = future<std::tuple<Futures...>>;\n    using promise_type = typename future_type::promise_type;\n\n    static void set_promise(promise_type& p, std::tuple<Futures...> futures) {\n        p.set_value(std::move(futures));\n    }\n\n    static future_type make_ready_future(std::tuple<Futures...> futures) noexcept {\n        return seastar::make_ready_future<std::tuple<Futures...>>(std::move(futures));\n    }\n\n    static future_type current_exception_as_future() noexcept {\n        return seastar::current_exception_as_future<std::tuple<Futures...>>();\n    }\n};\n\nclass when_all_state_base;\n\n// If the future is ready, return true\n// if the future is not ready, chain a continuation to it, and return false\nusing when_all_process_element_func = bool (*)(void* future, void* continuation, when_all_state_base* wasb) noexcept;\n\nstruct when_all_process_element {\n    when_all_process_element_func func;\n    void* future;\n};\n\nclass when_all_state_base {\n    size_t _nr_remain;\n    const when_all_process_element* _processors;\n    void* _continuation;\npublic:\n    virtual ~when_all_state_base() {}\n    when_all_state_base(size_t nr_remain, const when_all_process_element* processors, void* continuation)\n            : _nr_remain(nr_remain), _processors(processors), _continuation(continuation) {\n    }\n    virtual task* waiting_task() = 0;\n    void complete_one() noexcept {\n        // We complete in reverse order; if the futures happen to complete\n        // in order, then waiting for the last one will find the rest ready\n        --_nr_remain;\n        while (_nr_remain) {\n            bool ready = process_one(_nr_remain - 1);\n            if (!ready) {\n                return;\n            }\n            --_nr_remain;\n        }\n        if (!_nr_remain) {\n            delete this;\n        }\n    }\n    void do_wait_all() noexcept {\n        ++_nr_remain; // fake pending completion for complete_one()\n        complete_one();\n    }\n    bool process_one(size_t idx) noexcept {\n        auto p = _processors[idx];\n        return p.func(p.future, _continuation, this);\n    }\n};\n\ntemplate <typename Future>\nclass when_all_state_component final : public continuation_base_from_future_t<Future> {\n    when_all_state_base* _base;\n    Future* _final_resting_place;\npublic:\n    static bool process_element_func(void* future, void* continuation, when_all_state_base* wasb) noexcept {\n        auto f = reinterpret_cast<Future*>(future);\n        if (f->available()) {\n            return true;\n        } else {\n            auto c = new (continuation) when_all_state_component(wasb, f);\n            set_callback(std::move(*f), c);\n            return false;\n        }\n    }\n    when_all_state_component(when_all_state_base *base, Future* future) noexcept : _base(base), _final_resting_place(future) {}\n    task* waiting_task() noexcept override { return _base->waiting_task(); }\n    virtual void run_and_dispose() noexcept override {\n        using futurator = futurize<Future>;\n        if (__builtin_expect(this->_state.failed(), false)) {\n            *_final_resting_place = futurator::make_exception_future(std::move(this->_state).get_exception());\n        } else {\n            *_final_resting_place = futurator::from_tuple(std::move(this->_state).get_value());\n        }\n        auto base = _base;\n        this->~when_all_state_component();\n        base->complete_one();\n    }\n};\n\ntemplate<typename ResolvedTupleTransform, typename... Futures>\nclass when_all_state : public when_all_state_base {\n    static constexpr size_t nr = sizeof...(Futures);\n    using type = std::tuple<Futures...>;\n    type tuple;\n    // We only schedule one continuation at a time, and store it in _cont.\n    // This way, if while the future we wait for completes, some other futures\n    // also complete, we won't need to schedule continuations for them.\n    alignas(when_all_state_component<Futures>...) std::byte _cont[std::max({sizeof(when_all_state_component<Futures>)...})];\n    when_all_process_element _processors[nr];\npublic:\n    typename ResolvedTupleTransform::promise_type p;\n    when_all_state(Futures&&... t) : when_all_state_base(nr, _processors, &_cont), tuple(std::make_tuple(std::move(t)...)) {\n        init_element_processors(std::make_index_sequence<nr>());\n    }\n    virtual ~when_all_state() {\n        ResolvedTupleTransform::set_promise(p, std::move(tuple));\n    }\n    task* waiting_task() noexcept override {\n        return p.waiting_task();\n    }\nprivate:\n    template <size_t... Idx>\n    void init_element_processors(std::index_sequence<Idx...>) {\n        auto ignore = {\n        0,\n            (_processors[Idx] = when_all_process_element{\n                when_all_state_component<std::tuple_element_t<Idx, type>>::process_element_func,\n                &std::get<Idx>(tuple)\n         }, 0)...\n        };\n        (void)ignore;\n    }\npublic:\n    static typename ResolvedTupleTransform::future_type wait_all(Futures&&... futures) noexcept {\n        if ((futures.available() && ...)) {\n            return ResolvedTupleTransform::make_ready_future(std::make_tuple(std::move(futures)...));\n        }\n        auto state = [&] () noexcept {\n            memory::scoped_critical_alloc_section _;\n            return new when_all_state(std::move(futures)...);\n        }();\n        auto ret = state->p.get_future();\n        state->do_wait_all();\n        return ret;\n    }\n};\n\n} // namespace internal\n\n/// \\cond internal\n\nnamespace impl {\n\n// Want: folds\n\ntemplate <typename T>\nstruct is_tuple_of_futures : std::false_type {\n};\n\ntemplate <>\nstruct is_tuple_of_futures<std::tuple<>> : std::true_type {\n};\n\ntemplate <typename... T, typename... Rest>\nstruct is_tuple_of_futures<std::tuple<future<T...>, Rest...>> : is_tuple_of_futures<std::tuple<Rest...>> {\n};\n\n}\n\ntemplate <typename... Futs>\nconcept AllAreFutures = impl::is_tuple_of_futures<std::tuple<Futs...>>::value;\n\ntemplate<Future Fut>\nauto futurize_invoke_if_func(Fut&& fut) noexcept {\n    return std::forward<Fut>(fut);\n}\n\ntemplate<std::invocable Func>\nauto futurize_invoke_if_func(Func&& func) noexcept {\n    return futurize_invoke(std::forward<Func>(func));\n}\n/// \\endcond\n\nnamespace internal {\n\ntemplate <typename... Futs>\nrequires seastar::AllAreFutures<Futs...>\ninline\nfuture<std::tuple<Futs...>>\nwhen_all_impl(Futs&&... futs) noexcept {\n    using state = when_all_state<identity_futures_tuple<Futs...>, Futs...>;\n    return state::wait_all(std::forward<Futs>(futs)...);\n}\n\n} // namespace internal\n\n/// Wait for many futures to complete, capturing possible errors (variadic version).\n///\n/// Each future can be passed directly, or a function that returns a\n/// future can be given instead.\n///\n/// If any function throws, an exceptional future is created for it.\n///\n/// Returns a tuple of futures so individual values or exceptions can be\n/// examined.\n///\n/// \\param fut_or_funcs futures or functions that return futures\n/// \\return an \\c std::tuple<> of all futures returned; when ready,\n///         all contained futures will be ready as well.\ntemplate <typename... FutOrFuncs>\ninline auto when_all(FutOrFuncs&&... fut_or_funcs) noexcept {\n    return internal::when_all_impl(futurize_invoke_if_func(std::forward<FutOrFuncs>(fut_or_funcs))...);\n}\n\nnamespace internal {\n\ntemplate<typename Future>\nstruct identity_futures_vector {\n    using future_type = future<std::vector<Future>>;\n    static future_type run(std::vector<Future> futures) noexcept {\n        return make_ready_future<std::vector<Future>>(std::move(futures));\n    }\n    static future_type current_exception_as_future() noexcept {\n        return seastar::current_exception_as_future<std::vector<Future>>();\n    }\n};\n\n// Internal function for when_all().\ntemplate <typename ResolvedVectorTransform, typename Future>\ninline\ntypename ResolvedVectorTransform::future_type\ncomplete_when_all(std::vector<Future>&& futures, typename std::vector<Future>::iterator pos) noexcept {\n    // If any futures are already ready, skip them.\n    while (pos != futures.end() && pos->available()) {\n        ++pos;\n    }\n    // Done?\n    if (pos == futures.end()) {\n        return ResolvedVectorTransform::run(std::move(futures));\n    }\n    // Wait for unready future, store, and continue.\n    return pos->then_wrapped([futures = std::move(futures), pos] (auto fut) mutable {\n        *pos++ = std::move(fut);\n        return complete_when_all<ResolvedVectorTransform>(std::move(futures), pos);\n    });\n}\n\ntemplate<typename ResolvedVectorTransform, typename FutureIterator, typename Sentinel>\ninline auto\ndo_when_all(FutureIterator begin, Sentinel end) noexcept {\n    using itraits = std::iterator_traits<FutureIterator>;\n    auto make_values_vector = [] (size_t size) noexcept {\n        memory::scoped_critical_alloc_section _;\n        std::vector<typename itraits::value_type> ret;\n        ret.reserve(size);\n        return ret;\n    };\n    std::vector<typename itraits::value_type> ret =\n            make_values_vector(iterator_range_estimate_vector_capacity(begin, end));\n    // Important to invoke the *begin here, in case it's a function iterator,\n    // so we launch all computation in parallel.\n    std::ranges::move(begin, end, std::back_inserter(ret));\n    return complete_when_all<ResolvedVectorTransform>(std::move(ret), ret.begin());\n}\n\n} // namespace internal\n\n/// Wait for many futures to complete, capturing possible errors (iterator version).\n///\n/// Given a range of futures as input, wait for all of them\n/// to resolve (either successfully or with an exception), and return\n/// them as a \\c std::vector so individual values or exceptions can be examined.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range of futures\n/// \\param end an \\c InputIterator designating the end of the range of futures\n/// \\return an \\c std::vector<> of all the futures in the input; when\n///         ready, all contained futures will be ready as well.\ntemplate <typename FutureIterator>\nrequires requires (FutureIterator i) { { *i++ }; requires is_future<std::remove_reference_t<decltype(*i)>>::value; }\ninline\nfuture<std::vector<typename std::iterator_traits<FutureIterator>::value_type>>\nwhen_all(FutureIterator begin, FutureIterator end) noexcept {\n    namespace si = internal;\n    using itraits = std::iterator_traits<FutureIterator>;\n    using result_transform = si::identity_futures_vector<typename itraits::value_type>;\n    try {\n        return si::do_when_all<result_transform>(std::move(begin), std::move(end));\n    } catch (...) {\n        return result_transform::current_exception_as_future();\n    }\n}\n\nnamespace internal {\n\ntemplate<typename Future>\nstruct future_has_value {\n    enum {\n        value = !std::is_same_v<std::decay_t<Future>, future<>>\n    };\n};\n\ntemplate<typename Tuple>\nstruct tuple_to_future;\n\ntemplate<typename... Elements>\nstruct tuple_to_future<std::tuple<Elements...>> {\n    using value_type = std::tuple<Elements...>;\n    using type = future<value_type>;\n    using promise_type = promise<value_type>;\n\n    // Elements... all come from futures, so we know they are nothrow move\n    // constructible. `future` also has a static assertion to that effect.\n\n    static auto make_ready(std::tuple<Elements...> t) noexcept {\n        return make_ready_future<value_type>(value_type(std::move(t)));\n    }\n\n    static auto make_failed(std::exception_ptr excp) noexcept {\n        return seastar::make_exception_future<value_type>(std::move(excp));\n    }\n};\n\ntemplate<typename... Futures>\nclass extract_values_from_futures_tuple {\n    static auto transform(std::tuple<Futures...> futures) noexcept {\n        auto prepare_result = [] (auto futures) noexcept {\n            auto fs = tuple_filter_by_type<internal::future_has_value>(std::move(futures));\n            return tuple_map(std::move(fs), [] (auto&& e) {\n                return e.get();\n            });\n        };\n\n        using tuple_futurizer = internal::tuple_to_future<decltype(prepare_result(std::move(futures)))>;\n\n        std::exception_ptr excp;\n        tuple_for_each(futures, [&excp] (auto& f) {\n            if (!excp) {\n                if (f.failed()) {\n                    excp = f.get_exception();\n                }\n            } else {\n                f.ignore_ready_future();\n            }\n        });\n        if (excp) {\n            return tuple_futurizer::make_failed(std::move(excp));\n        }\n\n        return tuple_futurizer::make_ready(prepare_result(std::move(futures)));\n    }\npublic:\n    using future_type = decltype(transform(std::declval<std::tuple<Futures...>>()));\n    using promise_type = typename future_type::promise_type;\n\n    static void set_promise(promise_type& p, std::tuple<Futures...> tuple) {\n        transform(std::move(tuple)).forward_to(std::move(p));\n    }\n\n    static future_type make_ready_future(std::tuple<Futures...> tuple) noexcept {\n        return transform(std::move(tuple));\n    }\n\n    static future_type current_exception_as_future() noexcept {\n        future_type (*type_deduct)() = current_exception_as_future;\n        return type_deduct();\n    }\n};\n\ntemplate<typename Future>\nstruct extract_values_from_futures_vector {\n    using value_type = decltype(untuple(std::declval<typename Future::tuple_type>()));\n\n    using future_type = future<std::vector<value_type>>;\n\n    static future_type run(std::vector<Future> futures) noexcept {\n        auto make_values_vector = [] (size_t size) noexcept {\n            memory::scoped_critical_alloc_section _;\n            std::vector<value_type> values;\n            values.reserve(size);\n            return values;\n        };\n        std::vector<value_type> values = make_values_vector(futures.size());\n\n        std::exception_ptr excp;\n        for (auto&& f : futures) {\n            if (!excp) {\n                if (f.failed()) {\n                    excp = f.get_exception();\n                } else {\n                    values.emplace_back(f.get());\n                }\n            } else {\n                f.ignore_ready_future();\n            }\n        }\n        if (excp) {\n            return seastar::make_exception_future<std::vector<value_type>>(std::move(excp));\n        }\n        return make_ready_future<std::vector<value_type>>(std::move(values));\n    }\n\n    static future_type current_exception_as_future() noexcept {\n        return seastar::current_exception_as_future<std::vector<value_type>>();\n    }\n};\n\ntemplate<>\nstruct extract_values_from_futures_vector<future<>> {\n    using future_type = future<>;\n\n    static future_type run(std::vector<future<>> futures) noexcept {\n        std::exception_ptr excp;\n        for (auto&& f : futures) {\n            if (!excp) {\n                if (f.failed()) {\n                    excp = f.get_exception();\n                }\n            } else {\n                f.ignore_ready_future();\n            }\n        }\n        if (excp) {\n            return seastar::make_exception_future<>(std::move(excp));\n        }\n        return make_ready_future<>();\n    }\n\n    static future_type current_exception_as_future() noexcept {\n        return seastar::current_exception_as_future<>();\n    }\n};\n\ntemplate<typename... Futures>\nrequires seastar::AllAreFutures<Futures...>\ninline auto when_all_succeed_impl(Futures&&... futures) noexcept {\n    using state = when_all_state<extract_values_from_futures_tuple<Futures...>, Futures...>;\n    return state::wait_all(std::forward<Futures>(futures)...);\n}\n\n} // namespace internal\n\n/// Wait for many futures to complete (variadic version).\n///\n/// Each future can be passed directly, or a function that returns a\n/// future can be given instead.\n///\n/// If any function throws, or if the returned future fails, one of\n/// the exceptions is returned by this function as a failed future.\n///\n/// \\param fut_or_funcs futures or functions that return futures\n/// \\return future containing values of futures returned by funcs\ntemplate <typename... FutOrFuncs>\ninline auto when_all_succeed(FutOrFuncs&&... fut_or_funcs) noexcept {\n    return internal::when_all_succeed_impl(futurize_invoke_if_func(std::forward<FutOrFuncs>(fut_or_funcs))...);\n}\n\n/// Wait for many futures to complete (iterator version).\n///\n/// Given a range of futures as input, wait for all of them\n/// to resolve, and return a future containing a vector of values of the\n/// original futures.\n/// In case any of the given futures fails one of the exceptions is returned\n/// by this function as a failed future.\n/// \\param begin an \\c InputIterator designating the beginning of the range of futures\n/// \\param end an \\c InputIterator designating the end of the range of futures\n/// \\return an \\c std::vector<> of all the valus in the input\ntemplate <typename FutureIterator, typename Sentinel, typename = typename std::iterator_traits<FutureIterator>::value_type>\nrequires requires (FutureIterator i) {\n     *i++;\n     { i != i } -> std::convertible_to<bool>;\n     requires is_future<std::remove_reference_t<decltype(*i)>>::value;\n}\ninline auto\nwhen_all_succeed(FutureIterator begin, Sentinel end) noexcept {\n    using itraits = std::iterator_traits<FutureIterator>;\n    using result_transform = internal::extract_values_from_futures_vector<typename itraits::value_type>;\n    try {\n        return internal::do_when_all<result_transform>(std::move(begin), std::move(end));\n    } catch (...) {\n        return result_transform::current_exception_as_future();\n    }\n}\n\n\n/// Wait for many futures to complete (vector version).\n///\n/// Given a vector of futures as input, wait for all of them\n/// to resolve, and return a future containing a vector of values of the\n/// original futures.\n///\n/// In case any of the given futures fails one of the exceptions is returned\n/// by this function as a failed future.\n///\n/// \\param futures a \\c std::vector containing the futures to wait for.\n/// \\return an \\c std::vector<> of all the values in the input\ntemplate <typename T>\ninline auto\nwhen_all_succeed(std::vector<future<T>>&& futures) noexcept {\n    using result_transform = internal::extract_values_from_futures_vector<future<T>>;\n    try {\n        return internal::complete_when_all<result_transform>(std::move(futures), futures.begin());\n    } catch (...) {\n        return result_transform::current_exception_as_future();\n    }\n}\n\n/// @}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/when_any.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * author: Niek J Bouman\n * reviewers: Avi Kivity, Benny Halevy\n * November 2021\n */\n\n#pragma once\n\n#include <iterator>\n#include <cstddef>\n#include <type_traits>\n#include <vector>\n#include <tuple>\n#include <utility>\n#include <seastar/core/future.hh>\n#include <seastar/core/shared_ptr.hh>\n\nnamespace seastar {\n\ntemplate <class Sequence>\nstruct when_any_result {\n    std::size_t index;\n    Sequence futures;\n};\n\nnamespace internal {\nclass waiter {\n    bool _done = false;\n    promise<std::size_t> _promise;\n\npublic:\n    void done(std::size_t index) {\n        if (!_done) {\n            _done = true;\n            _promise.set_value(index);\n        }\n    }\n    auto get_future() { return _promise.get_future(); }\n};\n\n} // namespace internal\n\n/// Wait for the first of multiple futures to complete (iterator version).\n///\n/// Given a range of futures as input, wait for the first of them\n/// to resolve (either successfully or with an exception), and return\n/// all of them in a \\c when_any_result (following the concurrency TS from\n/// the standard library), containing a std::vector to all futures\n/// and the index (into the vector) of the future that resolved.\n///\n/// \\param begin an \\c InputIterator designating the beginning of the range of futures\n/// \\param end an \\c InputIterator designating the end of the range of futures\n/// \\return a \\c when_any_result of all the futures in the input; when\n///         ready, at least one of the contained futures (the one indicated by index) will be ready.\ntemplate <class FutureIterator>\nrequires requires (FutureIterator i) { { *i++ }; requires is_future<std::remove_reference_t<decltype(*i)>>::value; }\nauto when_any(FutureIterator begin, FutureIterator end) noexcept\n  -> future<when_any_result<std::vector<std::decay_t<typename std::iterator_traits<FutureIterator>::value_type>>>>\n{\n    using ReturnType = when_any_result<std::vector<typename std::iterator_traits<FutureIterator>::value_type>>;\n    if (begin == end) {\n        return make_ready_future<ReturnType>();\n    }\n    ReturnType result;\n    result.futures.reserve(std::distance(begin, end));\n    auto waiter_obj = make_lw_shared<internal::waiter>();\n    std::size_t index{0};\n    for (auto it = begin; it != end; ++it) {\n        if (it->available()) {\n            result.futures.push_back(std::move(*it));\n            waiter_obj->done(index);\n        } else {\n            result.futures.push_back(it->finally([waiter_obj, index] {\n                waiter_obj->done(index);\n            }));\n        }\n        index++;\n    }\n    return waiter_obj->get_future().then(\n        [result = std::move(result)](std::size_t index) mutable {\n            result.index = index;\n            return std::move(result);\n        }\n    );\n}\n\nnamespace internal {\n\ntemplate <class... Futures, std::size_t... I>\nfuture<when_any_result<std::tuple<Futures...>>>\nwhen_any_impl(std::index_sequence<I...>, Futures&&... futs) noexcept\n{\n    auto waiter_obj = make_lw_shared<waiter>();\n    auto attach_notifier = [&](auto&& fut, size_t index) {\n        if (fut.available()) {\n            waiter_obj->done(index);\n            return std::move(fut);\n        }\n        else {\n            return fut.finally([waiter_obj, index] { waiter_obj->done(index); });\n        }\n    };\n\n    auto result =\n        when_any_result<std::tuple<Futures...>>{0, std::make_tuple(attach_notifier(std::forward<Futures>(futs), I)...)};\n    return waiter_obj->get_future().then([result = std::move(result)](std::size_t index) mutable {\n        result.index = index;\n        return std::move(result);\n    });\n}\n\n} // namespace internal\n\n/// Wait for the first of multiple futures to complete (variadic version).\n///\n/// Each future can be passed directly, or a function that returns a\n/// future can be given instead.\n///\n/// Returns a \\c when_any_result (following the concurrency TS from\n/// the standard library), containing a std::tuple to all futures\n/// and the index (into the vector) of the future that resolved.\n///\n/// \\param fut_or_funcs futures or functions that return futures\n/// \\return a \\c when_any_result containing a tuple of all futures\n///  and and index; when ready, at least one of the contained futures\n///  (the one indicated by index) will be ready.\ntemplate <class... FutOrFuncs>\nauto when_any(FutOrFuncs&&... fut_or_funcs) noexcept\n{\n    return internal::when_any_impl(std::make_index_sequence<sizeof...(FutOrFuncs)>{},\n                                   futurize_invoke_if_func(std::forward<FutOrFuncs>(fut_or_funcs))...);\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/with_scheduling_group.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/make_task.hh>\n#include <tuple>\n#include <utility>\n\nnamespace seastar {\n\n/// \\addtogroup future-util\n/// @{\n\nnamespace internal {\n\ntemplate <typename Func>\nrequires std::is_nothrow_move_constructible_v<Func>\nauto\nschedule_in_group(scheduling_group sg, Func func) noexcept {\n    static_assert(std::is_nothrow_move_constructible_v<Func>);\n    auto tsk = make_task(sg, std::move(func));\n    schedule_checked(tsk);\n    return tsk->get_future();\n}\n\n\n}\n\n/// \\brief run a callable (with some arbitrary arguments) in a scheduling group\n///\n/// If the conditions are suitable (see scheduling_group::may_run_immediately()),\n/// then the function is run immediately. Otherwise, the function is queued to run\n/// when its scheduling group next runs.\n///\n/// \\param sg  scheduling group that controls execution time for the function\n/// \\param func function to run; must be movable or copyable\n/// \\param args arguments to the function; may be copied or moved, so use \\c std::ref()\n///             to force passing references\ntemplate <typename Func, typename... Args>\nrequires std::is_nothrow_move_constructible_v<Func>\ninline\nauto\nwith_scheduling_group(scheduling_group sg, Func func, Args&&... args) noexcept {\n    static_assert(std::is_nothrow_move_constructible_v<Func>);\n    using return_type = decltype(func(std::forward<Args>(args)...));\n    using futurator = futurize<return_type>;\n    if (sg.active()) {\n        return futurator::invoke(func, std::forward<Args>(args)...);\n    } else {\n        return internal::schedule_in_group(sg, [func = std::move(func), args = std::make_tuple(std::forward<Args>(args)...)] () mutable {\n            return futurator::apply(func, std::move(args));\n        });\n    }\n}\n\n/// @}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/core/with_timeout.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n\n#pragma once\n\n#include <chrono>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <seastar/core/timer.hh>\n\nnamespace seastar {\n\n/// \\addtogroup future-util\n/// @{\n\n/// \\brief Wait for either a future, or a timeout, whichever comes first\n///\n/// When timeout is reached the returned future resolves with an exception\n/// produced by ExceptionFactory::timeout(). By default it is \\ref timed_out_error exception.\n///\n/// Note that timing out doesn't cancel any tasks associated with the original future.\n/// It also doesn't cancel the callback registerred on it.\n///\n/// \\param f future to wait for\n/// \\param timeout time point after which the returned future should be failed\n///\n/// \\return a future which will be either resolved with f or a timeout exception\ntemplate<typename ExceptionFactory = default_timeout_exception_factory, typename Clock, typename Duration, typename... T>\nfuture<T...> with_timeout(std::chrono::time_point<Clock, Duration> timeout, future<T...> f) {\n    if (f.available()) {\n        return f;\n    }\n    auto pr = std::make_unique<promise<T...>>();\n    auto result = pr->get_future();\n    timer<Clock> timer([&pr = *pr] {\n        pr.set_exception(std::make_exception_ptr(ExceptionFactory::timeout()));\n    });\n    timer.arm(timeout);\n    // Future is returned indirectly.\n    (void)f.then_wrapped([pr = std::move(pr), timer = std::move(timer)] (auto&& f) mutable {\n        if (timer.cancel()) {\n            f.forward_to(std::move(*pr));\n        } else {\n            f.ignore_ready_future();\n        }\n    });\n    return result;\n}\n\n/// @}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/coroutine/all.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021-present ScyllaDB\n */\n\n#pragma once\n\n#include <cstddef>\n#include <concepts>\n#include <tuple>\n#include <seastar/core/coroutine.hh>\n\nnamespace seastar::coroutine {\n\ntemplate <typename Future>\nconstexpr inline bool is_future_v = is_future<Future>::value;\n\ntemplate <typename Future>\nconcept future_type = is_future_v<Future>;\n\nnamespace internal {\n\n// Given a bunch of futures, find the indexes of the ones that are not avoid\n// and store them in member type `type` as an std::integer_sequence.\n//\n// `IndexSequence` and `current` are intermediates used for recursion.\ntemplate <typename IndexSequence, size_t current, typename... Futures>\nstruct index_sequence_for_non_void_futures_helper;\n\n// Terminate recursion be returning the accumulated `IndexSequence`\ntemplate <typename IndexSequence, size_t current>\nstruct index_sequence_for_non_void_futures_helper<IndexSequence, current> {\n    using type = IndexSequence;\n};\n\n// Process a future<T> by adding it to the current IndexSequence and recursing\ntemplate <size_t... Existing, size_t current, typename T, typename... Futures>\nstruct index_sequence_for_non_void_futures_helper<std::integer_sequence<size_t, Existing...>, current, future<T>, Futures...> {\n    using type = typename index_sequence_for_non_void_futures_helper<std::integer_sequence<size_t, Existing..., current>, current + 1, Futures...>::type;\n};\n\n// Process a future<void> by ignoring it and recursing\ntemplate <size_t... Existing, size_t current, typename... Futures>\nstruct index_sequence_for_non_void_futures_helper<std::integer_sequence<size_t, Existing...>, current, future<>, Futures...> {\n    using type = typename index_sequence_for_non_void_futures_helper<std::integer_sequence<size_t, Existing...>, current + 1, Futures...>::type;\n};\n\n// Simple interface for the above.\ntemplate <typename... Futures>\nusing index_sequence_for_non_void_futures = typename index_sequence_for_non_void_futures_helper<std::integer_sequence<size_t>, 0, Futures...>::type;\n\n// Given a tuple of futures, return a tuple of the value types, excluding future<void>.\ntemplate <typename IndexSequence, typename FutureTuple>\nstruct value_tuple_for_non_void_futures_helper;\n\ntemplate <size_t... Idx, typename FutureTuple>\nstruct value_tuple_for_non_void_futures_helper<std::integer_sequence<size_t, Idx...>, FutureTuple> {\n    using type = std::tuple<typename std::tuple_element_t<Idx, FutureTuple>::value_type...>;\n};\n\n// Simple interface for the above\ntemplate <typename... Futures>\nusing value_tuple_for_non_void_futures = typename value_tuple_for_non_void_futures_helper<index_sequence_for_non_void_futures<Futures...>, std::tuple<Futures...>>::type;\n\n}\n\n/// Wait for serveral futures to complete in a coroutine.\n///\n/// `all` can be used to launch several computations concurrently\n/// and wait for all of them to complete. Computations are provided\n/// as callable objects (typically lambda coroutines) that are invoked\n/// by `all`. Waiting is performend by `co_await` and returns a tuple\n/// of values, one for each non-void future.\n///\n/// If one or more of the function objects throws an exception, or if one\n/// or more of the futures resolves to an exception, then the exception is\n/// thrown. All of the futures are waited for, even in the case of exceptions.\n/// If more than one exception is present, an arbitrary one is thrown.\n///\n/// Example\n///\n/// ```\n/// future<int> add() {\n///     auto [a, b] = co_await all(\n///         [] () -> future<int> {\n///             co_await sleep(1ms);\n///             co_return 2;\n///         },\n///         [] () -> future<int> {\n///             co_await sleep(1ms);\n///             co_return 3;\n///         }\n///     );\n///     co_return a + b;\n/// };\n/// ```\n///\n/// Safe for use with lambda coroutines.\ntemplate <typename... Futures>\nrequires (sizeof ...(Futures) > 0)\nclass [[nodiscard(\"must co_await an all() object\")]] all {\n    using tuple = std::tuple<Futures...>;\n    using value_tuple = typename internal::value_tuple_for_non_void_futures<Futures...>;\n    struct awaiter;\n    template <size_t idx>\n    struct intermediate_task final : continuation_base_from_future_t<std::tuple_element_t<idx, tuple>> {\n        awaiter& container;\n        explicit intermediate_task(awaiter& container) : container(container) {}\n        virtual void run_and_dispose() noexcept {\n            using value_type = typename std::tuple_element_t<idx, tuple>::value_type;\n            if (__builtin_expect(this->_state.failed(), false)) {\n                using futurator = futurize<std::tuple_element_t<idx, tuple>>;\n                std::get<idx>(container.state._futures) = futurator::make_exception_future(std::move(this->_state).get_exception());\n            } else {\n                if constexpr (std::same_as<std::tuple_element_t<idx, tuple>, future<>>) {\n                    std::get<idx>(container.state._futures) = make_ready_future<>();\n                } else {\n                    std::get<idx>(container.state._futures) = make_ready_future<value_type>(std::move(this->_state).get());\n                }\n            }\n            awaiter& c = container;\n            this->~intermediate_task();\n            c.template process<idx+1>();\n        }\n    };\n    template <typename IndexSequence>\n    struct generate_aligned_union;\n    template <size_t... idx>\n    struct generate_aligned_union<std::integer_sequence<size_t, idx...>> {\n        static constexpr std::size_t alignment_value = std::max({alignof(intermediate_task<idx>)...});\n        using type = std::byte[std::max({sizeof(intermediate_task<idx>)...})];\n    };\n    using continuation_storage = generate_aligned_union<std::make_index_sequence<std::tuple_size_v<tuple>>>;\n    using coroutine_handle_t = std::coroutine_handle<void>;\nprivate:\n    tuple _futures;\nprivate:\n    struct awaiter {\n        all& state;\n        alignas(continuation_storage::alignment_value) typename continuation_storage::type _continuation_storage;\n        coroutine_handle_t when_ready;\n        awaiter(all& state) : state(state) {}\n        bool await_ready() const {\n            return std::apply([] (const Futures&... futures) {\n                return (... && futures.available());\n            }, state._futures);\n        }\n        template <typename U>\n        void await_suspend(std::coroutine_handle<U> h SEASTAR_COROUTINE_LOC_PARAM) {\n            SEASTAR_COROUTINE_LOC_STORE(h.promise());\n            when_ready = h;\n            process<0>();\n        }\n        value_tuple await_resume() {\n            std::apply([] (Futures&... futures) {\n                std::exception_ptr e;\n                // Call get_exception for every failed future, to avoid exceptional future\n                // ignored warnings.\n                (void)(..., (futures.failed() ? (e = futures.get_exception(), 0) : 0));\n                if (e) {\n                    std::rethrow_exception(std::move(e));\n                }\n            }, state._futures);\n            // This immediately-invoked lambda is used to materialize the indexes\n            // of non-void futures in the tuple.\n            return [&] <size_t... Idx> (std::integer_sequence<size_t, Idx...>) {\n                return value_tuple(std::get<Idx>(state._futures).get()...);\n            } (internal::index_sequence_for_non_void_futures<Futures...>());\n        }\n        template <unsigned idx>\n        void process() {\n            if constexpr (idx == sizeof...(Futures)) {\n                when_ready.resume();\n            } else {\n                if (!std::get<idx>(state._futures).available()) {\n                    auto task = new (&_continuation_storage) intermediate_task<idx>(*this);\n                    seastar::internal::set_callback(std::move(std::get<idx>(state._futures)), task);\n                } else {\n                    process<idx + 1>();\n                }\n            }\n        }\n    };\npublic:\n    template <typename... Func>\n    requires (... && std::invocable<Func>) && (... && future_type<std::invoke_result_t<Func>>)\n    explicit all(Func&&... funcs)\n            : _futures(futurize_invoke(funcs)...) {\n    }\n    awaiter operator co_await() { return awaiter{*this}; }\n};\n\ntemplate <typename FirstFunc, typename... MoreFuncs>\nexplicit all(FirstFunc&&, MoreFuncs&&...) -> all<std::invoke_result_t<FirstFunc>,\n                                                 std::invoke_result_t<MoreFuncs>...>;\n\n}\n"
  },
  {
    "path": "include/seastar/coroutine/as_future.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022-present ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/coroutine.hh>\n#include <source_location>\n\nnamespace seastar {\n\nnamespace internal {\n\ntemplate <bool CheckPreempt, typename T>\nclass [[nodiscard]] as_future_awaiter {\n    seastar::future<T> _future;\n\npublic:\n    explicit as_future_awaiter(seastar::future<T>&& f) noexcept : _future(std::move(f)) {}\n\n    as_future_awaiter(const as_future_awaiter&) = delete;\n    as_future_awaiter(as_future_awaiter&&) = delete;\n\n    bool await_ready() const noexcept {\n        return _future.available() && (!CheckPreempt || !need_preempt());\n    }\n\n    template<typename U>\n    void await_suspend(std::coroutine_handle<U> hndl SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(hndl.promise());\n        if (!CheckPreempt || !_future.available()) {\n            _future.set_coroutine(hndl.promise());\n        } else {\n            schedule(&hndl.promise());\n        }\n    }\n\n    seastar::future<T> await_resume() {\n        return std::move(_future);\n    }\n};\n\n} // namespace seastar::internal\n\nnamespace coroutine {\n\n/// \\brief co_await:s a \\ref future, returning it as result.\n///\n/// Similar to \\ref seastar::future::then_wrapped, `coroutine::as_future`\n/// waits for the \\ref future to resolve either to a ready future\n/// or to an exceptional one. It then returns it as the co_await result.\n///\n/// For example:\n/// ```\n/// static future<bool> did_future_fail(future<> fut) {\n///     auto f = co_await coroutine::as_future(std::move(fut));\n///     if (f.failed()) {\n///         mylog.warn(\"Future failed: {}\", f.get_exception());\n///         co_return true;\n///     } else {\n///         co_return false;\n///     }\n/// }\n/// ```\n///\n/// Note that by default, `as_future` checks for if the task quota is depleted,\n/// which means that it will yield if the future is ready and \\ref seastar::need_preempt()\n/// returns true.  Use \\ref coroutine::as_future_without_preemption_check\n/// to disable preemption checking.\ntemplate<typename T = void>\nclass [[nodiscard]] as_future : public seastar::internal::as_future_awaiter<true, T> {\npublic:\n    explicit as_future(seastar::future<T>&& f) noexcept : seastar::internal::as_future_awaiter<true, T>(std::move(f)) {}\n};\n\n/// \\brief co_await:s a \\ref future, returning it as result, without\n/// checking if preemption is needed.\n///\n/// Like \\ref coroutine::as_future, co_await-ing as_future_without_preemption_check\n/// returns the input `future` as the co_await result.\n/// However, it bypasses checking if the task quota is depleted, which means that\n/// a ready `future` will be handled immediately.\ntemplate<typename T = void>\nclass [[nodiscard]] as_future_without_preemption_check : public seastar::internal::as_future_awaiter<false, T> {\npublic:\n    explicit as_future_without_preemption_check(seastar::future<T>&& f) noexcept : seastar::internal::as_future_awaiter<false, T>(std::move(f)) {}\n};\n\n} // namespace seastar::coroutine\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/coroutine/exception.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021-present ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <coroutine>\n#include <exception>\n#include <source_location>\n\nnamespace seastar {\n\nnamespace internal {\n\nstruct exception_awaiter {\n    std::exception_ptr eptr;\n\n    explicit exception_awaiter(std::exception_ptr&& eptr) noexcept : eptr(std::move(eptr)) {}\n\n    exception_awaiter(const exception_awaiter&) = delete;\n    exception_awaiter(exception_awaiter&&) = delete;\n\n    bool await_ready() const noexcept {\n        return false;\n    }\n\n    template<typename U>\n    void await_suspend(std::coroutine_handle<U> hndl SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n      SEASTAR_COROUTINE_LOC_STORE(hndl.promise());\n      execute_involving_handle_destruction_in_await_suspend([hndl, eptr = std::move(eptr)] () mutable {\n        hndl.promise().set_exception(std::move(eptr));\n        hndl.destroy();\n      });\n    }\n\n    void await_resume() noexcept {}\n};\n\n} // internal\n\nnamespace coroutine {\n\n/// Wrapper for propagating an exception directly rather than\n/// throwing it. The wrapper can be used with both co_await and co_return.\n///\n/// \\note It is not possible to co_return the wrapper in coroutines which\n/// return future<> due to language limitations (it's not possible to specify\n/// both return_value and return_void in the promise_type). You can use co_await\n/// instead which works in coroutines which return either future<> or future<T>.\n///\n/// Example usage:\n///\n/// ```\n/// co_await coroutine::exception(std::make_exception_ptr(std::runtime_error(\"something failed miserably\")));\n/// co_return coroutine::exception(std::make_exception_ptr(std::runtime_error(\"something failed miserably\")));\n/// ```\nstruct exception {\n    std::exception_ptr eptr;\n    explicit exception(std::exception_ptr eptr) noexcept : eptr(std::move(eptr)) {}\n};\n\n/// Allows propagating an exception from a coroutine directly rather than\n/// throwing it.\n///\n/// `make_exception()` returns an object which must be co_returned.\n/// Co_returning the object will immediately resolve the current coroutine\n/// to the given exception.\n///\n/// \\note Due to language limitations, this function doesn't work in coroutines\n/// which return future<>. Consider using return_exception instead.\n///\n/// Example usage:\n///\n/// ```\n/// co_return coroutine::make_exception(std::runtime_error(\"something failed miserably\"));\n/// ```\n[[deprecated(\"Use co_await coroutine::return_exception_ptr or co_return coroutine::exception instead\")]]\n[[nodiscard]]\ninline exception make_exception(std::exception_ptr ex) noexcept {\n    return exception(std::move(ex));\n}\n\ntemplate<typename T>\n[[deprecated(\"Use co_await coroutine::return_exception_ptr or co_return coroutine::exception instead\")]]\n[[nodiscard]]\nexception make_exception(T&& t) noexcept {\n    log_exception_trace(log_level::trace);\n    return exception(std::make_exception_ptr(std::forward<T>(t)));\n}\n\n/// Allows propagating an exception from a coroutine directly rather than\n/// throwing it.\n///\n/// `return_exception_ptr()` returns an object which must be co_awaited.\n/// Co_awaiting the object will immediately resolve the current coroutine\n/// to the given exception.\n///\n/// Example usage:\n///\n/// ```\n/// std::exception_ptr ex;\n/// try {\n///   //\n/// } catch (...) {\n///   ex = std::current_exception();\n/// }\n/// if (ex) {\n///   co_await coroutine::return_exception_ptr(std::move(ex));\n/// }\n/// ```\n[[nodiscard]]\ninline exception return_exception_ptr(std::exception_ptr ex) noexcept {\n    return exception(std::move(ex));\n}\n\n/// Allows propagating an exception from a coroutine directly rather than\n/// throwing it.\n///\n/// `return_exception()` returns an object which must be co_awaited.\n/// Co_awaiting the object will immediately resolve the current coroutine\n/// to the given exception.\n///\n/// Example usage:\n///\n/// ```\n/// co_await coroutine::return_exception(std::runtime_error(\"something failed miserably\"));\n/// ```\n[[deprecated(\"Use co_await coroutine::return_exception_ptr instead\")]]\n[[nodiscard]]\ninline exception return_exception(std::exception_ptr ex) noexcept {\n    return exception(std::move(ex));\n}\n\ntemplate<typename T>\n[[nodiscard]]\nexception return_exception(T&& t) noexcept {\n    log_exception_trace(log_level::trace);\n    return exception(std::make_exception_ptr(std::forward<T>(t)));\n}\n\ninline auto operator co_await(exception ex) noexcept {\n    return internal::exception_awaiter(std::move(ex.eptr));\n}\n\n} // coroutine\n} // seastar\n"
  },
  {
    "path": "include/seastar/coroutine/generator.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 Kefu Chai ( tchaikov@gmail.com )\n */\n\n#pragma once\n\n#include <coroutine>\n#include <exception>\n#include <iterator>\n#include <memory>\n#include <optional>\n#include <type_traits>\n#include <utility>\n#include <seastar/core/future.hh>\n#include <seastar/util/assert.hh>\n\n// seastar::coroutine::generator is inspired by the C++23 proposal\n// P2502R2 (https://wg21.link/P2502R2), which introduced std::generator for\n// synchronous coroutine-based range generation.\n//\n// Similar to P2502R2's generator, seastar::coroutine::experimental::generator\n// prioritizes storing references to yielded objects instead of copying them.\n//\n// However, there are key differences in seastar::coroutine::experimental::generator:\n//\n// * Allocator support:\n//   Seastar's generator does not support the Allocator template parameter.\n//   Seastar uses its built-in allocator eliminating the need for\n//   additional flexibility.\n// * Asynchronous Operations:\n//   - generator::operator() is a coroutine that returns std::optional<reference_type>\n//   - Unlike P2502R2's synchronous iterator-based approach, Seastar's generator\n//     uses a simpler function-call API\n//   Note: Due to its asynchronous nature, this generator cannot be used in\n//   range-based for loops. Instead, use: while (auto val = co_await gen()) { ... }\n// * Ranges Integration:\n//   Seastar's generator is not a std::ranges::view_interface. It lacks\n//   integration with the C++20 Ranges library due to its asynchronous operations.\n// * Nesting:\n//   Nesting generators is not supported. You cannot yield another generator\n//   from within a generator. This prevents implementation of asynchronous,\n//   recursive algorithms like depth-first search on trees.\n// * Range Yielding:\n//   The buffered variant supports yielding both individual elements and ranges/slices.\n//   This provides flexibility to yield data in whatever form is most convenient for\n//   the producer, while the generator handles efficient batching internally.\nnamespace seastar::coroutine::experimental {\n\nnamespace internal {\n\nnamespace unbuffered {\n\ntemplate <typename Yielded> class next_awaiter;\n\ntemplate <typename Yielded>\nclass generator_promise_base : public seastar::task {\n    using yielded_deref_type = std::remove_reference_t<Yielded>;\n    using yielded_decvref_type = std::remove_cvref_t<Yielded>;\n    using value_ptr_type = std::add_pointer_t<Yielded>;\n\nprotected:\n    // a glvalue yield expression is passed to co_yield as its operand. and\n    // the object denoted by this expression is guaranteed to live until the\n    // coroutine resumes. we take advantage of this fact by storing only a\n    // pointer to the denoted object in the promise as long as the result of\n    // dereferencing this pointer is convertible to the Ref type.\n    std::add_pointer_t<Yielded> _value = nullptr;\n\nprotected:\n    std::exception_ptr _exception;\n    std::coroutine_handle<> _consumer;\n    task* _waiting_task = nullptr;\n\n    /// awaiter returned by the generator when it produces a new element\n    ///\n    /// There are different combinations of expression types passed to\n    /// \\c co_yield and \\c Ref. In most cases, zero copies are made. Copies\n    /// are only necessary when \\c co_yield requires type conversion.\n    ///\n    /// The following table summarizes the number of copies made for different\n    /// scenarios:\n    ///\n    /// | Ref       | co_yield const T& | co_yield T& | co_yield T&& | co_yield U&& |\n    /// | --------- | ----------------- | ----------- | ------------ | ------------ |\n    /// | T         | 0                 | 0           | 0            | 1            |\n    /// | const T&  | 0                 | 0           | 0            | 1            |\n    /// | T&        | ill-formed        | 0           | ill-formed   | ill-formed   |\n    /// | T&&       | ill-formed        | ill-formed  | 0            | 1            |\n    /// | const T&& | ill-formed        | ill-formed  | 0            | 1            |\n    ///\n    /// When no copies are required, \\c yield_awaiter is used. Otherwise,\n    /// \\c copy_awaiter is used. The latter converts \\c U to \\c T, and keeps the converted\n    /// value in it.\n    struct yield_awaiter;\n    struct copy_awaiter;\n\npublic:\n    generator_promise_base() noexcept = default;\n    generator_promise_base(const generator_promise_base &) = delete;\n    generator_promise_base& operator=(const generator_promise_base &) = delete;\n    generator_promise_base(generator_promise_base &&) noexcept = default;\n    generator_promise_base& operator=(generator_promise_base &&) noexcept = default;\n    virtual ~generator_promise_base() = default;\n\n    // lazily-started coroutine, do not execute the coroutine until\n    // the coroutine is awaited.\n    std::suspend_always initial_suspend() const noexcept {\n        return {};\n    }\n\n    yield_awaiter final_suspend() noexcept {\n        _value = nullptr;\n        return {this, this->_consumer};\n    }\n\n    void unhandled_exception() noexcept {\n        _exception = std::current_exception();\n    }\n\n    yield_awaiter yield_value(Yielded&& value) noexcept {\n        this->_value = std::addressof(value);\n        return {this, this->_consumer};\n    }\n\n    copy_awaiter yield_value(const yielded_deref_type& value)\n        noexcept (std::is_nothrow_constructible_v<\n                    yielded_decvref_type,\n                    const yielded_deref_type&>)\n        requires (std::is_rvalue_reference_v<Yielded> &&\n                  std::constructible_from<\n                    yielded_decvref_type,\n                    const yielded_deref_type&>) {\n        return {this, this->_consumer, yielded_decvref_type(value), _value};\n    }\n\n    void return_void() noexcept {}\n\n    // @return if the generator has reached the end of the sequence\n    bool finished() const noexcept {\n        return _value == nullptr;\n    }\n\n    void rethrow_if_unhandled_exception() {\n        if (_exception) {\n            std::rethrow_exception(std::move(_exception));\n        }\n    }\n\n    void run_and_dispose() noexcept final {\n        using handle_type = std::coroutine_handle<generator_promise_base>;\n        handle_type::from_promise(*this).resume();\n    }\n\n    seastar::task* waiting_task() noexcept final {\n        return _waiting_task;\n    }\n\nprivate:\n    template<typename, typename> friend class generator;\n    friend class next_awaiter<Yielded>;\n};\n\ntemplate <typename Yielded>\nstruct generator_promise_base<Yielded>::yield_awaiter final {\n    generator_promise_base* _promise;\n    std::coroutine_handle<> _consumer;\n\n    bool await_ready() const noexcept {\n        return false;\n    }\n    template <typename Promise>\n    std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> producer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(producer.promise());\n        _promise->_waiting_task = &producer.promise();\n        if (seastar::need_preempt()) {\n            auto consumer = std::coroutine_handle<seastar::task>::from_address(\n                _consumer.address());\n            seastar::schedule(&consumer.promise());\n            return std::noop_coroutine();\n        }\n        return _consumer;\n    }\n    void await_resume() noexcept {}\n};\n\ntemplate <typename Yielded>\nstruct generator_promise_base<Yielded>::copy_awaiter final {\n    generator_promise_base* _promise;\n    std::coroutine_handle<> _consumer;\n    yielded_decvref_type _value;\n    value_ptr_type& _value_ptr;\n\n    constexpr bool await_ready() const noexcept {\n        return false;\n    }\n    template <typename Promise>\n    std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> producer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(producer.promise());\n        _value_ptr = std::addressof(_value);\n        auto& current = producer.promise();\n        _promise->_waiting_task = &current;\n        if (seastar::need_preempt()) {\n            auto consumer = std::coroutine_handle<seastar::task>::from_address(\n                _consumer.address());\n            seastar::schedule(&consumer.promise());\n            return std::noop_coroutine();\n        }\n        return _consumer;\n    }\n    constexpr void await_resume() const noexcept {}\n};\n\n/// awaiter returned when the consumer calls \\c operator() to get the next value.\ntemplate <typename Yielded>\nclass [[nodiscard]] next_awaiter {\nprotected:\n    generator_promise_base<Yielded>* _promise = nullptr;\n    std::coroutine_handle<> _producer = nullptr;\n\n    explicit next_awaiter(std::nullptr_t) noexcept {}\n    next_awaiter(generator_promise_base<Yielded>& promise,\n                 std::coroutine_handle<> producer) noexcept\n        : _promise{std::addressof(promise)}\n        , _producer{producer} {}\n\npublic:\n    bool await_ready() const noexcept {\n        return false;\n    }\n\n    template <typename Promise>\n    std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> consumer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(consumer.promise());\n        _promise->_consumer = consumer;\n        // Check if we need to preempt. If not, directly resume producer.\n        // If yes, schedule the producer through the scheduler.\n        if (!seastar::need_preempt()) {\n            return _producer;\n        }\n        auto producer_handle = std::coroutine_handle<seastar::task>::from_address(\n            _producer.address());\n        seastar::schedule(&producer_handle.promise());\n        return std::noop_coroutine();\n    }\n\n    void await_resume() noexcept {}\n};\n\n/// unbuffered generator provides a simple async API for generating values.\n///\n/// generator has 2 template parameters:\n///\n/// - Ref\n/// - Value\n///\n/// From Ref and Value, we derive types:\n/// - value_type: a cv-unqualified object type that specifies the value type\n/// - reference_type: the reference type returned by \\c operator()\n/// - yielded_type: the type of the parameter to the primary overload of \\c\n///   yield_value in the generator's associated promise type\n///\n/// Under the most circumstances, only the first parameter is specified: like\n/// \\c generator<meow>. The resulting generator:\n/// - has a value type of \\c remove_cvref_t<meow>\n/// - has a reference type of \\c meow, if it is a reference type, or \\c meow&&\n///   otherwise\n/// - the operand of \\c co_yield in the body of the generator should be\n///   convertible to \\c meow when \\c meow is a reference type; when \\c meow\n///   is not a reference type, the operand type should be <tt>const meow&</tt>\n///\n/// Consider following code snippet:\n/// \\code\n/// generator<const std::string&> send_query(std::string query) {\n///   auto result_set = db.execute(query);\n///   for (auto row : result_set) {\n///       co_yield std::format(\"{}\", row);\n///   }\n/// }\n/// \\endcode\n///\n/// In this case, \\c Ref is a reference type of \\c <tt>const std::string&</tt>,\n/// and \\c Value is the default value of \\c void. So the \\c value_type is\n/// \\c std::string. The generator returns a reference to the yielded string.\n///\n/// But if some rare users want to use a proxy reference type, they should use\n/// the two-argument \\c generator, like <tt>generator<meow, woof></tt>.\n/// The resulting generator:\n/// - has a value type of \\c woof\n/// - has a reference type of \\c meow\n///\n/// For instance, consider following code snippet:\n/// \\code\n/// generator<std::string_view, std::string> generate_strings() {\n///   co_yield \"[\";\n///   std::string s;\n///   for (auto sv : {\"1\"sv, \"2\"sv}) {\n///     s = sv;\n///     s.push_back(',');\n///     co_yield s;\n///   }\n///   co_yield \"]\";\n/// }\n/// \\endcode\n///\n/// In this case, \\c Ref is \\c std::string_view, and \\c Value is \\c std::string.\n/// So we can ensure that the caller cannot invalidate the yielded values by\n/// mutating the returned value, as \\c std::string_view is immutable. But in\n/// the meanwhile, the generator can \\c co_yield either a \\c std::string_view\n/// or a \\c std::string. The caller receives values as \\c std::string_view.\n///\n/// This unbuffered generator implementation minimizes ping-pong overhead by:\n/// 1. Avoiding the scheduler when preemption is not needed - directly transferring\n///    control between producer and consumer coroutine handles\n/// 2. Only going through the scheduler when \\c need_preempt() indicates it's time\n///    to yield to other tasks\n/// 3. Yielding elements in-place via pointer - zero copy/move overhead\n///\n/// However, the unbuffered design inherently requires one coroutine suspension/resumption\n/// pair per yielded value (producer suspends on co_yield, consumer suspends on co_await).\n/// While direct handle transfer avoids scheduler overhead, the suspension/resumption itself\n/// still impacts icache and branch prediction in high-throughput scenarios.\n///\n/// Performance tradeoffs:\n/// - Use unbuffered generator for latency-sensitive code or when element moves are expensive\n/// - Use buffered generator (third template parameter) for throughput-critical code where\n///   move cost is acceptable\n///\n/// The buffered generator amortizes suspension overhead by batching: elements are moved into\n/// an internal buffer until it's full or need_preempt() returns true. The consumer then drains\n/// the buffer without coroutine suspensions (tight loop). This trades per-element move overhead\n/// for reduced suspension overhead, similar to how flat_mutation_reader batches mutations.\ntemplate<typename Ref, typename Value>\nclass [[nodiscard]] generator {\n    using value_type = std::conditional_t<std::is_void_v<Value>,\n                                          std::remove_cvref_t<Ref>,\n                                          Value>;\n    using reference_type = std::conditional_t<std::is_void_v<Value>,\n                                              Ref&&,\n                                              Ref>;\n    using yielded_type = std::conditional_t<std::is_reference_v<reference_type>,\n                                            reference_type,\n                                            const reference_type&>;\n    // optional_type is used to return values: for references, wrap in reference_wrapper\n    using optional_type = std::conditional_t<std::is_reference_v<reference_type>,\n                                             std::optional<std::reference_wrapper<std::remove_reference_t<reference_type>>>,\n                                             std::optional<reference_type>>;\n\npublic:\n    class promise_type;\n\nprivate:\n    using handle_type = std::coroutine_handle<promise_type>;\n    handle_type _coro = {};\n    bool _started = false;\n\npublic:\n    generator() noexcept = default;\n    explicit generator(promise_type& promise) noexcept\n        : _coro(std::coroutine_handle<promise_type>::from_promise(promise))\n    {}\n    generator(generator&& other) noexcept\n        : _coro{std::exchange(other._coro, {})}\n        , _started{std::exchange(other._started, false)}\n    {}\n    generator(const generator&) = delete;\n    generator& operator=(const generator&) = delete;\n\n    ~generator() {\n        if (_coro) {\n            _coro.destroy();\n        }\n    }\n\n    friend void swap(generator& lhs, generator& rhs) noexcept {\n        std::swap(lhs._coro, rhs._coro);\n        std::swap(lhs._started, rhs._started);\n    }\n\n    generator& operator=(generator&& other) noexcept {\n        if (this == &other) {\n            return *this;\n        }\n        if (_coro) {\n            _coro.destroy();\n        }\n        _coro = std::exchange(other._coro, nullptr);\n        _started = std::exchange(other._started, false);\n        return *this;\n    }\n\n    /// Get the next value from the generator.\n    /// Returns std::optional containing the next value, or std::nullopt if done.\n    /// For reference types, returns std::optional<std::reference_wrapper<T>>.\n    ///\n    /// Example usage:\n    /// \\code\n    /// auto gen = my_generator();\n    /// while (auto value = co_await gen()) {\n    ///     process(value->get());  // for references\n    ///     process(*value);        // for values\n    /// }\n    /// \\endcode\n    class [[nodiscard]] call_awaiter {\n        generator* _gen;\n\n    public:\n        explicit call_awaiter(generator* gen) noexcept\n            : _gen(gen)\n        {}\n\n        bool await_ready() const noexcept {\n            return !_gen->_coro || _gen->_coro.done();\n        }\n\n        template <typename Promise>\n        std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> consumer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n            SEASTAR_COROUTINE_LOC_STORE(consumer.promise());\n            auto& promise = _gen->_coro.promise();\n            promise._consumer = consumer;\n            // Check if we need to preempt. If not, directly resume producer.\n            // If yes, schedule the producer through the scheduler.\n            if (!seastar::need_preempt()) {\n                return _gen->_coro;\n            }\n            auto producer_handle = std::coroutine_handle<seastar::task>::from_address(\n                _gen->_coro.address());\n            seastar::schedule(&producer_handle.promise());\n            return std::noop_coroutine();\n        }\n\n        optional_type await_resume() {\n            if (!_gen->_coro) {\n                return std::nullopt;\n            }\n\n            if (!_gen->_started) {\n                _gen->_started = true;\n            }\n\n            auto& promise = _gen->_coro.promise();\n\n            // Check for exceptions first, even if coroutine is done\n            if (promise.finished() || _gen->_coro.done()) {\n                promise.rethrow_if_unhandled_exception();\n                return std::nullopt;\n            }\n\n            if constexpr (std::is_reference_v<reference_type>) {\n                // promise.value() returns a reference, convert to lvalue ref\n                auto&& val = promise.value();\n                return std::reference_wrapper<std::remove_reference_t<reference_type>>(val);\n            } else {\n                return promise.value();\n            }\n        }\n    };\n\n    [[nodiscard]] call_awaiter operator()() noexcept {\n        return call_awaiter{this};\n    }\n};\n\ntemplate <typename Ref, typename Value>\nclass generator<Ref, Value>::promise_type final : public generator_promise_base<yielded_type> {\npublic:\n    generator get_return_object() noexcept {\n        return generator{*this};\n    }\n\n    yielded_type value() const noexcept {\n        return static_cast<yielded_type>(*this->_value);\n    }\n};\n\n} // namespace unbuffered\n\nnamespace buffered::detail {\n\n// Customization point object for checking if more elements can be pushed to the buffer\n//\n// This CPO follows the C++20 ranges library pattern and provides a flexible\n// customization mechanism with the following priority:\n//\n// 1. Member function: container.can_push_more()\n//    Use this if your container has internal state for measuring capacity\n//    (e.g., memory usage tracking)\n//\n// 2. ADL-found free function: can_push_more(container)\n//    Use this for non-intrusive customization of third-party containers\n//\n// 3. Default implementation: container.size() < container.capacity()\n//    Element count-based measurement for standard containers\n//\n// Example member function customization:\n//   struct memory_aware_buffer {\n//       bool can_push_more() const { return memory_used < memory_limit; }\n//   };\n//\n// Example ADL customization:\n//   namespace my_ns {\n//       struct my_container { ... };\n//       bool can_push_more(const my_container& c) { return ...; }\n//   }\nstruct can_push_more_fn {\n    template <typename Container>\n    constexpr bool operator()(const Container& container) const {\n        // Priority 1: Member function\n        if constexpr (requires { { container.can_push_more() } -> std::convertible_to<bool>; }) {\n            return container.can_push_more();\n        }\n        // Priority 2: ADL-found free function\n        else if constexpr (requires { { can_push_more(container) } -> std::convertible_to<bool>; }) {\n            return can_push_more(container);\n        }\n        // Priority 3: Default implementation\n        else {\n            return container.size() < container.capacity();\n        }\n    }\n};\n\n} // namespace buffered::detail\n\nnamespace buffered {\n\n/// Customization point object for buffer capacity checking\ninline constexpr detail::can_push_more_fn can_push_more{};\n\n/// Concept for bounded containers suitable for buffering generator elements\ntemplate <typename T>\nconcept bounded_container = requires(T container, typename T::value_type element) {\n    // Must have value_type\n    typename T::value_type;\n\n    // Must support capacity queries\n    { container.capacity() } -> std::convertible_to<size_t>;\n    { container.size() } -> std::convertible_to<size_t>;\n\n    // Must support element addition and removal\n    { container.push_back(std::move(element)) } -> std::same_as<void>;\n    { container.clear() } -> std::same_as<void>;\n\n    // Must support indexed access\n    { container[size_t()] } -> std::convertible_to<typename T::value_type&>;\n};\n\ntemplate <bounded_container Container> class next_awaiter;\n\ntemplate <bounded_container Container>\nclass generator_promise_base : public seastar::task {\n    using element_type = typename Container::value_type;\n\nprotected:\n    // buffer holds elements yielded by the producer until consumed\n    Container _buffer;\n    bool _finished = false;\n\nprotected:\n    std::exception_ptr _exception;\n    std::coroutine_handle<> _consumer;\n    task* _waiting_task = nullptr;\n\n    struct yield_awaiter;\n\npublic:\n    generator_promise_base() noexcept = default;\n    generator_promise_base(const generator_promise_base &) = delete;\n    generator_promise_base& operator=(const generator_promise_base &) = delete;\n    generator_promise_base(generator_promise_base &&) noexcept = default;\n    generator_promise_base& operator=(generator_promise_base &&) noexcept = default;\n    virtual ~generator_promise_base() = default;\n\n    // lazily-started coroutine, do not execute the coroutine until\n    // the coroutine is awaited.\n    std::suspend_always initial_suspend() const noexcept {\n        return {};\n    }\n\n    yield_awaiter final_suspend() noexcept {\n        _finished = true;\n        return yield_awaiter{this, this->_consumer, true};\n    }\n\n    void unhandled_exception() noexcept {\n        _exception = std::current_exception();\n    }\n\n    // Yield a single element\n    yield_awaiter yield_value(element_type element) {\n        _buffer.push_back(std::move(element));\n\n        // Should we suspend and let consumer drain the buffer?\n        // Suspend if: buffer is full OR we need to yield to other tasks\n        bool should_suspend = !can_push_more(_buffer) || seastar::need_preempt();\n        return yield_awaiter{this, this->_consumer, should_suspend};\n    }\n\n    // Yield a range/slice of elements (C++23 ranges support)\n    // IMPORTANT: The entire range must fit in the buffer. If the range is larger\n    // than the buffer capacity, elements will be lost when the buffer overflows.\n    // For large datasets, yield elements individually instead of as a range.\n    template <std::ranges::range Range>\n    requires std::convertible_to<std::ranges::range_value_t<Range>, element_type>\n    yield_awaiter yield_value(Range&& range) {\n        // Add all elements from the range to the buffer\n        // Note: We cannot break mid-range because rvalue ranges would be destroyed,\n        // losing the remaining elements. The buffer must have sufficient capacity.\n        for (auto&& element : range) {\n            // Move from rvalue ranges, copy/move from lvalue ranges based on element type\n            if constexpr (std::is_rvalue_reference_v<decltype(range)>) {\n                _buffer.push_back(std::move(element));\n            } else {\n                _buffer.push_back(std::forward<decltype(element)>(element));\n            }\n        }\n\n        // All elements added, check if we should suspend\n        bool should_suspend = !can_push_more(_buffer) || seastar::need_preempt();\n        return yield_awaiter{this, this->_consumer, should_suspend};\n    }\n\n    void return_void() noexcept {}\n\n    // @return if the generator has reached the end of the sequence\n    bool finished() const noexcept {\n        return _finished && _buffer.empty();\n    }\n\n    // @return the buffer containing accumulated elements\n    Container& buffer() noexcept {\n        return _buffer;\n    }\n\n    void rethrow_if_unhandled_exception() {\n        if (_exception) {\n            std::rethrow_exception(std::move(_exception));\n        }\n    }\n\n    void run_and_dispose() noexcept final {\n        using handle_type = std::coroutine_handle<generator_promise_base>;\n        handle_type::from_promise(*this).resume();\n    }\n\n    seastar::task* waiting_task() noexcept final {\n        return _waiting_task;\n    }\n\nprivate:\n    template<typename, typename, bounded_container> friend class generator;\n    friend class next_awaiter<Container>;\n};\n\ntemplate <bounded_container Container>\nstruct generator_promise_base<Container>::yield_awaiter final {\n    generator_promise_base* _promise;\n    std::coroutine_handle<> _consumer;\n    bool _should_suspend;\npublic:\n    yield_awaiter(generator_promise_base* promise,\n                  std::coroutine_handle<> consumer,\n                  bool should_suspend) noexcept\n        : _promise{promise}\n        , _consumer{consumer}\n        , _should_suspend{should_suspend}\n    {}\n    bool await_ready() const noexcept {\n        return !_should_suspend;  // If we shouldn't suspend, we're ready immediately\n    }\n    template <typename Promise>\n    std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> producer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(producer.promise());\n        _promise->_waiting_task = &producer.promise();\n        if (seastar::need_preempt()) {\n            auto consumer = std::coroutine_handle<seastar::task>::from_address(\n                _consumer.address());\n            seastar::schedule(&consumer.promise());\n            return std::noop_coroutine();\n        }\n        return _consumer;\n    }\n    void await_resume() noexcept {}\n};\n\ntemplate <bounded_container Container>\nclass [[nodiscard]] next_awaiter {\nprotected:\n    generator_promise_base<Container>* _promise = nullptr;\n    std::coroutine_handle<> _producer = nullptr;\n\npublic:\n    explicit next_awaiter(std::nullptr_t) noexcept {}\n    next_awaiter(generator_promise_base<Container>& promise,\n                 std::coroutine_handle<> producer) noexcept\n        : _promise{std::addressof(promise)}\n        , _producer{producer} {}\n\n    bool await_ready() const noexcept {\n        return false;\n    }\n\n    template <typename Promise>\n    std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> consumer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(consumer.promise());\n        _promise->_consumer = consumer;\n        // Check if we need to preempt. If not, directly resume producer.\n        // If yes, schedule the producer through the scheduler.\n        if (!seastar::need_preempt()) {\n            return _producer;\n        }\n        auto producer_handle = std::coroutine_handle<seastar::task>::from_address(\n            _producer.address());\n        seastar::schedule(&producer_handle.promise());\n        return std::noop_coroutine();\n    }\n\n    void await_resume() noexcept {}\n};\n\ntemplate<typename Ref, typename Value, bounded_container Container>\nclass [[nodiscard]] generator {\n    using value_type = std::conditional_t<std::is_void_v<Value>,\n                                          std::remove_cvref_t<Ref>,\n                                          Value>;\n    using reference_type = std::conditional_t<std::is_void_v<Value>,\n                                              Ref&&,\n                                              Ref>;\n    using container_type = Container;\n    // optional_type for returning values\n    using optional_type = std::conditional_t<std::is_reference_v<reference_type>,\n                                             std::optional<std::reference_wrapper<std::remove_reference_t<reference_type>>>,\n                                             std::optional<reference_type>>;\n\npublic:\n    class promise_type;\n\nprivate:\n    using handle_type = std::coroutine_handle<promise_type>;\n    handle_type _coro = {};\n    bool _started = false;\n    size_t _buffer_index = 0;  // Current position in the buffer\n\npublic:\n    generator() noexcept = default;\n    explicit generator(promise_type& promise) noexcept\n        : _coro(std::coroutine_handle<promise_type>::from_promise(promise))\n    {}\n    generator(generator&& other) noexcept\n        : _coro{std::exchange(other._coro, {})}\n        , _started{std::exchange(other._started, false)}\n        , _buffer_index{std::exchange(other._buffer_index, 0)}\n    {}\n    generator(const generator&) = delete;\n    generator& operator=(const generator&) = delete;\n\n    ~generator() {\n        if (_coro) {\n            _coro.destroy();\n        }\n    }\n\n    friend void swap(generator& lhs, generator& rhs) noexcept {\n        std::swap(lhs._coro, rhs._coro);\n        std::swap(lhs._started, rhs._started);\n        std::swap(lhs._buffer_index, rhs._buffer_index);\n    }\n\n    generator& operator=(generator&& other) noexcept {\n        if (this == &other) {\n            return *this;\n        }\n        if (_coro) {\n            _coro.destroy();\n        }\n        _coro = std::exchange(other._coro, nullptr);\n        _started = std::exchange(other._started, false);\n        _buffer_index = std::exchange(other._buffer_index, 0);\n        return *this;\n    }\n\n    /// Get the next value from the generator.\n    /// Returns std::optional containing the next value, or std::nullopt if done.\n    /// For reference types, returns std::optional<std::reference_wrapper<T>>.\n    ///\n    /// The buffered generator accumulates elements internally until the buffer\n    /// is full or need_preempt() returns true, then transfers control to the consumer.\n    /// The consumer drains elements one at a time from the buffer without suspension.\n    ///\n    /// Example usage:\n    /// \\code\n    /// generator<const T&, T, circular_buffer_fixed_capacity<T, 128>> gen = my_generator();\n    /// while (auto value = co_await gen()) {\n    ///     process(value->get());  // for references\n    ///     process(*value);        // for values\n    /// }\n    /// \\endcode\n    class [[nodiscard]] call_awaiter {\n        generator* _gen;\n\n    public:\n        explicit call_awaiter(generator* gen) noexcept\n            : _gen(gen)\n        {}\n\n        bool await_ready() const noexcept {\n            // Empty or done generator\n            if (!_gen->_coro || _gen->_coro.done()) {\n                return true;\n            }\n\n            auto& buffer = _gen->_coro.promise().buffer();\n            // If we have elements in the buffer, we're ready (no suspension needed)\n            if (_gen->_buffer_index < buffer.size()) {\n                return true;\n            }\n\n            return false;\n        }\n\n        template <typename Promise>\n        std::coroutine_handle<> await_suspend(std::coroutine_handle<Promise> consumer SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n            SEASTAR_COROUTINE_LOC_STORE(consumer.promise());\n            // Buffer is empty, need to resume producer to get more elements\n            auto& promise = _gen->_coro.promise();\n            promise._consumer = consumer;\n\n            // Clear the buffer before resuming producer\n            promise.buffer().clear();\n            _gen->_buffer_index = 0;\n\n            // Check if we need to preempt. If not, directly resume producer.\n            // If yes, schedule the producer through the scheduler.\n            if (!seastar::need_preempt()) {\n                return _gen->_coro;\n            }\n            auto producer_handle = std::coroutine_handle<seastar::task>::from_address(\n                _gen->_coro.address());\n            seastar::schedule(&producer_handle.promise());\n            return std::noop_coroutine();\n        }\n\n        optional_type await_resume() {\n            // Check for invalid/null coroutine first\n            if (!_gen->_coro) {\n                return std::nullopt;\n            }\n\n            if (!_gen->_started) {\n                _gen->_started = true;\n            }\n\n            auto& promise = _gen->_coro.promise();\n            auto& buffer = promise.buffer();\n\n            // Check buffer first - there might be elements even if coroutine is done\n            if (_gen->_buffer_index < buffer.size()) {\n                auto& element = buffer[_gen->_buffer_index++];\n                if constexpr (std::is_reference_v<reference_type>) {\n                    return std::reference_wrapper<std::remove_reference_t<reference_type>>(element);\n                } else {\n                    // For value types, convert/cast the element if needed\n                    return static_cast<reference_type>(element);\n                }\n            }\n\n            // Buffer is empty - check if producer is finished or has exception\n            if (promise.finished() || _gen->_coro.done()) {\n                promise.rethrow_if_unhandled_exception();\n                return std::nullopt;\n            }\n\n            // This shouldn't happen - we resumed producer but got no elements\n            return std::nullopt;\n        }\n    };\n\n    [[nodiscard]] call_awaiter operator()() noexcept {\n        return call_awaiter{this};\n    }\n};\n\n/// buffered generator has 3 template parameters:\n///\n/// - Ref: The reference type returned by operator()\n/// - Value: The value type\n/// - Container: A bounded container type (must satisfy bounded_container concept)\n///\n/// The buffered generator supports two yielding patterns:\n/// 1. Yield individual elements: co_yield element\n/// 2. Yield ranges/slices: co_yield range (using C++23 ranges)\n///\n/// Elements are accumulated in an internal buffer until:\n/// 1. The buffer reaches its capacity, OR\n/// 2. need_preempt() returns true\n///\n/// Only then does the producer suspend. The consumer drains elements from the\n/// buffer one at a time without any coroutine suspensions (tight loop within buffer).\n///\n/// This provides the convenience of yielding/consuming individual elements while\n/// achieving the performance benefits of batching.\n///\n/// Example (yielding individual elements):\n/// \\code\n/// generator<const directory_entry&, directory_entry,\n///           circular_buffer_fixed_capacity<directory_entry, 128>> list_directory() {\n///     for (auto& entry : entries) {\n///         co_yield entry;  // Yields one at a time, batched internally\n///     }\n/// }\n/// \\endcode\n///\n/// Example (yielding ranges):\n/// \\code\n/// generator<const int&, int, circular_buffer_fixed_capacity<int, 128>> generate() {\n///     std::vector<int> batch = get_batch();\n///     co_yield batch;  // Yields entire range, appended to buffer\n///\n///     co_yield std::span(data, 10);  // Can yield spans, views, etc.\n/// }\n/// \\endcode\n///\n/// Consumer usage (same for both patterns):\n/// \\code\n/// auto gen = list_directory();\n/// while (auto entry = co_await gen()) {\n///     process(entry->get());  // Receives one at a time, no suspension within batch\n/// }\n/// \\endcode\ntemplate <typename Ref, typename Value, bounded_container Container>\nclass generator<Ref, Value, Container>::promise_type final : public generator_promise_base<container_type> {\npublic:\n    generator get_return_object() noexcept {\n        return generator{*this};\n    }\n};\n\n} // namespace buffered\n\n} // namespace internal\n\n// Helper to select generator implementation based on third parameter\ntemplate <typename Ref, typename Value, typename ContainerOrVoid>\nstruct generator_selector {\n    using type = std::conditional_t<!std::is_void_v<ContainerOrVoid> && internal::buffered::bounded_container<ContainerOrVoid>,\n        internal::buffered::generator<Ref, Value, ContainerOrVoid>,\n        internal::unbuffered::generator<Ref, Value>>;\n};\n\n// Specialization for void (unbuffered)\ntemplate <typename Ref, typename Value>\nstruct generator_selector<Ref, Value, void> {\n    using type = internal::unbuffered::generator<Ref, Value>;\n};\n\ntemplate<typename Ref, typename Value = void, typename ContainerOrVoid = void>\nusing generator = typename generator_selector<Ref, Value, ContainerOrVoid>::type;\n\n} // namespace seastar::coroutine::experimental\n"
  },
  {
    "path": "include/seastar/coroutine/maybe_yield.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021-present ScyllaDB\n */\n\n#pragma once\n\n#include <concepts>\n#include <type_traits>\n#include <seastar/core/coroutine.hh>\n\nnamespace seastar::coroutine {\n\nnamespace internal {\n\nstruct maybe_yield_awaiter final {\n    bool await_ready() const {\n        return !need_preempt();\n    }\n\n    template <typename T>\n    void await_suspend(std::coroutine_handle<T> h SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(h.promise());\n        schedule(&h.promise());\n    }\n\n    void await_resume() {\n    }\n};\n\n}\n\n/// Preempt if the current task quota expired.\n///\n/// `maybe_yield()` can be used to break a long computation in a\n/// coroutine and allow the reactor to preempt its execution. This\n/// allows other tasks to gain access to the CPU. If the task quota\n/// did not expire, the coroutine continues execution.\n///\n/// It should be used in long loops that do not contain other `co_await`\n/// calls.\n///\n/// Example\n///\n/// ```\n/// seastar::future<int> long_loop(int n) {\n///     float acc = 0;\n///     for (int i = 0; i < n; ++i) {\n///         acc += std::sin(float(i));\n///         co_await seastar::coroutine::maybe_yield();\n///     }\n///     co_return acc;\n/// }\n/// ```\nclass [[nodiscard(\"must co_await an maybe_yield() object\")]] maybe_yield {\npublic:\n    auto operator co_await() { return internal::maybe_yield_awaiter(); }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/coroutine/parallel_for_each.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022-present ScyllaDB\n */\n\n#pragma once\n\n#include <ranges>\n\n#include <boost/container/small_vector.hpp>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/internal/current_task.hh>\n\nnamespace seastar::coroutine {\n\n/// Invoke a function on all elements in a range in parallel and wait for all futures to complete in a coroutine.\n///\n/// `parallel_for_each` can be used to launch a function concurrently\n/// on all elements in a given range and wait for all of them to complete.\n/// Waiting is performend by `co_await` and returns a future.\n///\n/// If one or more of the function invocations resolve to an exception\n/// then the one of the exceptions is re-thrown.\n/// All of the futures are waited for, even in the case of exceptions.\n///\n/// Example\n///\n/// ```\n/// future<int> sum_of_squares(std::vector<int> v) {\n///     int sum = 0;\n///     return co_await parallel_for_each(v, [&sum] (int& x) {\n///         sum += x * x;\n///     });\n///     co_return sum;\n/// };\n/// ```\n///\n/// Safe for use with lambda coroutines.\n///\n/// \\note parallel_for_each() schedules all invocations of \\c func on the\n///       current shard. If you want to run a function on all shards in parallel,\n///       have a look at \\ref smp::invoke_on_all() instead.\ntemplate <typename Func>\n// constaints for Func are defined at the parallel_for_each constructor\nclass [[nodiscard(\"must co_await an parallel_for_each() object\")]] parallel_for_each final : continuation_base<> {\n    using coroutine_handle_t = std::coroutine_handle<void>;\n\n    Func _func;\n    boost::container::small_vector<future<>, 5> _futures;\n    std::exception_ptr _ex;\n    coroutine_handle_t _when_ready;\n    task* _waiting_task = nullptr;\n\n    // Consume futures in reverse order.\n    // Since futures at the front are expected\n    // to become ready before futures at the back,\n    // therefore it is less likely we will have\n    // to wait on them, after the back futures\n    // become available.\n    //\n    // Return true iff all futures were consumed.\n    bool consume_next() noexcept {\n        while (!_futures.empty()) {\n            auto& fut = _futures.back();\n            if (!fut.available()) {\n                return false;\n            }\n            if (fut.failed()) {\n                _ex = fut.get_exception();\n            }\n            _futures.pop_back();\n        }\n        return true;\n    }\n\n    void set_callback() noexcept {\n        // To reuse `this` as continuation_base<>\n        // we must reset _state, to allow setting\n        // it again.\n        this->_state = {};\n        seastar::internal::set_callback(std::move(_futures.back()), reinterpret_cast<continuation_base<>*>(this));\n        _futures.pop_back();\n    }\n\n    void resume_or_set_callback() noexcept {\n        if (consume_next()) {\n            seastar::internal::set_current_task(_waiting_task);\n            _when_ready.resume();\n        } else {\n            set_callback();\n        }\n    }\n\npublic:\n    // clang 13.0.1 doesn't support subrange\n    // so provide also a Iterator/Sentinel based constructor.\n    // See https://github.com/llvm/llvm-project/issues/46091\n    template <typename Iterator, typename Sentinel, typename Func1>\n    requires (std::same_as<Sentinel, Iterator> || std::sentinel_for<Sentinel, Iterator>)\n        && std::same_as<future<>, futurize_t<std::invoke_result_t<Func, typename std::iterator_traits<Iterator>::reference>>>\n    explicit parallel_for_each(Iterator begin, Sentinel end, Func1&& func) noexcept\n        : _func(std::forward<Func1>(func))\n    {\n        for (auto it = begin; it != end; ++it) {\n            auto fut = futurize_invoke(_func, *it);\n            if (fut.available()) {\n                if (fut.failed()) {\n                    _ex = fut.get_exception();\n                }\n            } else {\n                memory::scoped_critical_alloc_section _;\n                if (_futures.empty()) {\n                    if constexpr (seastar::internal::has_iterator_category<Iterator>::value) {\n                        auto n = seastar::internal::iterator_range_estimate_vector_capacity(it, end);\n                        _futures.reserve(n);\n                    }\n                }\n                _futures.push_back(std::move(fut));\n            }\n        }\n    }\n\n    template <std::ranges::range Range, typename Func1>\n    requires std::invocable<Func, std::ranges::range_reference_t<Range>>\n    explicit parallel_for_each(Range&& range, Func1&& func) noexcept\n        : parallel_for_each(std::ranges::begin(range), std::ranges::end(range), std::forward<Func1>(func))\n    { }\n\n    bool await_ready() const noexcept {\n        if (_futures.empty()) {\n            return !_ex;\n        }\n        return false;\n    }\n\n    template<typename T>\n    void await_suspend(std::coroutine_handle<T> h SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(h.promise());\n        _when_ready = h;\n        _waiting_task = &h.promise();\n        resume_or_set_callback();\n    }\n\n    void await_resume() const {\n        if (_ex) [[unlikely]] {\n            std::rethrow_exception(std::move(_ex));\n        }\n    }\n\n    virtual void run_and_dispose() noexcept override {\n        if (this->_state.failed()) {\n            _ex = std::move(this->_state).get_exception();\n        }\n        resume_or_set_callback();\n    }\n\n    virtual task* waiting_task() noexcept override {\n        return _waiting_task;\n    }\n};\n\ntemplate <typename Iterator, typename Sentinel, typename Func>\nrequires (std::same_as<Sentinel, Iterator> || std::sentinel_for<Sentinel, Iterator>)\n    && std::same_as<future<>, futurize_t<std::invoke_result_t<Func, typename std::iterator_traits<Iterator>::reference>>>\nparallel_for_each(Iterator begin, Sentinel end, Func&& func) -> parallel_for_each<Func>;\n\ntemplate <std::ranges::range Range,\n          std::invocable<std::ranges::range_reference_t<Range>> Func>\nparallel_for_each(Range&& range, Func&& func) -> parallel_for_each<Func>;\n\n\n\n}\n"
  },
  {
    "path": "include/seastar/coroutine/switch_to.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022-present ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/scheduling.hh>\n\nnamespace seastar::coroutine {\n\n/// Switch the current task scheduling group.\n///\n/// `switch_to(new_scheduling_group)` can be used to change\n/// the \\ref scheduling_group of the currently running coroutine.\n///\n/// If the new scheduling group is different than the current scheduling_group,\n/// the coroutine is re-scheduled using the new scheduling group.\n/// Otherwise, the coroutine just continues to run with\n/// the current scheduling group.\n///\n/// `switch_to` returns the current scheduling group\n/// to make it easy to switch back to it if needed.\n///\n/// Example\n///\n/// ```\n/// seastar::future<> cor() {\n///     ... // do some preliminary work\n///     auto prev_sg = co_await coroutine::switch_to(other_sg);\n///     ... // do some work using another scheduling group\n///     co_await coroutine::switch_to(prev_sg);\n///     ... // do some more work\n///     co_return;\n/// }\n/// ```\n\nstruct [[nodiscard(\"must co_await a switch_to object\")]] switch_to final : task {\n    scheduling_group _prev_sg;\n    scheduling_group _switch_to_sg;\n    task* _task = nullptr;\npublic:\n    explicit switch_to(scheduling_group new_sg) noexcept\n        : _prev_sg(current_scheduling_group())\n        , _switch_to_sg(std::move(new_sg))\n    { }\n\n    switch_to(const switch_to&) = delete;\n    switch_to(switch_to&&) = delete;\n\n    bool await_ready() const noexcept {\n        return current_scheduling_group() == _switch_to_sg;\n    }\n\n    template<typename T>\n    void await_suspend(std::coroutine_handle<T> hndl SEASTAR_COROUTINE_LOC_PARAM) noexcept {\n        SEASTAR_COROUTINE_LOC_STORE(hndl.promise());\n        auto& t = hndl.promise();\n        t.set_scheduling_group(_switch_to_sg);\n        _task = &t;\n        schedule(_task);\n    }\n\n    scheduling_group await_resume() {\n        return _prev_sg;\n    }\n\n    virtual void run_and_dispose() noexcept override { }\n\n    virtual task* waiting_task() noexcept override {\n        return _task;\n    }\n};\n\n} // namespace seastar::coroutine\n"
  },
  {
    "path": "include/seastar/coroutine/try_future.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025-present ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/internal/current_task.hh>\n\nnamespace seastar::internal {\n\ntemplate <typename T, typename U>\nvoid try_future_resume_or_destroy_coroutine(seastar::future<T>& fut, seastar::task& coroutine_task) {\n    auto promise_ptr = static_cast<U*>(&coroutine_task);\n    auto hndl = std::coroutine_handle<U>::from_promise(*promise_ptr);\n\n    if (fut.failed()) {\n        hndl.promise().set_exception(std::move(fut).get_exception());\n        hndl.destroy();\n    } else {\n        set_current_task(promise_ptr);\n        hndl.resume();\n    }\n}\n\ntemplate <bool CheckPreempt, typename T>\nclass [[nodiscard]] try_future_awaiter : public seastar::task {\n    seastar::future<T> _future;\n    void (*_resume_or_destroy)(seastar::future<T>&, seastar::task&){};\n    seastar::task* _coroutine_task{};\n    seastar::task* _waiting_task{};\n\npublic:\n    explicit try_future_awaiter(seastar::future<T>&& f) noexcept : _future(std::move(f)) {}\n\n    try_future_awaiter(const try_future_awaiter&) = delete;\n    try_future_awaiter(try_future_awaiter&&) = delete;\n\n    bool await_ready() const noexcept {\n        // Will suspend+schedule for ready failed futures too.\n        return _future.available() && !_future.failed() && (!CheckPreempt || !need_preempt());\n    }\n\n    template<typename U>\n    void await_suspend(std::coroutine_handle<U> hndl) noexcept {\n        _resume_or_destroy = try_future_resume_or_destroy_coroutine<T, U>;\n        _coroutine_task = &hndl.promise();\n        _waiting_task = hndl.promise().waiting_task();\n\n        if (_future.available()) {\n            execute_involving_handle_destruction_in_await_suspend(this);\n        } else {\n            _future.set_coroutine(*this);\n        }\n    }\n\n    T await_resume() {\n        if constexpr (std::is_void_v<T>) {\n            _future.get();\n        } else {\n            return std::move(_future).get();\n        }\n    }\n\n    virtual void run_and_dispose() noexcept final override {\n        _resume_or_destroy(_future, *_coroutine_task);\n    }\n\n    virtual task* waiting_task() noexcept override {\n        return _waiting_task;\n    }\n};\n\n} // namespace seastar::internal\n\nnamespace seastar::coroutine {\n\n/// \\brief co_await:s a \\ref future and returns the wrapped result if successful,\n/// terminates the coroutine otherwise, propagating the exception directly to the\n/// waiter.\n///\n/// If the future was successful, this is identical to co_await-ing the future\n/// directly. If the future failed, the coroutine is not resumed and instead the\n/// exception from the future is forwarded to the waiter directly and the\n/// coroutine is destroyed.\n///\n/// For example:\n/// ```\n/// // Function careful to not throw exceptions, instead returning failed futures.\n/// future<int> bar() {\n///    if (something_bad_happened) {\n///        return make_exception_future<>(std::runtime_error(\"error\"));\n///    }\n///    return result;\n/// }\n///\n/// future<> foo() {\n///     auto result = co_await coroutine::try_future(bar());\n///     // This code is only executed if bar() returned a successful future.\n///     // Otherwise the exception is forwarded to the waiter future directly\n///     // and the coroutine is destroyed.\n///     check_result(result);\n/// }\n/// ```\n///\n/// Note that by default, `try_future` checks for if the task quota is depleted,\n/// which means that it will yield if the future is ready and \\ref seastar::need_preempt()\n/// returns true.  Use \\ref coroutine::try_future_without_preemption_check\n/// to disable preemption checking.\ntemplate<typename T>\nclass [[nodiscard]] try_future : public seastar::internal::try_future_awaiter<true, T> {\npublic:\n    explicit try_future(seastar::future<T>&& f) noexcept\n        : seastar::internal::try_future_awaiter<true, T>(std::move(f))\n    {}\n};\n\n/// \\brief co_await:s a \\ref future, returns the wrapped result if successful,\n/// terminates the coroutine otherwise, propagating the exception to the waiter.\n///\n/// Same as \\ref coroutine::try_future, but does not check for preemption.\ntemplate<typename T>\nclass [[nodiscard]] try_future_without_preemption_check : public seastar::internal::try_future_awaiter<false, T> {\npublic:\n    explicit try_future_without_preemption_check(seastar::future<T>&& f) noexcept\n        : seastar::internal::try_future_awaiter<false, T>(std::move(f))\n    {}\n};\n\n} // namespace seastar::coroutine\n"
  },
  {
    "path": "include/seastar/http/api_docs.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n#include <seastar/json/json_elements.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/http/routes.hh>\n#include <seastar/http/transformers.hh>\n#include <string>\n#include <seastar/util/noncopyable_function.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nstruct api_doc : public json::json_base {\n    json::json_element<std::string> path;\n    json::json_element<std::string> description;\n\n    void register_params() {\n        add(&path, \"path\");\n        add(&description, \"description\");\n\n    }\n    api_doc() {\n        register_params();\n    }\n    api_doc(const api_doc & e)\n      : json::json_base()\n    {\n        register_params();\n        path = e.path;\n        description = e.description;\n    }\n    template<class T>\n    api_doc& operator=(const T& e) {\n        path = e.path;\n        description = e.description;\n        return *this;\n    }\n    api_doc& operator=(const api_doc& e) {\n        path = e.path;\n        description = e.description;\n        return *this;\n    }\n};\n\nstruct api_docs : public json::json_base {\n    json::json_element<std::string> apiVersion;\n    json::json_element<std::string> swaggerVersion;\n    json::json_list<api_doc> apis;\n\n    void register_params() {\n        add(&apiVersion, \"apiVersion\");\n        add(&swaggerVersion, \"swaggerVersion\");\n        add(&apis, \"apis\");\n\n    }\n    api_docs() {\n        apiVersion = \"0.0.1\";\n        swaggerVersion = \"1.2\";\n        register_params();\n    }\n    api_docs(const api_docs & e)\n        : apiVersion{e.apiVersion}\n        , swaggerVersion{e.swaggerVersion}\n        , apis{e.apis}\n    {\n        register_params();\n    }\n    template<class T>\n    api_docs& operator=(const T& e) {\n        apis = e.apis;\n        return *this;\n    }\n    api_docs& operator=(const api_docs& e) {\n        apis = e.apis;\n        return *this;\n    }\n};\n\nclass api_registry_base : public handler_base {\nprotected:\n    sstring _base_path;\n    sstring _file_directory;\n    routes& _routes;\n\npublic:\n    api_registry_base(routes& routes, const sstring& file_directory,\n            const sstring& base_path)\n            : _base_path(base_path), _file_directory(file_directory), _routes(\n                    routes) {\n    }\n\n    void set_route(handler_base* h) {\n        _routes.put(GET, _base_path, h);\n    }\n    virtual ~api_registry_base() = default;\n};\n\nclass api_registry : public api_registry_base {\n    api_docs _docs;\npublic:\n    api_registry(routes& routes, const sstring& file_directory,\n            const sstring& base_path)\n            : api_registry_base(routes, file_directory, base_path) {\n        set_route(this);\n    }\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        rep->_content = json::formatter::to_json(_docs);\n        rep->done(\"json\");\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n\n    void reg(const sstring& api, const sstring& description,\n            const sstring& alternative_path = \"\") {\n        api_doc doc;\n        doc.description = description;\n        doc.path = \"/\" + api;\n        _docs.apis.push(doc);\n        sstring path =\n                (alternative_path == \"\") ?\n                        _file_directory + api + \".json\" : alternative_path;\n        file_handler* index = new file_handler(path,\n                new content_replace(\"json\"));\n        _routes.put(GET, _base_path + \"/\" + api, index);\n    }\n};\n\nclass api_registry_builder_base {\nprotected:\n    sstring _file_directory;\n    sstring _base_path;\n    static const sstring DEFAULT_DIR;\n    static const sstring DEFAULT_PATH;\npublic:\n    api_registry_builder_base(const sstring& file_directory = DEFAULT_DIR,\n            const sstring& base_path = DEFAULT_PATH)\n            : _file_directory(file_directory), _base_path(base_path) {\n    }\n};\n\nclass api_registry_builder : public api_registry_builder_base {\npublic:\n    api_registry_builder(const sstring& file_directory = DEFAULT_DIR,\n            const sstring& base_path = DEFAULT_PATH)\n            : api_registry_builder_base(file_directory, base_path) {\n    }\n\n    void set_api_doc(routes& r) {\n        new api_registry(r, _file_directory, _base_path);\n    }\n\n    void register_function(routes& r, const sstring& api,\n            const sstring& description, const sstring& alternative_path = \"\") {\n        auto h = r.get_exact_match(GET, _base_path);\n        if (h) {\n            // if a handler is found, it was added there by the api_registry_builder\n            // with the set_api_doc method, so we know it's the type\n            static_cast<api_registry*>(h)->reg(api, description, alternative_path);\n        };\n    }\n};\n\nusing doc_entry = noncopyable_function<future<>(output_stream<char>&)>;\n\n/*!\n * \\brief a helper function that creates a reader from a file\n */\n\ndoc_entry get_file_reader(sstring file_name);\n\n/*!\n * \\brief An api doc that support swagger version 2.0\n *\n * The result is a unified JSON file with the swagger definitions.\n *\n * The file content is a concatenation of the doc_entry by the order of\n * their entry.\n *\n * Definitions will be added under the definition section\n *\n * typical usage:\n *\n * First entry:\n *\n  {\n  \"swagger\": \"2.0\",\n  \"host\": \"localhost:10000\",\n  \"basePath\": \"/v2\",\n  \"paths\": {\n\n * entry:\n \"/config/{id}\": {\n      \"get\": {\n        \"description\": \"Return a config value\",\n        \"operationId\": \"findConfigId\",\n        \"produces\": [\n          \"application/json\"\n        ],\n        }\n        }\n *\n * Closing the entries:\n  },\n\n  \"definitions\": {\n  .....\n\n  .....\n  }\n}\n *\n */\nclass api_docs_20 {\n    std::vector<doc_entry> _apis;\n    content_replace _transform;\n    std::vector<doc_entry> _definitions;\n\npublic:\n    future<> write(output_stream<char>&&, std::unique_ptr<http::request> req);\n\n    void add_api(doc_entry&& f) {\n        _apis.emplace_back(std::move(f));\n    }\n\n    void add_definition(doc_entry&& f) {\n        _definitions.emplace_back(std::move(f));\n    }\n};\n\nclass api_registry_20 : public api_registry_base {\n    api_docs_20 _docs;\npublic:\n    api_registry_20(routes& routes, const sstring& file_directory,\n            const sstring& base_path)\n            : api_registry_base(routes, file_directory, base_path) {\n        set_route(this);\n    }\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        rep->write_body(\"json\", [this, req = std::move(req)] (output_stream<char>&& os) mutable {\n            return _docs.write(std::move(os), std::move(req));\n        });\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n\n    virtual void reg(doc_entry&& f) {\n        _docs.add_api(std::move(f));\n    }\n\n    virtual void add_definition(doc_entry&& f) {\n        _docs.add_definition(std::move(f));\n    }\n};\n\nclass api_registry_builder20 : public api_registry_builder_base {\n    api_registry_20* get_register_base(routes& r) {\n        auto h = r.get_exact_match(GET, _base_path);\n        if (h) {\n            // if a handler is found, it was added there by the api_registry_builder\n            // with the set_api_doc method, so we know it's the type\n            return static_cast<api_registry_20*>(h);\n        }\n        return nullptr;\n    }\n\npublic:\n    api_registry_builder20(const sstring& file_directory = DEFAULT_DIR,\n            const sstring& base_path = DEFAULT_PATH)\n            : api_registry_builder_base(file_directory, base_path) {\n    }\n\n    void set_api_doc(routes& r) {\n        new api_registry_20(r, _file_directory, _base_path);\n    }\n\n    /*!\n     * \\brief register a doc_entry\n     * This doc_entry can be used to either take the definition from a file\n     * or generate them dynamically.\n     */\n    void register_function(routes& r, doc_entry&& f) {\n        auto h = get_register_base(r);\n        if (h) {\n            h->reg(std::move(f));\n        }\n    }\n    /*!\n     * \\brief register an API\n     */\n    void register_api_file(routes& r, const sstring& api) {\n        register_function(r, get_file_reader(_file_directory + \"/\" + api + \".json\"));\n    }\n\n\n    /*!\n     * Add a footer doc_entry\n     */\n    void add_definition(routes& r, doc_entry&& f) {\n        auto h = get_register_base(r);\n        if (h) {\n            h->add_definition(std::move(f));\n        }\n\n    }\n\n    /*!\n     * Add a definition file\n     */\n    void add_definitions_file(routes& r, const sstring& file) {\n        add_definition(r, get_file_reader(_file_directory + file + \".def.json\" ));\n    }\n\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/client.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <boost/intrusive/list.hpp>\n#include <seastar/net/api.hh>\n#include <seastar/http/connection_factory.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/http/retry_strategy.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\n\nnamespace tls { class certificate_credentials; }\n\nnamespace http {\n\nnamespace experimental { class client; }\nstruct request;\nstruct reply;\n\nnamespace internal {\n\nclass client_ref {\n    http::experimental::client* _c;\npublic:\n    client_ref(http::experimental::client* c) noexcept;\n    ~client_ref();\n    client_ref(client_ref&& o) noexcept : _c(std::exchange(o._c, nullptr)) {}\n    client_ref(const client_ref&) = delete;\n};\n\n}\n\nnamespace experimental {\n\n/**\n * \\brief Class connection represents an HTTP connection over a given transport\n *\n * Check the demos/http_client_demo.cc for usage example\n */\n\nclass connection : public enable_shared_from_this<connection> {\n    friend class client;\n    using hook_t = bi::list_member_hook<bi::link_mode<bi::auto_unlink>>;\n    using reply_ptr = std::unique_ptr<reply>;\n\n    connected_socket _fd;\n    input_stream<char> _read_buf;\n    output_stream<char> _write_buf;\n    hook_t _hook;\n    future<> _closed;\n    internal::client_ref _ref;\n    // Client sends HTTP-1.1 version and assumes the server is 1.1-compatible\n    // too and thus the connection will be persistent by default. If the server\n    // responds with older version, this flag will be dropped (see recv_reply())\n    bool _persistent = true;\n\npublic:\n    /**\n     * \\brief Create an http connection\n     *\n     * Construct the connection that will work over the provided \\fd transport socket\n     *\n     */\n    connection(connected_socket&& fd, internal::client_ref cr);\n\n    /**\n     * \\brief Send the request and wait for response\n     *\n     * Sends the provided request to the server, and returns a future that will resolve\n     * into the server response.\n     *\n     * If the request was configured with the set_expects_continue() and the server replied\n     * early with some error code, this early reply will be returned back.\n     *\n     * The returned reply only contains the status and headers. To get the reply body the\n     * caller should read it via the input_stream provided by the connection.in() method.\n     *\n     * \\param rq -- request to be sent\n     *\n     */\n    future<reply> make_request(request rq);\n\n    /**\n     * \\brief Get a reference on the connection input stream\n     *\n     * The stream can be used to get back the server response body. After the stream is\n     * finished, the reply can be additionally updated with trailing headers and chunk\n     * extentions\n     *\n     */\n    input_stream<char> in(reply& rep);\n\n    /**\n     * \\brief Closes the connection\n     *\n     * Connection must be closed regardless of whether there was an exception making the\n     * request or not\n     */\n    future<> close();\n\nprivate:\n    future<reply_ptr> do_make_request(const request& rq);\n    future<> send_request_head(const request& rq);\n    future<reply_ptr> maybe_wait_for_continue(const request& req);\n    future<> write_body(const request& rq);\n    future<reply_ptr> recv_reply();\n\n    void shutdown() noexcept;\n};\n\n/**\n * \\brief Class client wraps communications using HTTP protocol\n *\n * The class allows making HTTP requests and handling replies. It's up to the caller to\n * provide a transport, though for simple cases the class provides out-of-the-box\n * facilities.\n *\n * The main benefit client provides against \\ref connection is the transparent support\n * for Keep-Alive transport sockets.\n */\n\nclass client {\npublic:\n    using reply_handler = noncopyable_function<future<>(const reply&, input_stream<char>&& body)>;\n    using retry_requests = bool_class<struct retry_requests_tag>;\n\nprivate:\n    friend class http::internal::client_ref;\n    using connections_list_t = bi::list<connection, bi::member_hook<connection, typename connection::hook_t, &connection::_hook>, bi::constant_time_size<false>>;\n    static constexpr unsigned default_max_connections = 100;\n    static constexpr size_t default_max_bytes_to_drain = 128 * 1024;\n\n    std::unique_ptr<connection_factory> _new_connections;\n    unsigned _nr_connections = 0;\n    unsigned _max_connections;\n    size_t _max_bytes_to_drain;\n    unsigned long _total_new_connections = 0;\n    std::unique_ptr<retry_strategy> _retry_strategy;\n    condition_variable _wait_con;\n    connections_list_t _pool;\n\n    using connection_ptr = seastar::shared_ptr<connection>;\n\n    future<connection_ptr> get_connection(abort_source* as);\n    future<connection_ptr> make_connection(abort_source* as);\n    future<> put_connection(connection_ptr con);\n    future<> shrink_connections();\n\n    template <std::invocable<connection&> Fn>\n    auto with_connection(Fn&& fn, abort_source*);\n\n    template <typename Fn>\n    requires std::invocable<Fn, connection&>\n    auto with_new_connection(Fn&& fn, abort_source*);\n\n    future<> maybe_retry_request(std::exception_ptr ex,\n                                 unsigned retry_count,\n                                 const request& req,\n                                 reply_handler& handle,\n                                 const retry_strategy& strategy,\n                                 std::optional<reply::status_type> expected,\n                                 abort_source* as);\n\n    future<> do_make_request(connection& con, const request& req, reply_handler& handle, abort_source*, std::optional<reply::status_type> expected);\n\npublic:\n    /**\n     * \\brief Construct a simple client\n     *\n     * This creates a simple client that connects to provided address via plain (non-TLS)\n     * socket\n     *\n     * \\param addr -- host address to connect to\n     *\n     */\n    explicit client(socket_address addr);\n\n    /**\n     * \\brief Construct a secure client\n     *\n     * This creates a client that connects to provided address via TLS socket with\n     * given credentials. In simple words -- this makes an HTTPS client\n     *\n     * \\param addr -- host address to connect to\n     * \\param creds -- credentials\n     * \\param host -- optional host name\n     *\n     */\n    client(socket_address addr, shared_ptr<tls::certificate_credentials> creds, sstring host = {});\n\n    /**\n     * \\brief Construct a client with connection factory\n     *\n     * This creates a client that uses factory to get \\ref connected_socket that is then\n     * used as transport. The client may withdraw more than one socket from the factory and\n     * may re-use the sockets on its own\n     *\n     * \\param f -- the factory pointer\n     * \\param max_connections -- maximum number of connection a client is allowed to maintain\n     * (both active and cached in pool)\n     * \\param retry_strategy -- optional custom logic for retrying failed requests\n     *\n     * The client uses connections provided by factory to send requests over and receive responses\n     * back. Once request-response cycle is over the connection used for that is kept by a client\n     * in a \"pool\". Making another http request may then pick up the existing connection from the\n     * pool thus avoiding the extra latency of establishing new connection. Pool may thus accumulate\n     * more than one connection if user sends several requests in parallel.\n     *\n     * HTTP servers may sometimes want to terminate the connections it keeps. This can happen in\n     * one of several ways.\n     *\n     * The \"gentle\" way is when server adds the \"connection: close\" header to its response. In that\n     * case client would handle the response and will just close the connection without putting it\n     * to pool.\n     *\n     * Less gentle way a server may terminate a connection is by closing it, so the underlying TCP\n     * stack would communicate regular TCP FIN-s. If the connection happens to be in pool when it\n     * happens the client would just clean the connection from pool in the background.\n     *\n     * Sometimes the least gentle closing occurs when server closes the connection on the fly and\n     * TCP starts communicating FIN-s in parallel with client using it. In that case, user would\n     * receive exception from the \\ref make_request() call and will have to do something about it.\n     * Client provides a transparent way of handling it called \"retry\".\n     *\n     * When enabled, it makes client catch the transport error, close the broken connection, open\n     * another one and retry the very same request one more time over this new connection. If the\n     * second attempt fails, this error is reported back to user.\n     */\n    explicit client(std::unique_ptr<connection_factory> f, unsigned max_connections = default_max_connections, retry_requests retry = retry_requests::no, size_t max_bytes_to_drain = default_max_bytes_to_drain);\n    client(std::unique_ptr<connection_factory> f, unsigned max_connections, size_t max_bytes_to_drain, std::unique_ptr<retry_strategy>&& retry_strategy);\n\n    /**\n     * \\brief Send the request and handle the response\n     *\n     * Sends the provided request to the server and calls the provided callback to handle\n     * the response when it arrives. If the expected status is specified and the response's\n     * status is not the expected one, the handler is not called and the method resolves\n     * with exceptional future. Otherwise returns the handler's future\n     *\n     * \\param req -- request to be sent\n     * \\param handle -- the response handler\n     * \\param expected -- the optional expected reply status code, default is std::nullopt\n     * \\param as -- abort source that aborts the request\n     *\n     * Note that the handle callback should be prepared to be called more than once, because\n     * client may restart the whole request processing in case server closes the connection\n     * in the middle of operation\n     */\n    future<> make_request(request&& req, reply_handler&& handle, std::optional<reply::status_type>&& expected = std::nullopt, abort_source* as = nullptr);\n\n    /**\n     * \\brief Send the request and handle the response with retry strategy\n     *\n     * Same as \\ref make_request()\n     * The retry strategy defines how the client should behave in case of transient failures.\n     *\n     * \\param req -- request to be sent\n     * \\param handle -- the response handler\n     * \\param strategy -- retry strategy to apply on transient failures\n     * \\param expected -- the optional expected reply status code, default is std::nullopt\n     * \\param as -- abort source that aborts the request\n     *\n     * Note that the handle callback should be prepared to be called more than once, because\n     * client may restart the whole request processing in case server closes the connection\n     * in the middle of operation\n     */\n    future<> make_request(request&& req, reply_handler&& handle, const retry_strategy& strategy, std::optional<reply::status_type>&& expected = std::nullopt, abort_source* as = nullptr);\n\n    /**\n     * \\brief Send the request and handle the response (abortable), same as \\ref make_request()\n     *\n     *  @attention Note that the method does not take the ownership of the\n     * `request and the `handle`, it caller's responsibility the make sure they\n     * are referencing valid instances\n     */\n    future<> make_request(const request& req, reply_handler& handle, std::optional<reply::status_type> expected = std::nullopt, abort_source* as = nullptr);\n\n    /**\n     * \\brief Send the request and handle the response with retry strategy (abortable), same as \\ref make_request()\n     *\n     * Same as \\ref make_request()\n     * The retry strategy defines how the client should behave in case of transient failures.\n     *\n     * \\param req -- request to be sent (non-owning reference)\n     * \\param handle -- the response handler (non-owning reference)\n     * \\param strategy -- retry strategy to apply on transient failures\n     * \\param expected -- the optional expected reply status code, default is std::nullopt\n     * \\param as -- abort source that aborts the request\n     *\n     * @attention Note that the method does not take the ownership of the\n     * `request` and the `handle`, it is the caller's responsibility to ensure they\n     * are referencing valid instances\n     */\n    future<> make_request(const request& req, reply_handler& handle, const retry_strategy& strategy, std::optional<reply::status_type> expected = std::nullopt, abort_source* as = nullptr);\n\n    /**\n     * \\brief Updates the maximum number of connections a client may have\n     *\n     * If the new limit is less than the amount of connections a client has, they will be\n     * closed. The returned future resolves when all excessive connections get closed\n     *\n     * \\param nr -- the new limit on the number of connections\n     */\n    future<> set_maximum_connections(unsigned nr);\n\n    /**\n     * \\brief Closes the client\n     *\n     * Client must be closed before destruction unconditionally\n     */\n    future<> close();\n\n    /**\n     * \\brief Returns the total number of connections\n     */\n\n    unsigned connections_nr() const noexcept {\n        return _nr_connections;\n    }\n\n    /**\n     * \\brief Returns the number of idle connections\n     */\n\n    unsigned idle_connections_nr() const noexcept {\n        return _pool.size();\n    }\n\n    /**\n     * \\brief Returns the total number of connection factory invocations made so far\n     *\n     * This is the monotonically-increasing counter describing how \"frequently\" the\n     * client kicks its factory for new connections.\n     */\n\n    unsigned long total_new_connections_nr() const noexcept {\n        return _total_new_connections;\n    }\n};\n\n} // experimental namespace\n\n} // http namespace\n\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/http/common.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <unordered_map>\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/http/url.hh>\n\nnamespace seastar {\n\nnamespace http {\nnamespace internal {\noutput_stream<char> make_http_chunked_output_stream(output_stream<char>& out);\n// @param total_len defines the maximum number of bytes to be written.\n// @param bytes_written after the stream is closed, it is updated with the\n//        actual number of bytes written.\noutput_stream<char> make_http_content_length_output_stream(output_stream<char>& out, size_t total_len, size_t& bytes_written);\n} // internal namespace\n} // http namespace\n\nnamespace httpd {\n\n\nclass parameters {\n    // Note: the path matcher adds parameters with the '/' prefix into the params map (eg. \"/param1\"), and some getters\n    // remove this '/' to return just the path parameter value\n    std::unordered_map<sstring, sstring> params;\npublic:\n    const sstring& path(const sstring& key) const {\n        return params.at(key);\n    }\n\n    [[deprecated(\"Use request::get_path_param() instead.\")]]\n    sstring operator[](const sstring& key) const {\n        return params.at(key).substr(1);\n    }\n\n    const sstring& at(const sstring& key) const {\n        return path(key);\n    }\n\n    sstring get_decoded_param(const sstring& key) const {\n        auto res = params.find(key);\n        if (res == params.end()) {\n            return \"\";\n        }\n        auto raw_path_param = res->second.substr(1);\n        auto decoded_path_param = sstring{};\n        auto ok = seastar::http::internal::path_decode(raw_path_param, decoded_path_param);\n        if (!ok) {\n            return \"\";\n        }\n        return decoded_path_param;\n    }\n\n    bool exists(const sstring& key) const {\n        return params.find(key) != params.end();\n    }\n\n    void set(const sstring& key, const sstring& value) {\n        params[key] = value;\n    }\n\n    void clear() {\n        params.clear();\n    }\n\n};\n\nenum operation_type {\n    GET, POST, PUT, DELETE, HEAD, OPTIONS, TRACE, CONNECT, PATCH, NUM_OPERATION\n};\n\n/**\n * Translate the string command to operation type\n * @param type the string \"GET\" or \"POST\"\n * @return the operation_type\n */\noperation_type str2type(const sstring& type);\n\n/**\n * Translate the operation type to command string\n * @param type the string GET or POST\n * @return the command string \"GET\" or \"POST\"\n */\nsstring type2str(operation_type type);\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/connection_factory.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/seastar.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/tls.hh>\n\nnamespace seastar::http::experimental {\n\n/**\n * \\brief Factory that provides transport for \\ref client\n *\n * This customization point allows callers provide its own transport for client. The\n * client code calls factory when it needs more connections to the server.\n */\n\nclass connection_factory {\npublic:\n    /**\n     * \\brief Make a \\ref connected_socket\n     *\n     * The implementations of this method should return ready-to-use socket that will\n     * be used by \\ref client as transport for its http connections\n     */\n    virtual future<connected_socket> make(abort_source*) = 0;\n    /**\n    * \\brief Shutdown the factory\n    *\n    * The implementations of this method should perform needed cleanup of the factory that will\n    * be called by \\ref client when it is being closed.\n    */\n    virtual future<> close() {\n        return make_ready_future<>();\n    };\n    virtual ~connection_factory() {}\n};\n\nclass basic_connection_factory : public connection_factory {\n    socket_address _addr;\npublic:\n    explicit basic_connection_factory(socket_address addr)\n        : _addr(std::move(addr))\n    {\n    }\n    virtual future<connected_socket> make(abort_source* as) override {\n        return seastar::connect(_addr, {}, transport::TCP);\n    }\n};\n\nclass tls_connection_factory : public connection_factory {\n    socket_address _addr;\n    shared_ptr<tls::certificate_credentials> _creds;\n    sstring _host;\npublic:\n    tls_connection_factory(socket_address addr, shared_ptr<tls::certificate_credentials> creds, sstring host)\n        : _addr(std::move(addr))\n        , _creds(std::move(creds))\n        , _host(std::move(host))\n    {\n    }\n    virtual future<connected_socket> make(abort_source* as) override {\n        return tls::connect(_creds, _addr, tls::tls_options{.server_name = _host});\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/http/exception.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/util/log.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/json/json_elements.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n\n/**\n * The base_exception is a base for all http exception.\n * It contains a message that will be return as the message content\n * and a status that will be return as a status code.\n */\nclass base_exception : public std::exception {\npublic:\n    base_exception(const std::string& msg, http::reply::status_type status)\n            : _msg(msg), _status(status) {\n    }\n\n    virtual const char* what() const noexcept {\n        return _msg.c_str();\n    }\n\n    http::reply::status_type status() const {\n        return _status;\n    }\n\n    virtual const std::string& str() const {\n        return _msg;\n    }\nprivate:\n    std::string _msg;\n    http::reply::status_type _status;\n\n};\n\n/**\n * Throwing this exception will result in a redirect to the given url\n */\nclass redirect_exception : public base_exception {\npublic:\n    redirect_exception(const std::string& url, http::reply::status_type status = http::reply::status_type::moved_permanently)\n            : base_exception(\"\", status), url(url) {\n    }\n    std::string url;\n};\n\n/**\n * Throwing this exception will result in a 404 not found result\n */\nclass not_found_exception : public base_exception {\npublic:\n    not_found_exception(const std::string& msg = \"Not found\")\n            : base_exception(msg, http::reply::status_type::not_found) {\n    }\n};\n\n/**\n * Throwing this exception will result in a 400 bad request result\n */\n\nclass bad_request_exception : public base_exception {\npublic:\n    bad_request_exception(const std::string& msg)\n            : base_exception(msg, http::reply::status_type::bad_request) {\n    }\n};\n\nclass bad_param_exception : public bad_request_exception {\npublic:\n    bad_param_exception(const std::string& msg)\n            : bad_request_exception(msg) {\n    }\n};\n\nclass missing_param_exception : public bad_request_exception {\npublic:\n    missing_param_exception(const std::string& param)\n            : bad_request_exception(\n                    std::string(\"Missing mandatory parameter '\") + param + \"'\") {\n    }\n};\n\nclass bad_chunk_exception : public bad_request_exception {\npublic:\n    bad_chunk_exception(const std::string& msg)\n            : bad_request_exception(\n                    std::string(\"Can't read body chunk in a 'chunked' request '\") + msg + \"'\") {\n    }\n};\n\nclass server_error_exception : public base_exception {\npublic:\n    server_error_exception(const std::string& msg)\n            : base_exception(msg, http::reply::status_type::internal_server_error) {\n    }\n};\n\nclass response_parsing_exception : public server_error_exception {\npublic:\n    explicit response_parsing_exception(const std::string& msg) : server_error_exception(msg) {}\n};\n\nclass [[deprecated(\"Use base_exception or any of its inheritants instead\")]] json_exception : public json::json_base {\npublic:\n    json::json_element<std::string> _msg;\n    json::json_element<int> _code;\n    void register_params() {\n        add(&_msg, \"message\");\n        add(&_code, \"code\");\n    }\n\n    json_exception(const base_exception & e) {\n        set(e.str(), e.status());\n    }\n\n    json_exception(std::exception_ptr e) {\n        std::ostringstream exception_description;\n        exception_description << e;\n        set(exception_description.str(), http::reply::status_type::internal_server_error);\n    }\nprivate:\n    void set(const std::string& msg, http::reply::status_type code) {\n        register_params();\n        _msg = msg;\n        _code = (int) code;\n    }\n};\n\n/**\n * Client-side exception to report unexpected server reply status\n */\nclass unexpected_status_error : public base_exception {\npublic:\n    unexpected_status_error(http::reply::status_type st)\n        : base_exception(fmt::to_string(st), st)\n    {}\n};\n\n}\n\n}\n\ntemplate <>\nstruct fmt::formatter<seastar::httpd::base_exception> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::httpd::base_exception& e, fmt::format_context& ctx) const {\n        return fmt::format_to(ctx.out(), \"{} ({})\", e.what(), e.status());\n    }\n};\n"
  },
  {
    "path": "include/seastar/http/file_handler.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/http/handlers.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * This is a base class for file transformer.\n *\n * File transformer adds the ability to modify a file content before returning\n * the results, it acts as a factory class for output_stream.\n *\n * The transformer decides according to the file extension if transforming is\n * needed.\n *\n * If a transformation is needed it would create a new output stream from the given stream.\n */\nclass file_transformer {\npublic:\n    /**\n     * Any file transformer should implement this method.\n     * @param req the request\n     * @param extension the file extension originating the content\n     * returns a new output stream to be used when writing the file to the reply\n     */\n    virtual output_stream<char> transform(std::unique_ptr<http::request> req,\n            const sstring& extension, output_stream<char>&& s) = 0;\n    file_transformer() = default;\n    file_transformer(file_transformer&&) = default;\n    virtual ~file_transformer() = default;\n};\n\n/**\n * A base class for handlers that interact with files.\n * directory and file handlers both share some common logic\n * with regards to file handling.\n * they both needs to read a file from the disk, optionally transform it,\n * and return the result or page not found on error\n */\nclass file_interaction_handler : public handler_base {\npublic:\n    file_interaction_handler(file_transformer* p = nullptr)\n            : transformer(p) {\n\n    }\n\n    ~file_interaction_handler();\n\n    /**\n     * Allows setting a transformer to be used with the files returned.\n     * @param t the file transformer to use\n     * @return this\n     */\n    file_interaction_handler* set_transformer(file_transformer* t) {\n        transformer = t;\n        return this;\n    }\n\n    /**\n     * if the url ends without a slash redirect\n     * @param req the request\n     * @param rep the reply\n     * @return true on redirect\n     */\n    bool redirect_if_needed(const http::request& req, http::reply& rep) const;\n\n    /**\n     * A helper method that returns the file extension.\n     * @param file the file to check\n     * @return the file extension\n     */\n    static sstring get_extension(const sstring& file);\n\nprotected:\n\n    /**\n     * read a file from the disk and return it in the replay.\n     * @param file the full path to a file on the disk\n     * @param req the reuest\n     * @param rep the reply\n     */\n    future<std::unique_ptr<http::reply> > read(sstring file,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep);\n    file_transformer* transformer;\n\n    output_stream<char> get_stream(std::unique_ptr<http::request> req,\n            const sstring& extension, output_stream<char>&& s);\n};\n\n/**\n * The directory handler get a disk path in the\n * constructor.\n * and expect a path parameter in the handle method.\n * it would concatenate the two and return the file\n * e.g. if the path is /usr/mgmt/public in the path\n * parameter is index.html\n * handle will return the content of /usr/mgmt/public/index.html\n */\nclass directory_handler : public file_interaction_handler {\npublic:\n\n    /**\n     * The directory handler map a base path and a path parameter to a file\n     * @param doc_root the root directory to search the file from.\n     * @param transformer an optional file transformer\n     * For example if the root is '/usr/mgmt/public' and the path parameter\n     * will be '/css/style.css' the file wil be /usr/mgmt/public/css/style.css'\n     */\n    explicit directory_handler(const sstring& doc_root,\n            file_transformer* transformer = nullptr);\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override;\n\nprivate:\n    sstring doc_root;\n};\n\n/**\n * The file handler get a path to a file on the disk\n * in the constructor.\n * it will always return the content of the file.\n */\nclass file_handler : public file_interaction_handler {\npublic:\n\n    /**\n     * The file handler map a file to a url\n     * @param file the full path to the file on the disk\n     * @param transformer an optional file transformer\n     * @param force_path check if redirect is needed upon `handle`\n     */\n    explicit file_handler(const sstring& file, file_transformer* transformer =\n            nullptr, bool force_path = true)\n            : file_interaction_handler(transformer), file(file), force_path(\n                    force_path) {\n    }\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override;\n\nprivate:\n    sstring file;\n    bool force_path;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/function_handlers.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <concepts>\n#include <seastar/http/handlers.hh>\n#include <functional>\n#include <seastar/json/json_elements.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * A request function is a lambda expression that gets only the request\n * as its parameter\n */\ntypedef std::function<sstring(const_req req)> request_function;\n\n/**\n * A handle function is a lambda expression that gets request and reply\n */\ntypedef std::function<sstring(const_req req, http::reply&)> handle_function;\n\n/**\n * A json request function is a lambda expression that gets only the request\n * as its parameter and return a json response.\n * Using the json response is done implicitly.\n */\ntypedef std::function<json::json_return_type(const_req req)> json_request_function;\n\n/**\n * A future_json_function is a function that returns a future json reponse.\n * Similiar to the json_request_function, using the json reponse is done\n * implicitly.\n */\ntypedef std::function<\n        future<json::json_return_type>(std::unique_ptr<http::request> req)> future_json_function;\n\ntypedef std::function<\n        future<std::unique_ptr<http::reply>>(std::unique_ptr<http::request> req,\n                std::unique_ptr<http::reply> rep)> future_handler_function;\n/**\n * The function handler get a lambda expression in the constructor.\n * it will call that expression to get the result\n * This is suited for very simple handlers\n *\n */\nclass function_handler : public handler_base {\npublic:\n\n    function_handler(const handle_function & f_handle, const sstring& type)\n            : _f_handle(\n                    [f_handle](std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                        rep->_content += f_handle(*req.get(), *rep.get());\n                        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                    }), _type(type) {\n    }\n\n    function_handler(const future_handler_function& f_handle, const sstring& type)\n        : _f_handle(f_handle), _type(type) {\n    }\n\n    function_handler(const request_function & _handle, const sstring& type)\n            : _f_handle(\n                    [_handle](std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                        rep->_content += _handle(*req.get());\n                        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                    }), _type(type) {\n    }\n\n    function_handler(const json_request_function& _handle)\n            : _f_handle(\n                    [_handle](std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                        return write_json_reply(std::move(rep), _handle(*req.get()));\n                    }), _type(\"json\") {\n    }\n\n    function_handler(const future_json_function& _handle)\n            : _f_handle(\n                    [_handle](std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                        return _handle(std::move(req)).then([rep = std::move(rep)](json::json_return_type&& res) mutable {\n                            return write_json_reply(std::move(rep), std::move(res));\n                        });\n                    }), _type(\"json\") {\n    }\n\n    function_handler(const function_handler&) = default;\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        return _f_handle(std::move(req), std::move(rep)).then(\n                [this](std::unique_ptr<http::reply> rep) {\n                    rep->done(_type);\n                    return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                });\n    }\n\nprivate:\n    // send the json payload of result to reply, return the reply pointer\n    static future<std::unique_ptr<http::reply>> write_json_reply(\n        std::unique_ptr<http::reply>&& reply,\n        std::same_as<json::json_return_type> auto&& result) {\n        if (result._body_writer) {\n            reply->write_body(\"json\", std::move(result._body_writer));\n        } else {\n            reply->_content += result._res;\n        }\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(reply));\n    }\n\nprotected:\n    future_handler_function _f_handle;\n    sstring _type;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/handlers.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/http/request.hh>\n#include <seastar/http/common.hh>\n#include <seastar/http/exception.hh>\n#include <seastar/http/reply.hh>\n\n\nnamespace seastar {\n\nnamespace httpd {\n\ntypedef const http::request& const_req;\n\n/**\n * handlers holds the logic for serving an incoming request.\n * All handlers inherit from the base handler_base and\n * implement the handle method.\n *\n */\nclass handler_base {\n    std::vector<sstring> _mandatory_param;\nprotected:\n    handler_base() = default;\n    handler_base(const handler_base&) = default;\npublic:\n    virtual ~handler_base() = default;\n    /**\n     * All handlers should implement this method.\n     *  It fill the reply according to the request.\n     * @param path the url path used in this call\n     * @param req the original request\n     * @param rep the reply\n     */\n    virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) = 0;\n\n\n    /**\n     * Add a mandatory parameter\n     * @param param a parameter name\n     * @return a reference to the handler\n     */\n    handler_base& mandatory(const sstring& param) {\n        _mandatory_param.push_back(param);\n        return *this;\n    }\n\n    /**\n     * Check if all mandatory parameters exist in the request. if any param\n     * does not exist, the function would throw a @c missing_param_exception\n     * @param params req the http request\n     */\n    void verify_mandatory_params(const http::request& req) const {\n        for (auto& param : _mandatory_param) {\n            if (req.get_query_param(param) == \"\") {\n                throw missing_param_exception(param);\n            }\n        }\n    }\n};\n\n}\n\n}\n\n"
  },
  {
    "path": "include/seastar/http/httpd.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <limits>\n#include <cctype>\n#include <vector>\n#include <boost/intrusive/list.hpp>\n#include <seastar/http/request_parser.hh>\n#include <seastar/http/request.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/http/routes.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/core/shared_ptr.hh>\n\nnamespace seastar {\n\nnamespace http {\nstruct reply;\n}\n\nnamespace httpd {\n\nclass http_server;\nclass http_stats;\n\nusing namespace std::chrono_literals;\n\nclass http_stats {\n    metrics::metric_groups _metric_groups;\npublic:\n    http_stats(http_server& server, const sstring& name);\n};\n\nclass connection : public boost::intrusive::list_base_hook<> {\n    http_server& _server;\n    connected_socket _fd;\n    input_stream<char> _read_buf;\n    output_stream<char> _write_buf;\n    socket_address _client_addr;\n    socket_address _server_addr;\n    static constexpr size_t limit = 4096;\n    using tmp_buf = temporary_buffer<char>;\n    http_request_parser _parser;\n    std::unique_ptr<http::reply> _resp;\n    // null element marks eof\n    queue<std::unique_ptr<http::reply>> _replies { 10 };\n    bool _done = false;\n    const bool _tls;\npublic:\n    [[deprecated(\"use connection(http_server&, connected_socket&&, bool tls)\")]]\n    connection(http_server& server, connected_socket&& fd, socket_address, bool tls)\n            : connection(server, std::move(fd), tls) {}\n    connection(http_server& server, connected_socket&& fd, bool tls)\n            : _server(server)\n            , _fd(std::move(fd))\n            , _read_buf(_fd.input())\n            , _write_buf(_fd.output())\n            , _client_addr(_fd.remote_address())\n            , _server_addr(_fd.local_address())\n            , _tls(tls) {\n        on_new_connection();\n    }\n    connection(http_server& server, connected_socket&& fd,\n            socket_address client_addr, socket_address server_addr, bool tls)\n            : _server(server)\n            , _fd(std::move(fd))\n            , _read_buf(_fd.input())\n            , _write_buf(_fd.output())\n            , _client_addr(std::move(client_addr))\n            , _server_addr(std::move(server_addr))\n            , _tls(tls) {\n        on_new_connection();\n    }\n    ~connection();\n    void on_new_connection();\n\n    future<> process();\n    void shutdown();\n    future<> read();\n    future<> read_one();\n    future<> respond();\n    future<> do_response_loop();\n\n    void set_headers(http::reply& resp);\n\n    future<> start_response();\n\n    future<bool> generate_reply(std::unique_ptr<http::request> req);\n    void generate_error_reply_and_close(std::unique_ptr<http::request> req, http::reply::status_type status, const sstring& msg);\n\n    output_stream<char>& out();\n};\n\nclass http_server_tester;\n\nclass http_server {\n    std::vector<server_socket> _listeners;\n    http_stats _stats;\n    uint64_t _total_connections = 0;\n    uint64_t _current_connections = 0;\n    uint64_t _requests_served = 0;\n    uint64_t _read_errors = 0;\n    uint64_t _respond_errors = 0;\n    shared_ptr<seastar::tls::server_credentials> _credentials;\n    sstring _date = http_date();\n    timer<> _date_format_timer { [this] {_date = http_date();} };\n    size_t _content_length_limit = std::numeric_limits<size_t>::max();\n    bool _content_streaming = false;\n    gate _task_gate;\n    std::optional<net::keepalive_params> _keepalive_params;\npublic:\n    routes _routes;\n    using connection = seastar::httpd::connection;\n    using server_credentials_ptr = shared_ptr<seastar::tls::server_credentials>;\n    explicit http_server(const sstring& name) : _stats(*this, name) {\n        _date_format_timer.arm_periodic(1s);\n    }\n    /*!\n     * \\brief set tls credentials for the server\n     * Setting the tls credentials will set the http-server to work in https mode.\n     *\n     * To use the https, create server credentials and pass it to the server before it starts.\n     *\n     * Use case example using seastar threads for clarity:\n\n        sharded<http_server> server; // typical server\n\n        seastar::shared_ptr<seastar::tls::credentials_builder> creds = seastar::make_shared<seastar::tls::credentials_builder>();\n        sstring ms_cert = \"MyCertificate.crt\";\n        sstring ms_key = \"MyKey.key\";\n\n        creds->set_dh_level(seastar::tls::dh_params::level::MEDIUM);\n\n        creds->set_x509_key_file(ms_cert, ms_key, seastar::tls::x509_crt_format::PEM).get();\n        creds->set_system_trust().get();\n\n\n        server.invoke_on_all([creds](http_server& server) {\n            server.set_tls_credentials(creds->build_server_credentials());\n            return make_ready_future<>();\n        }).get();\n     *\n     */\n    [[deprecated(\"use listen(socket_address addr, server_credentials_ptr credentials)\")]]\n    void set_tls_credentials(server_credentials_ptr credentials);\n\n    void set_keepalive_parameters(std::optional<net::keepalive_params> params) {\n        _keepalive_params = std::move(params);\n    }\n\n    size_t get_content_length_limit() const;\n\n    void set_content_length_limit(size_t limit);\n\n    bool get_content_streaming() const;\n\n    void set_content_streaming(bool b);\n\n    future<> listen(socket_address addr, server_credentials_ptr credentials);\n    future<> listen(socket_address addr, listen_options lo, server_credentials_ptr credentials);\n    future<> listen(socket_address addr, listen_options lo);\n    future<> listen(socket_address addr);\n    future<> stop();\n\n    future<> do_accepts(int which);\n    future<> do_accepts(int which, bool with_tls);\n\n    uint64_t total_connections() const;\n    uint64_t current_connections() const;\n    uint64_t requests_served() const;\n    uint64_t read_errors() const;\n    uint64_t reply_errors() const;\n    // Write the current date in the specific \"preferred format\" defined in\n    // RFC 7231, Section 7.1.1.1.\n    static sstring http_date();\nprivate:\n    future<> do_accept_one(int which, bool with_tls);\n    boost::intrusive::list<connection> _connections;\n    friend class seastar::httpd::connection;\n    friend class http_server_tester;\n};\n\nclass http_server_tester {\npublic:\n    static std::vector<server_socket>& listeners(http_server& server) {\n        return server._listeners;\n    }\n};\n\n/*\n * A helper class to start, set and listen an http server\n * typical use would be:\n *\n * auto server = new http_server_control();\n *                 server->start().then([server] {\n *                 server->set_routes(set_routes);\n *              }).then([server, port] {\n *                  server->listen(port);\n *              }).then([port] {\n *                  std::cout << \"Seastar HTTP server listening on port \" << port << \" ...\\n\";\n *              });\n */\nclass http_server_control {\n    std::unique_ptr<sharded<http_server>> _server_dist;\nprivate:\n    static sstring generate_server_name();\npublic:\n    http_server_control() : _server_dist(new sharded<http_server>) {\n    }\n\n    future<> start(const sstring& name = generate_server_name());\n    future<> stop() noexcept;\n    future<> set_routes(std::function<void(routes& r)> fun);\n    future<> listen(socket_address addr);\n    future<> listen(socket_address addr, http_server::server_credentials_ptr credentials);\n    future<> listen(socket_address addr, listen_options lo);\n    future<> listen(socket_address addr, listen_options lo, http_server::server_credentials_ptr credentials);\n    sharded<http_server>& server();\n};\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/internal/content_source.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/http/chunk_parsers.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/http/common.hh>\n#include <seastar/util/log.hh>\n#include <seastar/http/exception.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nnamespace internal {\n\n/*\n * An input_stream wrapper that allows to read only \"length\" bytes\n * from it, used to handle requests with large bodies.\n * */\nclass content_length_source_impl : public data_source_impl {\n    input_stream<char>& _inp;\n    /*\n     * The _remaining_bytes references the counter of yet unread body bytes.\n     * The external counter variable is provided by users that want to track\n     * this value. If it's not the case and the external counter is not provided,\n     * this reference points to _builtin_remaining_bytes member.\n     */\n    size_t _builtin_remaining_bytes = 0;\n    size_t& _remaining_bytes;\npublic:\n    content_length_source_impl(input_stream<char>& inp, size_t length, size_t& remaining)\n        : _inp(inp), _remaining_bytes(remaining)\n    {\n        _remaining_bytes = length;\n    }\n\n    content_length_source_impl(input_stream<char>& inp, size_t length)\n        : _inp(inp), _builtin_remaining_bytes(length), _remaining_bytes(_builtin_remaining_bytes) {\n    }\n\n    virtual future<temporary_buffer<char>> get() override {\n        if (_remaining_bytes == 0) {\n            return make_ready_future<temporary_buffer<char>>();\n        }\n        return _inp.read_up_to(_remaining_bytes).then([this] (temporary_buffer<char> tmp_buf) {\n            _remaining_bytes -= tmp_buf.size();\n            return tmp_buf;\n        });\n    }\n\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        uint64_t skip_bytes = std::min(n, _remaining_bytes);\n        _remaining_bytes -= skip_bytes;\n        return _inp.skip(skip_bytes).then([] {\n            return temporary_buffer<char>();\n        });\n    }\n\n    virtual future<> close() override {\n        return make_ready_future<>();\n    }\n};\n\n/*\n * An input_stream wrapper that decodes a request body\n * with \"chunked\" encoding.\n * */\nclass chunked_source_impl : public data_source_impl {\n    class chunk_parser {\n        enum class parsing_state\n            : uint8_t {\n                size_and_ext,\n                body,\n                trailer_part\n        };\n        http_chunk_size_and_ext_parser _size_and_ext_parser;\n        http_chunk_trailer_parser _trailer_parser;\n\n        temporary_buffer<char> _buf;\n        size_t _current_chunk_bytes_read = 0;\n        size_t _current_chunk_length;\n        parsing_state _ps = parsing_state::size_and_ext;\n        bool _end_of_request = false;\n        // references to fields in the request structure\n        std::unordered_map<sstring, sstring>& _chunk_extensions;\n        std::unordered_map<sstring, sstring>& _trailing_headers;\n        /*\n         * Reference to the counter of yet unread body bytes, similar to the one\n         * from content_length_source_impl. Similarly, when the external counter is\n         * not provided it points to the _builtin_remaining_bytes, which sits on the\n         * chunked_source_impl itself, not in the chunk_parser.\n         *\n         * This counter differs from the content_length_source_impl one -- since body\n         * size is not known for chunked body, this counter has only two values. It\n         * starts with uint64_t::max and remains such up until all the body is read.\n         * Once end-of-body is met, the counter is reset to zero.\n         */\n        size_t& _remaining_bytes;\n        using consumption_result_type = consumption_result<char>;\n    public:\n        chunk_parser(std::unordered_map<sstring, sstring>& chunk_extensions, std::unordered_map<sstring, sstring>& trailing_headers, size_t& remaining)\n            : _chunk_extensions(chunk_extensions), _trailing_headers(trailing_headers), _remaining_bytes(remaining) {\n                _remaining_bytes = std::numeric_limits<size_t>::max();\n                _size_and_ext_parser.init();\n        }\n        temporary_buffer<char> buf() {\n            _current_chunk_bytes_read += _buf.size();\n            return std::move(_buf);\n        }\n\n        future<consumption_result_type> operator()(temporary_buffer<char> data) {\n            if (_buf.size() || _end_of_request || data.empty()) {\n                // return if we have already read some content (_buf.size()), we have already reached the end of the chunked request (_end_of_request),\n                // or the underlying stream reached eof (data.empty())\n                return make_ready_future<consumption_result_type>(stop_consuming(std::move(data)));\n            }\n            switch (_ps) {\n            // \"data\" buffer is non-empty\n            case parsing_state::size_and_ext:\n                return _size_and_ext_parser(std::move(data)).then([this] (std::optional<temporary_buffer<char>> res) {\n                    if (res.has_value()) {\n                        if (_size_and_ext_parser.failed()) {\n                            return make_exception_future<consumption_result_type>(bad_request_exception(\"Can't parse chunk size and extensions\"));\n                        }\n                        // save extensions\n                        auto parsed_extensions = _size_and_ext_parser.get_parsed_extensions();\n                        _chunk_extensions.merge(parsed_extensions);\n                        for (auto& key_val : parsed_extensions) {\n                            _chunk_extensions[key_val.first] += sstring(\",\") + key_val.second;\n                        }\n\n                        // save size\n                        auto size_string = _size_and_ext_parser.get_size();\n                        if (size_string.size() > 16) {\n                            return make_exception_future<consumption_result_type>(bad_chunk_exception(\"Chunk length too big\"));\n                        }\n                        _current_chunk_bytes_read = 0;\n                        _current_chunk_length = strtol(size_string.c_str(), nullptr, 16);\n\n                        if (_current_chunk_length == 0) {\n                            _ps = parsing_state::trailer_part;\n                            _trailer_parser.init();\n                        } else {\n                            _ps = parsing_state::body;\n                        }\n                        if (res->empty()) {\n                            return make_ready_future<consumption_result_type>(continue_consuming{});\n                        }\n                        return this->operator()(std::move(res.value()));\n                    } else {\n                        return make_ready_future<consumption_result_type>(continue_consuming{});\n                    }\n                });\n            case parsing_state::body:\n                // read the new data into _buf\n                if (_current_chunk_bytes_read < _current_chunk_length) {\n                    size_t to_read = std::min(_current_chunk_length - _current_chunk_bytes_read, data.size());\n                    if (_buf.empty()) {\n                        _buf = data.share(0, to_read);\n                    }\n                    data.trim_front(to_read);\n                    return make_ready_future<consumption_result_type>(stop_consuming(std::move(data)));\n                }\n\n                // chunk body is finished, we haven't entered the previous if, so \"data\" is still non-empty\n                if (_current_chunk_bytes_read == _current_chunk_length) {\n                    // we haven't read \\r yet\n                    if (data.get()[0] != '\\r') {\n                        return make_exception_future<consumption_result_type>(bad_chunk_exception(\"The actual chunk length exceeds the specified length\"));\n                    } else {\n                        _current_chunk_bytes_read++;\n                        data.trim_front(1);\n                        if (data.empty()) {\n                            return make_ready_future<consumption_result_type>(continue_consuming{});\n                        }\n                    }\n                }\n                if (_current_chunk_bytes_read == _current_chunk_length + 1) {\n                    // we haven't read \\n but have \\r\n                    if (data.get()[0] != '\\n') {\n                        return make_exception_future<consumption_result_type>(bad_chunk_exception(\"The actual chunk length exceeds the specified length\"));\n                    } else {\n                        _ps = parsing_state::size_and_ext;\n                        _size_and_ext_parser.init();\n                        data.trim_front(1);\n                        if (data.empty()) {\n                            return make_ready_future<consumption_result_type>(continue_consuming{});\n                        }\n                    }\n                }\n                return this->operator()(std::move(data));\n            case parsing_state::trailer_part:\n                return _trailer_parser(std::move(data)).then([this] (std::optional<temporary_buffer<char>> res) {\n                    if (res.has_value()) {\n                        if (_trailer_parser.failed()) {\n                            return make_exception_future<consumption_result_type>(bad_request_exception(\"Can't parse chunked request trailer\"));\n                        }\n                        // save trailing headers\n                        _trailing_headers = _trailer_parser.get_parsed_headers();\n                        _end_of_request = true;\n                        _remaining_bytes = 0;\n                        return make_ready_future<consumption_result_type>(stop_consuming(std::move(*res)));\n                    } else {\n                        return make_ready_future<consumption_result_type>(continue_consuming{});\n                    }\n                });\n            }\n            __builtin_unreachable();\n        }\n    };\n    input_stream<char>& _inp;\n    chunk_parser _chunk;\n    size_t _builtin_remaining_bytes = 0;\n\npublic:\n    chunked_source_impl(input_stream<char>& inp, std::unordered_map<sstring, sstring>& chunk_extensions, std::unordered_map<sstring, sstring>& trailing_headers, size_t& remaining)\n        : _inp(inp), _chunk(chunk_extensions, trailing_headers, remaining) {\n    }\n\n    chunked_source_impl(input_stream<char>& inp, std::unordered_map<sstring, sstring>& chunk_extensions, std::unordered_map<sstring, sstring>& trailing_headers)\n        : _inp(inp), _chunk(chunk_extensions, trailing_headers, _builtin_remaining_bytes) {\n    }\n\n    virtual future<temporary_buffer<char>> get() override {\n        return _inp.consume(_chunk).then([this] () mutable {\n            return _chunk.buf();\n        });\n    }\n\n    virtual future<> close() override {\n        return make_ready_future<>();\n    }\n};\n\n} // namespace internal\n\n} // namespace httpd\n\n}\n"
  },
  {
    "path": "include/seastar/http/json_path.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <vector>\n\n#include <seastar/http/common.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/http/routes.hh>\n#include <seastar/http/function_handlers.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * A json_operation contain a method and a nickname.\n * operation are associated to a path, that can\n * have multiple methods\n */\nstruct json_operation {\n    /**\n     * default constructor\n     */\n    json_operation()\n            : method(GET) {\n    }\n\n    /**\n     * Construct with assignment\n     * @param method the http method type\n     * @param nickname the http nickname\n     */\n    json_operation(operation_type method, const sstring& nickname)\n            : method(method), nickname(nickname) {\n    }\n\n    operation_type method;\n    sstring nickname;\n\n};\n\n/**\n * path description holds the path in the system.\n * It maps a nickname to an operation, which allows\n * defining the operation (path and method) by its\n * nickname.\n *\n * A path_description has a type, a base path and a list of\n * url components.\n * Each component can be a regular path parameter, a path parameter that\n * contains everything until the end of the path or a fixed string.\n *\n * the description are taken from the json swagger\n * definition file, during auto code generation in the\n * compilation.\n */\nstruct path_description {\n    //\n    enum class url_component_type {\n        PARAM, // a normal path parameter (starts with / and end with / or end of path)\n        PARAM_UNTIL_END_OF_PATH, // a parameter that contains all the path entil its end\n        FIXED_STRING, // a fixed string inside the path, must be a full match and does not count\n                      // as a parameter\n    };\n\n    // path_part is either a parameter or a fixed string\n    struct path_part {\n        sstring name;\n        url_component_type type = url_component_type::PARAM;\n    };\n\n    /**\n     * default empty constructor\n     */\n    path_description() = default;\n\n    /**\n     * constructor for path with parameters\n     * The constructor is used by\n     * @param path the url path\n     * @param method the http method\n     * @param nickname the nickname\n     * @param path_parameters path parameters and url parts of the path\n     * @param mandatory_params the names of the mandatory query parameters\n     */\n    path_description(const sstring& path, operation_type method,\n            const sstring& nickname,\n            const std::vector<std::pair<sstring, bool>>& path_parameters,\n            const std::vector<sstring>& mandatory_params);\n\n    /**\n     * constructor for path with parameters\n     * The constructor is used by\n     * @param path the url path\n     * @param method the http method\n     * @param nickname the method nickname\n     * @param path_parameters path parameters and url parts of the path\n     * @param mandatory_params the names of the mandatory query parameters\n     */\n    path_description(const sstring& path, operation_type method,\n            const sstring& nickname,\n            const std::initializer_list<path_part>& path_parameters,\n            const std::vector<sstring>& mandatory_params);\n\n    /**\n     * Add a parameter to the path definition\n     * for example, if the url should match /file/{path}\n     * The constructor would be followed by a call to\n     * pushparam(\"path\")\n     *\n     * @param param the name of the parameters, this name will\n     * be used by the handler to identify the parameters.\n     * A name can appear at most once in a description\n     * @param all_path when set to true the parameter will assume to match\n     * until the end of the url.\n     * This is useful for situation like file path with\n     * a rule like /file/{path} and a url /file/etc/hosts.\n     * path should be equal to /ets/hosts and not only /etc\n     * @return the current path description\n     */\n    path_description* pushparam(const sstring& param,\n    bool all_path = false) {\n        params.push_back( { param, (all_path) ? url_component_type::PARAM_UNTIL_END_OF_PATH : url_component_type::PARAM});\n        return this;\n    }\n\n    /*!\n     * \\brief adds a fixed string as part of the path\n     * This will allow to combine fixed string URL parts and path parameters.\n     *\n     * For example to map a path like:\n     * /mypath/{param1}/morepath/{param2}\n     * path_description p(\"/mypath\", operation_type::GET);\n     * p.pushparam(\"param1)->pushurl(\"morepath\")->pushparam(\"param2\");\n     */\n    path_description* push_static_path_part(const sstring& url) {\n        params.push_back( { url, url_component_type::FIXED_STRING});\n        return this;\n    }\n    /**\n     * adds a mandatory query parameter to the path\n     * this parameter will be check before calling a handler\n     * @param param the parameter to head\n     * @return a pointer to the current path description\n     */\n    path_description* pushmandatory_param(const sstring& param) {\n        mandatory_queryparams.push_back(param);\n        return this;\n    }\n\n    std::vector<path_part> params;\n    sstring path;\n    json_operation operations;\n    mutable routes::rule_cookie _cookie;\n\n    std::vector<sstring> mandatory_queryparams;\n\n    void set(routes& _routes, handler_base* handler) const;\n\n    void set(routes& _routes, const json_request_function& f) const;\n\n    void set(routes& _routes, const future_json_function& f) const;\n\n    void unset(routes& _routes) const;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/matcher.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/http/common.hh>\n\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * a base class for the url matching.\n * Each implementation check if the given url matches a criteria\n */\nclass matcher {\npublic:\n\n    virtual ~matcher() = default;\n\n    /**\n     * check if the given url matches the rule\n     * @param url the url to check\n     * @param ind the position to start from\n     * @param param fill the parameters hash\n     * @return the end of of the matched part, or sstring::npos if not matched\n     */\n    virtual size_t match(const sstring& url, size_t ind, parameters& param) = 0;\n};\n\n/**\n * Check if the url match a parameter and fill the parameters object\n *\n * Note that a non empty url will always return true with the parameters\n * object filled\n *\n * Assume that the rule is /file/{path}/ and the param_matcher identify\n * the /{path}\n *\n * For all non empty values, match will return true.\n * If the entire url is /file/etc/hosts, and the part that is passed to\n * param_matcher is /etc/hosts, if entire_path is true, the match will be\n * '/etc/hosts' If entire_path is false, the match will be '/etc'\n */\nclass param_matcher : public matcher {\npublic:\n    /**\n     * Constructor\n     * @param name the name of the parameter, will be used as the key\n     * in the parameters object\n     * @param entire_path when set to true, the matched parameters will\n     * include all the remaining url until the end of it.\n     * when set to false the match will terminate at the next slash\n     */\n    explicit param_matcher(const sstring& name, bool entire_path = false)\n            : _name(name), _entire_path(entire_path) {\n    }\n\n    virtual size_t match(const sstring& url, size_t ind, parameters& param)\n            override;\nprivate:\n    sstring _name;\n    bool _entire_path;\n};\n\n/**\n * Check if the url match a predefine string.\n *\n * When parsing a match rule such as '/file/{path}' the str_match would parse\n * the '/file' part\n */\nclass str_matcher : public matcher {\npublic:\n    /**\n     * Constructor\n     * @param cmp the string to match\n     */\n    explicit str_matcher(const sstring& cmp)\n            : _cmp(cmp), _len(cmp.size()) {\n    }\n\n    virtual size_t match(const sstring& url, size_t ind, parameters& param)\n            override;\nprivate:\n    sstring _cmp;\n    unsigned _len;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/matchrules.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/http/handlers.hh>\n#include <seastar/http/matcher.hh>\n#include <seastar/http/common.hh>\n\n#include <seastar/core/sstring.hh>\n#include <vector>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * match_rule check if a url matches criteria, that can contains\n * parameters.\n * the routes object would call the get method with a url and if\n * it matches, the method will return a handler\n * during the matching process, the method fill the parameters object.\n */\nclass match_rule {\npublic:\n    /**\n     * The destructor deletes matchers.\n     */\n    ~match_rule() {\n        for (auto m : _match_list) {\n            delete m;\n        }\n        delete _handler;\n    }\n\n    /**\n     * Constructor with a handler\n     * @param handler the handler to return when this match rule is met\n     */\n    explicit match_rule(handler_base* handler)\n            : _handler(handler) {\n    }\n\n    /**\n     * Check if url match the rule and return a handler if it does\n     * @param url a url to compare against the rule\n     * @param params the parameters object, matches parameters will fill\n     * the object during the matching process\n     * @return a handler if there is a full match or nullptr if not\n     */\n    handler_base* get(const sstring& url, parameters& params) {\n        size_t ind = 0;\n        if (_match_list.empty()) {\n            return _handler;\n        }\n        for (unsigned int i = 0; i < _match_list.size(); i++) {\n            ind = _match_list.at(i)->match(url, ind, params);\n            if (ind == sstring::npos) {\n                return nullptr;\n            }\n        }\n        return (ind + 1 >= url.length()) ? _handler : nullptr;\n    }\n\n    /**\n     * Add a matcher to the rule\n     * @param match the matcher to add\n     * @return this\n     */\n    match_rule& add_matcher(matcher* match) {\n        _match_list.push_back(match);\n        return *this;\n    }\n\n    /**\n     * Add a static url matcher\n     * @param str the string to search for\n     * @return this\n     */\n    match_rule& add_str(const sstring& str) {\n        add_matcher(new str_matcher(str));\n        return *this;\n    }\n\n    /**\n     * add a parameter matcher to the rule\n     * @param str the parameter name\n     * @param fullpath when set to true, parameter will included all the\n     * remaining url until its end\n     * @return this\n     */\n    match_rule& add_param(const sstring& str, bool fullpath = false) {\n        add_matcher(new param_matcher(str, fullpath));\n        return *this;\n    }\n\nprivate:\n    std::vector<matcher*> _match_list;\n    handler_base* _handler;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/mime_types.hh",
    "content": "//\n// mime_types.hpp\n// ~~~~~~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n//\n\n#pragma once\n\n#include <string_view>\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\n\nnamespace http {\n\nnamespace mime_types {\n\n/**\n * Convert a file extension into a MIME type.\n *\n * @param extension the file extension\n * @return the mime type as a string\n */\nconst char* extension_to_type(std::string_view extension);\n\n} // namespace mime_types\n\n} // namespace httpd\n\nnamespace httpd {\nnamespace mime_types {\n[[deprecated(\"Use http::mime_types::extension_to_type instead\")]]\ninline const char* extension_to_type(const sstring& extension) {\n    return http::mime_types::extension_to_type(extension);\n}\n}\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/reply.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n// This file was modified from boost http example\n//\n// reply.hpp\n// ~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n//\n#pragma once\n\n#include <unordered_map>\n#include <string_view>\n#include <seastar/core/sstring.hh>\n#include <seastar/http/mime_types.hh>\n#include <seastar/http/types.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/util/string_utils.hh>\n#include <seastar/util/iostream.hh>\n\nnamespace seastar {\n\n\nnamespace httpd {\n\nclass connection;\nclass routes;\n\n}\n\nnamespace http {\n\n/**\n * A reply to be sent to a client.\n */\nstruct reply {\n    /**\n     * The status of the reply.\n     */\n    enum class status_type {\n        continue_ = 100, //!< continue\n        switching_protocols = 101, //!< switching_protocols\n        ok = 200, //!< ok\n        created = 201, //!< created\n        accepted = 202, //!< accepted\n        nonauthoritative_information = 203, //!< nonauthoritative_information\n        no_content = 204, //!< no_content\n        reset_content = 205, //!< reset_content\n        partial_content = 206, //! partial_content\n        multiple_choices = 300, //!< multiple_choices\n        moved_permanently = 301, //!< moved_permanently\n        moved_temporarily = 302, //!< moved_temporarily\n        see_other = 303, //!< see_other\n        not_modified = 304, //!< not_modified\n        use_proxy = 305, //!< use_proxy\n        temporary_redirect = 307, //!< temporary_redirect\n        permanent_redirect = 308, //!< permanent_redirect\n        bad_request = 400, //!< bad_request\n        unauthorized = 401, //!< unauthorized\n        payment_required = 402, //!< payment_required\n        forbidden = 403, //!< forbidden\n        not_found = 404, //!< not_found\n        method_not_allowed = 405, //!< method_not_allowed\n        not_acceptable = 406, //!< not_acceptable\n        request_timeout = 408, //!< request_timeout\n        conflict = 409, //!< conflict\n        gone = 410, //!< gone\n        length_required = 411, //!< length_required\n        payload_too_large = 413, //!< payload_too_large\n        uri_too_long = 414, //!< uri_too_long\n        unsupported_media_type = 415, //!< unsupported_media_type\n        expectation_failed = 417, //!< expectation_failed\n        page_expired = 419, //!< page_expired\n        unprocessable_entity = 422, //!< unprocessable_entity\n        upgrade_required = 426, //!< upgrade_required\n        too_many_requests = 429, //!< too_many_requests\n        login_timeout = 440, //!< login_timeout\n        internal_server_error = 500, //!< internal_server_error\n        not_implemented = 501, //!< not_implemented\n        bad_gateway = 502, //!< bad_gateway\n        service_unavailable = 503,  //!< service_unavailable\n        gateway_timeout = 504, //!< gateway_timeout\n        http_version_not_supported = 505, //!< http_version_not_supported\n        insufficient_storage = 507, //!< insufficient_storage\n        bandwidth_limit_exceeded = 509, //!< bandwidth_limit_exceeded\n        network_read_timeout = 598, //!< network_read_timeout\n        network_connect_timeout = 599, //!< network_connect_timeout\n    } _status;\n\n    /**\n     * HTTP status classes\n     * See https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml\n     *\n     * 1xx: Informational - Request received, continuing process\n     * 2xx: Success - The action was successfully received, understood, and accepted\n     * 3xx: Redirection - Further action must be taken in order to complete the request\n     * 4xx: Client Error - The request contains bad syntax or cannot be fulfilled\n     * 5xx: Server Error - The server failed to fulfill an apparently valid request\n     */\n    enum class status_class : uint8_t {\n        informational = 1,\n        success = 2,\n        redirection = 3,\n        client_error = 4,\n        server_error = 5,\n        unclassified\n    };\n\n    /**\n     * Classify http status\n     * @param http_status the http status \\ref status_type\n     * @return one of the \\ref status_class values\n     */\n    static constexpr status_class classify_status(status_type http_status) {\n        auto sc = static_cast<std::underlying_type_t<status_type>>(http_status) / 100;\n        if (sc < 1 || sc > 5) [[unlikely]] {\n            return status_class::unclassified;\n        }\n        return static_cast<status_class>(sc);\n    }\n\n    /**\n     * The headers to be included in the reply.\n     */\n    std::unordered_map<sstring, sstring, seastar::internal::case_insensitive_hash, seastar::internal::case_insensitive_cmp> _headers;\n\n    std::unordered_map<sstring, sstring> _cookies;\n\n    sstring _version;\n    /**\n     * The content to be sent in the reply.\n     */\n    sstring _content;\n    size_t content_length = 0; // valid when received via client connection\n    size_t left_content_length = std::numeric_limits<size_t>::max();\n    bool _skip_body = false;\n\n    sstring _response_line;\n    std::unordered_map<sstring, sstring> trailing_headers;\n    std::unordered_map<sstring, sstring> chunk_extensions;\n\n    reply()\n            : _status(status_type::ok) {\n    }\n\n    reply& add_header(const sstring& h, const sstring& value) {\n        _headers[h] = value;\n        return *this;\n    }\n\n    /**\n     * Search for the first header of a given name\n     * @param name the header name\n     * @return a pointer to the header value, if it exists or empty string\n     */\n    sstring get_header(const sstring& name) const {\n        auto res = _headers.find(name);\n        if (res == _headers.end()) {\n            return \"\";\n        }\n        return res->second;\n    }\n\n    reply& set_version(const sstring& version) {\n        _version = version;\n        return *this;\n    }\n\n    reply& set_status(status_type status, sstring content = \"\") {\n        _status = status;\n        if (content != \"\") {\n            _content = std::move(content);\n        }\n        return *this;\n    }\n\n    /**\n     * Set the content type. The content_type string can be one of:\n     * 1. A MIME (RFC 2045) Content-Type - looking like \"type/subtype\", e.g.,\n     *   \"text/html\"\n     * 2. If a \"/\" is missing in the given string, we look it up in a list of\n     *    common file extensions listed in the http::mime_types map. For\n     *    example \"html\" will be mapped to \"text/html\".\n     */\n    reply& set_content_type(std::string_view content_type) {\n        if (content_type.find('/') == std::string_view::npos) {\n            content_type = http::mime_types::extension_to_type(content_type);\n        }\n        _headers[\"Content-Type\"] = sstring(content_type);\n        return *this;\n    }\n\n    [[deprecated(\"Use set_content_type(std::string_view) instead\")]]\n    reply& set_content_type() {\n        return set_content_type(\"html\");\n    }\n\n    [[deprecated(\"Use set_content_type(std::string_view) instead\")]]\n    reply& set_mime_type(const sstring& mime) {\n        return set_content_type(mime);\n    }\n\n    /**\n     * Set the cookie with the given name and value.\n     */\n    reply& set_cookie(const sstring& name, const sstring& value) {\n        _cookies[name] = value;\n        return *this;\n    }\n\n    reply& done(const sstring& content_type) {\n        return set_content_type(content_type).done();\n    }\n    /**\n     * Done should be called before using the reply.\n     * It would set the response line\n     */\n    reply& done() {\n        _response_line = response_line();\n        return *this;\n    }\n    sstring response_line() const;\n\n    /*!\n     * \\brief use an output stream to write the message body\n     *\n     * When a handler needs to use an output stream it should call this method\n     * with a function.\n     *\n     * \\param content_type - is used to choose the content type of the body. Use the file extension\n     *  you would have used for such a content, (i.e. \"txt\", \"html\", \"json\", etc')\n     * \\param body_writer - a function that accept an output stream and use that stream to write the body.\n     *   The function should take ownership of the stream while using it and must close the stream when it\n     *   is done.\n     *\n     * Message would use chunked transfer encoding in the reply.\n     *\n     */\n\n    void write_body(const sstring& content_type, http::body_writer_type&& body_writer);\n\n    /*!\n     * \\brief use and output stream to write the message body\n     *\n     * The same as above, but the handler can only .write() data into the stream, it will be\n     * closed (and flushed) automatically after the writer fn resolves\n     */\n    template <typename W>\n    requires std::is_invocable_r_v<future<>, W, output_stream<char>&>\n    void write_body(const sstring& content_type, W&& body_writer) {\n        write_body(content_type, [body_writer = std::move(body_writer)] (output_stream<char>&& out) mutable -> future<> {\n            return util::write_to_stream_and_close(std::move(out), std::move(body_writer));\n        });\n    }\n\n    /*!\n     * \\brief Write a string as the reply\n     *\n     * \\param content_type - is used to choose the content type of the body. Use the file extension\n     *  you would have used for such a content, (i.e. \"txt\", \"html\", \"json\", etc')\n     * \\param content - the message content.\n     * This would set the the content and content type of the message along\n     * with any additional information that is needed to send the message.\n     */\n    void write_body(const sstring& content_type, sstring content);\n\n    // RFC7231 Sec. 4.3.2\n    // For HEAD replies collect everything from the handler, but don't write the body itself\n    void skip_body() noexcept {\n        _skip_body = true;\n    }\n\n    future<> write_reply(output_stream<char>& out);\n    future<> write_reply_headers(output_stream<char>& out);\n\nprivate:\n    http::body_writer_type _body_writer;\n    friend class httpd::routes;\n};\n\nstd::ostream& operator<<(std::ostream& os, reply::status_type st);\n\n} // namespace http\n\nnamespace httpd {\nusing reply [[deprecated(\"Use http::reply instead\")]] = http::reply;\n}\n\n}\n\ntemplate <> struct fmt::formatter<seastar::http::reply::status_type> : fmt::ostream_formatter {};\n"
  },
  {
    "path": "include/seastar/http/request.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n//\n// request.hpp\n// ~~~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n//\n#pragma once\n\n#include <seastar/core/iostream.hh>\n#include <seastar/core/sstring.hh>\n#include <string_view>\n#include <strings.h>\n#include <seastar/http/common.hh>\n#include <seastar/http/mime_types.hh>\n#include <seastar/http/types.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/util/string_utils.hh>\n#include <seastar/util/iostream.hh>\n\nnamespace seastar {\n\nnamespace http {\n\nnamespace experimental { class connection; }\n\n/**\n * A request received from a client.\n */\nstruct request {\n    enum class ctclass\n        : char {\n            other, multipart, app_x_www_urlencoded,\n    };\n\n    socket_address _client_address;\n    socket_address _server_address;\n    sstring _method;\n    sstring _url;\n    sstring _version;\n    ctclass content_type_class;\n    size_t content_length = 0;\n    mutable size_t _bytes_written = 0;\n    std::unordered_map<sstring, sstring, seastar::internal::case_insensitive_hash, seastar::internal::case_insensitive_cmp> _headers;\n    // deprecated: it is used to store last value of query parameters, but will be removed in the future\n    [[deprecated(\"Use helper methods instead\")]] std::unordered_map<sstring, sstring> query_parameters;\n    httpd::parameters param;\n    [[deprecated(\"use content_stream (server-side) / write_body (client-side) instead\")]]\n    sstring content; // server-side deprecated: use content_stream instead\n    /*\n     * The handler should read the contents of this stream till reaching eof (i.e., the end of this request's content). Failing to do so\n     * will force the server to close this connection, and the client will not be able to reuse this connection for the next request.\n     * The stream should not be closed by the handler, the server will close it for the handler.\n     * */\n    input_stream<char>* content_stream;\n    std::unordered_map<sstring, sstring> trailing_headers;\n    std::unordered_map<sstring, sstring> chunk_extensions;\n    sstring protocol_name = \"http\";\n    http::body_writer_type body_writer; // for client\n\n    using query_parameters_type = std::unordered_map<sstring, std::vector<sstring>, seastar::internal::string_view_hash, std::equal_to<>>;\nprivate:\n    query_parameters_type _query_params;\npublic:\n\n// NOTE: Remove this once both `query_parameters` and `content` are removed\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n    request() = default;\n    request(request&&) = default;\n    request& operator=(request&&) = default;\n    ~request() = default;\n#pragma GCC diagnostic pop\n\n    /**\n     * Get the address of the client that generated the request\n     * @return The address of the client that generated the request\n     */\n    const socket_address & get_client_address() const {\n        return _client_address;\n    }\n\n    /**\n     * Get the address of the server that handled the request\n     * @return The address of the server that handled the request\n     */\n    const socket_address & get_server_address() const {\n        return _server_address;\n    }\n\n    /**\n     * Search for the first header of a given name\n     * @param name the header name\n     * @return a pointer to the header value, if it exists or empty string\n     */\n    sstring get_header(const sstring& name) const {\n        auto res = _headers.find(name);\n        if (res == _headers.end()) {\n            return \"\";\n        }\n        return res->second;\n    }\n\n    /**\n     * Does the query parameters contain a given key?\n     * @param key the query parameter key\n     * @return true if the key exists, false otherwise\n     */\n    bool has_query_param(std::string_view key) const {\n        return _query_params.contains(key);\n    }\n\n    /**\n     * Search for the last query parameter of a given key\n     * @param key the query parameter key\n     * @return the query parameter value, if it exists or the default_value otherwise\n     */\n    sstring get_query_param(std::string_view key, std::string_view default_value = \"\") const {\n        auto res = _query_params.find(key);\n        if (res != _query_params.end()) {\n            return res->second.back();\n        }\n\n        return sstring(default_value);\n    }\n\n    /**\n     * Search for all query parameters of a given key\n     * @param key the query parameter key\n     * @return a vector of all query parameter values, if it exists or an empty vector\n     */\n    const std::vector<sstring>& get_query_param_array(std::string_view key) const {\n        if (auto res = _query_params.find(key); res != _query_params.end()) {\n            return res->second;\n        }\n        static const std::vector<sstring> empty_vector;\n        return empty_vector;\n    }\n\n    /**\n     * Get all query parameters\n     * @return a map of all query parameters\n     */\n    const query_parameters_type& get_query_params() const {\n        return _query_params;\n    }\n\n    /**\n     * Set a query parameter value\n     * @param key the query parameter key\n     * @param value the query parameter value\n     * @return a reference to this request object\n     */\n    request& set_query_param(std::string_view key, std::string_view value) {\n        return set_query_param(key, {value});\n    }\n\n    /**\n     * Set a query parameter value\n     * @param key the query parameter key\n     * @param values the query parameter values\n     * @return a reference to this request object\n    */\n    request& set_query_param(std::string_view key, std::initializer_list<std::string_view> values) {\n        _query_params[sstring(key)] = std::vector<sstring>(values.begin(), values.end());\n        return *this;\n    }\n\n    /**\n     * Set a query parameter value\n     * @param key the query parameter key\n     * @param values the query parameter values\n     * @return a reference to this request object\n    */\n    request& set_query_param(std::string_view key, std::vector<sstring> values) {\n        _query_params[sstring(key)] = std::move(values);\n        return *this;\n    }\n\n    /**\n     * Set the query parameters in the request objects.\n     * @param params a map of query parameters\n     * @return a reference to this request object\n     */\n    request& set_query_params(query_parameters_type params) {\n        _query_params = std::move(params);\n        return *this;\n    }\n\n    /**\n     * Search for the last path parameter of a given key\n     * @param key the path paramerter key\n     * @return the unescaped path parameter value, if it exists and can be path decoded successfully, otherwise it\n     *  returns an empty string\n     */\n    sstring get_path_param(const sstring& key) const {\n        return param.get_decoded_param(key);\n    }\n\n    /**\n     * Get the request protocol name. Can be either \"http\" or \"https\".\n     */\n    sstring get_protocol_name() const {\n        return protocol_name;\n    }\n\n    /**\n     * Get the request url.\n     * @return the request url\n     */\n    sstring get_url() const {\n        return get_protocol_name() + \"://\" + get_header(\"Host\") + _url;\n    }\n\n    bool is_multi_part() const {\n        return content_type_class == ctclass::multipart;\n    }\n\n    bool is_form_post() const {\n        return content_type_class == ctclass::app_x_www_urlencoded;\n    }\n\n    bool should_keep_alive() const {\n        if (_version == \"0.9\") {\n            return false;\n        }\n\n        // TODO: handle HTTP/2.0 when it releases\n\n        auto it = _headers.find(\"Connection\");\n        if (_version == \"1.0\") {\n            return it != _headers.end()\n                 && seastar::internal::case_insensitive_cmp()(it->second, \"keep-alive\");\n        } else { // HTTP/1.1\n            return it == _headers.end() || !seastar::internal::case_insensitive_cmp()(it->second, \"close\");\n        }\n    }\n\n    /**\n     * Set the query parameters in the request objects.\n     * Returns the URL path part, i.e. -- without the query paremters\n     * query param appear after the question mark and are separated\n     * by the ampersand sign\n     */\n    sstring parse_query_param();\n\n    /**\n     * Generates the URL string from the _url and query_parameters\n     * values in a form parseable by the above method\n     */\n    sstring format_url() const;\n\n    /**\n     * Set the content type. The content_type string can be one of:\n     * 1. A MIME (RFC 2045) Content-Type - looking like \"type/subtype\", e.g.,\n     *   \"text/html\"\n     * 2. If a \"/\" is missing in the given string, we look it up in a list of\n     *    common file extensions listed in the http::mime_types map. For\n     *    example \"html\" will be mapped to \"text/html\".\n     */\n    void set_content_type(std::string_view content_type) {\n        if (content_type.find('/') == std::string_view::npos) {\n            content_type = http::mime_types::extension_to_type(content_type);\n        }\n        _headers[\"Content-Type\"] = sstring(content_type);\n    }\n\n    [[deprecated(\"Use set_content_type(std::string_view) instead\")]]\n    void set_content_type() {\n        set_content_type(\"html\");\n    }\n\n    [[deprecated(\"Use set_content_type(std::string_view) instead\")]]\n    void set_mime_type(const sstring& mime) {\n        set_content_type(mime);\n    }\n\n    /**\n     * \\brief Write a string as the body\n     *\n     * \\param content_type - is used to choose the content type of the body. Use the file extension\n     *  you would have used for such a content, (i.e. \"txt\", \"html\", \"json\", etc')\n     * \\param content - the message content.\n     * This would set the the content, conent length and content type of the message along\n     * with any additional information that is needed to send the message.\n     *\n     * This method is good to be used if the body is available as a contiguous buffer.\n     */\n    void write_body(const sstring& content_type, sstring content);\n\n    /**\n     * \\brief Use an output stream to write the message body\n     *\n     * When a handler needs to use an output stream it should call this method\n     * with a function.\n     *\n     * \\param content_type - is used to choose the content type of the body. Use the file extension\n     *  you would have used for such a content, (i.e. \"txt\", \"html\", \"json\", etc')\n     * \\param body_writer - a function that accept an output stream and use that stream to write the body.\n     *   The function should take ownership of the stream while using it and must close the stream when it\n     *   is done.\n     *\n     * This method can be used to write body of unknown or hard to evaluate length. For example,\n     * when sending the contents of some other input_stream or when the body is available as a\n     * collection of memory buffers. Message would use chunked transfer encoding.\n     *\n     */\n    void write_body(const sstring& content_type, http::body_writer_type&& body_writer);\n\n    /*!\n     * \\brief use and output stream to write the message body\n     *\n     * The same as above, but the caller can only .write() data into the stream, it will be\n     * closed (and flushed) automatically after the writer fn resolves\n     */\n    template <typename W>\n    requires std::is_invocable_r_v<future<>, W, output_stream<char>&>\n    void write_body(const sstring& content_type, W&& body_writer) {\n        write_body(content_type, [body_writer = std::move(body_writer)] (output_stream<char>&& out) mutable -> future<> {\n            return util::write_to_stream_and_close(std::move(out), std::move(body_writer));\n        });\n    }\n\n    /**\n     * \\brief Use an output stream to write the message body\n     *\n     * When a handler needs to use an output stream it should call this method\n     * with a function.\n     *\n     * \\param content_type - is used to choose the content type of the body. Use the file extension\n     *  you would have used for such a content, (i.e. \"txt\", \"html\", \"json\", etc')\n     * \\param len - known in advance content length\n     * \\param body_writer - a function that accept an output stream and use that stream to write the body.\n     *   The function should take ownership of the stream while using it and must close the stream when it\n     *   is done.\n     *\n     * This method is to be used when the body is not available of a single contiguous buffer, but the\n     * size of it is known and it's desirable to provide it to the server, or when the server strongly\n     * requires the content-length header for any reason.\n     *\n     * Message would use plain encoding in the the reply with Content-Length header set accordingly.\n     * If the body_writer doesn't generate enough bytes into the stream or tries to put more data into\n     * the stream, sending the request would resolve with exceptional future.\n     *\n     */\n    void write_body(const sstring& content_type, size_t len, http::body_writer_type&& body_writer);\n\n    /*!\n     * \\brief use and output stream to write the message body\n     *\n     * The same as above, but the caller can only .write() data into the stream, it will be\n     * closed (and flushed) automatically after the writer fn resolves\n     */\n    template <typename W>\n    requires std::is_invocable_r_v<future<>, W, output_stream<char>&>\n    void write_body(const sstring& content_type, size_t len, W&& body_writer) {\n        write_body(content_type, len, [body_writer = std::move(body_writer)] (output_stream<char>&& out) mutable -> future<> {\n            return util::write_to_stream_and_close(std::move(out), std::move(body_writer));\n        });\n    }\n\n    /**\n     * \\brief Make request send Expect header\n     *\n     * When set, the connection::make_request will send the Expect header and will wait for the\n     * server resply before tranferring the body\n     *\n     */\n    void set_expects_continue();\n\n    /**\n     * \\brief Make simple request\n     *\n     * \\param method - method to use, e.g. \"GET\" or \"POST\"\n     * \\param host - host to contact. This value will be used as the \"Host\" header\n     * \\path - the URL to send the request to\n     *\n     */\n    static request make(sstring method, sstring host, sstring path);\n\n    /**\n     * \\brief Make simple request\n     *\n     * \\param method - method to use, e.g. operation_type::GET\n     * \\param host - host to contact. This value will be used as the \"Host\" header\n     * \\path - the URL to send the request to\n     *\n     */\n    static request make(httpd::operation_type type, sstring host, sstring path);\n\n    sstring request_line() const;\n    future<> write_request_headers(output_stream<char>& out) const;\nprivate:\n    void add_query_param(std::string_view param);\n    friend class experimental::connection;\n};\n\nnamespace internal {\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\ninline sstring& deprecated_content(request& req) noexcept { return req.content; }\ninline const sstring& deprecated_content(const request& req) noexcept { return req.content; }\n#pragma GCC diagnostic pop\n}\n\n} // namespace httpd\n\nnamespace httpd {\nusing request [[deprecated(\"Use http::request instead\")]] = http::request;\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/retry_strategy.hh",
    "content": "/*\n * Copyright (C) 2025-present ScyllaDB\n */\n\n/*\n * SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0\n */\n\n#pragma once\n#include <seastar/core/future.hh>\n#include <seastar/http/reply.hh>\n\nnamespace seastar {\n\nnamespace http::experimental {\n\nclass retry_strategy {\npublic:\n    virtual ~retry_strategy() = default;\n    // Returns true if the error can be retried given the error and the number of times already tried.\n    virtual future<bool> should_retry(std::exception_ptr error, unsigned attempted_retries) const = 0;\n};\n\nclass default_retry_strategy final : public retry_strategy {\n    unsigned _max_retries;\n\npublic:\n    default_retry_strategy();\n    explicit default_retry_strategy(unsigned max_retries);\n\n    future<bool> should_retry(std::exception_ptr error, unsigned attempted_retries) const override;\n};\n\nclass no_retry_strategy final : public retry_strategy {\npublic:\n    future<bool> should_retry(std::exception_ptr error, unsigned attempted_retries) const override;\n};\n} // namespace http::experimental\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/http/routes.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <boost/program_options/variables_map.hpp>\n#include <unordered_map>\n\n#include <seastar/http/matchrules.hh>\n#include <seastar/http/handlers.hh>\n#include <seastar/http/common.hh>\n#include <seastar/http/reply.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * The url helps defining a route.\n */\nclass url {\npublic:\n    /**\n     * Move constructor\n     */\n    url(url&&) = default;\n\n    /**\n     * Construct with a url path as it's parameter\n     * @param path the url path to be used\n     */\n    url(const sstring& path)\n            : _path(path) {\n    }\n\n    /**\n     * Adds a parameter that matches untill the end of the URL.\n     * @param param the parmaeter name\n     * @return the current url\n     */\n    url& remainder(const sstring& param) {\n        this->_param = param;\n        return *this;\n    }\n\n    sstring _path;\n    sstring _param;\n};\n\nstruct path_description;\n\n/**\n * routes object do the request dispatching according to the url.\n * It uses two decision mechanism exact match, if a url matches exactly\n * (an optional leading slash is permitted) it is chosen\n * If not, the matching rules are used.\n * matching rules are evaluated by their insertion order\n */\nclass routes {\npublic:\n    /**\n     * The destructor deletes the match rules and handlers\n     */\n    ~routes();\n\n    /**\n     * adding a handler as an exact match\n     * @param url the url to match (note that url should start with /)\n     * @param handler the desire handler\n     * @return itself\n     * @attention If successful, this method takes ownership of the handler\n        pointer. It will be automatically deleted when the routes instance is\n        destroyed. However, if the method throws, the caller is responsible\n        for deleting the handler pointer.\n     */\n    routes& put(operation_type type, const sstring& url, handler_base* handler);\n\n    /**\n     * removing a handler from exact match\n     * @param url the url to match (note that url should start with /)\n     * @return the current handler (to be removed by caller)\n     */\n    handler_base* drop(operation_type type, const sstring& url);\n\n    /**\n     * add a rule to be used.\n     * rules are search only if an exact match was not found.\n     * rules are search by the order they were added.\n     * First in higher priority\n     * @param rule a rule to add\n     * @param type the operation type\n     * @return itself\n     * @attention This method takes ownership of the match_rule pointer. It will be automatically deleted when the\n     * routes instance is destroyed.\n     */\n    routes& add(match_rule* rule, operation_type type = GET) {\n        _rules[type][_rover++] = rule;\n        return *this;\n    }\n\n    /**\n     * Add a url match to a handler:\n     * Example  routes.add(GET, url(\"/api\").remainder(\"path\"), handler);\n     * @param type\n     * @param url\n     * @param handler\n     * @return\n     * @attention This method takes ownership of the handler's pointer.It will be automatically deleted when the routes\n     * instance is destroyed.\n     */\n    routes& add(operation_type type, const url& url, handler_base* handler);\n\n    /**\n     * Add a default handler - handles any HTTP Method and Path (/\\*) combination:\n     * Example  routes.add_default_handler(handler);\n     * @param handler\n     * @return\n     * @attention The caller must keep the @c handler active as long as the route instance is active.\n     */\n    routes& add_default_handler(handler_base* handler);\n\n    /**\n     * the main entry point.\n     * the general handler calls this method with the request\n     * the method takes the headers from the request and find the\n     * right handler.\n     * It then calls the handler with the parameters (if they exist) found in the url\n     * @param path the url path found\n     * @param req the http request\n     * @param rep the http reply\n     */\n    future<std::unique_ptr<http::reply> > handle(const sstring& path, std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep);\n\n    /**\n     * Search and return an exact match\n     * @param url the request url\n     * @return the handler if exists or nullptr if it does not\n     */\n    handler_base* get_exact_match(operation_type type, const sstring& url) const {\n        auto i = _map[type].find(url);\n        return (i == _map[type].end()) ? nullptr : i->second;\n    }\n\n    /**\n     * Search and return a handler by the operation type and url\n     * @param type the http operation type\n     * @param url the request url\n     * @param params a parameter object that will be filled during the match\n     * @return a handler based on the type/url match\n     */\n    handler_base* get_handler(operation_type type, const sstring& url,\n            parameters& params);\n\nprivate:\n    /**\n     * Normalize the url to remove the last / if exists\n     * and get the parameter part\n     * @param url the full url path\n     * @return the url from the request without the last /\n     */\n    sstring normalize_url(const sstring& url);\n\n    std::unordered_map<sstring, handler_base*> _map[NUM_OPERATION];\npublic:\n    using rule_cookie = uint64_t;\nprivate:\n    rule_cookie _rover = 0;\n    std::map<rule_cookie, match_rule*> _rules[NUM_OPERATION];\n    //default Handler -- for any HTTP Method and Path (/*)\n    handler_base* _default_handler = nullptr;\npublic:\n    using exception_handler_fun = std::function<std::unique_ptr<http::reply>(std::exception_ptr eptr)>;\n    using exception_handler_id = size_t;\nprivate:\n    std::map<exception_handler_id, exception_handler_fun> _exceptions;\n    exception_handler_id _exception_id = 0;\n    // for optimization reason, the lambda function\n    // that calls the exception_reply of the current object\n    // is stored\n    exception_handler_fun _general_handler;\npublic:\n    /**\n     * The exception_handler_fun expect to call\n     * std::rethrow_exception(eptr);\n     * and catch only the exception it handles\n     */\n    exception_handler_id register_exeption_handler(exception_handler_fun fun) {\n        auto current = _exception_id++;\n        _exceptions[current] = fun;\n        return current;\n    }\n\n    void remove_exception_handler(exception_handler_id id) {\n        _exceptions.erase(id);\n    }\n\n    std::unique_ptr<http::reply> exception_reply(std::exception_ptr eptr);\n\n    routes();\n\n    /*!\n     * \\brief add an alias to an already registered path.\n     * After registering a handler to a path, use this method\n     * to add an alias to that handler.\n     *\n     */\n    void add_alias(const path_description& old_path, const path_description& new_path);\n\n    /**\n     * Add a rule to be used.\n     * @param rule a rule to add\n     * @param type the operation type\n     * @return a cookie using which the rule can be removed\n     * @attention This method takes ownership of the match_rule pointer. It will be automatically deleted when the\n     * routes instance is destroyed.\n     */\n    rule_cookie add_cookie(match_rule* rule, operation_type type) {\n        auto pos = _rover++;\n        _rules[type][pos] = rule;\n        return pos;\n    }\n\n    /**\n     * Del a rule by cookie\n     * @param cookie a cookie returned previously by add_cookie\n     * @param type the operation type\n     * @return the pointer to the rule\n     * @attention Ownership of the pointer is returned to the caller.\n     */\n    match_rule* del_cookie(rule_cookie cookie, operation_type type);\n};\n\n/**\n * The handler_registration object facilitates registration and auto\n * unregistration of an exact-match handler_base into \\ref routes \"routes\"\n */\nclass handler_registration {\n    routes& _routes;\n    const sstring _url;\n    operation_type _op;\n\npublic:\n    /**\n     * Registers the handler_base into routes with routes::put\n     * @param rs the routes object reference\n     * @param h the desire handler\n     * @param url the url to match\n     * @param op the operation type (`GET` by default)\n     */\n    handler_registration(routes& rs, handler_base& h, const sstring& url, operation_type op = GET);\n\n    /**\n     * Unregisters the handler from routes with routes::drop\n     */\n    ~handler_registration();\n};\n\n/**\n * The rule_registration object facilitates registration and auto\n * unregistration of a match_rule handler into \\ref routes \"routes\"\n */\nclass rule_registration {\n    routes& _routes;\n    operation_type _op;\n    routes::rule_cookie _cookie;\n\npublic:\n    /**\n     * Registers the match_rule into routes with routes::add_cookie\n     * @param rs the routes object reference\n     * @param rule a rule to add\n     * @param op the operation type (`GET` by default)\n     */\n    rule_registration(routes& rs, match_rule& rule, operation_type op = GET);\n\n    /**\n     * Unregisters the rule from routes with routes::del_cookie\n     */\n    ~rule_registration();\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/short_streams.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/util/short_streams.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/// Returns all bytes from the stream until eof, accessible in chunks\n[[deprecated(\"seastar::httpd::read_entire_stream was moved to seastar::util, #include <seastar/util/short_streams.hh> instead\")]]\ninline\nfuture<std::vector<temporary_buffer<char>>> read_entire_stream(input_stream<char>& inp) {\n    return util::read_entire_stream(inp);\n}\n\n/// Returns all bytes from the stream until eof as a single buffer, use only on short streams\n[[deprecated(\"seastar::httpd::read_entire_stream_contiguous was moved to seastar::util, #include <seastar/util/short_streams.hh> instead\")]]\ninline\nfuture<sstring> read_entire_stream_contiguous(input_stream<char>& inp) {\n    return util::read_entire_stream_contiguous(inp);\n}\n\n/// Ignores all bytes until eof\n[[deprecated(\"seastar::httpd::skip_entire_stream was moved to seastar::util, #include <seastar/util/short_streams.hh> instead\")]]\ninline\nfuture<> skip_entire_stream(input_stream<char>& inp) {\n    return util::skip_entire_stream(inp);\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/transformers.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/http/handlers.hh>\n#include <seastar/http/file_handler.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\n/**\n * content_replace replaces variable in a file with a dynamic value.\n * It would take the host from request and will replace the variable\n * in a file\n *\n * The replacement can be restricted to an extension.\n *\n * We are currently support only one file type for replacement.\n * It could be extend if we will need it\n *\n */\nclass content_replace : public file_transformer {\npublic:\n    virtual output_stream<char> transform(std::unique_ptr<http::request> req,\n            const sstring& extension, output_stream<char>&& s);\n    /**\n     * the constructor get the file extension the replace would work on.\n     * @param extension file extension, when not set all files extension\n     */\n    explicit content_replace(const sstring& extension = \"\")\n            : extension(extension) {\n    }\nprivate:\n    sstring extension;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/http/types.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025-present ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/util/noncopyable_function.hh>\n\nnamespace seastar::http {\n\n/// body_writer_type - a function that accepts an output stream and uses that stream to write the body.\n/// The function should take ownership of the stream while using it and must close the stream when done.\nusing body_writer_type = noncopyable_function<future<>(output_stream<char>&&)>;\n\n} // namespace seastar::http\n"
  },
  {
    "path": "include/seastar/http/url.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\n\nnamespace http {\nnamespace internal {\n\nbool url_decode(std::string_view in, sstring& out);\n\nbool path_decode(std::string_view in, sstring& out);\n\n/**\n * Makes a percent-encoded string out of the given parameter\n *\n * Note, that it does NOT parse and encode a URL correctly handling\n * all the delimeters in arguments. It's up to the caller to split\n * the URL into path and arguments (both names and values) and use\n * this helper to encode individual strings\n */\nsstring url_encode(std::string_view in);\n\n} // internal namespace\n} // http namespace\n\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/json/formatter.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <ranges>\n#include <string>\n#include <tuple>\n#include <vector>\n#include <unordered_map>\n#include <map>\n#include <time.h>\n#include <sstream>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\ntemplate<typename T>\nconcept is_map = requires {\n    typename std::remove_reference_t<T>::mapped_type;\n};\n\ntemplate<typename T>\nconcept is_pair_like = requires {\n    typename std::tuple_size<T>::type;\n    requires std::tuple_size_v<T> == 2;\n};\n\ntemplate<typename T>\nconcept is_string_like =\n    std::convertible_to<const T&, std::string_view> &&\n    requires (T s) {\n        { s.data() } -> std::convertible_to<const char*>;\n        // sstring::length() and sstring::find() return size_t instead of\n        // size_type (i.e., uint32_t), so we cannot check their return type\n        // with T::size_type\n        s.find('a');\n        s.length();\n    };\n\n}\n\nnamespace json {\n\nclass jsonable;\n\ntypedef struct tm date_time;\n\n/**\n * The formatter prints json values in a json format\n * it overload to_json method for each of the supported format\n * all to_json parameters are passed as a pointer\n */\nclass formatter {\n    enum class state {\n        none, array, map\n    };\n    static sstring begin(state);\n    static sstring end(state);\n\n    template<internal::is_pair_like T>\n    static sstring to_json(state s, const T& p) {\n        auto& [key, value] = p;\n        return s == state::array ?\n                        \"{\" + to_json(state::none, p) + \"}\" :\n                        to_json(key) + \":\" + to_json(value);\n    }\n\n    template<typename Iterator, typename Sentinel>\n    static sstring to_json(state s, Iterator i, Sentinel e) {\n        std::stringstream res;\n        res << begin(s);\n        size_t n = 0;\n        while (i != e) {\n            if (n++ != 0) {\n                res << \",\";\n            }\n            res << to_json(s, *i++);\n        }\n        res << end(s);\n        return res.str();\n    }\n\n    // fallback template\n    template<typename T>\n    static sstring to_json(state, const T& t) {\n        return to_json(t);\n    }\n\n    template<internal::is_pair_like T>\n    static future<> write(output_stream<char>& stream, state s, T&& p) {\n        if (s == state::array) {\n            return stream.write(\"{\").then([&stream, &p] {\n                return write(stream, state::none, p).then([&stream] {\n                   return stream.write(\"}\");\n                });\n            });\n        } else {\n            auto& [key, value] = p;\n            return stream.write(to_json(key) + \":\").then([&value, &stream] {\n                return write(stream, value);\n            });\n        }\n    }\n\n    template<internal::is_pair_like T>\n    static future<> write(output_stream<char>& stream, state s, const T& p) {\n        if (s == state::array) {\n            return stream.write(\"{\").then([&stream, p] {\n                return write(stream, state::none, p).then([&stream] {\n                   return stream.write(\"}\");\n                });\n            });\n        } else {\n            auto& [key, value] = p;\n            return stream.write(to_json(key) + \":\").then([&stream, value] {\n                return write(stream, value);\n            });\n        }\n    }\n\n    template<typename Iterator, typename Sentinel>\n    static future<> write(output_stream<char>& stream, state s, Iterator i, Sentinel e) {\n        return do_with(true, [&stream, s, i, e] (bool& first) {\n            return stream.write(begin(s)).then([&first, &stream, s, i, e] {\n                using ref_t = std::iter_reference_t<Iterator>;\n                return do_for_each(i, e, [&first, &stream, s] (ref_t m) {\n                    auto f = (first) ? make_ready_future<>() : stream.write(\",\");\n                    first = false;\n                    if constexpr (std::is_lvalue_reference_v<ref_t>) {\n                       return f.then([&m, &stream, s] {\n                           return write(stream, s, m);\n                       });\n                    } else {\n                        using value_t = std::iter_value_t<Iterator>;\n                        return f.then([m = std::forward<value_t>(m), &stream, s] {\n                            return write(stream, s, m);\n                        });\n                    }\n                }).then([&stream, s] {\n                    return stream.write(end(s));\n                });\n            });\n        });\n    }\n\n    // fallback template\n    template<typename T>\n    static future<> write(output_stream<char>& stream, state, const T& t) {\n        return write(stream, t);\n    }\n\npublic:\n\n    /**\n     * return a json formatted string\n     * @param str the string_view to format\n     * @return the given string in a json format\n     */\n    static sstring to_json(std::string_view str);\n\n    /**\n     * return a json formatted int\n     * @param n the int to format\n     * @return the given int in a json format\n     */\n    static sstring to_json(int n);\n\n    /**\n     * return a json formatted unsigned\n     * @param n the unsigned to format\n     * @return the given unsigned in a json format\n     */\n    static sstring to_json(unsigned n);\n\n    /**\n     * return a json formatted long\n     * @param n the long to format\n     * @return the given long in a json format\n     */\n    static sstring to_json(long n);\n\n    /**\n     * return a json formatted float\n     * @param f the float to format\n     * @return the given float in a json format\n     */\n    static sstring to_json(float f);\n\n    /**\n     * return a json formatted double\n     * @param d the double to format\n     * @return the given double in a json format\n     */\n    static sstring to_json(double d);\n\n    /**\n     * return a json formatted char* (treated as string), possibly with zero-chars in the middle\n     * @param str the char* to format\n     * @param len number of bytes to read from the \\p str\n     * @return the given char* in a json format\n     */\n    static sstring to_json(const char* str, size_t len);\n\n    /**\n     * return a json formatted char* (treated as string), assuming there are no zero-chars in the middle\n     * @param str the char* to format\n     * @return the given char* in a json format\n     * @deprecated A more robust overload should be preferred: \\ref to_json(const char*, size_t)\n     */\n    static sstring to_json(const char* str);\n\n    /**\n     * return a json formatted bool\n     * @param d the bool to format\n     * @return the given bool in a json format\n     */\n    static sstring to_json(bool d);\n\n    /**\n     * converts a given range to a JSON-formatted string\n     * @param range A standard range type\n     * @return A string containing the JSON representation of the input range\n     */\n    template<std::ranges::input_range Range>\n    requires (!internal::is_string_like<Range>)\n    static sstring to_json(const Range& range) {\n        if constexpr (internal::is_map<Range>) {\n            return to_json(state::map, std::ranges::begin(range), std::ranges::end(range));\n        } else {\n            return to_json(state::array, std::ranges::begin(range), std::ranges::end(range));\n        }\n    }\n\n    /**\n     * return a json formatted date_time\n     * @param d the date_time to format\n     * @return the given date_time in a json format\n     */\n    static sstring to_json(const date_time& d);\n\n    /**\n     * return a json formatted json object\n     * @param obj the date_time to format\n     * @return the given json object in a json format\n     */\n    static sstring to_json(const jsonable& obj);\n\n    /**\n     * return a json formatted unsigned long\n     * @param l unsigned long to format\n     * @return the given unsigned long in a json format\n     */\n    static sstring to_json(unsigned long l);\n\n    /**\n     * return a json formatted string\n     * @param str the string_view to format\n     * @return the given string in a json format\n     */\n    static future<> write(output_stream<char>& s, std::string_view str) {\n        return s.write(to_json(str));\n    }\n\n    /**\n     * return a json formatted int\n     * @param n the int to format\n     * @return the given int in a json format\n     */\n    static future<> write(output_stream<char>& s, int n) {\n        return s.write(to_json(n));\n    }\n\n    /**\n     * return a json formatted long\n     * @param n the long to format\n     * @return the given long in a json format\n     */\n    static future<> write(output_stream<char>& s, long n) {\n        return s.write(to_json(n));\n    }\n\n    /**\n     * return a json formatted float\n     * @param f the float to format\n     * @return the given float in a json format\n     */\n    static future<> write(output_stream<char>& s, float f) {\n        return s.write(to_json(f));\n    }\n\n    /**\n     * return a json formatted double\n     * @param d the double to format\n     * @return the given double in a json format\n     */\n    static future<> write(output_stream<char>& s, double d) {\n        return s.write(to_json(d));\n    }\n\n    /**\n     * return a json formatted char* (treated as string)\n     * @param str the char* to format\n     * @return the given char* in a json format\n     */\n    static future<> write(output_stream<char>& s, const char* str) {\n        return s.write(to_json(str));\n    }\n\n    /**\n     * return a json formatted bool\n     * @param d the bool to format\n     * @return the given bool in a json format\n     */\n    static future<> write(output_stream<char>& s, bool d) {\n        return s.write(to_json(d));\n    }\n\n    /**\n     * Converts a range to a JSON array or object and writes it to an output stream.\n     * @param s     The output stream that will receive the JSON-formatted string\n     * @param range The range to convert. If the range contains key-value pairs (like std::map),\n     *              it will be formatted as a JSON object. Otherwise, it will be formatted as\n     *              a JSON array.\n     * @returns     A future that will be resolved when the write operation completes\n     *\n     */\n    template<std::ranges::input_range Range>\n    requires (!internal::is_string_like<Range>)\n    static future<> write(output_stream<char>& s, Range&& range) {\n        return do_with(std::forward<Range>(range), [&s] (const auto& range) {\n            if constexpr (internal::is_map<Range>) {\n                return write(s, state::map, std::ranges::begin(range), std::ranges::end(range));\n            } else {\n                return write(s, state::array, std::ranges::begin(range), std::ranges::end(range));\n            }\n        });\n    }\n\n    /**\n     * return a json formatted date_time\n     * @param d the date_time to format\n     * @return the given date_time in a json format\n     */\n    static future<> write(output_stream<char>& s, const date_time& d) {\n      return s.write(to_json(d));\n     }\n\n    /**\n     * return a json formatted json object\n     * @param obj the date_time to format\n     * @return the given json object in a json format\n     */\n    template <std::derived_from<jsonable> Jsonable>\n    static future<> write(output_stream<char>& s, Jsonable obj) {\n        return do_with(std::move(obj), [&s] (const auto& obj) {\n            return obj.write(s);\n        });\n    }\n\n    /**\n     * return a json formatted unsigned long\n     * @param l unsigned long to format\n     * @return the given unsigned long in a json format\n     */\n    static future<> write(output_stream<char>& s, unsigned long l) {\n      return s.write(to_json(l));\n     }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/json/json_elements.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#pragma once\n\n#include <string>\n#include <vector>\n\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/http/types.hh>\n\nnamespace seastar {\n\nnamespace json {\n\n\n/**\n * The base class for all json element.\n * Every json element has a name\n * An indication if it was set or not\n * And is this element is mandatory.\n * When a mandatory element is not set\n * this is not a valid object\n */\nclass json_base_element {\nprotected:\n    /**\n     * The constructors\n     */\n    json_base_element() noexcept\n            : _mandatory(false), _set(false) {\n    }\n\n    json_base_element(const json_base_element& o) noexcept = default;\n    json_base_element& operator=(const json_base_element& o) noexcept {\n        // Names and mandatory are never changed after creation\n        _set = o._set;\n        return *this;\n    }\n\n    json_base_element(json_base_element&&) = delete;\n    json_base_element& operator=(json_base_element&&) = delete;\npublic:\n    virtual ~json_base_element() = default;\n    /**\n     * Check if it's a mandatory parameter\n     * and if it's set.\n     * @return true if this is not a mandatory parameter\n     * or if it is and it's value is set\n     */\n    virtual bool is_verify() noexcept {\n        return !(_mandatory && !_set);\n    }\n\n    /**\n     * returns the internal value in a json format\n     * Each inherit class must implement this method\n     * @return formated internal value\n     */\n    virtual std::string to_string() = 0;\n\n    virtual future<> write(output_stream<char>& s) const = 0;\n    std::string _name;\n    bool _mandatory;\n    bool _set;\n};\n\n/**\n * Basic json element instantiate\n * the json_element template.\n * it adds a value to the base definition\n * and the to_string implementation using the formatter\n */\ntemplate<class T>\nclass json_element : public json_base_element {\npublic:\n\n    /**\n     * the assignment operator also set\n     * the set value to true.\n     * @param new_value the new value\n     * @return the value itself\n     */\n    json_element &operator=(const T& new_value) {\n        _value = new_value;\n        _set = true;\n        return *this;\n    }\n    /**\n     * the assignment operator also set\n     * the set value to true.\n     * @param new_value the new value\n     * @return the value itself\n     */\n    template<class C>\n    json_element &operator=(const C& new_value) {\n        _value = new_value;\n        _set = true;\n        return *this;\n    }\n    /**\n     * The brackets operator\n     * @return the value\n     */\n    const T& operator()() const noexcept {\n        return _value;\n    }\n\n    /**\n     * The to_string return the value\n     * formated as a json value\n     * @return the value formatted for json\n     */\n    virtual std::string to_string() override\n    {\n        return formatter::to_json(_value);\n    }\n\n    virtual future<> write(output_stream<char>& s) const override {\n        return formatter::write(s, _value);\n    }\nprivate:\n    T _value;\n};\n\n/**\n * json_list_template is an array type based on a\n * container type passed as a template parameter, as we want to\n * have flavors based on both vector and chunked_fifo.\n *\n * When values are added with push it is set the \"set\" flag to true\n * hence will be included in the parsed object\n */\ntemplate <class T, class Container>\nclass json_list_template : public json_base_element {\npublic:\n\n    /**\n     * Add an element to the list.\n     * @param element a new element that will be added to the end of the list\n     */\n    void push(const T& element) {\n        _set = true;\n        _elements.push_back(element);\n    }\n\n    /**\n     * Move an element into the list.\n     * @param element a new element that will be added to the list using move-construction\n     */\n    void push(T&& element) {\n        _set = true;\n        _elements.push_back(std::move(element));\n    }\n\n    virtual std::string to_string() override\n    {\n        return formatter::to_json(_elements);\n    }\n\n    /**\n     * Assignment can be done from any object that support const range\n     * iteration and that it's elements can be assigned to the list elements\n     */\n    template<class C>\n    json_list_template& operator=(const C& list) {\n        _elements.clear();\n        _elements.reserve(std::size(list));\n        for  (auto i : list) {\n            push(i);\n        }\n        return *this;\n    }\n\n    template<class C>\n    json_list_template& operator=(json_list_template<T, Container>&& list) noexcept {\n        if (this != &list) {\n            _elements = std::move(list._elements);\n        }\n        return *this;\n    }\n\n    virtual future<> write(output_stream<char>& s) const override {\n        return formatter::write(s, _elements);\n    }\n\n    Container _elements;\n};\n\ntemplate <typename T>\nusing json_list = json_list_template<T, std::vector<T>>;\n\ntemplate <typename T>\nusing json_chunked_list = json_list_template<T, seastar::chunked_fifo<T>>;\n\nclass jsonable {\npublic:\n    jsonable() = default;\n    jsonable(const jsonable&) = default;\n    jsonable& operator=(const jsonable&) = default;\n    virtual ~jsonable() = default;\n    /**\n     * create a formatted string of the object.\n     * @return the object formated.\n     */\n    virtual std::string to_json() const = 0;\n\n    /*!\n     * \\brief write an object to the output stream\n     *\n     * The defult implementation uses the to_json\n     * Object implementation override it.\n     */\n    virtual future<> write(output_stream<char>& s) const {\n        return s.write(to_json());\n    }\n};\n\n/**\n * The base class for all json objects\n * It holds a list of all the element in it,\n * allowing it implement the to_json method.\n *\n * It also allows iterating over the element\n * in the object, even if not all the member\n * are known in advance and in practice mimic\n * reflection\n */\nstruct json_base : public jsonable {\n\n    virtual ~json_base() = default;\n\n    json_base() = default;\n\n    json_base(const json_base&) = delete;\n\n    json_base operator=(const json_base&) = delete;\n\n    /**\n     * create a formatted string of the object.\n     * @return the object formatted.\n     */\n    virtual std::string to_json() const;\n\n    /*!\n     * \\brief write to an output stream\n     */\n    virtual future<> write(output_stream<char>&) const;\n\n    /**\n     * Check that all mandatory elements are set\n     * @return true if all mandatory parameters are set\n     */\n    virtual bool is_verify() const;\n\n    /**\n     * Register an element in an object\n     * @param element the element to be added\n     * @param name the element name\n     * @param mandatory is this element mandatory.\n     */\n    virtual void add(json_base_element* element, std::string name,\n            bool mandatory = false);\n\n    std::vector<json_base_element*> _elements;\n};\n\n/**\n * There are cases where a json request needs to return a successful\n * empty reply.\n * The json_void class will be used to mark that the reply should be empty.\n *\n */\nstruct json_void : public jsonable{\n    virtual std::string to_json() const {\n        return \"\";\n    }\n\n    /*!\n     * \\brief write to an output stream\n     */\n    virtual future<> write(output_stream<char>& s) const {\n        return s.close();\n    }\n};\n\n\n/**\n * The json return type, is a helper class to return a json\n * formatted string.\n * It uses autoboxing in its constructor so when a function return\n * type is json_return_type, it could return a type that would be converted\n * ie.\n * json_return_type foo() {\n *     return \"hello\";\n * }\n *\n * would return a json formatted string: \"hello\" (rather then hello)\n */\nstruct json_return_type {\n    sstring _res;\n#if SEASTAR_API_LEVEL >= 8\n    using body_writer_type = http::body_writer_type;\n#else\n    using body_writer_type = std::function<future<>(output_stream<char>&&)>;\n#endif\n    body_writer_type _body_writer;\n    json_return_type(body_writer_type&& body_writer) : _body_writer(std::move(body_writer)) {\n    }\n    template<class T>\n    json_return_type(const T& res) {\n        _res = formatter::to_json(res);\n    }\n\n   json_return_type(json_return_type&& o) noexcept : _res(std::move(o._res)), _body_writer(std::move(o._body_writer)) {\n   }\n    json_return_type& operator=(json_return_type&& o) noexcept {\n        if (this != &o) {\n            _res = std::move(o._res);\n            _body_writer = std::move(o._body_writer);\n        }\n        return *this;\n    }\n\n#if SEASTAR_API_LEVEL < 8\n    json_return_type(const json_return_type&) = default;\n    json_return_type& operator=(const json_return_type&) = default;\n#endif\n};\n\n/*!\n * \\brief capture a range and return a serialize function for it as a json array.\n *\n * To use it, pass a range and a mapping function.\n * For example, if res is a map:\n *\n * return make_ready_future<json::json_return_type>(stream_range_as_array(res, [](const auto&i) {return i.first}));\n */\ntemplate<typename Container, typename Func>\nrequires requires (Container c, Func aa, output_stream<char> s) { { formatter::write(s, aa(*c.begin())) } -> std::same_as<future<>>; }\njson_return_type::body_writer_type stream_range_as_array(Container val, Func fun) {\n    return [val = std::move(val), fun = std::move(fun)](output_stream<char>&& s) mutable {\n        return do_with(output_stream<char>(std::move(s)), Container(std::move(val)), Func(std::move(fun)), true, [](output_stream<char>& s, const Container& val, const Func& f, bool& first){\n            return s.write(\"[\").then([&val, &s, &first, &f] () {\n                return do_for_each(val, [&s, &first, &f](const typename Container::value_type& v){\n                    auto fut = first ? make_ready_future<>() : s.write(\", \");\n                    first = false;\n                    return fut.then([&s, &f, &v]() {\n                        return formatter::write(s, f(v));\n                    });\n                });\n            }).then([&s](){\n                return s.write(\"]\");\n            }).finally([&s] {\n                return s.close();\n            });\n        });\n    };\n}\n\n/*!\n * \\brief capture an object and return a serialize function for it.\n *\n * To use it:\n * return make_ready_future<json::json_return_type>(stream_object(res));\n */\ntemplate<class T>\njson_return_type::body_writer_type stream_object(T val) {\n    return [val = std::move(val)](output_stream<char>&& s) mutable {\n        return do_with(output_stream<char>(std::move(s)), T(std::move(val)), [](output_stream<char>& s, T& val){\n            return formatter::write(s, std::move(val)).finally([&s] {\n                return s.close();\n            });\n        });\n    };\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/api.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <chrono>\n#include <memory>\n#include <vector>\n#include <cstring>\n#include <sys/types.h>\n\n#include <seastar/core/future.hh>\n#include <seastar/net/byteorder.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/core/internal/api-level.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/file-types.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/program-options.hh>\n\nnamespace seastar {\n\ninline\nbool is_ip_unspecified(const ipv4_addr& addr) noexcept {\n    return addr.is_ip_unspecified();\n}\n\ninline\nbool is_port_unspecified(const ipv4_addr& addr) noexcept {\n    return addr.is_port_unspecified();\n}\n\ninline\nsocket_address make_ipv4_address(const ipv4_addr& addr) noexcept {\n    return socket_address(addr);\n}\n\ninline\nsocket_address make_ipv4_address(uint32_t ip, uint16_t port) noexcept {\n    return make_ipv4_address(ipv4_addr(ip, port));\n}\n\nnamespace net {\n\n// see linux tcp(7) for parameter explanation\nstruct tcp_keepalive_params {\n    std::chrono::seconds idle; // TCP_KEEPIDLE\n    std::chrono::seconds interval; // TCP_KEEPINTVL\n    unsigned count; // TCP_KEEPCNT\n};\n\n// see linux sctp(7) for parameter explanation\nstruct sctp_keepalive_params {\n    std::chrono::seconds interval; // spp_hbinterval\n    unsigned count; // spp_pathmaxrt\n};\n\nusing keepalive_params = std::variant<tcp_keepalive_params, sctp_keepalive_params>;\n\n/// \\cond internal\nclass connected_socket_impl;\nclass socket_impl;\n\nclass server_socket_impl;\nclass datagram_channel_impl;\nclass get_impl;\n/// \\endcond\n\nclass datagram_impl {\npublic:\n    virtual ~datagram_impl() {};\n    virtual socket_address get_src() = 0;\n    virtual socket_address get_dst() = 0;\n    virtual uint16_t get_dst_port() = 0;\n    virtual std::span<temporary_buffer<char>> get_buffers() = 0;\n};\n\nusing udp_datagram_impl = datagram_impl;\n\nclass datagram final {\nprivate:\n    std::unique_ptr<datagram_impl> _impl;\n    // The get_data() below will need to tell\n    //  - _p wasn't initialized from get_buffers() span\n    //  - _p was initialized, but was then release()-d by caller\n    // from each other. thus std::optional\n    std::optional<net::packet> _p;\npublic:\n    datagram(std::unique_ptr<datagram_impl>&& impl) noexcept : _impl(std::move(impl)) {};\n    socket_address get_src() { return _impl->get_src(); }\n    socket_address get_dst() { return _impl->get_dst(); }\n    uint16_t get_dst_port() { return _impl->get_dst_port(); }\n    [[deprecated(\"Use get_buf() instead\")]]\n    packet& get_data() {\n        if (!_p) {\n            _p.emplace(_impl->get_buffers());\n        }\n        return _p.value();\n    }\n    std::span<temporary_buffer<char>> get_buffers() { return _impl->get_buffers(); }\n};\n\nusing udp_datagram = datagram;\n\nclass datagram_channel {\nprivate:\n    std::unique_ptr<datagram_channel_impl> _impl;\npublic:\n    datagram_channel() noexcept;\n    datagram_channel(std::unique_ptr<datagram_channel_impl>) noexcept;\n    ~datagram_channel();\n\n    datagram_channel(datagram_channel&&) noexcept;\n    datagram_channel& operator=(datagram_channel&&) noexcept;\n\n    socket_address local_address() const;\n\n    future<datagram> receive();\n    future<> send(const socket_address& dst, const char* msg);\n    [[deprecated(\"Use send(std::span<temporary_buffer<char>>) overload\")]]\n    future<> send(const socket_address& dst, packet p);\n    /**\n     * \\brief Send a datagram composed of multiple buffers to the specified destination.\n     *\n     * The temporary_buffers objects referenced must remain valid only for the duration\n     * of the call. The implementation transfers the buffers ownership before returning\n     * the future.\n     *\n     * \\param dst The destination socket address.\n     * \\param bufs A span of temporary_buffer<char> objects containing the data to send.\n     * \\return A future that completes when the send operation is finished.\n     */\n    future<> send(const socket_address& dst, std::span<temporary_buffer<char>> bufs);\n    bool is_closed() const;\n    /// Causes a pending receive() to complete (possibly with an exception)\n    void shutdown_input();\n    /// Causes a pending send() to complete (possibly with an exception)\n    void shutdown_output();\n    /// Close the channel and releases all resources.\n    ///\n    /// Must be called only when there are no unfinished send() or receive() calls. You\n    /// can force pending calls to complete soon by calling shutdown_input() and\n    /// shutdown_output().\n    void close();\n};\n\nusing udp_channel = datagram_channel;\n\nclass network_interface_impl;\n\n} /* namespace net */\n\n/// \\addtogroup networking-module\n/// @{\n\n/// Configuration for buffered connected_socket input operations\n///\n/// This structure allows tuning of buffered input operations done via\n/// connected_socket. It is a hint to the implementation and may be\n/// ignored (e.g. the zero-copy native stack does not allocate buffers,\n/// so it ignores buffer-size parameters).\nstruct connected_socket_input_stream_config final {\n    /// Initial buffer size to use for input buffering\n    unsigned buffer_size = 8192;\n    /// Minimum buffer size to use for input buffering. The system will decrease\n    /// buffer sizes if it sees a tendency towards small requests, but will not go\n    /// below this buffer size.\n    unsigned min_buffer_size = 512;\n    /// Maximum buffer size to use for input buffering. The system will increase\n    /// buffer sizes if it sees a tendency towards large requests, but will not go\n    /// above this buffer size.\n    unsigned max_buffer_size = 128 * 1024;\n};\n\n/// Distinguished name\nstruct session_dn {\n    sstring subject;\n    sstring issuer;\n};\n\n/// A TCP (or other stream-based protocol) connection.\n///\n/// A \\c connected_socket represents a full-duplex stream between\n/// two endpoints, a local endpoint and a remote endpoint.\nclass connected_socket {\n    friend class net::get_impl;\n    std::unique_ptr<net::connected_socket_impl> _csi;\npublic:\n    /// Constructs a \\c connected_socket not corresponding to a connection\n    connected_socket() noexcept;\n    ~connected_socket();\n\n    /// \\cond internal\n    explicit connected_socket(std::unique_ptr<net::connected_socket_impl> csi) noexcept;\n    /// \\endcond\n    /// Moves a \\c connected_socket object.\n    connected_socket(connected_socket&& cs) noexcept;\n    /// Move-assigns a \\c connected_socket object.\n    connected_socket& operator=(connected_socket&& cs) noexcept;\n    /// Gets the input stream.\n    ///\n    /// \\param csisc Configuration for the input_stream returned\n    ///\n    /// Gets an object returning data sent from the remote endpoint.\n    input_stream<char> input(connected_socket_input_stream_config csisc = {});\n    /// Gets the output stream.\n    ///\n    /// Gets an object that sends data to the remote endpoint.\n    /// \\param buffer_size how much data to buffer\n    output_stream<char> output(size_t buffer_size = 8192);\n    /// Sets the TCP_NODELAY option (disabling Nagle's algorithm)\n    void set_nodelay(bool nodelay);\n    /// Gets the TCP_NODELAY option (Nagle's algorithm)\n    ///\n    /// \\return whether the nodelay option is enabled or not\n    bool get_nodelay() const;\n    /// Sets SO_KEEPALIVE option (enable keepalive timer on a socket)\n    void set_keepalive(bool keepalive);\n    /// Gets O_KEEPALIVE option\n    /// \\return whether the keepalive option is enabled or not\n    bool get_keepalive() const;\n    /// Sets TCP keepalive parameters\n    void set_keepalive_parameters(const net::keepalive_params& p);\n    /// Get TCP keepalive parameters\n    net::keepalive_params get_keepalive_parameters() const;\n    /// Sets custom socket options. Based on setsockopt function.\n    /// Linux users should refer to protocol-specific manuals\n    /// to see available options, e.g. tcp(7), ip(7), etc.\n    void set_sockopt(int level, int optname, const void* data, size_t len);\n    /// Gets custom socket options. Based on getsockopt function.\n    /// Linux users should refer to protocol-specific manuals\n    /// to see available options, e.g. tcp(7), ip(7), etc.\n    int get_sockopt(int level, int optname, void* data, size_t len) const;\n    /// Local address of the socket\n    socket_address local_address() const noexcept;\n    /// Remote address of the socket\n    socket_address remote_address() const noexcept;\n\n    /// Disables output to the socket.\n    ///\n    /// Current or future writes that have not been successfully flushed\n    /// will immediately fail with an error.  This is useful to abort\n    /// operations on a socket that is not making progress due to a\n    /// peer failure.\n    void shutdown_output();\n    /// Disables input from the socket.\n    ///\n    /// Current or future reads will immediately fail with an error.\n    /// This is useful to abort operations on a socket that is not making\n    /// progress due to a peer failure.\n    void shutdown_input();\n    /// Check whether the \\c connected_socket is initialized.\n    ///\n    /// \\return true if this \\c connected_socket socket_address is bound initialized\n    /// false otherwise.\n    ///\n    /// \\see connect(socket_address sa)\n    /// \\see connect(socket_address sa, socket_address local, transport proto)\n    explicit operator bool() const noexcept {\n        return static_cast<bool>(_csi);\n    }\n    /// Waits for the peer of this socket to disconnect\n    ///\n    /// \\return future that resolves when the peer closes connection or shuts it down\n    /// for writing or when local socket is called \\ref shutdown_input().\n    ///\n    /// Note, that when the returned future is resolved for whatever reason socket\n    /// may still be readable from, so the caller may want to wait for both events\n    /// -- this one and EOF from read.\n    ///\n    /// Calling it several times per socket is not allowed (undefined behavior)\n    ///\n    /// \\see poll(2) about POLLRDHUP for more details\n    future<> wait_input_shutdown();\n};\n/// @}\n\n/// \\addtogroup networking-module\n/// @{\n\n/// The seastar socket.\n///\n/// A \\c socket that allows a connection to be established between\n/// two endpoints.\nclass socket {\n    std::unique_ptr<net::socket_impl> _si;\npublic:\n    socket() noexcept = default;\n    ~socket();\n\n    /// \\cond internal\n    explicit socket(std::unique_ptr<net::socket_impl> si) noexcept;\n    /// \\endcond\n    /// Moves a \\c seastar::socket object.\n    socket(socket&&) noexcept;\n    /// Move-assigns a \\c seastar::socket object.\n    socket& operator=(socket&&) noexcept;\n\n    /// Attempts to establish the connection.\n    ///\n    /// \\return a \\ref connected_socket representing the connection.\n    future<connected_socket> connect(socket_address sa, socket_address local = {}, transport proto = transport::TCP);\n\n    /// Sets SO_REUSEADDR option (enable reuseaddr option on a socket)\n    void set_reuseaddr(bool reuseaddr);\n    /// Gets O_REUSEADDR option\n    /// \\return whether the reuseaddr option is enabled or not\n    bool get_reuseaddr() const;\n    /// Stops any in-flight connection attempt.\n    ///\n    /// Cancels the connection attempt if it's still in progress, and\n    /// terminates the connection if it has already been established.\n    void shutdown();\n};\n\n/// @}\n\n/// \\addtogroup networking-module\n/// @{\n\n/// The result of an server_socket::accept() call\nstruct accept_result {\n    connected_socket connection;  ///< The newly-accepted connection\n    socket_address remote_address;  ///< The address of the peer that connected to us\n};\n\n/// A listening socket, waiting to accept incoming network connections.\nclass server_socket {\n    std::unique_ptr<net::server_socket_impl> _ssi;\n    bool _aborted = false;\npublic:\n    enum class load_balancing_algorithm {\n        // This algorithm tries to distribute all connections equally between all shards.\n        // It does this by sending new connections to a shard with smallest amount of connections.\n        connection_distribution,\n        // This algorithm distributes new connection based on peer's tcp port. Destination shard\n        // is calculated as a port number modulo number of shards. This allows a client to connect\n        // to a specific shard in a server given it knows how many shards server has by choosing\n        // src port number accordingly.\n        port,\n        // This algorithm distributes all new connections to listen_options::fixed_cpu shard only.\n        fixed,\n        default_ = connection_distribution\n    };\n    /// Constructs a \\c server_socket without being bound to any address\n    server_socket() noexcept;\n    /// \\cond internal\n    explicit server_socket(std::unique_ptr<net::server_socket_impl> ssi) noexcept;\n    /// \\endcond\n    /// Moves a \\c server_socket object.\n    server_socket(server_socket&& ss) noexcept;\n    ~server_socket();\n    /// Move-assigns a \\c server_socket object.\n    server_socket& operator=(server_socket&& cs) noexcept;\n\n    /// Accepts the next connection to successfully connect to this socket.\n    ///\n    /// \\return an accept_result representing the connection and\n    ///         the socket_address of the remote endpoint.\n    ///\n    /// \\see listen(socket_address sa)\n    /// \\see listen(socket_address sa, listen_options opts)\n    future<accept_result> accept();\n\n    /// Stops any \\ref accept() in progress.\n    ///\n    /// Current and future \\ref accept() calls will terminate immediately\n    /// with an error.\n    void abort_accept();\n\n    /// Local bound address\n    ///\n    /// \\return the local bound address if the \\c server_socket is listening,\n    /// an empty address constructed with \\c socket_address() otherwise.\n    ///\n    /// \\see listen(socket_address sa)\n    /// \\see listen(socket_address sa, listen_options opts)\n    socket_address local_address() const noexcept;\n\n    /// Check whether the \\c server_socket is listening on any address.\n    ///\n    /// \\return true if this \\c socket_address is bound to an address,\n    /// false if it is just created with the default constructor.\n    ///\n    /// \\see listen(socket_address sa)\n    /// \\see listen(socket_address sa, listen_options opts)\n    explicit operator bool() const noexcept {\n        return static_cast<bool>(_ssi);\n    }\n};\n\n/// @}\n\n/// Options for creating a listening socket.\n///\n/// WARNING: these options currently only have an effect when using\n/// the POSIX stack: all options are ignored on the native stack as they\n/// are not implemented there.\nstruct listen_options {\n    bool reuse_address = false;\n    server_socket::load_balancing_algorithm lba = server_socket::load_balancing_algorithm::default_;\n    transport proto = transport::TCP;\n    int listen_backlog = 100;\n    unsigned fixed_cpu = 0u;\n    std::optional<file_permissions> unix_domain_socket_permissions;\n\n    /// If set, the SO_SNDBUF size will be set to the given value on the listening socket\n    /// via setsockopt. This buffer size is inherited by the sockets returned by\n    /// accept and is the preferred way to set the buffer size for these sockets since\n    /// setting it directly on the already-accepted socket is ineffective (see TCP(7)).\n    std::optional<int> so_sndbuf;\n\n    /// If set, the SO_RCVBUF size will be set to the given value on the listening socket\n    /// via setsockopt. This buffer size is inherited by the sockets returned by\n    /// accept and is the preferred way to set the buffer size for these sockets since\n    /// setting it directly on the already-accepted socket is ineffective (see TCP(7)).\n    std::optional<int> so_rcvbuf;\n\n    void set_fixed_cpu(unsigned cpu) {\n        lba = server_socket::load_balancing_algorithm::fixed;\n        fixed_cpu = cpu;\n    }\n\n    // The connection is encapsulated with proxy protocol (which is just\n    // a header prepended to the data stream). connected_socket::remote_address()\n    // and connected_socket::local_address() will return the addresses\n    // as specified in the proxy protocol header. load_balancing_algorithm::port\n    // will use the port from the proxy protocol header.\n    //\n    // Currently only proxy protocol v2 binary format is supported.\n    //\n    // The proxy protocol is defined in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt\n    bool proxy_protocol = false;\n};\n\nclass network_interface {\nprivate:\n    shared_ptr<net::network_interface_impl> _impl;\npublic:\n    network_interface() = delete;\n    network_interface(shared_ptr<net::network_interface_impl>) noexcept;\n    network_interface(network_interface&&) noexcept;\n\n    network_interface& operator=(network_interface&&) noexcept;\n\n    uint32_t index() const;\n    uint32_t mtu() const;\n\n    const sstring& name() const;\n    const sstring& display_name() const;\n    const std::vector<net::inet_address>& addresses() const;\n    const std::vector<uint8_t> hardware_address() const;\n\n    bool is_loopback() const;\n    bool is_virtual() const;\n    bool is_up() const;\n    bool supports_ipv6() const;\n};\n\nstruct statistics {\n    uint64_t bytes_sent = 0;\n    uint64_t bytes_received = 0;\n};\n\nnamespace metrics {\nclass metric_groups;\nclass label_instance;\n}\n\nvoid register_net_metrics_for_scheduling_group(\n    metrics::metric_groups& m, unsigned sg_id, const metrics::label_instance& name);\n\nclass network_stack {\npublic:\n    virtual ~network_stack() {}\n    virtual server_socket listen(socket_address sa, listen_options opts) = 0;\n    // FIXME: local parameter assumes ipv4 for now, fix when adding other AF\n    future<connected_socket> connect(socket_address sa, socket_address = {}, transport proto = transport::TCP);\n    virtual ::seastar::socket socket() = 0;\n\n    [[deprecated(\"Use `make_[un]bound_datagram_channel` instead\")]]\n    virtual net::udp_channel make_udp_channel(const socket_address& = {}) = 0;\n\n    virtual net::datagram_channel make_unbound_datagram_channel(sa_family_t) = 0;\n    virtual net::datagram_channel make_bound_datagram_channel(const socket_address& local) = 0;\n    virtual future<> initialize() {\n        return make_ready_future();\n    }\n    virtual bool has_per_core_namespace() = 0;\n    // NOTE: this is not a correct query approach.\n    // This question should be per NIC, but we have no such\n    // abstraction, so for now this is \"stack-wide\"\n    virtual bool supports_ipv6() const {\n        return false;\n    }\n\n    // Return network stats (bytes sent/received etc.) for this stack and scheduling group\n    virtual statistics stats(unsigned scheduling_group_id) = 0;\n    // Clears the stats for this stack and scheduling group\n    virtual void clear_stats(unsigned scheduling_group_id) = 0;\n\n    /**\n     * Returns available network interfaces. This represents a\n     * snapshot of interfaces available at call time, hence the\n     * return by value.\n     */\n    virtual std::vector<network_interface> network_interfaces();\n};\n\nstruct network_stack_entry {\n    using factory_func = noncopyable_function<future<std::unique_ptr<network_stack>> (const program_options::option_group&)>;\n\n    sstring name;\n    std::unique_ptr<program_options::option_group> opts;\n    factory_func factory;\n    bool is_default;\n};\n\n}\n"
  },
  {
    "path": "include/seastar/net/arp.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#pragma once\n\n#include <unordered_map>\n#include <seastar/net/net.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/net/ethernet.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nclass arp;\nclass arp_for_protocol;\ntemplate <typename L3>\nclass arp_for;\n\nclass arp_for_protocol {\nprotected:\n    arp& _arp;\n    uint16_t _proto_num;\npublic:\n    arp_for_protocol(arp& a, uint16_t proto_num);\n    virtual ~arp_for_protocol();\n    virtual future<> received(packet p) = 0;\n    virtual bool forward(forward_hash& out_hash_data, packet& p, size_t off) {\n      std::ignore = out_hash_data;\n      std::ignore = p;\n      std::ignore = off;\n      return false;\n    }\n};\n\nclass arp {\n    interface* _netif;\n    l3_protocol _proto;\n    std::unordered_map<uint16_t, arp_for_protocol*> _arp_for_protocol;\n    circular_buffer<l3_protocol::l3packet> _packetq;\nprivate:\n    struct arp_hdr {\n        uint16_t htype;\n        uint16_t ptype;\n\n        static arp_hdr read(const char* p) {\n            arp_hdr ah;\n            ah.htype = consume_be<uint16_t>(p);\n            ah.ptype = consume_be<uint16_t>(p);\n            return ah;\n        }\n        static constexpr size_t size() { return 4; }\n    };\npublic:\n    explicit arp(interface* netif);\n    void add(uint16_t proto_num, arp_for_protocol* afp);\n    void del(uint16_t proto_num);\nprivate:\n    ethernet_address l2self() const noexcept { return _netif->hw_address(); }\n    future<> process_packet(packet p, ethernet_address from);\n    bool forward(forward_hash& out_hash_data, packet& p, size_t off);\n    std::optional<l3_protocol::l3packet> get_packet();\n    template <class l3_proto>\n    friend class arp_for;\n};\n\ntemplate <typename L3>\nclass arp_for : public arp_for_protocol {\npublic:\n    using l2addr = ethernet_address;\n    using l3addr = typename L3::address_type;\nprivate:\n    static constexpr auto max_waiters = 512;\n    enum oper {\n        op_request = 1,\n        op_reply = 2,\n    };\n    struct arp_hdr {\n        uint16_t htype;\n        uint16_t ptype;\n        uint8_t hlen;\n        uint8_t plen;\n        uint16_t oper;\n        l2addr sender_hwaddr;\n        l3addr sender_paddr;\n        l2addr target_hwaddr;\n        l3addr target_paddr;\n\n        static arp_hdr read(const char* p) {\n            arp_hdr ah;\n            ah.htype = consume_be<uint16_t>(p);\n            ah.ptype = consume_be<uint16_t>(p);\n            ah.hlen = consume_be<uint8_t>(p);\n            ah.plen = consume_be<uint8_t>(p);\n            ah.oper = consume_be<uint16_t>(p);\n            ah.sender_hwaddr = l2addr::consume(p);\n            ah.sender_paddr = l3addr::consume(p);\n            ah.target_hwaddr = l2addr::consume(p);\n            ah.target_paddr = l3addr::consume(p);\n            return ah;\n        }\n        void write(char* p) const {\n            produce_be<uint16_t>(p, htype);\n            produce_be<uint16_t>(p, ptype);\n            produce_be<uint8_t>(p, hlen);\n            produce_be<uint8_t>(p, plen);\n            produce_be<uint16_t>(p, oper);\n            sender_hwaddr.produce(p);\n            sender_paddr.produce(p);\n            target_hwaddr.produce(p);\n            target_paddr.produce(p);\n        }\n        static constexpr size_t size() {\n            return 8 + 2 * (l2addr::size() + l3addr::size());\n        }\n    };\n    struct resolution {\n        std::vector<promise<l2addr>> _waiters;\n        timer<> _timeout_timer;\n    };\nprivate:\n    l3addr _l3self = L3::broadcast_address();\n    std::unordered_map<l3addr, l2addr> _table;\n    std::unordered_map<l3addr, resolution> _in_progress;\nprivate:\n    packet make_query_packet(l3addr paddr);\n    virtual future<> received(packet p) override;\n    future<> handle_request(arp_hdr* ah);\n    l2addr l2self() const noexcept { return _arp.l2self(); }\n    void send(l2addr to, packet p);\npublic:\n    future<> send_query(const l3addr& paddr);\n    explicit arp_for(arp& a) : arp_for_protocol(a, L3::arp_protocol_type()) {\n        _table[L3::broadcast_address()] = ethernet::broadcast_address();\n    }\n    future<ethernet_address> lookup(const l3addr& addr);\n    void learn(l2addr l2, l3addr l3);\n    void run();\n    void set_self_addr(l3addr addr) {\n        _table.erase(_l3self);\n        _table[addr] = l2self();\n        _l3self = addr;\n    }\n    friend class arp;\n};\n\ntemplate <typename L3>\npacket\narp_for<L3>::make_query_packet(l3addr paddr) {\n    arp_hdr hdr;\n    hdr.htype = ethernet::arp_hardware_type();\n    hdr.ptype = L3::arp_protocol_type();\n    hdr.hlen = sizeof(l2addr);\n    hdr.plen = sizeof(l3addr);\n    hdr.oper = op_request;\n    hdr.sender_hwaddr = l2self();\n    hdr.sender_paddr = _l3self;\n    hdr.target_hwaddr = ethernet::broadcast_address();\n    hdr.target_paddr = paddr;\n    auto p = packet();\n    p.prepend_uninitialized_header(hdr.size());\n    hdr.write(p.get_header(0, hdr.size()));\n    return p;\n}\n\ntemplate <typename L3>\nvoid arp_for<L3>::send(l2addr to, packet p) {\n    _arp._packetq.push_back(l3_protocol::l3packet{eth_protocol_num::arp, to, std::move(p)});\n}\n\ntemplate <typename L3>\nfuture<>\narp_for<L3>::send_query(const l3addr& paddr) {\n    send(ethernet::broadcast_address(), make_query_packet(paddr));\n    return make_ready_future<>();\n}\n\nclass arp_error : public std::runtime_error {\npublic:\n    arp_error(const std::string& msg) : std::runtime_error(msg) {}\n};\n\nclass arp_timeout_error : public arp_error {\npublic:\n    arp_timeout_error() : arp_error(\"ARP timeout\") {}\n};\n\nclass arp_queue_full_error : public arp_error {\npublic:\n    arp_queue_full_error() : arp_error(\"ARP waiter's queue is full\") {}\n};\n\ntemplate <typename L3>\nfuture<ethernet_address>\narp_for<L3>::lookup(const l3addr& paddr) {\n    auto i = _table.find(paddr);\n    if (i != _table.end()) {\n        return make_ready_future<ethernet_address>(i->second);\n    }\n    auto j = _in_progress.find(paddr);\n    auto first_request = j == _in_progress.end();\n    auto& res = first_request ? _in_progress[paddr] : j->second;\n\n    if (first_request) {\n        res._timeout_timer.set_callback([paddr, this, &res] {\n            // FIXME: future is discarded\n            (void)send_query(paddr);\n            for (auto& w : res._waiters) {\n                w.set_exception(arp_timeout_error());\n            }\n            res._waiters.clear();\n        });\n        res._timeout_timer.arm_periodic(std::chrono::seconds(1));\n        // FIXME: future is discarded\n        (void)send_query(paddr);\n    }\n\n    if (res._waiters.size() >= max_waiters) {\n        return make_exception_future<ethernet_address>(arp_queue_full_error());\n    }\n\n    res._waiters.emplace_back();\n    return res._waiters.back().get_future();\n}\n\ntemplate <typename L3>\nvoid\narp_for<L3>::learn(l2addr hwaddr, l3addr paddr) {\n    _table[paddr] = hwaddr;\n    auto i = _in_progress.find(paddr);\n    if (i != _in_progress.end()) {\n        auto& res = i->second;\n        res._timeout_timer.cancel();\n        for (auto &&pr : res._waiters) {\n            pr.set_value(hwaddr);\n        }\n        _in_progress.erase(i);\n    }\n}\n\ntemplate <typename L3>\nfuture<>\narp_for<L3>::received(packet p) {\n    auto ah = p.get_header(0, arp_hdr::size());\n    if (!ah) {\n        return make_ready_future<>();\n    }\n    auto h = arp_hdr::read(ah);\n    if (h.hlen != sizeof(l2addr) || h.plen != sizeof(l3addr)) {\n        return make_ready_future<>();\n    }\n    switch (h.oper) {\n    case op_request:\n        return handle_request(&h);\n    case op_reply:\n        arp_learn(h.sender_hwaddr, h.sender_paddr);\n        return make_ready_future<>();\n    default:\n        return make_ready_future<>();\n    }\n}\n\ntemplate <typename L3>\nfuture<>\narp_for<L3>::handle_request(arp_hdr* ah) {\n    if (ah->target_paddr == _l3self\n            && _l3self != L3::broadcast_address()) {\n        ah->oper = op_reply;\n        ah->target_hwaddr = ah->sender_hwaddr;\n        ah->target_paddr = ah->sender_paddr;\n        ah->sender_hwaddr = l2self();\n        ah->sender_paddr = _l3self;\n        auto p = packet();\n        ah->write(p.prepend_uninitialized_header(ah->size()));\n        send(ah->target_hwaddr, std::move(p));\n    }\n    return make_ready_future<>();\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/byteorder.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <arpa/inet.h>  // for ntohs() and friends\n#include <iosfwd>\n#include <utility>\n\n#include <seastar/core/unaligned.hh>\n\nnamespace seastar {\n\ninline uint64_t ntohq(uint64_t v) {\n#if defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__\n    // big endian, nothing to do\n    return v;\n#else\n    // little endian, reverse bytes\n    return __builtin_bswap64(v);\n#endif\n}\ninline uint64_t htonq(uint64_t v) {\n    // htonq and ntohq have identical implementations\n    return ntohq(v);\n}\n\nnamespace net {\n\ninline void ntoh() {}\ninline void hton() {}\n\ninline uint8_t ntoh(uint8_t x) { return x; }\ninline uint8_t hton(uint8_t x) { return x; }\ninline uint16_t ntoh(uint16_t x) { return ntohs(x); }\ninline uint16_t hton(uint16_t x) { return htons(x); }\ninline uint32_t ntoh(uint32_t x) { return ntohl(x); }\ninline uint32_t hton(uint32_t x) { return htonl(x); }\ninline uint64_t ntoh(uint64_t x) { return ntohq(x); }\ninline uint64_t hton(uint64_t x) { return htonq(x); }\n\ninline int8_t ntoh(int8_t x) { return x; }\ninline int8_t hton(int8_t x) { return x; }\ninline int16_t ntoh(int16_t x) { return ntohs(x); }\ninline int16_t hton(int16_t x) { return htons(x); }\ninline int32_t ntoh(int32_t x) { return ntohl(x); }\ninline int32_t hton(int32_t x) { return htonl(x); }\ninline int64_t ntoh(int64_t x) { return ntohq(x); }\ninline int64_t hton(int64_t x) { return htonq(x); }\n\n// Deprecated alias net::packed<> for unaligned<> from unaligned.hh.\n// TODO: get rid of this alias.\ntemplate <typename T> using packed = unaligned<T>;\n\ntemplate <typename T>\ninline T ntoh(const packed<T>& x) {\n    T v = x;\n    return ntoh(v);\n}\n\ntemplate <typename T>\ninline T hton(const packed<T>& x) {\n    T v = x;\n    return hton(v);\n}\n\ntemplate <typename T>\ninline std::ostream& operator<<(std::ostream& os, const packed<T>& v) {\n    auto x = v.raw;\n    return os << x;\n}\n\ninline\nvoid ntoh_inplace() {}\ninline\nvoid hton_inplace() {};\n\ntemplate <typename First, typename... Rest>\ninline\nvoid ntoh_inplace(First& first, Rest&... rest) {\n    first = ntoh(first);\n    ntoh_inplace(std::forward<Rest&>(rest)...);\n}\n\ntemplate <typename First, typename... Rest>\ninline\nvoid hton_inplace(First& first, Rest&... rest) {\n    first = hton(first);\n    hton_inplace(std::forward<Rest&>(rest)...);\n}\n\ntemplate <class T>\ninline\nT ntoh(const T& x) {\n    T tmp = x;\n    tmp.adjust_endianness([] (auto&&... what) { ntoh_inplace(std::forward<decltype(what)&>(what)...); });\n    return tmp;\n}\n\ntemplate <class T>\ninline\nT hton(const T& x) {\n    T tmp = x;\n    tmp.adjust_endianness([] (auto&&... what) { hton_inplace(std::forward<decltype(what)&>(what)...); });\n    return tmp;\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/config.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 Marek Waszkiewicz ( marek.waszkiewicz77@gmail.com )\n */\n\n#pragma once\n\n#include <optional>\n#include <istream>\n#include <string>\n#include <unordered_map>\n\nnamespace seastar {\nnamespace net {\n\n    struct ipv4_config {\n        std::string ip;\n        std::string netmask;\n        std::string gateway;\n        bool dhcp{ false };\n    };\n\n    struct hw_config {\n        std::string pci_address;\n        std::optional<unsigned> port_index;\n        bool lro{ true };\n        bool tso{ true };\n        bool ufo{ true };\n        bool hw_fc{ true };\n        bool event_index{ true };\n        bool csum_offload{ true };\n        std::optional<unsigned> ring_size;\n    };\n\n    struct device_config {\n        ipv4_config ip_cfg;\n        hw_config hw_cfg;\n    };\n\n    std::unordered_map<std::string, device_config> parse_config(std::istream& input);\n\n    class config_exception : public std::runtime_error {\n    public:\n        config_exception(const std::string& msg)\n            : std::runtime_error(msg) {\n        }\n    };\n}\n}\n"
  },
  {
    "path": "include/seastar/net/const.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cstdint>\n\nnamespace seastar {\n\nnamespace net {\n\nenum class ip_protocol_num : uint8_t {\n    icmp = 1, tcp = 6, udp = 17, unused = 255\n};\n\nenum class eth_protocol_num : uint16_t {\n    ipv4 = 0x0800, arp = 0x0806, ipv6 = 0x86dd\n};\n\nconst uint8_t eth_hdr_len = 14;\nconst uint8_t tcp_hdr_len_min = 20;\nconst uint8_t ipv4_hdr_len_min = 20;\nconst uint8_t ipv6_hdr_len_min = 40;\nconst uint16_t ip_packet_len_max = 65535;\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/dhcp.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#pragma once\n\n#include <seastar/net/ip.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\nnamespace net {\n\n/*\n * Simplistic DHCP query class.\n * Due to the nature of the native stack,\n * it operates on an \"ipv4\" object instead of,\n * for example, an interface.\n */\nclass dhcp {\npublic:\n    dhcp(ipv4 &);\n    dhcp(dhcp &&) noexcept;\n    ~dhcp();\n\n    static const steady_clock_type::duration default_timeout;\n\n    struct lease {\n        ipv4_address ip;\n        ipv4_address netmask;\n        ipv4_address broadcast;\n\n        ipv4_address gateway;\n        ipv4_address dhcp_server;\n\n        std::vector<ipv4_address> name_servers;\n\n        std::chrono::seconds lease_time;\n        std::chrono::seconds renew_time;\n        std::chrono::seconds rebind_time;\n\n        uint16_t mtu = 0;\n    };\n\n    typedef future<std::optional<lease>> result_type;\n\n    /**\n     * Runs a discover/request sequence on the ipv4 \"stack\".\n     * During this execution the ipv4 will be \"hijacked\"\n     * more or less (through packet filter), and while not\n     * inoperable, most likely quite less efficient.\n     *\n     * Please note that this does _not_ modify the ipv4 object bound.\n     * It only makes queries and records replys for the related NIC.\n     * It is up to caller to use the returned information as he se fit.\n     */\n    result_type discover(const steady_clock_type::duration & = default_timeout);\n    result_type renew(const lease &, const steady_clock_type::duration & = default_timeout);\n    ip_packet_filter* get_ipv4_filter();\nprivate:\n    class impl;\n    std::unique_ptr<impl> _impl;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/dns.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 Cloudius Systems\n */\n\n#pragma once\n\n#include <system_error>\n#include <vector>\n#include <unordered_map>\n#include <memory>\n#include <seastar/util/std-compat.hh>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/net/inet_address.hh>\n\nnamespace seastar {\n\nstruct ipv4_addr;\n\nclass socket_address;\nclass network_stack;\n\n/**\n * C-ares based dns query support.\n * Handles name- and ip-based resolution.\n *\n */\n\nnamespace net {\n\n/**\n * A c++-esque version of a hostent\n */\nstruct hostent {\n    // Primary name is always first\n    std::vector<sstring> names;\n    // Primary address is also always first.\n    [[deprecated(\"Use `addr_entries` instead\")]] std::vector<inet_address> addr_list;\n    struct address_entry {\n        inet_address addr;\n        // https://datatracker.ietf.org/doc/html/rfc1035#section-2.3.4\n        // https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.3\n        // https://datatracker.ietf.org/doc/html/rfc2181#section-8\n        std::chrono::seconds ttl{std::numeric_limits<signed int>::max()};\n    };\n    std::vector<address_entry> addr_entries;\n\n    // Remove the whole section below once we drop `addr_list`\n    // from the struct.\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n    hostent() = default;\n    hostent(std::vector<sstring>&& _names, std::vector<inet_address>&& _addr_list, std::vector<address_entry>&& _addr_entries)\n        : names(std::move(_names))\n        , addr_list(std::move(_addr_list))\n        , addr_entries(std::move(_addr_entries)) {\n    }\n    ~hostent() = default;\n    hostent(const hostent&) = default;\n    hostent& operator=(const hostent&) = default;\n    hostent(hostent&&) noexcept = default;\n    hostent& operator=(hostent&&) noexcept = default;\n#pragma GCC diagnostic pop\n};\n\ntypedef std::optional<inet_address::family> opt_family;\n\nstruct srv_record {\n    unsigned short priority;\n    unsigned short weight;\n    unsigned short port;\n    sstring target;\n};\n\n/**\n * A DNS resolver object.\n * Wraps the query logic & networking.\n * Can be instantiated with options and your network\n * stack of choice, though for \"normal\" non-test\n * querying, you are probably better of with the\n * global calls further down.\n */\nclass dns_resolver {\npublic:\n    struct options {\n        std::optional<bool>\n            use_tcp_query;\n        std::optional<std::vector<inet_address>>\n            servers;\n        std::optional<std::chrono::milliseconds>\n            timeout;\n        std::optional<uint16_t>\n            tcp_port, udp_port;\n        std::optional<std::vector<sstring>>\n            domains;\n    };\n\n    enum class srv_proto {\n        tcp, udp\n    };\n    using srv_records = std::vector<srv_record>;\n\n    dns_resolver();\n    dns_resolver(dns_resolver&&) noexcept;\n    explicit dns_resolver(const options&);\n    explicit dns_resolver(network_stack&, const options& = {});\n    ~dns_resolver();\n\n    dns_resolver& operator=(dns_resolver&&) noexcept;\n\n    /**\n     * Resolves a hostname to one or more addresses and aliases\n     */\n    future<hostent> get_host_by_name(const sstring&, opt_family = {});\n    /**\n     * Resolves an address to one or more addresses and aliases\n     */\n    future<hostent> get_host_by_addr(const inet_address&);\n\n    /**\n     * Resolves a hostname to one (primary) address\n     */\n    future<inet_address> resolve_name(const sstring&, opt_family = {});\n    /**\n     * Resolves an address to one (primary) name\n     */\n    future<sstring> resolve_addr(const inet_address&);\n\n    /**\n     * Resolve a service in given domain to one or more SRV records\n     */\n    future<srv_records> get_srv_records(srv_proto proto,\n                                        const sstring& service,\n                                        const sstring& domain);\n\n    /**\n     * Shuts the object down. Great for tests.\n     */\n    future<> close();\nprivate:\n    class impl;\n    shared_ptr<impl> _impl;\n};\n\nnamespace dns {\n\n// See above. These functions simply queries using a shard-local\n// default-stack, default-opts resolver\nfuture<hostent> get_host_by_name(const sstring&, opt_family = {});\nfuture<hostent> get_host_by_addr(const inet_address&);\n\nfuture<inet_address> resolve_name(const sstring&, opt_family = {});\nfuture<sstring> resolve_addr(const inet_address&);\n\nfuture<std::vector<srv_record>> get_srv_records(dns_resolver::srv_proto proto,\n                                                const sstring& service,\n                                                const sstring& domain);\n\n/**\n * Error handling.\n *\n * The error_category instance used by exceptions thrown by DNS\n */\n const std::error_category& error_category();\n\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/dpdk.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <memory>\n#include <seastar/net/config.hh>\n#include <seastar/net/net.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/program-options.hh>\n\nnamespace seastar {\n\nnamespace net {\n\n/// DPDK configuration.\nstruct dpdk_options : public program_options::option_group {\n    /// DPDK Port Index.\n    ///\n    /// Default: 0.\n    program_options::value<unsigned> dpdk_port_index;\n    /// \\brief Enable HW Flow Control (on / off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> hw_fc;\n\n    /// \\cond internal\n    dpdk_options(program_options::option_group* parent_group);\n    /// \\endcond\n};\n\n}\n\n/// \\cond internal\n\n#ifdef SEASTAR_HAVE_DPDK\n\nstd::unique_ptr<net::device> create_dpdk_net_device(\n                                    uint16_t port_idx = 0,\n                                    uint16_t num_queues = 1,\n                                    bool use_lro = true,\n                                    bool enable_fc = true);\n\nstd::unique_ptr<net::device> create_dpdk_net_device(\n                                    const net::hw_config& hw_cfg);\n\nnamespace dpdk {\n/**\n * @return Number of bytes needed for mempool objects of each QP.\n */\nuint32_t qp_mempool_obj_size(bool hugetlbfs_membackend);\n}\n\n/// \\endcond\n\n#endif // SEASTAR_HAVE_DPDK\n\n}\n"
  },
  {
    "path": "include/seastar/net/ethernet.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <fmt/ostream.h>\n\n#include <array>\n#include <algorithm>\n#include <seastar/net/byteorder.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nstruct ethernet_address {\n    ethernet_address() noexcept\n        : mac{} {}\n\n    ethernet_address(const uint8_t *eaddr) noexcept {\n        std::copy(eaddr, eaddr + 6, mac.begin());\n    }\n\n    ethernet_address(std::initializer_list<uint8_t> eaddr) noexcept {\n        SEASTAR_ASSERT(eaddr.size() == mac.size());\n        std::copy(eaddr.begin(), eaddr.end(), mac.begin());\n    }\n\n    std::array<uint8_t, 6> mac;\n\n    template <typename Adjuster>\n    void adjust_endianness(Adjuster) noexcept {}\n\n    static ethernet_address read(const char* p) noexcept {\n        ethernet_address ea;\n        std::copy_n(p, size(), reinterpret_cast<char*>(ea.mac.data()));\\\n        return ea;\n    }\n    static ethernet_address consume(const char*& p) noexcept {\n        auto ea = read(p);\n        p += size();\n        return ea;\n    }\n    void write(char* p) const noexcept {\n        std::copy_n(reinterpret_cast<const char*>(mac.data()), size(), p);\n    }\n    void produce(char*& p) const noexcept {\n        write(p);\n        p += size();\n    }\n    static constexpr size_t size() noexcept {\n        return 6;\n    }\n} __attribute__((packed));\n\nstd::ostream& operator<<(std::ostream& os, ethernet_address ea);\n\nstruct ethernet {\n    using address = ethernet_address;\n    static address broadcast_address() {\n        return  {0xff, 0xff, 0xff, 0xff, 0xff, 0xff};\n    }\n    static constexpr uint16_t arp_hardware_type() { return 1; }\n};\n\nstruct eth_hdr {\n    ethernet_address dst_mac;\n    ethernet_address src_mac;\n    packed<uint16_t> eth_proto;\n    template <typename Adjuster>\n    auto adjust_endianness(Adjuster a) {\n        return a(eth_proto);\n    }\n} __attribute__((packed));\n\nethernet_address parse_ethernet_address(std::string addr);\n}\n\n}\n\ntemplate <> struct fmt::formatter<seastar::net::ethernet_address> : fmt::ostream_formatter {};\n"
  },
  {
    "path": "include/seastar/net/inet_address.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n\n#include <iosfwd>\n#include <sys/types.h>\n#include <netinet/in.h>\n#include <optional>\n#include <stdexcept>\n#include <vector>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\nnamespace net {\n\n\nstruct ipv4_address;\nstruct ipv6_address;\n\nclass unknown_host : public std::invalid_argument {\npublic:\n    using invalid_argument::invalid_argument;\n};\n\nclass inet_address {\npublic:\n    enum class family : sa_family_t {\n        INET = AF_INET, INET6 = AF_INET6\n    };\nprivate:\n    family _in_family;\n\n    union {\n        ::in_addr _in;\n        ::in6_addr _in6;\n    };\n\n    uint32_t _scope = invalid_scope;\n\n    static future<std::vector<inet_address>> find_all_impl(const sstring& name, std::optional<family> family);\npublic:\n    static constexpr uint32_t invalid_scope = std::numeric_limits<uint32_t>::max();\n\n    inet_address() noexcept;\n    inet_address(family) noexcept;\n    inet_address(::in_addr i) noexcept;\n    inet_address(::in6_addr i, uint32_t scope = invalid_scope) noexcept;\n    // NOTE: does _not_ resolve the address. Only parses\n    // ipv4/ipv6 numerical address\n    // throws std::invalid_argument if sstring is invalid\n    inet_address(const sstring&);\n    inet_address(inet_address&&) noexcept = default;\n    inet_address(const inet_address&) noexcept = default;\n\n    inet_address(const ipv4_address&) noexcept;\n    inet_address(const ipv6_address&, uint32_t scope = invalid_scope) noexcept;\n\n    // throws iff ipv6\n    ipv4_address as_ipv4_address() const;\n    ipv6_address as_ipv6_address() const noexcept;\n\n    inet_address& operator=(const inet_address&) noexcept = default;\n    bool operator==(const inet_address&) const noexcept;\n\n    family in_family() const noexcept {\n        return _in_family;\n    }\n\n    bool is_ipv6() const noexcept {\n        return _in_family == family::INET6;\n    }\n    bool is_ipv4() const noexcept {\n        return _in_family == family::INET;\n    }\n\n    size_t size() const noexcept;\n    const void * data() const noexcept;\n\n    uint32_t scope() const noexcept {\n        return _scope;\n    }\n\n    // throws iff ipv6\n    operator ::in_addr() const;\n    operator ::in6_addr() const noexcept;\n\n    operator ipv6_address() const noexcept;\n\n    future<sstring> hostname() const;\n    future<std::vector<sstring>> aliases() const;\n\n    static future<inet_address> find(const sstring&);\n    static future<inet_address> find(const sstring&, family);\n    static future<std::vector<inet_address>> find_all(const sstring&);\n    static future<std::vector<inet_address>> find_all(const sstring&, family);\n\n    static std::optional<inet_address> parse_numerical(const sstring&);\n\n    bool is_loopback() const noexcept;\n    bool is_addr_any() const noexcept;\n};\n\nstd::ostream& operator<<(std::ostream&, const inet_address&);\nstd::ostream& operator<<(std::ostream&, const inet_address::family&);\n\n}\n}\n\nnamespace std {\ntemplate<>\nstruct hash<seastar::net::inet_address> {\n    size_t operator()(const seastar::net::inet_address&) const;\n};\n}\n\ntemplate <> struct fmt::formatter<seastar::net::inet_address> : fmt::ostream_formatter {};\ntemplate <> struct fmt::formatter<seastar::net::inet_address::family> : fmt::ostream_formatter {};\n"
  },
  {
    "path": "include/seastar/net/ip.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#pragma once\n\n#include <boost/asio/ip/address_v4.hpp>\n#include <arpa/inet.h>\n#include <unordered_map>\n#include <cstdint>\n#include <array>\n#include <map>\n#include <list>\n#include <chrono>\n\n#include <seastar/util/internal/array_map.hh>\n#include <seastar/net/byteorder.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/net/arp.hh>\n#include <seastar/net/ip_checksum.hh>\n#include <seastar/net/const.hh>\n#include <seastar/net/packet-util.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/net/toeplitz.hh>\n#include <seastar/net/udp.hh>\n#include <seastar/core/metrics_registration.hh>\n\n#include \"ipv4_address.hh\"\n#include \"ipv6_address.hh\"\n\nnamespace seastar {\n\nnamespace net {\n\nclass ipv4;\ntemplate <ip_protocol_num ProtoNum>\nclass ipv4_l4;\n\ntemplate <typename InetTraits>\nclass tcp;\n\nstruct ipv4_traits {\n    using address_type = ipv4_address;\n    using inet_type = ipv4_l4<ip_protocol_num::tcp>;\n    struct l4packet {\n        ipv4_address to;\n        packet p;\n        ethernet_address e_dst;\n        ip_protocol_num proto_num;\n    };\n    using packet_provider_type = std::function<std::optional<l4packet> ()>;\n    static void tcp_pseudo_header_checksum(checksummer& csum, ipv4_address src, ipv4_address dst, uint16_t len) {\n        csum.sum_many(src.ip.raw, dst.ip.raw, uint8_t(0), uint8_t(ip_protocol_num::tcp), len);\n    }\n    static void udp_pseudo_header_checksum(checksummer& csum, ipv4_address src, ipv4_address dst, uint16_t len) {\n        csum.sum_many(src.ip.raw, dst.ip.raw, uint8_t(0), uint8_t(ip_protocol_num::udp), len);\n    }\n    static constexpr uint8_t ip_hdr_len_min = ipv4_hdr_len_min;\n};\n\ntemplate <ip_protocol_num ProtoNum>\nclass ipv4_l4 {\npublic:\n    ipv4& _inet;\npublic:\n    ipv4_l4(ipv4& inet) : _inet(inet) {}\n    void register_packet_provider(ipv4_traits::packet_provider_type func);\n    future<ethernet_address> get_l2_dst_address(ipv4_address to);\n    const ipv4& inet() const {\n        return _inet;\n    }\n};\n\nclass ip_protocol {\npublic:\n    virtual ~ip_protocol() {}\n    virtual void received(packet p, ipv4_address from, ipv4_address to) = 0;\n    virtual bool forward(forward_hash& out_hash_data, packet& p, size_t off) {\n      std::ignore = out_hash_data;\n      std::ignore = p;\n      std::ignore = off;\n      return true;\n    }\n};\n\ntemplate <typename InetTraits>\nstruct l4connid {\n    using ipaddr = typename InetTraits::address_type;\n    using inet_type = typename InetTraits::inet_type;\n    struct connid_hash;\n\n    ipaddr local_ip;\n    ipaddr foreign_ip;\n    uint16_t local_port;\n    uint16_t foreign_port;\n\n    bool operator==(const l4connid& x) const {\n        return local_ip == x.local_ip\n                && foreign_ip == x.foreign_ip\n                && local_port == x.local_port\n                && foreign_port == x.foreign_port;\n    }\n\n    uint32_t hash(rss_key_type rss_key) {\n        forward_hash hash_data;\n        hash_data.push_back(hton(foreign_ip.ip));\n        hash_data.push_back(hton(local_ip.ip));\n        hash_data.push_back(hton(foreign_port));\n        hash_data.push_back(hton(local_port));\n        return toeplitz_hash(rss_key, hash_data);\n    }\n};\n\nclass ipv4_tcp final : public ip_protocol {\n    ipv4_l4<ip_protocol_num::tcp> _inet_l4;\n    std::unique_ptr<tcp<ipv4_traits>> _tcp;\npublic:\n    ipv4_tcp(ipv4& inet);\n    ~ipv4_tcp();\n    virtual void received(packet p, ipv4_address from, ipv4_address to) override;\n    virtual bool forward(forward_hash& out_hash_data, packet& p, size_t off) override;\n    friend class ipv4;\n};\n\nstruct icmp_hdr {\n    enum class msg_type : uint8_t {\n        echo_reply = 0,\n        echo_request = 8,\n    };\n    msg_type type;\n    uint8_t code;\n    packed<uint16_t> csum;\n    packed<uint32_t> rest;\n    template <typename Adjuster>\n    auto adjust_endianness(Adjuster a) {\n        return a(csum);\n    }\n} __attribute__((packed));\n\n\nclass icmp {\npublic:\n    using ipaddr = ipv4_address;\n    using inet_type = ipv4_l4<ip_protocol_num::icmp>;\n    explicit icmp(inet_type& inet) : _inet(inet) {\n        _inet.register_packet_provider([this] {\n            std::optional<ipv4_traits::l4packet> l4p;\n            if (!_packetq.empty()) {\n                l4p = std::move(_packetq.front());\n                _packetq.pop_front();\n                _queue_space.signal(l4p.value().p.len());\n            }\n            return l4p;\n        });\n    }\n    void received(packet p, ipaddr from, ipaddr to);\nprivate:\n    inet_type& _inet;\n    circular_buffer<ipv4_traits::l4packet> _packetq;\n    semaphore _queue_space = {212992};\n};\n\nclass ipv4_icmp final : public ip_protocol {\n    ipv4_l4<ip_protocol_num::icmp> _inet_l4;\n    icmp _icmp;\npublic:\n    ipv4_icmp(ipv4& inet) : _inet_l4(inet), _icmp(_inet_l4) {}\n    virtual void received(packet p, ipv4_address from, ipv4_address to) {\n        _icmp.received(std::move(p), from, to);\n    }\n    friend class ipv4;\n};\n\nclass ipv4_udp : public ip_protocol {\n    using connid = l4connid<ipv4_traits>;\n    using connid_hash = typename connid::connid_hash;\n\npublic:\n    static const int default_queue_size;\nprivate:\n    static const uint16_t min_anonymous_port = 32768;\n    ipv4 &_inet;\n    std::unordered_map<uint16_t, lw_shared_ptr<udp_channel_state>> _channels;\n    int _queue_size = default_queue_size;\n    uint16_t _next_anonymous_port = min_anonymous_port;\n    circular_buffer<ipv4_traits::l4packet> _packetq;\nprivate:\n    uint16_t next_port(uint16_t port);\npublic:\n    class registration {\n    private:\n        ipv4_udp &_proto;\n        uint16_t _port;\n    public:\n        registration(ipv4_udp &proto, uint16_t port) : _proto(proto), _port(port) {};\n\n        void unregister() {\n            _proto._channels.erase(_proto._channels.find(_port));\n        }\n\n        uint16_t port() const {\n            return _port;\n        }\n    };\n\n    ipv4_udp(ipv4& inet);\n    udp_channel make_channel(ipv4_addr addr);\n    virtual void received(packet p, ipv4_address from, ipv4_address to) override;\n    void send(uint16_t src_port, ipv4_addr dst, packet &&p);\n    bool forward(forward_hash& out_hash_data, packet& p, size_t off) override;\n    void set_queue_size(int size) { _queue_size = size; }\n\n    const ipv4& inet() const {\n        return _inet;\n    }\n};\n\nstruct ip_hdr;\n\nstruct ip_packet_filter {\n    virtual ~ip_packet_filter() {};\n    virtual future<> handle(packet& p, ip_hdr* iph, ethernet_address from, bool & handled) = 0;\n};\n\nstruct ipv4_frag_id {\n    struct hash;\n    ipv4_address src_ip;\n    ipv4_address dst_ip;\n    uint16_t identification;\n    uint8_t protocol;\n    bool operator==(const ipv4_frag_id& x) const {\n        return src_ip == x.src_ip &&\n               dst_ip == x.dst_ip &&\n               identification == x.identification &&\n               protocol == x.protocol;\n    }\n};\n\nstruct ipv4_frag_id::hash : private std::hash<ipv4_address>,\n    private std::hash<uint16_t>, private std::hash<uint8_t> {\n    size_t operator()(const ipv4_frag_id& id) const noexcept {\n        using h1 = std::hash<ipv4_address>;\n        using h2 = std::hash<uint16_t>;\n        using h3 = std::hash<uint8_t>;\n        return h1::operator()(id.src_ip) ^\n               h1::operator()(id.dst_ip) ^\n               h2::operator()(id.identification) ^\n               h3::operator()(id.protocol);\n    }\n};\n\nstruct ipv4_tag {};\nusing ipv4_packet_merger = packet_merger<uint32_t, ipv4_tag>;\n\nclass ipv4 {\npublic:\n    using clock_type = lowres_clock;\n    using address_type = ipv4_address;\n    using proto_type = uint16_t;\n    static address_type broadcast_address() { return ipv4_address(0xffffffff); }\n    static proto_type arp_protocol_type() { return proto_type(eth_protocol_num::ipv4); }\nprivate:\n    interface* _netif;\n    std::vector<ipv4_traits::packet_provider_type> _pkt_providers;\n    arp _global_arp;\n    arp_for<ipv4> _arp;\n    ipv4_address _host_address;\n    ipv4_address _gw_address;\n    ipv4_address _netmask;\n    l3_protocol _l3;\n    ipv4_tcp _tcp;\n    ipv4_icmp _icmp;\n    ipv4_udp _udp;\n    internal::array_map<ip_protocol*, 256> _l4;\n    ip_packet_filter * _packet_filter = nullptr;\n    struct frag {\n        packet header;\n        ipv4_packet_merger data;\n        clock_type::time_point rx_time;\n        uint32_t mem_size = 0;\n        // fragment with MF == 0 inidates it is the last fragment\n        bool last_frag_received = false;\n\n        packet get_assembled_packet(ethernet_address from, ethernet_address to);\n        int32_t merge(ip_hdr &h, uint16_t offset, packet p);\n        bool is_complete();\n    };\n    std::unordered_map<ipv4_frag_id, frag, ipv4_frag_id::hash> _frags;\n    std::list<ipv4_frag_id> _frags_age;\n    static constexpr std::chrono::seconds _frag_timeout{30};\n    static constexpr uint32_t _frag_low_thresh{3 * 1024 * 1024};\n    static constexpr uint32_t _frag_high_thresh{4 * 1024 * 1024};\n    uint32_t _frag_mem{0};\n    timer<lowres_clock> _frag_timer;\n    circular_buffer<l3_protocol::l3packet> _packetq;\n    unsigned _pkt_provider_idx = 0;\n    metrics::metric_groups _metrics;\nprivate:\n    future<> handle_received_packet(packet p, ethernet_address from);\n    bool forward(forward_hash& out_hash_data, packet& p, size_t off);\n    std::optional<l3_protocol::l3packet> get_packet();\n    bool in_my_netmask(ipv4_address a) const;\n    void frag_limit_mem();\n    void frag_timeout();\n    void frag_drop(ipv4_frag_id frag_id, uint32_t dropped_size);\n    void frag_arm(clock_type::time_point now) {\n        auto tp = now + _frag_timeout;\n        _frag_timer.arm(tp);\n    }\n    void frag_arm() {\n        auto now = clock_type::now();\n        frag_arm(now);\n    }\npublic:\n    explicit ipv4(interface* netif);\n    void set_host_address(ipv4_address ip);\n    ipv4_address host_address() const;\n    void set_gw_address(ipv4_address ip);\n    ipv4_address gw_address() const;\n    void set_netmask_address(ipv4_address ip);\n    ipv4_address netmask_address() const;\n    interface * netif() const {\n        return _netif;\n    }\n    // TODO or something. Should perhaps truly be a list\n    // of filters. With ordering. And blackjack. Etc.\n    // But for now, a simple single raw pointer suffices\n    void set_packet_filter(ip_packet_filter *);\n    ip_packet_filter * packet_filter() const;\n    void send(ipv4_address to, ip_protocol_num proto_num, packet p, ethernet_address e_dst);\n    tcp<ipv4_traits>& get_tcp() { return *_tcp._tcp; }\n    ipv4_udp& get_udp() { return _udp; }\n    void register_l4(proto_type id, ip_protocol* handler);\n    const net::hw_features& hw_features() const { return _netif->hw_features(); }\n    static bool needs_frag(packet& p, ip_protocol_num proto_num, net::hw_features hw_features);\n    void learn(ethernet_address l2, ipv4_address l3) {\n        _arp.learn(l2, l3);\n    }\n    void register_packet_provider(ipv4_traits::packet_provider_type&& func) {\n        _pkt_providers.push_back(std::move(func));\n    }\n    future<ethernet_address> get_l2_dst_address(ipv4_address to);\n};\n\ntemplate <ip_protocol_num ProtoNum>\ninline\nvoid ipv4_l4<ProtoNum>::register_packet_provider(ipv4_traits::packet_provider_type func) {\n    _inet.register_packet_provider([func = std::move(func)] {\n        auto l4p = func();\n        if (l4p) {\n            l4p.value().proto_num = ProtoNum;\n        }\n        return l4p;\n    });\n}\n\ntemplate <ip_protocol_num ProtoNum>\ninline\nfuture<ethernet_address> ipv4_l4<ProtoNum>::get_l2_dst_address(ipv4_address to) {\n    return _inet.get_l2_dst_address(to);\n}\n\nstruct ip_hdr {\n    uint8_t ihl : 4;\n    uint8_t ver : 4;\n    uint8_t dscp : 6;\n    uint8_t ecn : 2;\n    packed<uint16_t> len;\n    packed<uint16_t> id;\n    packed<uint16_t> frag;\n    enum class frag_bits : uint8_t { mf = 13, df = 14, reserved = 15, offset_shift = 3 };\n    uint8_t ttl;\n    uint8_t ip_proto;\n    packed<uint16_t> csum;\n    ipv4_address src_ip;\n    ipv4_address dst_ip;\n    uint8_t options[0];\n    template <typename Adjuster>\n    auto adjust_endianness(Adjuster a) {\n        return a(len, id, frag, csum, src_ip, dst_ip);\n    }\n    bool mf() { return frag & (1 << uint8_t(frag_bits::mf)); }\n    bool df() { return frag & (1 << uint8_t(frag_bits::df)); }\n    uint16_t offset() { return frag << uint8_t(frag_bits::offset_shift); }\n} __attribute__((packed));\n\ntemplate <typename InetTraits>\nstruct l4connid<InetTraits>::connid_hash : private std::hash<ipaddr>, private std::hash<uint16_t> {\n    size_t operator()(const l4connid<InetTraits>& id) const noexcept {\n        using h1 = std::hash<ipaddr>;\n        using h2 = std::hash<uint16_t>;\n        return h1::operator()(id.local_ip)\n            ^ h1::operator()(id.foreign_ip)\n            ^ h2::operator()(id.local_port)\n            ^ h2::operator()(id.foreign_port);\n    }\n};\n\nvoid arp_learn(ethernet_address l2, ipv4_address l3);\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/ip_checksum.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/net/packet.hh>\n#include <cstdint>\n#include <cstddef>\n#include <arpa/inet.h>\n\nnamespace seastar {\n\nnamespace net {\n\nuint16_t ip_checksum(const void* data, size_t len);\n\nstruct checksummer {\n    __int128 csum = 0;\n    bool odd = false;\n    void sum(const char* data, size_t len);\n    void sum(const packet& p);\n    void sum(uint8_t data) {\n        if (!odd) {\n            csum += data << 8;\n        } else {\n            csum += data;\n        }\n        odd = !odd;\n    }\n    void sum(uint16_t data) {\n        if (odd) {\n            sum(uint8_t(data >> 8));\n            sum(uint8_t(data));\n        } else {\n            csum += data;\n        }\n    }\n    void sum(uint32_t data) {\n        if (odd) {\n            sum(uint16_t(data));\n            sum(uint16_t(data >> 16));\n        } else {\n            csum += data;\n        }\n    }\n    void sum_many() {}\n    template <typename T0, typename... T>\n    void sum_many(T0 data, T... rest) {\n        sum(data);\n        sum_many(rest...);\n    }\n    uint16_t get() const;\n};\n\n}\n\n}\n\n"
  },
  {
    "path": "include/seastar/net/ipv4_address.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB\n *\n */\n\n#pragma once\n\n#include <seastar/net/socket_defs.hh>\n#include <seastar/core/byteorder.hh>\n\nnamespace seastar {\n\nnamespace net {\n\n\nstruct ipv4_address {\n    ipv4_address() noexcept : ip(0) {}\n    explicit ipv4_address(uint32_t ip) noexcept : ip(ip) {}\n    // throws if addr is not a valid ipv4 address\n    explicit ipv4_address(const std::string& addr);\n    ipv4_address(ipv4_addr addr) noexcept {\n        ip = addr.ip;\n    }\n\n    packed<uint32_t> ip;\n\n    template <typename Adjuster>\n    auto adjust_endianness(Adjuster a) { return a(ip); }\n\n    friend bool operator==(ipv4_address x, ipv4_address y) noexcept {\n        return x.ip == y.ip;\n    }\n    friend bool operator!=(ipv4_address x, ipv4_address y) noexcept {\n        return x.ip != y.ip;\n    }\n\n    static ipv4_address read(const char* p) noexcept {\n        ipv4_address ia;\n        ia.ip = read_be<uint32_t>(p);\n        return ia;\n    }\n    static ipv4_address consume(const char*& p) noexcept {\n        auto ia = read(p);\n        p += 4;\n        return ia;\n    }\n    void write(char* p) const noexcept {\n        write_be<uint32_t>(p, ip);\n    }\n    void produce(char*& p) const noexcept {\n        produce_be<uint32_t>(p, ip);\n    }\n    static constexpr size_t size() {\n        return 4;\n    }\n} __attribute__((packed));\n\ninline bool is_unspecified(ipv4_address addr) noexcept { return addr.ip == 0; }\n\nstd::ostream& operator<<(std::ostream& os, const ipv4_address& a);\n\n\n}\n\n}\n\nnamespace std {\n\ntemplate <>\nstruct hash<seastar::net::ipv4_address> {\n    size_t operator()(seastar::net::ipv4_address a) const { return a.ip; }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/net/ipv6_address.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB\n *\n */\n\n#pragma once\n\n#include <seastar/net/socket_defs.hh>\n#include <seastar/core/byteorder.hh>\n\n#include <array>\n\nnamespace seastar {\n\nnamespace net {\n\nstruct ipv6_address {\n    using ipv6_bytes = std::array<uint8_t, 16>;\n\n    static_assert(alignof(ipv6_bytes) == 1, \"ipv6_bytes should be byte-aligned\");\n    static_assert(sizeof(ipv6_bytes) == 16, \"ipv6_bytes should be 16 bytes\");\n\n    ipv6_address() noexcept;\n    explicit ipv6_address(const ::in6_addr&) noexcept;\n    explicit ipv6_address(const ipv6_bytes&) noexcept;\n    // throws if addr is not a valid ipv6 address\n    explicit ipv6_address(const std::string&);\n    ipv6_address(const ipv6_addr& addr) noexcept;\n\n    // No need to use packed - we only store\n    // as byte array. If we want to read as\n    // uints or whatnot, we must copy\n    ipv6_bytes ip;\n\n    template <typename Adjuster>\n    auto adjust_endianness(Adjuster a) { return a(ip); }\n\n    bool operator==(const ipv6_address& y) const noexcept {\n        return bytes() == y.bytes();\n    }\n    bool operator!=(const ipv6_address& y) const noexcept {\n        return !(*this == y);\n    }\n\n    const ipv6_bytes& bytes() const noexcept {\n        return ip;\n    }\n\n    bool is_unspecified() const noexcept;\n\n    static ipv6_address read(const char*) noexcept;\n    static ipv6_address consume(const char*& p) noexcept;\n    void write(char* p) const noexcept;\n    void produce(char*& p) const noexcept;\n    static constexpr size_t size() {\n        return sizeof(ipv6_bytes);\n    }\n} __attribute__((packed));\n\nstd::ostream& operator<<(std::ostream&, const ipv6_address&);\n\n}\n\n}\n\nnamespace std {\n\ntemplate <>\nstruct hash<seastar::net::ipv6_address> {\n    size_t operator()(const seastar::net::ipv6_address&) const;\n};\n\n}\n"
  },
  {
    "path": "include/seastar/net/native-stack.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/net/net.hh>\n#include <seastar/net/virtio.hh>\n#include <seastar/net/dpdk.hh>\n#include <seastar/util/program-options.hh>\n\nnamespace seastar {\n\nstruct network_stack_entry;\n\nnamespace net {\n\n/// Native stack configuration.\nstruct native_stack_options : public program_options::option_group {\n    program_options::value<std::string> net_config;\n    program_options::value<std::string> net_config_file;\n    /// \\brief Tap device to connect to.\n    ///\n    /// Default: \\p tap0.\n    program_options::value<std::string> tap_device;\n    /// \\brief Static IPv4 address to use.\n    ///\n    /// Default: \\p 192.168.122.2.\n    program_options::value<std::string> host_ipv4_addr;\n    /// \\brief Static IPv4 gateway to use.\n    ///\n    /// Default: \\p 192.168.122.1.\n    program_options::value<std::string> gw_ipv4_addr;\n    /// \\brief Static IPv4 netmask to use.\n    ///\n    /// Default: \\p 255.255.255.0.\n    program_options::value<std::string> netmask_ipv4_addr;\n    /// \\brief Default size of the UDPv4 per-channel packet queue.\n    ///\n    /// Default: \\ref ipv4_udp::default_queue_size.\n    program_options::value<int> udpv4_queue_size;\n    /// \\brief Use DHCP discovery.\n    ///\n    /// Default: \\p true.\n    program_options::value<bool> dhcp;\n    /// \\brief Weighing of a hardware network queue relative to a software queue.\n    ///\n    /// Values:\n    /// * 0.0: no work\n    /// * 1.0: equal share\n    ///\n    /// Default: 1.0.\n    program_options::value<float> hw_queue_weight;\n    /// \\brief Use DPDK PMD drivers.\n    ///\n    /// \\note Unused when seastar is compiled without DPDK support.\n    program_options::value<> dpdk_pmd;\n    /// \\brief Enable LRO (on/off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> lro;\n\n    /// Virtio configuration.\n    virtio_options virtio_opts;\n    /// DPDK configuration.\n    ///\n    /// \\note Unused when seastar is compiled without DPDK support.\n    dpdk_options dpdk_opts;\n\n    /// \\cond internal\n    bool _hugepages;\n\n    native_stack_options();\n    /// \\endcond\n};\n\nvoid create_native_stack(const native_stack_options& opts, std::shared_ptr<device> dev);\nnetwork_stack_entry register_native_stack();\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/net.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/smp.hh>\n#include <seastar/core/deleter.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/net/toeplitz.hh>\n#include <seastar/net/ethernet.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/net/const.hh>\n#include <seastar/util/assert.hh>\n#include <map>\n#include <unordered_map>\n\nnamespace seastar {\n\nnamespace internal {\n\nclass poller;\n\n}\n\nnamespace net {\n\nclass packet;\nclass interface;\nclass device;\nclass qp;\nclass l3_protocol;\n\nclass forward_hash {\n    uint8_t data[64];\n    size_t end_idx = 0;\npublic:\n    size_t size() const {\n        return end_idx;\n    }\n    void push_back(uint8_t b) {\n        SEASTAR_ASSERT(end_idx < sizeof(data));\n        data[end_idx++] = b;\n    }\n    void push_back(uint16_t b) {\n        push_back(uint8_t(b));\n        push_back(uint8_t(b >> 8));\n    }\n    void push_back(uint32_t b) {\n        push_back(uint16_t(b));\n        push_back(uint16_t(b >> 16));\n    }\n    const uint8_t& operator[](size_t idx) const {\n        return data[idx];\n    }\n};\n\nstruct hw_features {\n    // Enable tx ip header checksum offload\n    bool tx_csum_ip_offload = false;\n    // Enable tx l4 (TCP or UDP) checksum offload\n    bool tx_csum_l4_offload = false;\n    // Enable rx checksum offload\n    bool rx_csum_offload = false;\n    // LRO is enabled\n    bool rx_lro = false;\n    // Enable tx TCP segment offload\n    bool tx_tso = false;\n    // Enable tx UDP fragmentation offload\n    bool tx_ufo = false;\n    // Maximum Transmission Unit\n    uint16_t mtu = 1500;\n    // Maximun packet len when TCP/UDP offload is enabled\n    uint16_t max_packet_len = ip_packet_len_max - eth_hdr_len;\n};\n\nclass l3_protocol {\npublic:\n    struct l3packet {\n        eth_protocol_num proto_num;\n        ethernet_address to;\n        packet p;\n    };\n    using packet_provider_type = std::function<std::optional<l3packet> ()>;\nprivate:\n    interface* _netif;\n    eth_protocol_num _proto_num;\npublic:\n    explicit l3_protocol(interface* netif, eth_protocol_num proto_num, packet_provider_type func);\n    future<> receive(\n            std::function<future<> (packet, ethernet_address)> rx_fn,\n            std::function<bool (forward_hash&, packet&, size_t)> forward);\nprivate:\n    friend class interface;\n};\n\nclass interface {\n    struct l3_rx_stream {\n        stream<packet, ethernet_address> packet_stream;\n        future<> ready;\n        std::function<bool (forward_hash&, packet&, size_t)> forward;\n        l3_rx_stream(std::function<bool (forward_hash&, packet&, size_t)>&& fw) : ready(packet_stream.started()), forward(fw) {}\n    };\n    std::unordered_map<uint16_t, l3_rx_stream> _proto_map;\n    std::shared_ptr<device> _dev;\n    ethernet_address _hw_address;\n    net::hw_features _hw_features;\n    std::vector<l3_protocol::packet_provider_type> _pkt_providers;\nprivate:\n    future<> dispatch_packet(packet p);\npublic:\n    explicit interface(std::shared_ptr<device> dev);\n    ethernet_address hw_address() const noexcept { return _hw_address; }\n    const net::hw_features& hw_features() const { return _hw_features; }\n    future<> register_l3(eth_protocol_num proto_num,\n            std::function<future<> (packet p, ethernet_address from)> next,\n            std::function<bool (forward_hash&, packet&, size_t)> forward);\n    void forward(unsigned cpuid, packet p);\n    unsigned hash2cpu(uint32_t hash);\n    void register_packet_provider(l3_protocol::packet_provider_type func) {\n        _pkt_providers.push_back(std::move(func));\n    }\n    uint16_t hw_queues_count();\n    rss_key_type rss_key() const;\n    friend class l3_protocol;\n};\n\nstruct qp_stats_good {\n    /**\n     * Update the packets bunch related statistics.\n     *\n     * Update the last packets bunch size and the total packets counter.\n     *\n     * @param count Number of packets in the last packets bunch.\n     */\n    void update_pkts_bunch(uint64_t count) {\n        last_bunch = count;\n        packets   += count;\n    }\n\n    /**\n     * Increment the appropriate counters when a few fragments have been\n     * processed in a copy-way.\n     *\n     * @param nr_frags Number of copied fragments\n     * @param bytes    Number of copied bytes\n     */\n    void update_copy_stats(uint64_t nr_frags, uint64_t bytes) {\n        copy_frags += nr_frags;\n        copy_bytes += bytes;\n    }\n\n    /**\n     * Increment total fragments and bytes statistics\n     *\n     * @param nfrags Number of processed fragments\n     * @param nbytes Number of bytes in the processed fragments\n     */\n    void update_frags_stats(uint64_t nfrags, uint64_t nbytes) {\n        nr_frags += nfrags;\n        bytes    += nbytes;\n    }\n\n    uint64_t bytes;      // total number of bytes\n    uint64_t nr_frags;   // total number of fragments\n    uint64_t copy_frags; // fragments that were copied on L2 level\n    uint64_t copy_bytes; // bytes that were copied on L2 level\n    uint64_t packets;    // total number of packets\n    uint64_t last_bunch; // number of packets in the last sent/received bunch\n};\n\nstruct qp_stats {\n    qp_stats() : rx{}, tx{} {}\n\n    struct {\n        struct qp_stats_good good;\n\n        struct {\n            void inc_csum_err() {\n                ++csum;\n                ++total;\n            }\n\n            void inc_no_mem() {\n                ++no_mem;\n                ++total;\n            }\n\n            uint64_t no_mem;       // Packets dropped due to allocation failure\n            uint64_t total;        // total number of erroneous packets\n            uint64_t csum;         // packets with bad checksum\n        } bad;\n    } rx;\n\n    struct {\n        struct qp_stats_good good;\n        uint64_t linearized;       // number of packets that were linearized\n    } tx;\n};\n\nclass qp {\n    using packet_provider_type = std::function<std::optional<packet> ()>;\n    std::vector<packet_provider_type> _pkt_providers;\n    std::optional<std::array<uint8_t, 128>> _sw_reta;\n    circular_buffer<packet> _proxy_packetq;\n    stream<packet> _rx_stream;\n    std::unique_ptr<internal::poller> _tx_poller;\n    circular_buffer<packet> _tx_packetq;\n\nprotected:\n    const std::string _stats_plugin_name;\n    const std::string _queue_name;\n    metrics::metric_groups _metrics;\n    qp_stats _stats;\n\npublic:\n    qp(bool register_copy_stats = false,\n       const std::string stats_plugin_name = std::string(\"network\"),\n       uint8_t qid = 0);\n    virtual ~qp();\n    virtual future<> send(packet p) = 0;\n    virtual uint32_t send(circular_buffer<packet>& p) {\n        uint32_t sent = 0;\n        while (!p.empty()) {\n            // FIXME: future is discarded\n            (void)send(std::move(p.front()));\n            p.pop_front();\n            sent++;\n        }\n        return sent;\n    }\n    virtual void rx_start() {};\n    void configure_proxies(const std::map<unsigned, float>& cpu_weights);\n    // build REdirection TAble for cpu_weights map: target cpu -> weight\n    void build_sw_reta(const std::map<unsigned, float>& cpu_weights);\n    void proxy_send(packet p) {\n        _proxy_packetq.push_back(std::move(p));\n    }\n    void register_packet_provider(packet_provider_type func) {\n        _pkt_providers.push_back(std::move(func));\n    }\n    bool poll_tx();\n    friend class device;\n};\n\nclass device {\nprotected:\n    std::unique_ptr<qp*[]> _queues;\n    size_t _rss_table_bits = 0;\npublic:\n    device() {\n        _queues = std::make_unique<qp*[]>(smp::count);\n    }\n    virtual ~device() {};\n    qp& queue_for_cpu(unsigned cpu) { return *_queues[cpu]; }\n    qp& local_queue() { return queue_for_cpu(this_shard_id()); }\n    void l2receive(packet p) {\n        // FIXME: future is discarded\n        (void)_queues[this_shard_id()]->_rx_stream.produce(std::move(p));\n    }\n    future<> receive(std::function<future<> (packet)> next_packet);\n    virtual ethernet_address hw_address() = 0;\n    virtual net::hw_features hw_features() = 0;\n    virtual rss_key_type rss_key() const { return default_rsskey_40bytes; }\n    virtual uint16_t hw_queues_count() { return 1; }\n    virtual future<> link_ready() { return make_ready_future<>(); }\n    virtual std::unique_ptr<qp> init_local_queue(const program_options::option_group& opts, uint16_t qid) = 0;\n    virtual unsigned hash2qid(uint32_t hash) {\n        return hash % hw_queues_count();\n    }\n    void set_local_queue(std::unique_ptr<qp> dev);\n    template <typename Func>\n    unsigned forward_dst(unsigned src_cpuid, Func&& hashfn) {\n        auto& qp = queue_for_cpu(src_cpuid);\n        if (!qp._sw_reta) {\n            return src_cpuid;\n        }\n        auto hash = hashfn() >> _rss_table_bits;\n        auto& reta = *qp._sw_reta;\n        return reta[hash % reta.size()];\n    }\n    virtual unsigned hash2cpu(uint32_t hash) {\n        // there is an assumption here that qid == cpu_id which will\n        // not necessary be true in the future\n        return forward_dst(hash2qid(hash), [hash] { return hash; });\n    }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/packet-data-source.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n#pragma once\n\n#include <seastar/util/memory-data-source.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nclass [[deprecated(\"Use util::memory_data_source\")]] packet_data_source final : public data_source_impl {\n    util::memory_data_source _mds;\n\npublic:\n    explicit packet_data_source(net::packet&& p)\n        : _mds(p.release())\n    {}\n\n    virtual future<temporary_buffer<char>> get() override {\n        return _mds.get();\n    }\n};\n\n[[deprecated(\"Use util::as_input_stream\")]]\nstatic inline\ninput_stream<char> as_input_stream(packet&& p) {\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n    return input_stream<char>(data_source(std::make_unique<packet_data_source>(std::move(p))));\n#pragma GCC diagnostic pop\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/packet-util.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/net/packet.hh>\n#include <map>\n\nnamespace seastar {\n\nnamespace net {\n\ntemplate <typename Offset, typename Tag>\nclass packet_merger {\nprivate:\n    static uint64_t& linearizations_ref() {\n        static thread_local uint64_t linearization_count;\n        return linearization_count;\n    }\npublic:\n    std::map<Offset, packet> map;\n\n    static uint64_t linearizations() {\n        return linearizations_ref();\n    }\n\n    void merge(Offset offset, packet p) {\n        bool insert = true;\n        auto beg = offset;\n        auto end = beg + p.len();\n        // Fisrt, try to merge the packet with existing segment\n        for (auto it = map.begin(); it != map.end();) {\n            auto& seg_pkt = it->second;\n            auto seg_beg = it->first;\n            auto seg_end = seg_beg + seg_pkt.len();\n            // There are 6 cases:\n            if (seg_beg <= beg && end <= seg_end) {\n                // 1) seg_beg beg end seg_end\n                // We already have data in this packet\n                return;\n            } else if (beg <= seg_beg && seg_end <= end) {\n                // 2) beg seg_beg seg_end end\n                // The new segment contains more data than this old segment\n                // Delete the old one, insert the new one\n                it = map.erase(it);\n                insert = true;\n                break;\n            } else if (beg < seg_beg && seg_beg <= end && end <= seg_end) {\n                // 3) beg seg_beg end seg_end\n                // Merge two segments, trim front of old segment\n                auto trim = end - seg_beg;\n                seg_pkt.trim_front(trim);\n                p.append(std::move(seg_pkt));\n                // Delete the old one, insert the new one\n                it = map.erase(it);\n                insert = true;\n                break;\n            } else if (seg_beg <= beg && beg <= seg_end && seg_end < end) {\n                // 4) seg_beg beg seg_end end\n                // Merge two segments, trim front of new segment\n                auto trim = seg_end - beg;\n                p.trim_front(trim);\n                // Append new data to the old segment, keep the old segment\n                seg_pkt.append(std::move(p));\n                seg_pkt.linearize();\n                ++linearizations_ref();\n                insert = false;\n                break;\n            } else {\n                // 5) beg end < seg_beg seg_end\n                //   or\n                // 6) seg_beg seg_end < beg end\n                // Can not merge with this segment, keep looking\n                it++;\n                insert = true;\n            }\n        }\n\n        if (insert) {\n            p.linearize();\n            ++linearizations_ref();\n            map.emplace(beg, std::move(p));\n        }\n\n        // Second, merge adjacent segments after this packet has been merged,\n        // becasue this packet might fill a \"whole\" and make two adjacent\n        // segments mergable\n        for (auto it = map.begin(); it != map.end();) {\n            // The first segment\n            auto& seg_pkt = it->second;\n            auto seg_beg = it->first;\n            auto seg_end = seg_beg + seg_pkt.len();\n\n            // The second segment\n            auto it_next = it;\n            it_next++;\n            if (it_next == map.end()) {\n                break;\n            }\n            auto& p = it_next->second;\n            auto beg = it_next->first;\n            auto end = beg + p.len();\n\n            // Merge the the second segment into first segment if possible\n            if (seg_beg <= beg && beg <= seg_end && seg_end < end) {\n                // Merge two segments, trim front of second segment\n                auto trim = seg_end - beg;\n                p.trim_front(trim);\n                // Append new data to the first segment, keep the first segment\n                seg_pkt.append(std::move(p));\n\n                // Delete the second segment\n                map.erase(it_next);\n\n                // Keep merging this first segment with its new next packet\n                // So we do not update the iterator: it\n                continue;\n            } else if (end <= seg_end) {\n                // The first segment has all the data in the second segment\n                // Delete the second segment\n                map.erase(it_next);\n                continue;\n            } else if (seg_end < beg) {\n                // Can not merge first segment with second segment\n                it = it_next;\n                continue;\n            } else {\n                // If we reach here, we have a bug with merge.\n                std::abort();\n                break;\n            }\n        }\n    }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/packet.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/deleter.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/net/const.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <span>\n#include <algorithm>\n#include <cstdint>\n#include <functional>\n#include <iosfwd>\n#include <memory>\n#include <optional>\n#include <vector>\n\nnamespace seastar {\n\nnamespace net {\n\n\nstruct fragment {\n    char* base;\n    size_t size;\n};\n\nstruct offload_info {\n    ip_protocol_num protocol = ip_protocol_num::unused;\n    bool needs_csum = false;\n    uint8_t ip_hdr_len = 20;\n    uint8_t tcp_hdr_len = 20;\n    uint8_t udp_hdr_len = 8;\n    bool needs_ip_csum = false;\n    bool reassembled = false;\n    uint16_t tso_seg_size = 0;\n    // HW stripped VLAN header (CPU order)\n    std::optional<uint16_t> vlan_tci;\n};\n\n// Zero-copy friendly packet class\n//\n// For implementing zero-copy, we need a flexible destructor that can\n// destroy packet data in different ways: decrementing a reference count,\n// or calling a free()-like function.\n//\n// Moreover, we need different destructors for each set of fragments within\n// a single fragment. For example, a header and trailer might need delete[]\n// to be called, while the internal data needs a reference count to be\n// released.  Matters are complicated in that fragments can be split\n// (due to virtual/physical translation).\n//\n// To implement this, we associate each packet with a single destructor,\n// but allow composing a packet from another packet plus a fragment to\n// be added, with its own destructor, causing the destructors to be chained.\n//\n// The downside is that the data needed for the destructor is duplicated,\n// if it is already available in the fragment itself.\n//\n// As an optimization, when we allocate small fragments, we allocate some\n// extra space, so prepending to the packet does not require extra\n// allocations.  This is useful when adding headers.\n//\nclass packet final {\n    // enough for lots of headers, not quite two cache lines:\n    static constexpr size_t internal_data_size = 128 - 16;\n    static constexpr size_t default_nr_frags = 4;\n\n    struct pseudo_vector {\n        fragment* _start;\n        fragment* _finish;\n        pseudo_vector(fragment* start, size_t nr) noexcept\n            : _start(start), _finish(_start + nr) {}\n        fragment* begin() noexcept { return _start; }\n        fragment* end() noexcept { return _finish; }\n        fragment& operator[](size_t idx) noexcept { return _start[idx]; }\n    };\n\n    struct impl {\n        // when destroyed, virtual destructor will reclaim resources\n        deleter _deleter;\n        unsigned _len = 0;\n        uint16_t _nr_frags = 0;\n        uint16_t _allocated_frags;\n        offload_info _offload_info;\n        std::optional<uint32_t> _rss_hash;\n        char _data[internal_data_size]; // only _frags[0] may use\n        unsigned _headroom = internal_data_size; // in _data\n        // FIXME: share _data/_frags space\n\n        fragment _frags[];\n\n        impl(size_t nr_frags = default_nr_frags) noexcept;\n        impl(const impl&) = delete;\n        impl(fragment frag, size_t nr_frags = default_nr_frags);\n\n        pseudo_vector fragments() noexcept { return { _frags, _nr_frags }; }\n\n        static std::unique_ptr<impl> allocate(size_t nr_frags) {\n            nr_frags = std::max(nr_frags, default_nr_frags);\n            return std::unique_ptr<impl>(new (nr_frags) impl(nr_frags));\n        }\n\n        static std::unique_ptr<impl> copy(impl* old, size_t nr) {\n            auto n = allocate(nr);\n            n->_deleter = std::move(old->_deleter);\n            n->_len = old->_len;\n            n->_nr_frags = old->_nr_frags;\n            n->_headroom = old->_headroom;\n            n->_offload_info = old->_offload_info;\n            n->_rss_hash = old->_rss_hash;\n            std::copy(old->_frags, old->_frags + old->_nr_frags, n->_frags);\n            old->copy_internal_fragment_to(n.get());\n            return n;\n        }\n\n        static std::unique_ptr<impl> copy(impl* old) {\n            return copy(old, old->_nr_frags);\n        }\n\n        static std::unique_ptr<impl> allocate_if_needed(std::unique_ptr<impl> old, size_t extra_frags) {\n            if (old->_allocated_frags >= old->_nr_frags + extra_frags) {\n                return old;\n            }\n            return copy(old.get(), std::max<size_t>(old->_nr_frags + extra_frags, 2 * old->_nr_frags));\n        }\n        void* operator new(size_t size, size_t nr_frags = default_nr_frags) {\n            SEASTAR_ASSERT(nr_frags == uint16_t(nr_frags));\n            return ::operator new(size + nr_frags * sizeof(fragment));\n        }\n        // Matching the operator new above\n        void operator delete(void* ptr, size_t) {\n            return ::operator delete(ptr);\n        }\n        // Since the above \"placement delete\" hides the global one, expose it\n        void operator delete(void* ptr) {\n            return ::operator delete(ptr);\n        }\n\n        bool using_internal_data() const noexcept {\n            return _nr_frags\n                    && _frags[0].base >= _data\n                    && _frags[0].base < _data + internal_data_size;\n        }\n\n        void unuse_internal_data() {\n            if (!using_internal_data()) {\n                return;\n            }\n            auto buf = static_cast<char*>(::malloc(_frags[0].size));\n            if (!buf) {\n                throw std::bad_alloc();\n            }\n            deleter d = make_free_deleter(buf);\n            std::copy(_frags[0].base, _frags[0].base + _frags[0].size, buf);\n            _frags[0].base = buf;\n            d.append(std::move(_deleter));\n            _deleter = std::move(d);\n            _headroom = internal_data_size;\n        }\n        void copy_internal_fragment_to(impl* to) noexcept {\n            if (!using_internal_data()) {\n                return;\n            }\n            to->_frags[0].base = to->_data + _headroom;\n            std::copy(_frags[0].base, _frags[0].base + _frags[0].size,\n                    to->_frags[0].base);\n        }\n    };\n    packet(std::unique_ptr<impl>&& impl) noexcept : _impl(std::move(impl)) {}\n    std::unique_ptr<impl> _impl;\npublic:\n    static packet from_static_data(const char* data, size_t len) noexcept {\n        return {fragment{const_cast<char*>(data), len}, deleter()};\n    }\n\n    // build empty packet\n    packet();\n    // build empty packet with nr_frags allocated\n    packet(size_t nr_frags);\n    // move existing packet\n    packet(packet&& x) noexcept;\n    // copy data into packet\n    packet(const char* data, size_t len);\n    // copy data into packet\n    packet(fragment frag);\n    // zero-copy single fragment\n    packet(fragment frag, deleter del);\n    // zero-copy multiple fragments\n    packet(std::vector<fragment> frag, deleter del);\n    // build packet with iterator\n    template <typename Iterator>\n    packet(Iterator begin, Iterator end, deleter del);\n    // append fragment (copying new fragment)\n    packet(packet&& x, fragment frag);\n    // prepend fragment (copying new fragment, with header optimization)\n    packet(fragment frag, packet&& x);\n    // prepend fragment (zero-copy)\n    packet(fragment frag, deleter del, packet&& x);\n    // append fragment (zero-copy)\n    packet(packet&& x, fragment frag, deleter d);\n    // append temporary_buffer (zero-copy)\n    packet(packet&& x, temporary_buffer<char> buf);\n    // create from temporary_buffer (zero-copy)\n    packet(temporary_buffer<char> buf);\n    // append deleter\n    packet(packet&& x, deleter d);\n    // create from span of buffers (zero-copy, ownership is transfered)\n    explicit packet(std::span<temporary_buffer<char>>);\n\n    packet& operator=(packet&& x) noexcept {\n        if (this != &x) {\n            this->~packet();\n            new (this) packet(std::move(x));\n        }\n        return *this;\n    }\n\n    unsigned len() const noexcept { return _impl->_len; }\n    unsigned memory() const noexcept { return len() +  sizeof(packet::impl); }\n\n    fragment frag(unsigned idx) const noexcept { return _impl->_frags[idx]; }\n    fragment& frag(unsigned idx) noexcept { return _impl->_frags[idx]; }\n\n    unsigned nr_frags() const noexcept { return _impl->_nr_frags; }\n    pseudo_vector fragments() const noexcept { return { _impl->_frags, _impl->_nr_frags }; }\n    fragment* fragment_array() const noexcept { return _impl->_frags; }\n\n    // share packet data (reference counted, non COW)\n    packet share();\n    packet share(size_t offset, size_t len);\n\n    void append(packet&& p);\n\n    void trim_front(size_t how_much) noexcept;\n    void trim_back(size_t how_much) noexcept;\n\n    // get a header pointer, linearizing if necessary\n    template <typename Header>\n    Header* get_header(size_t offset = 0);\n\n    // get a header pointer, linearizing if necessary\n    char* get_header(size_t offset, size_t size);\n\n    // prepend a header (default-initializing it)\n    template <typename Header>\n    Header* prepend_header(size_t extra_size = 0);\n\n    // prepend a header (uninitialized!)\n    char* prepend_uninitialized_header(size_t size);\n\n    packet free_on_cpu(unsigned cpu, std::function<void()> cb = []{});\n\n    void linearize() { return linearize(0, len()); }\n\n    void reset() noexcept { _impl.reset(); }\n\n    void reserve(int n_frags) {\n        if (n_frags > _impl->_nr_frags) {\n            auto extra = n_frags - _impl->_nr_frags;\n            _impl = impl::allocate_if_needed(std::move(_impl), extra);\n        }\n    }\n    std::optional<uint32_t> rss_hash() const noexcept {\n        return _impl->_rss_hash;\n    }\n    std::optional<uint32_t> set_rss_hash(uint32_t hash) noexcept {\n        return _impl->_rss_hash = hash;\n    }\n    // Call `func` for each fragment, avoiding data copies when possible\n    // `func` is called with a temporary_buffer<char> parameter\n    template <typename Func>\n    void release_into(Func&& func) {\n        unsigned idx = 0;\n        if (_impl->using_internal_data()) {\n            auto&& f = frag(idx++);\n            func(temporary_buffer<char>(f.base, f.size));\n        }\n        while (idx < nr_frags()) {\n            auto&& f = frag(idx++);\n            func(temporary_buffer<char>(f.base, f.size, _impl->_deleter.share()));\n        }\n    }\n    std::vector<temporary_buffer<char>> release() {\n        std::vector<temporary_buffer<char>> ret;\n        ret.reserve(_impl->_nr_frags);\n        release_into([&ret] (temporary_buffer<char>&& frag) {\n            ret.push_back(std::move(frag));\n        });\n        return ret;\n    }\n    explicit operator bool() const noexcept {\n        return bool(_impl);\n    }\n    static packet make_null_packet() noexcept {\n        return net::packet(nullptr);\n    }\nprivate:\n    void linearize(size_t at_frag, size_t desired_size);\n    bool allocate_headroom(size_t size);\npublic:\n    struct offload_info get_offload_info() const noexcept { return _impl->_offload_info; }\n    struct offload_info& offload_info_ref() noexcept { return _impl->_offload_info; }\n    void set_offload_info(struct offload_info oi) noexcept { _impl->_offload_info = oi; }\n};\n\nstd::ostream& operator<<(std::ostream& os, const packet& p);\n\n\ninline\npacket::packet(packet&& x) noexcept\n    : _impl(std::move(x._impl)) {\n}\n\ninline\npacket::impl::impl(size_t nr_frags) noexcept\n    : _len(0), _allocated_frags(nr_frags) {\n}\n\ninline\npacket::impl::impl(fragment frag, size_t nr_frags)\n    : _len(frag.size), _allocated_frags(nr_frags) {\n    SEASTAR_ASSERT(_allocated_frags > _nr_frags);\n    if (frag.size <= internal_data_size) {\n        _headroom -= frag.size;\n        _frags[0] = { _data + _headroom, frag.size };\n    } else {\n        auto buf = static_cast<char*>(::malloc(frag.size));\n        if (!buf) {\n            throw std::bad_alloc();\n        }\n        deleter d = make_free_deleter(buf);\n        _frags[0] = { buf, frag.size };\n        _deleter.append(std::move(d));\n    }\n    std::copy(frag.base, frag.base + frag.size, _frags[0].base);\n    ++_nr_frags;\n}\n\ninline\npacket::packet()\n    : _impl(impl::allocate(1)) {\n}\n\ninline\npacket::packet(size_t nr_frags)\n    : _impl(impl::allocate(nr_frags)) {\n}\n\ninline packet::packet(std::span<temporary_buffer<char>> bufs)\n    : _impl(impl::allocate(bufs.size()))\n{\n    for (auto& b : bufs) {\n        temporary_buffer<char> buf = std::move(b);\n        _impl->_len += buf.size();\n        _impl->_frags[_impl->_nr_frags++] = fragment{buf.get_write(), buf.size()};\n        deleter d = buf.release();\n        d.append(std::move(_impl->_deleter));\n        _impl->_deleter = std::move(d);\n    }\n}\n\ninline\npacket::packet(fragment frag) : _impl(new impl(frag)) {\n}\n\ninline\npacket::packet(const char* data, size_t size) : packet(fragment{const_cast<char*>(data), size}) {\n}\n\ninline\npacket::packet(fragment frag, deleter d)\n    : _impl(impl::allocate(1)) {\n    _impl->_deleter = std::move(d);\n    _impl->_frags[_impl->_nr_frags++] = frag;\n    _impl->_len = frag.size;\n}\n\ninline\npacket::packet(std::vector<fragment> frag, deleter d)\n    : _impl(impl::allocate(frag.size())) {\n    _impl->_deleter = std::move(d);\n    std::copy(frag.begin(), frag.end(), _impl->_frags);\n    _impl->_nr_frags = frag.size();\n    _impl->_len = 0;\n    for (auto&& f : _impl->fragments()) {\n        _impl->_len += f.size;\n    }\n}\n\ntemplate <typename Iterator>\ninline\npacket::packet(Iterator begin, Iterator end, deleter del) {\n    unsigned nr_frags = 0, len = 0;\n    nr_frags = std::distance(begin, end);\n    std::for_each(begin, end, [&] (const fragment& frag) { len += frag.size; });\n    _impl = impl::allocate(nr_frags);\n    _impl->_deleter = std::move(del);\n    _impl->_len = len;\n    _impl->_nr_frags = nr_frags;\n    std::copy(begin, end, _impl->_frags);\n}\n\ninline\npacket::packet(packet&& x, fragment frag)\n    : _impl(impl::allocate_if_needed(std::move(x._impl), 1)) {\n    _impl->_len += frag.size;\n    std::unique_ptr<char[]> buf(new char[frag.size]);\n    std::copy(frag.base, frag.base + frag.size, buf.get());\n    _impl->_frags[_impl->_nr_frags++] = {buf.get(), frag.size};\n    _impl->_deleter = make_deleter(std::move(_impl->_deleter), [buf = buf.release()] {\n        delete[] buf;\n    });\n}\n\ninline\nbool\npacket::allocate_headroom(size_t size) {\n    if (_impl->_headroom >= size) {\n        _impl->_len += size;\n        if (!_impl->using_internal_data()) {\n            _impl = impl::allocate_if_needed(std::move(_impl), 1);\n            std::copy_backward(_impl->_frags, _impl->_frags + _impl->_nr_frags,\n                    _impl->_frags + _impl->_nr_frags + 1);\n            _impl->_frags[0] = { _impl->_data + internal_data_size, 0 };\n            ++_impl->_nr_frags;\n        }\n        _impl->_headroom -= size;\n        _impl->_frags[0].base -= size;\n        _impl->_frags[0].size += size;\n        return true;\n    } else {\n        return false;\n    }\n}\n\n\ninline\npacket::packet(fragment frag, packet&& x)\n    : _impl(std::move(x._impl)) {\n    // try to prepend into existing internal fragment\n    if (allocate_headroom(frag.size)) {\n        std::copy(frag.base, frag.base + frag.size, _impl->_frags[0].base);\n        return;\n    } else {\n        // didn't work out, allocate and copy\n        _impl->unuse_internal_data();\n        _impl = impl::allocate_if_needed(std::move(_impl), 1);\n        _impl->_len += frag.size;\n        std::unique_ptr<char[]> buf(new char[frag.size]);\n        std::copy(frag.base, frag.base + frag.size, buf.get());\n        std::copy_backward(_impl->_frags, _impl->_frags + _impl->_nr_frags,\n                _impl->_frags + _impl->_nr_frags + 1);\n        ++_impl->_nr_frags;\n        _impl->_frags[0] = {buf.get(), frag.size};\n        _impl->_deleter = make_deleter(std::move(_impl->_deleter),\n                [buf = std::move(buf)] {});\n    }\n}\n\ninline\npacket::packet(packet&& x, fragment frag, deleter d)\n    : _impl(impl::allocate_if_needed(std::move(x._impl), 1)) {\n    _impl->_len += frag.size;\n    _impl->_frags[_impl->_nr_frags++] = frag;\n    d.append(std::move(_impl->_deleter));\n    _impl->_deleter = std::move(d);\n}\n\ninline\npacket::packet(packet&& x, deleter d)\n    : _impl(std::move(x._impl)) {\n    _impl->_deleter.append(std::move(d));\n}\n\ninline\npacket::packet(packet&& x, temporary_buffer<char> buf)\n    : packet(std::move(x), fragment{buf.get_write(), buf.size()}, buf.release()) {\n}\n\ninline\npacket::packet(temporary_buffer<char> buf)\n    : packet(fragment{buf.get_write(), buf.size()}, buf.release()) {}\n\ninline\nvoid packet::append(packet&& p) {\n    if (!_impl->_len) {\n        *this = std::move(p);\n        return;\n    }\n    _impl = impl::allocate_if_needed(std::move(_impl), p._impl->_nr_frags);\n    _impl->_len += p._impl->_len;\n    p._impl->unuse_internal_data();\n    std::copy(p._impl->_frags, p._impl->_frags + p._impl->_nr_frags,\n            _impl->_frags + _impl->_nr_frags);\n    _impl->_nr_frags += p._impl->_nr_frags;\n    p._impl->_deleter.append(std::move(_impl->_deleter));\n    _impl->_deleter = std::move(p._impl->_deleter);\n}\n\ninline\nchar* packet::get_header(size_t offset, size_t size) {\n    if (offset + size > _impl->_len) {\n        return nullptr;\n    }\n    size_t i = 0;\n    while (i != _impl->_nr_frags && offset >= _impl->_frags[i].size) {\n        offset -= _impl->_frags[i++].size;\n    }\n    if (i == _impl->_nr_frags) {\n        return nullptr;\n    }\n    if (offset + size > _impl->_frags[i].size) {\n        linearize(i, offset + size);\n    }\n    return _impl->_frags[i].base + offset;\n}\n\ntemplate <typename Header>\ninline\nHeader* packet::get_header(size_t offset) {\n    return reinterpret_cast<Header*>(get_header(offset, sizeof(Header)));\n}\n\ninline\nvoid packet::trim_front(size_t how_much) noexcept {\n    SEASTAR_ASSERT(how_much <= _impl->_len);\n    _impl->_len -= how_much;\n    size_t i = 0;\n    while (how_much && how_much >= _impl->_frags[i].size) {\n        how_much -= _impl->_frags[i++].size;\n    }\n    std::copy(_impl->_frags + i, _impl->_frags + _impl->_nr_frags, _impl->_frags);\n    _impl->_nr_frags -= i;\n    if (!_impl->using_internal_data()) {\n        _impl->_headroom = internal_data_size;\n    }\n    if (how_much) {\n        if (_impl->using_internal_data()) {\n            _impl->_headroom += how_much;\n        }\n        _impl->_frags[0].base += how_much;\n        _impl->_frags[0].size -= how_much;\n    }\n}\n\ninline\nvoid packet::trim_back(size_t how_much) noexcept {\n    SEASTAR_ASSERT(how_much <= _impl->_len);\n    _impl->_len -= how_much;\n    size_t i = _impl->_nr_frags - 1;\n    while (how_much && how_much >= _impl->_frags[i].size) {\n        how_much -= _impl->_frags[i--].size;\n    }\n    _impl->_nr_frags = i + 1;\n    if (how_much) {\n        _impl->_frags[i].size -= how_much;\n        if (i == 0 && _impl->using_internal_data()) {\n            _impl->_headroom += how_much;\n        }\n    }\n}\n\ntemplate <typename Header>\nHeader*\npacket::prepend_header(size_t extra_size) {\n    auto h = prepend_uninitialized_header(sizeof(Header) + extra_size);\n    return new (h) Header{};\n}\n\n// prepend a header (uninitialized!)\ninline\nchar* packet::prepend_uninitialized_header(size_t size) {\n    if (!allocate_headroom(size)) {\n        // didn't work out, allocate and copy\n        _impl->unuse_internal_data();\n        // try again, after unuse_internal_data we may have space after all\n        if (!allocate_headroom(size)) {\n            // failed\n            _impl->_len += size;\n            _impl = impl::allocate_if_needed(std::move(_impl), 1);\n            std::unique_ptr<char[]> buf(new char[size]);\n            std::copy_backward(_impl->_frags, _impl->_frags + _impl->_nr_frags,\n                    _impl->_frags + _impl->_nr_frags + 1);\n            ++_impl->_nr_frags;\n            _impl->_frags[0] = {buf.get(), size};\n            _impl->_deleter = make_deleter(std::move(_impl->_deleter),\n                    [buf = std::move(buf)] {});\n        }\n    }\n    return _impl->_frags[0].base;\n}\n\ninline\npacket packet::share() {\n    return share(0, _impl->_len);\n}\n\ninline\npacket packet::share(size_t offset, size_t len) {\n    _impl->unuse_internal_data(); // FIXME: eliminate?\n    packet n;\n    n._impl = impl::allocate_if_needed(std::move(n._impl), _impl->_nr_frags);\n    size_t idx = 0;\n    while (offset > 0 && offset >= _impl->_frags[idx].size) {\n        offset -= _impl->_frags[idx++].size;\n    }\n    while (n._impl->_len < len) {\n        auto& f = _impl->_frags[idx++];\n        auto fsize = std::min(len - n._impl->_len, f.size - offset);\n        n._impl->_frags[n._impl->_nr_frags++] = { f.base + offset, fsize };\n        n._impl->_len += fsize;\n        offset = 0;\n    }\n    n._impl->_offload_info = _impl->_offload_info;\n    SEASTAR_ASSERT(!n._impl->_deleter);\n    n._impl->_deleter = _impl->_deleter.share();\n    return n;\n}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/posix-stack.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n#include <unordered_set>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/internal/pollable_fd.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/core/polymorphic_temporary_buffer.hh>\n#include <seastar/core/internal/buffer_allocator.hh>\n#include <seastar/util/program-options.hh>\n\n#include <unordered_set>\n\nnamespace seastar {\n\nnamespace internal {\n\n// Holder of iovec-s for vectorized IO syscalls\n// Gets populated from temporary_buffers, providing deleter object\n// that controls the lifetime of the buffers it points to\nstruct wrapped_iovecs {\n    std::vector<iovec> v;\n    std::pair<size_t, deleter> populate(std::span<temporary_buffer<char>> bufs);\n\n    wrapped_iovecs() = default;\n    wrapped_iovecs(const wrapped_iovecs&) = delete;\n    wrapped_iovecs(wrapped_iovecs&&) = delete;\n};\n\n}\n\nnamespace net {\n\nusing namespace seastar;\n\n// We can't keep this in any of the socket servers as instance members, because a connection can\n// outlive the socket server. To avoid having the whole socket_server tracked as a shared pointer,\n// we will have a conntrack structure.\n//\n// Right now this class is used by the posix_server_socket_impl, but it could be used by any other.\nclass conntrack {\n    class load_balancer {\n        std::vector<unsigned> _cpu_load;\n    public:\n        load_balancer() : _cpu_load(size_t(smp::count), 0) {}\n        void closed_cpu(shard_id cpu) {\n            _cpu_load[cpu]--;\n        }\n        shard_id next_cpu() {\n            // FIXME: The naive algorithm will just round robin the connections around the shards.\n            // A more complex version can keep track of the amount of activity in each connection,\n            // and use that information.\n            auto min_el = std::min_element(_cpu_load.begin(), _cpu_load.end());\n            auto cpu = shard_id(std::distance(_cpu_load.begin(), min_el));\n            _cpu_load[cpu]++;\n            return cpu;\n        }\n        shard_id force_cpu(shard_id cpu) {\n            _cpu_load[cpu]++;\n            return cpu;\n        }\n    };\n\n    lw_shared_ptr<load_balancer> _lb;\n    void closed_cpu(shard_id cpu) {\n        _lb->closed_cpu(cpu);\n    }\npublic:\n    class handle {\n        shard_id _host_cpu;\n        shard_id _target_cpu;\n        foreign_ptr<lw_shared_ptr<load_balancer>> _lb;\n    public:\n        handle() : _lb(nullptr) {}\n        handle(shard_id cpu, lw_shared_ptr<load_balancer> lb)\n            : _host_cpu(this_shard_id())\n            , _target_cpu(cpu)\n            , _lb(make_foreign(std::move(lb))) {}\n\n        handle(const handle&) = delete;\n        handle(handle&&) = default;\n        handle& operator=(const handle&) = delete;\n        handle& operator=(handle&&) = default;\n        ~handle() {\n            if (!_lb) {\n                return;\n            }\n            // FIXME: future is discarded\n            (void)smp::submit_to(_host_cpu, [cpu = _target_cpu, lb = std::move(_lb)] {\n                lb->closed_cpu(cpu);\n            });\n        }\n        shard_id cpu() {\n            return _target_cpu;\n        }\n    };\n    friend class handle;\n\n    conntrack() : _lb(make_lw_shared<load_balancer>()) {}\n    handle get_handle() {\n        return handle(_lb->next_cpu(), _lb);\n    }\n    handle get_handle(shard_id cpu) {\n        return handle(_lb->force_cpu(cpu), _lb);\n    }\n};\n\nclass posix_data_source_impl final : public data_source_impl, private internal::buffer_allocator {\n    std::pmr::polymorphic_allocator<char>* _buffer_allocator;\n    pollable_fd _fd;\n    connected_socket_input_stream_config _config;\nprivate:\n    virtual temporary_buffer<char> allocate_buffer() override;\npublic:\n    explicit posix_data_source_impl(pollable_fd fd, connected_socket_input_stream_config config,\n            std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator)\n            : _buffer_allocator(allocator), _fd(std::move(fd)), _config(config) {\n    }\n    future<temporary_buffer<char>> get() override;\n    future<> close() override;\n};\n\nclass posix_data_sink_impl : public data_sink_impl {\n    pollable_fd _fd;\n#if SEASTAR_API_LEVEL >= 9\n    internal::wrapped_iovecs _vecs;\n#else\n    packet _p{net::packet::make_null_packet()};\n#endif\npublic:\n    explicit posix_data_sink_impl(pollable_fd fd) : _fd(std::move(fd)) {}\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>>) override;\n#else\n    using data_sink_impl::put;\n    future<> put(packet p) override;\n    future<> put(temporary_buffer<char> buf) override;\n#endif\n    future<> close() override;\n    bool can_batch_flushes() const noexcept override { return true; }\n    void on_batch_flush_error() noexcept override;\n};\n\nstruct proxy_data {\n    socket_address remote_address;\n    socket_address local_address;\n};\n\nclass posix_ap_server_socket_impl : public server_socket_impl {\n    using protocol_and_socket_address = std::tuple<int, socket_address>;\n    struct connection {\n        pollable_fd fd;\n        socket_address addr;\n        conntrack::handle connection_tracking_handle;\n        std::optional<proxy_data> proxy_protocol_header_opt;\n        connection(pollable_fd xfd, socket_address xaddr, conntrack::handle cth, std::optional<proxy_data> addr_data_opt) : fd(std::move(xfd)), addr(xaddr), connection_tracking_handle(std::move(cth)), proxy_protocol_header_opt(std::move(addr_data_opt)) {}\n    };\n    using port_map_t = std::unordered_set<protocol_and_socket_address>;\n    using sockets_map_t = std::unordered_map<protocol_and_socket_address, promise<accept_result>>;\n    using conn_map_t = std::unordered_multimap<protocol_and_socket_address, connection>;\n    static thread_local port_map_t ports;\n    static thread_local sockets_map_t sockets;\n    static thread_local conn_map_t conn_q;\n    int _protocol;\n    socket_address _sa;\n    std::pmr::polymorphic_allocator<char>* _allocator;\npublic:\n    explicit posix_ap_server_socket_impl(int protocol, socket_address sa, std::pmr::polymorphic_allocator<char>* allocator = memory::malloc_allocator);\n    ~posix_ap_server_socket_impl();\n    virtual future<accept_result> accept() override;\n    virtual void abort_accept() override;\n    socket_address local_address() const override {\n        return _sa;\n    }\n    static void move_connected_socket(int protocol, socket_address sa, pollable_fd fd, socket_address addr, conntrack::handle handle, std::optional<proxy_data> addr_data_opt, std::pmr::polymorphic_allocator<char>* allocator);\n\n    template <typename T>\n    friend class std::hash;\n};\n\nclass posix_server_socket_impl : public server_socket_impl {\n    socket_address _sa;\n    int _protocol;\n    pollable_fd _lfd;\n    conntrack _conntrack;\n    server_socket::load_balancing_algorithm _lba;\n    shard_id _fixed_cpu;\n    bool _proxy_protocol;\n    std::pmr::polymorphic_allocator<char>* _allocator;\npublic:\n    explicit posix_server_socket_impl(int protocol, socket_address sa, pollable_fd lfd,\n        server_socket::load_balancing_algorithm lba, shard_id fixed_cpu,\n        bool proxy_protocol,\n        std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) : _sa(sa), _protocol(protocol), _lfd(std::move(lfd)), _lba(lba), _fixed_cpu(fixed_cpu), _proxy_protocol(proxy_protocol), _allocator(allocator) {}\n    virtual future<accept_result> accept() override;\n    virtual void abort_accept() override;\n    virtual socket_address local_address() const override;\n};\n\nclass posix_reuseport_server_socket_impl : public server_socket_impl {\n    socket_address _sa;\n    int _protocol;\n    pollable_fd _lfd;\n    std::pmr::polymorphic_allocator<char>* _allocator;\npublic:\n    explicit posix_reuseport_server_socket_impl(int protocol, socket_address sa, pollable_fd lfd,\n        std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) : _sa(sa), _protocol(protocol), _lfd(std::move(lfd)), _allocator(allocator) {}\n    virtual future<accept_result> accept() override;\n    virtual void abort_accept() override;\n    virtual socket_address local_address() const override;\n};\n\nclass posix_network_stack : public network_stack {\nprotected:\n    const bool _reuseport;\n    const bool _sock_need_nonblock;\n    std::pmr::polymorphic_allocator<char>* _allocator;\npublic:\n    explicit posix_network_stack(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator);\n    virtual server_socket listen(socket_address sa, listen_options opts) override;\n    virtual ::seastar::socket socket() override;\n    virtual net::udp_channel make_udp_channel(const socket_address&) override;\n    virtual net::datagram_channel make_unbound_datagram_channel(sa_family_t) override;\n    virtual net::datagram_channel make_bound_datagram_channel(const socket_address& local) override;\n    static future<std::unique_ptr<network_stack>> create(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) {\n        return make_ready_future<std::unique_ptr<network_stack>>(std::unique_ptr<network_stack>(new posix_network_stack(opts, allocator)));\n    }\n    virtual bool has_per_core_namespace() override { return _reuseport; };\n    bool supports_ipv6() const override;\n    std::vector<network_interface> network_interfaces() override;\n    virtual statistics stats(unsigned scheduling_group_id) override;\n    virtual void clear_stats(unsigned scheduling_group_id) override;\n};\n\nclass posix_ap_network_stack : public posix_network_stack {\npublic:\n    posix_ap_network_stack(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator);\n    virtual server_socket listen(socket_address sa, listen_options opts) override;\n    static future<std::unique_ptr<network_stack>> create(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) {\n        return make_ready_future<std::unique_ptr<network_stack>>(std::unique_ptr<network_stack>(new posix_ap_network_stack(opts, allocator)));\n    }\n};\n\nnetwork_stack_entry register_posix_stack();\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/proxy.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n#pragma once\n\n#include <memory>\n#include <seastar/net/net.hh>\n#include <seastar/net/packet.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nstd::unique_ptr<qp> create_proxy_net_device(unsigned master_cpu, device* dev);\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/socket_defs.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n#pragma once\n\n#include <sys/socket.h>\n#include <sys/un.h>\n#include <netinet/ip.h>\n#include <array>\n#include <cassert>\n#include <functional>\n#include <iosfwd>\n#include <fmt/ostream.h>\n#include <seastar/net/byteorder.hh>\n#include <seastar/net/unix_address.hh>\n\nnamespace seastar {\n\n\nnamespace net {\nclass inet_address;\n}\n\nstruct ipv4_addr;\nstruct ipv6_addr;\n\nclass socket_address {\npublic:\n    socklen_t addr_length; ///!< actual size of the relevant 'u' member\n    union {\n        ::sockaddr_storage sas;\n        ::sockaddr sa;\n        ::sockaddr_in in;\n        ::sockaddr_in6 in6;\n        ::sockaddr_un un;\n    } u;\n    socket_address(const sockaddr_in& sa) noexcept : addr_length{sizeof(::sockaddr_in)} {\n        u.in = sa;\n    }\n    socket_address(const sockaddr_in6& sa) noexcept : addr_length{sizeof(::sockaddr_in6)} {\n        u.in6 = sa;\n    }\n    socket_address(uint16_t) noexcept;\n    socket_address(ipv4_addr) noexcept;\n    socket_address(const ipv6_addr&) noexcept;\n    socket_address(const ipv6_addr&, uint32_t scope) noexcept;\n    socket_address(const net::inet_address&, uint16_t p = 0) noexcept;\n    explicit socket_address(const unix_domain_addr&) noexcept;\n    /** creates an uninitialized socket_address. this can be written into, or used as\n     *  \"unspecified\" for such addresses as bind(addr) or local address in socket::connect\n     *  (i.e. system picks)\n     */\n    socket_address() noexcept;\n\n    ::sockaddr& as_posix_sockaddr() noexcept { return u.sa; }\n    ::sockaddr_in& as_posix_sockaddr_in() noexcept { return u.in; }\n    ::sockaddr_in6& as_posix_sockaddr_in6() noexcept { return u.in6; }\n    const ::sockaddr& as_posix_sockaddr() const noexcept { return u.sa; }\n    const ::sockaddr_in& as_posix_sockaddr_in() const noexcept { return u.in; }\n    const ::sockaddr_in6& as_posix_sockaddr_in6() const noexcept { return u.in6; }\n\n    socket_address(uint32_t, uint16_t p = 0) noexcept;\n\n    socklen_t length() const noexcept { return addr_length; };\n\n    bool is_af_unix() const noexcept {\n        return u.sa.sa_family == AF_UNIX;\n    }\n\n    bool is_unspecified() const noexcept;\n\n    sa_family_t family() const noexcept {\n        return u.sa.sa_family;\n    }\n\n    net::inet_address addr() const noexcept;\n    ::in_port_t port() const noexcept;\n    bool is_wildcard() const noexcept;\n\n    bool operator==(const socket_address&) const noexcept;\n    bool operator!=(const socket_address& a) const noexcept {\n        return !(*this == a);\n    }\n};\n\nstd::ostream& operator<<(std::ostream&, const socket_address&);\n\nenum class transport {\n    TCP = IPPROTO_TCP,\n    SCTP = IPPROTO_SCTP\n};\n\nstruct ipv4_addr {\n    uint32_t ip;\n    uint16_t port;\n\n    ipv4_addr() noexcept : ip(0), port(0) {}\n    ipv4_addr(uint32_t ip, uint16_t port) noexcept : ip(ip), port(port) {}\n    ipv4_addr(uint16_t port) noexcept : ip(0), port(port) {}\n    // throws if not a valid ipv4 addr\n    ipv4_addr(const std::string &addr);\n    ipv4_addr(const std::string &addr, uint16_t port);\n    // throws if not an ipv4 addr\n    ipv4_addr(const net::inet_address&, uint16_t);\n    ipv4_addr(const socket_address &) noexcept;\n    ipv4_addr(const ::in_addr&, uint16_t = 0) noexcept;\n\n    bool is_ip_unspecified() const noexcept {\n        return ip == 0;\n    }\n    bool is_port_unspecified() const noexcept {\n        return port == 0;\n    }\n};\n\nstruct ipv6_addr {\n    using ipv6_bytes = std::array<uint8_t, 16>;\n\n    ipv6_bytes ip;\n    uint16_t port;\n\n    ipv6_addr(const ipv6_bytes&, uint16_t port = 0) noexcept;\n    ipv6_addr(uint16_t port = 0) noexcept;\n    // throws if not a valid ipv6 addr\n    ipv6_addr(const std::string&);\n    ipv6_addr(const std::string&, uint16_t port);\n    ipv6_addr(const net::inet_address&, uint16_t = 0) noexcept;\n    ipv6_addr(const ::in6_addr&, uint16_t = 0) noexcept;\n    ipv6_addr(const ::sockaddr_in6&) noexcept;\n    ipv6_addr(const socket_address&) noexcept;\n\n    bool is_ip_unspecified() const noexcept;\n    bool is_port_unspecified() const noexcept {\n        return port == 0;\n    }\n};\n\nstd::ostream& operator<<(std::ostream&, const ipv4_addr&);\nstd::ostream& operator<<(std::ostream&, const ipv6_addr&);\n\ninline bool operator==(const ipv4_addr &lhs, const ipv4_addr& rhs) noexcept {\n    return lhs.ip == rhs.ip && lhs.port == rhs.port;\n}\n}\n\nnamespace std {\ntemplate<>\nstruct hash<seastar::socket_address> {\n    size_t operator()(const seastar::socket_address&) const;\n};\ntemplate<>\nstruct hash<seastar::ipv4_addr> {\n    size_t operator()(const seastar::ipv4_addr&) const;\n};\ntemplate<>\nstruct hash<seastar::unix_domain_addr> {\n    size_t operator()(const seastar::unix_domain_addr&) const;\n};\ntemplate<>\nstruct hash<::sockaddr_un> {\n    size_t operator()(const ::sockaddr_un&) const;\n};\n\ntemplate <>\nstruct hash<seastar::transport> {\n    size_t operator()(seastar::transport tr) const {\n        return static_cast<size_t>(tr);\n    }\n};\n\n}\n\ntemplate <> struct fmt::formatter<seastar::socket_address> : fmt::ostream_formatter {};\ntemplate <> struct fmt::formatter<seastar::ipv4_addr> : fmt::ostream_formatter {};\ntemplate <> struct fmt::formatter<seastar::ipv6_addr> : fmt::ostream_formatter {};\n"
  },
  {
    "path": "include/seastar/net/stack.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#pragma once\n\n#include <chrono>\n#include <seastar/net/api.hh>\n#include <seastar/core/internal/api-level.hh>\n#include <seastar/core/memory.hh>\n\nnamespace seastar {\n\nnamespace net {\n\n/// \\cond internal\nclass connected_socket_impl {\npublic:\n    virtual ~connected_socket_impl() {}\n    virtual data_source source() = 0;\n    virtual data_source source(connected_socket_input_stream_config csisc);\n    virtual data_sink sink() = 0;\n    virtual void shutdown_input() = 0;\n    virtual void shutdown_output() = 0;\n    virtual void set_nodelay(bool nodelay) = 0;\n    virtual bool get_nodelay() const = 0;\n    virtual void set_keepalive(bool keepalive) = 0;\n    virtual bool get_keepalive() const = 0;\n    virtual void set_keepalive_parameters(const keepalive_params&) = 0;\n    virtual keepalive_params get_keepalive_parameters() const = 0;\n    virtual void set_sockopt(int level, int optname, const void* data, size_t len) = 0;\n    virtual int get_sockopt(int level, int optname, void* data, size_t len) const = 0;\n    virtual socket_address local_address() const noexcept = 0;\n    virtual socket_address remote_address() const noexcept = 0;\n    virtual future<> wait_input_shutdown() = 0;\n};\n\nclass socket_impl {\npublic:\n    socket_impl() = default;\n    socket_impl(const socket_impl&) = delete;\n    socket_impl(socket_impl&&) = default;\n    virtual ~socket_impl() {}\n    virtual future<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) = 0;\n    virtual void set_reuseaddr(bool reuseaddr) = 0;\n    virtual bool get_reuseaddr() const = 0;\n    virtual void shutdown() = 0;\n};\n\n\nclass server_socket_impl {\npublic:\n    virtual ~server_socket_impl() {}\n    virtual future<accept_result> accept() = 0;\n    virtual void abort_accept() = 0;\n    virtual socket_address local_address() const = 0;\n};\n\nclass datagram_channel_impl {\npublic:\n    virtual ~datagram_channel_impl() {}\n    virtual socket_address local_address() const = 0;\n    virtual future<datagram> receive() = 0;\n    virtual future<> send(const socket_address& dst, const char* msg) = 0;\n    // The ownership of temporary_buffer-s referenced by span must be transferred\n    // synchronously before returning the future\n    virtual future<> send(const socket_address& dst, std::span<temporary_buffer<char>> bufs) = 0;\n    virtual void shutdown_input() = 0;\n    virtual void shutdown_output() = 0;\n    virtual bool is_closed() const = 0;\n    virtual void close() = 0;\n};\n\nusing udp_channel_impl = datagram_channel_impl;\n\nclass network_interface_impl {\nprotected:\n    network_interface_impl() = default;\n    network_interface_impl(const network_interface_impl&) = default;\n    virtual ~network_interface_impl() {}\npublic:\n    virtual uint32_t index() const = 0;\n    virtual uint32_t mtu() const = 0;\n\n    virtual const sstring& name() const = 0;\n    virtual const sstring& display_name() const = 0;\n    virtual const std::vector<net::inet_address>& addresses() const = 0;\n    virtual const std::vector<uint8_t> hardware_address() const = 0;\n\n    virtual bool is_loopback() const = 0;\n    virtual bool is_virtual() const = 0;\n    virtual bool is_up() const = 0;\n    virtual bool supports_ipv6() const = 0;\n};\n\n/// \\endcond\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/tcp-stack.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n// tcp/network-stack integration\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/internal/api-level.hh>\n\nnamespace seastar {\n\nstruct listen_options;\n\nclass server_socket;\nclass socket;\nclass connected_socket;\n\nnamespace net {\n\nstruct ipv4_traits;\ntemplate <typename InetTraits>\nclass tcp;\n\nserver_socket\ntcpv4_listen(tcp<ipv4_traits>& tcpv4, uint16_t port, listen_options opts);\n\nseastar::socket\ntcpv4_socket(tcp<ipv4_traits>& tcpv4);\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/tcp.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <unordered_map>\n#include <map>\n#include <functional>\n#include <deque>\n#include <chrono>\n#include <random>\n#include <span>\n#include <stdexcept>\n#include <system_error>\n#include <gnutls/crypto.h>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/net/net.hh>\n#include <seastar/net/ip_checksum.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/const.hh>\n#include <seastar/net/packet-util.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\nusing namespace std::chrono_literals;\n\nnamespace net {\n\nstruct tcp_hdr;\n\ninline auto tcp_error(int err) {\n    return std::system_error(err, std::system_category());\n}\n\ninline auto tcp_reset_error() {\n    return tcp_error(ECONNRESET);\n};\n\ninline auto tcp_connect_error() {\n    return tcp_error(ECONNABORTED);\n}\n\ninline auto tcp_refused_error() {\n    return tcp_error(ECONNREFUSED);\n};\n\nenum class tcp_state : uint16_t {\n    CLOSED          = (1 << 0),\n    LISTEN          = (1 << 1),\n    SYN_SENT        = (1 << 2),\n    SYN_RECEIVED    = (1 << 3),\n    ESTABLISHED     = (1 << 4),\n    FIN_WAIT_1      = (1 << 5),\n    FIN_WAIT_2      = (1 << 6),\n    CLOSE_WAIT      = (1 << 7),\n    CLOSING         = (1 << 8),\n    LAST_ACK        = (1 << 9),\n    TIME_WAIT       = (1 << 10)\n};\n\ninline tcp_state operator|(tcp_state s1, tcp_state s2) {\n    return tcp_state(uint16_t(s1) | uint16_t(s2));\n}\n\ntemplate <typename... Args>\nvoid tcp_debug(const char* fmt, Args&&... args) {\n#if TCP_DEBUG\n    print(fmt, std::forward<Args>(args)...);\n#endif\n}\n\nstruct tcp_option {\n    // The kind and len field are fixed and defined in TCP protocol\n    enum class option_kind: uint8_t { mss = 2, win_scale = 3, sack = 4, timestamps = 8,  nop = 1, eol = 0 };\n    enum class option_len:  uint8_t { mss = 4, win_scale = 3, sack = 2, timestamps = 10, nop = 1, eol = 1 };\n    static void write(char* p, option_kind kind, option_len len) {\n        p[0] = static_cast<uint8_t>(kind);\n        if (static_cast<uint8_t>(len) > 1) {\n            p[1] = static_cast<uint8_t>(len);\n        }\n    }\n    struct mss {\n        static constexpr option_kind kind = option_kind::mss;\n        static constexpr option_len len = option_len::mss;\n        uint16_t mss;\n        static tcp_option::mss read(const char* p) {\n            tcp_option::mss x;\n            x.mss = read_be<uint16_t>(p + 2);\n            return x;\n        }\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n            write_be<uint16_t>(p + 2, mss);\n        }\n    };\n    struct win_scale {\n        static constexpr option_kind kind = option_kind::win_scale;\n        static constexpr option_len len = option_len::win_scale;\n        uint8_t shift;\n        static tcp_option::win_scale read(const char* p) {\n            tcp_option::win_scale x;\n            x.shift = p[2];\n            return x;\n        }\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n            p[2] = shift;\n        }\n    };\n    struct sack {\n        static constexpr option_kind kind = option_kind::sack;\n        static constexpr option_len len = option_len::sack;\n        static tcp_option::sack read(const char* p) {\n            return {};\n        }\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n        }\n    };\n    struct timestamps {\n        static constexpr option_kind kind = option_kind::timestamps;\n        static constexpr option_len len = option_len::timestamps;\n        uint32_t t1;\n        uint32_t t2;\n        static tcp_option::timestamps read(const char* p) {\n            tcp_option::timestamps ts;\n            ts.t1 = read_be<uint32_t>(p + 2);\n            ts.t2 = read_be<uint32_t>(p + 6);\n            return ts;\n        }\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n            write_be<uint32_t>(p + 2, t1);\n            write_be<uint32_t>(p + 6, t2);\n        }\n    };\n    struct nop {\n        static constexpr option_kind kind = option_kind::nop;\n        static constexpr option_len len = option_len::nop;\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n        }\n    };\n    struct eol {\n        static constexpr option_kind kind = option_kind::eol;\n        static constexpr option_len len = option_len::eol;\n        void write(char* p) const {\n            tcp_option::write(p, kind, len);\n        }\n    };\n    static const uint8_t align = 4;\n\n    void parse(uint8_t* beg, uint8_t* end);\n    uint8_t fill(void* h, const tcp_hdr* th, uint8_t option_size);\n    uint8_t get_size(bool syn_on, bool ack_on);\n\n    // For option negotiattion\n    bool _mss_received = false;\n    bool _win_scale_received = false;\n    bool _timestamps_received = false;\n    bool _sack_received = false;\n\n    // Option data\n    uint16_t _remote_mss = 536;\n    uint16_t _local_mss;\n    uint8_t _remote_win_scale = 0;\n    uint8_t _local_win_scale = 0;\n};\ninline char*& operator+=(char*& x, tcp_option::option_len len) { x += uint8_t(len); return x; }\ninline const char*& operator+=(const char*& x, tcp_option::option_len len) { x += uint8_t(len); return x; }\ninline uint8_t& operator+=(uint8_t& x, tcp_option::option_len len) { x += uint8_t(len); return x; }\n\nstruct tcp_seq {\n    uint32_t raw;\n};\n\ninline tcp_seq ntoh(tcp_seq s) {\n    return tcp_seq { ntoh(s.raw) };\n}\n\ninline tcp_seq hton(tcp_seq s) {\n    return tcp_seq { hton(s.raw) };\n}\n\ninline\nstd::ostream& operator<<(std::ostream& os, tcp_seq s) {\n    return os << s.raw;\n}\n\ninline tcp_seq make_seq(uint32_t raw) { return tcp_seq{raw}; }\ninline tcp_seq& operator+=(tcp_seq& s, int32_t n) { s.raw += n; return s; }\ninline tcp_seq& operator-=(tcp_seq& s, int32_t n) { s.raw -= n; return s; }\ninline tcp_seq operator+(tcp_seq s, int32_t n) { return s += n; }\ninline tcp_seq operator-(tcp_seq s, int32_t n) { return s -= n; }\ninline int32_t operator-(tcp_seq s, tcp_seq q) { return s.raw - q.raw; }\ninline bool operator==(tcp_seq s, tcp_seq q)  { return s.raw == q.raw; }\ninline bool operator!=(tcp_seq s, tcp_seq q) { return !(s == q); }\ninline bool operator<(tcp_seq s, tcp_seq q) { return s - q < 0; }\ninline bool operator>(tcp_seq s, tcp_seq q) { return q < s; }\ninline bool operator<=(tcp_seq s, tcp_seq q) { return !(s > q); }\ninline bool operator>=(tcp_seq s, tcp_seq q) { return !(s < q); }\n\nstruct tcp_hdr {\n    static constexpr size_t len = 20;\n    uint16_t src_port;\n    uint16_t dst_port;\n    tcp_seq seq;\n    tcp_seq ack;\n    uint8_t rsvd1 : 4;\n    uint8_t data_offset : 4;\n    uint8_t f_fin : 1;\n    uint8_t f_syn : 1;\n    uint8_t f_rst : 1;\n    uint8_t f_psh : 1;\n    uint8_t f_ack : 1;\n    uint8_t f_urg : 1;\n    uint8_t rsvd2 : 2;\n    uint16_t window;\n    uint16_t checksum;\n    uint16_t urgent;\n    static tcp_hdr read(const char* p) {\n        tcp_hdr h;\n        h.src_port = read_be<uint16_t>(p + 0);\n        h.dst_port = read_be<uint16_t>(p + 2);\n        h.seq = tcp_seq{read_be<uint32_t>(p + 4)};\n        h.ack = tcp_seq{read_be<uint32_t>(p + 8)};\n        h.rsvd1 = p[12] & 15;\n        h.data_offset = uint8_t(p[12]) >> 4;\n        h.f_fin = (uint8_t(p[13]) >> 0) & 1;\n        h.f_syn = (uint8_t(p[13]) >> 1) & 1;\n        h.f_rst = (uint8_t(p[13]) >> 2) & 1;\n        h.f_psh = (uint8_t(p[13]) >> 3) & 1;\n        h.f_ack = (uint8_t(p[13]) >> 4) & 1;\n        h.f_urg = (uint8_t(p[13]) >> 5) & 1;\n        h.rsvd2 = (uint8_t(p[13]) >> 6) & 3;\n        h.window = read_be<uint16_t>(p + 14);\n        h.checksum = read_be<uint16_t>(p + 16);\n        h.urgent = read_be<uint16_t>(p + 18);\n        return h;\n    }\n    void write(char* p) const {\n        write_be<uint16_t>(p + 0, src_port);\n        write_be<uint16_t>(p + 2, dst_port);\n        write_be<uint32_t>(p + 4, seq.raw);\n        write_be<uint32_t>(p + 8, ack.raw);\n        p[12] = rsvd1 | (data_offset << 4);\n        p[13] = (f_fin << 0)\n                | (f_syn << 1)\n                | (f_rst << 2)\n                | (f_psh << 3)\n                | (f_ack << 4)\n                | (f_urg << 5)\n                | (rsvd2 << 6);\n        write_be<uint16_t>(p + 14, window);\n        write_be<uint16_t>(p + 16, checksum);\n        write_be<uint16_t>(p + 18, urgent);\n    }\n    static void write_nbo_checksum(char* p, uint16_t checksum_in_network_byte_order) {\n        std::copy_n(reinterpret_cast<const char*>(&checksum_in_network_byte_order), 2, p + 16);\n    }\n};\n\nstruct tcp_tag {};\nusing tcp_packet_merger = packet_merger<tcp_seq, tcp_tag>;\n\ntemplate <typename InetTraits>\nclass tcp {\npublic:\n    using ipaddr = typename InetTraits::address_type;\n    using inet_type = typename InetTraits::inet_type;\n    using connid = l4connid<InetTraits>;\n    using connid_hash = typename connid::connid_hash;\n    class connection;\n    class listener;\nprivate:\n    class tcb;\n\n    class tcb : public enable_lw_shared_from_this<tcb> {\n        using clock_type = lowres_clock;\n        static constexpr tcp_state CLOSED         = tcp_state::CLOSED;\n        static constexpr tcp_state LISTEN         = tcp_state::LISTEN;\n        static constexpr tcp_state SYN_SENT       = tcp_state::SYN_SENT;\n        static constexpr tcp_state SYN_RECEIVED   = tcp_state::SYN_RECEIVED;\n        static constexpr tcp_state ESTABLISHED    = tcp_state::ESTABLISHED;\n        static constexpr tcp_state FIN_WAIT_1     = tcp_state::FIN_WAIT_1;\n        static constexpr tcp_state FIN_WAIT_2     = tcp_state::FIN_WAIT_2;\n        static constexpr tcp_state CLOSE_WAIT     = tcp_state::CLOSE_WAIT;\n        static constexpr tcp_state CLOSING        = tcp_state::CLOSING;\n        static constexpr tcp_state LAST_ACK       = tcp_state::LAST_ACK;\n        static constexpr tcp_state TIME_WAIT      = tcp_state::TIME_WAIT;\n        tcp_state _state = CLOSED;\n        tcp& _tcp;\n        connection* _conn = nullptr;\n        promise<> _connect_done;\n        std::optional<promise<>> _fin_recvd_promise = promise<>();\n        ipaddr _local_ip;\n        ipaddr _foreign_ip;\n        uint16_t _local_port;\n        uint16_t _foreign_port;\n        struct unacked_segment {\n            packet p;\n            uint16_t data_len;\n            unsigned nr_transmits;\n            clock_type::time_point tx_time;\n        };\n        struct send {\n            tcp_seq unacknowledged;\n            tcp_seq next;\n            uint32_t window;\n            uint8_t window_scale;\n            uint16_t mss;\n            tcp_seq urgent;\n            tcp_seq wl1;\n            tcp_seq wl2;\n            tcp_seq initial;\n            std::deque<unacked_segment> data;\n            std::deque<temporary_buffer<char>> unsent;\n            uint32_t unsent_len = 0;\n            bool closed = false;\n            promise<> _window_opened;\n            // Wait for all data are acked\n            std::optional<promise<>> _all_data_acked_promise;\n            // Limit number of data queued into send queue\n            size_t max_queue_space = 212992;\n            size_t current_queue_space = 0;\n            // wait for there is at least one byte available in the queue\n            std::optional<promise<>> _send_available_promise;\n            // Round-trip time variation\n            std::chrono::milliseconds rttvar;\n            // Smoothed round-trip time\n            std::chrono::milliseconds srtt;\n            bool first_rto_sample = true;\n            clock_type::time_point syn_tx_time;\n            // Congestion window\n            uint32_t cwnd;\n            // Slow start threshold\n            uint32_t ssthresh;\n            // Duplicated ACKs\n            uint16_t dupacks = 0;\n            unsigned syn_retransmit = 0;\n            unsigned fin_retransmit = 0;\n            uint32_t limited_transfer = 0;\n            uint32_t partial_ack = 0;\n            tcp_seq recover;\n            bool window_probe = false;\n            uint8_t zero_window_probing_out = 0;\n        } _snd;\n        struct receive {\n            tcp_seq next;\n            uint32_t window;\n            uint8_t window_scale;\n            uint16_t mss;\n            tcp_seq urgent;\n            tcp_seq initial;\n            std::deque<packet> data;\n            // The total size of data stored in std::deque<packet> data\n            size_t data_size = 0;\n            tcp_packet_merger out_of_order;\n            std::optional<promise<>> _data_received_promise;\n            // The maximun memory buffer size allowed for receiving\n            // Currently, it is the same as default receive window size when window scaling is enabled\n            size_t max_receive_buf_size = 3737600;\n        } _rcv;\n        tcp_option _option;\n        timer<lowres_clock> _delayed_ack;\n        // Retransmission timeout\n        std::chrono::milliseconds _rto{1000};\n        std::chrono::milliseconds _persist_time_out{1000};\n        static constexpr std::chrono::milliseconds _rto_min{1000};\n        static constexpr std::chrono::milliseconds _rto_max{60000};\n        // Clock granularity\n        static constexpr std::chrono::milliseconds _rto_clk_granularity{1};\n        static constexpr uint16_t _max_nr_retransmit{5};\n        timer<lowres_clock> _retransmit;\n        timer<lowres_clock> _persist;\n        uint16_t _nr_full_seg_received = 0;\n        struct isn_secret {\n            // 512 bits secretkey for ISN generating\n            uint32_t key[16];\n            isn_secret () {\n                std::random_device rd;\n                std::default_random_engine e(rd());\n                std::uniform_int_distribution<uint32_t> dist{};\n                for (auto& k : key) {\n                    k = dist(e);\n                }\n            }\n        };\n        static isn_secret _isn_secret;\n        tcp_seq get_isn();\n        circular_buffer<typename InetTraits::l4packet> _packetq;\n        bool _poll_active = false;\n        uint32_t get_default_receive_window_size() {\n            // Linux's default window size\n            constexpr uint32_t size = 29200;\n            return size << _rcv.window_scale;\n        }\n        // Returns the current receive window according to available receiving buffer size\n        uint32_t get_modified_receive_window_size() {\n            uint32_t left =  _rcv.data_size > _rcv.max_receive_buf_size ? 0 : _rcv.max_receive_buf_size - _rcv.data_size;\n            return std::min(left, get_default_receive_window_size());\n        }\n    public:\n        tcb(tcp& t, connid id);\n        void input_handle_listen_state(tcp_hdr* th, packet p);\n        void input_handle_syn_sent_state(tcp_hdr* th, packet p);\n        void input_handle_other_state(tcp_hdr* th, packet p);\n        void output_one(bool data_retransmit = false);\n        future<> wait_for_data();\n        future<> wait_input_shutdown();\n        void abort_reader() noexcept;\n        future<> wait_for_all_data_acked();\n        future<> wait_send_available();\n        future<> send(std::span<temporary_buffer<char>> data);\n        void connect();\n        packet read();\n        void close() noexcept;\n        void remove_from_tcbs() {\n            auto id = connid{_local_ip, _foreign_ip, _local_port, _foreign_port};\n            _tcp._tcbs.erase(id);\n        }\n        std::optional<typename InetTraits::l4packet> get_packet();\n        void output() {\n            if (!_poll_active) {\n                _poll_active = true;\n                // FIXME: future is discarded\n                (void)_tcp.poll_tcb(_foreign_ip, this->shared_from_this()).then_wrapped([this] (auto&& f) {\n                    try {\n                        f.get();\n                    } catch(arp_queue_full_error& ex) {\n                        // retry later\n                        _poll_active = false;\n                        this->start_retransmit_timer();\n                    } catch(arp_timeout_error& ex) {\n                        if (this->in_state(SYN_SENT)) {\n                            _connect_done.set_exception(ex);\n                            this->cleanup();\n                        }\n                        // in other states connection should time out\n                    }\n                });\n            }\n        }\n        future<> connect_done() {\n            return _connect_done.get_future();\n        }\n        tcp_state& state() {\n            return _state;\n        }\n    private:\n        void respond_with_reset(tcp_hdr* th);\n        bool merge_out_of_order();\n        void insert_out_of_order(tcp_seq seq, packet p);\n        void trim_receive_data_after_window();\n        bool should_send_ack(uint16_t seg_len);\n        void clear_delayed_ack() noexcept;\n        packet get_transmit_packet();\n        void retransmit_one() {\n            bool data_retransmit = true;\n            output_one(data_retransmit);\n        }\n        void start_retransmit_timer() {\n            auto now = clock_type::now();\n            start_retransmit_timer(now);\n        };\n        void start_retransmit_timer(clock_type::time_point now) {\n            auto tp = now + _rto;\n            _retransmit.rearm(tp);\n        };\n        void stop_retransmit_timer() noexcept {\n            _retransmit.cancel();\n        };\n        void start_persist_timer() {\n            auto now = clock_type::now();\n            start_persist_timer(now);\n        };\n        void start_persist_timer(clock_type::time_point now) {\n            auto tp = now + _persist_time_out;\n            _persist.rearm(tp);\n        };\n        void stop_persist_timer() {\n            _persist.cancel();\n        };\n        void persist();\n        void retransmit();\n        void fast_retransmit();\n        void update_rto(clock_type::time_point tx_time);\n        void update_cwnd(uint32_t acked_bytes);\n        void cleanup();\n        uint32_t can_send() {\n            if (_snd.window_probe) {\n                return 1;\n            }\n\n            // Can not send if send window is zero\n            if (_snd.window == 0) {\n                return 0;\n            }\n\n            // Can not send if send window is less than unacknowledged data size\n            auto window_used = uint32_t(_snd.next - _snd.unacknowledged);\n            if (window_used > _snd.window) {\n                return 0;\n            }\n\n            // Can not send more than advertised window allows or unsent data size\n            auto x = std::min(_snd.window - window_used, _snd.unsent_len);\n\n            // Can not send more than congestion window allows\n            x = std::min(_snd.cwnd, x);\n            if (_snd.dupacks == 1 || _snd.dupacks == 2) {\n                // RFC5681 Step 3.1\n                // Send cwnd + 2 * smss per RFC3042\n                auto flight = flight_size();\n                auto max = _snd.cwnd + 2 * _snd.mss;\n                x = flight <= max ? std::min(x, max - flight) : 0;\n                _snd.limited_transfer += x;\n            } else if (_snd.dupacks >= 3) {\n                // RFC5681 Step 3.5\n                // Sent 1 full-sized segment at most\n                x = std::min(uint32_t(_snd.mss), x);\n            }\n            return x;\n        }\n        uint32_t flight_size() {\n            uint32_t size = 0;\n            std::for_each(_snd.data.begin(), _snd.data.end(), [&] (unacked_segment& seg) { size += seg.p.len(); });\n            return size;\n        }\n        uint16_t local_mss() {\n            return _tcp.hw_features().mtu - net::tcp_hdr_len_min - InetTraits::ip_hdr_len_min;\n        }\n        void queue_packet(packet p) {\n            _packetq.emplace_back(typename InetTraits::l4packet{_foreign_ip, std::move(p)});\n        }\n        void signal_data_received() {\n            if (_rcv._data_received_promise) {\n                _rcv._data_received_promise->set_value();\n                _rcv._data_received_promise = {};\n            }\n        }\n        void signal_all_data_acked() {\n            if (_snd._all_data_acked_promise && _snd.unsent_len == 0) {\n                _snd._all_data_acked_promise->set_value();\n                _snd._all_data_acked_promise = {};\n            }\n        }\n        void signal_send_available() {\n            if (_snd._send_available_promise && _snd.max_queue_space > _snd.current_queue_space) {\n                _snd._send_available_promise->set_value();\n                _snd._send_available_promise = {};\n            }\n        }\n        void do_syn_sent() {\n            _state = SYN_SENT;\n            _snd.syn_tx_time = clock_type::now();\n            // Send <SYN> to remote\n            output();\n        }\n        void do_syn_received() {\n            _state = SYN_RECEIVED;\n            _snd.syn_tx_time = clock_type::now();\n            // Send <SYN,ACK> to remote\n            output();\n        }\n        void do_established() {\n            _state = ESTABLISHED;\n            update_rto(_snd.syn_tx_time);\n            _connect_done.set_value();\n        }\n        void do_reset() {\n            _state = CLOSED;\n            cleanup();\n            if (_rcv._data_received_promise) {\n                _rcv._data_received_promise->set_exception(tcp_reset_error());\n                _rcv._data_received_promise = std::nullopt;\n            }\n            if (_snd._all_data_acked_promise) {\n                _snd._all_data_acked_promise->set_exception(tcp_reset_error());\n                _snd._all_data_acked_promise = std::nullopt;\n            }\n            if (_snd._send_available_promise) {\n                _snd._send_available_promise->set_exception(tcp_reset_error());\n                _snd._send_available_promise = std::nullopt;\n            }\n        }\n        void do_time_wait() {\n            // FIXME: Implement TIME_WAIT state timer\n            _state = TIME_WAIT;\n            cleanup();\n        }\n        void do_closed() {\n            _state = CLOSED;\n            cleanup();\n        }\n        void do_setup_isn() {\n            _snd.initial = get_isn();\n            _snd.unacknowledged = _snd.initial;\n            _snd.next = _snd.initial + 1;\n            _snd.recover = _snd.initial;\n        }\n        void do_local_fin_acked() {\n            _snd.unacknowledged += 1;\n            _snd.next += 1;\n        }\n        bool syn_needs_on() const noexcept {\n            return in_state(SYN_SENT | SYN_RECEIVED);\n        }\n        bool fin_needs_on() const noexcept {\n            return in_state(FIN_WAIT_1 | CLOSING | LAST_ACK) && _snd.closed &&\n                   _snd.unsent_len == 0;\n        }\n        bool ack_needs_on() const noexcept {\n            return !in_state(CLOSED | LISTEN | SYN_SENT);\n        }\n        bool foreign_will_not_send() const noexcept {\n            return in_state(CLOSING | TIME_WAIT | CLOSE_WAIT | LAST_ACK | CLOSED);\n        }\n        bool in_state(tcp_state state) const noexcept {\n            return uint16_t(_state) & uint16_t(state);\n        }\n        void exit_fast_recovery() {\n            _snd.dupacks = 0;\n            _snd.limited_transfer = 0;\n            _snd.partial_ack = 0;\n        }\n        uint32_t data_segment_acked(tcp_seq seg_ack);\n        bool segment_acceptable(tcp_seq seg_seq, unsigned seg_len);\n        void init_from_options(tcp_hdr* th, uint8_t* opt_start, uint8_t* opt_end);\n        friend class connection;\n    };\n    inet_type& _inet;\n    std::unordered_map<connid, lw_shared_ptr<tcb>, connid_hash> _tcbs;\n    std::unordered_map<uint16_t, listener*> _listening;\n    std::random_device _rd;\n    std::default_random_engine _e;\n    std::uniform_int_distribution<uint16_t> _port_dist{41952, 65535};\n    circular_buffer<std::pair<lw_shared_ptr<tcb>, ethernet_address>> _poll_tcbs;\n    // queue for packets that do not belong to any tcb\n    circular_buffer<ipv4_traits::l4packet> _packetq;\n    semaphore _queue_space = {212992};\n    metrics::metric_groups _metrics;\npublic:\n    const inet_type& inet() const {\n        return _inet;\n    }\n    class connection {\n        lw_shared_ptr<tcb> _tcb;\n    public:\n        explicit connection(lw_shared_ptr<tcb> tcbp) : _tcb(std::move(tcbp)) { _tcb->_conn = this; }\n        connection(const connection&) = delete;\n        connection(connection&& x) noexcept : _tcb(std::move(x._tcb)) {\n            _tcb->_conn = this;\n        }\n        ~connection();\n        void operator=(const connection&) = delete;\n        connection& operator=(connection&& x) {\n            if (this != &x) {\n                this->~connection();\n                new (this) connection(std::move(x));\n            }\n            return *this;\n        }\n        future<> connected() {\n            return _tcb->connect_done();\n        }\n        future<> send(std::span<temporary_buffer<char>> data) {\n            return _tcb->send(data);\n        }\n        future<> wait_for_data() {\n            return _tcb->wait_for_data();\n        }\n        future<> wait_input_shutdown() {\n            return _tcb->wait_input_shutdown();\n        }\n        packet read() {\n            return _tcb->read();\n        }\n        ipaddr foreign_ip() {\n            return _tcb->_foreign_ip;\n        }\n        uint16_t foreign_port() {\n            return _tcb->_foreign_port;\n        }\n        ipaddr local_ip() {\n            return _tcb->_local_ip;\n        }\n        uint16_t local_port() {\n            return _tcb->_local_port;\n        }\n        void shutdown_connect();\n        void close_read() noexcept;\n        void close_write() noexcept;\n    };\n    class listener {\n        tcp& _tcp;\n        uint16_t _port;\n        queue<connection> _q;\n        size_t _pending = 0;\n    private:\n        listener(tcp& t, uint16_t port, size_t queue_length)\n            : _tcp(t), _port(port), _q(queue_length) {\n            _tcp._listening.emplace(_port, this);\n        }\n    public:\n        listener(listener&& x)\n            : _tcp(x._tcp), _port(x._port), _q(std::move(x._q)) {\n            _tcp._listening[_port] = this;\n            x._port = 0;\n        }\n        ~listener() {\n            if (_port) {\n                _tcp._listening.erase(_port);\n            }\n        }\n        future<connection> accept() {\n            return _q.pop_eventually();\n        }\n        void abort_accept() {\n            _q.abort(std::make_exception_ptr(std::system_error(ECONNABORTED, std::system_category())));\n        }\n        bool full() { return _pending + _q.size() >= _q.max_size(); }\n        void inc_pending() { _pending++; }\n        void dec_pending() { _pending--; }\n\n        const tcp& get_tcp() const {\n            return _tcp;\n        }\n        uint16_t port() const {\n            return _port;\n        }\n        friend class tcp;\n    };\npublic:\n    explicit tcp(inet_type& inet);\n    void received(packet p, ipaddr from, ipaddr to);\n    bool forward(forward_hash& out_hash_data, packet& p, size_t off);\n    listener listen(uint16_t port, size_t queue_length = 100);\n    connection connect(socket_address sa);\n    const net::hw_features& hw_features() const { return _inet._inet.hw_features(); }\n    future<> poll_tcb(ipaddr to, lw_shared_ptr<tcb> tcb);\n    void add_connected_tcb(lw_shared_ptr<tcb> tcbp, uint16_t local_port) {\n        auto it = _listening.find(local_port);\n        if (it != _listening.end()) {\n            it->second->_q.push(connection(tcbp));\n            it->second->dec_pending();\n        }\n    }\nprivate:\n    void send_packet_without_tcb(ipaddr from, ipaddr to, packet p);\n    void respond_with_reset(tcp_hdr* rth, ipaddr local_ip, ipaddr foreign_ip);\n    friend class listener;\n};\n\ntemplate <typename InetTraits>\ntcp<InetTraits>::tcp(inet_type& inet)\n    : _inet(inet)\n    , _e(_rd()) {\n    namespace sm = metrics;\n\n    _metrics.add_group(\"tcp\", {\n        sm::make_counter(\"linearizations\", [] { return tcp_packet_merger::linearizations(); },\n                        sm::description(\"Counts a number of times a buffer linearization was invoked during the buffers merge process. \"\n                                        \"Divide it by a total TCP receive packet rate to get an everage number of lineraizations per TCP packet.\"))\n    });\n\n    _inet.register_packet_provider([this, tcb_polled = 0u] () mutable {\n        std::optional<typename InetTraits::l4packet> l4p;\n        auto c = _poll_tcbs.size();\n        if (!_packetq.empty() && (!(tcb_polled % 128) || c == 0)) {\n            l4p = std::move(_packetq.front());\n            _packetq.pop_front();\n            _queue_space.signal(l4p.value().p.len());\n        } else {\n            while (c--) {\n                tcb_polled++;\n                lw_shared_ptr<tcb> tcb;\n                ethernet_address dst;\n                std::tie(tcb, dst) = std::move(_poll_tcbs.front());\n                _poll_tcbs.pop_front();\n                l4p = tcb->get_packet();\n                if (l4p) {\n                    l4p.value().e_dst = dst;\n                    break;\n                }\n            }\n        }\n        return l4p;\n    });\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::poll_tcb(ipaddr to, lw_shared_ptr<tcb> tcb) {\n    return  _inet.get_l2_dst_address(to).then([this, tcb = std::move(tcb)] (ethernet_address dst) {\n            _poll_tcbs.emplace_back(std::move(tcb), dst);\n    });\n}\n\ntemplate <typename InetTraits>\nauto tcp<InetTraits>::listen(uint16_t port, size_t queue_length) -> listener {\n    return listener(*this, port, queue_length);\n}\n\ntemplate <typename InetTraits>\nauto tcp<InetTraits>::connect(socket_address sa) -> connection {\n    connid id;\n    auto src_ip = _inet._inet.host_address();\n    auto dst_ip = ipv4_address(sa);\n    auto dst_port = net::ntoh(sa.u.in.sin_port);\n\n    if (smp::count > 1) {\n        do {\n            id = connid{src_ip, dst_ip, _port_dist(_e), dst_port};\n        } while (_inet._inet.netif()->hash2cpu(id.hash(_inet._inet.netif()->rss_key())) != this_shard_id()\n                 || _tcbs.find(id) != _tcbs.end());\n    } else {\n        id = connid{src_ip, dst_ip, _port_dist(_e), dst_port};\n    }\n\n    auto tcbp = make_lw_shared<tcb>(*this, id);\n    _tcbs.insert({id, tcbp});\n    tcbp->connect();\n    return connection(tcbp);\n}\n\ntemplate <typename InetTraits>\nbool tcp<InetTraits>::forward(forward_hash& out_hash_data, packet& p, size_t off) {\n    auto th = p.get_header(off, tcp_hdr::len);\n    if (th) {\n        // src_port, dst_port in network byte order\n        out_hash_data.push_back(uint8_t(th[0]));\n        out_hash_data.push_back(uint8_t(th[1]));\n        out_hash_data.push_back(uint8_t(th[2]));\n        out_hash_data.push_back(uint8_t(th[3]));\n    }\n    return true;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::received(packet p, ipaddr from, ipaddr to) {\n    auto th = p.get_header(0, tcp_hdr::len);\n    if (!th) {\n        return;\n    }\n    // data_offset is correct even before ntoh()\n    auto data_offset = uint8_t(th[12]) >> 4;\n    if (size_t(data_offset * 4) < tcp_hdr::len) {\n        return;\n    }\n\n    if (!hw_features().rx_csum_offload) {\n        checksummer csum;\n        InetTraits::tcp_pseudo_header_checksum(csum, from, to, p.len());\n        csum.sum(p);\n        if (csum.get() != 0) {\n            return;\n        }\n    }\n    auto h = tcp_hdr::read(th);\n    auto id = connid{to, from, h.dst_port, h.src_port};\n    auto tcbi = _tcbs.find(id);\n    lw_shared_ptr<tcb> tcbp;\n    if (tcbi == _tcbs.end()) {\n        auto listener = _listening.find(id.local_port);\n        if (listener == _listening.end() || listener->second->full()) {\n            // 1) In CLOSE state\n            // 1.1 all data in the incoming segment is discarded.  An incoming\n            // segment containing a RST is discarded. An incoming segment not\n            // containing a RST causes a RST to be sent in response.\n            // FIXME:\n            //      if ACK off: <SEQ=0><ACK=SEG.SEQ+SEG.LEN><CTL=RST,ACK>\n            //      if ACK on:  <SEQ=SEG.ACK><CTL=RST>\n            return respond_with_reset(&h, id.local_ip, id.foreign_ip);\n        } else {\n            // 2) In LISTEN state\n            // 2.1 first check for an RST\n            if (h.f_rst) {\n                // An incoming RST should be ignored\n                return;\n            }\n            // 2.2 second check for an ACK\n            if (h.f_ack) {\n                // Any acknowledgment is bad if it arrives on a connection\n                // still in the LISTEN state.\n                // <SEQ=SEG.ACK><CTL=RST>\n                return respond_with_reset(&h, id.local_ip, id.foreign_ip);\n            }\n            // 2.3 third check for a SYN\n            if (h.f_syn) {\n                // check the security\n                // NOTE: Ignored for now\n                tcbp = make_lw_shared<tcb>(*this, id);\n                _tcbs.insert({id, tcbp});\n                // TODO: we need to remove the tcb and decrease the pending if\n                // it stays SYN_RECEIVED state forever.\n                listener->second->inc_pending();\n\n                return tcbp->input_handle_listen_state(&h, std::move(p));\n            }\n            // 2.4 fourth other text or control\n            // So you are unlikely to get here, but if you do, drop the\n            // segment, and return.\n            return;\n        }\n    } else {\n        tcbp = tcbi->second;\n        if (tcbp->state() == tcp_state::SYN_SENT) {\n            // 3) In SYN_SENT State\n            return tcbp->input_handle_syn_sent_state(&h, std::move(p));\n        } else {\n            // 4) In other state, can be one of the following:\n            // SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2\n            // CLOSE_WAIT, CLOSING, LAST_ACK, TIME_WAIT\n            return tcbp->input_handle_other_state(&h, std::move(p));\n        }\n    }\n}\n\n// Send packet does not belong to any tcb\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::send_packet_without_tcb(ipaddr from, ipaddr to, packet p) {\n    if (_queue_space.try_wait(p.len())) { // drop packets that do not fit the queue\n        // FIXME: future is discarded\n        (void)_inet.get_l2_dst_address(to).then([this, to, p = std::move(p)] (ethernet_address e_dst) mutable {\n                _packetq.emplace_back(ipv4_traits::l4packet{to, std::move(p), e_dst, ip_protocol_num::tcp});\n        });\n    }\n}\n\ntemplate <typename InetTraits>\ntcp<InetTraits>::connection::~connection() {\n    if (_tcb) {\n        _tcb->_conn = nullptr;\n        close_read();\n        close_write();\n    }\n}\n\ntemplate <typename InetTraits>\ntcp<InetTraits>::tcb::tcb(tcp& t, connid id)\n    : _tcp(t)\n    , _local_ip(id.local_ip)\n    , _foreign_ip(id.foreign_ip)\n    , _local_port(id.local_port)\n    , _foreign_port(id.foreign_port)\n    , _delayed_ack([this] { _nr_full_seg_received = 0; output(); })\n    , _retransmit([this] { retransmit(); })\n    , _persist([this] { persist(); }) {\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::respond_with_reset(tcp_hdr* rth) {\n    _tcp.respond_with_reset(rth, _local_ip, _foreign_ip);\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::respond_with_reset(tcp_hdr* rth, ipaddr local_ip, ipaddr foreign_ip) {\n    if (rth->f_rst) {\n        return;\n    }\n    packet p;\n    auto th = p.prepend_uninitialized_header(tcp_hdr::len);\n    auto h = tcp_hdr{};\n    h.src_port = rth->dst_port;\n    h.dst_port = rth->src_port;\n    if (rth->f_ack) {\n        h.seq = rth->ack;\n    }\n    // If this RST packet is in response to a SYN packet. We ACK the ISN.\n    if (rth->f_syn) {\n        h.ack = rth->seq + 1;\n        h.f_ack = true;\n    }\n    h.f_rst = true;\n    h.data_offset = tcp_hdr::len / 4;\n    h.checksum = 0;\n    h.write(th);\n\n    checksummer csum;\n    offload_info oi;\n    InetTraits::tcp_pseudo_header_checksum(csum, local_ip, foreign_ip, tcp_hdr::len);\n    uint16_t checksum;\n    if (hw_features().tx_csum_l4_offload) {\n        checksum = ~csum.get();\n        oi.needs_csum = true;\n    } else {\n        csum.sum(p);\n        checksum = csum.get();\n        oi.needs_csum = false;\n    }\n    tcp_hdr::write_nbo_checksum(th, checksum);\n\n    oi.protocol = ip_protocol_num::tcp;\n    oi.tcp_hdr_len = tcp_hdr::len;\n    p.set_offload_info(oi);\n\n    send_packet_without_tcb(local_ip, foreign_ip, std::move(p));\n}\n\ntemplate <typename InetTraits>\nuint32_t tcp<InetTraits>::tcb::data_segment_acked(tcp_seq seg_ack) {\n    uint32_t total_acked_bytes = 0;\n    // Full ACK of segment\n    while (!_snd.data.empty()\n            && (_snd.unacknowledged + _snd.data.front().p.len() <= seg_ack)) {\n        auto acked_bytes = _snd.data.front().p.len();\n        _snd.unacknowledged += acked_bytes;\n        // Ignore retransmitted segments when setting the RTO\n        if (_snd.data.front().nr_transmits == 0) {\n            update_rto(_snd.data.front().tx_time);\n        }\n        update_cwnd(acked_bytes);\n        total_acked_bytes += acked_bytes;\n        _snd.current_queue_space -= _snd.data.front().data_len;\n        signal_send_available();\n        _snd.data.pop_front();\n    }\n    // Partial ACK of segment\n    if (_snd.unacknowledged < seg_ack) {\n        auto acked_bytes = seg_ack - _snd.unacknowledged;\n        if (!_snd.data.empty()) {\n            auto& unacked_seg = _snd.data.front();\n            unacked_seg.p.trim_front(acked_bytes);\n        }\n        _snd.unacknowledged = seg_ack;\n        update_cwnd(acked_bytes);\n        total_acked_bytes += acked_bytes;\n    }\n    return total_acked_bytes;\n}\n\ntemplate <typename InetTraits>\nbool tcp<InetTraits>::tcb::segment_acceptable(tcp_seq seg_seq, unsigned seg_len) {\n    if (seg_len == 0 && _rcv.window == 0) {\n        // SEG.SEQ = RCV.NXT\n        return seg_seq == _rcv.next;\n    } else if (seg_len == 0 && _rcv.window > 0) {\n        // RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND\n        return (_rcv.next <= seg_seq) && (seg_seq < _rcv.next + _rcv.window);\n    } else if (seg_len > 0 && _rcv.window > 0) {\n        // RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND\n        //    or\n        // RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND\n        bool x = (_rcv.next <= seg_seq) && seg_seq < (_rcv.next + _rcv.window);\n        bool y = (_rcv.next <= seg_seq + seg_len - 1) && (seg_seq + seg_len - 1 < _rcv.next + _rcv.window);\n        return x || y;\n    } else  {\n        // SEG.LEN > 0 RCV.WND = 0, not acceptable\n        return false;\n    }\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::init_from_options(tcp_hdr* th, uint8_t* opt_start, uint8_t* opt_end) {\n    // Handle tcp options\n    _option.parse(opt_start, opt_end);\n\n    // Remote receive window scale factor\n    _snd.window_scale = _option._remote_win_scale;\n    // Local receive window scale factor\n    _rcv.window_scale = _option._local_win_scale;\n\n    // Maximum segment size remote can receive\n    _snd.mss = _option._remote_mss;\n    // Maximum segment size local can receive\n    _rcv.mss = _option._local_mss = local_mss();\n\n    _rcv.window = get_default_receive_window_size();\n    _snd.window = th->window << _snd.window_scale;\n\n    // Segment sequence number used for last window update\n    _snd.wl1 = th->seq;\n    // Segment acknowledgment number used for last window update\n    _snd.wl2 = th->ack;\n\n    // Setup initial congestion window\n    if (2190 < _snd.mss) {\n        _snd.cwnd = 2 * _snd.mss;\n    } else if (1095 < _snd.mss && _snd.mss <= 2190) {\n        _snd.cwnd = 3 * _snd.mss;\n    } else {\n        _snd.cwnd = 4 * _snd.mss;\n    }\n\n    // Setup initial slow start threshold\n    _snd.ssthresh = th->window << _snd.window_scale;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::input_handle_listen_state(tcp_hdr* th, packet p) {\n    auto opt_len = th->data_offset * 4 - tcp_hdr::len;\n    auto opt_start = reinterpret_cast<uint8_t*>(p.get_header(0, th->data_offset * 4)) + tcp_hdr::len;\n    auto opt_end = opt_start + opt_len;\n    p.trim_front(th->data_offset * 4);\n    tcp_seq seg_seq = th->seq;\n\n    // Set RCV.NXT to SEG.SEQ+1, IRS is set to SEG.SEQ\n    _rcv.next = seg_seq + 1;\n    _rcv.initial = seg_seq;\n\n    // ISS should be selected and a SYN segment sent of the form:\n    // <SEQ=ISS><ACK=RCV.NXT><CTL=SYN,ACK>\n    // SND.NXT is set to ISS+1 and SND.UNA to ISS\n    // NOTE: In previous code, _snd.next is set to ISS + 1 only when SYN is\n    // ACKed. Now, we set _snd.next to ISS + 1 here, so in output_one(): we\n    // have\n    //     th->seq = syn_on ? _snd.initial : _snd.next\n    // to make sure retransmitted SYN has correct SEQ number.\n    do_setup_isn();\n\n    _rcv.urgent = _rcv.next;\n\n    tcp_debug(\"listen: LISTEN -> SYN_RECEIVED\\n\");\n    init_from_options(th, opt_start, opt_end);\n    do_syn_received();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::input_handle_syn_sent_state(tcp_hdr* th, packet p) {\n    auto opt_len = th->data_offset * 4 - tcp_hdr::len;\n    auto opt_start = reinterpret_cast<uint8_t*>(p.get_header(0, th->data_offset * 4)) + tcp_hdr::len;\n    auto opt_end = opt_start + opt_len;\n    p.trim_front(th->data_offset * 4);\n    tcp_seq seg_seq = th->seq;\n    auto seg_ack = th->ack;\n\n    bool acceptable = false;\n    // 3.1 first check the ACK bit\n    if (th->f_ack) {\n        // If SEG.ACK =< ISS, or SEG.ACK > SND.NXT, send a reset (unless the\n        // RST bit is set, if so drop the segment and return)\n        if (seg_ack <= _snd.initial || seg_ack > _snd.next) {\n            return respond_with_reset(th);\n        }\n\n        // If SND.UNA =< SEG.ACK =< SND.NXT then the ACK is acceptable.\n        acceptable = _snd.unacknowledged <= seg_ack && seg_ack <= _snd.next;\n    }\n\n    // 3.2 second check the RST bit\n    if (th->f_rst) {\n        // If the ACK was acceptable then signal the user \"error: connection\n        // reset\", drop the segment, enter CLOSED state, delete TCB, and\n        // return.  Otherwise (no ACK) drop the segment and return.\n        if (acceptable) {\n            _connect_done.set_exception(tcp_refused_error());\n            return do_reset();\n        } else {\n            return;\n        }\n    }\n\n    // 3.3 third check the security and precedence\n    // NOTE: Ignored for now\n\n    // 3.4 fourth check the SYN bit\n    if (th->f_syn) {\n        // RCV.NXT is set to SEG.SEQ+1, IRS is set to SEG.SEQ.  SND.UNA should\n        // be advanced to equal SEG.ACK (if there is an ACK), and any segments\n        // on the retransmission queue which are thereby acknowledged should be\n        // removed.\n        _rcv.next = seg_seq + 1;\n        _rcv.initial = seg_seq;\n        if (th->f_ack) {\n            // TODO: clean retransmission queue\n            _snd.unacknowledged = seg_ack;\n        }\n        if (_snd.unacknowledged > _snd.initial) {\n            // If SND.UNA > ISS (our SYN has been ACKed), change the connection\n            // state to ESTABLISHED, form an ACK segment\n            // <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>\n            tcp_debug(\"syn: SYN_SENT -> ESTABLISHED\\n\");\n            init_from_options(th, opt_start, opt_end);\n            do_established();\n            output();\n        } else {\n            // Otherwise enter SYN_RECEIVED, form a SYN,ACK segment\n            // <SEQ=ISS><ACK=RCV.NXT><CTL=SYN,ACK>\n            tcp_debug(\"syn: SYN_SENT -> SYN_RECEIVED\\n\");\n            do_syn_received();\n        }\n    }\n\n    // 3.5 fifth, if neither of the SYN or RST bits is set then drop the\n    // segment and return.\n    return;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::input_handle_other_state(tcp_hdr* th, packet p) {\n    p.trim_front(th->data_offset * 4);\n    bool do_output = false;\n    bool do_output_data = false;\n    tcp_seq seg_seq = th->seq;\n    auto seg_ack = th->ack;\n    auto seg_len = p.len();\n\n    // 4.1 first check sequence number\n    if (!segment_acceptable(seg_seq, seg_len)) {\n        //<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>\n        return output();\n    }\n\n    // In the following it is assumed that the segment is the idealized\n    // segment that begins at RCV.NXT and does not exceed the window.\n    if (seg_seq < _rcv.next) {\n        // ignore already acknowledged data\n        auto dup = std::min(uint32_t(_rcv.next - seg_seq), seg_len);\n        p.trim_front(dup);\n        seg_len -= dup;\n        seg_seq += dup;\n    }\n    // FIXME: We should trim data outside the right edge of the receive window as well\n\n    if (seg_seq != _rcv.next) {\n        insert_out_of_order(seg_seq, std::move(p));\n        // A TCP receiver SHOULD send an immediate duplicate ACK\n        // when an out-of-order segment arrives.\n        return output();\n    }\n\n    // 4.2 second check the RST bit\n    if (th->f_rst) {\n        if (in_state(SYN_RECEIVED)) {\n            // If this connection was initiated with a passive OPEN (i.e.,\n            // came from the LISTEN state), then return this connection to\n            // LISTEN state and return.  The user need not be informed.  If\n            // this connection was initiated with an active OPEN (i.e., came\n            // from SYN_SENT state) then the connection was refused, signal\n            // the user \"connection refused\".  In either case, all segments\n            // on the retransmission queue should be removed.  And in the\n            // active OPEN case, enter the CLOSED state and delete the TCB,\n            // and return.\n            _connect_done.set_exception(tcp_refused_error());\n            return do_reset();\n        }\n        if (in_state(ESTABLISHED | FIN_WAIT_1 | FIN_WAIT_2 | CLOSE_WAIT)) {\n            // If the RST bit is set then, any outstanding RECEIVEs and SEND\n            // should receive \"reset\" responses.  All segment queues should be\n            // flushed.  Users should also receive an unsolicited general\n            // \"connection reset\" signal.  Enter the CLOSED state, delete the\n            // TCB, and return.\n            return do_reset();\n        }\n        if (in_state(CLOSING | LAST_ACK | TIME_WAIT)) {\n            // If the RST bit is set then, enter the CLOSED state, delete the\n            // TCB, and return.\n            return do_closed();\n        }\n    }\n\n    // 4.3 third check security and precedence\n    // NOTE: Ignored for now\n\n    // 4.4 fourth, check the SYN bit\n    if (th->f_syn) {\n        // SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2\n        // CLOSE_WAIT, CLOSING, LAST_ACK, TIME_WAIT\n\n        // If the SYN is in the window it is an error, send a reset, any\n        // outstanding RECEIVEs and SEND should receive \"reset\" responses,\n        // all segment queues should be flushed, the user should also\n        // receive an unsolicited general \"connection reset\" signal, enter\n        // the CLOSED state, delete the TCB, and return.\n        respond_with_reset(th);\n        return do_reset();\n\n        // If the SYN is not in the window this step would not be reached\n        // and an ack would have been sent in the first step (sequence\n        // number check).\n    }\n\n    // 4.5 fifth check the ACK field\n    if (!th->f_ack) {\n        // if the ACK bit is off drop the segment and return\n        return;\n    } else {\n        // SYN_RECEIVED STATE\n        if (in_state(SYN_RECEIVED)) {\n            // If SND.UNA =< SEG.ACK =< SND.NXT then enter ESTABLISHED state\n            // and continue processing.\n            if (_snd.unacknowledged <= seg_ack && seg_ack <= _snd.next) {\n                tcp_debug(\"SYN_RECEIVED -> ESTABLISHED\\n\");\n                do_established();\n                _tcp.add_connected_tcb(this->shared_from_this(), _local_port);\n            } else {\n                // <SEQ=SEG.ACK><CTL=RST>\n                return respond_with_reset(th);\n            }\n        }\n        auto update_window = [this, th, seg_seq, seg_ack] {\n            tcp_debug(\"window update seg_seq=%d, seg_ack=%d, old window=%d new window=%d\\n\",\n                      seg_seq, seg_ack, _snd.window, th->window << _snd.window_scale);\n            _snd.window = th->window << _snd.window_scale;\n            _snd.wl1 = seg_seq;\n            _snd.wl2 = seg_ack;\n            _snd.zero_window_probing_out = 0;\n            if (_snd.window == 0) {\n                _persist_time_out = _rto;\n                start_persist_timer();\n            } else {\n                stop_persist_timer();\n            }\n        };\n        // ESTABLISHED STATE or\n        // CLOSE_WAIT STATE: Do the same processing as for the ESTABLISHED state.\n        if (in_state(ESTABLISHED | CLOSE_WAIT)){\n            // When we are in zero window probing phase and packets_out = 0 we bypass \"duplicated ack\" check\n            auto packets_out = _snd.next - _snd.unacknowledged - _snd.zero_window_probing_out;\n            // If SND.UNA < SEG.ACK =< SND.NXT then, set SND.UNA <- SEG.ACK.\n            if (_snd.unacknowledged < seg_ack && seg_ack <= _snd.next) {\n                // Remote ACKed data we sent\n                auto acked_bytes = data_segment_acked(seg_ack);\n\n                // If SND.UNA < SEG.ACK =< SND.NXT, the send window should be updated.\n                if (_snd.wl1 < seg_seq || (_snd.wl1 == seg_seq && _snd.wl2 <= seg_ack)) {\n                    update_window();\n                }\n\n                // some data is acked, try send more data\n                do_output_data = true;\n\n                auto set_retransmit_timer = [this] {\n                    if (_snd.data.empty()) {\n                        // All outstanding segments are acked, turn off the timer.\n                        stop_retransmit_timer();\n                        // Signal the waiter of this event\n                        signal_all_data_acked();\n                    } else {\n                        // Restart the timer becasue new data is acked.\n                        start_retransmit_timer();\n                    }\n                };\n\n                if (_snd.dupacks >= 3) {\n                    // We are in fast retransmit / fast recovery phase\n                    uint32_t smss = _snd.mss;\n                    if (seg_ack > _snd.recover) {\n                        tcp_debug(\"ack: full_ack\\n\");\n                        // Set cwnd to min (ssthresh, max(FlightSize, SMSS) + SMSS)\n                        _snd.cwnd = std::min(_snd.ssthresh, std::max(flight_size(), smss) + smss);\n                        // Exit the fast recovery procedure\n                        exit_fast_recovery();\n                        set_retransmit_timer();\n                    } else {\n                        tcp_debug(\"ack: partial_ack\\n\");\n                        // Retransmit the first unacknowledged segment\n                        fast_retransmit();\n                        // Deflate the congestion window by the amount of new data\n                        // acknowledged by the Cumulative Acknowledgment field\n                        _snd.cwnd -= acked_bytes;\n                        // If the partial ACK acknowledges at least one SMSS of new\n                        // data, then add back SMSS bytes to the congestion window\n                        if (acked_bytes >= smss) {\n                            _snd.cwnd += smss;\n                        }\n                        // Send a new segment if permitted by the new value of\n                        // cwnd.  Do not exit the fast recovery procedure For\n                        // the first partial ACK that arrives during fast\n                        // recovery, also reset the retransmit timer.\n                        if (++_snd.partial_ack == 1) {\n                            start_retransmit_timer();\n                        }\n                    }\n                } else {\n                    // RFC5681: The fast retransmit algorithm uses the arrival\n                    // of 3 duplicate ACKs (as defined in section 2, without\n                    // any intervening ACKs which move SND.UNA) as an\n                    // indication that a segment has been lost.\n                    //\n                    // So, here we reset dupacks to zero becasue this ACK moves\n                    // SND.UNA.\n                    exit_fast_recovery();\n                    set_retransmit_timer();\n                }\n            } else if ((packets_out > 0) && !_snd.data.empty() && seg_len == 0 &&\n                th->f_fin == 0 && th->f_syn == 0 &&\n                th->ack == _snd.unacknowledged &&\n                uint32_t(th->window << _snd.window_scale) == _snd.window) {\n                // Note:\n                // RFC793 states:\n                // If the ACK is a duplicate (SEG.ACK < SND.UNA), it can be ignored\n                // RFC5681 states:\n                // The TCP sender SHOULD use the \"fast retransmit\" algorithm to detect\n                // and repair loss, based on incoming duplicate ACKs.\n                // Here, We follow RFC5681.\n                _snd.dupacks++;\n                uint32_t smss = _snd.mss;\n                // 3 duplicated ACKs trigger a fast retransmit\n                if (_snd.dupacks == 1 || _snd.dupacks == 2) {\n                    // RFC5681 Step 3.1\n                    // Send cwnd + 2 * smss per RFC3042\n                    do_output_data = true;\n                } else if (_snd.dupacks == 3) {\n                    // RFC6582 Step 3.2\n                    if (seg_ack - 1 > _snd.recover) {\n                        _snd.recover = _snd.next - 1;\n                        // RFC5681 Step 3.2\n                        _snd.ssthresh = std::max((flight_size() - _snd.limited_transfer) / 2, 2 * smss);\n                        fast_retransmit();\n                    } else {\n                        // Do not enter fast retransmit and do not reset ssthresh\n                    }\n                    // RFC5681 Step 3.3\n                    _snd.cwnd = _snd.ssthresh + 3 * smss;\n                } else if (_snd.dupacks > 3) {\n                    // RFC5681 Step 3.4\n                    _snd.cwnd += smss;\n                    // RFC5681 Step 3.5\n                    do_output_data = true;\n                }\n            } else if (seg_ack > _snd.next) {\n                // If the ACK acks something not yet sent (SEG.ACK > SND.NXT)\n                // then send an ACK, drop the segment, and return\n                return output();\n            } else if (_snd.window == 0 && th->window > 0) {\n                update_window();\n                do_output_data = true;\n            }\n        }\n        // FIN_WAIT_1 STATE\n        if (in_state(FIN_WAIT_1)) {\n            // In addition to the processing for the ESTABLISHED state, if\n            // our FIN is now acknowledged then enter FIN-WAIT-2 and continue\n            // processing in that state.\n            if (seg_ack == _snd.next + 1) {\n                tcp_debug(\"ack: FIN_WAIT_1 -> FIN_WAIT_2\\n\");\n                _state = FIN_WAIT_2;\n                do_local_fin_acked();\n            }\n        }\n        // FIN_WAIT_2 STATE\n        if (in_state(FIN_WAIT_2)) {\n            // In addition to the processing for the ESTABLISHED state, if\n            // the retransmission queue is empty, the user’s CLOSE can be\n            // acknowledged (\"ok\") but do not delete the TCB.\n            // TODO\n        }\n        // CLOSING STATE\n        if (in_state(CLOSING)) {\n            if (seg_ack == _snd.next + 1) {\n                tcp_debug(\"ack: CLOSING -> TIME_WAIT\\n\");\n                do_local_fin_acked();\n                return do_time_wait();\n            } else {\n                return;\n            }\n        }\n        // LAST_ACK STATE\n        if (in_state(LAST_ACK)) {\n            if (seg_ack == _snd.next + 1) {\n                tcp_debug(\"ack: LAST_ACK -> CLOSED\\n\");\n                do_local_fin_acked();\n                return do_closed();\n            }\n        }\n        // TIME_WAIT STATE\n        if (in_state(TIME_WAIT)) {\n            // The only thing that can arrive in this state is a\n            // retransmission of the remote FIN. Acknowledge it, and restart\n            // the 2 MSL timeout.\n            // TODO\n        }\n    }\n\n    // 4.6 sixth, check the URG bit\n    if (th->f_urg) {\n        // TODO\n    }\n\n    // 4.7 seventh, process the segment text\n    if (in_state(ESTABLISHED | FIN_WAIT_1 | FIN_WAIT_2)) {\n        if (p.len()) {\n            // Once the TCP takes responsibility for the data it advances\n            // RCV.NXT over the data accepted, and adjusts RCV.WND as\n            // apporopriate to the current buffer availability.  The total of\n            // RCV.NXT and RCV.WND should not be reduced.\n            _rcv.data_size += p.len();\n            _rcv.data.push_back(std::move(p));\n            _rcv.next += seg_len;\n            auto merged = merge_out_of_order();\n            _rcv.window = get_modified_receive_window_size();\n            signal_data_received();\n            // Send an acknowledgment of the form:\n            // <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>\n            // This acknowledgment should be piggybacked on a segment being\n            // transmitted if possible without incurring undue delay.\n            if (merged) {\n                // TCP receiver SHOULD send an immediate ACK when the\n                // incoming segment fills in all or part of a gap in the\n                // sequence space.\n                do_output = true;\n            } else {\n                do_output = should_send_ack(seg_len);\n            }\n        }\n    } else if (in_state(CLOSE_WAIT | CLOSING | LAST_ACK | TIME_WAIT)) {\n        // This should not occur, since a FIN has been received from the\n        // remote side. Ignore the segment text.\n        return;\n    }\n\n    // 4.8 eighth, check the FIN bit\n    if (th->f_fin) {\n        if (_fin_recvd_promise) {\n            _fin_recvd_promise->set_value();\n            _fin_recvd_promise.reset();\n        }\n        if (in_state(CLOSED | LISTEN | SYN_SENT)) {\n            // Do not process the FIN if the state is CLOSED, LISTEN or SYN-SENT\n            // since the SEG.SEQ cannot be validated; drop the segment and return.\n            return;\n        }\n        auto fin_seq = seg_seq + seg_len;\n        if (fin_seq == _rcv.next) {\n            _rcv.next = fin_seq + 1;\n            signal_data_received();\n\n            // If this <FIN> packet contains data as well, we can ACK both data\n            // and <FIN> in a single packet, so canncel the previous ACK.\n            clear_delayed_ack();\n            do_output = false;\n            // Send ACK for the FIN!\n            output();\n\n            if (in_state(SYN_RECEIVED | ESTABLISHED)) {\n                tcp_debug(\"fin: SYN_RECEIVED or ESTABLISHED -> CLOSE_WAIT\\n\");\n                _state = CLOSE_WAIT;\n            }\n            if (in_state(FIN_WAIT_1)) {\n                // If our FIN has been ACKed (perhaps in this segment), then\n                // enter TIME-WAIT, start the time-wait timer, turn off the other\n                // timers; otherwise enter the CLOSING state.\n                // Note: If our FIN has been ACKed, we should be in FIN_WAIT_2\n                // not FIN_WAIT_1 if we reach here.\n                tcp_debug(\"fin: FIN_WAIT_1 -> CLOSING\\n\");\n                _state = CLOSING;\n            }\n            if (in_state(FIN_WAIT_2)) {\n                tcp_debug(\"fin: FIN_WAIT_2 -> TIME_WAIT\\n\");\n                return do_time_wait();\n            }\n        }\n    }\n    if (do_output || (do_output_data && can_send())) {\n        // Since we will do output, we can canncel scheduled delayed ACK.\n        clear_delayed_ack();\n        output();\n    }\n}\n\ntemplate <typename InetTraits>\npacket tcp<InetTraits>::tcb::get_transmit_packet() {\n    // easy case: empty queue\n    if (_snd.unsent.empty()) {\n        return packet();\n    }\n    auto can_send = this->can_send();\n    // Max number of TCP payloads we can pass to NIC\n    uint32_t len;\n    if (_tcp.hw_features().tx_tso) {\n        // FIXME: Info tap device the size of the splitted packet\n        len = _tcp.hw_features().max_packet_len - net::tcp_hdr_len_min - InetTraits::ip_hdr_len_min;\n    } else {\n        len = std::min(uint16_t(_tcp.hw_features().mtu - net::tcp_hdr_len_min - InetTraits::ip_hdr_len_min), _snd.mss);\n    }\n    can_send = std::min(can_send, len);\n    auto p = packet();\n    while (!_snd.unsent.empty() && _snd.unsent.front().size() <= can_send) {\n        can_send -= _snd.unsent.front().size();\n        p.append(std::move(_snd.unsent.front()));\n        _snd.unsent.pop_front();\n    }\n    if (!_snd.unsent.empty() && can_send) {\n        auto& q = _snd.unsent.front();\n        p.append(q.share(0, can_send));\n        q.trim_front(can_send);\n    }\n    _snd.unsent_len -= p.len();\n    return p;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::output_one(bool data_retransmit) {\n    if (in_state(CLOSED)) {\n        return;\n    }\n\n    packet p = data_retransmit ? _snd.data.front().p.share() : get_transmit_packet();\n    packet clone = p.share();  // early clone to prevent share() from calling packet::unuse_internal_data() on header.\n    uint16_t len = p.len();\n    bool syn_on = syn_needs_on();\n    bool ack_on = ack_needs_on();\n\n    auto options_size = _option.get_size(syn_on, ack_on);\n    auto th = p.prepend_uninitialized_header(tcp_hdr::len + options_size);\n    auto h = tcp_hdr{};\n\n    h.src_port = _local_port;\n    h.dst_port = _foreign_port;\n\n    h.f_syn = syn_on;\n    h.f_ack = ack_on;\n    if (ack_on) {\n        clear_delayed_ack();\n    }\n    h.f_urg = false;\n    h.f_psh = false;\n\n    tcp_seq seq;\n    if (data_retransmit) {\n        seq = _snd.unacknowledged;\n    } else {\n        seq = syn_on ? _snd.initial : _snd.next;\n        _snd.next += len;\n    }\n    h.seq = seq;\n    h.ack = _rcv.next;\n    h.data_offset = (tcp_hdr::len + options_size) / 4;\n    h.window = _rcv.window >> _rcv.window_scale;\n    h.checksum = 0;\n\n    // FIXME: does the FIN have to fit in the window?\n    bool fin_on = fin_needs_on();\n    h.f_fin = fin_on;\n\n    // Add tcp options\n    _option.fill(th, &h, options_size);\n    h.write(th);\n\n    offload_info oi;\n    checksummer csum;\n    uint16_t pseudo_hdr_seg_len = 0;\n\n    oi.tcp_hdr_len = tcp_hdr::len + options_size;\n\n    if (_tcp.hw_features().tx_csum_l4_offload) {\n        oi.needs_csum = true;\n\n        //\n        // tx checksum offloading: both virtio-net's VIRTIO_NET_F_CSUM dpdk's\n        // PKT_TX_TCP_CKSUM - requires th->checksum to be initialized to ones'\n        // complement sum of the pseudo header.\n        //\n        // For TSO the csum should be calculated for a pseudo header with\n        // segment length set to 0. All the rest is the same as for a TCP Tx\n        // CSUM offload case.\n        //\n        if (_tcp.hw_features().tx_tso && len > _snd.mss) {\n            oi.tso_seg_size = _snd.mss;\n        } else {\n            pseudo_hdr_seg_len = tcp_hdr::len + options_size + len;\n        }\n    } else {\n        pseudo_hdr_seg_len = tcp_hdr::len + options_size + len;\n        oi.needs_csum = false;\n    }\n\n    InetTraits::tcp_pseudo_header_checksum(csum, _local_ip, _foreign_ip,\n                                           pseudo_hdr_seg_len);\n\n    uint16_t checksum;\n    if (_tcp.hw_features().tx_csum_l4_offload) {\n        checksum = ~csum.get();\n    } else {\n        csum.sum(p);\n        checksum = csum.get();\n    }\n    tcp_hdr::write_nbo_checksum(th, checksum);\n\n    oi.protocol = ip_protocol_num::tcp;\n\n    p.set_offload_info(oi);\n\n    if (!data_retransmit && (len || syn_on || fin_on)) {\n        auto now = clock_type::now();\n        if (len) {\n            unsigned nr_transmits = 0;\n            _snd.data.emplace_back(unacked_segment{std::move(clone),\n                                   len, nr_transmits, now});\n        }\n        if (!_retransmit.armed()) {\n            start_retransmit_timer(now);\n        }\n    }\n\n\n    // if advertised TCP receive window is 0 we may only transmit zero window probing segment.\n    // Payload size of this segment is 1. Queueing anything bigger when _snd.window == 0 is bug\n    // and violation of RFC\n    SEASTAR_ASSERT((_snd.window > 0) || ((_snd.window == 0) && (len <= 1)));\n    queue_packet(std::move(p));\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::tcb::wait_for_data() {\n    if (!_rcv.data.empty() || foreign_will_not_send()) {\n        return make_ready_future<>();\n    }\n    _rcv._data_received_promise = promise<>();\n    return _rcv._data_received_promise->get_future();\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::tcb::wait_input_shutdown() {\n    if (!_fin_recvd_promise) {\n        return make_ready_future<>();\n    }\n    return _fin_recvd_promise->get_future();\n}\n\ntemplate <typename InetTraits>\nvoid\ntcp<InetTraits>::tcb::abort_reader() noexcept {\n    if (_rcv._data_received_promise) {\n        _rcv._data_received_promise->set_exception(\n                std::make_exception_ptr(std::system_error(ECONNABORTED, std::system_category())));\n        _rcv._data_received_promise = std::nullopt;\n    }\n    if (_fin_recvd_promise) {\n        _fin_recvd_promise->set_value();\n        _fin_recvd_promise.reset();\n    }\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::tcb::wait_for_all_data_acked() {\n    if (_snd.data.empty() && _snd.unsent_len == 0) {\n        return make_ready_future<>();\n    }\n    _snd._all_data_acked_promise = promise<>();\n    return _snd._all_data_acked_promise->get_future();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::connect() {\n    // An initial send sequence number (ISS) is selected.  A SYN segment of the\n    // form <SEQ=ISS><CTL=SYN> is sent.  Set SND.UNA to ISS, SND.NXT to ISS+1,\n    // enter SYN-SENT state, and return.\n    do_setup_isn();\n\n    // Local receive window scale factor\n    _rcv.window_scale = _option._local_win_scale = 7;\n    // Maximum segment size local can receive\n    _rcv.mss = _option._local_mss = local_mss();\n    _rcv.window = get_default_receive_window_size();\n\n    do_syn_sent();\n}\n\ntemplate <typename InetTraits>\npacket tcp<InetTraits>::tcb::read() {\n    packet p;\n    for (auto&& q : _rcv.data) {\n        p.append(std::move(q));\n    }\n    _rcv.data_size = 0;\n    _rcv.data.clear();\n    _rcv.window = get_default_receive_window_size();\n    return p;\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::tcb::wait_send_available() {\n    if (_snd.max_queue_space > _snd.current_queue_space) {\n        return make_ready_future<>();\n    }\n    _snd._send_available_promise = promise<>();\n    return _snd._send_available_promise->get_future();\n}\n\ntemplate <typename InetTraits>\nfuture<> tcp<InetTraits>::tcb::send(std::span<temporary_buffer<char>> data) {\n    // We can not send after the connection is closed\n    if (_snd.closed || in_state(CLOSED)) {\n        return make_exception_future<>(tcp_reset_error());\n    }\n\n    auto sizes = data | std::views::transform(&temporary_buffer<char>::size);\n    auto len = std::accumulate(sizes.begin(), sizes.end(), size_t(0));\n\n    _snd.current_queue_space += len;\n    _snd.unsent_len += len;\n    // FIXME: use append_range with C++23\n    std::ranges::move(data, std::back_inserter(_snd.unsent));\n\n    if (can_send() > 0) {\n        output();\n    }\n\n    return wait_send_available();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::close() noexcept {\n    if (in_state(CLOSED) || _snd.closed) {\n        return;\n    }\n    // TODO: We should return a future to upper layer\n    (void)wait_for_all_data_acked().then([this, zis = this->shared_from_this()] () mutable {\n        _snd.closed = true;\n        tcp_debug(\"close: unsent_len=%d\\n\", _snd.unsent_len);\n        if (in_state(CLOSE_WAIT)) {\n            tcp_debug(\"close: CLOSE_WAIT -> LAST_ACK\\n\");\n            _state = LAST_ACK;\n        } else if (in_state(ESTABLISHED)) {\n            tcp_debug(\"close: ESTABLISHED -> FIN_WAIT_1\\n\");\n            _state = FIN_WAIT_1;\n        }\n        // Send <FIN> to remote\n        // Note: we call output_one to make sure a packet with FIN actually\n        // sent out. If we only call output() and _packetq is not empty,\n        // tcp::tcb::get_packet(), packet with FIN will not be generated.\n        output_one();\n        output();\n    });\n}\n\ntemplate <typename InetTraits>\nbool tcp<InetTraits>::tcb::should_send_ack(uint16_t seg_len) {\n    // We've received a TSO packet, do ack immediately\n    if (seg_len > _rcv.mss) {\n        _nr_full_seg_received = 0;\n        _delayed_ack.cancel();\n        return true;\n    }\n\n    // We've received a full sized segment, ack for every second full sized segment\n    if (seg_len == _rcv.mss) {\n        if (_nr_full_seg_received++ >= 1) {\n            _nr_full_seg_received = 0;\n            _delayed_ack.cancel();\n            return true;\n        }\n    }\n\n    // If the timer is armed and its callback hasn't been run.\n    if (_delayed_ack.armed()) {\n        return false;\n    }\n\n    // If the timer is not armed, schedule a delayed ACK.\n    // The maximum delayed ack timer allowed by RFC1122 is 500ms, most\n    // implementations use 200ms.\n    _delayed_ack.arm(200ms);\n    return false;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::clear_delayed_ack() noexcept {\n    _delayed_ack.cancel();\n}\n\ntemplate <typename InetTraits>\nbool tcp<InetTraits>::tcb::merge_out_of_order() {\n    bool merged = false;\n    if (_rcv.out_of_order.map.empty()) {\n        return merged;\n    }\n    for (auto it = _rcv.out_of_order.map.begin(); it != _rcv.out_of_order.map.end();) {\n        auto& p = it->second;\n        auto seg_beg = it->first;\n        auto seg_len = p.len();\n        auto seg_end = seg_beg + seg_len;\n        if (seg_beg <= _rcv.next && _rcv.next < seg_end) {\n            // This segment has been received out of order and its previous\n            // segment has been received now\n            auto trim = _rcv.next - seg_beg;\n            if (trim) {\n                p.trim_front(trim);\n                seg_len -= trim;\n            }\n            _rcv.next += seg_len;\n            _rcv.data_size += p.len();\n            _rcv.data.push_back(std::move(p));\n            // Since c++11, erase() always returns the value of the following element\n            it = _rcv.out_of_order.map.erase(it);\n            merged = true;\n        } else if (_rcv.next >= seg_end) {\n            // This segment has been receive already, drop it\n            it = _rcv.out_of_order.map.erase(it);\n        } else {\n            // seg_beg > _rcv.need, can not merge. Note, seg_beg can grow only,\n            // so we can stop looking here.\n            it++;\n            break;\n        }\n    }\n    return merged;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::insert_out_of_order(tcp_seq seg, packet p) {\n    _rcv.out_of_order.merge(seg, std::move(p));\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::trim_receive_data_after_window() {\n    abort();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::persist() {\n    tcp_debug(\"persist timer fired\\n\");\n    // Send 1 byte packet to probe peer's window size\n    _snd.window_probe = true;\n    _snd.zero_window_probing_out++;\n    output_one();\n    _snd.window_probe = false;\n\n    output();\n    // Perform binary exponential back-off per RFC1122\n    _persist_time_out = std::min(_persist_time_out * 2, _rto_max);\n    start_persist_timer();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::retransmit() {\n    auto output_update_rto = [this] {\n        output();\n        // According to RFC6298, Update RTO <- RTO * 2 to perform binary exponential back-off\n        this->_rto = std::min(this->_rto * 2, this->_rto_max);\n        start_retransmit_timer();\n    };\n\n    // Retransmit SYN\n    if (syn_needs_on()) {\n        if (_snd.syn_retransmit++ < _max_nr_retransmit) {\n            output_update_rto();\n        } else {\n            _connect_done.set_exception(tcp_connect_error());\n            cleanup();\n            return;\n        }\n    }\n\n    // Retransmit FIN\n    if (fin_needs_on()) {\n        if (_snd.fin_retransmit++ < _max_nr_retransmit) {\n            output_update_rto();\n        } else {\n            cleanup();\n            return;\n        }\n    }\n\n    // Retransmit Data\n    if (_snd.data.empty()) {\n        return;\n    }\n\n    // If there are unacked data, retransmit the earliest segment\n    auto& unacked_seg = _snd.data.front();\n\n    // According to RFC5681\n    // Update ssthresh only for the first retransmit\n    uint32_t smss = _snd.mss;\n    if (unacked_seg.nr_transmits == 0) {\n        _snd.ssthresh = std::max(flight_size() / 2, 2 * smss);\n    }\n    // RFC6582 Step 4\n    _snd.recover = _snd.next - 1;\n    // Start the slow start process\n    _snd.cwnd = smss;\n    // End fast recovery\n    exit_fast_recovery();\n\n    if (unacked_seg.nr_transmits < _max_nr_retransmit) {\n        unacked_seg.nr_transmits++;\n    } else {\n        // Delete connection when max num of retransmission is reached\n        do_reset();\n        return;\n    }\n    retransmit_one();\n\n    output_update_rto();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::fast_retransmit() {\n    if (!_snd.data.empty()) {\n        auto& unacked_seg = _snd.data.front();\n        unacked_seg.nr_transmits++;\n        retransmit_one();\n        output();\n    }\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::update_rto(clock_type::time_point tx_time) {\n    // Update RTO according to RFC6298\n    auto R = std::chrono::duration_cast<std::chrono::milliseconds>(clock_type::now() - tx_time);\n    if (_snd.first_rto_sample) {\n        _snd.first_rto_sample = false;\n        // RTTVAR <- R/2\n        // SRTT <- R\n        _snd.rttvar = R / 2;\n        _snd.srtt = R;\n    } else {\n        // RTTVAR <- (1 - beta) * RTTVAR + beta * |SRTT - R'|\n        // SRTT <- (1 - alpha) * SRTT + alpha * R'\n        // where alpha = 1/8 and beta = 1/4\n        auto delta = _snd.srtt > R ? (_snd.srtt - R) : (R - _snd.srtt);\n        _snd.rttvar = _snd.rttvar * 3 / 4 + delta / 4;\n        _snd.srtt = _snd.srtt * 7 / 8 +  R / 8;\n    }\n    // RTO <- SRTT + max(G, K * RTTVAR)\n    _rto =  _snd.srtt + std::max(_rto_clk_granularity, 4 * _snd.rttvar);\n\n    // Make sure 1 sec << _rto << 60 sec\n    _rto = std::max(_rto, _rto_min);\n    _rto = std::min(_rto, _rto_max);\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::update_cwnd(uint32_t acked_bytes) {\n    uint32_t smss = _snd.mss;\n    if (_snd.cwnd < _snd.ssthresh) {\n        // In slow start phase\n        _snd.cwnd += std::min(acked_bytes, smss);\n    } else {\n        // In congestion avoidance phase\n        uint32_t round_up = 1;\n        _snd.cwnd += std::max(round_up, smss * smss / _snd.cwnd);\n    }\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::tcb::cleanup() {\n    _snd.unsent.clear();\n    _snd.data.clear();\n    _rcv.out_of_order.map.clear();\n    _rcv.data_size = 0;\n    _rcv.data.clear();\n    stop_retransmit_timer();\n    clear_delayed_ack();\n    remove_from_tcbs();\n}\n\ntemplate <typename InetTraits>\ntcp_seq tcp<InetTraits>::tcb::get_isn() {\n    // Per RFC6528, TCP SHOULD generate its Initial Sequence Numbers\n    // with the expression:\n    //   ISN = M + F(localip, localport, remoteip, remoteport, secretkey)\n    //   M is the 4 microsecond timer\n    using namespace std::chrono;\n    uint32_t hash[4];\n    hash[0] = _local_ip.ip;\n    hash[1] = _foreign_ip.ip;\n    hash[2] = (_local_port << 16) + _foreign_port;\n    gnutls_hash_hd_t md5_hash_handle;\n    // GnuTLS digests do not init at all, so this should never fail.\n    gnutls_hash_init(&md5_hash_handle, GNUTLS_DIG_MD5);\n    gnutls_hash(md5_hash_handle, hash, 3 * sizeof(hash[0]));\n    gnutls_hash(md5_hash_handle, _isn_secret.key, sizeof(_isn_secret.key));\n    // reuse \"hash\" for the output of digest\n    SEASTAR_ASSERT(sizeof(hash) == gnutls_hash_get_len(GNUTLS_DIG_MD5));\n    gnutls_hash_deinit(md5_hash_handle, hash);\n    auto seq = hash[0];\n    auto m = duration_cast<microseconds>(clock_type::now().time_since_epoch());\n    seq += m.count() / 4;\n    return make_seq(seq);\n}\n\ntemplate <typename InetTraits>\nstd::optional<typename InetTraits::l4packet> tcp<InetTraits>::tcb::get_packet() {\n    _poll_active = false;\n    if (_packetq.empty()) {\n        output_one();\n    }\n\n    if (in_state(CLOSED)) {\n        return std::optional<typename InetTraits::l4packet>();\n    }\n\n    SEASTAR_ASSERT(!_packetq.empty());\n\n    auto p = std::move(_packetq.front());\n    _packetq.pop_front();\n    if (!_packetq.empty() || (_snd.dupacks < 3 && can_send() > 0 && (_snd.window > 0))) {\n        // If there are packets to send in the queue or tcb is allowed to send\n        // more add tcp back to polling set to keep sending. In addition, dupacks >= 3\n        // is an indication that an segment is lost, stop sending more in this case.\n        // Finally - we can't send more until window is opened again.\n        output();\n    }\n    return p;\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::connection::close_read() noexcept {\n    _tcb->abort_reader();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::connection::close_write() noexcept {\n    _tcb->close();\n}\n\ntemplate <typename InetTraits>\nvoid tcp<InetTraits>::connection::shutdown_connect() {\n    if (_tcb->syn_needs_on()) {\n      _tcb->_connect_done.set_exception(tcp_refused_error());\n      _tcb->cleanup();\n    } else {\n        close_read();\n        close_write();\n    }\n}\n\ntemplate <typename InetTraits>\ntypename tcp<InetTraits>::tcb::isn_secret tcp<InetTraits>::tcb::_isn_secret;\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/tls.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n#pragma once\n\n#include <functional>\n#include <unordered_set>\n#include <map>\n#include <any>\n#include <fmt/format.h>\n\n#include <seastar/core/future.hh>\n#include <seastar/core/internal/api-level.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/net/api.hh>\n\nnamespace seastar {\n\nclass socket;\n\nclass server_socket;\nclass connected_socket;\nclass socket_address;\n\n/**\n * Relatively thin SSL wrapper for socket IO.\n * (Can be expanded to other IO forms).\n *\n * The current underlying mechanism is\n * gnutls, however, all interfaces are kept\n * agnostic, so in theory it could be replaced\n * with OpenSSL or similar.\n *\n */\nnamespace tls {\n\n    enum class x509_crt_format {\n        DER,\n        PEM,\n    };\n\n    typedef std::basic_string_view<char> blob;\n\n    class session;\n    class server_session;\n    class server_credentials;\n    class certificate_credentials;\n    class credentials_builder;\n\n    /**\n     * Diffie-Hellman parameters for\n     * wire encryption.\n     */\n    class dh_params {\n    public:\n        // Key strength\n        enum class level {\n            LEGACY = 2,\n            MEDIUM = 3,\n            HIGH = 4,\n            ULTRA = 5\n        };\n        dh_params(level = level::LEGACY);\n        // loads a key from data\n        dh_params(const blob&, x509_crt_format);\n        ~dh_params();\n\n        dh_params(dh_params&&) noexcept;\n        dh_params& operator=(dh_params&&) noexcept;\n\n        dh_params(const dh_params&) = delete;\n        dh_params& operator=(const dh_params&) = delete;\n\n        /** loads a key from file */\n        static future<dh_params> from_file(const sstring&, x509_crt_format);\n    private:\n        class impl;\n        friend class server_credentials;\n        friend class certificate_credentials;\n        std::unique_ptr<impl> _impl;\n    };\n\n    class x509_cert {\n        x509_cert(const blob&, x509_crt_format);\n\n        static future<x509_cert> from_file(const sstring&, x509_crt_format);\n    private:\n        class impl;\n        x509_cert(shared_ptr<impl>);\n        shared_ptr<impl> _impl;\n    };\n\n    class abstract_credentials {\n    protected:\n        abstract_credentials() = default;\n        abstract_credentials(const abstract_credentials&) = default;\n        abstract_credentials& operator=(abstract_credentials&) = default;\n        abstract_credentials& operator=(abstract_credentials&&) = default;\n        virtual ~abstract_credentials() {};\n    public:\n        virtual void set_x509_trust(const blob&, x509_crt_format) = 0;\n        virtual void set_x509_crl(const blob&, x509_crt_format) = 0;\n        virtual void set_x509_key(const blob& cert, const blob& key, x509_crt_format) = 0;\n\n        virtual void set_simple_pkcs12(const blob&, x509_crt_format, const sstring& password) = 0;\n\n        virtual future<> set_x509_trust_file(const sstring& cafile, x509_crt_format);\n        virtual future<> set_x509_crl_file(const sstring& crlfile, x509_crt_format);\n        virtual future<> set_x509_key_file(const sstring& cf, const sstring& kf, x509_crt_format);\n\n        virtual future<> set_simple_pkcs12_file(const sstring& pkcs12file, x509_crt_format, const sstring& password);\n    };\n\n    template<typename Base>\n    class reloadable_credentials;\n\n    /**\n     * Enum like tls::session::type but independent of gnutls headers\n     *\n     * \\warning Uses a different internal encoding than tls::session::type\n     */\n    enum class session_type {\n        CLIENT, SERVER,\n    };\n\n    /**\n     * Callback prototype for receiving Distinguished Name (DN) information\n     *\n     * \\param type Our own role in the TLS handshake (client vs. server)\n     * \\param subject The subject DN string\n     * \\param issuer The issuer DN string\n     */\n    using dn_callback = noncopyable_function<void(session_type type, sstring subject, sstring issuer)>;\n\n    /**\n     * Holds certificates and keys.\n     *\n     * Typically, credentials are shared for multiple client/server\n     * sessions. Changes to the credentials object will affect all\n     * sessions instantiated with it.\n     * You should probably set it up once, before starting client/server\n     * connections.\n     */\n    class certificate_credentials : public abstract_credentials {\n    public:\n        certificate_credentials();\n        ~certificate_credentials();\n\n        certificate_credentials(certificate_credentials&&) noexcept;\n        certificate_credentials& operator=(certificate_credentials&&) noexcept;\n\n        certificate_credentials(const certificate_credentials&) = delete;\n        certificate_credentials& operator=(const certificate_credentials&) = delete;\n\n        void set_x509_trust(const blob&, x509_crt_format) override;\n        void set_x509_crl(const blob&, x509_crt_format) override;\n        void set_x509_key(const blob& cert, const blob& key, x509_crt_format) override;\n        void set_simple_pkcs12(const blob&, x509_crt_format, const sstring& password) override;\n\n        /**\n         * Loads default system cert trust file\n         * into this object.\n         */\n        future<> set_system_trust();\n\n        // TODO add methods for certificate verification\n\n        /**\n         * TLS handshake priority string. See gnutls docs and syntax at\n         * https://gnutls.org/manual/html_node/Priority-Strings.html\n         *\n         * Allows specifying order and allowance for handshake alg.\n         */\n        void set_priority_string(const sstring&);\n\n        /**\n         * Register a callback for receiving Distinguished Name (DN) information\n         * during the TLS handshake, extracted from the certificate as sent by the peer.\n         *\n         * The callback is not invoked in case the peer did not send a certificate.\n         * (This could e.g. happen when we are the server, and a client connects while\n         * client_auth is not set to REQUIRE.)\n         *\n         * If, based upon the extracted DN information, you want to abort the handshake,\n         * then simply throw an exception (e.g., from the callback) like verification_error.\n         *\n         * Registering this callback does not bypass the 'standard' certificate verification\n         * procedure; instead it merely extracts the DN information from the peer certificate\n         * (i.e., the 'leaf' certificate from the chain of certificates sent by the peer)\n         * and allows for extra checks.\n         *\n         * To keep the API simple, you can unregister the callback by means of registering\n         * an empty callback, i.e. dn_callback{}\n         *\n         * The callback prototype is documented in the dn_callback typedef.\n         */\n        void set_dn_verification_callback(dn_callback);\n\n        /**\n         * Optional override to disable certificate verification\n         */\n        void set_enable_certificate_verification(bool enable);\n\n    private:\n        class impl;\n        friend class session;\n        friend class server_session;\n        friend class server_credentials;\n        friend class credentials_builder;\n        template<typename Base>\n        friend class reloadable_credentials;\n        shared_ptr<impl> _impl;\n    };\n\n    /** Exception thrown on certificate validation error */\n    class verification_error : public std::runtime_error {\n    public:\n        using runtime_error::runtime_error;\n    };\n\n    enum class client_auth {\n        NONE, REQUEST, REQUIRE\n    };\n\n    /**\n     * Session resumption support.\n     * We only support TLS1.3 session tickets.\n    */\n    enum class session_resume_mode {\n        NONE, TLS13_SESSION_TICKET\n    };\n\n    /**\n     * Extending certificates and keys for server usage.\n     * More probably goes in here...\n     */\n    class server_credentials : public certificate_credentials {\n    public:\n        server_credentials();\n        server_credentials(shared_ptr<dh_params>);\n        server_credentials(const dh_params&);\n\n        server_credentials(server_credentials&&) noexcept;\n        server_credentials& operator=(server_credentials&&) noexcept;\n\n        server_credentials(const server_credentials&) = delete;\n        server_credentials& operator=(const server_credentials&) = delete;\n\n        void set_client_auth(client_auth);\n\n        /**\n         * Sets session resume mode.\n         * If session resumption is set to TLS13 session tickets,\n         * calling this also functions as key rotation, i.e. creates\n         * a new window of TLS session keys.\n        */\n        void set_session_resume_mode(session_resume_mode);\n\n        /**\n         * Sets Application-Layer Protocol Name (ALPN) supported by the server,\n         * in preference order.\n         */\n        void set_alpn_protocols(const std::vector<sstring>& protocols);\n    };\n\n    class reloadable_credentials_base;\n    class credentials_builder;\n\n    using reload_callback = std::function<void(const std::unordered_set<sstring>&, std::exception_ptr)>;\n    using reload_callback_ex = std::function<future<>(const credentials_builder&, const std::unordered_set<sstring>&, std::exception_ptr)>;\n\n    /**\n     * Intentionally \"primitive\", and more importantly, copyable\n     * container for certificate credentials options.\n     * The intendend use case is to be able to use across shards,\n     * at, say, initialization of tls objects\n     *\n     * Note that loading invalid objects (malformed certs etc) will\n     * _not_ generate exceptions until, earliest, the build functions\n     * are called.\n     */\n    class credentials_builder : public abstract_credentials {\n    public:\n        void set_dh_level(dh_params::level = dh_params::level::LEGACY);\n\n        void set_x509_trust(const blob&, x509_crt_format) override ;\n        void set_x509_crl(const blob&, x509_crt_format) override;\n        void set_x509_key(const blob& cert, const blob& key, x509_crt_format) override;\n        void set_simple_pkcs12(const blob&, x509_crt_format, const sstring& password) override;\n\n        future<> set_x509_trust_file(const sstring& cafile, x509_crt_format) override;\n        future<> set_x509_crl_file(const sstring& crlfile, x509_crt_format) override;\n        future<> set_x509_key_file(const sstring& cf, const sstring& kf, x509_crt_format) override;\n        future<> set_simple_pkcs12_file(const sstring& pkcs12file, x509_crt_format, const sstring& password) override;\n\n        future<> set_system_trust();\n        void set_client_auth(client_auth);\n        void set_priority_string(const sstring&);\n        /**\n         * Sets session resume mode to be applied to all created server credential sets\n         * Note: setting this will generate a session key that will be reused across all\n         * built server credentials, i.e. they will share resume key.\n         * If you wish to reuse a builder to create disparate server crendentials,\n         * simply call this method again to regenerate the key.\n         */\n        void set_session_resume_mode(session_resume_mode);\n\n        /**\n         * Sets Application-Layer Protocol Name (ALPN) supported by the server,\n         * in preference order.\n         */\n        void set_alpn_protocols(const std::vector<sstring>& protocols);\n\n        void apply_to(certificate_credentials&) const;\n\n        shared_ptr<certificate_credentials> build_certificate_credentials() const;\n        shared_ptr<server_credentials> build_server_credentials() const;\n\n        void rebuild(certificate_credentials&) const;\n        void rebuild(server_credentials&) const;\n\n        // same as above, but any files used for certs/keys etc will be watched\n        // for modification and reloaded if changed\n        future<shared_ptr<certificate_credentials>> build_reloadable_certificate_credentials(reload_callback_ex = {}, std::optional<std::chrono::milliseconds> tolerance = {}) const;\n        future<shared_ptr<server_credentials>> build_reloadable_server_credentials(reload_callback_ex = {}, std::optional<std::chrono::milliseconds> tolerance = {}) const;\n\n        future<shared_ptr<certificate_credentials>> build_reloadable_certificate_credentials(reload_callback, std::optional<std::chrono::milliseconds> tolerance = {}) const;\n        future<shared_ptr<server_credentials>> build_reloadable_server_credentials(reload_callback, std::optional<std::chrono::milliseconds> tolerance = {}) const;\n    private:\n        friend class reloadable_credentials_base;\n\n        std::multimap<std::string_view, std::any> _blobs;\n        client_auth _client_auth = client_auth::NONE;\n        session_resume_mode _session_resume_mode = session_resume_mode::NONE;\n        sstring _priority;\n        std::vector<uint8_t> _session_resume_key;\n        std::vector<sstring> _alpn_protocols;\n    };\n\n    using session_data = std::vector<uint8_t>;\n\n    /// TLS configuration options\n    struct tls_options {\n    private:\n        struct deprecated_wait_for_eof_on_shutdown {\n            bool _value = true;\n            deprecated_wait_for_eof_on_shutdown() noexcept : _value(true) {}\n            [[deprecated(\"Use tls::options::bye_timeout instead\")]]\n            deprecated_wait_for_eof_on_shutdown(bool value) noexcept : _value(value) {}\n            [[deprecated(\"Use tls::options::bye_timeout instead\")]]\n            void operator=(bool value) noexcept { _value = value; }\n            [[deprecated(\"Use tls::options::bye_timeout instead\")]]\n            operator bool() const noexcept { return _value; }\n        };\n\n    public:\n        /// \\brief whether to wait for EOF from server on session termination\n        deprecated_wait_for_eof_on_shutdown wait_for_eof_on_shutdown;\n        /// \\brief server name to be used for the SNI TLS extension\n        sstring server_name = {};\n\n        /// \\brief whether server certificate should be verified. May be set to false\n        /// in test environments.\n        bool verify_certificate = true;\n\n        /// \\brief Optional session resume data. Must be retrieved via\n        /// get_session_resume_data below.\n        session_data session_resume_data;\n\n        /// \\brief Optional list of ALPN protocols to offer to the server,\n        /// in order of preference.\n        std::vector<sstring> alpn_protocols;\n\n        // \\brief Time to wait for correct session wrap-up\n        // If set to zero, the TLS-level closing is not performed, the\n        // connection is just aborted\n        std::chrono::seconds bye_timeout = std::chrono::seconds(10);\n\n        // \\brief Whether or not to pick up any unread data during shutdown\n        // When false and any data arrives at the socket, the connection is\n        // immediately aborted without waiting for graceful wrap-up\n        bool wait_for_data_on_shutdown = false;\n    };\n\n    /**\n     * Creates a TLS client connection using the default network stack and the\n     * supplied credentials.\n     * Typically these should contain enough information\n     * to validate the remote certificate (i.e. trust info).\n     *\n     * ATTN: The method is going to be deprecated\n     *\n     * \\param name The expected server name for the remote end point\n     */\n    /// @{\n    [[deprecated(\"Use overload with tls_options parameter\")]]\n    future<connected_socket> connect(shared_ptr<certificate_credentials>, socket_address, sstring name);\n    [[deprecated(\"Use overload with tls_options parameter\")]]\n    future<connected_socket> connect(shared_ptr<certificate_credentials>, socket_address, socket_address local, sstring name);\n    /// @}\n\n    /**\n     * Creates a TLS client connection using the default network stack and the\n     * supplied credentials.\n     * Typically these should contain enough information\n     * to validate the remote certificate (i.e. trust info).\n     *\n     * \\param options Optional additional session configuration\n     */\n    /// @{\n    future<connected_socket> connect(shared_ptr<certificate_credentials>, socket_address, tls_options option = {});\n    future<connected_socket> connect(shared_ptr<certificate_credentials>, socket_address, socket_address local, tls_options options = {});\n    /// @}\n\n    /**\n     * Creates a socket through which a TLS client connection can be created,\n     * using the default network stack and the supplied credentials.\n     * Typically these should contain enough information\n     * to validate the remote certificate (i.e. trust info).\n     *\n     * ATTN: The method is going to be deprecated\n     *\n     * \\param name The expected server name for the remote end point\n     */\n    /// @{\n    [[deprecated(\"Use overload with tls_options parameter\")]]\n    ::seastar::socket socket(shared_ptr<certificate_credentials>, sstring name);\n    /// @}\n\n    /**\n     * Creates a socket through which a TLS client connection can be created,\n     * using the default network stack and the supplied credentials.\n     * Typically these should contain enough information\n     * to validate the remote certificate (i.e. trust info).\n     *\n     * \\param options Optional additional session configuration\n     */\n    /// @{\n    ::seastar::socket socket(shared_ptr<certificate_credentials>, tls_options options = {});\n    /// @}\n\n    /**\n     * Wraps an existing connection in SSL/TLS.\n     *\n     * ATTN: The method is going to be deprecated\n     *\n     * \\param name The expected server name for the remote end point\n     */\n    /// @{\n    [[deprecated(\"Use overload with tls_options parameter\")]]\n    future<connected_socket> wrap_client(shared_ptr<certificate_credentials>, connected_socket&&, sstring name);\n    future<connected_socket> wrap_server(shared_ptr<server_credentials>, connected_socket&&);\n    /// @}\n\n    /**\n     * Wraps an existing connection in SSL/TLS.\n     *\n     * \\param options Optional additional session configuration\n     */\n    /// @{\n    future<connected_socket> wrap_client(shared_ptr<certificate_credentials>, connected_socket&&, tls_options options = {});\n    /// @}\n\n    /**\n     * Creates a server socket that accepts SSL/TLS clients using default network stack\n     * and the supplied credentials.\n     * The credentials object should contain certificate info\n     * for the server and optionally trust/crl data.\n     */\n    /// @{\n    server_socket listen(shared_ptr<server_credentials>, socket_address sa, listen_options opts = listen_options());\n    // Wraps an existing server socket in SSL\n    server_socket listen(shared_ptr<server_credentials>, server_socket);\n    /// @}\n\n    /**\n     * Get distinguished name from the leaf certificate in the certificate chain that\n     * the connected peer is using.\n     * This function forces the TLS handshake. If the handshake didn't happen before the\n     * call to 'get_dn_information' it will be completed when the returned future will become\n     * ready.\n     * The function returns DN information on success. If the peer didn't send the certificate\n     * during the handshake the function returns nullopt. If the socket is not connected the\n     * system_error exception will be thrown.\n     */\n    future<std::optional<session_dn>> get_dn_information(connected_socket& socket);\n\n    /**\n     * Force a re-handshake (session key renegotiotion on TLS1.3).\n     * Can only be called on a server side socket.\n     * Mainly for testing purposes.\n     */\n    future<> force_rehandshake(connected_socket& socket);\n\n    /**\n     * Subject alt name types.\n    */\n    enum class subject_alt_name_type {\n        dnsname = 1, // string value representing a 'DNS' entry\n        rfc822name, // string value representing an 'email' entry\n        uri, // string value representing an 'uri' entry\n        ipaddress, // inet_address value representing an 'IP' entry\n        othername, // string value\n        dn, // string value\n    };\n\n    // Subject alt name entry\n    struct subject_alt_name {\n        using value_type = std::variant<\n            sstring,\n            net::inet_address\n        >;\n        subject_alt_name_type type;\n        value_type value;\n    };\n\n    /**\n     * Returns the alt name entries of matching types, or all entries if 'types' is empty\n     * The values are extracted from the client authentication certificate, if available.\n     * If no certificate authentication is used in the connection, en empty list is returned.\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n    */\n    future<std::vector<subject_alt_name>> get_alt_name_information(connected_socket& socket, std::unordered_set<subject_alt_name_type> types = {});\n\n    using certificate_data = std::vector<uint8_t>;\n\n    /**\n     * Get the raw certificate (chain) that the connected peer is using.\n     * This function forces the TLS handshake. If the handshake didn't happen before the\n     * call to 'get_peer_certificate_chain' it will be completed when the returned future\n     * will become ready.\n     * The function returns the certificate chain on success. If the peer didn't send the\n     * certificate during the handshake, the function returns an empty certificate chain.\n     * If the socket is not connected the system_error exception will be thrown.\n     */\n    future<std::vector<certificate_data>> get_peer_certificate_chain(connected_socket& socket);\n\n    /**\n     * Checks if the socket was connected using session resume.\n     * Will force handshake if not already done.\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n    */\n    future<bool> check_session_is_resumed(connected_socket& socket);\n\n    /**\n     * Get session resume data from a connected client socket. Will force handshake if not already done.\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n     * If no session resumption data is available, returns empty buffer.\n     *\n     * Note: TLS13 session tickets most of the time require data to have been transferred\n     * between client/server. To ensure getting the session data, it is advisable to\n     * delay this call to sometime before shutting down/closing the socket.\n    */\n    future<session_data> get_session_resume_data(connected_socket&);\n\n    /**\n     * Gets the Application-Layer Protocol Name (ALPN) selected during the TLS handshake.\n     * Will force handshake if not already done.\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n    */\n    future<std::optional<sstring>> get_selected_alpn_protocol(connected_socket&);\n\n    /**\n     * Returns the cipher suite used in the connection. This string depends on the internal implementation:\n     * it's e.g. \"TLS_AES_256_GCM_SHA384\" for gnutls and may be different for OpenSSL.\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n    */\n    future<sstring> get_cipher_suite(connected_socket& socket);\n\n    /**\n     * Returns the protocol version used in the connection, e.g. \"TLS1.3\"\n     *\n     * If the socket is not connected a system_error exception will be thrown.\n     * If the socket is not a TLS socket an exception will be thrown.\n    */\n    future<sstring> get_protocol_version(connected_socket& socket);\n\n    std::ostream& operator<<(std::ostream&, const subject_alt_name::value_type&);\n    std::ostream& operator<<(std::ostream&, const subject_alt_name&);\n\n    /**\n     * Alt name to string.\n     * Note: because naming of alternative names is inconsistent between tools,\n     * and because openssl is probably more popular when creating certs anyway,\n     * this routine will be inconsistent with both gnutls and openssl (though more\n     * in line with the latter) and name the constants as follows:\n     *\n     * dnsname: \"DNS\"\n     * rfc822name: \"EMAIL\"\n     * uri: \"URI\"\n     * ipaddress \"IP\"\n     * othername: \"OTHERNAME\"\n     * dn: \"DIRNAME\"\n    */\n    std::string_view format_as(subject_alt_name_type);\n    std::ostream& operator<<(std::ostream&, subject_alt_name_type);\n\n    /**\n     * Error handling.\n     *\n     * The error_category instance used by exceptions thrown by TLS\n     */\n    const std::error_category& error_category();\n\n    /**\n     * The more common error codes encountered in TLS.\n     * Not an exhaustive list. Add exports as needed.\n     */\n    extern const int ERROR_UNKNOWN_COMPRESSION_ALGORITHM;\n    extern const int ERROR_UNKNOWN_CIPHER_TYPE;\n    extern const int ERROR_INVALID_SESSION;\n    extern const int ERROR_UNEXPECTED_HANDSHAKE_PACKET;\n    extern const int ERROR_UNKNOWN_CIPHER_SUITE;\n    extern const int ERROR_UNKNOWN_ALGORITHM;\n    extern const int ERROR_UNSUPPORTED_SIGNATURE_ALGORITHM;\n    extern const int ERROR_SAFE_RENEGOTIATION_FAILED;\n    extern const int ERROR_UNSAFE_RENEGOTIATION_DENIED;\n    extern const int ERROR_UNKNOWN_SRP_USERNAME;\n    extern const int ERROR_PREMATURE_TERMINATION;\n    extern const int ERROR_PUSH;\n    extern const int ERROR_PULL;\n    extern const int ERROR_UNEXPECTED_PACKET;\n    extern const int ERROR_UNSUPPORTED_VERSION;\n    extern const int ERROR_NO_CIPHER_SUITES;\n    extern const int ERROR_DECRYPTION_FAILED;\n    extern const int ERROR_MAC_VERIFY_FAILED;\n}\n}\n\ntemplate <> struct fmt::formatter<seastar::tls::subject_alt_name_type> : fmt::formatter<string_view> {\n    template <typename FormatContext>\n    auto format(seastar::tls::subject_alt_name_type type, FormatContext& ctx) const {\n        return formatter<string_view>::format(format_as(type), ctx);\n    }\n};\n\ntemplate <> struct fmt::formatter<seastar::tls::subject_alt_name::value_type> : fmt::formatter<string_view> {\n    template <typename FormatContext>\n    auto format(const seastar::tls::subject_alt_name::value_type& value, FormatContext& ctx) const {\n        return std::visit([&](const auto& v) {\n            return fmt::format_to(ctx.out(), \"{}\", v);\n        }, value);\n    }\n};\n\ntemplate <> struct fmt::formatter<seastar::tls::subject_alt_name> : fmt::formatter<string_view> {\n    template <typename FormatContext>\n    auto format(const seastar::tls::subject_alt_name& name, FormatContext& ctx) const {\n        return fmt::format_to(ctx.out(), \"{}={}\", name.type, name.value);\n    }\n};\n"
  },
  {
    "path": "include/seastar/net/toeplitz.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*-\n * Copyright (c) 2010 David Malone <dwmalone@FreeBSD.org>\n * All rights reserved.\n *\n * Redistribution and use in source and binary forms, with or without\n * modification, are permitted provided that the following conditions\n * are met:\n * 1. Redistributions of source code must retain the above copyright\n *    notice, this list of conditions and the following disclaimer.\n * 2. Redistributions in binary form must reproduce the above copyright\n *    notice, this list of conditions and the following disclaimer in the\n *    documentation and/or other materials provided with the distribution.\n *\n * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE\n * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\n * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n * SUCH DAMAGE.\n */\n\n#pragma once\n\n#include <cstdint>\n#include <span>\n#include <vector>\n\nnamespace seastar {\n\nusing rss_key_type = std::span<const uint8_t>;\n\n// Mellanox Linux's driver key\nstatic constexpr uint8_t default_rsskey_40bytes_v[] = {\n    0xd1, 0x81, 0xc6, 0x2c, 0xf7, 0xf4, 0xdb, 0x5b,\n    0x19, 0x83, 0xa2, 0xfc, 0x94, 0x3e, 0x1a, 0xdb,\n    0xd9, 0x38, 0x9e, 0x6b, 0xd1, 0x03, 0x9c, 0x2c,\n    0xa7, 0x44, 0x99, 0xad, 0x59, 0x3d, 0x56, 0xd9,\n    0xf3, 0x25, 0x3c, 0x06, 0x2a, 0xdc, 0x1f, 0xfc\n};\n\nstatic constexpr rss_key_type default_rsskey_40bytes{default_rsskey_40bytes_v, sizeof(default_rsskey_40bytes_v)};\n\n// Intel's i40e PMD default RSS key\nstatic constexpr uint8_t default_rsskey_52bytes_v[] = {\n    0x44, 0x39, 0x79, 0x6b, 0xb5, 0x4c, 0x50, 0x23,\n    0xb6, 0x75, 0xea, 0x5b, 0x12, 0x4f, 0x9f, 0x30,\n    0xb8, 0xa2, 0xc0, 0x3d, 0xdf, 0xdc, 0x4d, 0x02,\n    0xa0, 0x8c, 0x9b, 0x33, 0x4a, 0xf6, 0x4a, 0x4c,\n    0x05, 0xc6, 0xfa, 0x34, 0x39, 0x58, 0xd8, 0x55,\n    0x7d, 0x99, 0x58, 0x3a, 0xe1, 0x38, 0xc9, 0x2e,\n    0x81, 0x15, 0x03, 0x66\n};\n\nstatic constexpr rss_key_type default_rsskey_52bytes{default_rsskey_52bytes_v, sizeof(default_rsskey_52bytes_v)};\n\ntemplate<typename T>\ninline uint32_t\ntoeplitz_hash(rss_key_type key, const T& data)\n{\n\tuint32_t hash = 0, v;\n\n\t/* XXXRW: Perhaps an assertion about key length vs. data length? */\n\n\tv = (key[0]<<24) + (key[1]<<16) + (key[2] <<8) + key[3];\n\tfor (unsigned i = 0; i < data.size(); i++) {\n\t\tfor (unsigned b = 0; b < 8; b++) {\n\t\t\tif (data[i] & (1<<(7-b)))\n\t\t\t\thash ^= v;\n\t\t\tv <<= 1;\n\t\t\tif ((i + 4) < key.size() &&\n\t\t\t    (key[i+4] & (1<<(7-b))))\n\t\t\t\tv |= 1;\n\t\t}\n\t}\n\treturn (hash);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/udp.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#pragma once\n\n#include <unordered_map>\n#include <assert.h>\n\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/const.hh>\n#include <seastar/net/net.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nstruct udp_hdr {\n    packed<uint16_t> src_port;\n    packed<uint16_t> dst_port;\n    packed<uint16_t> len;\n    packed<uint16_t> cksum;\n\n    template<typename Adjuster>\n    auto adjust_endianness(Adjuster a) {\n        return a(src_port, dst_port, len, cksum);\n    }\n} __attribute__((packed));\n\nstruct udp_channel_state {\n    queue<datagram> _queue;\n    // Limit number of data queued into send queue\n    semaphore _user_queue_space = {212992};\n    udp_channel_state(size_t queue_size) : _queue(queue_size) {}\n    future<> wait_for_send_buffer(size_t len) { return _user_queue_space.wait(len); }\n    void complete_send(size_t len) { _user_queue_space.signal(len); }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/net/unix_address.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Red Hat, Inc.\n */\n#pragma once\n\n#include <iosfwd>\n#include <sys/types.h>\n#include <sys/un.h>\n#include <string>\n\nnamespace seastar {\n/*!\n    A helper struct for creating/manipulating UNIX-domain sockets.\n\n    A UNIX-domain socket is either named or unnamed. If named, the name is either\n    a path in the filesystem namespace, or an abstract-domain identifier. Abstract-domain\n    names start with a null byte, and may contain non-printable characters.\n\n    std::string() can hold a sequence of arbitrary bytes, and has a length() attribute\n    that does not rely on using strlen(). Thus it is used here to hold the address.\n */\nstruct unix_domain_addr {\n    const std::string name;\n    const int path_count;  //  either name.length() or name.length()+1. See path_length_aux() below.\n\n    explicit unix_domain_addr(const std::string& fn) : name{fn}, path_count{path_length_aux()} {}\n\n    explicit unix_domain_addr(const char* fn) : name{fn}, path_count{path_length_aux()} {}\n\n    int path_length() const { return path_count; }\n\n    //  the following holds:\n    //  for abstract name: name.length() == number of meaningful bytes, including the null in name[0].\n    //  for filesystem path: name.length() does not count the implicit terminating null.\n    //  Here we tweak the outside-visible length of the address.\n    int path_length_aux() const {\n        auto pl = (int)name.length();\n        if (!pl || (name[0] == '\\0')) {\n            // unnamed, or abstract-namespace\n            return pl;\n        }\n        return 1 + pl;\n    }\n\n    const char* path_bytes() const { return name.c_str(); }\n\n    bool operator==(const unix_domain_addr& a) const {\n        return name == a.name;\n    }\n    bool operator!=(const unix_domain_addr& a) const {\n        return !(*this == a);\n    }\n};\n\nstd::ostream& operator<<(std::ostream&, const unix_domain_addr&);\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/net/virtio-interface.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#pragma once\n\n#ifndef _LINUX_VIRTIO_RING_H\n#define _LINUX_VIRTIO_RING_H\n\n/* An interface for efficient virtio implementation, currently for use by KVM\n * and lguest, but hopefully others soon.  Do NOT change this since it will\n * break existing servers and clients.\n *\n * This header is BSD licensed so anyone can use the definitions to implement\n * compatible drivers/servers.\n *\n * Redistribution and use in source and binary forms, with or without\n * modification, are permitted provided that the following conditions\n * are met:\n * 1. Redistributions of source code must retain the above copyright\n *    notice, this list of conditions and the following disclaimer.\n * 2. Redistributions in binary form must reproduce the above copyright\n *    notice, this list of conditions and the following disclaimer in the\n *    documentation and/or other materials provided with the distribution.\n * 3. Neither the name of IBM nor the names of its contributors\n *    may be used to endorse or promote products derived from this software\n *    without specific prior written permission.\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n * ARE DISCLAIMED.  IN NO EVENT SHALL IBM OR CONTRIBUTORS BE LIABLE\n * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\n * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n * SUCH DAMAGE.\n *\n * Copyright Rusty Russell IBM Corporation 2007. */\n\n/* This marks a buffer as continuing via the next field. */\n#define VRING_DESC_F_NEXT       1\n/* This marks a buffer as write-only (otherwise read-only). */\n#define VRING_DESC_F_WRITE      2\n/* This means the buffer contains a list of buffer descriptors. */\n#define VRING_DESC_F_INDIRECT   4\n\n/* The Host uses this in used->flags to advise the Guest: don't kick me when\n * you add a buffer.  It's unreliable, so it's simply an optimization.  Guest\n * will still kick if it's out of buffers. */\n#define VRING_USED_F_NO_NOTIFY  1\n/* The Guest uses this in avail->flags to advise the Host: don't interrupt me\n * when you consume a buffer.  It's unreliable, so it's simply an\n * optimization.  */\n#define VRING_AVAIL_F_NO_INTERRUPT      1\n\n/* We support indirect buffer descriptors */\n#define VIRTIO_RING_F_INDIRECT_DESC     (1 << 28)\n\n/* The Guest publishes the used index for which it expects an interrupt\n * at the end of the avail ring. Host should ignore the avail->flags field. */\n/* The Host publishes the avail index for which it expects a kick\n * at the end of the used ring. Guest should ignore the used->flags field. */\n#define VIRTIO_RING_F_EVENT_IDX         (1 << 29)\n\n/* The standard layout for the ring is a continuous chunk of memory which looks\n * like this.  We assume num is a power of 2.\n *\n * struct vring\n * {\n *      // The actual descriptors (16 bytes each)\n *      struct vring_desc desc[num];\n *\n *      // A ring of available descriptor heads with free-running index.\n *      uint16_t avail_flags;\n *      uint16_t avail_idx;\n *      uint16_t available[num];\n *      uint16_t used_event_idx;\n *\n *      // Padding to the next align boundary.\n *      char pad[];\n *\n *      // A ring of used descriptor heads with free-running index.\n *      uint16_t used_flags;\n *      uint16_t used_idx;\n *      struct vring_used_elem used[num];\n *      uint16_t avail_event_idx;\n * };\n */\n\n#endif\n\n#define VIRTIO_NET_F_CSUM (1 << 0)\n#define VIRTIO_NET_F_GUEST_CSUM (1 << 1)\n#define VIRTIO_NET_F_CTRL_GUEST_OFFLOADS (1 << 2)\n#define VIRTIO_NET_F_MAC (1 << 5)\n#define VIRTIO_NET_F_GUEST_TSO4 (1 << 7)\n#define VIRTIO_NET_F_GUEST_TSO6 (1 << 8)\n#define VIRTIO_NET_F_GUEST_ECN (1 << 9)\n#define VIRTIO_NET_F_GUEST_UFO (1 << 10)\n#define VIRTIO_NET_F_HOST_TSO4 (1 << 11)\n#define VIRTIO_NET_F_HOST_TSO6 (1 << 12)\n#define VIRTIO_NET_F_HOST_ECN (1 << 13)\n#define VIRTIO_NET_F_HOST_UFO (1 << 14)\n#define VIRTIO_NET_F_MRG_RXBUF (1 << 15)\n#define VIRTIO_NET_F_STATUS (1 << 16)\n#define VIRTIO_NET_F_CTRL_VQ (1 << 17)\n#define VIRTIO_NET_F_CTRL_RX (1 << 18)\n#define VIRTIO_NET_F_CTRL_VLAN (1 << 19)\n#define VIRTIO_NET_F_GUEST_ANNOUNCE (1 << 21)\n#define VIRTIO_NET_F_MQ (1 << 22)\n#define VIRTIO_NET_F_CTRL_MAC_ADDR (1 << 23)\n"
  },
  {
    "path": "include/seastar/net/virtio.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <memory>\n#include <seastar/net/net.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/program-options.hh>\n\nnamespace seastar {\n\nnamespace net {\n\n/// Virtio configuration.\nstruct virtio_options : public program_options::option_group {\n    /// \\brief Enable event-index feature (on / off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> event_index;\n    /// \\brief Enable checksum offload feature (on / off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> csum_offload;\n    /// \\brief Enable TCP segment offload feature (on / off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> tso;\n    /// \\brief Enable UDP fragmentation offload feature (on / off).\n    ///\n    /// Default: \\p on.\n    program_options::value<std::string> ufo;\n    /// \\brief Virtio ring size (must be power-of-two).\n    ///\n    /// Default: 256.\n    program_options::value<unsigned> virtio_ring_size;\n\n    /// \\cond internal\n    virtio_options(program_options::option_group* parent_group);\n    /// \\endcond\n};\n\n}\n\n/// \\cond internal\nstd::unique_ptr<net::device> create_virtio_net_device(const net::virtio_options& opts, const program_options::value<std::string>& lro);\n/// \\endcond\n\n}\n"
  },
  {
    "path": "include/seastar/rpc/lz4_compressor.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/rpc/rpc_types.hh>\n\nnamespace seastar {\n\nnamespace rpc {\n    class lz4_compressor : public compressor {\n    public:\n        class factory: public rpc::compressor::factory {\n        public:\n            virtual const sstring& supported() const override;\n            virtual std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server) const override;\n        };\n    public:\n        ~lz4_compressor() {}\n        // compress data, leaving head_space empty in returned buffer\n        snd_buf compress(size_t head_space, snd_buf data) override;\n        // decompress data\n        rcv_buf decompress(rcv_buf data) override;\n        sstring name() const override;\n    };\n}\n\n}\n"
  },
  {
    "path": "include/seastar/rpc/lz4_fragmented_compressor.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/rpc/rpc_types.hh>\n\nnamespace seastar {\nnamespace rpc {\n\nclass lz4_fragmented_compressor final : public compressor {\npublic:\n    class factory final : public rpc::compressor::factory {\n    public:\n        virtual const sstring& supported() const override;\n        virtual std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server) const override;\n    };\npublic:\n    virtual snd_buf compress(size_t head_space, snd_buf data) override;\n    virtual rcv_buf decompress(rcv_buf data) override;\n    sstring name() const override;\n};\n\n}\n}\n"
  },
  {
    "path": "include/seastar/rpc/multi_algo_compressor_factory.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylladb, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/rpc/rpc_types.hh>\n\nnamespace seastar {\n\nnamespace rpc {\n\n// This is meta compressor factory. It gets an array of regular factories that\n// support one compression algorithm each and negotiates common compression algorithm\n// that is supported both by a client and a server. The order of algorithm preferences\n// is the order they appear in clien's list\nclass multi_algo_compressor_factory : public rpc::compressor::factory {\n    std::vector<const rpc::compressor::factory*> _factories;\n    sstring _features;\n\npublic:\n    multi_algo_compressor_factory(std::vector<const rpc::compressor::factory*> factories);\n    multi_algo_compressor_factory(std::initializer_list<const rpc::compressor::factory*> factories) :\n        multi_algo_compressor_factory(std::vector<const rpc::compressor::factory*>(std::move(factories))) {}\n    multi_algo_compressor_factory(const rpc::compressor::factory* factory) : multi_algo_compressor_factory({factory}) {}\n    // return feature string that will be sent as part of protocol negotiation\n    const sstring& supported() const override {\n        return _features;\n    }\n    // negotiate compress algorithm\n    std::unique_ptr<compressor> negotiate(sstring feature, bool is_server) const override {\n        return negotiate(feature, is_server, nullptr);\n    }\n    std::unique_ptr<compressor> negotiate(sstring feature, bool is_server, std::function<future<>()> send_empty_frame) const override;\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/rpc/rpc.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <map>\n#include <unordered_map>\n#include <unordered_set>\n#include <list>\n#include <variant>\n#include <boost/intrusive/list.hpp>\n#include <seastar/core/future.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/net/api.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/rpc/rpc_types.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/shared_future.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/weak_ptr.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/deleter.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/log.hh>\n\nnamespace bi = boost::intrusive;\n\nnamespace seastar {\n\nnamespace rpc {\n\n/// \\defgroup rpc rpc - remote procedure call framework\n///\n/// \\brief\n/// rpc is a framework that can be used to define client-server communication\n/// protocols.\n/// For a high-level description of the RPC features see\n/// [doc/rpc.md](./md_rpc.html),\n/// [doc/rpc-streaming.md](./md_rpc-streaming.html) and\n/// [doc/rpc-compression.md](./md_rpc-compression.html)\n///\n/// The entry point for setting up an rpc protocol is\n/// seastar::rpc::protocol.\n\nusing id_type = int64_t;\n\nusing rpc_semaphore = basic_semaphore<semaphore_default_exception_factory, rpc_clock_type>;\nusing resource_permit = semaphore_units<semaphore_default_exception_factory, rpc_clock_type>;\n\nstatic constexpr char rpc_magic[] = \"SSTARRPC\";\n\n/// \\addtogroup rpc\n/// @{\n\n/// Specifies resource isolation for a connection.\nstruct isolation_config {\n    /// Specifies a scheduling group under which the connection (and all its\n    /// verb handlers) will execute.\n    scheduling_group sched_group = current_scheduling_group();\n};\n\n/// Default isolation configuration - run everything in the default scheduling group.\n///\n/// In the scheduling_group that the protocol::server was created in.\nisolation_config default_isolate_connection(sstring isolation_cookie);\n\n/// \\brief Resource limits for an RPC server\n///\n/// A request's memory use will be estimated as\n///\n///     req_mem = basic_request_size * sizeof(serialized_request) * bloat_factor\n///\n/// Concurrent requests will be limited so that\n///\n///     sum(req_mem) <= max_memory\n///\n/// \\see server\nstruct resource_limits {\n    size_t basic_request_size = 0; ///< Minimum request footprint in memory\n    unsigned bloat_factor = 1;     ///< Serialized size multiplied by this to estimate memory used by request\n    size_t max_memory = rpc_semaphore::max_counter(); ///< Maximum amount of memory that may be consumed by all requests\n    /// Configures isolation for a connection based on its isolation cookie. May throw,\n    /// in which case the connection will be terminated.\n    using syncronous_isolation_function = std::function<isolation_config (sstring isolation_cookie)>;\n    using asyncronous_isolation_function = std::function<future<isolation_config> (sstring isolation_cookie)>;\n    using isolation_function_alternatives = std::variant<syncronous_isolation_function, asyncronous_isolation_function>;\n    isolation_function_alternatives isolate_connection = default_isolate_connection;\n};\n\nstruct client_options {\n    std::optional<net::tcp_keepalive_params> keepalive;\n    bool tcp_nodelay = true;\n    bool reuseaddr = false;\n    compressor::factory* compressor_factory = nullptr;\n    bool send_timeout_data = true;\n    connection_id stream_parent = invalid_connection_id;\n    /// Configures how this connection is isolated from other connection on the same server.\n    ///\n    /// \\see resource_limits::isolate_connection\n    sstring isolation_cookie;\n    sstring metrics_domain = \"default\";\n    bool send_handler_duration = true;\n};\n\n/// @}\n\n// RPC call that passes stream connection id as a parameter\n// may arrive to a different shard from where the stream connection\n// was opened, so the connection id is not known to a server that handles\n// the RPC call. The shard that the stream connection belong to is know\n// since it is a part of connection id, but this is not enough to locate\n// a server instance the connection belongs to if there are more than one\n// server on the shard. Stream domain parameter is here to help with that.\n// Different servers on all shards logically belonging to the same service should\n// belong to the same streaming domain. Only one server on each shard can belong to\n// a particulr streaming domain.\nclass streaming_domain_type {\n    uint64_t _id;\npublic:\n    explicit streaming_domain_type(uint64_t id) : _id(id) {}\n    bool operator==(const streaming_domain_type& o) const {\n        return _id == o._id;\n    }\n    friend struct std::hash<streaming_domain_type>;\n    friend std::ostream& operator<<(std::ostream&, const streaming_domain_type&);\n};\n\n/// \\addtogroup rpc\n/// @{\n\nclass server;\n\nstruct server_options {\n    compressor::factory* compressor_factory = nullptr;\n    bool tcp_nodelay = true;\n    std::optional<streaming_domain_type> streaming_domain;\n    server_socket::load_balancing_algorithm load_balancing_algorithm = server_socket::load_balancing_algorithm::default_;\n    // optional filter function. If set, will be called with remote\n    // (connecting) address.\n    // Returning false will refuse the incoming connection.\n    // Returning true will allow the mechanism to proceed.\n    std::function<bool(const socket_address&)> filter_connection = {};\n};\n\n/// @}\n\ninline\nsize_t\nestimate_request_size(const resource_limits& lim, size_t serialized_size) {\n    return lim.basic_request_size + serialized_size * lim.bloat_factor;\n}\n\nstruct negotiation_frame {\n    char magic[sizeof(rpc_magic) - 1];\n    uint32_t len; // additional negotiation data length; multiple negotiation_frame_feature_record structs\n};\n\nenum class protocol_features : uint32_t {\n    COMPRESS = 0,\n    TIMEOUT = 1,\n    CONNECTION_ID = 2,\n    STREAM_PARENT = 3,\n    ISOLATION = 4,\n    HANDLER_DURATION = 5,\n};\n\n// internal representation of feature data\nusing feature_map = std::map<protocol_features, sstring>;\n\n// An rpc signature, in the form signature<Ret (In0, In1, In2)>.\ntemplate <typename Function>\nstruct signature;\n\nclass logger {\n    ::seastar::logger* _seastar_logger = nullptr;\n\n    void log(const sstring& str) const {\n        if (_seastar_logger) {\n            // default level for log messages is `info`\n            _seastar_logger->info(\"{}\", str);\n        }\n    }\n\n    // _seastar_logger will always be used first if it's available\n    template <typename... Args>\n#ifdef SEASTAR_LOGGER_COMPILE_TIME_FMT\n    void log(log_level level, fmt::format_string<Args...> fmt, Args&&... args) const {\n#else\n    void log(log_level level, const char* fmt, Args&&... args) const {\n#endif\n        if (_seastar_logger) {\n            _seastar_logger->log(level, fmt, std::forward<Args>(args)...);\n        }\n    }\n\npublic:\n    void set(::seastar::logger* logger) {\n        _seastar_logger = logger;\n    }\n\n    void operator()(const client_info& info, id_type msg_id, const sstring& str) const;\n    void operator()(const client_info& info, id_type msg_id, log_level level, std::string_view str) const;\n\n    void operator()(const client_info& info, const sstring& str) const;\n    void operator()(const client_info& info, log_level level, std::string_view str) const;\n\n    void operator()(const socket_address& addr, const sstring& str) const;\n    void operator()(const socket_address& addr, log_level level, std::string_view str) const;\n};\n\nnamespace internal {\ntemplate<typename Serializer, typename... Out>\nclass sink_impl;\n\ntemplate<typename Serializer, typename... In>\nclass source_impl;\n}\n\nclass connection {\nprotected:\n    struct socket_and_buffers {\n        connected_socket fd;\n        input_stream<char> read_buf;\n        output_stream<char> write_buf;\n        socket_and_buffers(connected_socket cs) noexcept\n                : fd(std::move(cs))\n                , read_buf(fd.input())\n                , write_buf(fd.output())\n        {}\n    };\n    bool _error = false;\n    std::optional<socket_and_buffers> _connected;\n    std::optional<shared_promise<>> _negotiated = shared_promise<>();\n    promise<> _stopped;\n    stats _stats;\n    const logger& _logger;\n    // The owner of the pointer below is an instance of rpc::protocol<typename Serializer> class.\n    // The type of the pointer is erased here, but the original type is Serializer\n    void* _serializer;\n    struct outgoing_entry : public bi::list_base_hook<bi::link_mode<bi::auto_unlink>> {\n        timer<rpc_clock_type> t;\n        snd_buf buf;\n        promise<> done;\n        cancellable* pcancel = nullptr;\n        outgoing_entry(snd_buf b) : buf(std::move(b)) {}\n\n        outgoing_entry(outgoing_entry&&) = delete;\n        outgoing_entry(const outgoing_entry&) = delete;\n\n        void uncancellable() {\n            t.cancel();\n            if (pcancel) {\n                pcancel->cancel_send = std::function<void()>();\n            }\n        }\n\n        ~outgoing_entry() {\n            if (pcancel) {\n                pcancel->cancel_send = std::function<void()>();\n                pcancel->send_back_pointer = nullptr;\n            }\n        }\n\n        using container_t = bi::list<outgoing_entry, bi::constant_time_size<false>>;\n    };\n    void withdraw(outgoing_entry::container_t::iterator it, std::exception_ptr ex = nullptr);\n    future<> _outgoing_queue_ready = _negotiated->get_shared_future();\n    outgoing_entry::container_t _outgoing_queue;\n    size_t _outgoing_queue_size = 0;\n    std::unique_ptr<compressor> _compressor;\n    bool _propagate_timeout = false;\n    bool _timeout_negotiated = false;\n    bool _handler_duration_negotiated = false;\n    // stream related fields\n    bool _is_stream = false;\n    connection_id _id = invalid_connection_id;\n\n    std::unordered_map<connection_id, xshard_connection_ptr> _streams;\n    queue<rcv_buf> _stream_queue = queue<rcv_buf>(max_queued_stream_buffers);\n    semaphore _stream_sem = semaphore(max_stream_buffers_memory);\n    bool _sink_closed = true;\n    bool _source_closed = true;\n    // the future holds if sink is already closed\n    // if it is not ready it means the sink is been closed\n    future<bool> _sink_closed_future = make_ready_future<bool>(false);\n\n    void set_negotiated() noexcept;\n\n    bool is_stream() const noexcept {\n        return _is_stream;\n    }\n\n    snd_buf compress(snd_buf buf);\n    future<> send_buffer(snd_buf buf);\n    future<> send(snd_buf buf, std::optional<rpc_clock_type::time_point> timeout = {}, cancellable* cancel = nullptr);\n    future<> send_entry(outgoing_entry& d) noexcept;\n    future<> stop_send_loop(std::exception_ptr ex);\n    future<std::optional<rcv_buf>>  read_stream_frame_compressed(input_stream<char>& in);\n    bool stream_check_twoway_closed() const noexcept {\n        return _sink_closed && _source_closed;\n    }\n    future<> stream_close();\n    future<> stream_process_incoming(rcv_buf&&);\n    future<> handle_stream_frame();\n    future<> send_negotiation_frame(feature_map features);\n\npublic:\n    connection(connected_socket&& fd, const logger& l, void* s, connection_id id = invalid_connection_id) : connection(l, s, id) {\n        set_socket(std::move(fd));\n    }\n    connection(const logger& l, void* s, connection_id id = invalid_connection_id) : _logger(l), _serializer(s), _id(id) {}\n    virtual ~connection() {}\n    size_t outgoing_queue_length() const noexcept {\n        return _outgoing_queue_size;\n    }\n\n    void set_socket(connected_socket&& fd);\n    bool error() const noexcept { return _error; }\n    void abort();\n    future<> stop() noexcept;\n\nprivate:\n    future<> stream_receive(circular_buffer<foreign_ptr<std::unique_ptr<rcv_buf>>>& bufs);\n    future<> close_sink() {\n        _sink_closed = true;\n        if (stream_check_twoway_closed()) {\n            return stream_close();\n        }\n        return make_ready_future();\n    }\n    bool sink_closed() const noexcept {\n        return _sink_closed;\n    }\n    future<> close_source() {\n        _source_closed = true;\n        if (stream_check_twoway_closed()) {\n            return stream_close();\n        }\n        return make_ready_future();\n    }\n\npublic:\n    connection_id get_connection_id() const noexcept {\n        return _id;\n    }\n    stats& get_stats_internal() noexcept {\n        return _stats;\n    }\n    xshard_connection_ptr get_stream(connection_id id) const;\n    void register_stream(connection_id id, xshard_connection_ptr c);\n    virtual socket_address peer_address() const = 0;\n\n    const logger& get_logger() const noexcept {\n        return _logger;\n    }\n\n    template<typename Serializer>\n    Serializer& serializer() {\n        return *static_cast<Serializer*>(_serializer);\n    }\n\n    template <typename FrameType>\n    future<typename FrameType::return_type> read_frame(socket_address info, input_stream<char>& in);\n\n    template <typename FrameType>\n    future<typename FrameType::return_type> read_frame_compressed(socket_address info, std::unique_ptr<compressor>& compressor, input_stream<char>& in);\n    friend class client;\n    template<typename Serializer, typename... Out>\n    friend class internal::sink_impl;\n    template<typename Serializer, typename... In>\n    friend class internal::source_impl;\n\n    void suspend_for_testing(promise<>& p) {\n        _outgoing_queue_ready.get();\n        auto dummy = std::make_unique<outgoing_entry>(snd_buf());\n        _outgoing_queue.push_back(*dummy);\n        _outgoing_queue_ready = dummy->done.get_future();\n        (void)p.get_future().then([dummy = std::move(dummy)] { dummy->done.set_value(); });\n    }\n};\n\nnamespace internal {\n\ntemplate <typename T>\nrequires std::is_base_of_v<bi::slist_base_hook<>, T>\nclass batched_queue {\n    using list_type = boost::intrusive::slist<T, boost::intrusive::cache_last<true>, boost::intrusive::constant_time_size<false>>;\n\n    std::function<future<>(T*)> _process_func;\n    shard_id _processing_shard;\n    list_type _queue;\n    list_type _cur_batch;\n    future<> _process_fut = make_ready_future();\n\npublic:\n    batched_queue(std::function<future<>(T*)> process_func, shard_id processing_shard)\n        : _process_func(std::move(process_func))\n        , _processing_shard(processing_shard)\n    {}\n\n    ~batched_queue() {\n        assert(_process_fut.available());\n    }\n\n    future<> stop() noexcept {\n        return std::exchange(_process_fut, make_ready_future());\n    }\n\n    void enqueue(T* buf) noexcept {\n        _queue.push_back(*buf);\n        if (_process_fut.available()) {\n            _process_fut = process_loop();\n        }\n    }\n\n    future<> process_loop() {\n        return seastar::do_until([this] { return _queue.empty(); }, [this] {\n            _cur_batch = std::exchange(_queue, list_type());\n            return smp::submit_to(_processing_shard, [this] {\n                return seastar::do_until([this] { return _cur_batch.empty(); }, [this] {\n                    auto* buf = &_cur_batch.front();\n                    _cur_batch.pop_front();\n                    return _process_func(buf);\n                });\n            });\n        });\n    }\n};\n\n// Safely delete the original allocation buffer on the local shard\n// When deleted after it was sent on the remote shard, we queue\n// up the buffer pointers to be destroyed and deleted as a batch\n// back on the local shard.\nclass snd_buf_deleter_impl final : public deleter::impl {\n    snd_buf* _obj_ptr;\n    batched_queue<snd_buf>& _delete_queue;\n\npublic:\n    snd_buf_deleter_impl(snd_buf* obj_ptr, batched_queue<snd_buf>& delete_queue)\n        : impl(deleter())\n        , _obj_ptr(obj_ptr)\n        , _delete_queue(delete_queue)\n    {}\n\n    virtual ~snd_buf_deleter_impl() override {\n        _delete_queue.enqueue(_obj_ptr);\n    }\n};\n\n// send data Out...\ntemplate<typename Serializer, typename... Out>\nclass sink_impl : public sink<Out...>::impl {\n    batched_queue<snd_buf> _send_queue;\n    batched_queue<snd_buf> _delete_queue;\n\npublic:\n    sink_impl(xshard_connection_ptr con);\n    future<> operator()(const Out&... args) override;\n    future<> close() noexcept override;\n    future<> flush() noexcept override;\n    ~sink_impl() override;\n\nprivate:\n    // Runs on connection shard\n    future<> send_buffer(snd_buf* buf);\n};\n\n// receive data In...\ntemplate<typename Serializer, typename... In>\nclass source_impl : public source<In...>::impl {\npublic:\n    source_impl(xshard_connection_ptr con) : source<In...>::impl(std::move(con)) { this->_con->get()->_source_closed = false; }\n    future<std::optional<std::tuple<In...>>> operator()() override;\n};\n\n} // namespace internal\n\nclass client : public rpc::connection, public weakly_referencable<client> {\n    socket _socket;\n    id_type _message_id = 1;\n    struct reply_handler_base {\n        timer<rpc_clock_type> t;\n        cancellable* pcancel = nullptr;\n        rpc_clock_type::time_point start;\n        virtual void operator()(client&, id_type, rcv_buf data) = 0;\n        virtual void timeout() {}\n        virtual void cancel() {}\n        virtual ~reply_handler_base() {\n            if (pcancel) {\n                pcancel->cancel_wait = std::function<void()>();\n                pcancel->wait_back_pointer = nullptr;\n            }\n        };\n    };\n\n    class metrics {\n        struct domain;\n\n        using domain_member_hook_t = boost::intrusive::list_member_hook<boost::intrusive::link_mode<boost::intrusive::auto_unlink>>;\n\n        const client& _c;\n        domain& _domain;\n        domain_member_hook_t _link;\n\n        using domain_list_t = boost::intrusive::list<metrics,\n              boost::intrusive::member_hook<metrics, metrics::domain_member_hook_t, &metrics::_link>,\n              boost::intrusive::constant_time_size<false>>;\n\n    public:\n        metrics(const client&);\n        ~metrics();\n    };\n\n    void enqueue_zero_frame();\n    future<> loop(client_options ops, const socket_address& addr, const socket_address& local);\npublic:\n    template<typename Reply, typename Func>\n    struct reply_handler final : reply_handler_base {\n        Func func;\n        Reply reply;\n        reply_handler(Func&& f) : func(std::move(f)) {}\n        virtual void operator()(client& client, id_type msg_id, rcv_buf data) override {\n            return func(reply, client, msg_id, std::move(data));\n        }\n        virtual void timeout() override {\n            reply.done = true;\n            reply.p.set_exception(timeout_error());\n        }\n        virtual void cancel() override {\n            reply.done = true;\n            reply.p.set_exception(canceled_error());\n        }\n        virtual ~reply_handler() {}\n    };\nprivate:\n    std::unordered_map<id_type, std::unique_ptr<reply_handler_base>> _outstanding;\n    socket_address _server_addr, _local_addr;\n    client_options _options;\n    weak_ptr<client> _parent; // for stream clients\n\n    metrics _metrics;\n\nprivate:\n    future<> negotiate_protocol(feature_map map);\n    void negotiate(feature_map server_features);\n    // Returned future is\n    // - message id\n    // - optional server-side handler duration\n    // - message payload\n    future<std::tuple<int64_t, std::optional<uint32_t>, std::optional<rcv_buf>>>\n    read_response_frame_compressed(input_stream<char>& in);\npublic:\n    /**\n     * Create client object which will attempt to connect to the remote address.\n     *\n     * @param l \\ref seastar::logger to use for logging error messages\n     * @param s an optional connection serializer\n     * @param addr the remote address identifying this client\n     * @param local the local address of this client\n     */\n    client(const logger& l, void* s, const socket_address& addr, const socket_address& local = {});\n    client(const logger& l, void* s, client_options options, const socket_address& addr, const socket_address& local = {});\n\n     /**\n     * Create client object which will attempt to connect to the remote address using the\n     * specified seastar::socket.\n     *\n     * @param l \\ref seastar::logger to use for logging error messages\n     * @param s an optional connection serializer\n     * @param addr the remote address identifying this client\n     * @param local the local address of this client\n     * @param socket the socket object use to connect to the remote address\n     */\n    client(const logger& l, void* s, socket socket, const socket_address& addr, const socket_address& local = {});\n    client(const logger& l, void* s, client_options options, socket socket, const socket_address& addr, const socket_address& local = {});\n\n    stats get_stats() const;\n    size_t incoming_queue_length() const noexcept {\n        return _outstanding.size();\n    }\n\n    auto next_message_id() { return _message_id++; }\n    void wait_for_reply(id_type id, std::unique_ptr<reply_handler_base>&& h, std::optional<rpc_clock_type::time_point> timeout, cancellable* cancel);\n    void wait_timed_out(id_type id);\n    future<> stop() noexcept;\n    void abort_all_streams();\n    void deregister_this_stream();\n    socket_address peer_address() const override {\n        return _server_addr;\n    }\n    future<> await_connection() {\n        if (!_negotiated) {\n            return make_ready_future<>();\n        } else {\n            return _negotiated->get_shared_future();\n        }\n    }\n    template<typename Serializer, typename... Out>\n    future<sink<Out...>> make_stream_sink(socket socket) {\n        return await_connection().then([this, socket = std::move(socket)] () mutable {\n            if (!this->get_connection_id()) {\n                return make_exception_future<sink<Out...>>(std::runtime_error(\"Streaming is not supported by the server\"));\n            }\n            client_options o = _options;\n            o.stream_parent = this->get_connection_id();\n            o.send_timeout_data = false;\n            o.metrics_domain += \"_stream\";\n            auto c = make_shared<client>(_logger, _serializer, o, std::move(socket), _server_addr, _local_addr);\n            c->_parent = this->weak_from_this();\n            c->_is_stream = true;\n            return c->await_connection().then([c, this] {\n                if (_error) {\n                    throw closed_error();\n                }\n                xshard_connection_ptr s = make_lw_shared(make_foreign(static_pointer_cast<rpc::connection>(c)));\n                this->register_stream(c->get_connection_id(), s);\n                return sink<Out...>(make_shared<internal::sink_impl<Serializer, Out...>>(std::move(s)));\n            }).handle_exception([c] (std::exception_ptr eptr) {\n                // If await_connection fails we need to stop the client\n                // before destroying it.\n                return c->stop().then([eptr, c] {\n                    return make_exception_future<sink<Out...>>(eptr);\n                });\n            });\n        });\n    }\n    template<typename Serializer, typename... Out>\n    future<sink<Out...>> make_stream_sink() {\n        return make_stream_sink<Serializer, Out...>(make_socket());\n    }\n\n    future<> request(uint64_t type, int64_t id, snd_buf buf, std::optional<rpc_clock_type::time_point> timeout = {}, cancellable* cancel = nullptr);\n};\n\nclass protocol_base;\n\nclass server {\nprivate:\n    static thread_local std::unordered_map<streaming_domain_type, server*> _servers;\n\npublic:\n    class connection : public rpc::connection, public enable_shared_from_this<connection> {\n        client_info _info;\n        connection_id _parent_id = invalid_connection_id;\n        std::optional<isolation_config> _isolation_config;\n    private:\n        future<> negotiate_protocol();\n        future<std::tuple<std::optional<uint64_t>, uint64_t, int64_t, std::optional<rcv_buf>>>\n        read_request_frame_compressed(input_stream<char>& in);\n        future<feature_map> negotiate(feature_map requested);\n        future<> send_unknown_verb_reply(std::optional<rpc_clock_type::time_point> timeout, int64_t msg_id, uint64_t type);\n    public:\n        connection(server& s, connected_socket&& fd, socket_address&& addr, const logger& l, void* seralizer, connection_id id);\n        future<> process();\n        future<> respond(int64_t msg_id, snd_buf&& data, std::optional<rpc_clock_type::time_point> timeout, std::optional<rpc_clock_type::duration> handler_duration);\n        client_info& info() { return _info; }\n        const client_info& info() const { return _info; }\n        stats get_stats() const {\n            stats res = _stats;\n            res.pending = outgoing_queue_length();\n            return res;\n        }\n        socket_address peer_address() const override {\n            return _info.addr;\n        }\n        // Resources will be released when this goes out of scope\n        future<resource_permit> wait_for_resources(size_t memory_consumed,  std::optional<rpc_clock_type::time_point> timeout) {\n            if (timeout) {\n                return get_units(get_server()._resources_available, memory_consumed, *timeout);\n            } else {\n                return get_units(get_server()._resources_available, memory_consumed);\n            }\n        }\n        size_t estimate_request_size(size_t serialized_size) {\n            return rpc::estimate_request_size(get_server()._limits, serialized_size);\n        }\n        size_t max_request_size() const {\n            return get_server()._limits.max_memory;\n        }\n        server& get_server() {\n            return _info.server;\n        }\n        const server& get_server() const {\n            return _info.server;\n        }\n        future<> deregister_this_stream();\n        future<> abort_all_streams();\n    };\nprivate:\n    protocol_base& _proto;\n    server_socket _ss;\n    resource_limits _limits;\n    rpc_semaphore _resources_available;\n    std::unordered_map<connection_id, shared_ptr<connection>> _conns;\n    promise<> _ss_stopped;\n    gate _reply_gate;\n    server_options _options;\n    bool _shutdown = false;\n    uint64_t _next_client_id = 1;\n\npublic:\n    server(protocol_base* proto, const socket_address& addr, resource_limits memory_limit = resource_limits());\n    server(protocol_base* proto, server_options opts, const socket_address& addr, resource_limits memory_limit = resource_limits());\n    server(protocol_base* proto, server_socket, resource_limits memory_limit = resource_limits(), server_options opts = server_options{});\n    server(protocol_base* proto, server_options opts, server_socket, resource_limits memory_limit = resource_limits());\n    void accept();\n    /**\n     * Stops the server.\n     *\n     * It makes sure that no new rpcs are admitted, no rpc handlers issued on this\n     * connection are running any longer and no replies on the previously running\n     * handlers will be sent.\n     */\n    future<> stop();\n    /**\n     * Shuts down the server.\n     *\n     * Light version of the stop, that just makes sure the server is not visible\n     * by remote clients, i.e. -- no new rpcs are admitted and no replies on the\n     * previously running handlers will be sent. Currently running handlers may\n     * still run.\n     *\n     * Caller of shutdown() mush wait for it to resolve before calling stop.\n     */\n    future<> shutdown();\n    template<typename Func>\n    void foreach_connection(Func&& f) {\n        for (auto c : _conns) {\n            f(*c.second);\n        }\n    }\n    /**\n     * Abort the given connection, causing it to stop receiving any further messages.\n     * It's safe to abort a connection from an RPC handler running on that connection.\n     * Does nothing if there is no connection with the given ID on this server.\n     *\n     * @param id the ID of the connection to abort.\n     */\n    void abort_connection(connection_id id);\n    gate& reply_gate() {\n        return _reply_gate;\n    }\n    friend connection;\n    friend client;\n};\n\nusing rpc_handler_func = std::function<future<> (shared_ptr<server::connection>, std::optional<rpc_clock_type::time_point> timeout, int64_t msgid,\n                                                 rcv_buf data, gate::holder guard)>;\n\nstruct rpc_handler {\n    scheduling_group sg;\n    rpc_handler_func func;\n    gate use_gate;\n};\n\nclass protocol_base {\npublic:\n    virtual ~protocol_base() {};\n    virtual shared_ptr<server::connection> make_server_connection(rpc::server& server, connected_socket fd, socket_address addr, connection_id id) = 0;\nprotected:\n    friend class server;\n\n    struct handler_with_holder {\n        rpc_handler& handler;\n        gate::holder holder;\n    };\n    virtual std::optional<handler_with_holder> get_handler(uint64_t msg_id) = 0;\n};\n\n/// \\addtogroup rpc\n/// @{\n\n/// Defines a protocol for communication between a server and a client.\n///\n/// A protocol is defined by a `Serializer` and a `MsgType`. The `Serializer` is\n/// responsible for serializing and unserializing all types used as arguments and\n/// return types used in the protocol. The `Serializer` is expected to define a\n/// `read()` and `write()` method for each such type `T` as follows:\n///\n///     template <typename Output>\n///     void write(const serializer&, Output& output, const T& data);\n///\n///     template <typename Input>\n///     T read(const serializer&, Input& input, type<T> type_tag);  // type_tag used to disambiguate\n///\n/// Where `Input` and `Output` have a `void read(char*, size_t)` and\n/// `write(const char*, size_t)` respectively.\n/// `MsgType` defines the type to be used as the message id, the id which is\n/// used to identify different messages used in the protocol. These are also\n/// often referred to as \"verbs\". The client will use the message id, to\n/// specify the remote method (verb) to invoke on the server. The server uses\n/// the message id to dispatch the incoming call to the right handler.\n/// `MsgType` should be hashable and serializable. It is preferable to use enum\n/// for message types, but do not forget to provide hash function for it.\n///\n/// Use register_handler() on the server to define the available verbs and the\n/// code to be executed when they are invoked by clients. Use make_client() on\n/// the client to create a matching callable that can be used to invoke the\n/// verb on the server and wait for its result. Note that register_handler()\n/// also returns a client, that can be used to invoke the registered verb on\n/// another node (given that the other node has the same verb). This is useful\n/// for symmetric protocols, where two or more nodes all have servers as well as\n/// connect to the other nodes as clients.\n///\n/// Use protocol::server to listen for and accept incoming connections on the\n/// server and protocol::client to establish connections to the server.\n/// Note that registering the available verbs can be done before/after\n/// listening for connections, but best to ensure that by the time incoming\n/// requests are to be expected, all the verbs are set-up.\n///\n/// ## Configuration\n///\n/// TODO\n///\n/// ## Isolation\n///\n/// RPC supports isolating verb handlers from each other. There are two ways to\n/// achieve this: per-handler isolation (the old way) and per-connection\n/// isolation (the new way). If no isolation is configured, all handlers will be\n/// executed in the context of the scheduling_group in which the\n/// protocol::server was created.\n///\n/// Per-handler isolation (the old way) can be configured by using the\n/// register_handler() overload which takes a scheduling_group. When invoked,\n/// the body of the handler will be executed from the context of the configured\n/// scheduling_group.\n///\n/// Per-connection isolation (the new way) is a more flexible mechanism that\n/// requires user application provided logic to determine how connections are\n/// isolated. This mechanism has two parts, the server and the client part.\n/// The client configures isolation by setting client_options::isolation_cookie.\n/// This cookie is an opaque (to the RPC layer) string that is to be interpreted\n/// on the server using user application provided logic. The application\n/// provides this logic to the server by setting\n/// resource_limits::isolate_connection to an appropriate handler function, that\n/// interprets the opaque cookie and resolves it to an isolation_config. The\n/// scheduling_group in the former will be used not just to execute all verb\n/// handlers, but also the connection loop itself, hence providing better\n/// isolation.\n///\n/// There a few gotchas related to mixing the two isolation mechanisms. This can\n/// happen when the application is updated and one of the client/server is\n/// still using the old/new mechanism. In general per-connection isolation\n/// overrides the per-handler one. If both are set up, the former will determine\n/// the scheduling_group context for the handlers. If the client is not\n/// configured to send an isolation cookie, the server's\n/// resource_limits::isolate_connection will not be invoked and the server will\n/// fall back to per-handler isolation if configured. If the client is\n/// configured to send an isolation cookie but the server doesn't have a\n/// resource_limits::isolate_connection configured, it will use\n/// default_isolate_connection() to interpret the cookie. Note that this still\n/// overrides the per-handler isolation if any is configured. If the server is\n/// so old that it doesn't have the per-connection isolation feature at all, it\n/// will of course just use the per-handler one, if configured.\n///\n/// ## Compatibility\n///\n/// TODO\n///\n/// \\tparam Serializer the serializer for the protocol.\n/// \\tparam MsgType the type to be used as the message id or verb id.\ntemplate<typename Serializer, typename MsgType = uint32_t>\nclass protocol final : public protocol_base {\npublic:\n    /// Represents the listening port and all accepted connections.\n    class server : public rpc::server {\n    public:\n        server(protocol& proto, const socket_address& addr, resource_limits memory_limit = resource_limits()) :\n            rpc::server(&proto, addr, memory_limit) {}\n        server(protocol& proto, server_options opts, const socket_address& addr, resource_limits memory_limit = resource_limits()) :\n            rpc::server(&proto, opts, addr, memory_limit) {}\n        server(protocol& proto, server_socket socket, resource_limits memory_limit = resource_limits(), server_options = server_options{}) :\n            rpc::server(&proto, std::move(socket), memory_limit) {}\n        server(protocol& proto, server_options opts, server_socket socket, resource_limits memory_limit = resource_limits()) :\n            rpc::server(&proto, opts, std::move(socket), memory_limit) {}\n    };\n    /// Represents a client side connection.\n    class client : public rpc::client {\n    public:\n        /*\n         * Create client object which will attempt to connect to the remote address.\n         *\n         * @param addr the remote address identifying this client\n         * @param local the local address of this client\n         */\n        client(protocol& p, const socket_address& addr, const socket_address& local = {}) :\n            rpc::client(p.get_logger(), &p._serializer, addr, local) {}\n        client(protocol& p, client_options options, const socket_address& addr, const socket_address& local = {}) :\n            rpc::client(p.get_logger(), &p._serializer, options, addr, local) {}\n\n        /**\n         * Create client object which will attempt to connect to the remote address using the\n         * specified seastar::socket.\n         *\n         * @param addr the remote address identifying this client\n         * @param local the local address of this client\n         * @param socket the socket object use to connect to the remote address\n         */\n        client(protocol& p, socket socket, const socket_address& addr, const socket_address& local = {}) :\n            rpc::client(p.get_logger(), &p._serializer, std::move(socket), addr, local) {}\n        client(protocol& p, client_options options, socket socket, const socket_address& addr, const socket_address& local = {}) :\n            rpc::client(p.get_logger(), &p._serializer, options, std::move(socket), addr, local) {}\n    };\n\n    friend server;\nprivate:\n    std::unordered_map<MsgType, rpc_handler> _handlers;\n    Serializer _serializer;\n    logger _logger;\n\npublic:\n    protocol(Serializer&& serializer) : _serializer(std::forward<Serializer>(serializer)) {}\n\n    /// Creates a callable that can be used to invoke the verb on the remote.\n    ///\n    /// \\tparam Func The signature of the verb. Has to be either the same or\n    ///     compatible with the one passed to register_handler on the server.\n    /// \\param t the verb to invoke on the remote.\n    ///\n    /// \\returns a callable whose signature is derived from Func as follows:\n    ///     given `Func == Ret(Args...)` the returned callable has the following\n    ///     signature: `future<Ret>(protocol::client&, Args...)`.\n    template<typename Func>\n    auto make_client(MsgType t);\n\n    /// Register a handler to be called when this verb is invoked.\n    ///\n    /// \\tparam Func the type of the handler for the verb. This determines the\n    ///     signature of the verb.\n    /// \\param t the verb to register the handler for.\n    /// \\param func the callable to be called when the verb is invoked by the\n    ///     remote.\n    ///\n    /// \\returns a client, a callable that can be used to invoke the verb. See\n    ///     make_client(). The client can be discarded, in fact this is what\n    ///     most callers will do as real clients will live on a remote node, not\n    ///     on the one where handlers are registered.\n    template<typename Func>\n    auto register_handler(MsgType t, Func&& func);\n\n    /// Register a handler to be called when this verb is invoked.\n    ///\n    /// \\tparam Func the type of the handler for the verb. This determines the\n    ///     signature of the verb.\n    /// \\param t the verb to register the handler for.\n    /// \\param sg the scheduling group that will be used to invoke the handler\n    ///     in. This can be used to execute different verbs in different\n    ///     scheduling groups. Note that there is a newer mechanism to determine\n    ///     the scheduling groups a handler will run it per invocation, see\n    ///     isolation_config.\n    /// \\param func the callable to be called when the verb is invoked by the\n    ///     remote.\n    ///\n    /// \\returns a client, a callable that can be used to invoke the verb. See\n    ///     make_client(). The client can be discarded, in fact this is what\n    ///     most callers will do as real clients will live on a remote node, not\n    ///     on the one where handlers are registered.\n    template <typename Func>\n    auto register_handler(MsgType t, scheduling_group sg, Func&& func);\n\n    /// Unregister the handler for the verb.\n    ///\n    /// Waits for all currently running handlers, then unregisters the handler.\n    /// Future attempts to invoke the verb will fail. This becomes effective\n    /// immediately after calling this function.\n    ///\n    /// \\param t the verb to unregister the handler for.\n    ///\n    /// \\returns a future that becomes available once all currently running\n    ///     handlers finished.\n    future<> unregister_handler(MsgType t);\n\n\n    /// Set a logger to be used to log messages.\n    void set_logger(::seastar::logger* logger) {\n        _logger.set(logger);\n    }\n\n    const logger& get_logger() const {\n        return _logger;\n    }\n\n    shared_ptr<rpc::server::connection> make_server_connection(rpc::server& server, connected_socket fd, socket_address addr, connection_id id) override {\n        return make_shared<rpc::server::connection>(server, std::move(fd), std::move(addr), _logger, &_serializer, id);\n    }\n\n    bool has_handler(MsgType msg_id);\n\n    /// Checks if any there are handlers registered.\n    /// Debugging helper, should only be used for debugging and not relied on.\n    ///\n    /// \\returns true if there are, false if there are no registered handlers.\n    bool has_handlers() const noexcept {\n        return !_handlers.empty();\n    }\n\nprivate:\n    std::optional<handler_with_holder> get_handler(uint64_t msg_id) override;\n\n    template<typename Ret, typename... In>\n    auto make_client(signature<Ret(In...)> sig, MsgType t);\n\n    void register_receiver(MsgType t, rpc_handler&& handler) {\n        auto r = _handlers.emplace(t, std::move(handler));\n        if (!r.second) {\n            throw_with_backtrace<std::runtime_error>(\"registered handler already exists\");\n        }\n    }\n};\n\n/// @}\n\n}\n\n}\n\n#include \"rpc_impl.hh\"\n"
  },
  {
    "path": "include/seastar/rpc/rpc_impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n#pragma once\n\n#include <seastar/core/format.hh>\n#include <seastar/core/function_traits.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/is_smart_ptr.hh>\n#include <seastar/core/simple-stream.hh>\n#include <seastar/core/deleter.hh>\n\n#include <boost/type.hpp> // for compatibility\n\n#include <concepts>\n\nnamespace seastar {\n\nnamespace rpc {\n\nenum class exception_type : uint32_t {\n    USER = 0,\n    UNKNOWN_VERB = 1,\n};\n\ntemplate<typename T>\nstruct remove_optional {\n    using type = T;\n};\n\ntemplate<typename T>\nstruct remove_optional<optional<T>> {\n    using type = T;\n};\n\nstruct wait_type {}; // opposite of no_wait_type\n\n// tags to tell whether we want a const client_info& parameter\nstruct do_want_client_info {};\nstruct dont_want_client_info {};\n\n// tags to tell whether we want a opt_time_point parameter\nstruct do_want_time_point {};\nstruct dont_want_time_point {};\n\n// General case\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature;\n    using want_client_info = dont_want_client_info;\n    using want_time_point = dont_want_time_point;\n};\n\n// Specialize 'clean' for handlers that receive client_info\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (const client_info&, In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature<Ret (In...)>;\n    using want_client_info = do_want_client_info;\n    using want_time_point = dont_want_time_point;\n};\n\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (client_info&, In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature<Ret (In...)>;\n    using want_client_info = do_want_client_info;\n    using want_time_point = dont_want_time_point;\n};\n\n// Specialize 'clean' for handlers that receive client_info and opt_time_point\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (const client_info&, opt_time_point, In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature<Ret (In...)>;\n    using want_client_info = do_want_client_info;\n    using want_time_point = do_want_time_point;\n};\n\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (client_info&, opt_time_point, In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature<Ret (In...)>;\n    using want_client_info = do_want_client_info;\n    using want_time_point = do_want_time_point;\n};\n\n// Specialize 'clean' for handlers that receive opt_time_point\ntemplate <typename Ret, typename... In>\nstruct signature<Ret (opt_time_point, In...)> {\n    using ret_type = Ret;\n    using arg_types = std::tuple<In...>;\n    using clean = signature<Ret (In...)>;\n    using want_client_info = dont_want_client_info;\n    using want_time_point = do_want_time_point;\n};\n\ntemplate <typename T>\nstruct wait_signature {\n    using type = wait_type;\n    using cleaned_type = T;\n};\n\ntemplate <typename... T>\nstruct wait_signature<future<T...>> {\n    using type = wait_type;\n    using cleaned_type = future<T...>;\n};\n\ntemplate <>\nstruct wait_signature<no_wait_type> {\n    using type = no_wait_type;\n    using cleaned_type = void;\n};\n\ntemplate <>\nstruct wait_signature<future<no_wait_type>> {\n    using type = no_wait_type;\n    using cleaned_type = future<>;\n};\n\ntemplate <typename T>\nusing wait_signature_t = typename wait_signature<T>::type;\n\ntemplate <typename... In>\ninline\nstd::tuple<In...>\nmaybe_add_client_info(dont_want_client_info, client_info&, std::tuple<In...>&& args) {\n    return std::move(args);\n}\n\ntemplate <typename... In>\ninline\nstd::tuple<std::reference_wrapper<client_info>, In...>\nmaybe_add_client_info(do_want_client_info, client_info& ci, std::tuple<In...>&& args) {\n    return std::tuple_cat(std::make_tuple(std::ref(ci)), std::move(args));\n}\n\ntemplate <typename... In>\ninline\nstd::tuple<In...>\nmaybe_add_time_point(dont_want_time_point, opt_time_point&, std::tuple<In...>&& args) {\n    return std::move(args);\n}\n\ntemplate <typename... In>\ninline\nstd::tuple<opt_time_point, In...>\nmaybe_add_time_point(do_want_time_point, opt_time_point& otp, std::tuple<In...>&& args) {\n    return std::tuple_cat(std::make_tuple(otp), std::move(args));\n}\n\ninline sstring serialize_connection_id(const connection_id& id) {\n    sstring p = uninitialized_string(sizeof(id));\n    auto c = p.data();\n    write_le(c, id.id());\n    return p;\n}\n\ninline connection_id deserialize_connection_id(const sstring& s) {\n    using id_type = decltype(connection_id{0}.id());\n    auto p = s.c_str();\n    auto id = read_le<id_type>(p);\n    return connection_id{id};\n}\n\ntemplate <bool IsSmartPtr>\nstruct serialize_helper;\n\ntemplate <>\nstruct serialize_helper<false> {\n    template <typename Serializer, typename Output, typename T>\n    static inline void serialize(Serializer& serializer, Output& out, const T& t) {\n        return write(serializer, out, t);\n    }\n};\n\ntemplate <>\nstruct serialize_helper<true> {\n    template <typename Serializer, typename Output, typename T>\n    static inline void serialize(Serializer& serializer, Output& out, const T& t) {\n        return write(serializer, out, *t);\n    }\n};\n\ntemplate <typename Serializer, typename Output, typename... T>\ninline void do_marshall(Serializer& serializer, Output& out, const T&... args);\n\ntemplate <typename Serializer, typename Output>\nstruct marshall_one {\n    template <typename T> struct helper {\n        static void doit(Serializer& serializer, Output& out, const T& arg) {\n            using serialize_helper_type = serialize_helper<is_smart_ptr<typename std::remove_reference_t<T>>::value>;\n            serialize_helper_type::serialize(serializer, out, arg);\n        }\n    };\n    template<typename T> struct helper<std::reference_wrapper<const T>> {\n        static void doit(Serializer& serializer, Output& out, const std::reference_wrapper<const T>& arg) {\n            helper<T>::doit(serializer, out, arg.get());\n        }\n    };\n    static void put_connection_id(const connection_id& cid, Output& out) {\n        sstring id = serialize_connection_id(cid);\n        out.write(id.c_str(), id.size());\n    }\n    template <typename... T> struct helper<sink<T...>> {\n        static void doit(Serializer&, Output& out, const sink<T...>& arg) {\n            put_connection_id(arg.get_id(), out);\n        }\n    };\n    template <typename... T> struct helper<source<T...>> {\n        static void doit(Serializer&, Output& out, const source<T...>& arg) {\n            put_connection_id(arg.get_id(), out);\n        }\n    };\n    template <typename... T> struct helper<tuple<T...>> {\n        static void doit(Serializer& serializer, Output& out, const tuple<T...>& arg) {\n            auto do_do_marshall = [&serializer, &out] (const auto&... args) {\n                do_marshall(serializer, out, args...);\n            };\n            // since C++23, std::apply() only accepts tuple-like types, while\n            // rpc::tuple is not a tuple-like type from the tuple-like C++\n            // concept's perspective. so we have to cast it to std::tuple to\n            // appease std::apply()\n            std::apply(do_do_marshall, static_cast<const std::tuple<T...>&>(arg));\n        }\n    };\n};\n\ntemplate <typename Serializer, typename Output, typename... T>\ninline void do_marshall(Serializer& serializer, Output& out, const T&... args) {\n    // C++ guarantees that brace-initialization expressions are evaluted in order\n    (void)std::initializer_list<int>{(marshall_one<Serializer, Output>::template helper<T>::doit(serializer, out, args), 1)...};\n}\n\nstatic inline memory_output_stream<snd_buf::iterator> make_serializer_stream(snd_buf& output) {\n    auto* b = std::get_if<temporary_buffer<char>>(&output.bufs);\n    if (b) {\n        return memory_output_stream<snd_buf::iterator>(memory_output_stream<snd_buf::iterator>::simple(b->get_write(), b->size()));\n    } else {\n        auto& ar = std::get<std::vector<temporary_buffer<char>>>(output.bufs);\n        return memory_output_stream<snd_buf::iterator>(memory_output_stream<snd_buf::iterator>::fragmented(ar.begin(), output.size));\n    }\n}\n\ntemplate <typename Serializer, typename... T>\ninline snd_buf marshall(Serializer& serializer, size_t head_space, const T&... args) {\n    measuring_output_stream measure;\n    do_marshall(serializer, measure, args...);\n    snd_buf ret(measure.size() + head_space);\n    auto out = make_serializer_stream(ret);\n    out.skip(head_space);\n    do_marshall(serializer, out, args...);\n    return ret;\n}\n\ntemplate <typename Serializer, typename Input, typename... T>\nstd::tuple<T...> do_unmarshall(connection& c, Input& in);\n\n// The protocol to call the serializer is read(serializer, stream, rpc::type<T>).\n// However, some users (ahem) used boost::type instead of rpc::type when the two\n// types were aliased, preventing us from moving to the newer std::type_identity.\n// To preserve compatibility, calls to read() are routed through\n// read_via_type_marker(), of which there are two variants, one for\n// boost::type (marked as deprecated) and one for std::type_identity.\n\ntemplate <typename T, typename... Args>\nrequires requires (Args... args, type<T> t) { read(std::forward<Args>(args)..., t); }\nauto\nread_via_type_marker(Args&&... args) {\n    return read(std::forward<Args>(args)..., type<T>());\n}\n\ntemplate <typename T, typename... Args>\nrequires requires (Args... args, boost::type<T> t) { read(std::forward<Args>(args)..., t); }\n[[deprecated(\"Use rpc::type<> instead of boost::type<>\")]]\nauto\nread_via_type_marker(Args&&... args) {\n    return read(std::forward<Args>(args)..., boost::type<T>());\n}\n\ntemplate<typename Serializer, typename Input>\nstruct unmarshal_one {\n    template<typename T> struct helper {\n        static T doit(connection& c, Input& in) {\n            return read_via_type_marker<T>(c.serializer<Serializer>(), in);\n        }\n    };\n    template<typename T> struct helper<optional<T>> {\n        static optional<T> doit(connection& c, Input& in) {\n            if (in.size()) {\n                return optional<T>(read_via_type_marker<typename remove_optional<T>::type>(c.serializer<Serializer>(), in));\n            } else {\n                return optional<T>();\n            }\n        }\n    };\n    template<typename T> struct helper<std::reference_wrapper<const T>> {\n        static T doit(connection& c, Input& in) {\n            return helper<T>::doit(c, in);\n        }\n    };\n    static connection_id get_connection_id(Input& in) {\n        sstring id = uninitialized_string(sizeof(connection_id));\n        in.read(id.data(), sizeof(connection_id));\n        return deserialize_connection_id(id);\n    }\n    template<typename... T> struct helper<sink<T...>> {\n        static sink<T...> doit(connection& c, Input& in) {\n            return sink<T...>(make_shared<internal::sink_impl<Serializer, T...>>(c.get_stream(get_connection_id(in))));\n        }\n    };\n    template<typename... T> struct helper<source<T...>> {\n        static source<T...> doit(connection& c, Input& in) {\n            return source<T...>(make_shared<internal::source_impl<Serializer, T...>>(c.get_stream(get_connection_id(in))));\n        }\n    };\n    template <typename... T> struct helper<tuple<T...>> {\n        static tuple<T...> doit(connection& c, Input& in) {\n            return do_unmarshall<Serializer, Input, T...>(c, in);\n        }\n    };\n};\n\ntemplate <typename... T>\nstruct default_constructible_tuple_except_first;\n\ntemplate <>\nstruct default_constructible_tuple_except_first<> {\n    using type = std::tuple<>;\n};\n\ntemplate <typename T0, typename... T>\nstruct default_constructible_tuple_except_first<T0, T...> {\n    using type = std::tuple<\n            T0,\n            std::conditional_t<\n                    std::is_default_constructible_v<T>,\n                    T,\n                    std::optional<T>\n            >...\n        >;\n};\n\ntemplate <typename... T>\nusing default_constructible_tuple_except_first_t = typename default_constructible_tuple_except_first<T...>::type;\n\n// Where Tin != Tout, apply std:optional::value()\ntemplate <typename... Tout, typename... Tin>\nauto\nunwrap_optional_if_needed(std::tuple<Tin...>&& tuple_in) {\n    using tuple_in_t = std::tuple<Tin...>;\n    using tuple_out_t = std::tuple<Tout...>;\n    return std::invoke([&] <size_t... Idx> (std::index_sequence<Idx...>) {\n        return tuple_out_t(\n            std::invoke([&] () {\n                if constexpr (std::same_as<std::tuple_element_t<Idx, tuple_in_t>, std::tuple_element_t<Idx, tuple_out_t>>) {\n                    return std::move(std::get<Idx>(tuple_in));\n                } else {\n                    return std::move(std::get<Idx>(tuple_in).value());\n                }\n            })...);\n    }, std::make_index_sequence<sizeof...(Tout)>());\n}\n\ntemplate <typename Serializer, typename Input, typename... T>\ninline std::tuple<T...> do_unmarshall(connection& c, Input& in) {\n    // Argument order processing is unspecified, but we need to deserialize\n    // left-to-right. So we deserialize into something that can be lazily\n    // constructed (and can conditionally destroy itself if we only constructed some\n    // of the arguments).\n    //\n    // The first element of the tuple has no ordering\n    // problem, and we can deserialize directly into a std::tuple<T...>.\n    //\n    // For the rest of the elements, if they are default-constructible, we leave\n    // them as is, and if not, we deserialize into std::optional<T>, and later\n    // unwrap them. If we're lucky and nothing was wrapped, we can return without\n    // any data movement.\n    using ret_type = std::tuple<T...>;\n    using temporary_type = default_constructible_tuple_except_first_t<T...>;\n    return std::invoke([&] <size_t... Idx> (std::index_sequence<Idx...>) {\n        auto tmp = temporary_type(\n            std::invoke([&] () -> std::tuple_element_t<Idx, temporary_type> {\n                if constexpr (Idx == 0) {\n                    // The first T has no ordering problem, so we can deserialize it directly into the tuple\n                    return unmarshal_one<Serializer, Input>::template helper<std::tuple_element_t<Idx, ret_type>>::doit(c, in);\n                } else {\n                    // Use default constructor for the rest of the Ts\n                    return {};\n                }\n            })...\n        );\n        // Deserialize the other Ts, comma-expression preserves left-to-right order.\n        (void)(...,  ((Idx == 0\n            ? 0\n            : ((std::get<Idx>(tmp) = unmarshal_one<Serializer, Input>::template helper<std::tuple_element_t<Idx, ret_type>>::doit(c, in), 0)))));\n        if constexpr (std::same_as<ret_type, temporary_type>) {\n            // Use Named Return Vale Optimization (NVRO) if we didn't have to wrap anything\n            return tmp;\n        } else {\n            return unwrap_optional_if_needed<T...>(std::move(tmp));\n        }\n    }, std::index_sequence_for<T...>());\n}\n\ntemplate <typename Serializer, typename... T>\ninline std::tuple<T...> unmarshall(connection& c, rcv_buf input) {\n    auto in = make_deserializer_stream(input);\n    return do_unmarshall<Serializer, decltype(in), T...>(c, in);\n}\n\ninline std::exception_ptr unmarshal_exception(rcv_buf& d) {\n    std::exception_ptr ex;\n    auto data = make_deserializer_stream(d);\n\n    uint32_t v32;\n    data.read(reinterpret_cast<char*>(&v32), 4);\n    exception_type ex_type = exception_type(le_to_cpu(v32));\n    data.read(reinterpret_cast<char*>(&v32), 4);\n    uint32_t ex_len = le_to_cpu(v32);\n\n    switch (ex_type) {\n    case exception_type::USER: {\n        std::string s(ex_len, '\\0');\n        data.read(&*s.begin(), ex_len);\n        ex = std::make_exception_ptr(remote_verb_error(std::move(s)));\n        break;\n    }\n    case exception_type::UNKNOWN_VERB: {\n        uint64_t v64;\n        data.read(reinterpret_cast<char*>(&v64), 8);\n        ex = std::make_exception_ptr(unknown_verb_error(le_to_cpu(v64)));\n        break;\n    }\n    default:\n        ex = std::make_exception_ptr(unknown_exception_error());\n        break;\n    }\n    return ex;\n}\n\ntemplate <typename Payload, typename... T>\nstruct rcv_reply_base  {\n    bool done = false;\n    promise<T...> p;\n    template<typename... V>\n    void set_value(V&&... v) {\n        done = true;\n        p.set_value(seastar::internal::untuple(std::forward<V>(v))...);\n    }\n    ~rcv_reply_base() {\n        if (!done) {\n            p.set_exception(closed_error());\n        }\n    }\n};\n\ntemplate<typename Serializer, typename T>\nstruct rcv_reply : rcv_reply_base<T, T> {\n    inline void get_reply(rpc::client& dst, rcv_buf input) {\n        this->set_value(unmarshall<Serializer, T>(dst, std::move(input)));\n    }\n};\n\ntemplate<typename Serializer, typename... T>\nstruct rcv_reply<Serializer, future<T...>> : rcv_reply_base<std::tuple<T...>, T...> {\n    inline void get_reply(rpc::client& dst, rcv_buf input) {\n        this->set_value(unmarshall<Serializer, T...>(dst, std::move(input)));\n    }\n};\n\ntemplate<typename Serializer>\nstruct rcv_reply<Serializer, void> : rcv_reply_base<void, void> {\n    inline void get_reply(rpc::client&, rcv_buf) {\n        this->set_value();\n    }\n};\n\ntemplate<typename Serializer>\nstruct rcv_reply<Serializer, future<>> : rcv_reply<Serializer, void> {};\n\ntemplate <typename Serializer, typename Ret, typename... InArgs>\ninline auto wait_for_reply(wait_type, std::optional<rpc_clock_type::time_point> timeout, rpc_clock_type::time_point start, cancellable* cancel, rpc::client& dst, id_type msg_id,\n        signature<Ret (InArgs...)>) {\n    using reply_type = rcv_reply<Serializer, Ret>;\n    auto lambda = [] (reply_type& r, rpc::client& dst, id_type msg_id, rcv_buf data) mutable {\n        if (msg_id >= 0) {\n            dst.get_stats_internal().replied++;\n            return r.get_reply(dst, std::move(data));\n        } else {\n            dst.get_stats_internal().exception_received++;\n            r.done = true;\n            r.p.set_exception(unmarshal_exception(data));\n        }\n    };\n    using handler_type = typename rpc::client::template reply_handler<reply_type, decltype(lambda)>;\n    auto r = std::make_unique<handler_type>(std::move(lambda));\n    r->start = start;\n    auto fut = r->reply.p.get_future();\n    dst.wait_for_reply(msg_id, std::move(r), timeout, cancel);\n    return fut;\n}\n\ntemplate<typename Serializer, typename... InArgs>\ninline auto wait_for_reply(no_wait_type, std::optional<rpc_clock_type::time_point>, rpc_clock_type::time_point start, cancellable*, rpc::client&, id_type,\n        signature<no_wait_type (InArgs...)>) {  // no_wait overload\n    return make_ready_future<>();\n}\n\ntemplate<typename Serializer, typename... InArgs>\ninline auto wait_for_reply(no_wait_type, std::optional<rpc_clock_type::time_point>, rpc_clock_type::time_point, cancellable*, rpc::client&, id_type,\n        signature<future<no_wait_type> (InArgs...)>) {  // future<no_wait> overload\n    return make_ready_future<>();\n}\n\n// Convert a relative timeout (a duration) to an absolute one (time_point).\n// Do the calculation safely so that a very large duration will be capped by\n// time_point::max, instead of wrapping around to ancient history.\ninline rpc_clock_type::time_point\nrelative_timeout_to_absolute(rpc_clock_type::duration relative) {\n    rpc_clock_type::time_point now = rpc_clock_type::now();\n    return now + std::min(relative, rpc_clock_type::time_point::max() - now);\n}\n\n// Refer to struct request_frame for more details\nstatic constexpr size_t request_frame_headroom = 28;\n\n// Returns lambda that can be used to send rpc messages.\n// The lambda gets client connection and rpc parameters as arguments, marshalls them sends\n// to a server and waits for a reply. After receiving reply it unmarshalls it and signal completion\n// to a caller.\ntemplate<typename Serializer, typename MsgType, typename Ret, typename... InArgs>\nauto send_helper(MsgType xt, signature<Ret (InArgs...)> xsig) {\n    struct shelper {\n        MsgType t;\n        signature<Ret (InArgs...)> sig;\n        auto send(rpc::client& dst, std::optional<rpc_clock_type::time_point> timeout, cancellable* cancel, const InArgs&... args) {\n            if (dst.error()) {\n                using cleaned_ret_type = typename wait_signature<Ret>::cleaned_type;\n                return futurize<cleaned_ret_type>::make_exception_future(closed_error());\n            }\n\n            auto start = rpc_clock_type::now();\n            // send message\n            auto msg_id = dst.next_message_id();\n            snd_buf data = marshall(dst.template serializer<Serializer>(), request_frame_headroom, args...);\n\n            // prepare reply handler, if return type is now_wait_type this does nothing, since no reply will be sent\n            using wait = wait_signature_t<Ret>;\n            return when_all(dst.request(uint64_t(t), msg_id, std::move(data), timeout, cancel), wait_for_reply<Serializer>(wait(), timeout, start, cancel, dst, msg_id, sig)).then([] (auto r) {\n                    std::get<0>(r).ignore_ready_future();\n                    return std::move(std::get<1>(r)); // return future of wait_for_reply\n            });\n        }\n        auto operator()(rpc::client& dst, const InArgs&... args) {\n            return send(dst, {}, nullptr, args...);\n        }\n        auto operator()(rpc::client& dst, rpc_clock_type::time_point timeout, const InArgs&... args) {\n            return send(dst, timeout, nullptr, args...);\n        }\n        auto operator()(rpc::client& dst, rpc_clock_type::time_point timeout, cancellable& cancel, const InArgs&... args) {\n            return send(dst, timeout, &cancel, args...);\n        }\n        auto operator()(rpc::client& dst, rpc_clock_type::duration timeout, const InArgs&... args) {\n            return send(dst, relative_timeout_to_absolute(timeout), nullptr, args...);\n        }\n        auto operator()(rpc::client& dst, rpc_clock_type::duration timeout, cancellable& cancel, const InArgs&... args) {\n            return send(dst, relative_timeout_to_absolute(timeout), &cancel, args...);\n        }\n        auto operator()(rpc::client& dst, cancellable& cancel, const InArgs&... args) {\n            return send(dst, {}, &cancel, args...);\n        }\n\n    };\n    return shelper{xt, xsig};\n}\n\n// Refer to struct response_frame for more details\nstatic constexpr size_t response_frame_headroom = 16;\n\ntemplate<typename Serializer, typename RetTypes>\ninline future<> reply(wait_type, future<RetTypes>&& ret, uint64_t verb, int64_t msg_id, shared_ptr<server::connection> client,\n        std::optional<rpc_clock_type::time_point> timeout, std::optional<rpc_clock_type::duration> handler_duration) {\n    if (!client->error()) {\n        snd_buf data;\n        try {\n            if constexpr (std::is_void_v<RetTypes>) {\n                ret.get();\n                data = std::invoke(marshall<Serializer>, std::ref(client->template serializer<Serializer>()), response_frame_headroom);\n            } else {\n                data = std::invoke(marshall<Serializer, const RetTypes&>, std::ref(client->template serializer<Serializer>()), response_frame_headroom, std::move(ret.get()));\n            }\n        } catch (std::exception& ex) {\n            uint32_t len = std::strlen(ex.what());\n            data = snd_buf(response_frame_headroom + 2 * sizeof(uint32_t) + len);\n            auto os = make_serializer_stream(data);\n            os.skip(response_frame_headroom);\n            uint32_t v32 = cpu_to_le(uint32_t(exception_type::USER));\n            os.write(reinterpret_cast<char*>(&v32), sizeof(v32));\n            v32 = cpu_to_le(len);\n            os.write(reinterpret_cast<char*>(&v32), sizeof(v32));\n            os.write(ex.what(), len);\n            msg_id = -msg_id;\n        }\n\n        return client->respond(msg_id, std::move(data), timeout, handler_duration);\n    } else {\n        ret.ignore_ready_future();\n        return make_ready_future<>();\n    }\n}\n\n// specialization for no_wait_type which does not send a reply\ntemplate<typename Serializer>\ninline future<> reply(no_wait_type, future<no_wait_type>&& r, uint64_t verb, int64_t msgid, shared_ptr<server::connection> client,\n        std::optional<rpc_clock_type::time_point>, std::optional<rpc_clock_type::duration>) {\n    try {\n        r.get();\n    } catch (std::exception& ex) {\n        client->get_logger()(client->info(), msgid, format(\"exception \\\"{}\\\" in no_wait handler of the verb {} ignored\", ex.what(), verb));\n    }\n    return make_ready_future<>();\n}\n\ntemplate<typename Ret, typename... InArgs, typename WantClientInfo, typename WantTimePoint, typename Func, typename ArgsTuple>\ninline futurize_t<Ret> apply(Func& func, client_info& info, opt_time_point time_point, WantClientInfo wci, WantTimePoint wtp, signature<Ret (InArgs...)>, ArgsTuple&& args) {\n    using futurator = futurize<Ret>;\n    return futurator::apply(func, maybe_add_client_info(wci, info, maybe_add_time_point(wtp, time_point, std::forward<ArgsTuple>(args))));\n}\n\n// lref_to_cref is a helper that encapsulates lvalue reference in std::ref() or does nothing otherwise\ntemplate<typename T>\nauto lref_to_cref(T&& x) {\n    return std::move(x);\n}\n\ntemplate<typename T>\nauto lref_to_cref(T& x) {\n    return std::ref(x);\n}\n\n// Creates lambda to handle RPC message on a server.\n// The lambda unmarshalls all parameters, calls a handler, marshall return values and sends them back to a client\ntemplate <typename Serializer, typename Func, typename Ret, typename... InArgs, typename WantClientInfo, typename WantTimePoint>\nauto recv_helper(uint64_t verb, signature<Ret (InArgs...)> sig, Func&& func, WantClientInfo, WantTimePoint) {\n    using signature = decltype(sig);\n    using wait_style = wait_signature_t<Ret>;\n    return [verb, func = lref_to_cref(std::forward<Func>(func))](shared_ptr<server::connection> client,\n                                                                 std::optional<rpc_clock_type::time_point> timeout,\n                                                                 int64_t msg_id,\n                                                                 rcv_buf data,\n                                                                 gate::holder guard) mutable {\n        auto memory_consumed = client->estimate_request_size(data.size);\n        if (memory_consumed > client->max_request_size()) {\n            auto err = format(\"request size {:d} large than memory limit {:d}, verb {}\", memory_consumed, client->max_request_size(), verb);\n            client->get_logger()(client->peer_address(), err);\n            // FIXME: future is discarded\n            (void)try_with_gate(client->get_server().reply_gate(), [verb, client, timeout, msg_id, err = std::move(err)] {\n                return reply<Serializer>(wait_style(), futurize<Ret>::make_exception_future(std::runtime_error(err.c_str())), verb, msg_id, client, timeout, std::nullopt).handle_exception([verb, client, msg_id] (std::exception_ptr eptr) {\n                    client->get_logger()(client->info(), msg_id, seastar::format(\"got exception while processing an oversized message: {} for verb {}\", eptr, verb));\n                });\n            }).handle_exception_type([] (gate_closed_exception&) {/* ignore */});\n            return make_ready_future();\n        }\n        // note: apply is executed asynchronously with regards to networking so we cannot chain futures here by doing \"return apply()\"\n        auto f = client->wait_for_resources(memory_consumed, timeout).then([verb, client, timeout, msg_id, data = std::move(data), &func, g = std::move(guard)] (auto permit) mutable {\n                // FIXME: future is discarded\n                (void)try_with_gate(client->get_server().reply_gate(), [verb, client, timeout, msg_id, data = std::move(data), permit = std::move(permit), &func] () mutable {\n                    try {\n                        auto args = unmarshall<Serializer, InArgs...>(*client, std::move(data));\n                        auto start = rpc_clock_type::now();\n                        return apply(func, client->info(), timeout, WantClientInfo(), WantTimePoint(), signature(), std::move(args)).then_wrapped([verb, client, timeout, msg_id, permit = std::move(permit), start] (futurize_t<Ret> ret) mutable {\n                            return reply<Serializer>(wait_style(), std::move(ret), verb, msg_id, client, timeout, rpc_clock_type::now() - start).handle_exception([verb, permit = std::move(permit), client, msg_id] (std::exception_ptr eptr) {\n                                client->get_logger()(client->info(), msg_id, seastar::format(\"got exception while processing a message: {}, verb {}\", eptr, verb));\n                            });\n                        });\n                    } catch (...) {\n                        client->get_logger()(client->info(), msg_id, seastar::format(\"caught exception while processing a message: {}, verb {}\", std::current_exception(), verb));\n                        return make_ready_future();\n                    }\n                }).handle_exception_type([g = std::move(g)] (gate_closed_exception&) {/* ignore */});\n        });\n\n        if (timeout) {\n            f = f.handle_exception_type([] (semaphore_timed_out&) { /* ignore */ });\n        }\n\n        return f;\n    };\n}\n\n// helper to create copy constructible lambda from non copy constructible one. std::function<> works only with former kind.\ntemplate<typename Func>\nauto make_copyable_function(Func&& func) {\n  auto p = make_lw_shared<typename std::decay_t<Func>>(std::forward<Func>(func));\n  return [p] (auto&&... args) { return (*p)( std::forward<decltype(args)>(args)... ); };\n}\n\ntemplate<typename Func>\nrequires std::copy_constructible<std::decay_t<Func>>\nauto make_copyable_function(Func&& func) {\n    return std::forward<Func>(func);\n}\n\n// This class is used to calculate client side rpc function signature.\n// Return type is converted from a smart pointer to a type it points to.\n// rpc::optional are converted to non optional type.\n//\n// Examples:\n// std::unique_ptr<int>(int, rpc::optional<long>) -> int(int, long)\n// double(float) -> double(float)\ntemplate<typename Ret, typename... In>\nclass client_function_type {\n    template<typename T, bool IsSmartPtr>\n    struct drop_smart_ptr_impl;\n    template<typename T>\n    struct drop_smart_ptr_impl<T, true> {\n        using type = typename T::element_type;\n    };\n    template<typename T>\n    struct drop_smart_ptr_impl<T, false> {\n        using type = T;\n    };\n    template<typename T>\n    using drop_smart_ptr = drop_smart_ptr_impl<T, is_smart_ptr<T>::value>;\n\n    // if return type is smart ptr take a type it points to instead\n    using return_type = typename drop_smart_ptr<Ret>::type;\npublic:\n    using type = return_type(typename remove_optional<In>::type...);\n};\n\ntemplate<typename Serializer, typename MsgType>\ntemplate<typename Ret, typename... In>\nauto protocol<Serializer, MsgType>::make_client(signature<Ret(In...)>, MsgType t) {\n    using sig_type = signature<typename client_function_type<Ret, In...>::type>;\n    return send_helper<Serializer>(t, sig_type());\n}\n\ntemplate<typename Serializer, typename MsgType>\ntemplate<typename Func>\nauto protocol<Serializer, MsgType>::make_client(MsgType t) {\n    return make_client(typename signature<typename function_traits<Func>::signature>::clean(), t);\n}\n\ntemplate<typename Serializer, typename MsgType>\ntemplate<typename Func>\nauto protocol<Serializer, MsgType>::register_handler(MsgType t, scheduling_group sg, Func&& func) {\n    using sig_type = signature<typename function_traits<Func>::signature>;\n    using clean_sig_type = typename sig_type::clean;\n    using want_client_info = typename sig_type::want_client_info;\n    using want_time_point = typename sig_type::want_time_point;\n    auto recv = recv_helper<Serializer>(static_cast<uint64_t>(t), clean_sig_type(), std::forward<Func>(func),\n            want_client_info(), want_time_point());\n    register_receiver(t, rpc_handler{sg, make_copyable_function(std::move(recv)), {}});\n    return make_client(clean_sig_type(), t);\n}\n\ntemplate<typename Serializer, typename MsgType>\ntemplate<typename Func>\nauto protocol<Serializer, MsgType>::register_handler(MsgType t, Func&& func) {\n    return register_handler(t, scheduling_group(), std::forward<Func>(func));\n}\n\ntemplate<typename Serializer, typename MsgType>\nfuture<> protocol<Serializer, MsgType>::unregister_handler(MsgType t) {\n    auto it = _handlers.find(t);\n    if (it != _handlers.end()) {\n        return it->second.use_gate.close().finally([this, t] {\n            _handlers.erase(t);\n        });\n    }\n    return make_ready_future<>();\n}\n\ntemplate<typename Serializer, typename MsgType>\nbool protocol<Serializer, MsgType>::has_handler(MsgType msg_id) {\n    auto it = _handlers.find(msg_id);\n    if (it == _handlers.end()) {\n        return false;\n    }\n    return !it->second.use_gate.is_closed();\n}\n\ntemplate<typename Serializer, typename MsgType>\nstd::optional<protocol_base::handler_with_holder> protocol<Serializer, MsgType>::get_handler(uint64_t msg_id) {\n    const auto it = _handlers.find(MsgType(msg_id));\n    if (it != _handlers.end()) {\n        try {\n            return handler_with_holder{it->second, it->second.use_gate.hold()};\n        } catch (gate_closed_exception&) {\n            // unregistered, just ignore\n        }\n    }\n    return std::nullopt;\n}\n\nnamespace internal {\n\ntemplate<typename Serializer, typename... Out>\nsink_impl<Serializer, Out...>::sink_impl(xshard_connection_ptr con)\n    : sink<Out...>::impl(std::move(con))\n    , _send_queue([this] (snd_buf* buf) { return send_buffer(buf); }, this->_con->get_owner_shard())\n    , _delete_queue([] (snd_buf* buf) { delete buf; return make_ready_future<>(); }, this_shard_id())\n{\n    this->_con->get()->_sink_closed = false;\n}\n\nsnd_buf make_shard_local_buffer_copy(snd_buf* org, std::function<deleter(snd_buf*)> make_deleter);\n\n// Runs on connection shard\ntemplate<typename Serializer, typename... Out>\nfuture<> sink_impl<Serializer, Out...>::send_buffer(snd_buf* data) {\n    auto local_data = make_shard_local_buffer_copy(data, [this] (snd_buf* org) {\n        return deleter(new snd_buf_deleter_impl(org, _delete_queue));\n    });\n    // Exceptions are allowed from here since destroying local_data will free the original data buffer\n    if (this->_ex) {\n        return make_ready_future<>();\n    }\n    connection* con = this->_con->get();\n    // Keep first error in _ex, but make sure to drain the whole batch\n    // and destroy all queued buffers\n    if (con->error()) {\n        this->_ex = std::make_exception_ptr(closed_error());\n        return make_ready_future<>();\n    }\n    if (con->sink_closed()) {\n        this->_ex = std::make_exception_ptr(stream_closed());\n        return make_ready_future<>();\n    }\n\n    return con->send(std::move(local_data), {}, nullptr);\n}\n\ntemplate<typename Serializer, typename... Out>\nfuture<> sink_impl<Serializer, Out...>::operator()(const Out&... args) {\n    // note that we use remote serializer pointer, so if serailizer needs a state\n    // it should have per-cpu one\n    auto data = std::make_unique<snd_buf>(marshall(this->_con->get()->template serializer<Serializer>(), 4, args...));\n    static_assert(snd_buf::chunk_size >= 4, \"send buffer chunk size is too small\");\n    auto p = data->front().get_write();\n    write_le<uint32_t>(p, data->size - 4);\n    // we do not want to dead lock on huge packets, so let them in\n    // but only one at a time\n    auto size = std::min(size_t(data->size), max_stream_buffers_memory);\n    return get_units(this->_sem, size).then([this, data = std::move(data)] (semaphore_units<> su) mutable {\n        if (this->_ex) {\n            return make_exception_future(this->_ex);\n        }\n        data->su = std::move(su);\n        _send_queue.enqueue(data.get());\n        data.release();\n        return make_ready_future<>();\n    });\n}\n\ntemplate<typename Serializer, typename... Out>\nfuture<> sink_impl<Serializer, Out...>::flush() noexcept {\n    // wait until everything is sent out before returning.\n    return with_semaphore(this->_sem, max_stream_buffers_memory, [this] {\n        if (this->_ex) {\n            return make_exception_future(this->_ex);\n        }\n        return make_ready_future();\n    });\n}\n\ntemplate<typename Serializer, typename... Out>\nfuture<> sink_impl<Serializer, Out...>::close() noexcept {\n    return with_semaphore(this->_sem, max_stream_buffers_memory, [this] {\n        // break the semaphore to prevent any new messages to be sent\n        this->_sem.broken(stream_closed());\n        return _send_queue.stop().finally([this] {\n            return smp::submit_to(this->_con->get_owner_shard(), [this] {\n                return _delete_queue.stop().finally([this] {\n                    connection* con = this->_con->get();\n                    if (con->sink_closed()) { // double close, should not happen!\n                        return make_exception_future(stream_closed());\n                    }\n                    future<> f = make_ready_future<>();\n                    if (!con->error() && !this->_ex) {\n                        snd_buf data = marshall(con->template serializer<Serializer>(), 4);\n                        static_assert(snd_buf::chunk_size >= 4, \"send buffer chunk size is too small\");\n                        auto p = data.front().get_write();\n                        write_le<uint32_t>(p, -1U); // max len fragment marks an end of a stream\n                        f = con->send(std::move(data), {}, nullptr);\n                    } else {\n                        f = this->_ex ? make_exception_future(this->_ex) : make_exception_future(closed_error());\n                    }\n                    return f.finally([con] { return con->close_sink(); });\n                });\n            });\n        });\n    });\n}\n\ntemplate<typename Serializer, typename... Out>\nsink_impl<Serializer, Out...>::~sink_impl() {\n    // A failure to close might leave some continuations running after\n    // this is destroyed, leading to use-after-free bugs.\n    SEASTAR_ASSERT(this->_con->get()->sink_closed());\n}\n\nrcv_buf make_shard_local_buffer_copy(foreign_ptr<std::unique_ptr<rcv_buf>> org);\n\ntemplate<typename Serializer, typename... In>\nfuture<std::optional<std::tuple<In...>>> source_impl<Serializer, In...>::operator()() {\n    auto process_one_buffer = [this] {\n        foreign_ptr<std::unique_ptr<rcv_buf>> buf = std::move(this->_bufs.front());\n        this->_bufs.pop_front();\n        return std::apply([] (In&&... args) {\n            auto ret = std::make_optional(std::make_tuple(std::move(args)...));\n            return make_ready_future<std::optional<std::tuple<In...>>>(std::move(ret));\n        }, unmarshall<Serializer, In...>(*this->_con->get(), make_shard_local_buffer_copy(std::move(buf))));\n    };\n\n    if (!this->_bufs.empty()) {\n        return process_one_buffer();\n    }\n\n    // refill buffers from remote cpu\n    return smp::submit_to(this->_con->get_owner_shard(), [this] () -> future<> {\n        connection* con = this->_con->get();\n        if (con->_source_closed) {\n            return make_exception_future<>(stream_closed());\n        }\n        return con->stream_receive(this->_bufs).then_wrapped([this, con] (future<>&& f) {\n            if (f.failed()) {\n                return con->close_source().then_wrapped([ex = f.get_exception()] (future<> f){\n                    f.ignore_ready_future();\n                    return make_exception_future<>(ex);\n                });\n            }\n            if (this->_bufs.empty()) { // nothing to read -> eof\n                return con->close_source().then_wrapped([] (future<> f) {\n                    f.ignore_ready_future();\n                    return make_ready_future<>();\n                });\n            }\n            return make_ready_future<>();\n        });\n    }).then([this, process_one_buffer] () {\n        if (this->_bufs.empty()) {\n            return make_ready_future<std::optional<std::tuple<In...>>>(std::nullopt);\n        } else {\n            return process_one_buffer();\n        }\n    });\n}\n\n} // namespace internal\n\ntemplate<typename... Out>\nconnection_id sink<Out...>::get_id() const {\n    return _impl->_con->get()->get_connection_id();\n}\n\ntemplate<typename... In>\nconnection_id source<In...>::get_id() const {\n    return _impl->_con->get()->get_connection_id();\n}\n\ntemplate<typename... In>\ntemplate<typename Serializer, typename... Out>\nsink<Out...> source<In...>::make_sink() {\n    return sink<Out...>(make_shared<internal::sink_impl<Serializer, Out...>>(_impl->_con));\n}\n\n}\n\n}\n\nnamespace std {\ntemplate<>\nstruct hash<seastar::rpc::streaming_domain_type> {\n    size_t operator()(const seastar::rpc::streaming_domain_type& domain) const {\n        size_t h = 0;\n        boost::hash_combine(h, std::hash<uint64_t>{}(domain._id));\n        return h;\n    }\n};\n}\n\n\n"
  },
  {
    "path": "include/seastar/rpc/rpc_types.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <fmt/ostream.h>\n#if FMT_VERSION >= 100000\n#include <fmt/std.h>\n#endif\n\n#include <seastar/net/api.hh>\n#include <stdexcept>\n#include <string>\n#include <any>\n#include <boost/intrusive/slist.hpp>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/variant_utils.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/simple-stream.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <boost/functional/hash.hpp>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/semaphore.hh>\n\nnamespace seastar {\n\nnamespace rpc {\n\nusing rpc_clock_type = lowres_clock;\n\n// used to tag a type for serializers\ntemplate<typename T>\nusing type = std::type_identity<T>;\n\nstruct stats {\n    using counter_type = uint64_t;\n    counter_type replied = 0;\n    counter_type pending = 0;\n    counter_type exception_received = 0;\n    counter_type sent_messages = 0;\n    counter_type wait_reply = 0;\n    counter_type timeout = 0;\n    counter_type delay_samples = 0;\n    std::chrono::duration<double> delay_total = std::chrono::duration<double>(0);\n};\n\nclass connection_id {\n    uint64_t _id;\n\npublic:\n    uint64_t id() const {\n        return _id;\n    }\n    bool operator==(const connection_id& o) const {\n        return _id == o._id;\n    }\n    explicit operator bool() const {\n        return shard() != 0xffff;\n    }\n    size_t shard() const {\n        return size_t(_id & 0xffff);\n    }\n    constexpr static connection_id make_invalid_id(uint64_t _id = 0) {\n        return make_id(_id, 0xffff);\n    }\n    constexpr static connection_id make_id(uint64_t _id, uint16_t shard) {\n        return {_id << 16 | shard};\n    }\n    constexpr connection_id(uint64_t id) : _id(id) {}\n};\n\nconstexpr connection_id invalid_connection_id = connection_id::make_invalid_id();\n\nstd::ostream& operator<<(std::ostream&, const connection_id&);\n\nclass server;\n\nstruct client_info {\n    socket_address addr;\n    rpc::server& server;\n    connection_id conn_id;\n    std::unordered_map<sstring, std::any> user_data;\n    template <typename T>\n    void attach_auxiliary(const sstring& key, T&& object) {\n        user_data.emplace(key, std::any(std::forward<T>(object)));\n    }\n    template <typename T>\n    T& retrieve_auxiliary(const sstring& key) {\n        auto it = user_data.find(key);\n        SEASTAR_ASSERT(it != user_data.end());\n        return std::any_cast<T&>(it->second);\n    }\n    template <typename T>\n    std::add_const_t<T>& retrieve_auxiliary(const sstring& key) const {\n        return const_cast<client_info*>(this)->retrieve_auxiliary<std::add_const_t<T>>(key);\n    }\n    template <typename T>\n    T* retrieve_auxiliary_opt(const sstring& key) noexcept {\n        auto it = user_data.find(key);\n        if (it == user_data.end()) {\n            return nullptr;\n        }\n        return &std::any_cast<T&>(it->second);\n    }\n    template <typename T>\n    const T* retrieve_auxiliary_opt(const sstring& key) const noexcept {\n        auto it = user_data.find(key);\n        if (it == user_data.end()) {\n            return nullptr;\n        }\n        return &std::any_cast<const T&>(it->second);\n    }\n};\n\nclass error : public std::runtime_error {\npublic:\n    error(const std::string& msg) : std::runtime_error(msg) {}\n};\n\nclass closed_error : public error {\npublic:\n    closed_error() : error(\"connection is closed\") {}\n    closed_error(const std::string& msg) : error(msg) {}\n};\n\nclass timeout_error : public error {\npublic:\n    timeout_error() : error(\"rpc call timed out\") {}\n};\n\nclass unknown_verb_error : public error {\npublic:\n    uint64_t type;\n    unknown_verb_error(uint64_t type_) : error(\"unknown verb\"), type(type_) {}\n};\n\nclass unknown_exception_error : public error {\npublic:\n    unknown_exception_error() : error(\"unknown exception\") {}\n};\n\nclass rpc_protocol_error : public error {\npublic:\n    rpc_protocol_error() : error(\"rpc protocol exception\") {}\n};\n\nclass canceled_error : public error {\npublic:\n    canceled_error() : error(\"rpc call was canceled\") {}\n};\n\nclass stream_closed : public error {\npublic:\n    stream_closed() : error(\"rpc stream was closed by peer\") {}\n};\n\nclass remote_verb_error : public error {\n    using error::error;\n};\n\nstruct no_wait_type {};\n\n// return this from a callback if client does not want to waiting for a reply\nextern no_wait_type no_wait;\n\n/// \\addtogroup rpc\n/// @{\n\ntemplate <typename T>\nclass optional : public std::optional<T> {\npublic:\n     using std::optional<T>::optional;\n};\n\nclass opt_time_point : public std::optional<rpc_clock_type::time_point> {\npublic:\n     using std::optional<rpc_clock_type::time_point>::optional;\n     opt_time_point(std::optional<rpc_clock_type::time_point> time_point) {\n         static_cast<std::optional<rpc_clock_type::time_point>&>(*this) = time_point;\n     }\n};\n\n/// @}\n\nstruct cancellable {\n    std::function<void()> cancel_send;\n    std::function<void()> cancel_wait;\n    cancellable** send_back_pointer = nullptr;\n    cancellable** wait_back_pointer = nullptr;\n    cancellable() = default;\n    cancellable(cancellable&& x) : cancel_send(std::move(x.cancel_send)), cancel_wait(std::move(x.cancel_wait)), send_back_pointer(x.send_back_pointer), wait_back_pointer(x.wait_back_pointer) {\n        if (send_back_pointer) {\n            *send_back_pointer = this;\n            x.send_back_pointer = nullptr;\n        }\n        if (wait_back_pointer) {\n            *wait_back_pointer = this;\n            x.wait_back_pointer = nullptr;\n        }\n    }\n    cancellable& operator=(cancellable&& x) {\n        if (&x != this) {\n            this->~cancellable();\n            new (this) cancellable(std::move(x));\n        }\n        return *this;\n    }\n    void cancel() {\n        if (cancel_send) {\n            cancel_send();\n        }\n        if (cancel_wait) {\n            cancel_wait();\n        }\n    }\n    ~cancellable() {\n        cancel();\n    }\n};\n\nstruct rcv_buf {\n    uint32_t size = 0;\n    std::optional<semaphore_units<>> su;\n    std::variant<std::vector<temporary_buffer<char>>, temporary_buffer<char>> bufs;\n    using iterator = std::vector<temporary_buffer<char>>::iterator;\n    rcv_buf() {}\n    explicit rcv_buf(size_t size_) : size(size_) {}\n    explicit rcv_buf(temporary_buffer<char> b) : size(b.size()), bufs(std::move(b)) {};\n    explicit rcv_buf(std::vector<temporary_buffer<char>> bufs, size_t size)\n        : size(size), bufs(std::move(bufs)) {};\n};\n\nstruct snd_buf : public boost::intrusive::slist_base_hook<> {\n    // Preferred, but not required, chunk size.\n    static constexpr size_t chunk_size = 128*1024;\n    uint32_t size = 0;\n    std::variant<std::vector<temporary_buffer<char>>, temporary_buffer<char>> bufs;\n    // Holds semaphore units to extend backpressure lifetime until snd_buf is destroyed.\n    semaphore_units<> su;\n    using iterator = std::vector<temporary_buffer<char>>::iterator;\n    snd_buf() {}\n    snd_buf(snd_buf&&) noexcept;\n    snd_buf& operator=(snd_buf&&) noexcept;\n    explicit snd_buf(size_t size_);\n    explicit snd_buf(temporary_buffer<char> b) : size(b.size()), bufs(std::move(b)) {};\n\n    explicit snd_buf(std::vector<temporary_buffer<char>> bufs, size_t size)\n        : size(size), bufs(std::move(bufs)) {};\n\n    temporary_buffer<char>& front();\n};\n\nstatic inline memory_input_stream<rcv_buf::iterator> make_deserializer_stream(rcv_buf& input) {\n    auto* b = std::get_if<temporary_buffer<char>>(&input.bufs);\n    if (b) {\n        return memory_input_stream<rcv_buf::iterator>(memory_input_stream<rcv_buf::iterator>::simple(b->begin(), b->size()));\n    } else {\n        auto& ar = std::get<std::vector<temporary_buffer<char>>>(input.bufs);\n        return memory_input_stream<rcv_buf::iterator>(memory_input_stream<rcv_buf::iterator>::fragmented(ar.begin(), input.size));\n    }\n}\n\nclass compressor {\npublic:\n    virtual ~compressor() {}\n    // compress data and leave head_space bytes at the beginning of returned buffer\n    virtual snd_buf compress(size_t head_space, snd_buf data) = 0;\n    // decompress data\n    virtual rcv_buf decompress(rcv_buf data) = 0;\n    virtual sstring name() const = 0;\n    virtual future<> close() noexcept { return make_ready_future<>(); };\n\n    // factory to create compressor for a connection\n    class factory {\n    public:\n        virtual ~factory() {}\n        // return feature string that will be sent as part of protocol negotiation\n        virtual const sstring& supported() const = 0;\n        // negotiate compress algorithm\n        // send_empty_frame() requests an empty frame to be sent to the peer compressor on the other side of the connection.\n        // By attaching a header to this empty frame, the compressor can communicate somthing to the peer,\n        // send_empty_frame() mustn't be called from inside compress() or decompress().\n        virtual std::unique_ptr<compressor> negotiate(sstring feature, bool is_server, std::function<future<>()> send_empty_frame) const {\n            return negotiate(feature, is_server);\n        }\n        virtual std::unique_ptr<compressor> negotiate(sstring feature, bool is_server) const = 0;\n    };\n};\n\nclass connection;\n\nusing xshard_connection_ptr = lw_shared_ptr<foreign_ptr<shared_ptr<connection>>>;\nconstexpr size_t max_queued_stream_buffers = 50;\nconstexpr size_t max_stream_buffers_memory = 100 * 1024;\n\n/// \\addtogroup rpc\n/// @{\n\n// send data Out...\ntemplate<typename... Out>\nclass sink {\npublic:\n    class impl {\n    protected:\n        xshard_connection_ptr _con;\n        semaphore _sem;\n        std::exception_ptr _ex;\n        impl(xshard_connection_ptr con) : _con(std::move(con)), _sem(max_stream_buffers_memory) {}\n    public:\n        virtual ~impl() {};\n        virtual future<> operator()(const Out&... args) = 0;\n        // Failures may be returned as an exceptional future\n        virtual future<> close() noexcept = 0;\n        virtual future<> flush() noexcept = 0;\n        friend sink;\n    };\n\nprivate:\n    shared_ptr<impl> _impl;\n\npublic:\n    sink(shared_ptr<impl> impl) : _impl(std::move(impl)) {}\n    future<> operator()(const Out&... args) {\n        return _impl->operator()(args...);\n    }\n    // Failures may be returned as an exceptional future\n    future<> close() noexcept {\n        return _impl->close();\n    }\n    // Calling this function makes sure that any data buffered\n    // by the stream sink will be flushed to the network.\n    // It does not mean the data was received by the corresponding\n    // source.\n    future<> flush() noexcept {\n        return _impl->flush();\n    }\n    connection_id get_id() const;\n};\n\n// receive data In...\ntemplate<typename... In>\nclass source {\npublic:\n    class impl {\n    protected:\n        xshard_connection_ptr _con;\n        circular_buffer<foreign_ptr<std::unique_ptr<rcv_buf>>> _bufs;\n        impl(xshard_connection_ptr con) : _con(std::move(con)) {\n            _bufs.reserve(max_queued_stream_buffers);\n        }\n    public:\n        virtual ~impl() {}\n        virtual future<std::optional<std::tuple<In...>>> operator()() = 0;\n        friend source;\n    };\nprivate:\n    shared_ptr<impl> _impl;\n\npublic:\n    source(shared_ptr<impl> impl) : _impl(std::move(impl)) {}\n    future<std::optional<std::tuple<In...>>> operator()() {\n        return _impl->operator()();\n    };\n    connection_id get_id() const;\n    template<typename Serializer, typename... Out> sink<Out...> make_sink();\n};\n\n/// Used to return multiple values in rpc without variadic futures\n///\n/// If you wish to return multiple values from an rpc procedure, use a\n/// signature `future<rpc::tuple<return type list> (argument list)>>`. This\n/// will be marshalled by rpc, so you do not need to have your Serializer\n/// serialize/deserialize this tuple type. The serialization format is\n/// compatible with the deprecated variadic future support, and is compatible\n/// with adding new return types in a backwards compatible way provided new\n/// parameters are appended only, and wrapped with rpc::optional:\n/// `future<rpc::tuple<existing return type list, rpc::optional<new_return_type>>> (argument list)`\n///\n/// You may also use another tuple type, such as std::tuple. In this case,\n/// your Serializer type must recognize your tuple type and provide serialization\n/// and deserialization for it.\ntemplate <typename... T>\nclass tuple : public std::tuple<T...> {\npublic:\n    using std::tuple<T...>::tuple;\n    tuple(std::tuple<T...>&& x) : std::tuple<T...>(std::move(x)) {}\n};\n\n/// @}\n\n#ifndef SEASTAR_P2581R1\ntemplate <typename... T>\ntuple(T&&...) ->  tuple<T...>;\n#endif\n\n} // namespace rpc\n\n}\n\nnamespace std {\ntemplate<>\nstruct hash<seastar::rpc::connection_id> {\n    size_t operator()(const seastar::rpc::connection_id& id) const {\n        size_t h = 0;\n        boost::hash_combine(h, std::hash<uint64_t>{}(id.id()));\n        return h;\n    }\n};\n\ntemplate <typename... T>\nstruct tuple_size<seastar::rpc::tuple<T...>> : tuple_size<tuple<T...>> {\n};\n\ntemplate <size_t I, typename... T>\nstruct tuple_element<I, seastar::rpc::tuple<T...>> : tuple_element<I, tuple<T...>> {\n};\n\n}\n\ntemplate <> struct fmt::formatter<seastar::rpc::connection_id> : fmt::ostream_formatter {};\n\n#if FMT_VERSION < 100000\n// fmt v10 introduced formatter for std::exception\ntemplate <std::derived_from<seastar::rpc::error> T>\nstruct fmt::formatter<T> : fmt::formatter<string_view> {\n    auto format(const T& e, fmt::format_context& ctx) const {\n        return fmt::format_to(ctx.out(), \"{}\", e.what());\n    }\n};\n#endif\n\n#if FMT_VERSION < 100000\ntemplate <typename T>\nstruct fmt::formatter<seastar::rpc::optional<T>> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::rpc::optional<T>& opt, fmt::format_context& ctx) const {\n        if (opt) {\n            return fmt::format_to(ctx.out(), \"optional({})\", *opt);\n        } else {\n            return fmt::format_to(ctx.out(), \"none\");\n        }\n    }\n};\n#else\ntemplate <typename T>\nstruct fmt::formatter<seastar::rpc::optional<T>> : private fmt::formatter<std::optional<T>> {\n    using fmt::formatter<std::optional<T>>::parse;\n    auto format(const seastar::rpc::optional<T>& opt, fmt::format_context& ctx) const {\n        return fmt::formatter<std::optional<T>>::format(opt, ctx);\n    }\n};\n#endif\n"
  },
  {
    "path": "include/seastar/testing/entry_point.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#pragma once\n\nnamespace seastar {\n\nnamespace testing {\n\nint entry_point(int argc, char **argv);\n\n}\n\n}"
  },
  {
    "path": "include/seastar/testing/exchanger.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <mutex>\n#include <condition_variable>\n#include <optional>\n\nnamespace seastar {\n\nnamespace testing {\n\nclass exchanger_base {\nprotected:\n    std::mutex _mutex;\n    std::condition_variable _cv;\n    std::exception_ptr _exception;\n    void interrupt_ptr(std::exception_ptr e) {\n        std::unique_lock<std::mutex> lock(_mutex);\n        if (!_exception) {\n            _exception = e;\n            _cv.notify_all();\n        }\n        // FIXME: log if already interrupted\n    }\n};\n\n// Single-element blocking queue\ntemplate <typename T>\nclass exchanger : public exchanger_base {\nprivate:\n    std::optional<T> _element;\n\npublic:\n    template <typename Exception>\n    void interrupt(Exception e) {\n        try {\n            throw e;\n        } catch (...) {\n            interrupt_ptr(std::current_exception());\n        }\n    }\n    void give(T value) {\n        std::unique_lock<std::mutex> lock(_mutex);\n        _cv.wait(lock, [this] { return !_element || _exception; });\n        if (_exception) {\n            std::rethrow_exception(_exception);\n        }\n        _element = value;\n        _cv.notify_one();\n    }\n    T take() {\n        std::unique_lock<std::mutex> lock(_mutex);\n        _cv.wait(lock, [this] { return bool(_element) || _exception; });\n        if (_exception) {\n            std::rethrow_exception(_exception);\n        }\n        auto v = *_element;\n        _element = {};\n        _cv.notify_one();\n        return v;\n    }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/testing/linux_perf_event.hh",
    "content": "/*\n * Copyright (C) 2021-present ScyllaDB\n */\n\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * This file was copied from Scylla (https://github.com/scylladb/scylla)\n */\n\n#pragma once\n\n\n#include <cstdint>\n#include <utility>\n#include <unistd.h>\n\nstruct perf_event_attr; // from <linux/perf_event.h>\n\nclass linux_perf_event {\n    int _fd = -1;\n    linux_perf_event() = default;\npublic:\n    linux_perf_event(const struct ::perf_event_attr& attr, pid_t pid, int cpu, int group_fd, unsigned long flags);\n    linux_perf_event(linux_perf_event&& x) noexcept : _fd(std::exchange(x._fd, -1)) {}\n    linux_perf_event& operator=(linux_perf_event&& x) noexcept;\n    ~linux_perf_event();\n    uint64_t read();\n    void enable();\n    void disable();\npublic:\n    /// Returns a no-op event that always reads as zero.\n    static linux_perf_event always_zero() { return linux_perf_event(); }\n    static linux_perf_event user_instructions_retired();\n    static linux_perf_event user_cpu_cycles_retired();\n};\n\n"
  },
  {
    "path": "include/seastar/testing/on_internal_error.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar {\nnamespace testing {\n\n// Disables aborting in on_internal_error() for a scope.\n//\n// Intended for tests, which want to test error paths that invoke\n// on_internal_error() without aborting, at the same time, having it enabled\n// for other, indirectly affected code paths, that are not a direct target of\n// the test.\nclass scoped_no_abort_on_internal_error {\n    bool _prev;\npublic:\n    scoped_no_abort_on_internal_error() noexcept;\n    ~scoped_no_abort_on_internal_error();\n};\n\n}\n}\n"
  },
  {
    "path": "include/seastar/testing/perf_tests.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <atomic>\n#include <memory>\n\n#include <fmt/format.h>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/testing/linux_perf_event.hh>\n#include <seastar/util/noncopyable_function.hh>\n\nusing namespace seastar;\n\nnamespace perf_tests {\n\n// The type of the pre-run hook. See PERF_PRE_RUN_HOOK.\nusing pre_run_hook = noncopyable_function<void(const sstring& test_group, const sstring& test_case)>;\n\nnamespace internal {\n\nstruct config;\n\nusing clock_type = std::chrono::steady_clock;\n\nclass perf_stats {\npublic:\n    uint64_t allocations = 0;\n    uint64_t tasks_executed = 0;\n    uint64_t instructions_retired = 0;\n    uint64_t cpu_cycles_retired = 0;\n\nprivate:\n    static uint64_t perf_mallocs();\n    static uint64_t perf_tasks_processed();\n\npublic:\n    perf_stats() = default;\n    perf_stats(uint64_t allocations_, uint64_t tasks_executed_, uint64_t instructions_retired_ = 0, uint64_t cpu_cycles_retired_ = 0)\n        : allocations(allocations_)\n        , tasks_executed(tasks_executed_)\n        , instructions_retired(instructions_retired_)\n        , cpu_cycles_retired(cpu_cycles_retired_)\n    {}\n    perf_stats(perf_stats&& o) noexcept\n        : allocations(std::exchange(o.allocations, 0))\n        , tasks_executed(std::exchange(o.tasks_executed, 0))\n        , instructions_retired(std::exchange(o.instructions_retired, 0))\n        , cpu_cycles_retired(std::exchange(o.cpu_cycles_retired, 0))\n    {}\n    perf_stats(const perf_stats& o) = default;\n\n    perf_stats& operator=(perf_stats&& o) = default;\n    perf_stats& operator=(const perf_stats& o) = default;\n\n    perf_stats& operator+=(perf_stats b);\n    perf_stats& operator-=(perf_stats b);\n\n    static perf_stats snapshot(linux_perf_event* instructions_retired_counter = nullptr, linux_perf_event* cpu_cycles_retired_counter = nullptr);\n};\n\ninline\nperf_stats\noperator+(perf_stats a, perf_stats b) {\n    a.allocations += b.allocations;\n    a.tasks_executed += b.tasks_executed;\n    a.instructions_retired += b.instructions_retired;\n    a.cpu_cycles_retired += b.cpu_cycles_retired;\n    return a;\n}\n\ninline\nperf_stats\noperator-(perf_stats a, perf_stats b) {\n    a.allocations -= b.allocations;\n    a.tasks_executed -= b.tasks_executed;\n    a.instructions_retired -= b.instructions_retired;\n    a.cpu_cycles_retired -= b.cpu_cycles_retired;\n    return a;\n}\n\ninline perf_stats& perf_stats::operator+=(perf_stats b) {\n    allocations += b.allocations;\n    tasks_executed += b.tasks_executed;\n    instructions_retired += b.instructions_retired;\n    cpu_cycles_retired += b.cpu_cycles_retired;\n    return *this;\n}\n\ninline perf_stats& perf_stats::operator-=(perf_stats b) {\n    allocations -= b.allocations;\n    tasks_executed -= b.tasks_executed;\n    instructions_retired -= b.instructions_retired;\n    cpu_cycles_retired -= b.cpu_cycles_retired;\n    return *this;\n}\n\nclass performance_test {\n    std::string _test_case;\n    std::string _test_group;\n\n    uint64_t _single_run_iterations = 0;\n    std::atomic<uint64_t> _max_single_run_iterations;\nprivate:\n    void do_run(const config&);\npublic:\n    struct run_result {\n        clock_type::duration duration;\n        perf_stats stats;\n        uint64_t start_stop_count = 0;\n    };\nprotected:\n    [[gnu::always_inline]] [[gnu::hot]]\n    bool stop_iteration() const {\n        return _single_run_iterations >= _max_single_run_iterations.load(std::memory_order_relaxed);\n    }\n\n    [[gnu::always_inline]] [[gnu::hot]]\n    void next_iteration(size_t n) {\n        _single_run_iterations += n;\n    }\n\n    virtual void set_up() = 0;\n    virtual void tear_down() noexcept = 0;\n    virtual future<run_result> do_single_run() = 0;\n\n    // execute the pre-run hooks\n    void run_hooks();\n\n    // start/stop timing for a single run - called by do_single_run()\n    void start_run();\n    run_result stop_run();\npublic:\n    performance_test(const std::string& test_case, const std::string& test_group)\n        : _test_case(test_case)\n        , _test_group(test_group)\n    { }\n\n    virtual ~performance_test() = default;\n\n    const std::string& test_case() const { return _test_case; }\n    const std::string& test_group() const { return _test_group; }\n    std::string name() const { return fmt::format(\"{}.{}\", test_group(), test_case()); }\n\n    void run(const config&);\npublic:\n    static void register_test(std::unique_ptr<performance_test>);\n};\n\n// Functions for time measurement iteration control - implementation in perf_tests.cc\nvoid time_measurement_start_iteration();\nvoid time_measurement_stop_iteration();\n\ntemplate<typename Test>\nclass concrete_performance_test final : public performance_test {\n    std::optional<Test> _test;\n\n    using test_ret_type = decltype(_test->run());\n    // true iff the test method returns future<...>\n    static constexpr bool is_async_test = is_future<test_ret_type>::value;\n    // true iff the test returns the number of iterations run, otherwise it returns\n    // void and we consider each invocation to be 1 iteration\n    static constexpr bool is_iteration_returning = !(std::is_same_v<test_ret_type, future<>> || std::is_void_v<test_ret_type>);\nprivate:\n\nprotected:\n    virtual void set_up() override {\n        _test.emplace();\n    }\n\n    virtual void tear_down() noexcept override {\n        _test = std::nullopt;\n    }\n\n    [[gnu::hot]]\n    virtual future<run_result> do_single_run() override {\n        run_hooks();\n        start_run();\n        while (!stop_iteration()) {\n            if constexpr (is_async_test) {\n                if constexpr (is_iteration_returning) {\n                    auto f = _test->run();\n                    next_iteration(f.available() ? std::move(f).get() : co_await std::move(f));\n                } else {\n                    auto f = _test->run();\n                    // The available() check is functionally redundant, but is significantly faster\n                    // than invoking the co_await machinery on a future-returning function.\n                    if (!f.available()) {\n                        co_await std::move(f);\n                    }\n                    next_iteration(1);\n                }\n            } else {\n                if constexpr (is_iteration_returning) {\n                    next_iteration(_test->run());\n                } else {\n                    _test->run();\n                    next_iteration(1);\n                }\n            }\n        }\n        co_return stop_run();\n    }\npublic:\n    using performance_test::performance_test;\n};\n\ntemplate<typename Test>\nstruct test_registrar {\n    test_registrar(const std::string& test_group, const std::string& test_case) {\n        auto test = std::make_unique<concrete_performance_test<Test>>(test_case, test_group);\n        performance_test::register_test(std::move(test));\n    }\n};\n\n\n// internal hook registration function\nint register_pre_run_hook(pre_run_hook hook);\n\n}\n\n[[gnu::always_inline]]\ninline void start_measuring_time()\n{\n    internal::time_measurement_start_iteration();\n}\n\n[[gnu::always_inline]]\ninline void stop_measuring_time()\n{\n    internal::time_measurement_stop_iteration();\n}\n\n\ntemplate<typename T>\nvoid do_not_optimize(const T& v)\n{\n    asm volatile(\"\" : : \"r,m\" (v));\n}\n\n}\n\n// PERF_TEST and PERF_TEST_F support both synchronous and asynchronous functions.\n// The former should return `void`, the latter `future<>`.\n// PERF_TEST_C executes a coroutine function, if enabled.\n// PERF_TEST_CN executes a coroutine function, if enabled, returning the number of inner-loops.\n//\n// Test cases may perform multiple operations in a single run, this may be desirable\n// if the cost of an individual operation is very small. This allows measuring either\n// the latency of throughput depending on how the test in written. In such cases,\n// the test function shall return either size_t or future<size_t> for synchronous and\n// asynchronous cases respectively. The returned value shall be the number of iterations\n// done in a single test run.\n\n#define PERF_TEST_F(test_group, test_case) \\\n    struct test_##test_group##_##test_case : test_group { \\\n        [[gnu::always_inline]] inline auto run(); \\\n    }; \\\n    static ::perf_tests::internal::test_registrar<test_##test_group##_##test_case> \\\n    test_##test_group##_##test_case##_registrar(#test_group, #test_case); \\\n    [[gnu::always_inline]] auto test_##test_group##_##test_case::run()\n\n#define PERF_TEST(test_group, test_case) \\\n    struct test_##test_group##_##test_case { \\\n        [[gnu::always_inline]] inline auto run(); \\\n    }; \\\n    static ::perf_tests::internal::test_registrar<test_##test_group##_##test_case> \\\n    test_##test_group##_##test_case##_registrar(#test_group, #test_case); \\\n    [[gnu::always_inline]] auto test_##test_group##_##test_case::run()\n\n\n#define PERF_TEST_C(test_group, test_case) \\\n    struct test_##test_group##_##test_case : test_group { \\\n        inline future<> run(); \\\n    }; \\\n    static ::perf_tests::internal::test_registrar<test_##test_group##_##test_case> \\\n    test_##test_group##_##test_case##_registrar(#test_group, #test_case); \\\n    future<> test_##test_group##_##test_case::run()\n\n#define PERF_TEST_CN(test_group, test_case) \\\n    struct test_##test_group##_##test_case : test_group { \\\n        inline future<size_t> run(); \\\n    }; \\\n    static ::perf_tests::internal::test_registrar<test_##test_group##_##test_case> \\\n    test_##test_group##_##test_case##_registrar(#test_group, #test_case); \\\n    future<size_t> test_##test_group##_##test_case::run()\n\n\n#define CONCAT_IMPL(a, b) a##b\n#define CONCAT(a, b) CONCAT_IMPL(a, b)\n\n// Installs a pre-run hook.\n//\n// Register a function to be called before each test run. These are called before\n// all runs include the \"dry run\" used to estimate the number of iterations, if\n// that is used.\n//\n// Multiple hooks may be registered, they will be called in order of registration.\n//\n// Hooks cannot be deregistered.\n#define PERF_PRE_RUN_HOOK(hook) \\\n    static int CONCAT(hook_registrar_, __COUNTER__) = ::perf_tests::internal::register_pre_run_hook(hook);\n"
  },
  {
    "path": "include/seastar/testing/random.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <random>\n\nnamespace seastar {\n\nnamespace testing {\n\nextern thread_local std::default_random_engine local_random_engine;\n\n} // namespace testing\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/testing/seastar_test.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <exception>\n#include <string_view>\n#include <boost/test/unit_test.hpp> // IWYU pragma: export\n#include <seastar/core/future.hh>\n#include <seastar/testing/entry_point.hh> // IWYU pragma: keep\n\n#define SEASTAR_TEST_INVOKE(func, ...) func(__VA_ARGS__)\n\nnamespace seastar {\n\nnamespace testing {\n\nclass seastar_test {\n    const std::string _test_file;\npublic:\n    seastar_test(const char* test_name, const char* test_file, int test_line);\n    seastar_test(const char* test_name, const char* test_file, int test_line,\n                 boost::unit_test::decorator::collector_t& decorators);\n    virtual ~seastar_test() {}\n    const std::string& get_test_file() const {\n        return _test_file;\n    }\n    static const std::string& get_name();\n    virtual future<> run_test_case() const = 0;\n    void run();\n};\n\n// BOOST_REQUIRE_EXCEPTION predicates\nnamespace exception_predicate {\n\nstd::function<bool(const std::exception&)> message_equals(std::string_view expected_message);\nstd::function<bool(const std::exception&)> message_contains(std::string_view expected_message);\n\n} // exception_predicate\n\n}\n\n}\n\n#ifdef SEASTAR_TESTING_MAIN\n\nint main(int argc, char** argv) {\n    return seastar::testing::entry_point(argc, argv);\n}\n\n#endif // SEASTAR_TESTING_MAIN\n"
  },
  {
    "path": "include/seastar/testing/test_case.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <boost/preprocessor/control/iif.hpp>\n#include <boost/preprocessor/comparison/equal.hpp>\n#include <boost/preprocessor/variadic/size.hpp>\n\n#include <seastar/core/future.hh> // IWYU pragma: keep\n\n#include <seastar/testing/seastar_test.hh>\n\n#define SEASTAR_TEST_CASE_WITH_DECO(name, decorators)               \\\n    struct name : public seastar::testing::seastar_test {           \\\n        using seastar::testing::seastar_test::seastar_test;         \\\n        seastar::future<> run_test_case() const override;           \\\n    };                                                              \\\n    static const name name ## _instance(                            \\\n        #name,                                                      \\\n        __FILE__,                                                   \\\n        __LINE__,                                                   \\\n        decorators); /* NOLINT(cert-err58-cpp) */                   \\\n    seastar::future<> name::run_test_case() const\n\n#define SEASTAR_TEST_CASE_WITHOUT_DECO(name)                        \\\n    SEASTAR_TEST_CASE_WITH_DECO(                                    \\\n        name,                                                       \\\n        boost::unit_test::decorator::collector_t::instance())\n\n#define SEASTAR_TEST_CASE(...)                                      \\\n    SEASTAR_TEST_INVOKE(                                            \\\n        BOOST_PP_IIF(                                               \\\n            BOOST_PP_EQUAL(BOOST_PP_VARIADIC_SIZE(__VA_ARGS__), 1), \\\n            SEASTAR_TEST_CASE_WITHOUT_DECO,                         \\\n            SEASTAR_TEST_CASE_WITH_DECO),                           \\\n        __VA_ARGS__)\n"
  },
  {
    "path": "include/seastar/testing/test_fixture.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2025 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <functional>\n\n#include <seastar/core/future.hh> // IWYU pragma: keep\n#include <seastar/core/thread.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/testing/seastar_test.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/util/defer.hh>\n\nnamespace seastar::testing {\nnamespace detail {\n\n    template<typename T>\nfuture<> conditional_invoke_setup(T& t) {\n    if constexpr (boost::unit_test::impl_fixture::has_setup<T>::value) {\n        return futurize_invoke(std::mem_fn(&T::setup), t);\n    }\n    return make_ready_future<>();\n}\n\ntemplate<typename T>\nfuture<> conditional_invoke_teardown(T& t) {\n    if constexpr (boost::unit_test::impl_fixture::has_teardown<T>::value) {\n        return futurize_invoke(std::mem_fn(&T::teardown), t);\n    }\n    return make_ready_future<>();\n}\nvoid warn_teardown_exception(const sstring&, std::exception_ptr);\n\ntemplate<typename Func, typename T>\nrequires std::is_invocable_v<Func, T>\nfuture<> do_fixture_test(Func&& f, T& t, const sstring& name) {\n    return conditional_invoke_setup(t).then([f = std::forward<Func>(f), &t]() mutable {\n        return futurize_invoke(std::forward<Func>(f), t);\n    }).finally([&]() {\n        return conditional_invoke_teardown(t).handle_exception([&](auto ep) {\n            warn_teardown_exception(name, ep);\n        });\n    });\n}\n} // detail\n\n// holder for decorator based fixture using class\ntemplate<typename F>\nclass async_class_based_fixture : public boost::unit_test::test_unit_fixture {\npublic:\n    template<typename... Args>\n    requires std::is_constructible_v<F, const Args&...>\n    async_class_based_fixture(Args&& ...args)\n        : _create([args = std::make_tuple(std::forward<Args>(args)...)] {\n            return std::apply([](auto&& ...args) {\n                return std::make_unique<F>(args...);\n            }, args);\n        })\n    {}\nprivate:\n    // Fixture interface\n    void setup() override {\n        // create and possibly init the fixture object in reactor, on the\n        // test thread.\n        global_test_runner().run_sync([this] {\n            _inst = _create();\n            return detail::conditional_invoke_setup(*_inst);\n        });\n    }\n    void teardown() override {\n        // possibly de-init and destroy the fixture object in reactor, on the\n        // test thread.\n        global_test_runner().run_sync([this] {\n            auto inst = std::exchange(_inst, {});\n            return detail::conditional_invoke_teardown(*inst).finally([inst = std::move(inst)] {});\n        });\n    }\n    // Data members\n    std::unique_ptr<F> _inst;\n    std::function<std::unique_ptr<F>()> _create;\n};\n\nusing fixture_function = std::function<future<>()>;\n\n// holder for decorator based fixture using functions\nclass async_function_based_fixture : public boost::unit_test::test_unit_fixture {\npublic:\n    async_function_based_fixture(fixture_function setup, fixture_function teardown)\n        : _setup(std::move(setup))\n        , _teardown(std::move(teardown))\n    {}\nprivate:\n    // Fixture interface\n    void setup() override {\n        if (_setup) {\n            // run in reactor thread\n            global_test_runner().run_sync([this] {\n                return _setup();\n            });\n        }\n    }\n    void teardown() override {\n        if (_teardown) {\n            // run in reactor thread\n            global_test_runner().run_sync([this] {\n                return _teardown();\n            });\n        }\n    }\n    fixture_function _setup, _teardown;\n};\n\n// our versions of \"fixture\" in boost. only differs in decorator used\ntemplate<typename F, typename... Args>\nrequires std::is_constructible_v<F, const Args&...>\ninline boost::unit_test::decorator::fixture_t async_fixture(Args&& ...args) {\n    return boost::unit_test::decorator::fixture_t(\n        boost::unit_test::test_unit_fixture_ptr(new async_class_based_fixture<F>(std::forward<Args>(args)...))\n    );\n}\ninline boost::unit_test::decorator::fixture_t async_fixture(fixture_function setup, fixture_function teardown) {\n    return boost::unit_test::decorator::fixture_t(\n        boost::unit_test::test_unit_fixture_ptr(new async_function_based_fixture(std::move(setup), std::move(teardown)))\n    );\n}\ntemplate<typename F1, typename F2>\nrequires (std::is_invocable_v<F1> && std::is_invocable_v<F2>)\ninline boost::unit_test::decorator::fixture_t async_func_fixture(F1 setup, F2 teardown) {\n    return boost::unit_test::decorator::fixture_t(\n        boost::unit_test::test_unit_fixture_ptr(new async_function_based_fixture(std::move(setup), std::move(teardown)))\n    );\n}\n}\n\n// macro for using a base-class type fixture with async setup/teardown\n// this just piggy-backs on normal seastar test case.\n#define SEASTAR_FIXTURE_TEST_CASE(name, F, ...)                     \\\n    struct name ## _fxt : public F {                                \\\n        seastar::future<> run_test_case() const;                    \\\n    };                                                              \\\n    SEASTAR_TEST_CASE(name, ##__VA_ARGS__) {                        \\\n        namespace td =  seastar::testing::detail;                   \\\n        using type = name ## _fxt;                                  \\\n        auto t = std::make_unique<type>();                          \\\n        return td::do_fixture_test(                                 \\\n                std::mem_fn(&type::run_test_case), *t, #F           \\\n            ).finally([t = std::move(t)] {});                       \\\n    }                                                               \\\n    seastar::future<> name ## _fxt::run_test_case() const\n\n#define SEASTAR_FIXTURE_THREAD_TEST_CASE(name, F, ...)              \\\n    struct name ## _fxt : public F {                                \\\n        void run_test_case() const;                                 \\\n    };                                                              \\\n    SEASTAR_TEST_CASE(name, ##__VA_ARGS__) {                        \\\n        return seastar::async([] {                                  \\\n            namespace td =  seastar::testing::detail;               \\\n            using type = name ## _fxt;                              \\\n            type t;                                                 \\\n            td::conditional_invoke_setup(t).get();                  \\\n            auto def = defer([&t]() noexcept {                      \\\n                try {                                               \\\n                    td::conditional_invoke_teardown(t).get();       \\\n                } catch (...) {                                     \\\n                    td::warn_teardown_exception(#name               \\\n                        , std::current_exception()                  \\\n                    );                                              \\\n                }                                                   \\\n            });                                                     \\\n            t.run_test_case();                                      \\\n        });                                                         \\\n    }                                                               \\\n    void name ## _fxt::run_test_case() const\n"
  },
  {
    "path": "include/seastar/testing/test_runner.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <memory>\n#include <functional>\n#include <atomic>\n#include <seastar/core/future.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/testing/exchanger.hh>\n\nnamespace seastar {\n\nnamespace testing {\n\nclass test_runner {\nprivate:\n    std::unique_ptr<posix_thread> _thread;\n    std::atomic<bool> _started{false};\n    exchanger<std::function<future<>()>> _task;\n    bool _done = false;\n    int _exit_code{0};\n\n    struct start_thread_args {\n        int ac;\n        char** av;\n        start_thread_args(int ac_, char** av_) noexcept : ac(ac_), av(av_) {}\n    };\n    std::unique_ptr<start_thread_args> _st_args;\n    bool start_thread(int ac, char** av);\npublic:\n    // Returns whether initialization was successful.\n    // Will return as soon as the seastar::app was started.\n    bool start(int argc, char** argv);\n    ~test_runner();\n    void run_sync(std::function<future<>()> task);\n    // Returns the return value of the underlying `seastar::app::run()`.\n    int finalize();\n};\n\ntest_runner& global_test_runner();\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/testing/thread_test_case.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#pragma once\n\n// IWYU pragma: begin_keep\n#include <seastar/core/future.hh>\n#include <seastar/core/thread.hh>\n// IWYU pragma: end_keep\n//\n#include <boost/preprocessor/control/iif.hpp>\n#include <boost/preprocessor/comparison/equal.hpp>\n#include <boost/preprocessor/variadic/size.hpp>\n\n#include <seastar/testing/seastar_test.hh>\n\n#define SEASTAR_THREAD_TEST_CASE_WITH_DECO(name, decorators) \\\n    struct name : public seastar::testing::seastar_test {    \\\n        using seastar::testing::seastar_test::seastar_test;  \\\n        seastar::future<> run_test_case() const override {   \\\n            return seastar::async([this] {                   \\\n                do_run_test_case();                          \\\n            });                                              \\\n        }                                                    \\\n        void do_run_test_case() const;                       \\\n    };                                                       \\\n    static const name name ## _instance(                     \\\n        #name,                                               \\\n        __FILE__,                                            \\\n        __LINE__,                                            \\\n        decorators); /* NOLINT(cert-err58-cpp) */            \\\n    void name::do_run_test_case() const\n\n#define SEASTAR_THREAD_TEST_CASE_WITHOUT_DECO(name)          \\\n    SEASTAR_THREAD_TEST_CASE_WITH_DECO(                      \\\n        name,                                                \\\n        boost::unit_test::decorator::collector_t::instance())\n\n#define SEASTAR_THREAD_TEST_CASE(...)                               \\\n    SEASTAR_TEST_INVOKE(                                            \\\n        BOOST_PP_IIF(                                               \\\n            BOOST_PP_EQUAL(BOOST_PP_VARIADIC_SIZE(__VA_ARGS__), 1), \\\n            SEASTAR_THREAD_TEST_CASE_WITHOUT_DECO,                  \\\n            SEASTAR_THREAD_TEST_CASE_WITH_DECO),                    \\\n        __VA_ARGS__)\n"
  },
  {
    "path": "include/seastar/util/alloc_failure_injector.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <limits>\n#include <cstdint>\n#include <functional>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/critical_alloc_section.hh>\n\nnamespace seastar {\nnamespace memory {\n\n///\n/// Allocation failure injection framework. Allows testing for exception safety.\n///\n/// To exhaustively inject failure at every allocation point:\n///\n///     uint64_t i = 0;\n///     while (true) {\n///         try {\n///             local_failure_injector().fail_after(i++);\n///             code_under_test();\n///             local_failure_injector().cancel();\n///             break;\n///         } catch (const std::bad_alloc&) {\n///             // expected\n///         }\n///     }\nclass alloc_failure_injector {\n    uint64_t _alloc_count = 0;\n    uint64_t _fail_at = std::numeric_limits<uint64_t>::max();\n    noncopyable_function<void()> _on_alloc_failure = [] { throw std::bad_alloc(); };\n    bool _failed = false;\nprivate:\n    void fail();\npublic:\n    /// \\brief Marks a point in code which should be considered for failure injection.\n    void on_alloc_point() {\n        if (is_critical_alloc_section()) {\n            return;\n        }\n        if (_alloc_count >= _fail_at) {\n            fail();\n        }\n        ++_alloc_count;\n    }\n\n    /// Counts encountered allocation points which didn't fail and didn't have failure suppressed.\n    uint64_t alloc_count() const {\n        return _alloc_count;\n    }\n\n    /// Will cause count-th allocation point from now to fail, counting from 0.\n    void fail_after(uint64_t count) {\n        _fail_at = _alloc_count + count;\n        _failed = false;\n    }\n\n    /// Cancels the failure scheduled by fail_after().\n    void cancel() {\n        _fail_at = std::numeric_limits<uint64_t>::max();\n    }\n\n    /// Returns true iff allocation was failed since last fail_after().\n    bool failed() const {\n        return _failed;\n    }\n\n    /// Runs given function with a custom failure action instead of the default std::bad_alloc throw.\n    void run_with_callback(noncopyable_function<void()> callback, noncopyable_function<void()> to_run);\n};\n\n/// \\cond internal\nextern thread_local alloc_failure_injector the_alloc_failure_injector;\n/// \\endcond\n\n/// \\brief Return the shard-local \\ref alloc_failure_injector instance.\ninline\nalloc_failure_injector& local_failure_injector() {\n    return the_alloc_failure_injector;\n}\n\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\n#ifdef SEASTAR_DEFAULT_ALLOCATOR\n#error SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION is not supported when using SEASTAR_DEFAULT_ALLOCATOR\n#endif\n\n#endif\n\n\nstruct [[deprecated(\"Use scoped_critical_section instead\")]] disable_failure_guard {\n    scoped_critical_alloc_section cs;\n};\n\n/// \\brief Marks a point in code which should be considered for failure injection.\ninline\nvoid on_alloc_point() {\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n    local_failure_injector().on_alloc_point();\n#endif\n}\n\n/// Repeatedly run func with allocation failures\n///\n/// Initially, allocations start to fail immediately. In each\n/// subsequent run the failures start one allocation later. This\n/// returns when func is run and no allocation failures are detected.\nvoid with_allocation_failures(noncopyable_function<void()> func);\n\n}\n}\n"
  },
  {
    "path": "include/seastar/util/assert.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 Redpanda Data.\n */\n\n#pragma once\n\nnamespace seastar::internal {\n[[noreturn]] void assert_fail(const char *msg, const char *file, int line,\n                              const char *func);\n}\n\n/// Like assert(), but independent of NDEBUG. Active in all build modes.\n#define SEASTAR_ASSERT(x)                                             \\\n    do {                                                              \\\n        if (!(x)) [[unlikely]] {                                      \\\n            seastar::internal::assert_fail(#x, __FILE__, __LINE__,    \\\n                                           __PRETTY_FUNCTION__);      \\\n        }                                                             \\\n    } while (0)\n"
  },
  {
    "path": "include/seastar/util/backtrace.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/format.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/util/assert.hh>\n\n#if __has_include(<execinfo.h>)\n#include <execinfo.h>\n#define HAVE_EXECINFO\n#endif\n#include <iosfwd>\n#include <memory>\n#include <variant>\n#include <boost/container/static_vector.hpp>\n#include <fmt/ostream.h>\n\nnamespace seastar {\n\nstruct shared_object {\n    sstring name;\n    uintptr_t begin;\n    uintptr_t end; // C++-style, last addr + 1\n};\n\nstruct frame {\n    const shared_object* so;\n    uintptr_t addr;\n};\n\nbool operator==(const frame& a, const frame& b) noexcept;\n\n\n// If addr doesn't seem to belong to any of the provided shared objects, it\n// will be considered as part of the executable.\nframe decorate(uintptr_t addr) noexcept;\n\n// Invokes func for each frame passing it as argument.\n// incremental=false is the default mode and simply calls ::backtrace once\n// and then calls func for each frame. If the ::backtrace call crashes,\n// func will not be called.\n// incremental=true will call ::backtrace in a loop, increasing the number of\n// frames to capture by one each time. This is useful for cases where\n// ::backtrace itself may crash, e.g., in a crash handler, in this case the\n// func will be called for each frame which didn't crash. See #2710. This\n// is much slower, however.\ntemplate<typename Func>\nvoid backtrace(Func&& func, bool incremental = false) noexcept(noexcept(func(frame()))) {\n#ifdef HAVE_EXECINFO\n    constexpr size_t max_backtrace = 100;\n    void* buffer[max_backtrace];\n\n    if (incremental) {\n        for (size_t last_frame = 1; last_frame <= max_backtrace; ++last_frame) {\n            int n = ::backtrace(buffer, last_frame);\n            if (n < static_cast<int>(last_frame)) {\n                return;\n            }\n            auto ip = reinterpret_cast<uintptr_t>(buffer[last_frame-1]);\n            func(decorate(ip - 1));\n        }\n    } else {\n        int n = ::backtrace(buffer, max_backtrace);\n        for (int i = 0; i < n; ++i) {\n            auto ip = reinterpret_cast<uintptr_t>(buffer[i]);\n            func(decorate(ip - 1));\n        }\n    }\n#else\n// Not implemented yet\n#define SEASTAR_BACKTRACE_UNIMPLEMENTED\n#endif\n}\n\n// Represents a call stack of a single thread.\nclass simple_backtrace {\npublic:\n    using vector_type = boost::container::static_vector<frame, 64>;\nprivate:\n    vector_type _frames;\n    size_t _hash = 0;\n    char _delimeter = ' ';\nprivate:\n    size_t calculate_hash() const noexcept;\npublic:\n    simple_backtrace(vector_type f) noexcept : _frames(std::move(f)), _hash(calculate_hash()) {}\n    simple_backtrace() noexcept = default;\n    [[deprecated]] simple_backtrace(vector_type f, char delimeter) : _frames(std::move(f)), _hash(calculate_hash()), _delimeter(delimeter) {}\n    [[deprecated]] simple_backtrace(char delimeter) : _delimeter(delimeter) {}\n\n    size_t hash() const noexcept { return _hash; }\n    char delimeter() const noexcept { return _delimeter; }\n\n    friend fmt::formatter<simple_backtrace>;\n\n    bool operator==(const simple_backtrace& o) const noexcept {\n        return _hash == o._hash && _frames == o._frames;\n    }\n\n    bool operator!=(const simple_backtrace& o) const noexcept {\n        return !(*this == o);\n    }\n};\n\nusing shared_backtrace = seastar::lw_shared_ptr<simple_backtrace>;\n\n// Represents a task object inside a tasktrace.\nclass task_entry {\n    const std::type_info* _task_type;\npublic:\n    task_entry(const std::type_info& ti) noexcept\n        : _task_type(&ti)\n    { }\n\n    friend fmt::formatter<task_entry>;\n\n    bool operator==(const task_entry& o) const noexcept {\n        return *_task_type == *o._task_type;\n    }\n\n    bool operator!=(const task_entry& o) const noexcept {\n        return !(*this == o);\n    }\n\n    size_t hash() const noexcept { return _task_type->hash_code(); }\n};\n\n// Extended backtrace which consists of a backtrace of the currently running task\n// and information about the chain of tasks waiting for the current operation to complete.\nclass tasktrace {\npublic:\n    struct entry {\n        std::variant<shared_backtrace, task_entry> backtrace_or_task_entry;\n        std::source_location resume_point;\n\n        bool operator == (const entry &other) const {\n            return backtrace_or_task_entry == other.backtrace_or_task_entry\n                    && resume_point.line() == other.resume_point.line()\n                    && resume_point.column() == other.resume_point.column()\n                    && std::string_view{ resume_point.file_name() } == std::string_view{ other.resume_point.file_name() };\n        }\n    };\n    using vector_type = boost::container::static_vector<entry, 16>;\nprivate:\n    simple_backtrace _main;\n    vector_type _prev;\n    scheduling_group _sg;\n    size_t _hash;\npublic:\n    tasktrace() = default;\n    tasktrace(simple_backtrace main, vector_type prev, size_t prev_hash, scheduling_group sg) noexcept;\n    tasktrace(const tasktrace&) = default;\n    tasktrace& operator=(const tasktrace&) = default;\n    ~tasktrace();\n\n    size_t hash() const noexcept { return _hash; }\n    char delimeter() const noexcept { return _main.delimeter(); }\n\n    friend fmt::formatter<tasktrace>;\n\n    bool operator==(const tasktrace& o) const noexcept;\n\n    bool operator!=(const tasktrace& o) const noexcept {\n        return !(*this == o);\n    }\n};\n\n}\n\nnamespace std {\n\ntemplate<>\nstruct hash<seastar::simple_backtrace> {\n    size_t operator()(const seastar::simple_backtrace& b) const {\n        return b.hash();\n    }\n};\n\ntemplate<>\nstruct hash<seastar::tasktrace> {\n    size_t operator()(const seastar::tasktrace& b) const {\n        return b.hash();\n    }\n};\n\n}\n\ntemplate <> struct fmt::formatter<seastar::frame> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::frame&, fmt::format_context& ctx) const -> decltype(ctx.out());\n};\ntemplate <> struct fmt::formatter<seastar::simple_backtrace> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::simple_backtrace&, fmt::format_context& ctx) const -> decltype(ctx.out());\n};\ntemplate <> struct fmt::formatter<seastar::tasktrace> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::tasktrace&, fmt::format_context& ctx) const -> decltype(ctx.out());\n};\ntemplate <> struct fmt::formatter<seastar::task_entry> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(const seastar::task_entry&, fmt::format_context& ctx) const -> decltype(ctx.out());\n};\n\nnamespace seastar {\n\nusing saved_backtrace = tasktrace;\n\nsaved_backtrace current_backtrace() noexcept;\n\ntasktrace current_tasktrace() noexcept;\n\n// Collects backtrace only within the currently executing task.\nsimple_backtrace current_backtrace_tasklocal() noexcept;\n\nstd::ostream& operator<<(std::ostream& out, const tasktrace& b);\n\nnamespace internal {\n\ntemplate<class Exc>\nclass backtraced : public Exc {\n    std::shared_ptr<sstring> _backtrace;\npublic:\n    template<typename... Args>\n    backtraced(Args&&... args)\n            : Exc(std::forward<Args>(args)...)\n            , _backtrace(std::make_shared<sstring>(format(\"{} Backtrace: {}\", Exc::what(), current_backtrace()))) {}\n\n    /**\n     * Returns the original exception message with a backtrace appended to it\n     *\n     * @return original exception message followed by a backtrace\n     */\n    virtual const char* what() const noexcept override {\n        SEASTAR_ASSERT(_backtrace);\n        return _backtrace->c_str();\n    }\n};\n\n}\n\n\n/// Create an exception pointer of unspecified type that is derived from Exc type\n/// with a backtrace attached to its message.\n///\n/// \\tparam Exc exception type to be caught at the receiving side\n/// \\tparam Args types of arguments forwarded to the constructor of Exc\n/// \\param args arguments forwarded to the constructor of Exc\n/// \\return std::exception_ptr containing the exception with the backtrace.\ntemplate <class Exc, typename... Args>\nstd::exception_ptr make_backtraced_exception_ptr(Args&&... args) {\n    using exc_type = std::decay_t<Exc>;\n    static_assert(std::is_base_of_v<std::exception, exc_type>,\n            \"throw_with_backtrace only works with exception types\");\n    return std::make_exception_ptr<internal::backtraced<exc_type>>(Exc(std::forward<Args>(args)...));\n}\n\n    /**\n     * Throws an exception of unspecified type that is derived from the Exc type\n     * with a backtrace attached to its message\n     *\n     * @tparam Exc exception type to be caught at the receiving side\n     * @tparam Args types of arguments forwarded to the constructor of Exc\n     * @param args arguments forwarded to the constructor of Exc\n     * @return never returns (throws an exception)\n     */\ntemplate <class Exc, typename... Args>\n[[noreturn]]\nvoid\nthrow_with_backtrace(Args&&... args) {\n    std::rethrow_exception(make_backtraced_exception_ptr<Exc>(std::forward<Args>(args)...));\n};\n\n}\n"
  },
  {
    "path": "include/seastar/util/bool_class.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n#pragma once\n\n#include <ostream>\n#include <fmt/core.h>\n\nnamespace seastar {\n\n/// \\addtogroup utilities\n/// @{\n\n/// \\brief Type-safe boolean\n///\n/// bool_class objects are type-safe boolean values that cannot be implicitly\n/// casted to untyped bools, integers or different bool_class types while still\n/// provides all relevant logical and comparison operators.\n///\n/// bool_class template parameter is a tag type that is going to be used to\n/// distinguish booleans of different types.\n///\n/// Usage examples:\n/// \\code\n/// struct foo_tag { };\n/// using foo = bool_class<foo_tag>;\n///\n/// struct bar_tag { };\n/// using bar = bool_class<bar_tag>;\n///\n/// foo v1 = foo::yes; // OK\n/// bar v2 = foo::yes; // ERROR, no implicit cast\n/// foo v4 = v1 || foo::no; // OK\n/// bar v5 = bar::yes && bar(true); // OK\n/// bool v6 = v5; // ERROR, no implicit cast\n/// \\endcode\n///\n/// \\tparam Tag type used as a tag\ntemplate<typename Tag>\nclass bool_class {\n    bool _value;\npublic:\n    static const bool_class yes;\n    static const bool_class no;\n\n    /// Constructs a bool_class object initialised to \\c false.\n    constexpr bool_class() noexcept : _value(false) { }\n\n    /// Constructs a bool_class object initialised to \\c v.\n    constexpr explicit bool_class(bool v) noexcept : _value(v) { }\n\n    /// Casts a bool_class object to an untyped \\c bool.\n    explicit operator bool() const noexcept { return _value; }\n\n    /// Logical OR.\n    friend bool_class operator||(bool_class x, bool_class y) noexcept {\n        return bool_class(x._value || y._value);\n    }\n\n    /// Logical AND.\n    friend bool_class operator&&(bool_class x, bool_class y) noexcept {\n        return bool_class(x._value && y._value);\n    }\n\n    /// Logical NOT.\n    friend bool_class operator!(bool_class x) noexcept {\n        return bool_class(!x._value);\n    }\n\n    /// Equal-to and not-equal-to operators.\n    friend bool operator==(bool_class x, bool_class y) noexcept = default;\n\n#if __cpp_lib_three_way_comparison\n    auto operator<=>(const bool_class& other) const noexcept = default;\n#endif\n\n    /// Prints bool_class value to an output stream.\n    friend std::ostream& operator<<(std::ostream& os, bool_class v) {\n        return os << (v._value ? \"true\" : \"false\");\n    }\n};\n\ntemplate<typename Tag>\nconst bool_class<Tag> bool_class<Tag>::yes { true };\ntemplate<typename Tag>\nconst bool_class<Tag> bool_class<Tag>::no { false };\n\n/// @}\n\n}\n\ntemplate<typename Tag>\nstruct fmt::formatter<seastar::bool_class<Tag>> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    template <typename FormatContext>\n    auto format(seastar::bool_class<Tag> v, FormatContext& ctx) const{\n        return fmt::format_to(ctx.out(), \"{}\", bool(v));\n    }\n};\n"
  },
  {
    "path": "include/seastar/util/closeable.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <concepts>\n#include <functional>\n#include <seastar/core/future.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n\n/// \\file\n\n/// \\example closeable_test.cc\n\nnamespace seastar {\n\ntemplate <typename Object>\nconcept closeable = requires (Object o) {\n    { o.close() } SEASTAR_DEFERRED_ACTION_NOEXCEPT -> std::same_as<future<>>;\n};\n\n/// Template helper to auto-close \\c obj when destroyed.\n///\n/// \\tparam Object a class exposing a \\c close() method that returns a \\c future<>\n///         that is called when the controller is destroyed.\n///\n/// Must be used in a seastar thread as the destructor\n/// needs to wait on the \\c obj close() future.\ntemplate <typename Object>\nrequires closeable<Object>\nclass [[nodiscard]] deferred_close {\n    std::reference_wrapper<Object> _obj;\n    bool _closed = false;\n\n    void do_close() noexcept {\n        if (!_closed) {\n            _closed = true;\n            _obj.get().close().get();\n        }\n    }\npublic:\n    /// Construct an object that will auto-close \\c obj when destroyed.\n    /// \\tparam obj the object to auto-close.\n    deferred_close(Object& obj) noexcept : _obj(obj) {}\n    /// Moves the \\c deferred_close into a new one, and\n    /// the old one is canceled.\n    deferred_close(deferred_close&& x) noexcept : _obj(x._obj), _closed(std::exchange(x._closed, true)) {}\n    deferred_close(const deferred_close&) = delete;\n    /// Move-assign another \\ref deferred_close.\n    /// The current \\ref deferred_close is closed before being assigned.\n    /// And the other one's state is transferred to the current one.\n    deferred_close& operator=(deferred_close&& x) noexcept {\n        do_close();\n        _obj = x._obj;\n        _closed = std::exchange(x._closed, true);\n        return *this;\n    }\n    /// Destruct the deferred_close object and auto-close \\c obj.\n    ~deferred_close() {\n        do_close();\n    }\n    /// Close \\c obj once now.\n    void close_now() noexcept {\n        SEASTAR_ASSERT(!_closed);\n        do_close();\n    }\n\n    /// Prevents close() from being called when this object is destroyed.\n    /// Cannot call close_now() any more after this.\n    void cancel() noexcept {\n        _closed = true;\n    }\n};\n\ntemplate <closeable Closeable, std::invocable<Closeable&> Func>\nrequires std::is_nothrow_move_constructible_v<Closeable> && std::is_nothrow_move_constructible_v<Func>\ninline futurize_t<std::invoke_result_t<Func, Closeable&>>\nwith_closeable(Closeable&& obj, Func func) noexcept {\n    return do_with(std::move(obj), [func = std::move(func)] (Closeable& obj) mutable {\n        return futurize_invoke(func, obj).finally([&obj] {\n            return obj.close();\n        });\n    });\n}\n\ntemplate <typename Object>\nconcept stoppable = requires (Object o) {\n    { o.stop() } SEASTAR_DEFERRED_ACTION_NOEXCEPT -> std::same_as<future<>>;\n};\n\n/// Template helper to auto-stop \\c obj when destroyed.\n///\n/// \\tparam Object a class exposing a \\c stop() method that returns a \\c future<>\n///         that is called when the controller is destroyed.\n///\n/// Must be used in a seastar thread as the destructor\n/// needs to wait on the \\c obj stop() future.\ntemplate <typename Object>\nrequires stoppable<Object>\nclass [[nodiscard]] deferred_stop {\n    std::reference_wrapper<Object> _obj;\n    bool _stopped = false;\n\n    void do_stop() noexcept {\n        if (!_stopped) {\n            _stopped = true;\n            _obj.get().stop().get();\n        }\n    }\npublic:\n    /// Construct an object that will auto-stop \\c obj when destroyed.\n    /// \\tparam obj the object to auto-stop.\n    deferred_stop(Object& obj) noexcept : _obj(obj) {}\n    /// Moves the \\c deferred_stop into a new one, and\n    /// the old one is canceled.\n    deferred_stop(deferred_stop&& x) noexcept : _obj(x._obj), _stopped(std::exchange(x._stopped, true)) {}\n    deferred_stop(const deferred_stop&) = delete;\n    /// Move-assign another \\ref deferred_stop.\n    /// The current \\ref deferred_stop is stopped before being assigned.\n    /// And the other one's state is transferred to the current one.\n    deferred_stop& operator=(deferred_stop&& x) noexcept {\n        do_stop();\n        _obj = x._obj;\n        _stopped = std::exchange(x._stopped, true);\n        return *this;\n    }\n    /// Destruct the deferred_stop object and auto-stop \\c obj.\n    ~deferred_stop() {\n        do_stop();\n    }\n    /// Stop \\c obj once now.\n    void stop_now() noexcept {\n        SEASTAR_ASSERT(!_stopped);\n        do_stop();\n    }\n\n    /// Prevents stop() from being called when this object is destroyed.\n    /// Cannot call stop_now() any more after this.\n    void cancel() noexcept {\n        _stopped = true;\n    }\n};\n\ntemplate <stoppable Stoppable, std::invocable<Stoppable&> Func>\nrequires std::is_nothrow_move_constructible_v<Stoppable> && std::is_nothrow_move_constructible_v<Func>\ninline futurize_t<std::invoke_result_t<Func, Stoppable&>>\nwith_stoppable(Stoppable&& obj, Func func) noexcept {\n    return do_with(std::move(obj), [func = std::move(func)] (Stoppable& obj) mutable {\n        return futurize_invoke(func, obj).finally([&obj] {\n            return obj.stop();\n        });\n    });\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/conversions.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <cstdlib>\n#include <string_view>\n#include <vector>\n\nnamespace seastar {\n\n// Convert a string to a memory size, allowing binary SI\n// suffixes (intentionally, even though SI suffixes are\n// decimal, to follow existing usage). A string matched\n// by following BNF is accetped:\n//\n// memory_size ::= <digit>+ <suffix>? \"i\"? \"B\"?\n// suffix ::= (\"k\" | \"K\" | \"M\" | \"G\" | \"T\")\n//\n// for instance:\n//\n// \"5\" -> 5\n// \"4k\" -> (4 << 10)\n// \"8Mi\" -> (8 << 20)\n// \"7GB\" -> (7 << 30)\n// \"1TiB\" -> (1 << 40)\n// anything else: exception\nsize_t parse_memory_size(std::string_view s);\n\nstatic inline std::vector<char> string2vector(std::string_view str) {\n    auto v = std::vector<char>(str.begin(), str.end());\n    v.push_back('\\0');\n    return v;\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/critical_alloc_section.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\nnamespace seastar {\nnamespace memory {\n\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\n/// \\cond internal\nnamespace internal {\n\n// This variable is used in hot paths so we want to avoid the compiler\n// generating TLS init guards for it. In C++20 we have constinit to tell the\n// compiler that it can be initialized compile time (although gcc still doesn't\n// completely drops the init guards - https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97848).\n// In < c++20 we use `__thread` which results in no TLS init guards generated.\n#ifdef __cpp_constinit\nextern thread_local constinit volatile int critical_alloc_section;\n#else\nextern __thread volatile int critical_alloc_section;\n#endif\n\n} // namespace internal\n/// \\endcond\n\n/// \\brief Marks scopes that contain critical allocations.\n///\n/// Critical allocations are those, whose failure the application cannot\n/// tolerate. In a perfect world, there should be no such allocation, but we\n/// don't live in a perfect world.\n/// This information is used by other parts of the memory subsystem:\n/// * \\ref alloc_failure_injector will not inject errors into these scopes.\n/// * A memory diagnostics report will be dumped if an allocation fails in these\n///   scopes when the memory diagnostics subsystem is configured to dump reports\n///   for \\ref alloc_failure_kind \\ref alloc_failure_kind::critical or above.\n///   See \\ref set_dump_memory_diagnostics_on_alloc_failure_kind().\nclass scoped_critical_alloc_section {\npublic:\n    scoped_critical_alloc_section() {\n        // we assume the critical_alloc_section is thread local\n        // and there's seastar threads are non-preemptive.\n        // Otherwise, this would require an atomic variable\n        internal::critical_alloc_section = internal::critical_alloc_section + 1;\n    }\n    ~scoped_critical_alloc_section() {\n        internal::critical_alloc_section = internal::critical_alloc_section - 1;\n    }\n};\n\n/// \\brief Is the current context inside a critical alloc section?\n///\n/// Will return true if there is at least one \\ref scoped_critical_alloc_section\n/// alive in the current scope or the scope of any of the caller functions.\ninline bool is_critical_alloc_section() {\n    return bool(internal::critical_alloc_section);\n}\n\n#else   // SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\nstruct [[maybe_unused]] scoped_critical_alloc_section {};\n\ninline bool is_critical_alloc_section() {\n    return false;\n}\n\n#endif  // SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\n} // namespace seastar\n} // namespace memory\n"
  },
  {
    "path": "include/seastar/util/defer.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <concepts>\n#include <type_traits>\n#include <utility>\n\n\n#ifdef SEASTAR_DEFERRED_ACTION_REQUIRE_NOEXCEPT\n#define SEASTAR_DEFERRED_ACTION_NOEXCEPT noexcept\n#else\n#define SEASTAR_DEFERRED_ACTION_NOEXCEPT\n#endif\n\ntemplate <typename Func>\nconcept deferrable_action = requires (Func func) {\n    { func() } SEASTAR_DEFERRED_ACTION_NOEXCEPT -> std::same_as<void>;\n} && std::is_nothrow_move_constructible_v<Func>;\n\nnamespace seastar {\n\ntemplate <typename Func>\nrequires deferrable_action<Func>\nclass [[nodiscard]] deferred_action {\n    Func _func;\n    bool _cancelled = false;\npublic:\n    deferred_action(Func&& func) noexcept : _func(std::move(func)) {}\n    deferred_action(deferred_action&& o) noexcept : _func(std::move(o._func)), _cancelled(o._cancelled) {\n        o._cancelled = true;\n    }\n    deferred_action& operator=(deferred_action&& o) noexcept {\n        if (this != &o) {\n            this->~deferred_action();\n            new (this) deferred_action(std::move(o));\n        }\n        return *this;\n    }\n    deferred_action(const deferred_action&) = delete;\n    ~deferred_action() { if (!_cancelled) { _func(); }; }\n    void cancel() { _cancelled = true; }\n};\n\ntemplate <typename Func>\nrequires deferrable_action<Func>\ninline\ndeferred_action<Func>\ndefer(Func&& func) {\n    return deferred_action<Func>(std::forward<Func>(func));\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/eclipse.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#pragma once\n\n// Workarounds for deficiencies in Eclipse's C++ parser\n//\n// Tell Eclipse that IN_ECLIPSE is defined so it will ignore all the unknown syntax.\n\n#ifndef IN_ECLIPSE\n\n#else\n\n// Eclipse doesn't grok alignof\n#define alignof sizeof\n\n#endif\n"
  },
  {
    "path": "include/seastar/util/exceptions.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <filesystem>\n\nnamespace seastar {\n\n/// \\brief make a filesystem_error for system calls with a single file operand.\n///\n/// \\param what - describes the action that failed\n/// \\param path - path of the file that hit the error\n/// \\param error - the system error number (see errno(3))\n///\nstd::filesystem::filesystem_error make_filesystem_error(const std::string& what, std::filesystem::path path, int error);\n\n/// \\brief make a filesystem_error for system calls with two file operands.\n///\n/// \\param what - describes the action that failed\n/// \\param path1, path2 - paths of the files that hit the error\n/// \\param error - the system error number (see errno(3))\n///\nstd::filesystem::filesystem_error make_filesystem_error(const std::string& what, std::filesystem::path path1, std::filesystem::path path2, int error);\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/file.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/short_streams.hh>\n\nnamespace seastar {\n\n/// Recursively removes a directory and all of its contents.\n///\n/// \\param path path of the directory to recursively remove\n///\n/// \\note\n/// Unlike `rm -rf` path has to be a directory and may not refer to a regular file.\n///\n/// The function flushes the parent directory of the removed path and so guaranteeing that\n/// the remove is stable on disk.\n///\n/// The function bails out on first error. In that case, some files and/or sub-directories\n/// (and their contents) may be left behind at the level in which the error was detected.\n///\nfuture<> recursive_remove_directory(std::filesystem::path path) noexcept;\n\n/// @}\n\n/// \\defgroup fileio-util File and Stream Utilities\n/// \\ingroup fileio-module\n///\n/// \\brief\n/// These utilities are provided to help perform operations on files and I/O streams.\n\nnamespace util {\n\n/// \\addtogroup fileio-util\n/// @{\n\ntemplate <typename Func>\nrequires std::invocable<Func, input_stream<char>&>\ntypename futurize<typename std::invoke_result_t<Func, input_stream<char>&>>::type with_file_input_stream(const std::filesystem::path& path, Func func, file_open_options file_opts = {}, file_input_stream_options input_stream_opts = {}) {\n    static_assert(std::is_nothrow_move_constructible_v<Func>);\n    auto f = co_await open_file_dma(path.native(), open_flags::ro, std::move(file_opts));\n    data_source ins;\n    std::exception_ptr ex;\n    try {\n        ins = make_file_data_source(f, std::move(input_stream_opts));\n    } catch (...) {\n        ex = std::current_exception();\n    }\n    if (ex) {\n        co_await f.close();\n        co_await coroutine::return_exception_ptr(std::move(ex));\n    }\n\n    input_stream<char> in(std::move(ins));\n    auto res = co_await coroutine::as_future(futurize_invoke(std::move(func), in));\n    co_await in.close();\n    co_await f.close();\n    co_return co_await std::move(res);\n}\n\n/// Returns all bytes from the file until eof, accessible in chunks.\n///\n/// \\note use only on short files to avoid running out of memory.\n///\n/// \\param path path of the file to be read.\nfuture<std::vector<temporary_buffer<char>>> read_entire_file(std::filesystem::path path);\n\n/// Returns all bytes from the file until eof as a single buffer.\n///\n/// \\note use only on short files to avoid running out of memory.\n///\n/// \\param path path of the file to be read.\nfuture<sstring> read_entire_file_contiguous(std::filesystem::path path);\n\n/// @}\n\n} // namespace util\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/function_input_iterator.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\nnamespace seastar {\n\ntemplate <typename Function, typename State>\nstruct function_input_iterator {\n    Function _func;\n    State _state;\npublic:\n    function_input_iterator(Function func, State state)\n        : _func(func), _state(state) {\n    }\n    function_input_iterator(const function_input_iterator&) = default;\n    function_input_iterator(function_input_iterator&&) = default;\n    function_input_iterator& operator=(const function_input_iterator&) = default;\n    function_input_iterator& operator=(function_input_iterator&&) = default;\n    auto operator*() const {\n        return _func();\n    }\n    function_input_iterator& operator++() {\n        ++_state;\n        return *this;\n    }\n    function_input_iterator operator++(int) {\n        function_input_iterator ret{*this};\n        ++_state;\n        return ret;\n    }\n    bool operator==(const function_input_iterator& x) const {\n        return _state == x._state;\n    }\n    bool operator!=(const function_input_iterator& x) const {\n        return !operator==(x);\n    }\n};\n\ntemplate <typename Function, typename State>\ninline\nfunction_input_iterator<Function, State>\nmake_function_input_iterator(Function func, State state) {\n    return function_input_iterator<Function, State>(func, state);\n}\n\ntemplate <typename Function, typename State>\ninline\nfunction_input_iterator<Function, State>\nmake_function_input_iterator(Function&& func) {\n    return function_input_iterator<Function, State>(func, State{});\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/indirect.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <memory>\n\nnamespace seastar {\n\n// This header defines functors for comparing and hashing pointers by pointed-to values instead of pointer addresses.\n//\n// Examples:\n//\n//  std::multiset<shared_ptr<sstring>, indirect_less<shared_ptr<sstring>>> _multiset;\n//\n//  std::unordered_map<shared_ptr<sstring>, bool,\n//      indirect_hash<shared_ptr<sstring>>, indirect_equal_to<shared_ptr<sstring>>> _unordered_map;\n//\n\ntemplate<typename Pointer, typename Equal = std::equal_to<typename std::pointer_traits<Pointer>::element_type>>\nstruct indirect_equal_to {\n    Equal _eq;\n    indirect_equal_to(Equal eq = Equal()) : _eq(std::move(eq)) {}\n    bool operator()(const Pointer& i1, const Pointer& i2) const {\n        if (bool(i1) ^ bool(i2)) {\n            return false;\n        }\n        return !i1 || _eq(*i1, *i2);\n    }\n};\n\ntemplate<typename Pointer, typename Less = std::less<typename std::pointer_traits<Pointer>::element_type>>\nstruct indirect_less {\n    Less _cmp;\n    indirect_less(Less cmp = Less()) : _cmp(std::move(cmp)) {}\n    bool operator()(const Pointer& i1, const Pointer& i2) const {\n        if (i1 && i2) {\n            return _cmp(*i1, *i2);\n        }\n        return !i1 && i2;\n    }\n};\n\ntemplate<typename Pointer, typename Hash = std::hash<typename std::pointer_traits<Pointer>::element_type>>\nstruct indirect_hash {\n    Hash _h;\n    indirect_hash(Hash h = Hash()) : _h(std::move(h)) {}\n    size_t operator()(const Pointer& p) const {\n        if (p) {\n            return _h(*p);\n        }\n        return 0;\n    }\n};\n\n}\n"
  },
  {
    "path": "include/seastar/util/integrated-length.hh",
    "content": "#pragma once\n\nnamespace seastar {\nnamespace util {\n\n//\n// Integrates frequently changing value over time\n//\n// Given a volatile value (e.g. -- length of task queue), calculates a\n// time function out of it; it can later be exported as a COUNTER metrics,\n// so that in monitoring its average value can be observed by using the\n// rate() function.\n//\n// The value is modified as if it was a plain unsigned integer, and at\n// some points it must be explicitly checkpointed by calling the respective\n// method of this class. Only the checkpoint-ed values contribute to the\n// final result, so observed in metrics will be those values either.\n// However, checkpointing the value on every update is possible.\n//\n// The integrator maintains three values:\n// - the value itself. It can be modified by direct assignment, or by\n//   incrementing/decrementing it. It's thus a transparent wrapper of\n//   unsigned integrals\n//\n// - a timestamp of the last checkpoint\n// - the final result. Produces the length * seconds unit\n//\n// The calculated value is the sum of value * (now - timestamp) for every\n// checkpoint seen so far. The integrated result is measured in units\n// multiplied by seconds.\n//\ntemplate <std::unsigned_integral T, typename Clock, typename Resolution = std::chrono::nanoseconds>\nclass integrated_length {\n    T _value;\n    uint64_t _integral;\n    Clock::time_point _ts;\n\npublic:\n    integrated_length() noexcept\n            : _value(0)\n            , _integral(0)\n            , _ts(Clock::now())\n    {}\n\n    integrated_length(T iv) noexcept\n            : _value(iv)\n            , _integral(0)\n            , _ts(Clock::now())\n    {}\n\n    bool operator==(const integrated_length& o) const noexcept {\n        return _value == o._value;\n    }\n\n    bool operator!=(const integrated_length& o) const noexcept {\n        return _value != o._value;\n    }\n\n    // Below methods transparently \"wrap\" the T type for arithmetics\n\n    void operator=(T v) noexcept {\n        _value = v;\n    }\n\n    integrated_length& operator++() noexcept {\n        _value++;\n        return *this;\n    }\n\n    integrated_length operator++(int) noexcept {\n        integrated_length previous(*this);\n        _value++;\n        return previous;\n    }\n\n    integrated_length& operator--() noexcept {\n        _value--;\n        return *this;\n    }\n\n    integrated_length operator--(int) noexcept {\n        integrated_length previous(*this);\n        _value--;\n        return previous;\n    }\n\n    bool operator==(const T& v) const noexcept {\n        return _value == v;\n    }\n\n    bool operator!=(const T& v) const noexcept {\n        return _value != v;\n    }\n\n    // Updates the integral value by adding yet another\n    // value * duration component. The \"now\" time-point can be\n    // provided (e.g. for tests or optimizations)\n    void checkpoint(Clock::time_point now = Clock::now()) noexcept {\n        Resolution dur = std::chrono::duration_cast<Resolution>(now - _ts);\n        _ts = now;\n        _integral += _value * dur.count();\n    }\n\n    // Returns the last seen (or current) value\n    T value() const noexcept {\n        return _value;\n    }\n\n    // Returns the integrated result\n    uint64_t integral() const noexcept {\n        // Implicitly casts time component to seconds\n        return _integral / std::chrono::duration_cast<Resolution>(std::chrono::seconds(1)).count();\n    }\n};\n\n} // util namespace\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/internal/array_map.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <array>\n#include <cstddef>\n#include <stdexcept>\n#include <string>\n#include <utility>\n\nnamespace seastar {\nnamespace internal {\n\n// unordered_map implemented as a simple array\n\ntemplate <typename Value, size_t Max>\nclass array_map {\n    std::array<Value, Max> _a {};\npublic:\n    array_map(std::initializer_list<std::pair<size_t, Value>> i) {\n        for (auto kv : i) {\n            _a[kv.first] = kv.second;\n        }\n    }\n    Value& operator[](size_t key) { return _a[key]; }\n    const Value& operator[](size_t key) const { return _a[key]; }\n\n    Value& at(size_t key) {\n        if (key >= Max) {\n            throw std::out_of_range(std::to_string(key) + \" >= \" + std::to_string(Max));\n        }\n        return _a[key];\n    }\n};\n\n} // internal namespace\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/internal/iovec_utils.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB.\n */\n\n#include <span>\n\n#include <sys/types.h>\n\n#pragma once\n\nnamespace seastar {\nnamespace internal {\n\ninline size_t iovec_len(const iovec* begin, size_t len)\n{\n    size_t ret = 0;\n    auto end = begin + len;\n    while (begin != end) {\n        ret += begin++->iov_len;\n    }\n    return ret;\n}\n\n\ninline size_t iovec_len(std::span<const iovec> iov) {\n    size_t ret = 0;\n    for (auto&& e : iov) {\n        ret += e.iov_len;\n    }\n    return ret;\n}\n\n// Skips first \\size bytes from the data \\iov points to\n// Can update some iovecs in-place\ninline std::span<iovec> iovec_trim_front(std::span<iovec> iov, size_t size) {\n    unsigned pos = 0;\n    while (pos < iov.size() && size > 0) {\n        if (iov[pos].iov_len > size) {\n            iov[pos].iov_base = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(iov[pos].iov_base) + size);\n            iov[pos].iov_len -= size;\n            break;\n        }\n\n        size -= iov[pos].iov_len;\n        pos++;\n    }\n    return iov.subspan(pos);\n}\n\n// Given a properly aligned vector of iovecs, ensures that it respects the\n// IOV_MAX limit, by trimming if necessary. The modified vector still satisfied\n// the alignment requirements.\n// Returns the final total length of all iovecs.\nsize_t sanitize_iovecs(std::vector<iovec>& iov, size_t disk_alignment) noexcept;\n\n} // internal namespace\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/internal/magic.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB.\n */\n\n#pragma once\n\n#include <linux/magic.h>\n\nnamespace seastar {\n\n#ifndef XFS_SUPER_MAGIC\n#define XFS_SUPER_MAGIC 0x58465342\n#endif\n\n#ifndef EXT2_SUPER_MAGIC\n#define EXT2_SUPER_MAGIC 0xEF53\n#endif\n\n#ifndef EXT3_SUPER_MAGIC\n#define EXT3_SUPER_MAGIC 0xEF53\n#endif\n\n#ifndef EXT4_SUPER_MAGIC\n#define EXT4_SUPER_MAGIC 0xEF53\n#endif\n\n#ifndef NFS_SUPER_MAGIC\n#define NFS_SUPER_MAGIC 0x6969\n#endif\n\n#ifndef TMPFS_MAGIC\n#define TMPFS_MAGIC 0x01021994\n#endif\n\n#ifndef FUSE_SUPER_MAGIC\n#define FUSE_SUPER_MAGIC 0x65735546\n#endif\n\n#ifndef BTRFS_SUPER_MAGIC\n#define BTRFS_SUPER_MAGIC 0x9123683E\n#endif\n\n#ifndef HFS_SUPER_MAGIC\n#define HFS_SUPER_MAGIC 0x4244\n#endif\n\nnamespace internal {\n\nclass fs_magic {\npublic:\n    static constexpr unsigned long xfs = XFS_SUPER_MAGIC;\n    static constexpr unsigned long ext2 = EXT2_SUPER_MAGIC;\n    static constexpr unsigned long ext3 = EXT3_SUPER_MAGIC;\n    static constexpr unsigned long ext4 = EXT4_SUPER_MAGIC;\n    static constexpr unsigned long nfs = NFS_SUPER_MAGIC;\n    static constexpr unsigned long tmpfs = TMPFS_MAGIC;\n    static constexpr unsigned long fuse = FUSE_SUPER_MAGIC;\n    static constexpr unsigned long btrfs = BTRFS_SUPER_MAGIC;\n    static constexpr unsigned long hfs = HFS_SUPER_MAGIC;\n};\n\n} // internal namespace\n\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/iostream.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2024 ScyllaDB\n */\n\n#pragma once\n\n#include <type_traits>\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\nnamespace util {\n\n/// Writes data to stream and properly flushes and closes one\n///\n/// \\param out \\ref output_stream to write to\n/// \\param writer callable invoked to write data itself\n///\n/// The helper takes ownership of the output_stream, flushes and closes it\n/// after the writer is done. The stream is not useable after it resolves.\ntemplate <typename W>\nrequires std::is_invocable_r_v<future<>, W, output_stream<char>&>\nfuture<> write_to_stream_and_close(output_stream<char>&& out_, W writer) {\n    output_stream<char> out = std::move(out_);\n    std::exception_ptr ex;\n    try {\n        co_await writer(out);\n        co_await out.flush();\n    } catch (...) {\n        ex = std::current_exception();\n    }\n    co_await out.close();\n    if (ex) {\n        co_await coroutine::return_exception_ptr(std::move(ex));\n    }\n}\n\n} // util namespace\n} // seastar namespace\n\n"
  },
  {
    "path": "include/seastar/util/is_smart_ptr.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <memory> // for std::unique_ptr\n#include <type_traits>\n\nnamespace seastar {\n\ntemplate<typename T>\nstruct is_smart_ptr : std::false_type {};\n\ntemplate<typename T>\nstruct is_smart_ptr<std::unique_ptr<T>> : std::true_type {};\n\n}\n"
  },
  {
    "path": "include/seastar/util/later.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n\n#pragma once\n\n#include <seastar/core/future.hh>\n\nnamespace seastar {\n\n/// \\addtogroup future-util\n/// @{\n\n/// \\brief Returns a ready future.\ninline\nfuture<> now() {\n    return make_ready_future<>();\n}\n\n/// \\brief Returns a future which is not ready but is scheduled to resolve soon.\n///\n/// Schedules a future to run \"soon\". yield() can be used to break long-but-finite\n/// loops into pieces. Note that if nothing else is runnable,\n/// It will not check for I/O, and so an infinite loop with yield() will just\n/// burn CPU.\nfuture<> yield() noexcept;\n\n/// Yield the cpu if the task quota is exhausted.\n///\n/// Check if the current continuation is preempted and yield if so. Otherwise\n/// return a ready future.\n///\n/// \\note Threads and coroutines (see seastar::thread::maybe_yield() and\n///       seastar::coroutine::maybe_yield() have their own custom variants,\n///       and the various continuation-based loops (do_for_each() and similar)\n///       do this automatically.\ninline\nfuture<> maybe_yield() noexcept {\n    if (need_preempt()) {\n        return yield();\n    } else {\n        return make_ready_future<>();\n    }\n}\n\n/// Force the reactor to check for pending I/O\n///\n/// Schedules a check for new I/O completions (disk operations completions\n/// or network packet arrival) immediately and return a future that is ready\n/// when the I/O has been polled for.\n///\n/// \\note It is very rare to need to call this function. It is better to let the\n///       reactor schedule I/O polls itself.\n/// \\note This has no effect on I/O polling on other shards.\nfuture<> check_for_io_immediately() noexcept;\n\n/// \\brief Returns a future which is not ready but is scheduled to resolve soon.\n///\n/// \\deprecated Use yield() instead, or check_for_io_immediately() if your really need it.\n[[deprecated(\"Use yield() or check_for_io_immediately()\")]]\nfuture<> later() noexcept;\n\n/// @}\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/lazy.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Cloudius Systems, Ltd.\n */\n#pragma once\n\n#include <ostream>\n#include <fmt/core.h>\n\n/// \\addtogroup logging\n/// @{\n\nnamespace seastar {\n\n/// \\brief This class is a wrapper for a lazy evaluation of a value.\n///\n/// The value is evaluated by a functor that gets no parameters, which is\n/// provided to a lazy_value constructor.\n///\n/// The instance may be created only using seastar::value_of helper function.\n///\n/// The evaluation is triggered by operator().\ntemplate<typename Func>\nclass lazy_eval {\nprivate:\n    Func _func;\n\nprivate:\n    lazy_eval(Func&& f) : _func(std::forward<Func>(f)) {}\n\npublic:\n    /// \\brief Evaluate a value.\n    ///\n    /// \\return the evaluated value\n    auto operator()() {\n        return _func();\n    }\n\n    /// \\brief Evaluate a value (const version).\n    ///\n    /// \\return the evaluated value\n    auto operator()() const {\n        return _func();\n    }\n\n    template <typename F>\n    friend lazy_eval<F> value_of(F&& func);\n};\n\n\n/// Create a seastar::lazy_eval object that will use a given functor for\n/// evaluating a value when the evaluation is triggered.\n///\n/// The actual evaluation is triggered by applying a () operator on a\n/// returned object.\n///\n/// \\param func a functor to evaluate the value\n///\n/// \\return a lazy_eval object that may be used for evaluating a value\ntemplate <typename Func>\ninline lazy_eval<Func> value_of(Func&& func) {\n    return lazy_eval<Func>(std::forward<Func>(func));\n}\n\n/// \\brief This struct is a wrapper for lazy dereferencing a pointer.\n///\n/// In particular this is to be used in situations where the value of a\n/// pointer has to be converted to string in a lazy manner. Since\n/// pointers can be null adding a check at the point of calling the\n/// log function for example, will introduce an unnecessary branch in\n/// potentially useless code. Using lazy_deref this check can be\n/// deferred to the point where the code is actually evaluated.\ntemplate <typename T>\nstruct lazy_deref_wrapper {\n    const T& p;\n\n    constexpr lazy_deref_wrapper(const T& p) : p(p) {\n    }\n};\n\n/// Create a seastar::lazy_deref_wrapper object.\n///\n/// The actual dereferencing will happen when the object is inserted\n/// into a stream. The pointer is not copied, only a reference is saved\n/// to it. Smart pointers are supported as well.\n///\n/// \\param p a raw pointer or a smart pointer\n///\n/// \\return a lazy_deref_wrapper object\ntemplate <typename T>\nlazy_deref_wrapper<T>\nlazy_deref(const T& p) {\n    return lazy_deref_wrapper<T>(p);\n}\n\n}\n\nnamespace std {\n/// Output operator for a seastar::lazy_eval<Func>\n/// This would allow printing a seastar::lazy_eval<Func> as if it's a regular\n/// value.\n///\n/// For example:\n///\n/// `logger.debug(\"heavy eval result:{}\", seastar::value_of([&] { return <heavy evaluation>; }));`\n///\n/// (If a logging level is lower than \"debug\" the evaluation will not take place.)\n///\n/// \\tparam Func a functor type\n/// \\param os ostream to print to\n/// \\param lf a reference to a lazy_eval<Func> to be printed\n///\n/// \\return os\ntemplate <typename Func>\nostream& operator<<(ostream& os, const seastar::lazy_eval<Func>& lf) {\n    return os << lf();\n}\n\ntemplate <typename Func>\nostream& operator<<(ostream& os, seastar::lazy_eval<Func>& lf) {\n    return os << lf();\n}\n\ntemplate <typename Func>\nostream& operator<<(ostream& os, seastar::lazy_eval<Func>&& lf) {\n    return os << lf();\n}\n\ntemplate <typename T>\nostream& operator<<(ostream& os, seastar::lazy_deref_wrapper<T> ld) {\n    if (ld.p) {\n        return os << *ld.p;\n    }\n\n    return os << \"null\";\n}\n}\n\ntemplate <typename Func>\nstruct fmt::formatter<seastar::lazy_eval<Func>> : fmt::formatter<string_view> {\n    template <typename FormatContext>\n    auto format(const seastar::lazy_eval<Func>& lf, FormatContext& ctx) const {\n        return fmt::format_to(ctx.out(), \"{}\", lf());\n    }\n};\n\ntemplate <typename T>\nstruct fmt::formatter<seastar::lazy_deref_wrapper<T>> : fmt::formatter<string_view> {\n    template <typename FormatContext>\n    auto format(const seastar::lazy_deref_wrapper<T>& ld, FormatContext& ctx) const {\n        if (ld.p) {\n            return fmt::format_to(ctx.out(), \"{}\", *ld.p);\n        } else {\n            return fmt::format_to(ctx.out(), \"null\");\n        }\n    }\n};\n\n/// @}\n"
  },
  {
    "path": "include/seastar/util/log-cli.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <boost/program_options.hpp>\n\n#include <algorithm>\n#include <unordered_map>\n\n#include <seastar/util/log.hh>\n#include <seastar/util/program-options.hh>\n\n#include <seastar/core/sstring.hh>\n\n\n/// \\addtogroup logging\n/// @{\nnamespace seastar {\n\n///\n/// \\brief Configure application logging at run-time with program options.\n///\nnamespace log_cli {\n\n///\n/// \\brief Options for controlling logging at run-time.\n///\nboost::program_options::options_description get_options_description();\n\nusing log_level_map = std::unordered_map<sstring, log_level>;\n\n/// Logging configuration.\nstruct options : public program_options::option_group {\n    /// \\brief Default log level for log messages.\n    ///\n    /// Valid values are trace, debug, info, warn, error.\n    /// Default: \\p info\n    /// \\see \\ref log_level.\n    program_options::value<log_level> default_log_level;\n    /// \\brief Map of logger name to log level.\n    ///\n    /// The format is `NAME0=LEVEL0[:NAME1=LEVEL1:...]`.\n    /// Valid logger names can be queried with \\p --help-loggers.\n    /// This option can be specified multiple times.\n    program_options::value<log_level_map> logger_log_level;\n    /// Select timestamp style for stdout logs.\n    ///\n    /// Default: \\p real.\n    /// \\see logger_timestamp_style.\n    program_options::value<logger_timestamp_style> logger_stdout_timestamps;\n    /// \\brief Send log output to output stream.\n    ///\n    /// As selected by \\ref logger_ostream_type.\n    /// Default: \\p true.\n    program_options::value<bool> log_to_stdout;\n    /// Send log output to.\n    ///\n    /// Default: \\p stderr.\n    /// \\see \\ref seastar::logger_ostream_type.\n    program_options::value<seastar::logger_ostream_type> logger_ostream_type;\n    /// Send log output to syslog.\n    ///\n    /// Default: \\p false.\n    program_options::value<bool> log_to_syslog;\n\n    /// Print colored tag prefix in log messages sent to output stream.\n    ///\n    /// Default: \\p true.\n    program_options::value<bool> log_with_color;\n    /// \\cond internal\n    options(program_options::option_group* parent_group);\n    /// \\endcond\n};\n\n///\n/// \\brief Print a human-friendly list of the available loggers.\n///\nvoid print_available_loggers(std::ostream& os);\n\n///\n/// \\brief Parse a log-level ({error, warn, info, debug, trace}) string, throwing \\c std::runtime_error for an invalid\n/// level.\n///\nlog_level parse_log_level(const sstring&);\n\n/// \\cond internal\nvoid parse_map_associations(const std::string& v, std::function<void(std::string, std::string)> consume_key_value);\n/// \\endcond\n\n//\n// \\brief Parse associations from loggers to log-levels and write the resulting pairs to the output iterator.\n//\n// \\throws \\c std::runtime_error for an invalid log-level.\n//\ntemplate <class OutputIter>\nvoid parse_logger_levels(const program_options::string_map& levels, OutputIter out) {\n    std::for_each(levels.begin(), levels.end(), [&out](auto&& pair) {\n        *out++ = std::make_pair(pair.first, parse_log_level(pair.second));\n    });\n}\n\n///\n/// \\brief Extract CLI options into a logging configuration.\n//\nlogging_settings extract_settings(const boost::program_options::variables_map&);\n\n///\n/// \\brief Extract \\ref options into a logging configuration.\n///\nlogging_settings extract_settings(const options&);\n\n}\n\n}\n\n/// @}\n"
  },
  {
    "path": "include/seastar/util/log-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <iterator>\n\n/// \\addtogroup logging\n/// @{\n\nnamespace seastar {\n\n/// \\cond internal\nnamespace internal {\n\n/// A buffer to format log messages into.\n///\n/// It was designed to allow formatting the entire message into it, without any\n/// intermediary buffers. To minimize the amount of reallocations it supports\n/// using an external buffer. When this is full it moves to using buffers\n/// allocated by itself.\n/// To accommodate the most widely used way of formatting messages -- fmt --,\n/// it provides an output iterator interface for writing into it.\nclass log_buf {\n    char* _begin;\n    char* _end;\n    char* _current;\n    bool _own_buf;\n    bool _alloc_failure = false;\nprivate:\n    void free_buffer() noexcept;\n    void realloc_buffer_and_append(char c) noexcept;\n\npublic:\n    // inserter_iterator is designed like std::back_insert_iterator:\n    // operator*, operator++ and operator++(int) are no-ops,\n    // and all the work happens in operator=, which pushes a character\n    // to the buffer.\n    // The iterator stores no state of its own.\n    //\n    // inserter_iterator is supposed to be used as an output_iterator.\n    // That is, assignment is expected to alternate with incrementing.\n    class inserter_iterator {\n    public:\n        using iterator_category = std::output_iterator_tag;\n        using difference_type = std::ptrdiff_t;\n        using value_type = void;\n        using pointer = void;\n        using reference = void;\n\n    private:\n        log_buf* _buf;\n\n    public:\n        explicit inserter_iterator(log_buf& buf) noexcept : _buf(&buf) { }\n\n        inserter_iterator& operator=(char c) noexcept {\n            if (__builtin_expect(_buf->_current == _buf->_end, false)) {\n                _buf->realloc_buffer_and_append(c);\n                return *this;\n            }\n            *_buf->_current++ = c;\n            return *this;\n        }\n        inserter_iterator& operator*() noexcept {\n            return *this;\n        }\n        inserter_iterator& operator++() noexcept {\n            return *this;\n        }\n        inserter_iterator operator++(int) noexcept {\n            return *this;\n        }\n    };\n\n    /// Default ctor.\n    ///\n    /// Allocates an internal buffer of 512 bytes.\n    log_buf();\n    /// External buffer ctor.\n    ///\n    /// Use the external buffer until its full, then switch to internally\n    /// allocated buffer. log_buf doesn't take ownership of the buffer.\n    log_buf(char* external_buf, size_t size) noexcept;\n    ~log_buf();\n    /// Clear the buffer, setting its position back to the start, but does not\n    /// free any buffers (after this called, size is zero, capacity is unchanged).\n    /// Any existing iterators are invalidated.\n    void clear() { _current = _begin; }\n    /// Create an output iterator which allows writing into the buffer.\n    inserter_iterator back_insert_begin() noexcept { return inserter_iterator(*this); }\n    /// The amount of data written so far.\n    size_t size() const noexcept { return _current - _begin; }\n    /// The size of the buffer.\n    size_t capacity() const noexcept { return _end - _begin; }\n    /// Read only pointer to the buffer.\n    /// Note that the buffer is not guaranteed to be null terminated. The writer\n    /// has to ensure that, should it wish to.\n    const char* data() const noexcept { return _begin; }\n    /// A view of the buffer content.\n    std::string_view view() const noexcept { return std::string_view(_begin, size()); }\n};\n\n} // namespace internal\n/// \\endcond\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/log-level.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 Cloudius Systems, Ltd.\n */\n#pragma once\n\n\n#include <iosfwd>\n#include <fmt/core.h>\n\n/// \\addtogroup logging\n/// @{\n\nnamespace seastar {\n\n\n/// \\brief log level used with \\see {logger}\n/// used with the logger.do_log method.\n/// Levels are in increasing order. That is if you want to see debug(3) logs you\n/// will also see error(0), warn(1), info(2).\n///\nenum class log_level {\n    error,\n    warn,\n    info,\n    debug,\n    trace,\n};\n\nstd::ostream& operator<<(std::ostream& out, log_level level);\nstd::istream& operator>>(std::istream& in, log_level& level);\n\n}\n\ntemplate <>\nstruct fmt::formatter<seastar::log_level> {\n    constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); }\n    auto format(seastar::log_level level, fmt::format_context& ctx) const -> decltype(ctx.out());\n};\n\n/// @}\n"
  },
  {
    "path": "include/seastar/util/log.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/log-impl.hh>\n#include <seastar/util/log-level.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/util/std-compat.hh>\n\n#include <concepts>\n#include <source_location>\n#include <unordered_map>\n#include <exception>\n#include <iosfwd>\n#include <atomic>\n#include <mutex>\n#include <type_traits>\n#include <fmt/core.h>\n#include <fmt/format.h>\n\n/// \\addtogroup logging\n/// @{\n\n\nnamespace seastar {\nclass logger;\nclass logger_registry;\n\n/// \\brief Logger class for ostream or syslog.\n///\n/// Java style api for logging.\n/// \\code {.cpp}\n/// static seastar::logger logger(\"lsa-api\");\n/// logger.info(\"Triggering compaction\");\n/// \\endcode\n/// The output format is: (depending on level)\n/// DEBUG  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n///\n/// It is possible to rate-limit log messages, see \\ref logger::rate_limit.\nclass logger {\n    sstring _name;\n    std::atomic<log_level> _level = { log_level::info };\n    static std::ostream* _out;\n    static std::atomic<bool> _ostream;\n    static std::atomic<bool> _syslog;\n    static unsigned _shard_field_width;\n#ifdef SEASTAR_BUILD_SHARED_LIBS\n    static thread_local bool silent;\n#else\n    static inline thread_local bool silent = false;\n#endif\n\npublic:\n    class log_writer {\n    public:\n        virtual ~log_writer() = default;\n        virtual internal::log_buf::inserter_iterator operator()(internal::log_buf::inserter_iterator) = 0;\n    };\n    template <typename Func>\n    requires requires (Func fn, internal::log_buf::inserter_iterator it) {\n        it = fn(it);\n    }\n    class lambda_log_writer : public log_writer {\n        Func _func;\n    public:\n        lambda_log_writer(Func&& func) : _func(std::forward<Func>(func)) { }\n        virtual ~lambda_log_writer() override = default;\n        virtual internal::log_buf::inserter_iterator operator()(internal::log_buf::inserter_iterator it) override { return _func(it); }\n    };\n\n    /// \\cond internal\n    /// \\brief used to hold the log format string and the caller's source_location.\n#ifdef SEASTAR_LOGGER_COMPILE_TIME_FMT\n    template<typename... Args>\n    struct format_info {\n        /// implicitly construct format_info from a constant format string\n        /// \\param fmt - {fmt} style format string\n        template <std::convertible_to<std::string_view> S>\n        FMT_CONSTEVAL inline format_info(const S& format,\n                           std::source_location loc = std::source_location::current()) noexcept\n            : format(format)\n            , loc(loc)\n        {}\n        /// construct format_info from an instance of \\c format_string\n        ///\n        /// this constructor is used by other printers which print to logger\n        /// \\param s a format_string\n        inline format_info(fmt::format_string<Args...> s,\n                           std::source_location loc = std::source_location::current()) noexcept\n            : format(s)\n            , loc(loc)\n        {}\n#if FMT_VERSION >= 100000\n        using runtime_format_string_t = fmt::runtime_format_string<char>;\n#else\n        using runtime_format_string_t = fmt::basic_runtime<char>;\n#endif\n        inline format_info(runtime_format_string_t s,\n                           std::source_location loc = std::source_location::current()) noexcept\n            : format(s)\n            , loc(loc)\n        {}\n        /// implicitly construct format_info with no format string.\n        FMT_CONSTEVAL format_info() noexcept\n            : format_info(\"\")\n        {}\n        fmt::format_string<Args...> format;\n        std::source_location loc;\n    };\n#ifdef __cpp_lib_type_identity\n    template <typename T>\n    using type_identity_t = typename std::type_identity<T>::type;\n#else\n    template <typename T> struct type_identity { using type = T; };\n    template <typename T> using type_identity_t = typename type_identity<T>::type;\n#endif\n    template <typename... Args>\n    using format_info_t = format_info<type_identity_t<Args>...>;\n#else  // SEASTAR_LOGGER_COMPILE_TIME_FMT\n    /// \\cond internal\n    /// \\brief used to hold the log format string and the caller's source_location.\n    struct format_info {\n        /// implicitly construct format_info from a const char* format string.\n        /// \\param fmt - {fmt} style format string\n        format_info(const char* format, std::source_location loc = std::source_location::current()) noexcept\n            : format(format)\n            , loc(loc)\n        {}\n        /// implicitly construct format_info from a std::string_view format string.\n        /// \\param fmt - {fmt} style format string_view\n        format_info(std::string_view format, std::source_location loc = std::source_location::current()) noexcept\n            : format(format)\n            , loc(loc)\n        {}\n        /// implicitly construct format_info with no format string.\n        format_info(std::source_location loc = std::source_location::current()) noexcept\n            : format()\n            , loc(loc)\n        {}\n        std::string_view format;\n        std::source_location loc;\n    };\n    // to reduce the number of #ifdefs, let's be compatible with the templated\n    // format_info\n    template <typename...>\n    using format_info_t = format_info;\n#endif // SEASTAR_LOGGER_COMPILE_TIME_FMT\n\nprivate:\n\n    // We can't use an std::function<> as it potentially allocates.\n    void do_log(log_level level, log_writer& writer);\n    void failed_to_log(std::exception_ptr ex,\n                       fmt::string_view fmt,\n                       std::source_location loc) noexcept;\n\n    class silencer {\n    public:\n        silencer() noexcept {\n            silent = true;\n        }\n\n        ~silencer() {\n            silent = false;\n        }\n    };\n\npublic:\n    /// Apply a rate limit to log message(s)\n    ///\n    /// Pass this to \\ref logger::log() to apply a rate limit to the message.\n    /// The rate limit is applied to all \\ref logger::log() calls this rate\n    /// limit is passed to. Example:\n    ///\n    ///     void handle_request() {\n    ///         static thread_local logger::rate_limit my_rl(std::chrono::seconds(10));\n    ///         // ...\n    ///         my_log.log(log_level::info, my_rl, \"a message we don't want to log on every request, only at most once each 10 seconds\");\n    ///         // ...\n    ///     }\n    ///\n    /// The rate limit ensures that at most one message per interval will be\n    /// logged. If there were messages dropped due to rate-limiting the\n    /// following snippet will be prepended to the first non-dropped log\n    /// messages:\n    ///\n    ///     (rate limiting dropped $N similar messages)\n    ///\n    /// Where $N is the number of messages dropped.\n    class rate_limit {\n        friend class logger;\n\n        using clock = lowres_clock;\n\n    private:\n        clock::duration _interval;\n        clock::time_point _next;\n        uint64_t _dropped_messages = 0;\n\n    private:\n        bool check();\n        bool has_dropped_messages() const { return bool(_dropped_messages); }\n        uint64_t get_and_reset_dropped_messages() {\n            return std::exchange(_dropped_messages, 0);\n        }\n\n    public:\n        explicit rate_limit(std::chrono::milliseconds interval);\n    };\n\npublic:\n    explicit logger(sstring name);\n    logger(logger&& x);\n    ~logger();\n\n    bool is_shard_zero() noexcept;\n\n    /// Test if desired log level is enabled\n    ///\n    /// \\param level - enum level value (info|error...)\n    /// \\return true if the log level has been enabled.\n    bool is_enabled(log_level level) const noexcept {\n        return __builtin_expect(level <= _level.load(std::memory_order_relaxed), false) && !silent;\n    }\n\n    /// logs to desired level if enabled, otherwise we ignore the log line\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void log(log_level level, format_info_t<Args...> fmt, Args&&... args) noexcept {\n        if (is_enabled(level)) {\n            try {\n                lambda_log_writer writer([&] (internal::log_buf::inserter_iterator it) {\n#ifdef SEASTAR_LOGGER_COMPILE_TIME_FMT\n                    return fmt::format_to(it, fmt.format, std::forward<Args>(args)...);\n#else\n                    return fmt::format_to(it, fmt::runtime(fmt.format), std::forward<Args>(args)...);\n#endif\n                });\n                do_log(level, writer);\n            } catch (...) {\n                failed_to_log(std::current_exception(), fmt::string_view(fmt.format), fmt.loc);\n            }\n        }\n    }\n\n    /// logs with a rate limit to desired level if enabled, otherwise we ignore the log line\n    ///\n    /// If there were messages dropped due to rate-limiting the following snippet\n    /// will be prepended to the first non-dropped log messages:\n    ///\n    ///     (rate limiting dropped $N similar messages)\n    ///\n    /// Where $N is the number of messages dropped.\n    ///\n    /// \\param rl - the \\ref rate_limit to apply to this log\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void log(log_level level, rate_limit& rl, format_info_t<Args...> fmt, Args&&... args) noexcept {\n        if (is_enabled(level) && rl.check()) {\n            try {\n                lambda_log_writer writer([&] (internal::log_buf::inserter_iterator it) {\n                    if (rl.has_dropped_messages()) {\n                        it = fmt::format_to(it, \"(rate limiting dropped {} similar messages) \", rl.get_and_reset_dropped_messages());\n                    }\n                    return fmt::format_to(it, fmt::runtime(fmt.format), std::forward<Args>(args)...);\n                });\n                do_log(level, writer);\n            } catch (...) {\n                failed_to_log(std::current_exception(), fmt::string_view(fmt.format), fmt.loc);\n            }\n        }\n    }\n\n    /// logs to desired level if enabled, otherwise we ignore the log line\n    ///\n    /// \\param writer a function which writes directly to the underlying log buffer\n    /// \\param fmt - optional logger::format_info passed down the call chain.\n    ///\n    /// This is a low level method for use cases where it is very important to\n    /// avoid any allocations. The \\arg writer will be passed a\n    /// internal::log_buf::inserter_iterator that allows it to write into the log\n    /// buffer directly, avoiding the use of any intermediary buffers.\n    /// This is rate-limited version, see \\ref rate_limit.\n    void log(log_level level, rate_limit& rl, log_writer& writer, format_info_t<> fmt = {}) noexcept {\n        if (is_enabled(level) && rl.check()) {\n            try {\n                lambda_log_writer writer_wrapper([&] (internal::log_buf::inserter_iterator it) {\n                    if (rl.has_dropped_messages()) {\n                        it = fmt::format_to(it, \"(rate limiting dropped {} similar messages) \", rl.get_and_reset_dropped_messages());\n                    }\n                    return writer(it);\n                });\n                do_log(level, writer_wrapper);\n            } catch (...) {\n                failed_to_log(std::current_exception(), \"\", fmt.loc);\n            }\n        }\n    }\n    /// \\endcond\n\n    /// Log with error tag:\n    /// ERROR  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void error(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        log(log_level::error, std::move(fmt), std::forward<Args>(args)...);\n    }\n    /// Log with warning tag:\n    /// WARN  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void warn(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        log(log_level::warn, std::move(fmt), std::forward<Args>(args)...);\n    }\n    /// Log with info tag:\n    /// INFO  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void info(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        log(log_level::info, std::move(fmt), std::forward<Args>(args)...);\n    }\n    /// Log with info tag on shard zero only:\n    /// INFO  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void info0(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        if (is_shard_zero()) {\n            log(log_level::info, std::move(fmt), std::forward<Args>(args)...);\n        }\n    }\n    /// Log with debug tag:\n    /// DEBUG  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void debug(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        log(log_level::debug, std::move(fmt), std::forward<Args>(args)...);\n    }\n    /// Log with trace tag:\n    /// TRACE  %Y-%m-%d %T,%03d [shard 0] - \"your msg\" \\n\n    ///\n    /// \\param fmt - {fmt} style format string (implictly converted to struct logger::format_info)\n    ///              or a logger::format_info passed down the call chain.\n    /// \\param args - args to print string\n    ///\n    template <typename... Args>\n    void trace(format_info_t<Args...> fmt, Args&&... args) noexcept {\n        log(log_level::trace, std::move(fmt), std::forward<Args>(args)...);\n    }\n\n    /// \\return name of the logger. Usually one logger per module\n    ///\n    const sstring& name() const noexcept {\n        return _name;\n    }\n\n    /// \\return current log level for this logger\n    ///\n    log_level level() const noexcept {\n        return _level.load(std::memory_order_relaxed);\n    }\n\n    /// \\param level - set the log level\n    ///\n    void set_level(log_level level) noexcept {\n        _level.store(level, std::memory_order_relaxed);\n    }\n\n    /// Set output stream, default is std::cerr\n    static void set_ostream(std::ostream& out) noexcept;\n\n    /// Also output to ostream. default is true\n    static void set_ostream_enabled(bool enabled) noexcept;\n\n    /// Also output to syslog. default is false\n    ///\n    /// NOTE: syslog() can block, which will stall the reactor thread.\n    ///       this should be rare (will have to fill the pipe buffer\n    ///       before syslogd can clear it) but can happen.\n    static void set_syslog_enabled(bool enabled) noexcept;\n\n    /// Set the width of shard id field in log messages\n    ///\n    /// \\c this_shard_id() is printed as a part of the prefix in logging\n    /// messages, like \"[shard 42]\", where \\c 42 is the decimal number of the\n    /// current shard id printed with a minimal width.\n    ///\n    /// \\param width the minimal width of the shard id field\n    static void set_shard_field_width(unsigned width) noexcept;\n\n    /// enable/disable the colored tag in ostream\n    ///\n    /// \\note this is a noop if fmtlib's version is less than 6.0\n    static void set_with_color(bool enabled) noexcept;\n};\n\n/// \\brief used to keep a static registry of loggers\n/// since the typical use case is to do:\n/// \\code {.cpp}\n/// static seastar::logger(\"my_module\");\n/// \\endcode\n/// this class is used to wrap around the static map\n/// that holds pointers to all logs\n///\nclass logger_registry {\n    mutable std::mutex _mutex;\n    std::unordered_map<sstring, logger*> _loggers;\npublic:\n    /// loops through all registered loggers and sets the log level\n    /// Note: this method locks\n    ///\n    /// \\param level - desired level: error,info,...\n    void set_all_loggers_level(log_level level);\n\n    /// Given a name for a logger returns the log_level enum\n    /// Note: this method locks\n    ///\n    /// \\return log_level for the given logger name\n    log_level get_logger_level(sstring name) const;\n\n    /// Sets the log level for a given logger\n    /// Note: this method locks\n    ///\n    /// \\param name - name of logger\n    /// \\param level - desired level of logging\n    void set_logger_level(sstring name, log_level level);\n\n    /// Returns a list of registered loggers\n    /// Note: this method locks\n    ///\n    /// \\return all registered loggers\n    std::vector<sstring> get_all_logger_names();\n\n    /// Registers a logger with the static map\n    /// Note: this method locks\n    ///\n    void register_logger(logger* l);\n    /// Unregisters a logger with the static map\n    /// Note: this method locks\n    ///\n    void unregister_logger(logger* l);\n    /// Swaps the logger given the from->name() in the static map\n    /// Note: this method locks\n    ///\n    void moved(logger* from, logger* to);\n};\n\nlogger_registry& global_logger_registry();\n\n/// \\brief Timestamp style.\nenum class logger_timestamp_style {\n    none,\n    boot,\n    real,\n};\n\n/// \\brief Output stream to use for logging.\nenum class logger_ostream_type {\n    none,\n    cout = 1,\n    cerr = 2,\n};\n\nstruct logging_settings final {\n    std::unordered_map<sstring, log_level> logger_levels;\n    log_level default_level;\n    bool stdout_enabled;\n    bool syslog_enabled;\n    bool with_color;\n    logger_timestamp_style stdout_timestamp_style = logger_timestamp_style::real;\n    logger_ostream_type logger_ostream = logger_ostream_type::cerr;\n};\n\n/// Shortcut for configuring the logging system all at once.\n///\nvoid apply_logging_settings(const logging_settings&);\n\n\n/// \\cond internal\n\nextern thread_local uint64_t logging_failures;\n\nsstring pretty_type_name(const std::type_info&);\n\nsstring level_name(log_level level);\n\ntemplate <typename T>\nclass logger_for : public logger {\npublic:\n    logger_for() : logger(pretty_type_name(typeid(T))) {}\n};\n\n/// \\endcond\n} // end seastar namespace\n\n// Pretty-printer for exceptions to be logged, e.g., std::current_exception().\nnamespace std {\nstd::ostream& operator<<(std::ostream&, const std::exception_ptr&);\nstd::ostream& operator<<(std::ostream&, const std::exception&);\nstd::ostream& operator<<(std::ostream&, const std::system_error&);\n}\n\ntemplate <> struct fmt::formatter<std::exception_ptr> : fmt::ostream_formatter {};\ntemplate <> struct fmt::formatter<std::exception> : fmt::ostream_formatter {};\ntemplate <> struct fmt::formatter<std::system_error> : fmt::ostream_formatter {};\n\n/// @}\n"
  },
  {
    "path": "include/seastar/util/memory-data-sink.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 ScyllaDB\n */\n\n#pragma once\n\n#include <algorithm>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\nnamespace util {\n\n#if SEASTAR_API_LEVEL >= 9\n\ntemplate <template <typename T> class Container>\nrequires requires (Container<temporary_buffer<char>>& col, size_t s) {\n    typename std::back_insert_iterator<Container<temporary_buffer<char>>>;\n}\ninline void append_buffers(Container<temporary_buffer<char>>& c, std::span<temporary_buffer<char>> bufs) {\n    std::ranges::move(bufs, std::back_inserter(c));\n}\n\ntemplate <typename Container, size_t DefaultBufferSize = 1024>\nrequires requires (Container& ct, std::span<temporary_buffer<char>> bufs) {\n    { append_buffers(ct, bufs) };\n}\nclass basic_memory_data_sink final : public data_sink_impl {\n    Container& _container;\n    const size_t _buffer_size;\n\npublic:\n    basic_memory_data_sink(Container& ct, size_t buffer_size = DefaultBufferSize)\n        : _container(ct)\n        , _buffer_size(buffer_size)\n    {\n    }\n\n    virtual future<> put(std::span<temporary_buffer<char>> bufs) override {\n        append_buffers(this->_container, bufs);\n        return make_ready_future<>();\n    }\n\n    virtual future<> flush() override {\n        return make_ready_future<>();\n    }\n\n    virtual future<> close() override {\n        return make_ready_future<>();\n    }\n\n    virtual size_t buffer_size() const noexcept override {\n        return _buffer_size;\n    }\n};\n\nusing memory_data_sink = basic_memory_data_sink<std::vector<temporary_buffer<char>>>;\n\n#endif\n\n}\n}\n"
  },
  {
    "path": "include/seastar/util/memory-data-source.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/iostream.hh>\n\nnamespace seastar {\nnamespace util {\n\ntemplate <template <typename T> class Container, typename Element>\nconcept CollectionOf = requires (Container<Element> col) {\n    { col.begin() };\n    { col.end() };\n    { *col.begin() } -> std::same_as<Element&>;\n};\n\ntemplate <template <typename T> class Container>\nrequires (CollectionOf<Container, temporary_buffer<char>> && std::is_nothrow_move_constructible_v<Container<temporary_buffer<char>>>)\nclass basic_memory_data_source final : public data_source_impl {\n    using real_container = Container<temporary_buffer<char>>;\n    real_container _bufs;\n    real_container::iterator _cur;\n\npublic:\n    explicit basic_memory_data_source(real_container&& b) noexcept\n        : _bufs(std::move(b))\n        , _cur(_bufs.begin())\n    {}\n\n    virtual future<temporary_buffer<char>> get() override {\n        return make_ready_future<temporary_buffer<char>>(_cur != _bufs.end() ? std::move(*_cur++) : temporary_buffer<char>());\n    }\n};\n\nusing memory_data_source = basic_memory_data_source<std::vector>;\n\ntemplate <template <typename T> class C>\nrequires requires (C<temporary_buffer<char>> t) { basic_memory_data_source(std::move(t)); }\ninline input_stream<char> as_input_stream(C<temporary_buffer<char>>&& bufs) {\n    return input_stream<char>(data_source(std::make_unique<basic_memory_data_source<C>>(std::move(bufs))));\n}\n\nclass temporary_buffer_data_source final : public data_source_impl {\n    temporary_buffer<char> _buf;\n\npublic:\n    explicit temporary_buffer_data_source(temporary_buffer<char>&& b) noexcept\n        : _buf(std::move(b))\n    {}\n\n    virtual future<temporary_buffer<char>> get() override {\n        return make_ready_future<temporary_buffer<char>>(std::exchange(_buf, {}));\n    }\n};\n\ninline input_stream<char> as_input_stream(temporary_buffer<char>&& buf) {\n    return input_stream<char>(data_source(std::make_unique<temporary_buffer_data_source>(std::move(buf))));\n}\n\n} // util namespace\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/memory_diagnostics.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/sstring.hh>\n#include <seastar/util/noncopyable_function.hh>\n\nnamespace seastar {\n\n\nenum class log_level;\n\nnamespace memory {\n\n/// \\brief The kind of allocation failures to dump diagnostics report for.\n///\n/// Note that if the seastar_memory logger is set to level debug, there will\n/// be a report dumped for any allocation failure, regardless of this\n/// configuration.\nenum class alloc_failure_kind {\n    /// Dump diagnostic error report for none of the allocation failures.\n    none,\n    /// Dump diagnostic error report for critical allocation failures, see\n    /// \\ref scoped_critical_alloc_section.\n    critical,\n    /// Dump diagnostic error report for all the allocation failures.\n    all,\n};\n\n/// \\brief Configure when memory diagnostics are dumped.\n///\n/// See \\ref alloc_failure_kind on available options.\n/// Applies configuration on all shards.\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind);\n\n/// \\brief Configure when memory diagnostics are dumped.\n///\n/// String version. See \\ref alloc_failure_kind on available options.\n/// Applies configuration on all shards.\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(std::string_view);\n\n/// \\brief A functor which writes its argument into the diagnostics report.\nusing memory_diagnostics_writer = noncopyable_function<void(std::string_view)>;\n\n/// \\brief Set a producer of additional diagnostic information.\n///\n/// This allows the application running on top of seastar to add its own part to\n/// the diagnostics dump. The application can supply higher level diagnostics\n/// information, that might help explain how the memory was consumed.\n///\n/// The application specific part will be added just below the main stats\n/// (free/used/total memory).\n///\n/// \\param producer - the functor to produce the additional diagnostics, specific\n///     to the application, to be added to the generated report. The producer is\n///     passed a writer functor, which it can use to add its parts to the report.\n///\n/// \\note As the report is generated at a time when allocations are failing, the\n///     producer should try as hard as possible to not allocate while producing\n///     the output.\nvoid set_additional_diagnostics_producer(noncopyable_function<void(memory_diagnostics_writer)> producer);\n\n/// Generate and return a diagnostics report as a string.\n///\n/// Note that contrary to the automated report generation (triggered by\n/// allocation failure), this method does allocate memory and can fail in\n/// low-memory conditions.\nsstring generate_memory_diagnostics_report();\n\n\nnamespace internal {\n/// Log the memory diagnostics to the internal logger in the same way as\n/// during an allocation failure, at the given log level. These reports\n/// are not rate limited, unlike the internally generated reports which\n/// are limited to 1 per 10 seconds.\n///\n/// This method attempts to avoid any allocations.\nvoid log_memory_diagnostics_report(log_level lvl);\n}\n\n} // namespace memory\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/noncopyable_function.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <seastar/util/used_size.hh>\n\n#include <concepts>\n#include <utility>\n#include <type_traits>\n#include <functional>\n#include <array>\n\nnamespace seastar {\n\ntemplate <typename Signature>\nclass noncopyable_function;\n\nnamespace internal {\n\nclass noncopyable_function_base {\nprivate:\n    noncopyable_function_base() = default;\n    static constexpr size_t nr_direct = 32;\n    union [[gnu::may_alias]] storage {\n        char direct[nr_direct];\n        void* indirect;\n    };\n    using move_type = void (*)(noncopyable_function_base* from, noncopyable_function_base* to);\n    using destroy_type = void (*)(noncopyable_function_base* func);\n\n    static void empty_move(noncopyable_function_base*, noncopyable_function_base*) {}\n    static void empty_destroy(noncopyable_function_base*) {}\n\n    static void indirect_move(noncopyable_function_base* from, noncopyable_function_base* to) {\n        using void_ptr = void*;\n        new (&to->_storage.indirect) void_ptr(from->_storage.indirect);\n    }\n\n    template <size_t N>\n    static void trivial_direct_move(noncopyable_function_base* from, noncopyable_function_base* to) {\n        // We use bytewise copy here since we lost the type. This means that\n        // we will copy any holes/padding not initialized by the move\n        // constructor in direct_vtable_for::initialize().  This is okay,\n        // since we won't use those holes/padding, but gcc doesn't know\n        // that, and complains. Silence it.\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n        // A bytewise loop doesn't optimize to a wordwise loop since\n        // the compiler can't guess if there is overlap or not. Help\n        // it out by using a temporary, which the compiler knows cannot\n        // overlap.\n        std::array<char, N> tmp;\n        // Avoid including <algorithm> just for this\n        for (size_t i = 0; i != N; ++i) {\n            tmp[i] = from->_storage.direct[i];\n        }\n        for (size_t i = 0; i != N; ++i) {\n            to->_storage.direct[i] = tmp[i];\n        }\n#pragma GCC diagnostic pop\n    }\n\n    static void trivial_direct_destroy(noncopyable_function_base*) {\n    }\n\nprivate:\n    storage _storage;\n\n    template <typename Signature>\n    friend class seastar::noncopyable_function;\n};\n\ntemplate<typename FirstArg = void, typename... RemainingArgs>\nstruct is_nothrow_if_object {\n    static constexpr bool value = is_nothrow_if_object<FirstArg>::value && is_nothrow_if_object<RemainingArgs...>::value;\n};\n\ntemplate<typename Arg>\nstruct is_nothrow_if_object<Arg> {\n    static constexpr bool value = !std::is_object_v<Arg> || std::is_nothrow_move_constructible_v<Arg>;\n};\n\ntemplate<>\nstruct is_nothrow_if_object<> {\n    static constexpr bool value = true;\n};\n\n}\n\n/// A clone of \\c std::function, but only invokes the move constructor\n/// of the contained function.\ntemplate <typename Ret, typename... Args, bool Noexcept>\nclass noncopyable_function<Ret (Args...) noexcept(Noexcept)> : private internal::noncopyable_function_base {\n    using call_type = Ret (*)(const noncopyable_function* func, Args...);\n    struct vtable {\n        const call_type call;\n        const move_type move;\n        const destroy_type destroy;\n    };\nprivate:\n    const vtable* _vtable;\nprivate:\n    static Ret empty_call(const noncopyable_function*, [[maybe_unused]] Args... args) {\n        throw std::bad_function_call();\n    }\n\n    static constexpr vtable _s_empty_vtable = {empty_call, empty_move, empty_destroy};\n\n    template <typename Func>\n    struct direct_vtable_for {\n        static Func* access(noncopyable_function* func) { return reinterpret_cast<Func*>(func->_storage.direct); }\n        static const Func* access(const noncopyable_function* func) { return reinterpret_cast<const Func*>(func->_storage.direct); }\n        static Func* access(noncopyable_function_base* func) { return access(static_cast<noncopyable_function*>(func)); }\n        static Ret call(const noncopyable_function* func, Args... args) noexcept(Noexcept) {\n            return (*access(const_cast<noncopyable_function*>(func)))(std::forward<Args>(args)...);\n        }\n        static void move(noncopyable_function_base* from, noncopyable_function_base* to) {\n            new (access(to)) Func(std::move(*access(from)));\n            destroy(from);\n        }\n        static constexpr move_type select_move_thunk() {\n            bool can_trivially_move = std::is_trivially_move_constructible_v<Func>\n                    && std::is_trivially_destructible_v<Func>;\n            return can_trivially_move ? trivial_direct_move<internal::used_size<Func>::value> : move;\n        }\n        static void destroy(noncopyable_function_base* func) {\n            access(func)->~Func();\n        }\n        static constexpr destroy_type select_destroy_thunk() {\n            return std::is_trivially_destructible_v<Func> ? trivial_direct_destroy : destroy;\n        }\n        static void initialize(Func&& from, noncopyable_function* to) {\n            new (access(to)) Func(std::move(from));\n        }\n        static constexpr vtable make_vtable() { return { call, select_move_thunk(), select_destroy_thunk() }; }\n        static const vtable s_vtable;\n    };\n    template <typename Func>\n    struct indirect_vtable_for {\n        static Func* access(noncopyable_function* func) { return reinterpret_cast<Func*>(func->_storage.indirect); }\n        static const Func* access(const noncopyable_function* func) { return reinterpret_cast<const Func*>(func->_storage.indirect); }\n        static Func* access(noncopyable_function_base* func) { return access(static_cast<noncopyable_function*>(func)); }\n        static Ret call(const noncopyable_function* func, Args... args) noexcept(Noexcept) {\n            return (*access(const_cast<noncopyable_function*>(func)))(std::forward<Args>(args)...);\n        }\n        static void destroy(noncopyable_function_base* func) {\n            delete access(func);\n        }\n        static void initialize(Func&& from, noncopyable_function* to) {\n            to->_storage.indirect = new Func(std::move(from));\n        }\n        static constexpr vtable make_vtable() { return { call, indirect_move, destroy }; }\n        static const vtable s_vtable;\n    };\n    template <typename Func, bool Direct = true>\n    struct select_vtable_for : direct_vtable_for<Func> {};\n    template <typename Func>\n    struct select_vtable_for<Func, false> : indirect_vtable_for<Func> {};\n    template <typename Func>\n    static constexpr bool is_direct() {\n        return sizeof(Func) <= nr_direct && alignof(Func) <= alignof(storage)\n                && std::is_nothrow_move_constructible_v<Func>;\n    }\n    template <typename Func>\n    struct vtable_for : select_vtable_for<Func, is_direct<Func>()> {};\npublic:\n    noncopyable_function() noexcept : _vtable(&_s_empty_vtable) {}\n    template <typename Func>\n    requires std::is_invocable_r_v<Ret, Func, Args...>\n    noncopyable_function(Func func) {\n        static_assert(!Noexcept || noexcept(std::declval<Func>()(std::declval<Args>()...)));\n        vtable_for<Func>::initialize(std::move(func), this);\n        _vtable = &vtable_for<Func>::s_vtable;\n    }\n    template <typename Object, typename... AllButFirstArg>\n    noncopyable_function(Ret (Object::*member)(AllButFirstArg...) noexcept(Noexcept)) : noncopyable_function(std::mem_fn(member)) {}\n    template <typename Object, typename... AllButFirstArg>\n    noncopyable_function(Ret (Object::*member)(AllButFirstArg...) const noexcept(Noexcept)) : noncopyable_function(std::mem_fn(member)) {}\n\n    ~noncopyable_function() {\n        _vtable->destroy(this);\n    }\n\n    noncopyable_function(const noncopyable_function&) = delete;\n    noncopyable_function& operator=(const noncopyable_function&) = delete;\n\n    noncopyable_function(noncopyable_function&& x) noexcept : _vtable(std::exchange(x._vtable, &_s_empty_vtable)) {\n        _vtable->move(&x, this);\n    }\n\n    noncopyable_function& operator=(noncopyable_function&& x) noexcept {\n        if (this != &x) {\n            this->~noncopyable_function();\n            new (this) noncopyable_function(std::move(x));\n        }\n        return *this;\n    }\n\n    Ret operator()(Args... args) const noexcept(Noexcept) {\n        static_assert(!Noexcept || internal::is_nothrow_if_object<Args...>::value);\n        return _vtable->call(this, std::forward<Args>(args)...);\n    }\n\n    explicit operator bool() const {\n        return _vtable != &_s_empty_vtable;\n    }\n};\n\n\ntemplate <typename Ret, typename... Args, bool Noexcept>\ntemplate <typename Func>\nconst typename noncopyable_function<Ret (Args...) noexcept(Noexcept)>::vtable noncopyable_function<Ret (Args...) noexcept(Noexcept)>::direct_vtable_for<Func>::s_vtable\n        = noncopyable_function<Ret (Args...) noexcept(Noexcept)>::direct_vtable_for<Func>::make_vtable();\n\n\ntemplate <typename Ret, typename... Args, bool Noexcept>\ntemplate <typename Func>\nconst typename noncopyable_function<Ret (Args...) noexcept(Noexcept)>::vtable noncopyable_function<Ret (Args...) noexcept(Noexcept)>::indirect_vtable_for<Func>::s_vtable\n        = noncopyable_function<Ret (Args...) noexcept(Noexcept)>::indirect_vtable_for<Func>::make_vtable();\n\n}\n\n"
  },
  {
    "path": "include/seastar/util/optimized_optional.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <concepts>\n#include <iostream>\n#include <optional>\n#include <type_traits>\n#include <fmt/core.h>\n\nnamespace seastar {\n\ntemplate<typename T>\nconcept OptimizableOptional =\n    std::is_default_constructible_v<T>\n        && std::is_nothrow_move_assignable_v<T>\n        && requires(const T& obj) {\n            { bool(obj) } noexcept;\n        };\n\n/// \\c optimized_optional<> is intended mainly for use with classes that store\n/// their data externally and expect pointer to this data to be always non-null.\n/// In such case there is no real need for another flag signifying whether\n/// the optional is engaged.\ntemplate<typename T>\nclass optimized_optional {\n    T _object;\npublic:\n    optimized_optional() = default;\n    optimized_optional(std::nullopt_t) noexcept { }\n    optimized_optional(const T& obj) : _object(obj) { }\n    optimized_optional(T&& obj) noexcept : _object(std::move(obj)) { }\n    optimized_optional(std::optional<T>&& obj) noexcept {\n        if (obj) {\n            _object = std::move(*obj);\n        }\n    }\n    optimized_optional(const optimized_optional&) = default;\n    optimized_optional(optimized_optional&&) = default;\n\n    optimized_optional& operator=(std::nullopt_t) noexcept {\n        _object = T();\n        return *this;\n    }\n    template<typename U>\n    requires std::same_as<std::decay_t<U>, T>\n    optimized_optional& operator=(U&& obj) noexcept {\n        _object = std::forward<U>(obj);\n        return *this;\n    }\n    optimized_optional& operator=(const optimized_optional&) = default;\n    optimized_optional& operator=(optimized_optional&&) = default;\n\n    explicit operator bool() const noexcept {\n        return bool(_object);\n    }\n\n    T* operator->() noexcept { return &_object; }\n    const T* operator->() const noexcept { return &_object; }\n\n    T& operator*() noexcept { return _object; }\n    const T& operator*() const noexcept { return _object; }\n\n    bool operator==(const optimized_optional& other) const {\n        return _object == other._object;\n    }\n    bool operator!=(const optimized_optional& other) const {\n        return _object != other._object;\n    }\n    friend std::ostream& operator<<(std::ostream& out, const optimized_optional& opt) {\n        if (!opt) {\n            return out << \"null\";\n        }\n        return out << *opt;\n    }\n};\n\n}\n\ntemplate <typename T>\nstruct fmt::formatter<seastar::optimized_optional<T>> : fmt::formatter<string_view> {\n    auto format(const seastar::optimized_optional<T>& opt, fmt::format_context& ctx) const {\n        if (opt) {\n            return fmt::format_to(ctx.out(), \"{}\", *opt);\n        }\n        return fmt::format_to(ctx.out(), \"null\");\n    }\n};\n"
  },
  {
    "path": "include/seastar/util/print_safe.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <cerrno>\n#include <concepts>\n#include <cstring>\n#include <stdio.h>\n#include <unistd.h>\n\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\n//\n// Collection of async-signal safe printing functions.\n//\n\n// Outputs string to stderr.\n// Async-signal safe.\ninline\nvoid print_safe(const char *str, size_t len) noexcept {\n    while (len) {\n        auto result = write(STDERR_FILENO, str, len);\n        if (result > 0) {\n            len -= result;\n            str += result;\n        } else if (result == 0) {\n            break;\n        } else {\n            if (errno == EINTR) {\n                // retry\n            } else {\n                break; // what can we do?\n            }\n        }\n    }\n}\n\n// Outputs string to stderr.\n// Async-signal safe.\ninline\nvoid print_safe(const char *str) noexcept {\n    print_safe(str, strlen(str));\n}\n\n// Fills a buffer with a hexadecimal representation of an integer\n// and returns a pointer to the first character.\n// For example, convert_hex_safe(buf, 4, uint16_t(12)) fills the buffer with \"   c\".\ntemplate<typename Integral, char Padding = ' '>\nrequires std::integral<Integral>\nchar* convert_hex_safe(char *buf, size_t bufsz, Integral n) noexcept {\n    const char *digits = \"0123456789abcdef\";\n    memset(buf, Padding, bufsz);\n    auto* p = buf + bufsz;\n    do {\n        SEASTAR_ASSERT(p > buf);\n        *--p = digits[n & 0xf];\n        n >>= 4;\n    } while (n);\n    return p;\n}\n\n// Fills a buffer with a zero-padded hexadecimal representation of an integer.\n// For example, convert_zero_padded_hex_safe(buf, 4, uint16_t(12)) fills the buffer with \"000c\".\ntemplate<typename Integral>\nrequires std::integral<Integral>\nvoid convert_zero_padded_hex_safe(char *buf, size_t bufsz, Integral n) noexcept {\n    convert_hex_safe<Integral, '0'>(buf, bufsz, n);\n}\n\n// Prints zero-padded hexadecimal representation of an integer to stderr.\n// For example, print_zero_padded_hex_safe(uint16_t(12)) prints \"000c\".\n// Async-signal safe.\ntemplate<typename Integral>\nrequires std::unsigned_integral<Integral>\nvoid print_zero_padded_hex_safe(Integral n) noexcept {\n    char buf[sizeof(n) * 2];\n    convert_zero_padded_hex_safe(buf, sizeof(buf), n);\n    print_safe(buf, sizeof(buf));\n}\n\n// Fills a buffer with a decimal representation of an integer.\n// The argument bufsz is the maximum size of the buffer.\n// For example, print_decimal_safe(buf, 16, 12) prints \"12\".\ntemplate<typename Integral>\nrequires std::unsigned_integral<Integral>\nsize_t convert_decimal_safe(char *buf, size_t bufsz, Integral n) noexcept {\n    char tmp[sizeof(n) * 3];\n    unsigned i = bufsz;\n    do {\n        SEASTAR_ASSERT(i > 0);\n        tmp[--i] = '0' + n % 10;\n        n /= 10;\n    } while (n);\n    memcpy(buf, tmp + i, sizeof(tmp) - i);\n    return sizeof(tmp) - i;\n}\n\n// Prints decimal representation of an integer to stderr.\n// For example, print_decimal_safe(12) prints \"12\".\n// Async-signal safe.\ntemplate<typename Integral>\nvoid print_decimal_safe(Integral n) noexcept {\n    char buf[sizeof(n) * 3];\n    unsigned i = sizeof(buf);\n    auto len = convert_decimal_safe(buf, i, n);\n    print_safe(buf, len);\n}\n}\n"
  },
  {
    "path": "include/seastar/util/process.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 Kefu Chai ( tchaikov@gmail.com )\n */\n\n#pragma once\n\n#include <sys/types.h>\n#include <algorithm>\n#include <filesystem>\n#include <initializer_list>\n#include <iterator>\n#include <string_view>\n#include <utility>\n#include <variant>\n#include <vector>\n#include <fmt/format.h>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/sstring.hh>\n\nnamespace seastar::experimental {\n\n/// The optional parameters for spawning a subprocess\n///\n/// \\note see \\c execve(2) for more details on \\c argv and \\c env.\nstruct spawn_parameters {\n    /// The arguments passed to the program\n    std::vector<sstring> argv;\n    /// The environment variables for the program\n    std::vector<sstring> env;\n};\n\n/// Interact with a spawned subprocess\n///\n/// \\note the spawned subprocess should always be \\c wait()'ed. Otherwise,\n/// the Seastar application spawning the subprocess will leave us with\n/// one ore more zombie subprocesses after it exists.\nclass process {\n    struct create_tag {};\n    /// Spawn a subprocess using \\c posix_spawn(3)\n    ///\n    /// \\param pathname the full path to the executable\n    /// \\param params parameters for spawning the subprocess\n    ///\n    /// \\returns a \\c process instance representing the spawned subprocess\n    static future<process> spawn(const std::filesystem::path& pathname,\n                                 spawn_parameters params);\n    /// Spawn a subprocess using \\c posix_spawn(3)\n    ///\n    /// \\param pathname the full path to the executable\n    ///\n    /// \\returns a \\c process instance representing the spawned subprocess\n    static future<process> spawn(const std::filesystem::path& pathname);\npublic:\n    process(create_tag, pid_t pid, file_desc&& cin, file_desc&& cout, file_desc&& cerr);\n    /// Return the process ID of the child process\n    pid_t pid() const { return _pid; }\n    /// Return an writable stream which provides input from the child process\n    output_stream<char> cin();\n    /// Return an writable stream which provides stdout output from the child process\n    input_stream<char> cout();\n    /// Return an writable stream which provides stderr output from the child process\n    input_stream<char> cerr();\n    struct wait_exited {\n        int exit_code;\n    };\n    struct wait_signaled {\n        int terminating_signal;\n    };\n    using wait_status = std::variant<wait_exited, wait_signaled>;\n    /// Wait until the child process exits or terminates\n    ///\n    /// \\returns the exit status\n    future<wait_status> wait();\n    /// Stop the process using SIGTERM\n    void terminate();\n    /// Force the process to exit using SIGKILL\n    void kill();\n\nprivate:\n    const pid_t _pid;\n    file_desc _stdin;\n    file_desc _stdout;\n    file_desc _stderr;\n\n    friend future<process> spawn_process(const std::filesystem::path&,\n                                         spawn_parameters);\n    friend future<process> spawn_process(const std::filesystem::path&);\n};\n}\n"
  },
  {
    "path": "include/seastar/util/program-options.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/format.hh>\n#include <seastar/core/sstring.hh>\n\n#include <fmt/format.h>\n\n#include <boost/any.hpp>\n#include <boost/intrusive/list.hpp>\n\n#include <string>\n#include <unordered_map>\n#include <variant>\n#include <vector>\n#include <optional>\n#include <set>\n#include <memory>\n\n/// \\defgroup program-options Program Options\n///\n/// \\brief Infrastructure for configuring a seastar application\n///\n/// The program-options infrastructure allows configuring seastar both by C++\n/// code and by command-line and/or config files. This is achieved by\n/// providing a set of self-describing and self-validating value types as well\n/// as option groups to allow grouping them into arbitrary tree structures.\n/// Seastar modules expose statically declared option structs, which derive from\n/// \\ref option_group and contain various concrete \\ref basic_value members\n/// comprising the required configuration. These structs are self-describing, and\n/// self-validating, the name of the option group as well as the list of its\n/// \\ref basic_value member can be queried run-time.\n\nnamespace seastar {\n\nnamespace program_options {\n\n///\n/// \\brief Wrapper for command-line options with arbitrary string associations.\n///\n/// This type, to be used with Boost.Program_options, will result in an option that stores an arbitrary number of\n/// string associations.\n///\n/// Values are specified in the form \"key0=value0:[key1=value1:...]\". Options of this type can be specified multiple\n/// times, and the values will be merged (with the last-provided value for a key taking precedence).\n///\n/// \\note We need a distinct type (rather than a simple type alias) for overload resolution in the implementation, but\n/// advertizing our inheritance of \\c std::unordered_map would introduce the possibility of memory leaks since STL\n/// containers do not declare virtual destructors.\n///\nclass string_map final : private std::unordered_map<sstring, sstring> {\nprivate:\n    using base = std::unordered_map<sstring, sstring>;\npublic:\n    using base::value_type;\n    using base::key_type;\n    using base::mapped_type;\n\n    using base::base;\n    using base::at;\n    using base::find;\n    using base::count;\n    using base::emplace;\n    using base::clear;\n    using base::operator[];\n    using base::begin;\n    using base::end;\n\n    friend bool operator==(const string_map&, const string_map&);\n    friend bool operator!=(const string_map&, const string_map&);\n};\n\ninline bool operator==(const string_map& lhs, const string_map& rhs) {\n    return static_cast<const string_map::base&>(lhs) == static_cast<const string_map::base&>(rhs);\n}\n\ninline bool operator!=(const string_map& lhs, const string_map& rhs) {\n    return !(lhs == rhs);\n}\n\n///\n/// \\brief Query the value of a key in a \\c string_map, or a default value if the key doesn't exist.\n///\nsstring get_or_default(const string_map&, const sstring& key, const sstring& def = sstring());\n\nstd::istream& operator>>(std::istream& is, string_map&);\nstd::ostream& operator<<(std::ostream& os, const string_map&);\n\n/// \\cond internal\n\n//\n// Required implementation hook for Boost.Program_options.\n//\nvoid validate(boost::any& out, const std::vector<std::string>& in, string_map*, int);\n\nusing list_base_hook = boost::intrusive::list_base_hook<boost::intrusive::link_mode<boost::intrusive::auto_unlink>>;\n\n} // namespace program_options\n\nenum class log_level;\nenum class logger_timestamp_style;\nenum class logger_ostream_type;\nnamespace memory {\n    enum class alloc_failure_kind;\n}\n\nnamespace program_options {\n\n/// \\endcond\n\n/// \\addtogroup program-options\n/// @{\n\nclass option_group;\n\n/// Visitor interface for \\ref option_group::describe().\n///\n/// See \\ref option_group::describe() for more details on the visiting algorithm.\nclass options_descriptor {\npublic:\n    /// Visit the start of the group.\n    ///\n    /// Called when entering a group. Groups can be nested, in which case there\n    /// will be another call to this method, before the current groups is closed.\n    /// \\returns whether visitor is interested in the group: \\p true - visit,\n    /// \\p false - skip.\n    virtual bool visit_group_start(const std::string& name, bool used) = 0;\n    /// Visit the end of the group.\n    ///\n    /// Called after all values and nested groups were visited in the current\n    /// group.\n    virtual void visit_group_end() = 0;\n\n    /// Visit value metadata, common across all value types.\n    ///\n    /// Called at the start of visiting a value. After this, a call to the\n    /// appropriate \\ref visit_value() overload (or \\ref visit_selection_value())\n    /// follows.\n    /// \\returns whether visitor is interested in the value: \\p true - visit,\n    /// \\p false - skip.\n    virtual bool visit_value_metadata(const std::string& name, const std::string& description, bool used) = 0;\n\n    /// Visit a switch (\\ref value<std::monostate>).\n    virtual void visit_value() = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const bool* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const int* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const unsigned* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const float* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const double* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const std::string* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const std::set<unsigned>* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const log_level* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const logger_timestamp_style* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const logger_ostream_type* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const memory::alloc_failure_kind* default_val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual void visit_value(const std::unordered_map<sstring, log_level>* default_val) = 0;\n\n    /// Visit a selection value (\\ref selection_value), \\p default_candidate is null when there is no default candidate.\n    virtual void visit_selection_value(const std::vector<std::string>& candidate_names, const std::size_t* default_candidate) = 0;\n};\n\n/// Visitor interface \\ref option_group::mutate().\n///\n/// See \\ref option_group::mutate() for more details on the visiting algorithm.\nclass options_mutator {\npublic:\n    /// Visit the start of the group.\n    ///\n    /// Called when entering a group. Groups can be nested, in which case there\n    /// will be another call to this method, before the current groups is closed.\n    /// \\returns whether visitor is interested in the group: \\p true - visit,\n    /// \\p false - skip.\n    virtual bool visit_group_start(const std::string& name, bool used) = 0;\n    /// Visit the end of the group.\n    ///\n    /// Called after all values and nested groups were visited in the current\n    /// group.\n    virtual void visit_group_end() = 0;\n\n    /// Visit value metadata, common across all value types.\n    ///\n    /// Called at the start of visiting a value. After this, a call to the\n    /// appropriate \\ref visit_value() overload (or \\ref visit_selection_value())\n    /// follows.\n    /// \\returns whether visitor is interested in the value: \\p true - visit,\n    /// \\p false - skip.\n    virtual bool visit_value_metadata(const std::string& name, bool used) = 0;\n\n    /// Visit a switch (\\ref value<std::monostate>), switch is set to returned value.\n    virtual bool visit_value() = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(bool& val) = 0;\n    /// Visit a value (\\ref value), \\p default_val is null when value has no default.\n    virtual bool visit_value(int& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(unsigned& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(float& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(double& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(std::string& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(std::set<unsigned>& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(log_level& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(logger_timestamp_style& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(logger_ostream_type& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(memory::alloc_failure_kind& val) = 0;\n    /// Visit and optionally mutate a value (\\ref value), should return true if value was mutated.\n    virtual bool visit_value(std::unordered_map<sstring, log_level>& val) = 0;\n\n    /// Visit and optionally mutate a selection value (\\ref selection_value), should return true if value was mutated.\n    virtual bool visit_selection_value(const std::vector<std::string>& candidate_names, std::size_t& selected_candidate) = 0;\n};\n\n/// A tag type used to construct unused \\ref option_group and \\ref basic_value objects.\nstruct unused {};\n\nclass basic_value;\n\n/// A group of options.\n///\n/// \\ref option_group is the basis for organizing options. It can hold a number\n/// of \\ref basic_value objects. These are typically also its members:\n///\n///     struct my_option_group : public option_group {\n///         value<> opt1;\n///         value<bool> opt2;\n///         ...\n///\n///         my_option_group()\n///             : option_group(nullptr, \"My option group\")\n///             , opt1(this, \"opt1\", ...\n///             , opt2(this, \"opt2\", ...\n///             , ...\n///         { }\n///     };\n///\n/// Option groups can also be nested and using this property one can build a\n/// tree of option groups and values. This tree then can be visited using the\n/// two visitor methods exposed by \\ref option_group:\n/// * \\ref describe()\n/// * \\ref mutate()\n///\n/// Using these two visitors one can easily implement glue code to expose an\n/// entire options tree to the command line. Use \\ref describe() to build the\n/// command-line level description of the objects (using e.g.\n/// boost::program_options) and after parsing the provided command-line options\n/// use \\ref mutate() to propagate the extracted values back into the options\n/// tree. How this is done is entirely up to the visitors, the above methods\n/// only offer an API to visit each group and value in the tree, they don't make\n/// any assumption about how the visitor works and what its purpose is.\nclass option_group : public list_base_hook {\n    friend class basic_value;\n\npublic:\n    using value_list_type = boost::intrusive::list<\n            basic_value,\n            boost::intrusive::base_hook<list_base_hook>,\n            boost::intrusive::constant_time_size<false>>;\n\n    using option_group_list_type = boost::intrusive::list<\n            option_group,\n            boost::intrusive::base_hook<list_base_hook>,\n            boost::intrusive::constant_time_size<false>>;\n\nprivate:\n    option_group* _parent;\n    bool _used = true;\n    std::string _name;\n    value_list_type _values;\n    option_group_list_type _subgroups;\n\npublic:\n    /// Construct an option group.\n    ///\n    /// \\param parent - the parent option-group, this option group will become a\n    /// sub option group of the parent group\n    /// \\param name - the name of the option group\n    explicit option_group(option_group* parent, std::string name);\n    /// Construct an unused option group.\n    ///\n    /// \\param parent - the parent option-group, this option group will become a\n    /// sub option group of the parent group\n    /// \\param name - the name of the option group\n    explicit option_group(option_group* parent, std::string name, unused);\n    option_group(option_group&&);\n    option_group(const option_group&) = delete;\n    virtual ~option_group() = default;\n\n    option_group& operator=(option_group&&) = delete;\n    option_group& operator=(const option_group&) = delete;\n\n    /// Does the option group has any values contained in it?\n    operator bool () const { return !_values.empty(); }\n    bool used() const { return _used; }\n    const std::string& name() const { return _name; }\n    const value_list_type& values() const { return _values; }\n    value_list_type& values() { return _values; }\n\n    /// Describe the content of this option group to the visitor.\n    ///\n    /// The content is visited in a depth-first manner:\n    /// * First the option groups itself is visited with\n    ///   \\ref options_descriptor::visit_group_start(). If this returns \\p false\n    ///   the entire content of the group, including all its subgroups and values\n    ///   are skipped and \\ref options_descriptor::visit_group_end() is called\n    ///   immediately. Otherwise visiting the content of the group proceeds.\n    /// * All the values contained therein are visited. For each value the\n    ///   following happens:\n    ///     - First \\ref options_descriptor::visit_value_metadata() is called\n    ///       with generic metadata that all values have. If this return\n    ///       \\p false the value is skipped, otherwise visiting the value\n    ///       proceeds.\n    ///     - Then the appropriate overload of\n    ///       \\ref options_descriptor::visit_value() is called, with a pointer\n    ///       to the default value of the respective value. The pointer is null\n    ///       if there is no default value.\n    ///     - For \\ref selection_value,\n    ///       \\ref options_descriptor::visit_selection_value() will be called\n    ///       instead of \\ref options_descriptor::visit_value(). After the value\n    ///       is visited, the \\ref option_group instance belonging to each\n    ///       candidate (if set) will be visited.\n    /// * All the nested \\ref option_group instances in the current group are\n    ///   visited.\n    /// * Finally \\ref options_descriptor::visit_group_end() is called.\n    void describe(options_descriptor& descriptor) const;\n    /// Mutate the content of this option group by the visitor.\n    ///\n    /// The visiting algorithm is identical to that of \\ref describe(), with the\n    /// following differences:\n    /// * \\ref options_mutator::visit_value() is allowed to mutate the value\n    ///   through the passed-in reference. It should return \\p true if it did so\n    ///   and \\p false otherwise.\n    /// * When visiting a selection value, only the nested group belonging to\n    ///   the selected value is visited afterwards.\n    void mutate(options_mutator& mutator);\n};\n\n/// A basic configuration option value.\n///\n/// This serves as the common base-class of all the concrete value types.\nclass basic_value : public list_base_hook {\n    friend class option_group;\n\npublic:\n    option_group* _group;\n    bool _used = true;\n    std::string _name;\n    std::string _description;\n\nprivate:\n    virtual void do_describe(options_descriptor& descriptor) const = 0;\n    virtual void do_mutate(options_mutator& mutator) = 0;\n\npublic:\n    basic_value(option_group& group, bool used, std::string name, std::string description);\n    basic_value(basic_value&&);\n    basic_value(const basic_value&) = delete;\n    virtual ~basic_value() = default;\n\n    basic_value& operator=(basic_value&&) = delete;\n    basic_value& operator=(const basic_value&) = delete;\n\n    bool used() const { return _used; }\n    const std::string& name() const { return _name; }\n    const std::string& description() const { return _description; }\n\n    void describe(options_descriptor& descriptor) const;\n    void mutate(options_mutator& mutator);\n};\n\n/// A configuration option value.\n///\n/// \\tparam T the type of the contained value.\ntemplate <typename T = std::monostate>\nclass value : public basic_value {\n    std::optional<T> _value;\n    bool _defaulted = true;\n\nprivate:\n    virtual void do_describe(options_descriptor& descriptor) const override {\n        auto* val = _value ? &*_value : nullptr;\n        descriptor.visit_value(val);\n    }\n    virtual void do_mutate(options_mutator& mutator) override {\n        T val;\n        if (mutator.visit_value(val)) {\n            _value = std::move(val);\n            _defaulted = false;\n        }\n    }\n    void do_set_value(T value, bool defaulted) {\n        _value = std::move(value);\n        _defaulted = defaulted;\n    }\n\npublic:\n    /// Construct a value.\n    ///\n    /// \\param group - the group containing this value\n    /// \\param name - the name of this value\n    /// \\param default_value - the default value, can be unset\n    /// \\param description - the description of the value\n    value(option_group& group, std::string name, std::optional<T> default_value, std::string description)\n        : basic_value(group, true, std::move(name), std::move(description))\n        , _value(std::move(default_value))\n    { }\n    /// Construct an unused value.\n    value(option_group& group, std::string name, unused)\n        : basic_value(group, false, std::move(name), {})\n    { }\n    value(value&&) = default;\n    /// Is there a contained value?\n    operator bool () const { return bool(_value); }\n    /// Does this value still contain a default-value?\n    bool defaulted() const { return _defaulted; }\n    /// Return the contained value, assumes there is one, see \\ref operator bool().\n    const T& get_value() const { return _value.value(); }\n    T& get_value() { return _value.value(); }\n    void set_default_value(T value) { do_set_value(std::move(value), true); }\n    void set_value(T value) { do_set_value(std::move(value), false); }\n};\n\n/// A switch-style configuration option value.\n///\n/// Contains no value, can be set or not.\ntemplate <>\nclass value<std::monostate> : public basic_value {\n    std::optional<bool> _set;\n\nprivate:\n    virtual void do_describe(options_descriptor& descriptor) const override {\n        descriptor.visit_value();\n    }\n    virtual void do_mutate(options_mutator& mutator) override {\n        bool is_set = mutator.visit_value();\n        if (_set.has_value()) {\n            // override the value only if it is not preset\n            if (is_set) {\n                _set = true;\n            }\n        } else {\n            _set = is_set;\n        }\n    }\n\npublic:\n    /// Construct a value.\n    ///\n    /// \\param group - the group containing this value\n    /// \\param name - the name of this value\n    /// \\param description - the description of the value\n    value(option_group& group, std::string name, std::string description)\n        : basic_value(group, true, std::move(name), std::move(description))\n    { }\n    /// Construct an unused value.\n    value(option_group& group, std::string name, unused)\n        : basic_value(group, false, std::move(name), {})\n    { }\n    /// Is the option set?\n    operator bool () const { return _set ? _set.value() : false; }\n    void set_value() { _set = true; }\n    void unset_value() { _set = false; }\n};\n\n/// A selection value, allows selection from multiple candidates.\n///\n/// The candidates objects are of an opaque type which may not accessible to\n/// whoever is choosing between the available candidates. This allows the user\n/// selecting between seastar internal types without exposing them.\n/// Each candidate has a name, which is what the users choose based on. Each\n/// candidate can also have an associated \\ref option_group containing related\n/// candidate-specific configuration options, allowing further configuring the\n/// selected candidate. The code exposing the candidates should document the\n/// concrete types these can be down-casted to.\ntemplate <typename T = std::monostate>\nclass selection_value : public basic_value {\npublic:\n    using deleter = std::function<void(T*)>;\n    using value_handle = std::unique_ptr<T, deleter>;\n    struct candidate {\n        std::string name;\n        value_handle value;\n        std::unique_ptr<option_group> opts;\n    };\n    using candidates = std::vector<candidate>;\n\nprivate:\n    static constexpr size_t no_selected_candidate = -1;\n\nprivate:\n    candidates _candidates;\n    size_t _selected_candidate = no_selected_candidate;\n    bool _defaulted = true;\n\npublic:\n    std::vector<std::string> get_candidate_names() const {\n        std::vector<std::string> candidate_names;\n        candidate_names.reserve(_candidates.size());\n        for (const auto& c : _candidates) {\n            candidate_names.push_back(c.name);\n        }\n        return candidate_names;\n    }\nprivate:\n    virtual void do_describe(options_descriptor& descriptor) const override {\n        descriptor.visit_selection_value(get_candidate_names(), _selected_candidate == no_selected_candidate ? nullptr : &_selected_candidate);\n        for (auto& c : _candidates) {\n            if (c.opts) {\n                c.opts->describe(descriptor);\n            }\n        }\n    }\n    virtual void do_mutate(options_mutator& mutator) override {\n        if (mutator.visit_selection_value(get_candidate_names(), _selected_candidate)) {\n            _defaulted = false;\n        }\n        if (_selected_candidate != no_selected_candidate) {\n            auto& c = _candidates.at(_selected_candidate);\n            if (c.opts) {\n                c.opts->mutate(mutator);\n            }\n        }\n    }\n    size_t find_candidate(const std::string& candidate_name) {\n        auto it = find_if(_candidates.begin(), _candidates.end(), [&] (const auto& candidate) {\n            return candidate.name == candidate_name;\n        });\n        if (it == _candidates.end()) {\n            throw std::invalid_argument(fmt::format(\"find_candidate(): failed to find candidate {}\", candidate_name));\n        }\n        return it - _candidates.begin();\n    }\n\n    option_group* do_select_candidate(std::string candidate_name, bool defaulted) {\n        _selected_candidate = find_candidate(candidate_name);\n        _defaulted = defaulted;\n        return _candidates.at(_selected_candidate).opts.get();\n    }\n\npublic:\n    /// Construct a value.\n    ///\n    /// \\param group - the group containing this value\n    /// \\param name - the name of this value\n    /// \\param candidates - the available candidates\n    /// \\param default_candidates - the name of the default candidate\n    /// \\param description - the description of the value\n    selection_value(option_group& group, std::string name, candidates candidates, std::string default_candidate, std::string description)\n        : basic_value(group, true, std::move(name), std::move(description))\n        , _candidates(std::move(candidates))\n        , _selected_candidate(find_candidate(default_candidate))\n    { }\n    selection_value(option_group& group, std::string name, candidates candidates, std::string description)\n        : basic_value(group, true, std::move(name), std::move(description))\n        , _candidates(std::move(candidates))\n    { }\n    /// Construct an unused value.\n    selection_value(option_group& group, std::string name, unused)\n        : basic_value(group, false, std::move(name), {})\n    { }\n    /// Was there a candidate selected (default also counts)?\n    operator bool () const { return _selected_candidate != no_selected_candidate; }\n    /// Is the currently selected candidate the default one?\n    bool defaulted() const { return _defaulted; }\n    /// Get the name of the currently selected candidate (assumes there is one selected, see \\operator bool()).\n    const std::string& get_selected_candidate_name() const { return _candidates.at(_selected_candidate).name; }\n    /// Get the options of the currently selected candidate (assumes there is one selected, see \\operator bool()).\n    const option_group* get_selected_candidate_opts() const { return _candidates.at(_selected_candidate).opts.get(); }\n    /// Get the options of the currently selected candidate (assumes there is one selected, see \\operator bool()).\n    option_group* get_selected_candidate_opts() { return _candidates.at(_selected_candidate).opts.get(); }\n    T& get_selected_candidate() const { return *_candidates.at(_selected_candidate).value; }\n    /// Select a candidate.\n    ///\n    /// \\param candidate_name - the name of the to-be-selected candidate.\n    option_group* select_candidate(std::string candidate_name) { return do_select_candidate(candidate_name, false); }\n    /// Select a candidate and make it the default.\n    ///\n    /// \\param candidate_name - the name of the to-be-selected candidate.\n    option_group* select_default_candidate(std::string candidate_name) { return do_select_candidate(candidate_name, true); }\n};\n\n/// @}\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/read_first_line.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <filesystem>\n#include <seastar/core/sstring.hh>\n#include <boost/lexical_cast.hpp>\n\nnamespace seastar {\n\nsstring read_first_line(std::filesystem::path sys_file);\n\ntemplate <typename Type>\nType read_first_line_as(std::filesystem::path sys_file) {\n    return boost::lexical_cast<Type>(read_first_line(sys_file));\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/reference_wrapper.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#pragma once\n\nnamespace seastar {\n\n/// \\addtogroup utilities\n/// @{\n\n/// Wrapper for lvalue references\n///\n/// reference_wrapper wraps a lvalue reference into a copyable and assignable\n/// object. It is very similar to std::reference_wrapper except that it doesn't\n/// allow implicit construction from a reference and the only way to construct\n/// it is to use either ref() or cref(). The reason for that discrepancy (and\n/// also the reason why seastar::reference_wrapper was introduced) is that it\n/// serves a different purpose than std::reference_wrapper. The latter protects\n/// references from decaying and allows copying and assigning them.\n/// seastar::reference_wrapper is used mainly to force the user to explicitly\n/// state that an object is passed by reference thus reducing the chances of\n/// the referred object being prematurely destroyed in case the execution\n/// is deferred to a continuation.\ntemplate<typename T>\nclass reference_wrapper {\n    T* _pointer;\n\n    explicit reference_wrapper(T& object) noexcept : _pointer(&object) { }\n\n    template<typename U>\n    friend reference_wrapper<U> ref(U&) noexcept;\n    template<typename U>\n    friend reference_wrapper<const U> cref(const U&) noexcept;\npublic:\n    using type = T;\n\n    operator T&() const noexcept { return *_pointer; }\n    T& get() const noexcept { return *_pointer; }\n\n};\n\n/// Wraps reference in a reference_wrapper\ntemplate<typename T>\ninline reference_wrapper<T> ref(T& object) noexcept {\n    return reference_wrapper<T>(object);\n}\n\n/// Wraps constant reference in a reference_wrapper\ntemplate<typename T>\ninline reference_wrapper<const T> cref(const T& object) noexcept {\n    return reference_wrapper<const T>(object);\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/util/sampler.hh",
    "content": "/*\n * Copyright (C) 2018 The Android Open Source Project\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *      http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\n// This file has been originally been imported from:\n// https://cs.android.com/android/platform/superproject/+/013901367630d3ec71c9f2bb3f3077bd11585301:external/perfetto/src/profiling/memory/sampler.h\n\n//\n// The code has been modified as follows:\n//\n//  - Integrated into seastar and adapted to coding style\n//  - Right now we don't account for samples multiple times (in case we have\n//  multiple loops of drawing from the exp distribution). The reason is that\n//  in our memory sampler we would have to store the weight in addition to the\n//  alloation site ptr as on free we need to know how much a sample accounted\n//  for. Hence, for now we simply always use the sampling interval.\n//  - Sampler can be turned \"off\" with a 0 sampling rate\n//  - The fast path is more optimized (as a consequence of the first point)\n//  - Provide a way to temporarily pause sampling\n//\n// Changes Copyright (C) 2023 ScyllaDB\n\n#pragma once\n\n#include <random>\n\n// See also: https://perfetto.dev/docs/design-docs/heapprofd-sampling for more\n// background of how the sampler works\n\nclass sampler {\npublic:\n    sampler() : random_gen(rd_device()) {\n        set_sampling_interval(0);\n    }\n    /// Sets the sampling interval in bytes. Setting it to 0 means to never sample\n    void set_sampling_interval(uint64_t sampling_interval) {\n        sampling_interval_ = sampling_interval;\n        if (sampling_interval_ == 0) {\n            // Set the interval very large. This means in practice we will\n            // likely never get this below zero and hence it's unlikely we will\n            // ever have to run the reset path with sampling off\n            interval_to_next_sample_ = std::numeric_limits<int64_t>::max();\n            return;\n        }\n        sampling_rate_ = 1.0 / static_cast<double>(sampling_interval_);\n        interval_to_next_sample_ = next_sampling_interval();\n    }\n    /// Updates the sampling state (byte remaining until next sample) and\n    /// returns true if this allocation of size `alloc_size` may be sampled.\n    /// Specifically, if it returns false, this allocation is definitely not\n    /// samples. However if it returns true, it is not yet determiined whether\n    /// a sample should be taken. Instead, definitely_sample should be called\n    /// and if it returns true, a sample is called for.\n    [[gnu::always_inline]]\n    bool maybe_sample(size_t alloc_size) {\n        return (interval_to_next_sample_ -= alloc_size) < 0;\n    }\n\n    /// This method should be called if maybe_sample returned true for particular\n    /// allocation. It returns true if a sample should be taken and handles\n    /// resetting the sample interval countdown.\n    bool definitely_sample(size_t alloc_size) {\n        // this will hold if maybe_sample returned false for this allocation\n        if (interval_to_next_sample_ >= 0) {\n            return false;\n        }\n        reset_interval_to_next_sample(alloc_size);\n        return sampling_interval_ != 0; // sampling interval 0 means off\n    }\n\n    uint64_t sampling_interval() const { return sampling_interval_; }\n\n    /// How much should an allocation of size `allocation_size` count for\n    size_t sample_size(size_t allocation_size) const {\n        return std::max(allocation_size, sampling_interval_);\n    }\n\n    /// RAII class to temporarily pause sampling\n    struct disable_sampling_temporarily {\n        disable_sampling_temporarily() = default;\n        disable_sampling_temporarily(sampler& sampler)\n        : sampler_(&sampler)\n        , previous_sampling_interval_(sampler_->sampling_interval_)\n        , previous_sampling_rate_(sampler_->sampling_rate_)\n        , previous_interval_to_next_sample_(sampler_->interval_to_next_sample_) {\n            sampler_->set_sampling_interval(0);\n        }\n\n        ~disable_sampling_temporarily() {\n            if (sampler_) {\n                sampler_->sampling_interval_ = previous_sampling_interval_;\n                sampler_->sampling_rate_ = previous_sampling_rate_;\n                sampler_->interval_to_next_sample_ = previous_interval_to_next_sample_;\n            }\n        }\n\n    private:\n        sampler* sampler_ = nullptr;\n        uint64_t previous_sampling_interval_ = 0; // sampling interval before pausing\n        double previous_sampling_rate_ = 0; // sampling rate before pausing\n        int64_t previous_interval_to_next_sample_ = 0; // interval to next sample before pausing\n    };\n\n    /// Pauses sampling temporarily until the returned object is destroyed. This\n    /// is more efficient and statisically more correct than doing a back and\n    /// fourth of set_sampling_interval(0) and set_sampling_interval(RATE). The\n    /// reason is that that would reset the interval to the next sample and\n    /// force a reevaluation of the exponential distribution. This method avoids\n    /// that.\n    disable_sampling_temporarily pause_sampling() {\n        return disable_sampling_temporarily(*this);\n    }\n\nprivate:\n    /// Resets interval_to_next_sample_ by repeatedly drawing from the\n    /// exponential distribution given an allocation of size `alloc_size`\n    /// breached the current interval\n    void reset_interval_to_next_sample(size_t alloc_size)\n    {\n        if (sampling_interval_ == 0) { // sampling is off\n            interval_to_next_sample_ = std::numeric_limits<int64_t>::max();\n        }\n        else {\n            // Large allocations we will just consider in whole. This avoids\n            // having to sample the distribution too many times if a large alloc\n            // took us very negative we just add the alloc size back on\n            if (alloc_size > sampling_interval_) {\n                interval_to_next_sample_ += alloc_size;\n            }\n            else {\n                while (interval_to_next_sample_ < 0) {\n                    interval_to_next_sample_ += next_sampling_interval();\n                }\n            }\n        }\n    }\n\n    int64_t next_sampling_interval() {\n        std::exponential_distribution<double> dist(sampling_rate_);\n        int64_t next = static_cast<int64_t>(dist(random_gen));\n        // We approximate the geometric distribution using an exponential\n        // distribution.\n        return next;\n    }\n\n    uint64_t sampling_interval_; // Sample every N bytes ; 0 means off\n    double sampling_rate_; // 1 / sampling_interval_ ; used by the exp distribution\n    // How many bytes remain to be allocated before we take a sample.\n    // Specifically, if this member has value N, a sample will be taken of the allocation\n    // that allocates the Nth+1 byte.\n    int64_t interval_to_next_sample_;\n    std::random_device rd_device;\n    std::mt19937_64 random_gen;\n};\n"
  },
  {
    "path": "include/seastar/util/shared_token_bucket.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#pragma once\n\n#include <atomic>\n#include <chrono>\n#include <cmath>\n#include <concepts>\n#include <cstdint>\n\nnamespace seastar {\nnamespace internal {\n\ninline uint64_t wrapping_difference(const uint64_t& a, const uint64_t& b) noexcept {\n    return std::max<int64_t>(a - b, 0);\n}\n\ninline uint64_t fetch_add(std::atomic<uint64_t>& a, uint64_t b) noexcept {\n    return a.fetch_add(b);\n}\n\ntemplate <typename T>\nconcept supports_wrapping_arithmetics = requires (T a, std::atomic<T> atomic_a, T b) {\n    { fetch_add(atomic_a, b) } noexcept -> std::same_as<T>;\n    { wrapping_difference(a, b) } noexcept -> std::same_as<T>;\n    { a + b } noexcept -> std::same_as<T>;\n};\n\nenum class capped_release { yes, no };\n\ntemplate <typename T, capped_release Capped>\nstruct rovers;\n\ntemplate <typename T>\nstruct rovers<T, capped_release::yes> {\n    using atomic_rover = std::atomic<T>;\n\n    atomic_rover tail;\n    atomic_rover head;\n    atomic_rover ceil;\n\n    rovers(T limit) noexcept : tail(0), head(0), ceil(limit) {}\n\n    T max_extra(T) const noexcept {\n        return wrapping_difference(ceil.load(std::memory_order_relaxed), head.load(std::memory_order_relaxed));\n    }\n\n    void release(T tokens) {\n        fetch_add(ceil, tokens);\n    }\n};\n\ntemplate <typename T>\nstruct rovers<T, capped_release::no> {\n    using atomic_rover = std::atomic<T>;\n\n    atomic_rover tail;\n    atomic_rover head;\n\n    rovers(T) noexcept : tail(0), head(0) {}\n\n    T max_extra(T limit) const noexcept {\n        return wrapping_difference(tail.load(std::memory_order_relaxed) + limit, head.load(std::memory_order_relaxed));\n    }\n\n    void release(T) = delete;\n};\n\ntemplate <typename T, typename Period, capped_release Capped, typename Clock = std::chrono::steady_clock>\nrequires std::is_nothrow_copy_constructible_v<T> && supports_wrapping_arithmetics<T>\nclass shared_token_bucket {\n    using rate_resolution = std::chrono::duration<double, Period>;\n\n    T _replenish_rate;\n    const T _replenish_limit;\n    const T _replenish_threshold;\n    std::atomic<typename Clock::time_point> _replenished;\n\n    /*\n     * The token bucket is implemented as a pair of wrapping monotonic\n     * counters (called rovers) one chasing the other. Getting a token\n     * from the bucket is increasing the tail, replenishing a token back\n     * is increasing the head. If increased tail overruns the head then\n     * the bucket is empty and we have to wait. The shard that grabs tail\n     * earlier will be \"woken up\" earlier, so they form a queue.\n     *\n     * The top rover is needed to implement two buckets actually. The\n     * tokens are not just replenished by timer. They are replenished by\n     * timer from the second bucket. And the second bucket only get a\n     * token in it after the request that grabbed it from the first bucket\n     * completes and returns it back.\n     */\n\n    using rovers_t = rovers<T, Capped>;\n    static_assert(rovers_t::atomic_rover::is_always_lock_free);\n    rovers_t _rovers;\n\n    T tail() const noexcept { return _rovers.tail.load(std::memory_order_relaxed); }\n    T head() const noexcept { return _rovers.head.load(std::memory_order_relaxed); }\n\n    /*\n     * Need to make sure that the multiplication in accumulated_in() doesn't\n     * overflow. Not to introduce an extra branch there, define that the\n     * replenish period is not larger than this delta and limit the rate with\n     * the value that can overflow it.\n     *\n     * The additional /=2 in max_rate math is to make extra sure that the\n     * overflow doesn't break wrapping_difference sign tricks.\n     */\n    static constexpr rate_resolution max_delta = std::chrono::duration_cast<rate_resolution>(std::chrono::hours(1));\npublic:\n    static constexpr T max_rate = std::numeric_limits<T>::max() / 2 / max_delta.count();\n    static constexpr capped_release is_capped = Capped;\n\nprivate:\n    static constexpr T accumulated(T rate, rate_resolution delta) noexcept {\n        return std::round(rate * delta.count());\n    }\n#ifndef __clang__\n    // std::round() is constexpr only since C++23 (but g++ doesn't care)\n    static_assert(accumulated(max_rate, max_delta) <= std::numeric_limits<T>::max());\n#endif\n\npublic:\n    shared_token_bucket(T rate, T limit, T threshold, bool add_replenish_iffset = true) noexcept\n            : _replenish_rate(std::min(rate, max_rate))\n            , _replenish_limit(limit)\n            , _replenish_threshold(std::clamp(threshold, (T)1, limit))\n            // pretend it was replenished yesterday to spot overflows early\n            , _replenished(Clock::now() - std::chrono::hours(add_replenish_iffset ? 24 : 0))\n            , _rovers(_replenish_limit)\n    {}\n\n    T grab(T tokens) noexcept {\n        return fetch_add(_rovers.tail, tokens) + tokens;\n    }\n\n    void release(T tokens) noexcept {\n        _rovers.release(tokens);\n    }\n\n    void refund(T tokens) noexcept {\n        fetch_add(_rovers.head, tokens);\n    }\n\n    void replenish(typename Clock::time_point now) noexcept {\n        auto ts = _replenished.load(std::memory_order_relaxed);\n\n        if (now <= ts) {\n            return;\n        }\n\n        auto delta = now - ts;\n        auto extra = accumulated_in(delta);\n\n        if (extra >= _replenish_threshold) {\n            if (!_replenished.compare_exchange_weak(ts, ts + delta)) {\n                return; // next time or another shard\n            }\n\n            fetch_add(_rovers.head, std::min(extra, _rovers.max_extra(_replenish_limit)));\n        }\n    }\n\n    T deficiency(T from) const noexcept {\n        return wrapping_difference(from, head());\n    }\n\n    template <typename Rep, typename Per>\n    static auto rate_cast(const std::chrono::duration<Rep, Per> delta) noexcept {\n        return std::chrono::duration_cast<rate_resolution>(delta);\n    }\n\n    // the number of tokens accumulated for the given time frame\n    template <typename Rep, typename Per>\n    T accumulated_in(const std::chrono::duration<Rep, Per> delta) const noexcept {\n       auto delta_at_rate = std::min(rate_cast(delta), max_delta);\n       return accumulated(_replenish_rate, delta_at_rate);\n    }\n\n    // Estimated time to process the given amount of tokens\n    // (peer of accumulated_in helper)\n    rate_resolution duration_for(T tokens) const noexcept {\n        return rate_resolution(double(tokens) / _replenish_rate);\n    }\n\n    T rate() const noexcept { return _replenish_rate; }\n    T limit() const noexcept { return _replenish_limit; }\n    T threshold() const noexcept { return _replenish_threshold; }\n    typename Clock::time_point replenished_ts() const noexcept { return _replenished; }\n\n    void update_rate(T rate) noexcept {\n        _replenish_rate = std::min(rate, max_rate);\n    }\n};\n\n} // internal namespace\n} // seastar namespace\n"
  },
  {
    "path": "include/seastar/util/short_streams.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/temporary_buffer.hh>\n\nnamespace seastar {\n\nnamespace util {\n\n/// Returns all bytes from the stream until eof, accessible in chunks.\n///\n/// \\note use only on short streams to avoid running out of memory.\n///\n/// \\param inp \\ref input_stream to be read.\nfuture<std::vector<temporary_buffer<char>>> read_entire_stream(input_stream<char>& inp);\n\n/// Returns all bytes from the stream until eof as a single buffer.\n///\n/// \\note use only on short streams to avoid running out of memory.\n///\n/// \\param inp \\ref input_stream to be read.\nfuture<sstring> read_entire_stream_contiguous(input_stream<char>& inp);\n\n/// Ignores all bytes in the stream, until eos.\n///\n/// \\param inp \\ref input_stream to be read.\nfuture<> skip_entire_stream(input_stream<char>& inp);\n\n} // namespace util\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/spinlock.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/util/assert.hh>\n\n#include <atomic>\n\n#if defined(__x86_64__) || defined(__i386__)\n#include <xmmintrin.h>\n#endif\n\nnamespace seastar {\n\nnamespace internal {\n#if defined(__x86_64__) || defined(__i386__)\n\n/// \\brief Puts the current CPU thread into a \"relaxed\" state.\n///\n/// This function is supposed to significantly improve the performance in situations like spinlocks when process spins\n/// in a tight loop waiting for a lock. The actual implementation is different on different platforms. For more details\n/// look for \"Pause Intrinsic\" for x86 version, and for \"yield\" assembly instruction documentation for Power platform.\n[[gnu::always_inline]]\ninline void cpu_relax() {\n    _mm_pause();\n}\n\n#elif defined(__PPC__)\n\n[[gnu::always_inline]]\ninline void cpu_relax() {\n    __asm__ volatile(\"yield\");\n}\n\n#elif defined(__s390x__) || defined(__zarch__)\n\n// FIXME: there must be a better way\n[[gnu::always_inline]]\ninline void cpu_relax() {}\n\n#elif defined(__aarch64__)\n\n[[gnu::always_inline]]\ninline void cpu_relax() {\n    __asm__ volatile(\"yield\");\n}\n\n#else\n\n[[gnu::always_inline]]\ninline void cpu_relax() {}\n#warning \"Using an empty cpu_relax() for this architecture\"\n\n#endif\n\n\n}\n\nnamespace util {\n\n// Spin lock implementation.\n// BasicLockable.\n// Async-signal safe.\n// unlock() \"synchronizes with\" lock().\nclass spinlock {\n    std::atomic<bool> _busy = { false };\npublic:\n    spinlock() = default;\n    spinlock(const spinlock&) = delete;\n    ~spinlock() { SEASTAR_ASSERT(!_busy.load(std::memory_order_relaxed)); }\n    bool try_lock() noexcept {\n        return !_busy.exchange(true, std::memory_order_acquire);\n    }\n    void lock() noexcept {\n        while (_busy.exchange(true, std::memory_order_acquire)) {\n            while (_busy.load(std::memory_order_relaxed)) {\n                internal::cpu_relax();\n            }\n        }\n    }\n    void unlock() noexcept {\n        _busy.store(false, std::memory_order_release);\n    }\n};\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/std-compat.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB\n */\n\n#pragma once\n\n\n\n#if __has_include(<memory_resource>)\n#include <memory_resource>\n#else\n#include <experimental/memory_resource>\nnamespace std::pmr {\n    using namespace std::experimental::pmr;\n}\n#endif\n\n#include <source_location>\n\n// Defining SEASTAR_ASAN_ENABLED in here is a bit of a hack, but\n// convenient since it is build system independent and in practice\n// everything includes this header.\n\n#ifndef __has_feature\n#define __has_feature(x) 0\n#endif\n\n// clang uses __has_feature, gcc defines __SANITIZE_ADDRESS__\n#if __has_feature(address_sanitizer) || defined(__SANITIZE_ADDRESS__)\n#define SEASTAR_ASAN_ENABLED\n#endif\n\nnamespace seastar::compat {\n\n// Deprecated: use std::source_location directly.\n// This alias is maintained for backwards compatibility with external users.\nusing source_location\n    [[deprecated(\"Use std::source_location instead of seastar::compat::source_location\")]]\n    = std::source_location;\n\n}\n\n#if defined(__GNUC__) && !defined(__clang__)\n// GCC Workaround: Strip source_location to prevent ICE during RTL expansion.\n// See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114675\n#define SEASTAR_COROUTINE_LOC_PARAM\n#define SEASTAR_COROUTINE_LOC_STORE(promise) (void)0\n#else\n// Standard/Clang: Capture source location naturally.\n// Includes the leading comma to mix cleanly into argument lists.\n#define SEASTAR_COROUTINE_LOC_PARAM \\\n    , std::source_location sl = std::source_location::current()\n\n#define SEASTAR_COROUTINE_LOC_STORE(promise) \\\n    (promise).update_resume_point(sl)\n#endif\n"
  },
  {
    "path": "include/seastar/util/string_utils.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n//\n// request.hpp\n// ~~~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n\n#pragma once\n\n#include <cstring>\n#include <stdio.h>\n\n#include <seastar/core/sstring.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\n//\n// Collection of utilities for working with strings .\n//\n\nstruct string_view_hash {\n    using is_transparent = void;\n    size_t operator()(const char *txt) const {\n        return std::hash<std::string_view>{}(txt);\n    }\n    size_t operator()(std::string_view txt) const {\n        return std::hash<std::string_view>{}(txt);\n    }\n    size_t operator()(const sstring &txt) const {\n        return std::hash<sstring>()(txt);\n    }\n};\n\nstruct case_insensitive_cmp {\n    bool operator()(const sstring& s1, const sstring& s2) const {\n        return std::equal(s1.begin(), s1.end(), s2.begin(), s2.end(),\n                [](char a, char b) { return ::tolower(a) == ::tolower(b); });\n    }\n};\n\nstruct case_insensitive_hash {\n    size_t operator()(sstring s) const {\n        std::transform(s.begin(), s.end(), s.begin(), ::tolower);\n        return std::hash<sstring>()(s);\n    }\n};\n\n\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/tmp_file.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace seastar {\n\nconst std::filesystem::path& default_tmpdir();\nvoid set_default_tmpdir(std::filesystem::path);\n\nclass tmp_file {\n    std::filesystem::path _path;\n    file _file;\n    bool _is_open = false;\n\n    static_assert(std::is_nothrow_constructible_v<std::filesystem::path>,\n        \"filesystem::path's constructor must not throw\");\n    static_assert(std::is_nothrow_move_constructible_v<std::filesystem::path>,\n        \"filesystem::path's move constructor must not throw\");\npublic:\n    tmp_file() noexcept = default;\n    tmp_file(const tmp_file&) = delete;\n    tmp_file(tmp_file&& x) noexcept;\n\n    tmp_file& operator=(tmp_file&&) noexcept = default;\n\n    ~tmp_file();\n\n    future<> open(std::filesystem::path path_template = default_tmpdir(),\n            open_flags oflags = open_flags::rw,\n            file_open_options options = {}) noexcept;\n    future<> close() noexcept;\n    future<> remove() noexcept;\n\n    template <typename Func>\n    static future<> do_with(std::filesystem::path path_template, Func&& func,\n            open_flags oflags = open_flags::rw,\n            file_open_options options = {}) noexcept {\n        static_assert(std::is_nothrow_move_constructible_v<Func>,\n            \"Func's move constructor must not throw\");\n        return seastar::do_with(tmp_file(), [func = std::move(func), path_template = std::move(path_template), oflags, options = std::move(options)] (tmp_file& t) mutable {\n            return t.open(std::move(path_template), oflags, std::move(options)).then([&t, func = std::move(func)] () mutable {\n                return func(t);\n            }).finally([&t] {\n                return t.close().finally([&t] {\n                    return t.remove();\n                });\n            });\n        });\n    }\n\n    template <typename Func>\n    static future<> do_with(Func&& func) noexcept {\n        return do_with(default_tmpdir(), std::move(func));\n    }\n\n    bool has_path() const {\n        return !_path.empty();\n    }\n\n    bool is_open() const {\n        return _is_open;\n    }\n\n    const std::filesystem::path& get_path() const {\n        return _path;\n    }\n\n    file& get_file() {\n        return _file;\n    }\n};\n\n/// Returns a future for an opened tmp_file exclusively created by the function.\n///\n/// \\param path_template - path where the file is to be created,\n///                        optionally including a template for the file name.\n/// \\param oflags - optional \\ref open_flags (open_flags::create | open_flags::exclusive are added to those by default)\n/// \\param options - additional \\ref file_open_options, e.g. for setting the created file permission.\n///\n/// \\note\n///    path_template may optionally include a filename template in the last component of the path.\n///    The template is indicated by two or more consecutive XX's.\n///    Those will be replaced in the result path by a unique string.\n///\n///    If no filename template is found, then path_template is assumed to refer to the directory where\n///    the temporary file is to be created at (a.k.a. the parent directory) and `default_tmp_name_template`\n///    is appended to the path as the filename template.\n///\n///    The parent directory must exist and be writable to the current process.\n///\nfuture<tmp_file> make_tmp_file(std::filesystem::path path_template = default_tmpdir(),\n        open_flags oflags = open_flags::rw, file_open_options options = {}) noexcept;\n\nclass tmp_dir {\n    std::filesystem::path _path;\n\npublic:\n    tmp_dir() = default;\n    tmp_dir(const tmp_dir&) = delete;\n    tmp_dir(tmp_dir&& x) = default;\n\n    tmp_dir& operator=(tmp_dir&&) noexcept = default;\n\n    ~tmp_dir();\n\n    future<> create(std::filesystem::path path_template = default_tmpdir(),\n            file_permissions create_permissions = file_permissions::default_dir_permissions) noexcept;\n    future<> remove() noexcept;\n\n    template <typename Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    static future<> do_with(std::filesystem::path path_template, Func&& func,\n            file_permissions create_permissions = file_permissions::default_dir_permissions) noexcept {\n        static_assert(std::is_nothrow_move_constructible_v<Func>,\n            \"Func's move constructor must not throw\");\n        return seastar::do_with(tmp_dir(), [func = std::move(func), path_template = std::move(path_template), create_permissions] (tmp_dir& t) mutable {\n            return t.create(std::move(path_template), create_permissions).then([&t, func = std::move(func)] () mutable {\n                return func(t);\n            }).finally([&t] {\n                return t.remove();\n            });\n        });\n    }\n\n    template <typename Func>\n    static future<> do_with(Func&& func) noexcept {\n        return do_with(default_tmpdir(), std::move(func));\n    }\n\n    template <typename Func>\n    requires std::is_nothrow_move_constructible_v<Func>\n    static future<> do_with_thread(Func&& func) noexcept {\n        static_assert(std::is_nothrow_move_constructible_v<Func>,\n            \"Func's move constructor must not throw\");\n        return async([func = std::move(func)] () mutable {\n            auto t = tmp_dir();\n            t.create().get();\n            futurize_invoke(func, t).finally([&t] {\n                return t.remove();\n            }).get();\n        });\n    }\n\n    bool has_path() const {\n        return !_path.empty();\n    }\n\n    const std::filesystem::path& get_path() const {\n        return _path;\n    }\n};\n\n/// Returns a future for a tmp_dir exclusively created by the function.\n///\n/// \\param path_template - path where the file is to be created,\n///                        optionally including a template for the file name.\n/// \\param create_permissions - optional permissions for the newly created directory.\n///\n/// \\note\n///    path_template may optionally include a name template in the last component of the path.\n///    The template is indicated by two or more consecutive XX's.\n///    Those will be replaced in the result path by a unique string.\n///\n///    If no name template is found, then path_template is assumed to refer to the directory where\n///    the temporary dir is to be created at (a.k.a. the parent directory) and `default_tmp_name_template`\n///    is appended to the path as the name template for the to-be-created directory.\n///\n///    The parent directory must exist and be writable to the current process.\n///\nfuture<tmp_dir> make_tmp_dir(std::filesystem::path path_template = default_tmpdir(),\n        file_permissions create_permissions = file_permissions::default_dir_permissions) noexcept;\n\n} // namespace seastar\n"
  },
  {
    "path": "include/seastar/util/transform_iterator.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\nnamespace seastar {\n\ntemplate <typename Iterator, typename Func>\nclass transform_iterator {\n    Iterator _i;\n    Func _f;\npublic:\n    transform_iterator(Iterator i, Func f) : _i(i), _f(f) {}\n    auto operator*() { return _f(*_i); }\n    transform_iterator& operator++() {\n        ++_i;\n        return *this;\n    }\n    transform_iterator operator++(int) {\n        transform_iterator ret(*this);\n        _i++;\n        return ret;\n    }\n    bool operator==(const transform_iterator& x) const {\n        return _i == x._i;\n    }\n    bool operator!=(const transform_iterator& x) const {\n        return !operator==(x);\n    }\n};\n\ntemplate <typename Iterator, typename Func>\ninline\ntransform_iterator<Iterator, Func>\nmake_transform_iterator(Iterator i, Func f) {\n    return transform_iterator<Iterator, Func>(i, f);\n}\n\n}\n"
  },
  {
    "path": "include/seastar/util/tuple_utils.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB.\n */\n\n#pragma once\n\n#include <tuple>\n#include <utility>\n#include <stddef.h>\n\nnamespace seastar {\n\n/// \\cond internal\nnamespace internal {\n\ntemplate<typename Tuple>\nTuple untuple(Tuple t) {\n    return t;\n}\n\ntemplate<typename T>\nT untuple(std::tuple<T> t) {\n    return std::get<0>(std::move(t));\n}\n\ntemplate<typename Tuple, typename Function, size_t... I>\nvoid tuple_for_each_helper(Tuple&& t, Function&& f, std::index_sequence<I...>&&) {\n    auto ignore_me = { (f(std::get<I>(std::forward<Tuple>(t))), 1)... };\n    (void)ignore_me;\n}\n\ntemplate<typename Tuple, typename MapFunction, size_t... I>\nauto tuple_map_helper(Tuple&& t, MapFunction&& f, std::index_sequence<I...>&&) {\n    return std::make_tuple(f(std::get<I>(std::forward<Tuple>(t)))...);\n}\n\ntemplate<size_t I, typename IndexSequence>\nstruct prepend;\n\ntemplate<size_t I, size_t... Is>\nstruct prepend<I, std::index_sequence<Is...>> {\n    using type = std::index_sequence<I, Is...>;\n};\n\ntemplate<template<typename> class Filter, typename Tuple, typename IndexSequence>\nstruct tuple_filter;\n\ntemplate<template<typename> class Filter, typename T, typename... Ts, size_t I, size_t... Is>\nstruct tuple_filter<Filter, std::tuple<T, Ts...>, std::index_sequence<I, Is...>> {\n    using tail = typename tuple_filter<Filter, std::tuple<Ts...>, std::index_sequence<Is...>>::type;\n    using type = std::conditional_t<Filter<T>::value, typename prepend<I, tail>::type, tail>;\n};\n\ntemplate<template<typename> class Filter>\nstruct tuple_filter<Filter, std::tuple<>, std::index_sequence<>> {\n    using type = std::index_sequence<>;\n};\n\ntemplate<typename Tuple, size_t... I>\nauto tuple_filter_helper(Tuple&& t, std::index_sequence<I...>&&) {\n    return std::make_tuple(std::get<I>(std::forward<Tuple>(t))...);\n}\n\n}\n/// \\endcond\n\n/// \\addtogroup utilities\n/// @{\n\n/// Applies type transformation to all types in tuple\n///\n/// Member type `type` is set to a tuple type which is a result of applying\n/// transformation `MapClass<T>::type` to each element `T` of the input tuple\n/// type.\n///\n/// \\tparam MapClass class template defining type transformation\n/// \\tparam Tuple input tuple type\ntemplate<template<typename> class MapClass, typename Tuple>\nstruct tuple_map_types;\n\n/// @}\n\ntemplate<template<typename> class MapClass, typename... Elements>\nstruct tuple_map_types<MapClass, std::tuple<Elements...>> {\n    using type = std::tuple<typename MapClass<Elements>::type...>;\n};\n\n/// \\addtogroup utilities\n/// @{\n\n/// Filters elements in tuple by their type\n///\n/// Returns a tuple containing only those elements which type `T` caused\n/// expression FilterClass<T>::value to be true.\n///\n/// \\tparam FilterClass class template having an element value set to true for elements that\n///                     should be present in the result\n/// \\param t tuple to filter\n/// \\return a tuple contaning elements which type passed the test\ntemplate<template<typename> class FilterClass, typename... Elements>\nauto tuple_filter_by_type(const std::tuple<Elements...>& t) {\n    using sequence = typename internal::tuple_filter<FilterClass, std::tuple<Elements...>,\n                                                     std::index_sequence_for<Elements...>>::type;\n    return internal::tuple_filter_helper(t, sequence());\n}\ntemplate<template<typename> class FilterClass, typename... Elements>\nauto tuple_filter_by_type(std::tuple<Elements...>&& t) {\n    using sequence = typename internal::tuple_filter<FilterClass, std::tuple<Elements...>,\n                                                     std::index_sequence_for<Elements...>>::type;\n    return internal::tuple_filter_helper(std::move(t), sequence());\n}\n\n/// Applies function to all elements in tuple\n///\n/// Applies given function to all elements in the tuple and returns a tuple\n/// of results.\n///\n/// \\param t original tuple\n/// \\param f function to apply\n/// \\return tuple of results returned by f for each element in t\ntemplate<typename Function, typename... Elements>\nauto tuple_map(const std::tuple<Elements...>& t, Function&& f) {\n    return internal::tuple_map_helper(t, std::forward<Function>(f),\n                                      std::index_sequence_for<Elements...>());\n}\ntemplate<typename Function, typename... Elements>\nauto tuple_map(std::tuple<Elements...>&& t, Function&& f) {\n    return internal::tuple_map_helper(std::move(t), std::forward<Function>(f),\n                                      std::index_sequence_for<Elements...>());\n}\n\n/// Iterate over all elements in tuple\n///\n/// Iterates over given tuple and calls the specified function for each of\n/// it elements.\n///\n/// \\param t a tuple to iterate over\n/// \\param f function to call for each tuple element\ntemplate<typename Function, typename... Elements>\nvoid tuple_for_each(const std::tuple<Elements...>& t, Function&& f) {\n    return internal::tuple_for_each_helper(t, std::forward<Function>(f),\n                                           std::index_sequence_for<Elements...>());\n}\ntemplate<typename Function, typename... Elements>\nvoid tuple_for_each(std::tuple<Elements...>& t, Function&& f) {\n    return internal::tuple_for_each_helper(t, std::forward<Function>(f),\n                                           std::index_sequence_for<Elements...>());\n}\ntemplate<typename Function, typename... Elements>\nvoid tuple_for_each(std::tuple<Elements...>&& t, Function&& f) {\n    return internal::tuple_for_each_helper(std::move(t), std::forward<Function>(f),\n                                           std::index_sequence_for<Elements...>());\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/util/used_size.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <stddef.h>\n#include <type_traits>\n\nnamespace seastar {\nnamespace internal {\n// Empty types have a size of 1, but that byte is not actually\n// used. This helper is used to avoid accessing that byte.\ntemplate<typename T>\nstruct used_size {\n    static constexpr size_t value = std::is_empty_v<T> ? 0 : sizeof(T);\n};\n}\n}\n"
  },
  {
    "path": "include/seastar/util/variant_utils.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB.\n */\n\n#pragma once\n\n#include <variant>\n\nnamespace seastar {\n\n/// \\cond internal\nnamespace internal {\n\ntemplate<typename... Args>\nstruct variant_visitor : Args... {\n    variant_visitor(Args&&... a) : Args(std::move(a))... {}\n    using Args::operator()...;\n};\n\ntemplate<typename... Args> variant_visitor(Args&&...) -> variant_visitor<Args...>;\n\n}\n/// \\endcond\n\n/// \\addtogroup utilities\n/// @{\n\n/// Creates a visitor from function objects.\n///\n/// Returns a visitor object comprised of the provided function objects. Can be\n/// used with std::variant or any other custom variant implementation.\n///\n/// \\param args function objects each accepting one or some types stored in the variant as input\ntemplate <typename... Args>\nauto make_visitor(Args&&... args)\n{\n    return internal::variant_visitor<Args...>(std::forward<Args>(args)...);\n}\n\n/// Applies a static visitor comprised of supplied lambdas to a variant.\n/// Note that the lambdas should cover all the types that the variant can possibly hold.\n///\n/// Returns the common type of return types of all lambdas.\n///\n/// \\tparam Variant the type of a variant\n/// \\tparam Args types of lambda objects\n/// \\param variant the variant object\n/// \\param args lambda objects each accepting one or some types stored in the variant as input\n/// \\return\ntemplate <typename Variant, typename... Args>\ninline decltype(auto) visit(Variant&& variant, Args&&... args)\n{\n    static_assert(sizeof...(Args) > 0, \"At least one lambda must be provided for visitation\");\n    return std::visit(\n        make_visitor(std::forward<Args>(args)...),\n        std::forward<Variant>(variant));\n}\n\nnamespace internal {\ntemplate<typename... Args>\nstruct castable_variant {\n    std::variant<Args...> var;\n\n    template<typename... SuperArgs>\n    operator std::variant<SuperArgs...>() && {\n        return std::visit([] (auto&& x) {\n            return std::variant<SuperArgs...>(std::move(x));\n        }, var);\n    }\n};\n}\n\ntemplate<typename... Args>\ninternal::castable_variant<Args...> variant_cast(std::variant<Args...>&& var) {\n    return {std::move(var)};\n}\n\ntemplate<typename... Args>\ninternal::castable_variant<Args...> variant_cast(const std::variant<Args...>& var) {\n    return {var};\n}\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/websocket/client.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/http/response_parser.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/websocket/common.hh>\n\nnamespace seastar::experimental::websocket {\n\n/// \\addtogroup websocket\n/// @{\n\n/*!\n * \\brief a client WebSocket connection\n *\n * Represents a single WebSocket connection initiated by a client.\n * Handles the HTTP upgrade handshake and WebSocket frame exchange.\n */\ntemplate <bool text_frame = false>\nclass client_connection : public basic_connection<true, text_frame> {\n    http_response_parser _http_parser;\n    sstring _resource;\n    sstring _host;\n    sstring _websocket_key;\n\n    future<> send_http_upgrade_request();\n    future<> read_http_upgrade_response();\n\npublic:\n    /*!\n     * \\param fd established socket used for communication\n     * \\param resource the URI path (e.g., \"/ws\")\n     * \\param host the Host header value\n     * \\param subprotocol optional subprotocol name\n     * \\param handler application handler for incoming/outgoing data\n     */\n    client_connection(connected_socket&& fd, sstring resource, sstring host,\n                      sstring subprotocol, handler_t handler);\n\n    /*!\n     * \\brief Run the WebSocket client connection.\n     */\n    future<> process();\n\n     /*!\n     * \\brief Perform the WebSocket opening handshake.\n     *\n     * Sends an HTTP Upgrade request to the server and waits for\n     * a valid HTTP 101 Switching Protocols response. Upon success,\n     * the connection is upgraded to the WebSocket protocol and is\n     * ready for frame-level communication.\n     *\n     * \\throws websocket::exception if the server rejects the upgrade\n     *         or returns an invalid handshake response.\n     */\n    future<> handshake();\n};\n\n/*!\n * \\brief a WebSocket client\n *\n * A client capable of establishing a WebSocket connection to a server.\n * Manages the connection lifecycle.\n */\ntemplate<bool text_frame = false>\nclass client {\n    std::unique_ptr<client_connection<text_frame>> _conn;\n    gate _task_gate;\n\npublic:\n    /*!\n     * \\brief Connect to a WebSocket server over plain TCP.\n     * \\param addr server address\n     * \\param resource the URI path (e.g., \"/ws\")\n     * \\param host the Host header value\n     * \\param subprotocol optional subprotocol name (empty for none)\n     * \\param handler application handler\n     */\n    future<> connect(socket_address addr, sstring resource, sstring host,\n                     sstring subprotocol, handler_t handler);\n\n    /*!\n     * \\brief Connect to a WebSocket server over TLS.\n     * \\param addr server address\n     * \\param creds TLS credentials\n     * \\param resource the URI path (e.g., \"/ws\")\n     * \\param host the Host header value\n     * \\param subprotocol optional subprotocol name (empty for none)\n     * \\param handler application handler\n     */\n    future<> connect(socket_address addr,\n                     shared_ptr<tls::certificate_credentials> creds,\n                     sstring resource, sstring host,\n                     sstring subprotocol, handler_t handler);\n\n    /*!\n     * \\brief Close the client and the underlying connection.\n     */\n    future<> close();\n};\n\n/// @}\n\n}\n"
  },
  {
    "path": "include/seastar/websocket/common.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2024 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/net/api.hh>\n#include <seastar/util/log.hh>\n#include <seastar/websocket/parser.hh>\n\nnamespace seastar::experimental::websocket {\n\nusing handler_t = std::function<future<>(input_stream<char>&, output_stream<char>&)>;\n\nclass server;\n\n/// \\defgroup websocket WebSocket\n/// \\addtogroup websocket\n/// @{\n\n/*!\n * \\brief an error in handling a WebSocket connection\n */\nclass exception : public std::exception {\n    std::string _msg;\npublic:\n    exception(std::string_view msg) : _msg(msg) {}\n    virtual const char* what() const noexcept {\n        return _msg.c_str();\n    }\n};\n\n/*!\n * \\brief a server WebSocket connection\n */\ntemplate <bool is_client = false, bool text_frame = false>\nclass basic_connection : public boost::intrusive::list_base_hook<> {\nprotected:\n    using buff_t = temporary_buffer<char>;\n\n    /*!\n     * \\brief Implementation of connection's data source.\n     */\n    class connection_source_impl final : public data_source_impl {\n        queue<buff_t>* data;\n\n    public:\n        connection_source_impl(queue<buff_t>* data) : data(data) {}\n\n        virtual future<buff_t> get() override {\n            return data->pop_eventually().then_wrapped([](future<buff_t> f){\n                try {\n                    return make_ready_future<buff_t>(std::move(f.get()));\n                } catch(...) {\n                    return current_exception_as_future<buff_t>();\n                }\n            });\n        }\n\n        virtual future<> close() override {\n            data->push(buff_t(0));\n            return make_ready_future<>();\n        }\n    };\n\n    /*!\n     * \\brief Implementation of connection's data sink.\n     */\n    class connection_sink_impl final : public data_sink_impl {\n        queue<buff_t>* data;\n    public:\n        connection_sink_impl(queue<buff_t>* data) : data(data) {}\n\n#if SEASTAR_API_LEVEL >= 9\n        future<> put(std::span<temporary_buffer<char>> d) override {\n            return data_sink_impl::fallback_put(d, [this] (temporary_buffer<char>&& buf) {\n                return data->push_eventually(std::move(buf));\n            });\n        }\n#else\n        virtual future<> put(net::packet d) override {\n            net::fragment f = d.frag(0);\n            return data->push_eventually(temporary_buffer<char>{std::move(f.base), f.size});\n        }\n#endif\n\n        size_t buffer_size() const noexcept override {\n            return data->max_size();\n        }\n\n        virtual future<> close() override {\n            data->push(buff_t(0));\n            return make_ready_future<>();\n        }\n    };\n\n    /*!\n     * \\brief This function processess received PING frame.\n     * https://datatracker.ietf.org/doc/html/rfc6455#section-5.5.2\n     */\n    future<> handle_ping(temporary_buffer<char>);\n    /*!\n     * \\brief This function processess received PONG frame.\n     * https://datatracker.ietf.org/doc/html/rfc6455#section-5.5.3\n     */\n    future<> handle_pong();\n\n    static const size_t PIPE_SIZE = 512;\n    connected_socket _fd;\n    input_stream<char> _read_buf;\n    output_stream<char> _write_buf;\n    bool _done = false;\n    // TODO: implement https://datatracker.ietf.org/doc/html/rfc6455#section-7.1.2\n    bool _half_close = false;\n\n    websocket_parser _websocket_parser;\n    queue <temporary_buffer<char>> _input_buffer;\n    input_stream<char> _input;\n    queue <temporary_buffer<char>> _output_buffer;\n    output_stream<char> _output;\n\n    sstring _subprotocol;\n    handler_t _handler;\npublic:\n    /*!\n     * \\param fd established socket used for communication\n     * \\param is_client if true, this is a client-side connection (sends masked\n     *        frames and expects unmasked frames from server)\n     */\n    basic_connection(connected_socket&& fd)\n        : _fd(std::move(fd))\n        , _read_buf(_fd.input())\n        , _write_buf(_fd.output())\n        , _websocket_parser(!is_client)\n        , _input_buffer{PIPE_SIZE}\n        , _input(data_source{std::make_unique<connection_source_impl>(&_input_buffer)})\n        , _output_buffer{PIPE_SIZE}\n        , _output(data_sink{std::make_unique<connection_sink_impl>(&_output_buffer)})\n    {\n    }\n\n    /*!\n     * \\brief close the socket\n     */\n    void shutdown_input();\n    future<> close(bool send_close = true);\n\nprotected:\n    future<> read_one();\n    future<> response_loop();\n    /*!\n     * \\brief Packs buff in websocket frame and sends it to the client.\n     */\n    future<> send_data(opcodes opcode, temporary_buffer<char> buff);\n};\n\nusing connection = basic_connection<false, false>;\n\nstd::string sha1_base64(std::string_view source);\nstd::string encode_base64(std::string_view source);\n\nextern logger websocket_logger;\n\n/// @}\n}\n"
  },
  {
    "path": "include/seastar/websocket/parser.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#pragma once\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/iostream.hh>\n\nnamespace seastar::experimental::websocket {\n\n/// \\addtogroup websocket\n/// @{\n\n/*!\n * \\brief Possible type of a websocket frame.\n */\nenum opcodes {\n    CONTINUATION = 0x0,\n    TEXT = 0x1,\n    BINARY = 0x2,\n    CLOSE = 0x8,\n    PING = 0x9,\n    PONG = 0xA,\n    INVALID = 0xFF,\n};\n\nstruct frame_header {\n    static constexpr uint8_t FIN = 7;\n    static constexpr uint8_t RSV1 = 6;\n    static constexpr uint8_t RSV2 = 5;\n    static constexpr uint8_t RSV3 = 4;\n    static constexpr uint8_t MASKED = 7;\n\n    uint8_t fin : 1;\n    uint8_t rsv1 : 1;\n    uint8_t rsv2 : 1;\n    uint8_t rsv3 : 1;\n    uint8_t opcode : 4;\n    uint8_t masked : 1;\n    uint8_t length : 7;\n    frame_header(const char* input) {\n        this->fin = (input[0] >> FIN) & 1;\n        this->rsv1 = (input[0] >> RSV1) & 1;\n        this->rsv2 = (input[0] >> RSV2) & 1;\n        this->rsv3 = (input[0] >> RSV3) & 1;\n        this->opcode = input[0] & 0b1111;\n        this->masked = (input[1] >> MASKED) & 1;\n        this->length = (input[1] & 0b1111111);\n    }\n    // Returns length of the rest of the header.\n    uint64_t get_rest_of_header_length() {\n        size_t next_read_length = masked ? sizeof(uint32_t) : 0;\n        if (length == 126) {\n            next_read_length += sizeof(uint16_t);\n        } else if (length == 127) {\n            next_read_length += sizeof(uint64_t);\n        }\n        return next_read_length;\n    }\n    uint8_t get_fin() {return fin;}\n    uint8_t get_rsv1() {return rsv1;}\n    uint8_t get_rsv2() {return rsv2;}\n    uint8_t get_rsv3() {return rsv3;}\n    uint8_t get_opcode() {return opcode;}\n    uint8_t get_masked() {return masked;}\n    uint8_t get_length() {return length;}\n\n    bool is_opcode_known() {\n        //https://datatracker.ietf.org/doc/html/rfc6455#section-5.1\n        return opcode < 0xA && !(opcode < 0x8 && opcode > 0x2);\n    }\n};\n\nclass websocket_parser {\n    enum class parsing_state : uint8_t {\n        flags_and_payload_data,\n        payload_length_and_mask,\n        payload\n    };\n    enum class connection_state : uint8_t {\n        valid,\n        closed,\n        error\n    };\n    using consumption_result_t = consumption_result<char>;\n    using buff_t = temporary_buffer<char>;\n    // What parser is currently doing.\n    parsing_state _state;\n    // State of connection - can be valid, closed or should be closed\n    // due to error.\n    connection_state _cstate;\n    sstring _buffer;\n    std::unique_ptr<frame_header> _header;\n    uint64_t _payload_length = 0;\n    uint64_t _consumed_payload_length = 0;\n    bool _require_mask;\n    uint32_t _masking_key;\n    buff_t _result;\n\n    static future<consumption_result_t> dont_stop() {\n        return make_ready_future<consumption_result_t>(continue_consuming{});\n    }\n    static future<consumption_result_t> stop(buff_t data) {\n        return make_ready_future<consumption_result_t>(stop_consuming(std::move(data)));\n    }\n    uint64_t remaining_payload_length() const {\n        return _payload_length - _consumed_payload_length;\n    }\n\n    // Removes mask from payload given in p.\n    void remove_mask(buff_t& p, size_t n) {\n        char *payload = p.get_write();\n        for (uint64_t i = 0, j = 0; i < n; ++i, j = (j + 1) % 4) {\n            payload[i] ^= static_cast<char>(((_masking_key << (j * 8)) >> 24));\n        }\n    }\npublic:\n    /*!\n     * \\brief Construct a websocket frame parser.\n     * \\param require_mask if true (default), incoming frames must be masked\n     *        (server-side). If false, incoming frames must not be masked\n     *        (client-side).\n     */\n    explicit websocket_parser(bool require_mask = true)\n        : _state(parsing_state::flags_and_payload_data)\n        , _cstate(connection_state::valid)\n        , _require_mask(require_mask)\n        , _masking_key(0) {}\n    future<consumption_result_t> operator()(temporary_buffer<char> data);\n    bool is_valid() { return _cstate == connection_state::valid; }\n    bool eof() { return _cstate == connection_state::closed; }\n    opcodes opcode() const;\n    buff_t result();\n};\n\n/// @}\n}\n"
  },
  {
    "path": "include/seastar/websocket/server.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#pragma once\n\n#include <map>\n\n#include <seastar/http/request_parser.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/websocket/common.hh>\n\nnamespace seastar::experimental::websocket {\n\n/// \\addtogroup websocket\n/// @{\n\n/*!\n * \\brief a server WebSocket connection\n */\nclass server_connection : public connection {\n\n    server& _server;\n    http_request_parser _http_parser;\n\npublic:\n    /*!\n     * \\param server owning \\ref server\n     * \\param fd established socket used for communication\n     */\n    server_connection(server& server, connected_socket&& fd)\n        : connection(std::move(fd))\n        , _server(server) {\n        on_new_connection();\n    }\n    ~server_connection();\n\n    /*!\n     * \\brief serve WebSocket protocol on a server_connection\n     */\n    future<> process();\n\nprotected:\n    future<> read_loop();\n    future<> read_http_upgrade_request();\n    void on_new_connection();\n};\n\n/*!\n * \\brief a WebSocket server\n *\n * A server capable of establishing and serving connections\n * over WebSocket protocol.\n */\nclass server {\n    std::vector<server_socket> _listeners;\n    boost::intrusive::list<server_connection> _connections;\n    std::map<std::string, handler_t> _handlers;\n    gate _task_gate;\npublic:\n    /*!\n     * \\brief listen for a WebSocket connection on given address\n     * \\param addr address to listen on\n     */\n    void listen(socket_address addr);\n    /*!\n     * \\brief listen for a WebSocket connection on given address with custom options\n     * \\param addr address to listen on\n     * \\param lo custom listen options (\\ref listen_options)\n     */\n    void listen(socket_address addr, listen_options lo);\n\n    /*!\n     * Stops the server and shuts down all active connections\n     */\n    future<> stop();\n\n    bool is_handler_registered(std::string const& name);\n\n    /*!\n     * \\brief Register a handler for specific subprotocol\n     * \\param name The name of the subprotocol. If it is empty string, then the handler is used\n     * when the protocol is not specified\n     * \\param handler Handler for incoming WebSocket messages.\n     */\n    void register_handler(const std::string& name, handler_t handler);\n\n    friend class server_connection;\nprotected:\n    void accept(server_socket &listener);\n    future<stop_iteration> accept_one(server_socket &listener);\n};\n\n/// }@\n\n}\n"
  },
  {
    "path": "install-dependencies.sh",
    "content": "#!/bin/bash\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n# os-release may be missing in container environment by default.\nif [ -f \"/etc/os-release\" ]; then\n    . /etc/os-release\nelif [ -f \"/etc/arch-release\" ]; then\n    export ID=arch\nelse\n    echo \"/etc/os-release missing.\"\n    exit 1\nfi\n\ndebian_packages=(\n    cmake\n    diffutils\n    doxygen\n    g++\n    gcc\n    libboost-all-dev\n    libc-ares-dev\n    libcrypto++-dev\n    libfmt-dev\n    libgnutls28-dev\n    libhwloc-dev\n    liblz4-dev\n    libnuma-dev\n    libpciaccess-dev\n    libprotobuf-dev\n    libsctp-dev\n    libtool\n    liburing-dev\n    libxml2-dev\n    libyaml-cpp-dev\n    make\n    meson\n    ninja-build\n    openssl\n    pkg-config\n    protobuf-compiler\n    python3\n    python3-pyelftools\n    python3-yaml\n    ragel\n    stow\n    systemtap-sdt-dev\n    valgrind\n    xfslibs-dev\n)\n\n# seastar doesn't directly depend on these packages. They are\n# needed because we want to link seastar statically and pkg-config\n# has no way of saying \"static seastar, but dynamic transitive\n# dependencies\". They provide the various .so -> .so.ver symbolic\n# links.\ntransitive=(\n    libidn2-devel\n    libtool-ltdl-devel\n    libunistring-devel\n    trousers-devel\n)\n\nredhat_packages=(\n    boost-devel\n    c-ares-devel\n    cmake\n    diffutils\n    doxygen\n    fmt-devel\n    gcc\n    gnutls-devel\n    hwloc-devel\n    libpciaccess-devel\n    libtool\n    liburing-devel\n    libxml2-devel\n    lksctp-tools-devel\n    lz4-devel\n    make\n    meson\n    numactl-devel\n    openssl\n    protobuf-compiler\n    protobuf-devel\n    python3\n    python3-pyelftools\n    python3-pyyaml\n    stow\n    systemtap-sdt-devel\n    valgrind-devel\n    xfsprogs-devel\n    yaml-cpp-devel\n    \"${transitive[@]}\"\n)\n\nfedora_packages=(\n    \"${redhat_packages[@]}\"\n    boost-devel\n    fmt-devel\n    gcc-c++\n    libasan\n    libatomic\n    libubsan\n    ninja-build\n    ragel\n    valgrind-devel\n)\n\ncentos7_packages=(\n    \"${redhat_packages[@]}\"\n    cmake3\n    devtoolset-11-gcc-c++\n    devtoolset-11-libasan\n    devtoolset-11-libatomic\n    devtoolset-11-libubsan\n    ninja-build\n    ragel\n    rh-mongodb36-boost-devel\n)\n\ncentos8_packages=(\n    \"${redhat_packages[@]}\"\n    gcc-toolset-11-gcc\n    gcc-toolset-11-gcc-c++\n    gcc-toolset-11-libasan-devel\n    gcc-toolset-11-libatomic-devel\n    gcc-toolset-11-libubsan-devel\n    ninja-build\n    ragel\n)\n\ncentos9_packages=(\n    \"${redhat_packages[@]}\"\n    gcc-toolset-13-gcc\n    gcc-toolset-13-gcc-c++\n    gcc-toolset-13-libasan-devel\n    gcc-toolset-13-libatomic-devel\n    gcc-toolset-13-libubsan-devel\n    ninja-build\n    ragel\n)\n\n# 1) glibc 2.30-3 has sys/sdt.h (systemtap include)\n#    some old containers may contain glibc older,\n#    so enforce update on that one.\n# 2) if problems with signatures, ensure having fresh\n#    archlinux-keyring: pacman -Sy archlinux-keyring && pacman -Syyu\n# 3) aur installations require having sudo and being\n#    a sudoer. makepkg does not work otherwise.\narch_packages=(\n    boost\n    boost-libs\n    c-ares\n    cmake\n    crypto++\n    filesystem\n    fmt\n    gcc\n    glibc\n    gnutls\n    hwloc\n    libpciaccess\n    libtool\n    liburing\n    libxml2\n    lksctp-tools\n    lz4\n    make\n    meson\n    ninja\n    numactl\n    openssl\n    pkgconf\n    protobuf\n    python3\n    python-pyelftools\n    python-yaml\n    ragel\n    stow\n    valgrind\n    xfsprogs\n    yaml-cpp\n)\n\nopensuse_packages=(\n    c-ares-devel\n    cmake\n    hwloc-devel\n    libboost_atomic1_66_0\n    libboost_atomic1_66_0-devel\n    libboost_chrono1_66_0\n    libboost_chrono1_66_0-devel\n    libboost_date_time1_66_0\n    libboost_date_time1_66_0-devel\n    libboost_filesystem1_66_0\n    libboost_filesystem1_66_0-devel\n    libboost_program_options1_66_0\n    libboost_program_options1_66_0-devel\n    libboost_system1_66_0\n    libboost_system1_66_0-devel\n    libboost_test1_66_0\n    libboost_test1_66_0-devel\n    libboost_thread1_66_0\n    libboost_thread1_66_0-devel\n    libgnutls-devel\n    libgnutlsxx28\n    liblz4-devel\n    libnuma-devel\n    libtool\n    lksctp-tools-devel\n    meson\n    ninja\n    openssl\n    protobuf-devel\n    python3-PyYAML\n    ragel\n    stow\n    xfsprogs-devel\n    yaml-cpp-devel\n)\n\ncase \"$ID\" in\n    ubuntu|debian|pop)\n        apt-get install -y \"${debian_packages[@]}\"\n    ;;\n    fedora)\n        dnf install -y \"${fedora_packages[@]}\"\n    ;;\n    rhel|centos|rocky)\n        if [ \"$VERSION_ID\" = \"7\" ]; then\n            yum install -y epel-release centos-release-scl scl-utils\n            yum install -y \"${centos7_packages[@]}\"\n        elif [ \"${VERSION_ID%%.*}\" = \"8\" ]; then\n            dnf install -y epel-release\n            dnf install -y \"${centos8_packages[@]}\"\n        elif [ \"${VERSION_ID%%.*}\" = \"9\" ]; then\n            dnf install -y epel-release\n            dnf install -y \"${centos9_packages[@]}\"\n        fi\n    ;;\n    opensuse-leap)\n        zypper install -y \"${opensuse_packages[@]}\"\n    ;;\n    arch|manjaro)\n        if [ \"$EUID\" -eq \"0\" ]; then\n            pacman -Sy --needed --noconfirm \"${arch_packages[@]}\"\n        else\n            echo \"seastar: running without root. Skipping main dependencies (pacman).\" 1>&2\n        fi\n    ;;\n    *)\n        echo \"Your system ($ID) is not supported by this script. Please install dependencies manually.\"\n        exit 1\n    ;;\nesac\n"
  },
  {
    "path": "kvm/README.md",
    "content": "# seastar-in-kvm\nCreate a VM for Seastar development environment\n\n# Why we need this\nSeaStar scores muximum performance with DPDK, but it cannot live with existing NIC driver/Linux kernel network stack.\nAlso it directly accesses NIC device, it's bit hard to try it on remote node.\n\nseastar-in-kvm offers Fedora VM with SeaStar + DPDK without setup, it's easiest way to begin SeaStar application development.\n\n### Prerequire\n\nOn Fedora 21:\n```\nyum install @virtualization\nsystemctl enable libvirtd\nsystemctl start libvirtd\nyum install libguestfs-tools-c virt-install\n```\n\n### How to build & run\n```\n./build.sh\n./register.sh\nvirsh start seastar-dev && virsh console seastar-dev\n(Try login as 'seastar' after firstboot.sh finished, Fedora will ask new password for the user)\n```\n\n### Usage of the VM\n\nWait until finish running setup script on first startup.\nThen login as 'seastar', login prompt will ask for entering new password.\n\nAfter login to seastar, initialize DPDK module by following instruction:\n```\nsudo su - # entering root user\nresize    # extend console to actual terminal window size\nexport TERM=xterm-256color # set terminal type\ncd ~/dpdk\n./tools/setup.sh\n\n# input numbers by following order:\n(type 9 to re-compile DPDK)\n(type 12 to insert IGB UIO module)\n(type 15, then input \"64\" to setup hugepage mappings)\n(type 18, then input PCI device id something like \"0000:xx:yy.z\",\nwhich is shown at 'Network devices using DPDK-compatible driver')\n(type 30 to exit)\n\ncd ~/seastar\n# httpd example\nenv LD_LIBRARY_PATH=~/dpdk/x86_64-native-linuxapp-gcc/lib/ \\\n./build/release/apps/httpd/httpd --network-stack native --dpdk-pmd --csum-offload off\n```\n\n\n"
  },
  {
    "path": "kvm/register.sh",
    "content": "#!/bin/sh\n\nvirt-install --import --noreboot --name seastar-dev --vcpus 4 --ram 4096 --disk path=`realpath seastar-dev.qcow2`,format=qcow2,bus=virtio --accelerate --network=network:default,model=virtio --serial pty --cpu host --rng=/dev/random\n"
  },
  {
    "path": "kvm/scripts/bootstrap.sh",
    "content": "#!/bin/sh -e\nnmcli c modify eth0 ipv4.ignore-auto-dns \"yes\"\nsystemctl restart network\necho nameserver 8.8.8.8 > /etc/resolv.conf\nuseradd -m -p \"\" -g wheel seastar\nchage -d 0 seastar\nyum install -y gcc gcc-c++ libaio-devel ninja-build ragel hwloc-devel numactl-devel libpciaccess-devel boost-devel kernel-devel libxml2-devel zlib-devel libasan libubsan git wget python3 tar pciutils xterm\ncd /root\nwget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.0.0.tar.gz\ntar -xpf dpdk-2.0.0.tar.gz\nmv dpdk-2.0.0 dpdk\ncd dpdk\ncat config/common_linuxapp | sed -e \"s/CONFIG_RTE_MBUF_REFCNT_ATOMIC=y/CONFIG_RTE_MBUF_REFCNT_ATOMIC=n/g\" | sed -e \"s/CONFIG_RTE_BUILD_SHARED_LIB=n/CONFIG_RTE_BUILD_SHARED_LIB=y/g\" > /tmp/common_linuxapp\nmv /tmp/common_linuxapp config\nmake T=x86_64-native-linuxapp-gcc install\ncd -\ncd seastar\n./configure.py --dpdk-target ~/dpdk/x86_64-native-linuxapp-gcc\nninja-build -j2\n"
  },
  {
    "path": "licenses/cmake.txt",
    "content": "CMake - Cross Platform Makefile Generator\nCopyright 2000-2018 Kitware, Inc. and Contributors\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions\nare met:\n\n* Redistributions of source code must retain the above copyright\n  notice, this list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright\n  notice, this list of conditions and the following disclaimer in the\n  documentation and/or other materials provided with the distribution.\n\n* Neither the name of Kitware, Inc. nor the names of Contributors\n  may be used to endorse or promote products derived from this\n  software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\nLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\nA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\nHOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\nSPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\nLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\nDATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\nTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n"
  },
  {
    "path": "licenses/dpdk.txt",
    "content": "This project contains files from DPDK project that are distributed under BSD license.\n"
  },
  {
    "path": "licenses/freebsd.txt",
    "content": "Redistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions\nare met:\n1. Redistributions of source code must retain the above copyright\n   notice, this list of conditions and the following disclaimer.\n2. Redistributions in binary form must reproduce the above copyright\n   notice, this list of conditions and the following disclaimer in the\n   documentation and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\nOR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\nHOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\nLIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\nOUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGE.\n"
  },
  {
    "path": "pkgconfig/seastar-testing.pc.in",
    "content": "prefix=@Seastar_PKG_CONFIG_PREFIX@\nexec_prefix=${prefix}\nlibdir=@Seastar_PKG_CONFIG_LIBDIR@\n\nName: Seastar testing\nUrl: http://seastar-project.org\nDescription: Testing tools for libraries and applications built with Seastar\nVersion: @PROJECT_VERSION@\n\n# Dependencies.\nboost_cflags=-I$<JOIN:@Boost_INCLUDE_DIRS@, -I>\nboost_unit_test_libs=$<TARGET_PROPERTY:Boost::unit_test_framework,LOCATION>\n\n# Us.\nseastar_testing_cflags=\nseastar_testing_libs=${libdir}/$<TARGET_FILE_NAME:seastar_testing>\n\nRequires: seastar >= 1.0\nConflicts:\nCflags: ${boost_cflags} ${seastar_testing_cflags}\nLibs: ${boost_unit_test_libs} ${seastar_testing_libs}\n"
  },
  {
    "path": "pkgconfig/seastar.pc.in",
    "content": "prefix=@Seastar_PKG_CONFIG_PREFIX@\nexec_prefix=${prefix}\nseastar_include_flags=@Seastar_PKG_CONFIG_SEASTAR_INCLUDE_FLAGS@\nlibdir=@Seastar_PKG_CONFIG_LIBDIR@\n\nName: Seastar\nUrl: http://seastar-project.org\nDescription: Advanced C++ framework for high-performance server applications on modern hardware.\nVersion: @PROJECT_VERSION@\n\n# Platform dependencies.\nstdatomic_libs=$<JOIN:@StdAtomic_LIBRARIES@, >\ndl_libs=-l$<JOIN:@CMAKE_DL_LIBS@,-l>\nrt_libs=$<JOIN:@rt_LIBRARIES@, >\n\n# Dependencies of dependencies.\nboost_system_libs=@Boost_SYSTEM_LIBRARY@\n\n# Dependencies.\nboost_cflags=-I$<JOIN:@Boost_INCLUDE_DIRS@, -I>\nboost_program_options_libs=$<TARGET_PROPERTY:Boost::program_options,LOCATION>\nc_ares_cflags=-I$<JOIN:@c-ares_INCLUDE_DIRS@, -I>\nc_ares_libs=$<JOIN:@c-ares_LIBRARIES@, >\nfmt_cflags=-I$<JOIN:$<TARGET_PROPERTY:fmt::fmt,INTERFACE_INCLUDE_DIRECTORIES>, -I>\nfmt_libs=$<TARGET_LINKER_FILE:fmt::fmt>\nlksctp_tools_cflags=-I$<JOIN:@lksctp-tools_INCLUDE_DIRS@, -I>\nlksctp_tools_libs=$<JOIN:@lksctp-tools_LIBRARIES@, >\nliburing_cflags=$<$<BOOL:@Seastar_IO_URING@>:-I$<JOIN:$<TARGET_PROPERTY:URING::uring,INTERFACE_INCLUDE_DIRECTORIES>, -I>>\nliburing_libs=$<$<BOOL:@Seastar_IO_URING@>:$<JOIN:@URING_LIBRARIES@, >>\ndpdk_libs=$<JOIN:@dpdk_LIBRARIES@, >\n# Us.\nseastar_cflags=${seastar_include_flags} $<JOIN:$<TARGET_PROPERTY:seastar,INTERFACE_COMPILE_OPTIONS>, > -D$<JOIN:$<TARGET_PROPERTY:seastar,INTERFACE_COMPILE_DEFINITIONS>, -D>\nseastar_libs=${libdir}/$<TARGET_FILE_NAME:seastar> @Seastar_SPLIT_DWARF_FLAG@ $<JOIN:@Seastar_Sanitizers_OPTIONS@, >\n\nRequires: liblz4 >= 1.7.3\nRequires.private: gnutls >= 3.2.26, protobuf >= 2.5.0, hwloc >= 1.11.2, $<$<BOOL:@Seastar_IO_URING@>:liburing $<ANGLE-R>= 2.0, >yaml-cpp >= 0.5.1\nConflicts:\nCflags: @Seastar_CXX_COMPILE_OPTION@ ${boost_cflags} ${c_ares_cflags} ${fmt_cflags} ${liburing_cflags} ${lksctp_tools_cflags} ${seastar_cflags}\nLibs: ${seastar_libs} ${boost_program_options_libs} ${c_ares_libs} ${fmt_libs}\nLibs.private: ${dl_libs} ${rt_libs} ${lksctp_tools_libs} ${liburing_libs} ${stdatomic_libs} ${dpdk_libs}\n"
  },
  {
    "path": "scripts/addr2line.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n# Copyright (C) 2017 ScyllaDB\n\nimport bisect\nimport collections\nimport re\nimport sys\nimport subprocess\nfrom enum import Enum\nfrom functools import cache\nimport time\nfrom typing import Any, Optional, TypeVar, Union, cast\n\n# special binary path/module indicating that the address is from the kernel\nKERNEL_MODULE = '<kernel>'\n\n\nT = TypeVar('T')\n\n\ndef notNone(o: Optional[T]) -> T:\n    \"\"\"Asserts the argument is not None then returns it.\"\"\"\n    assert o is not None\n    return o\n\n\nclass Addr2Line:\n\n    # Matcher for a line that appears at the end a single decoded\n    # address, which we force by adding a dummy 0x0 address. The\n    # pattern varies between binutils addr2line and llvm-addr2line\n    # so we match both.\n    # The LLVM output is usually literally:\n    #  0x0: ?? at ??\n    # but not always, e.g., when ASAN is linked it may be (for example):\n    #  0x0: ?? at /v/llvm/llvm/src/compiler-rt/lib/asan/asan_fake_stack.h:133\n    # so that's why we liberally accept .* as the part after \"at\" below\n    dummy_pattern = re.compile(\n        r\"(.*0x0000000000000000: \\?\\? \\?\\?:0\\n)\"  # addr2line pattern\n        r\"|\"\n        r\"(\\?\\? at \\?\\?:0\\n)\"  # llvm-addr2line pattern for LLVM 18 and newer\n        r\"|\"\n        r\"(,\\n)\"  # llvm-addr2line pattern for LLVM 17 and older\n    )\n\n    def __init__(\n        self,\n        parent: 'BacktraceResolver',\n        binary: str,\n        concise: bool = False,\n        cmd_path: str = \"llvm-addr2line\",\n    ):\n        self._parent = parent\n        self._binary = binary\n\n        # Print warning if binary has no debug info according to `file`.\n        # Note: no message is printed for system errors as they will be\n        # printed also by addr2line later on.\n        output = subprocess.check_output([\"file\", self._binary])\n        s = output.decode(\"utf-8\")\n        if s.find('ELF') >= 0 and s.find('debug_info', len(self._binary)) < 0:\n            print('{}'.format(s))\n\n        args = [cmd_path, f\"-{'C' if not concise else ''}fpia\", \"-e\", self._binary]\n        self._parent.debug(f\"Addr2line invoking: {' '.join(args)}\")\n        self._input_proc = subprocess.Popen(\n            args,\n            stdin=subprocess.PIPE,\n            stdout=subprocess.PIPE,\n            universal_newlines=True,\n        )\n        if concise:\n            self._output_proc = subprocess.Popen(\n                [\"c++filt\", \"-p\"],\n                stdin=self._input_proc.stdout,\n                stdout=subprocess.PIPE,\n                universal_newlines=True,\n            )\n        else:\n            self._output_proc = self._input_proc\n\n        # If a library doesn't exist in a particular path, addr2line\n        # will just exit.  We need to be robust against that.  We\n        # can't just wait on self._addr2line since there is no\n        # guarantee on what timeout is sufficient.\n        self._input.write(',\\n')\n        self._input.flush()\n        res = self._output.readline()\n        self._missing = res == ''\n\n    @property\n    def _input(self):\n        \"\"\"Returns the input stream for the process/pipe.\"\"\"\n        return notNone(self._input_proc.stdin)\n\n    @property\n    def _output(self):\n        \"\"\"Returns the output stream for the process/pipe.\"\"\"\n        return notNone(self._output_proc.stdout)\n\n    def _read_resolved_address(self):\n        first = self._output.readline()\n        self._parent.debug('Addr2Line read output (first): ', first)\n        # remove the address\n        res = first.split(': ', 1)[1]\n        while True:\n            line = self._output.readline()\n            if Addr2Line.dummy_pattern.fullmatch(line):\n                self._parent.debug('Addr2Line read output (    dummy): ', line)\n                break\n            self._parent.debug('Addr2Line read output (non-dummy): ', line)\n            res += line\n        return res\n\n    def __call__(self, address: str):\n        if self._missing:\n            return \" \".join([self._binary, address, '\\n'])\n        # We trigger a dummy \"invalid\" address printout after the address we are interested in\n        # which we can look for in _read_address\n        inputline = address + '\\n,\\n'\n        self._parent.debug('Add2Line sending input to stdin:', inputline)\n        self._input.write(inputline)\n        self._input.flush()\n        return self._read_resolved_address()\n\n\nclass KernelResolver:\n    \"\"\"A resolver for kernel addresses which tries to read from /proc/kallsyms.\"\"\"\n\n    LAST_SYMBOL_MAX_SIZE = 1024\n\n    def __init__(self, parent: 'BacktraceResolver', kallsyms: str = '/proc/kallsyms'):\n        syms: list[tuple[int, str]] = []\n        ksym_re = re.compile(r'(?P<addr>[0-9a-f]+) (?P<type>.+) (?P<name>\\S+)')\n        warnings_left = 10\n\n        self.error = None\n\n        try:\n            f = open(kallsyms, 'r')\n        except OSError as e:\n            self.error = f'Cannot open {kallsyms}: {e}'\n            print(self.error)\n            return\n\n        try:\n            for line in f:\n                m = ksym_re.match(line)\n                if not m:\n                    if warnings_left > 0:  # don't spam too much\n                        print(\n                            f'WARNING: {kallsyms} regex match failure: {line.strip()}',\n                            file=sys.stdout,\n                        )\n                        warnings_left -= 1\n                else:\n                    syms.append((int(m.group('addr'), 16), m.group('name')))\n        finally:\n            f.close()\n\n        if not syms:\n            # make empty kallsyms (?) an error so we can assum len >= 1 below\n            self.error = 'kallsyms was empty'\n            print(self.error)\n            return\n\n        syms.sort()\n\n        if syms[-1][0] == 0:\n            # zero values for all symbols means that kptr_restrict blocked you\n            # from seeing the kernel symbol addresses\n            print('kallsyms is restricted, set /proc/sys/kernel/kptr_restrict to 0 to decode')\n            self.error = 'kallsyms is restricted'\n            return\n\n        # split because bisect can't take a key func before 3.10\n        self.sym_addrs: tuple[int]\n        self.sym_names: tuple[str]\n        self.sym_addrs, self.sym_names = zip(*syms)  # type: ignore\n\n    def __call__(self, addrstr: str):\n        if self.error:\n            return addrstr + '\\n'\n\n        sa = self.sym_addrs\n        sn = self.sym_names\n        slen = len(sa)\n        address = int(addrstr, 16)\n        idx = bisect.bisect_right(sa, address) - 1\n        assert -1 <= idx < slen\n        if idx == -1:\n            return f'{addrstr} ({sa[0] - address} bytes before first symbol)\\n'\n        if idx == slen - 1:\n            # We can easily detect symbol addresses which are too small: they fall before\n            # the first symbol in kallsyms, but for too large it is harder: we can't really\n            # distinguish between an address that is in the *very last* function in the symbol map\n            # and one which is beyond that, since kallsyms doesn't include symbol size. Instead\n            # we use a bit of a quick and dirty heuristic: if the symbol is *far enough* beyond\n            # the last symbol we assume it is not valid. Most likely, the overwhelming majority\n            # of cases are invalid (e.g., due to KASLR) as the final symbol in the map is usually\n            # something obscure.\n            lastsym = sa[-1]\n            if address - lastsym > self.LAST_SYMBOL_MAX_SIZE:\n                return f'{addrstr} ({address - lastsym} bytes after last symbol)\\n'\n        saddr = sa[idx]\n        assert saddr <= address\n        return f'{sn[idx]}+0x{address - saddr:x}\\n'\n\n\nLineResult = dict[\n    str, Union[None, 'BacktraceResolver.BacktraceParser.Type', str, list[dict[str, Any]]]\n]\n\n\nclass BacktraceResolver:\n\n    class BacktraceParser:\n        class Type(Enum):\n            ADDRESS = 1\n            SEPARATOR = 2\n\n        def __init__(self):\n            addr = \"0x[0-9a-f]+\"\n            path = r\"\\S+\"\n            token = fr\"(?:{path}\\+)?{addr}\"\n            full_addr_match = fr\"(?:(?P<path>{path})\\s*\\+\\s*)?(?P<addr>{addr})\"\n            ignore_addr_match = fr\"(?:(?P<path>{path})\\s*\\+\\s*)?(?:{addr})\"\n\n            self.asan_ignore_re = re.compile(f\"^=.*$\", flags=re.IGNORECASE)\n            self.separator_re = re.compile(r'^\\W*-+\\W*$')\n\n            # All of the below except for separator_re must only match lines containing 0x in them\n            # as we use that as a quick first filter\n            self.oneline_re = re.compile(\n                fr\"^((?:.*(?:(?:at|backtrace):?|:))?(?:\\s+))?({token}(?:\\s+{token})*)(?:\\).*|\\s*)$\",\n                flags=re.IGNORECASE,\n            )\n            self.address_re = re.compile(full_addr_match, flags=re.IGNORECASE)\n            self.syslog_re = re.compile(\n                fr\"^(?:#\\d+\\s+)(?P<addr>{addr})(?:.*\\s+)\\({ignore_addr_match}\\)\\s*$\",\n                flags=re.IGNORECASE,\n            )\n            self.kernel_re = re.compile(fr'^.*kernel callstack: (?P<addrs>(?:{addr}\\s*)+)$')\n            self.asan_re = re.compile(\n                fr\"^(?:.*\\s+)\\({full_addr_match}\\)(\\s+\\(BuildId: [0-9a-fA-F]+\\))?$\",\n                flags=re.IGNORECASE,\n            )\n            self.generic_re = re.compile(fr\"^(?:.*\\s+){full_addr_match}\\s*$\", flags=re.IGNORECASE)\n\n        def split_addresses(self, addrstring: str, default_path: Optional[str] = None):\n            addresses: list[dict[str, Any]] = []\n            for obj in addrstring.split():\n                m = re.match(self.address_re, obj)\n                assert m, f'addr did not match address regex: {obj}'\n                # print(f\"  >>> '{obj}': address {m.groups()}\")\n                addresses.append({'path': m.group(1) or default_path, 'addr': m.group(2)})\n            return addresses\n\n        def __call__(self, line: str) -> dict[str, Any] | None:\n\n            # quick up front check to eliminate a line from contention, which is at least\n            # 30x faster than just diving into the regex matching (for one test file)\n            if not (\"0x\" in line or \"0X\" in line or self.separator_re.match(line)):\n                # no addresses in this line, so it is not a backtrace line\n                return None\n\n            def get_prefix(s: Optional[str]):\n                if s is not None:\n                    s = s.strip()\n                return s or None\n\n            ret: LineResult\n\n            # order here is important: the kernel callstack regex\n            # needs to come first since it is more specific and would\n            # otherwise be matched by the online regex which comes next\n            m = self.kernel_re.match(line)\n            if m:\n                return {\n                    'type': self.Type.ADDRESS,\n                    'prefix': 'kernel callstack: ',\n                    'addresses': self.split_addresses(m.group('addrs'), KERNEL_MODULE),\n                }\n\n            m = re.match(self.oneline_re, line)\n            if m:\n                # print(f\">>> '{line}': oneline {m.groups()}\")\n                return {\n                    'type': self.Type.ADDRESS,\n                    'prefix': get_prefix(m.group(1)),\n                    'addresses': self.split_addresses(m.group(2)),\n                }\n\n            m = re.match(self.syslog_re, line)\n            if m:\n                # print(f\">>> '{line}': syslog {m.groups()}\")\n                ret = {'type': self.Type.ADDRESS}\n                ret['prefix'] = None\n                ret['addresses'] = [{'path': m.group('path'), 'addr': m.group('addr')}]\n                return ret\n\n            m = re.match(self.asan_ignore_re, line)\n            if m:\n                # print(f\">>> '{line}': asan ignore\")\n                return None\n\n            m = re.match(self.asan_re, line)\n            if m:\n                # print(f\">>> '{line}': asan {m.groups()}\")\n                ret = {'type': self.Type.ADDRESS}\n                ret['prefix'] = None\n                ret['addresses'] = [{'path': m.group('path'), 'addr': m.group('addr')}]\n                return ret\n\n            m = re.match(self.generic_re, line)\n            if m:\n                # print(f\">>> '{line}': generic {m.groups()}\")\n                ret = {'type': self.Type.ADDRESS}\n                ret['prefix'] = None\n                ret['addresses'] = [{'path': m.group('path'), 'addr': m.group('addr')}]\n                return ret\n\n            match = re.match(self.separator_re, line)\n            if match:\n                return {'type': self.Type.SEPARATOR}\n\n            # print(f\">>> '{line}': None\")\n            return None\n\n    def __init__(\n        self,\n        executable: str,\n        kallsyms: str = '/proc/kallsyms',\n        before_lines: int = 1,\n        context_re: Optional[str] = '',\n        verbose: bool = False,\n        concise: bool = False,\n        cmd_path: str = 'addr2line',\n        debug: bool = False,\n        timing: bool = False,\n    ):\n        self._debug = debug\n        self._timing = timing\n        self._total_resolve_time = 0.0\n        self._executable = executable\n        self._kallsyms = kallsyms\n        self._current_backtrace: list[tuple[str, str]] = []\n        self._prefix = None\n        self._before_lines = before_lines\n        self._before_lines_queue: collections.deque[str] = collections.deque(maxlen=before_lines)\n        self._i = 0\n        self._known_backtraces: dict[str, int] = {}\n        if context_re is not None:\n            self._context_re = re.compile(context_re)\n        else:\n            self._context_re = None\n        self._verbose = verbose\n        self._concise = concise\n        self._cmd_path = cmd_path\n        self._known_modules: dict[str, Union[Addr2Line, KernelResolver]] = {}\n        self._get_resolver_for_module(\n            self._executable\n        )  # fail fast if there is something wrong with the exe resolver\n        self.parser = self.BacktraceParser()\n\n    def debug(self, *args: Any):\n        if self._debug:\n            print('DEBUG >>', *args, file=sys.stderr)\n\n    def timing_now(self):\n        return time.perf_counter() if self._timing else 0.0\n\n    def timing_print_from_start(self, start: float, *args: Any):\n        self.timing_print(self.timing_now() - start, *args)\n\n    def timing_print(self, duration: float, *args: Any):\n        if self._timing:\n            print(f\"TIMING >> {duration:.4f} seconds : \", *args, file=sys.stderr)\n\n    def print_resolve_time(self):\n        self.timing_print(self._total_resolve_time, 'resolve time (addr2line subprocess time)')\n\n    def _get_resolver_for_module(self, module: str):\n        if not module in self._known_modules:\n            if module == KERNEL_MODULE:\n                resolver = KernelResolver(self, kallsyms=self._kallsyms)\n            else:\n                resolver = Addr2Line(self, module, self._concise, self._cmd_path)\n            self.debug(f'Adding resolver {resolver} for module: {module}')\n            self._known_modules[module] = resolver\n        return self._known_modules[module]\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, *_):\n        self._print_current_backtrace()\n\n    @cache\n    def resolve_address(\n        self, address: str, module: Optional[str] = None, verbose: Optional[bool] = None\n    ):\n        if module is None:\n            module = self._executable\n        if verbose is None:\n            verbose = self._verbose\n        resolver = self._get_resolver_for_module(module)\n        resolve_start = self.timing_now()\n        resolved_address = resolver(address)\n        self._total_resolve_time += self.timing_now() - resolve_start\n        if verbose:\n            resolved_address = '{{{}}} {}: {}'.format(module, address, resolved_address)\n        return resolved_address\n\n    def _print_resolved_address(self, module: Optional[str], address: str):\n        sys.stdout.write(self.resolve_address(address, module))\n\n    def _backtrace_context_matches(self):\n        if self._context_re is None:\n            return True\n\n        if any(self._context_re.search(x) for x in self._before_lines_queue):\n            return True\n\n        if (not self._prefix is None) and self._context_re.search(self._prefix):\n            return True\n\n        return False\n\n    def _print_current_backtrace(self):\n        if len(self._current_backtrace) == 0:\n            return\n\n        if not self._backtrace_context_matches():\n            self._current_backtrace = []\n            return\n\n        for line in self._before_lines_queue:\n            sys.stdout.write(line)\n\n        if not self._prefix is None:\n            print(self._prefix)\n            self._prefix = None\n\n        backtrace = \"\".join(map(str, self._current_backtrace))\n        if backtrace in self._known_backtraces:\n            print(\n                \"[Backtrace #{}] Already seen, not resolving again.\".format(\n                    self._known_backtraces[backtrace]\n                )\n            )\n            print(\"\")  # To separate traces with an empty line\n            self._current_backtrace = []\n            return\n\n        self._known_backtraces[backtrace] = self._i\n\n        self.debug(\n            f'Resolving and printing parsed backtrace with {len(self._current_backtrace)} frames'\n        )\n\n        print(\"[Backtrace #{}]\".format(self._i))\n\n        for module, addr in self._current_backtrace:\n            self._print_resolved_address(module, addr)\n\n        print(\"\")  # To separate traces with an empty line\n\n        self._current_backtrace = []\n        self._i += 1\n\n    def __call__(self, line: str):\n        res = self.parser(line)\n\n        if not res:\n            self.debug('INPUT LINE [NO MATCH]:', line)\n            self._print_current_backtrace()\n            if self._before_lines > 0:\n                self._before_lines_queue.append(line)\n            elif self._before_lines < 0:\n                sys.stdout.write(line)  # line already has a trailing newline\n            else:\n                pass  # when == 0 no non-backtrace lines are printed\n        elif res['type'] == self.BacktraceParser.Type.SEPARATOR:\n            self.debug('INPUT LINE [SEPARATOR]:', line)\n            pass\n        elif res['type'] == self.BacktraceParser.Type.ADDRESS:\n            self.debug('INPUT LINE [ADDRESS]:', line)\n            addresses = cast(list[dict[str, Any]], res['addresses'])\n            if len(addresses) > 1:\n                self._print_current_backtrace()\n            if len(self._current_backtrace) == 0:\n                self._prefix = cast(Union[str, None], res['prefix'])\n            for r in addresses:\n                if r['path']:\n                    self._current_backtrace.append((r['path'], r['addr']))\n                else:\n                    self._current_backtrace.append((self._executable, r['addr']))\n            if len(addresses) > 1:\n                self._print_current_backtrace()\n        else:\n            self.debug('INPUT LINE [UNKNOWN]:', line)\n            print(f\"Unknown '{line}': {res}\")\n            raise RuntimeError(\"Unknown result type {res}\")\n"
  },
  {
    "path": "scripts/dpdk_nic_bind.py",
    "content": "#!/usr/bin/env python2\n#\n#   BSD LICENSE\n#\n#   Copyright(c) 2010-2014 Intel Corporation. All rights reserved.\n#   All rights reserved.\n#\n#   Redistribution and use in source and binary forms, with or without\n#   modification, are permitted provided that the following conditions\n#   are met:\n#\n#     * Redistributions of source code must retain the above copyright\n#       notice, this list of conditions and the following disclaimer.\n#     * Redistributions in binary form must reproduce the above copyright\n#       notice, this list of conditions and the following disclaimer in\n#       the documentation and/or other materials provided with the\n#       distribution.\n#     * Neither the name of Intel Corporation nor the names of its\n#       contributors may be used to endorse or promote products derived\n#       from this software without specific prior written permission.\n#\n#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n#   \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n\nimport sys, os, getopt, subprocess\nfrom os.path import exists, abspath, dirname, basename\n\n\n# The PCI device class for ETHERNET devices\nETHERNET_CLASS = \"0200\"\n\n# global dict ethernet devices present. Dictionary indexed by PCI address.\n# Each device within this is itself a dictionary of device properties\ndevices = {}\n# list of supported DPDK drivers\ndpdk_drivers = [ \"igb_uio\", \"vfio-pci\", \"uio_pci_generic\" ]\n\n# command-line arg flags\nb_flag = None\nstatus_flag = False\nforce_flag = False\nargs = []\n\ndef usage():\n    '''Print usage information for the program'''\n    argv0 = basename(sys.argv[0])\n    print \"\"\"\nUsage:\n------\n\n     %(argv0)s [options] DEVICE1 DEVICE2 ....\n\nwhere DEVICE1, DEVICE2 etc, are specified via PCI \"domain:bus:slot.func\" syntax\nor \"bus:slot.func\" syntax. For devices bound to Linux kernel drivers, they may\nalso be referred to by Linux interface name e.g. eth0, eth1, em0, em1, etc.\n\nOptions:\n    --help, --usage:\n        Display usage information and quit\n\n    --status:\n        Print the current status of all known network interfaces.\n        For each device, it displays the PCI domain, bus, slot and function,\n        along with a text description of the device. Depending upon whether the\n        device is being used by a kernel driver, the igb_uio driver, or no\n        driver, other relevant information will be displayed:\n        * the Linux interface name e.g. if=eth0\n        * the driver being used e.g. drv=igb_uio\n        * any suitable drivers not currently using that device\n            e.g. unused=igb_uio\n        NOTE: if this flag is passed along with a bind/unbind option, the status\n        display will always occur after the other operations have taken place.\n\n    -b driver, --bind=driver:\n        Select the driver to use or \\\"none\\\" to unbind the device\n\n    -u, --unbind:\n        Unbind a device (Equivalent to \\\"-b none\\\")\n\n    --force:\n        By default, devices which are used by Linux - as indicated by having\n        routes in the routing table - cannot be modified. Using the --force\n        flag overrides this behavior, allowing active links to be forcibly\n        unbound.\n        WARNING: This can lead to loss of network connection and should be used\n        with caution.\n\nExamples:\n---------\n\nTo display current device status:\n        %(argv0)s --status\n\nTo bind eth1 from the current driver and move to use igb_uio\n        %(argv0)s --bind=igb_uio eth1\n\nTo unbind 0000:01:00.0 from using any driver\n        %(argv0)s -u 0000:01:00.0\n\nTo bind 0000:02:00.0 and 0000:02:00.1 to the ixgbe kernel driver\n        %(argv0)s -b ixgbe 02:00.0 02:00.1\n\n    \"\"\" % locals() # replace items from local variables\n\n# This is roughly compatible with check_output function in subprocess module\n# which is only available in python 2.7.\ndef check_output(args, stderr=None):\n    '''Run a command and capture its output'''\n    return subprocess.Popen(args, stdout=subprocess.PIPE,\n                            stderr=stderr).communicate()[0]\n\ndef find_module(mod):\n    '''find the .ko file for kernel module named mod.\n    Searches the $RTE_SDK/$RTE_TARGET directory, the kernel\n    modules directory and finally under the parent directory of\n    the script '''\n    # check $RTE_SDK/$RTE_TARGET directory\n    if 'RTE_SDK' in os.environ and 'RTE_TARGET' in os.environ:\n        path = \"%s/%s/kmod/%s.ko\" % (os.environ['RTE_SDK'],\\\n                                     os.environ['RTE_TARGET'], mod)\n        if exists(path):\n            return path\n\n    # check using depmod\n    try:\n        depmod_out = check_output([\"modinfo\", \"-n\", mod], \\\n                                  stderr=subprocess.STDOUT).lower()\n        if \"error\" not in depmod_out:\n            path = depmod_out.strip()\n            if exists(path):\n                return path\n    except: # if modinfo can't find module, it fails, so continue\n        pass\n\n    # check for a copy based off current path\n    tools_dir = dirname(abspath(sys.argv[0]))\n    if (tools_dir.endswith(\"tools\")):\n        base_dir = dirname(tools_dir)\n        find_out = check_output([\"find\", base_dir, \"-name\", mod + \".ko\"])\n        if len(find_out) > 0: #something matched\n            path = find_out.splitlines()[0]\n            if exists(path):\n                return path\n\ndef check_modules():\n    '''Checks that igb_uio is loaded'''\n    global dpdk_drivers\n\n    fd = file(\"/proc/modules\")\n    loaded_mods = fd.readlines()\n    fd.close()\n\n    # list of supported modules\n    mods =  [{\"Name\" : driver, \"Found\" : False} for driver in dpdk_drivers]\n\n    # first check if module is loaded\n    for line in loaded_mods:\n        for mod in mods:\n            if line.startswith(mod[\"Name\"]):\n                mod[\"Found\"] = True\n            # special case for vfio_pci (module is named vfio-pci,\n            # but its .ko is named vfio_pci)\n            elif line.replace(\"_\", \"-\").startswith(mod[\"Name\"]):\n                mod[\"Found\"] = True\n\n    # check if we have at least one loaded module\n    if True not in [mod[\"Found\"] for mod in mods] and b_flag is not None:\n        if b_flag in dpdk_drivers:\n            print \"Error - no supported modules(DPDK driver) are loaded\"\n            sys.exit(1)\n        else:\n            print \"Warning - no supported modules(DPDK driver) are loaded\"\n\n    # change DPDK driver list to only contain drivers that are loaded\n    dpdk_drivers = [mod[\"Name\"] for mod in mods if mod[\"Found\"]]\n\ndef has_driver(dev_id):\n    '''return true if a device is assigned to a driver. False otherwise'''\n    return \"Driver_str\" in devices[dev_id]\n\ndef get_pci_device_details(dev_id):\n    '''This function gets additional details for a PCI device'''\n    device = {}\n\n    extra_info = check_output([\"lspci\", \"-vmmks\", dev_id]).splitlines()\n\n    # parse lspci details\n    for line in extra_info:\n        if len(line) == 0:\n            continue\n        name, value = line.split(\"\\t\", 1)\n        name = name.strip(\":\") + \"_str\"\n        device[name] = value\n    # check for a unix interface name\n    sys_path = \"/sys/bus/pci/devices/%s/net/\" % dev_id\n    if exists(sys_path):\n        device[\"Interface\"] = \",\".join(os.listdir(sys_path))\n    else:\n        device[\"Interface\"] = \"\"\n    # check if a port is used for ssh connection\n    device[\"Ssh_if\"] = False\n    device[\"Active\"] = \"\"\n\n    return device\n\ndef get_nic_details():\n    '''This function populates the \"devices\" dictionary. The keys used are\n    the pci addresses (domain:bus:slot.func). The values are themselves\n    dictionaries - one for each NIC.'''\n    global devices\n    global dpdk_drivers\n\n    # clear any old data\n    devices = {}\n    # first loop through and read details for all devices\n    # request machine readable format, with numeric IDs\n    dev = {};\n    dev_lines = check_output([\"lspci\", \"-Dvmmn\"]).splitlines()\n    for dev_line in dev_lines:\n        if (len(dev_line) == 0):\n            if dev[\"Class\"] == ETHERNET_CLASS:\n                #convert device and vendor ids to numbers, then add to global\n                dev[\"Vendor\"] = int(dev[\"Vendor\"],16)\n                dev[\"Device\"] = int(dev[\"Device\"],16)\n                devices[dev[\"Slot\"]] = dict(dev) # use dict to make copy of dev\n        else:\n            name, value = dev_line.split(\"\\t\", 1)\n            dev[name.rstrip(\":\")] = value\n\n    # check what is the interface if any for an ssh connection if\n    # any to this host, so we can mark it later.\n    ssh_if = []\n    route = check_output([\"ip\", \"-o\", \"route\"])\n    # filter out all lines for 169.254 routes\n    route = \"\\n\".join(filter(lambda ln: not ln.startswith(\"169.254\"),\n                             route.splitlines()))\n    rt_info = route.split()\n    for i in xrange(len(rt_info) - 1):\n        if rt_info[i] == \"dev\":\n            ssh_if.append(rt_info[i+1])\n\n    # based on the basic info, get extended text details\n    for d in devices.keys():\n        # get additional info and add it to existing data\n        devices[d] = dict(devices[d].items() +\n                          get_pci_device_details(d).items())\n\n        for _if in ssh_if:\n            if _if in devices[d][\"Interface\"].split(\",\"):\n                devices[d][\"Ssh_if\"] = True\n                devices[d][\"Active\"] = \"*Active*\"\n                break;\n\n        # add igb_uio to list of supporting modules if needed\n        if \"Module_str\" in devices[d]:\n            for driver in dpdk_drivers:\n                if driver not in devices[d][\"Module_str\"]:\n                    devices[d][\"Module_str\"] = devices[d][\"Module_str\"] + \",%s\" % driver\n        else:\n            devices[d][\"Module_str\"] = \",\".join(dpdk_drivers)\n\n        # make sure the driver and module strings do not have any duplicates\n        if has_driver(d):\n            modules = devices[d][\"Module_str\"].split(\",\")\n            if devices[d][\"Driver_str\"] in modules:\n                modules.remove(devices[d][\"Driver_str\"])\n                devices[d][\"Module_str\"] = \",\".join(modules)\n\ndef dev_id_from_dev_name(dev_name):\n    '''Take a device \"name\" - a string passed in by user to identify a NIC\n    device, and determine the device id - i.e. the domain:bus:slot.func - for\n    it, which can then be used to index into the devices array'''\n    dev = None\n    # check if it's already a suitable index\n    if dev_name in devices:\n        return dev_name\n    # check if it's an index just missing the domain part\n    elif \"0000:\" + dev_name in devices:\n        return \"0000:\" + dev_name\n    else:\n        # check if it's an interface name, e.g. eth1\n        for d in devices.keys():\n            if dev_name in devices[d][\"Interface\"].split(\",\"):\n                return devices[d][\"Slot\"]\n    # if nothing else matches - error\n    print \"Unknown device: %s. \" \\\n        \"Please specify device in \\\"bus:slot.func\\\" format\" % dev_name\n    sys.exit(1)\n\ndef unbind_one(dev_id, force):\n    '''Unbind the device identified by \"dev_id\" from its current driver'''\n    dev = devices[dev_id]\n    if not has_driver(dev_id):\n        print \"%s %s %s is not currently managed by any driver\\n\" % \\\n            (dev[\"Slot\"], dev[\"Device_str\"], dev[\"Interface\"])\n        return\n\n    # prevent us disconnecting ourselves\n    if dev[\"Ssh_if\"] and not force:\n        print \"Routing table indicates that interface %s is active\" \\\n            \". Skipping unbind\" % (dev_id)\n        return\n\n    # write to /sys to unbind\n    filename = \"/sys/bus/pci/drivers/%s/unbind\" % dev[\"Driver_str\"]\n    try:\n        f = open(filename, \"a\")\n    except:\n        print \"Error: unbind failed for %s - Cannot open %s\" % (dev_id, filename)\n        sys/exit(1)\n    f.write(dev_id)\n    f.close()\n\ndef bind_one(dev_id, driver, force):\n    '''Bind the device given by \"dev_id\" to the driver \"driver\". If the device\n    is already bound to a different driver, it will be unbound first'''\n    dev = devices[dev_id]\n    saved_driver = None # used to rollback any unbind in case of failure\n\n    # prevent disconnection of our ssh session\n    if dev[\"Ssh_if\"] and not force:\n        print \"Routing table indicates that interface %s is active\" \\\n            \". Not modifying\" % (dev_id)\n        return\n\n    # unbind any existing drivers we don't want\n    if has_driver(dev_id):\n        if dev[\"Driver_str\"] == driver:\n            print \"%s already bound to driver %s, skipping\\n\" % (dev_id, driver)\n            return\n        else:\n            saved_driver = dev[\"Driver_str\"]\n            unbind_one(dev_id, force)\n            dev[\"Driver_str\"] = \"\" # clear driver string\n\n    # if we are binding to one of DPDK drivers, add PCI id's to that driver\n    if driver in dpdk_drivers:\n        filename = \"/sys/bus/pci/drivers/%s/new_id\" % driver\n        try:\n            f = open(filename, \"w\")\n        except:\n            print \"Error: bind failed for %s - Cannot open %s\" % (dev_id, filename)\n            return\n        try:\n            f.write(\"%04x %04x\" % (dev[\"Vendor\"], dev[\"Device\"]))\n            f.close()\n        except:\n            print \"Error: bind failed for %s - Cannot write new PCI ID to \" \\\n                \"driver %s\" % (dev_id, driver)\n            return\n\n    # do the bind by writing to /sys\n    filename = \"/sys/bus/pci/drivers/%s/bind\" % driver\n    try:\n        f = open(filename, \"a\")\n    except:\n        print \"Error: bind failed for %s - Cannot open %s\" % (dev_id, filename)\n        if saved_driver is not None: # restore any previous driver\n            bind_one(dev_id, saved_driver, force)\n        return\n    try:\n        f.write(dev_id)\n        f.close()\n    except:\n        # for some reason, closing dev_id after adding a new PCI ID to new_id\n        # results in IOError. however, if the device was successfully bound,\n        # we don't care for any errors and can safely ignore IOError\n        tmp = get_pci_device_details(dev_id)\n        if \"Driver_str\" in tmp and tmp[\"Driver_str\"] == driver:\n            return\n        print \"Error: bind failed for %s - Cannot bind to driver %s\" % (dev_id, driver)\n        if saved_driver is not None: # restore any previous driver\n            bind_one(dev_id, saved_driver, force)\n        return\n\n\ndef unbind_all(dev_list, force=False):\n    \"\"\"Unbind method, takes a list of device locations\"\"\"\n    dev_list = map(dev_id_from_dev_name, dev_list)\n    for d in dev_list:\n        unbind_one(d, force)\n\ndef bind_all(dev_list, driver, force=False):\n    \"\"\"Bind method, takes a list of device locations\"\"\"\n    global devices\n\n    dev_list = map(dev_id_from_dev_name, dev_list)\n\n    for d in dev_list:\n        bind_one(d, driver, force)\n\n    # when binding devices to a generic driver (i.e. one that doesn't have a\n    # PCI ID table), some devices that are not bound to any other driver could\n    # be bound even if no one has asked them to. hence, we check the list of\n    # drivers again, and see if some of the previously-unbound devices were\n    # erroneously bound.\n    for d in devices.keys():\n        # skip devices that were already bound or that we know should be bound\n        if \"Driver_str\" in devices[d] or d in dev_list:\n            continue\n\n        # update information about this device\n        devices[d] = dict(devices[d].items() +\n                          get_pci_device_details(d).items())\n\n        # check if updated information indicates that the device was bound\n        if \"Driver_str\" in devices[d]:\n            unbind_one(d, force)\n\ndef display_devices(title, dev_list, extra_params = None):\n    '''Displays to the user the details of a list of devices given in \"dev_list\"\n    The \"extra_params\" parameter, if given, should contain a string with\n    %()s fields in it for replacement by the named fields in each device's\n    dictionary.'''\n    strings = [] # this holds the strings to print. We sort before printing\n    print \"\\n%s\" % title\n    print   \"=\"*len(title)\n    if len(dev_list) == 0:\n        strings.append(\"<none>\")\n    else:\n        for dev in dev_list:\n            if extra_params is not None:\n                strings.append(\"%s '%s' %s\" % (dev[\"Slot\"], \\\n                                dev[\"Device_str\"], extra_params % dev))\n            else:\n                strings.append(\"%s '%s'\" % (dev[\"Slot\"], dev[\"Device_str\"]))\n    # sort before printing, so that the entries appear in PCI order\n    strings.sort()\n    print \"\\n\".join(strings) # print one per line\n\ndef show_status():\n    '''Function called when the script is passed the \"--status\" option. Displays\n    to the user what devices are bound to the igb_uio driver, the kernel driver\n    or to no driver'''\n    global dpdk_drivers\n    kernel_drv = []\n    dpdk_drv = []\n    no_drv = []\n\n    # split our list of devices into the three categories above\n    for d in devices.keys():\n        if not has_driver(d):\n            no_drv.append(devices[d])\n            continue\n        if devices[d][\"Driver_str\"] in dpdk_drivers:\n            dpdk_drv.append(devices[d])\n        else:\n            kernel_drv.append(devices[d])\n\n    # print each category separately, so we can clearly see what's used by DPDK\n    display_devices(\"Network devices using DPDK-compatible driver\", dpdk_drv, \\\n                    \"drv=%(Driver_str)s unused=%(Module_str)s\")\n    display_devices(\"Network devices using kernel driver\", kernel_drv,\n                    \"if=%(Interface)s drv=%(Driver_str)s unused=%(Module_str)s %(Active)s\")\n    display_devices(\"Other network devices\", no_drv,\\\n                    \"unused=%(Module_str)s\")\n\ndef parse_args():\n    '''Parses the command-line arguments given by the user and takes the\n    appropriate action for each'''\n    global b_flag\n    global status_flag\n    global force_flag\n    global args\n    if len(sys.argv) <= 1:\n        usage()\n        sys.exit(0)\n\n    try:\n        opts, args = getopt.getopt(sys.argv[1:], \"b:u\",\n                               [\"help\", \"usage\", \"status\", \"force\",\n                                \"bind=\", \"unbind\"])\n    except getopt.GetoptError, error:\n        print str(error)\n        print \"Run '%s --usage' for further information\" % sys.argv[0]\n        sys.exit(1)\n\n    for opt, arg in opts:\n        if opt == \"--help\" or opt == \"--usage\":\n            usage()\n            sys.exit(0)\n        if opt == \"--status\":\n            status_flag = True\n        if opt == \"--force\":\n            force_flag = True\n        if opt == \"-b\" or opt == \"-u\" or opt == \"--bind\" or opt == \"--unbind\":\n            if b_flag is not None:\n                print \"Error - Only one bind or unbind may be specified\\n\"\n                sys.exit(1)\n            if opt == \"-u\" or opt == \"--unbind\":\n                b_flag = \"none\"\n            else:\n                b_flag = arg\n\ndef do_arg_actions():\n    '''do the actual action requested by the user'''\n    global b_flag\n    global status_flag\n    global force_flag\n    global args\n\n    if b_flag is None and not status_flag:\n        print \"Error: No action specified for devices. Please give a -b or -u option\"\n        print \"Run '%s --usage' for further information\" % sys.argv[0]\n        sys.exit(1)\n\n    if b_flag is not None and len(args) == 0:\n        print \"Error: No devices specified.\"\n        print \"Run '%s --usage' for further information\" % sys.argv[0]\n        sys.exit(1)\n\n    if b_flag == \"none\" or b_flag == \"None\":\n        unbind_all(args, force_flag)\n    elif b_flag is not None:\n        bind_all(args, b_flag, force_flag)\n    if status_flag:\n        if b_flag is not None:\n            get_nic_details() # refresh if we have changed anything\n        show_status()\n\ndef main():\n    '''program main function'''\n    parse_args()\n    check_modules()\n    get_nic_details()\n    do_arg_actions()\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "scripts/io-trace-parse.py",
    "content": "#!/bin/env python3\n#\n# Script to parse IO trace logs and show some stats\n#\n\nimport sys\nimport statistics\n\n\n# prints average, .99 quantile and maximum value for an array\ndef print_stat_line(what, st):\n    def q99(arr):\n        return statistics.quantiles(arr, n=100)[-1]\n\n    print(\"\\t{:18}: avg:{:12.6f}  .99:{:12.6f}  max:{:12.6f}\".format(what,\n          statistics.fmean(st), q99(st), max(st)))\n\n\n# Inc/Dec counter that also collects its value history\nclass counter:\n    def __init__(self):\n        self._v = 0\n        self._stat = []\n\n    def inc(self):\n        self._v += 1\n        self._stat.append(self._v)\n\n    def dec(self):\n        self._v -= 1\n        self._stat.append(self._v)\n\n    def stat(self):\n        return self._stat\n\n\nclass req:\n    def __init__(self, rqlen):\n        self.len = rqlen\n        self.queue = None\n        self.submit = None\n        self.complete = None\n\n\n# Timings for requests\nclass req_stat:\n    def __init__(self):\n        self.qtimes = []        # time in queue\n        self.xtimes = []        # time in disk\n        self.latencies = []     # sum of the above\n        self.delays = []        # time between submits\n        self.prev = None        # helper for the above\n        self.in_queue = counter()\n        self.in_disk = counter()\n\n    def queue(self, rq):\n        self.in_queue.inc()\n\n    def submit(self, rq):\n        if self.prev:\n            self.delays.append(rq.submit - self.prev)\n        self.prev = rq.submit\n        self.qtimes.append(rq.submit - rq.queue)\n        self.in_queue.dec()\n        self.in_disk.inc()\n\n    def complete(self, rq):\n        self.xtimes.append(rq.complete - rq.submit)\n        self.latencies.append(rq.complete - rq.queue)\n        self.in_disk.dec()\n\n    def show(self, rqlen):\n        print(\"{}k requests\".format(int(rqlen/1024)))\n        print(\"\\ttotal: {}\".format(len(self.latencies)))\n        print_stat_line('in queue usec', self.qtimes)\n        print_stat_line('      `- num ', self.in_queue.stat())\n        print_stat_line('in disk usec', self.xtimes)\n        print_stat_line('     `- num ', self.in_disk.stat())\n        print_stat_line('latency', self.latencies)\n        print_stat_line('period', self.delays)\n\n\n# Stats for a device. Umbrella-object for the above stats\nclass device_stat:\n    def __init__(self):\n        self.reqs = {}          # collection of req's\n        self.req_stats = {}     # statistics by request size\n        self.in_queue = counter()\n        self.in_disk = counter()\n\n    def queue(self, rqid, ts, rqlen):\n        rq = req(rqlen)\n        self.reqs[rqid] = rq\n        rq.queue = ts\n        if rq.len not in self.req_stats:\n            self.req_stats[rq.len] = req_stat()\n        st = self.req_stats[rq.len]\n        st.queue(rq)\n        self.in_queue.inc()\n\n    def submit(self, rqid, ts):\n        rq = self.reqs[rqid]\n        rq.submit = ts\n        st = self.req_stats[rq.len]\n        st.submit(rq)\n        self.in_queue.dec()\n        self.in_disk.inc()\n\n    def complete(self, rqid, ts):\n        rq = self.reqs[rqid]\n        rq.complete = ts\n        st = self.req_stats[rq.len]\n        st.complete(rq)\n        del self.reqs[rqid]\n        self.in_disk.dec()\n\n    def _show_req_stats(self):\n        for rlen in self.req_stats:\n            st = self.req_stats[rlen]\n            st.show(rlen)\n\n    def _show_queue_stats(self):\n        print(\"queue\")\n        print_stat_line('in queue num:', self.in_queue.stat())\n        print_stat_line('in disk num:', self.in_disk.stat())\n\n    def show(self, devid):\n        print(\"{}\".format(devid).center(80, \"-\"))\n        self._show_req_stats()\n        self._show_queue_stats()\n\n\nclass parser:\n    def __init__(self, f):\n        self._file = f\n        self._dev_stats = {}\n\n    def _get_dev_stats(self, devid):\n        if devid not in self._dev_stats:\n            self._dev_stats[devid] = device_stat()\n\n        return self._dev_stats[devid]\n\n    def _parse_req_event(self, ln):\n        req_id = ln[10]\n        ts = float(ln[1])\n        st = self._get_dev_stats(int(ln[7]))\n\n        if ln[11] == 'queue':\n            st.queue(req_id, ts, int(ln[13]))\n        elif ln[11] == 'submit':\n            st.submit(req_id, ts)\n        elif ln[11] == 'complete':\n            st.complete(req_id, ts)\n\n    def _parse_line(self, ln):\n        if ln[4] == 'io':\n            if ln[9] == 'req':\n                self._parse_req_event(ln)\n\n    def parse(self):\n        for ln in self._file:\n            if ln.startswith('TRACE'):\n                self._parse_line(ln.strip().split())\n\n        return self._dev_stats\n\n\nif __name__ == \"__main__\":\n    p = parser(sys.stdin)\n    stats = p.parse()\n    for devid in stats:\n        stats[devid].show(devid)\n"
  },
  {
    "path": "scripts/perftune.py",
    "content": "#!/usr/bin/env python3\n\nimport abc\nimport argparse\nimport distutils.util\nimport enum\nimport functools\nimport glob\nimport itertools\nimport logging\nimport math\nimport multiprocessing\nimport os\nimport pathlib\nimport pyudev\nimport re\nimport shutil\nimport subprocess\nimport sys\nimport urllib.request\nimport yaml\nimport platform\nimport shlex\nimport psutil\nimport mmap\nimport datetime\n\ndry_run_mode = False\ndef perftune_print(log_msg, *args, **kwargs):\n    if dry_run_mode:\n        log_msg = \"# \" + log_msg\n    print(log_msg, *args, **kwargs)\n\ndef __run_one_command(prog_args, stderr=None, check=True):\n    proc = subprocess.Popen(prog_args, stdout = subprocess.PIPE, stderr = stderr)\n    outs, errs = proc.communicate()\n    outs = str(outs, 'utf-8')\n\n    if check and proc.returncode != 0:\n        raise subprocess.CalledProcessError(returncode=proc.returncode, cmd=\" \".join(prog_args), output=outs, stderr=errs)\n\n    return outs\n\ndef run_one_command(prog_args, stderr=None, check=True):\n    if dry_run_mode:\n        print(\" \".join([shlex.quote(x) for x in prog_args]))\n    else:\n        __run_one_command(prog_args, stderr=stderr, check=check)\n\ndef run_read_only_command(prog_args, stderr=None, check=True):\n    return __run_one_command(prog_args, stderr=stderr, check=check)\n\ndef run_hwloc_distrib(prog_args):\n    \"\"\"\n    Returns a list of strings - each representing a single line of hwloc-distrib output.\n    \"\"\"\n    return run_read_only_command(['hwloc-distrib'] + prog_args).splitlines()\n\ndef run_hwloc_calc(prog_args):\n    \"\"\"\n    Returns a single string with the result of the execution.\n    \"\"\"\n    return run_read_only_command(['hwloc-calc'] + prog_args).rstrip()\n\ndef run_ethtool(prog_args):\n    \"\"\"\n    Returns a list of strings - each representing a single line of ethtool output.\n    \"\"\"\n    return run_read_only_command(['ethtool'] + prog_args).splitlines()\n\ndef fwriteln(fname, line, log_message, log_errors=True):\n    try:\n        if dry_run_mode:\n            print(\"echo {} > {}\".format(line, fname))\n            return\n        else:\n            with open(fname, 'w') as f:\n                f.write(line)\n            print(log_message)\n    except:\n        if log_errors:\n            print(\"{}: failed to write into {}: {}\".format(log_message, fname, sys.exc_info()))\n\ndef readlines(fname):\n    try:\n        with open(fname, 'r') as f:\n            return f.readlines()\n    except:\n        print(\"Failed to read {}: {}\".format(fname, sys.exc_info()))\n        return []\n\ndef fwriteln_and_log(fname, line, log_errors=True):\n    msg = \"Writing '{}' to {}\".format(line, fname)\n    fwriteln(fname, line, log_message=msg, log_errors=log_errors)\n\ndouble_commas_pattern = re.compile(',,')\n\ndef set_one_mask(conf_file, mask, log_errors=True):\n    if not os.path.exists(conf_file):\n        raise Exception(\"Configure file to set mask doesn't exist: {}\".format(conf_file))\n    mask = re.sub('0x', '', mask)\n\n    while double_commas_pattern.search(mask):\n        mask = double_commas_pattern.sub(',0,', mask)\n\n    msg = \"Setting mask {} in {}\".format(mask, conf_file)\n    fwriteln(conf_file, mask, log_message=msg, log_errors=log_errors)\n\ndef distribute_irqs(irqs, cpu_mask, log_errors=True):\n    # If IRQs' list is empty - do nothing\n    if not irqs:\n        return\n\n    for i, mask in enumerate(run_hwloc_distrib([\"{}\".format(len(irqs)), '--single', '--restrict', cpu_mask])):\n        set_one_mask(\"/proc/irq/{}/smp_affinity\".format(irqs[i]), mask, log_errors=log_errors)\n\ndef is_process_running(name):\n    return len(list(filter(lambda ps_line : not re.search('<defunct>', ps_line), run_read_only_command(['ps', '--no-headers', '-C', name], check=False).splitlines()))) > 0\n\ndef restart_irqbalance(banned_irqs):\n    \"\"\"\n    Restart irqbalance if it's running and ban it from moving the IRQs from the\n    given list.\n    \"\"\"\n    config_file = '/etc/default/irqbalance'\n    options_key = 'OPTIONS'\n    systemd = False\n    banned_irqs_list = list(banned_irqs)\n\n    # If there is nothing to ban - quit\n    if not banned_irqs_list:\n        return\n\n    # return early if irqbalance is not running\n    if not is_process_running('irqbalance'):\n        perftune_print(\"irqbalance is not running\")\n        return\n\n    # If this file exists - this a \"new (systemd) style\" irqbalance packaging.\n    # This type of packaging uses IRQBALANCE_ARGS as an option key name, \"old (init.d) style\"\n    # packaging uses an OPTION key.\n    if os.path.exists('/lib/systemd/system/irqbalance.service') or \\\n        os.path.exists('/usr/lib/systemd/system/irqbalance.service'):\n        options_key = 'IRQBALANCE_ARGS'\n        systemd = True\n\n    if not os.path.exists(config_file):\n        if os.path.exists('/etc/sysconfig/irqbalance'):\n            config_file = '/etc/sysconfig/irqbalance'\n        elif os.path.exists('/etc/conf.d/irqbalance'):\n            config_file = '/etc/conf.d/irqbalance'\n            options_key = 'IRQBALANCE_OPTS'\n            with open('/proc/1/comm', 'r') as comm:\n                systemd = 'systemd' in comm.read()\n        else:\n            perftune_print(\"Unknown system configuration - not restarting irqbalance!\")\n            perftune_print(\"You have to prevent it from moving IRQs {} manually!\".format(banned_irqs_list))\n            return\n\n    orig_file = \"{}.scylla.orig\".format(config_file)\n\n    # Save the original file\n    if not dry_run_mode:\n        if not os.path.exists(orig_file):\n            print(\"Saving the original irqbalance configuration is in {}\".format(orig_file))\n            shutil.copyfile(config_file, orig_file)\n        else:\n            print(\"File {} already exists - not overwriting.\".format(orig_file))\n\n    # Read the config file lines\n    cfile_lines = open(config_file, 'r').readlines()\n\n    # Build the new config_file contents with the new options configuration\n    perftune_print(\"Restarting irqbalance: going to ban the following IRQ numbers: {} ...\".format(\", \".join(banned_irqs_list)))\n\n    # Search for the original options line\n    opt_lines = list(filter(lambda line : re.search(r\"^\\s*{}\".format(options_key), line), cfile_lines))\n    if not opt_lines:\n        new_options = \"{}=\\\"\".format(options_key)\n    elif len(opt_lines) == 1:\n        # cut the last \"\n        new_options = re.sub(r'\"\\s*$', \"\", opt_lines[0].rstrip())\n        opt_lines = opt_lines[0].strip()\n    else:\n        raise Exception(\"Invalid format in {}: more than one lines with {} key\".format(config_file, options_key))\n\n    for irq in banned_irqs_list:\n        # prevent duplicate \"ban\" entries for the same IRQ\n        opt = f\"--banirq={irq}\"\n        patt_str = rf\"{opt}\\Z|{opt}\\s\"\n        if not re.search(patt_str, new_options):\n            new_options += f\" {opt}\"\n\n    new_options += \"\\\"\"\n\n    if dry_run_mode:\n        if opt_lines:\n            print(\"sed -i 's/^{}/#{}/g' {}\".format(options_key, options_key, config_file))\n        print(\"echo {} | tee -a {}\".format(new_options, config_file))\n    else:\n        with open(config_file, 'w') as cfile:\n            for line in cfile_lines:\n                if not re.search(r\"^\\s*{}\".format(options_key), line):\n                    cfile.write(line)\n\n            cfile.write(new_options + \"\\n\")\n\n    if systemd:\n        perftune_print(\"Restarting irqbalance via systemctl...\")\n        run_one_command(['systemctl', 'try-restart', 'irqbalance'])\n    else:\n        perftune_print(\"Restarting irqbalance directly (init.d)...\")\n        run_one_command(['/etc/init.d/irqbalance', 'restart'])\n\ndef learn_irqs_from_proc_interrupts(pattern, irq2procline):\n    return [ irq for irq, proc_line in filter(lambda irq_proc_line_pair : re.search(pattern, irq_proc_line_pair[1]), irq2procline.items()) ]\n\ndef learn_all_irqs_one(irq_conf_dir, irq2procline, xen_dev_name):\n    \"\"\"\n    Returns a list of IRQs of a single device.\n\n    irq_conf_dir: a /sys/... directory with the IRQ information for the given device\n    irq2procline: a map of IRQs to the corresponding lines in the /proc/interrupts\n    xen_dev_name: a device name pattern as it appears in the /proc/interrupts on Xen systems\n    \"\"\"\n    msi_irqs_dir_name = os.path.join(irq_conf_dir, 'msi_irqs')\n    # Device uses MSI IRQs\n    if os.path.exists(msi_irqs_dir_name):\n        return os.listdir(msi_irqs_dir_name)\n\n    irq_file_name = os.path.join(irq_conf_dir, 'irq')\n    # Device uses INT#x\n    if os.path.exists(irq_file_name):\n        return [ line.lstrip().rstrip() for line in open(irq_file_name, 'r').readlines() ]\n\n    # No irq file detected\n    modalias = open(os.path.join(irq_conf_dir, 'modalias'), 'r').readline()\n\n    # virtio case\n    if re.search(\"^virtio\", modalias):\n        return list(itertools.chain.from_iterable(\n            map(lambda dirname : learn_irqs_from_proc_interrupts(dirname, irq2procline),\n                filter(lambda dirname : re.search('virtio', dirname),\n                       itertools.chain.from_iterable([ dirnames for dirpath, dirnames, filenames in os.walk(os.path.join(irq_conf_dir, 'driver')) ])))))\n\n    # xen case\n    if re.search(\"^xen:\", modalias):\n        return learn_irqs_from_proc_interrupts(xen_dev_name, irq2procline)\n\n    return []\n\ndef get_irqs2procline_map():\n    return { line.split(':')[0].lstrip().rstrip() : line for line in open('/proc/interrupts', 'r').readlines() }\n\n\nclass AutodetectError(Exception):\n    pass\n\n\ndef auto_detect_irq_mask(cpu_mask, cores_per_irq_core):\n    \"\"\"\n    The logic of auto-detection of what was once a 'mode' is generic and is all about the amount of CPUs and NUMA\n    nodes that are present and a restricting 'cpu_mask'.\n    This function implements this logic:\n\n    * up to 4 CPU threads: use 'cpu_mask'\n    * up to 4 CPU cores (on x86 this would translate to 8 CPU threads): use a single CPU thread out of allowed\n    * up to 16 CPU cores: use a single CPU core out of allowed\n    * more than 16 CPU cores: use a single CPU core for each 16 CPU cores and distribute them evenly among all\n      present NUMA nodes. We will also make sure all NUMA nodes have exactly the same amount of IRQ cores.\n\n    An AutodetectError exception is raised if 'cpu_mask' is defined in a way that there is a different number of threads\n    and/or cores among different NUMA nodes. In such a case a user needs to provide\n    an IRQ CPUs definition explicitly using 'irq_cpu_mask' parameter.\n\n    :param cpu_mask: CPU mask that defines which out of present CPUs can be considered for tuning\n    :param cores_per_irq_core number of cores to allocate a single IRQ core out of, e.g. 6 means allocate a single IRQ\n                              core out of every 6 CPU cores.\n    :return: CPU mask to bind IRQs to, a.k.a. irq_cpu_mask\n    \"\"\"\n    cores_key = 'cores'\n    PUs_key = 'PUs'\n\n    # List of NUMA IDs that own CPUs from the given CPU mask\n    numa_ids_list = run_hwloc_calc(['-I', 'numa', cpu_mask]).split(\",\")\n\n    # Let's calculate number of HTs and cores on each NUMA node belonging to the given CPU set\n    cores_PUs_per_numa = {} # { <numa_id> : {'cores':  <number of cores>, 'PUs': <number of PUs>}}\n    for n in numa_ids_list:\n        num_cores = int(run_hwloc_calc(['--restrict', cpu_mask, '--number-of', 'core', f'numa:{n}']))\n        num_PUs = int(run_hwloc_calc(['--restrict', cpu_mask, '--number-of', 'PU', f'numa:{n}']))\n        cores_PUs_per_numa[n] = {cores_key: num_cores, PUs_key: num_PUs}\n\n    # Let's check if configuration on each NUMA is the same. If it's not then we can't auto-detect the IRQs CPU set\n    # and a user needs to provide it explicitly\n    num_cores0 = cores_PUs_per_numa[numa_ids_list[0]][cores_key]\n    num_PUs0 = cores_PUs_per_numa[numa_ids_list[0]][PUs_key]\n    for n in numa_ids_list:\n        if cores_PUs_per_numa[n][cores_key] != num_cores0 or cores_PUs_per_numa[n][PUs_key] != num_PUs0:\n            raise AutodetectError(f\"NUMA{n} has a different configuration from NUMA0 for a given CPU mask {cpu_mask}: \"\n                                  f\"{cores_PUs_per_numa[n][cores_key]}:{cores_PUs_per_numa[n][PUs_key]} vs \"\n                                  f\"{num_cores0}:{num_PUs0}. Auto-detection of IRQ CPUs in not possible. \"\n                                  f\"Please, provide irq_cpu_mask explicitly.\")\n\n    # Auto-detection of IRQ CPU set is possible - let's get to it!\n    #\n    # Total counts for the whole machine\n    num_cores = int(run_hwloc_calc(['--restrict', cpu_mask, '--number-of', 'core', 'machine:0']))\n    num_PUs = int(run_hwloc_calc(['--restrict', cpu_mask, '--number-of', 'PU', 'machine:0']))\n\n    if num_PUs <= 4:\n        return cpu_mask\n    elif num_cores <= 4:\n        return run_hwloc_calc(['--restrict', cpu_mask, 'PU:0'])\n    elif num_cores <= cores_per_irq_core:\n        return run_hwloc_calc(['--restrict', cpu_mask, 'core:0'])\n    else:\n        # Big machine.\n        # Let's allocate a full core out of every cores_per_irq_core cores.\n        # Let's distribute IRQ cores among present NUMA nodes\n        num_irq_cores = len(numa_ids_list) * math.ceil(num_cores0 / cores_per_irq_core)\n        hwloc_args = []\n        numa_cores_count = {n: 0 for n in numa_ids_list}\n        added_cores = 0\n        while added_cores < num_irq_cores:\n            for numa in numa_ids_list:\n                hwloc_args.append(f\"node:{numa}.core:{numa_cores_count[numa]}\")\n                added_cores += 1\n                numa_cores_count[numa] += 1\n\n                if added_cores >= num_irq_cores:\n                    break\n\n        return run_hwloc_calc(['--restrict', cpu_mask] + hwloc_args)\n\n\ndef check_sysfs_numa_topology_is_valid():\n    # Verify that the sysfs entry exists correctly, same as the checks\n    # performed by hwloc code (check_sysfs_cpu_path() on topology-linux.c)\n    if os.path.isdir(\"/sys/devices/system/cpu\"):\n        if os.path.exists(\"/sys/devices/system/cpu/cpu0/topology/package_cpus\") or os.path.exists(\"/sys/devices/system/cpu/cpu0/topology/core_cpus\"):\n            return True\n        if os.path.exists(\"/sys/devices/system/cpu/cpu0/topology/core_siblings\") or os.path.exists(\"/sys/devices/system/cpu/cpu0/topology/thread_siblings\"):\n            return True\n    return False\n\n################################################################################\nclass PerfTunerBase(metaclass=abc.ABCMeta):\n    def __init__(self, args):\n        self.__args = args\n        self.__args.cpu_mask = run_hwloc_calc(['--restrict', self.__args.cpu_mask, 'all'])\n        self.__mode = None\n        self.__compute_cpu_mask = None\n\n        if self.args.mode:\n            self.mode = PerfTunerBase.SupportedModes[self.args.mode]\n        elif args.irq_cpu_mask:\n            self.irqs_cpu_mask = args.irq_cpu_mask\n        else:\n            if not check_sysfs_numa_topology_is_valid():\n                raise PerfTunerBase.InvalidNUMATopologyException(\"NUMA topology information is corrupted\")\n            self.irqs_cpu_mask = auto_detect_irq_mask(self.cpu_mask, self.cores_per_irq_core)\n\n        self.__is_aws_i3_nonmetal_instance = None\n        self.__metadata_token_value = None\n        self.__metadata_token_time = None\n\n#### Public methods ##########################\n    class CPUMaskIsZeroException(Exception):\n        \"\"\"Thrown if CPU mask turns out to be zero\"\"\"\n        pass\n\n    class InvalidNUMATopologyException(Exception):\n        \"\"\"Thrown if NUMA Topology is invalid\"\"\"\n        pass\n\n    class SupportedModes(enum.IntEnum):\n        \"\"\"\n        Modes are ordered from the one that cuts the biggest number of CPUs\n        from the compute CPUs' set to the one that takes the smallest ('mq' doesn't\n        cut any CPU from the compute set).\n\n        This fact is used when we calculate the 'common quotient' mode out of a\n        given set of modes (e.g. default modes of different Tuners) - this would\n        be the smallest among the given modes.\n        \"\"\"\n        sq_split = 0\n        sq = 1\n        mq = 2\n\n        # Note: no_irq_restrictions should always have the greatest value in the enum since it's the least restricting mode.\n        no_irq_restrictions = 9999\n\n        @staticmethod\n        def names():\n            return PerfTunerBase.SupportedModes.__members__.keys()\n\n        @staticmethod\n        def combine(modes):\n            \"\"\"\n            :param modes: a set of modes of the PerfTunerBase.SupportedModes type\n            :return: the mode that is the \"common ground\" for a given set of modes.\n            \"\"\"\n\n            # Perform an explicit cast in order to verify that the values in the 'modes' are compatible with the\n            # expected PerfTunerBase.SupportedModes type.\n            return min([PerfTunerBase.SupportedModes(m) for m in modes])\n\n    @staticmethod\n    def cpu_mask_is_zero(cpu_mask):\n        \"\"\"\n        The cpu_mask is a comma-separated list of 32-bit hex values with possibly omitted zero components,\n        e.g. 0xffff,,0xffff\n        We want to estimate if the whole mask is all-zeros.\n        :param cpu_mask: hwloc-calc generated CPU mask\n        :return: True if mask is zero, False otherwise\n        \"\"\"\n        for cur_cpu_mask in cpu_mask.split(','):\n            if cur_cpu_mask and int(cur_cpu_mask, 16) != 0:\n                return False\n\n        return True\n\n    @staticmethod\n    def compute_cpu_mask_for_mode(mq_mode, cpu_mask):\n        mq_mode = PerfTunerBase.SupportedModes(mq_mode)\n\n        if mq_mode == PerfTunerBase.SupportedModes.sq:\n            # all but CPU0\n            compute_cpu_mask = run_hwloc_calc([cpu_mask, '~PU:0'])\n        elif mq_mode == PerfTunerBase.SupportedModes.sq_split:\n            # all but CPU0 and its HT siblings\n            compute_cpu_mask = run_hwloc_calc([cpu_mask, '~core:0'])\n        elif mq_mode == PerfTunerBase.SupportedModes.mq:\n            # all available cores\n            compute_cpu_mask = cpu_mask\n        elif mq_mode == PerfTunerBase.SupportedModes.no_irq_restrictions:\n            # all available cores\n            compute_cpu_mask = cpu_mask\n        else:\n            raise Exception(\"Unsupported mode: {}\".format(mq_mode))\n\n        if PerfTunerBase.cpu_mask_is_zero(compute_cpu_mask):\n            raise PerfTunerBase.CPUMaskIsZeroException(\"Bad configuration mode ({}) and cpu-mask value ({}): this results in a zero-mask for compute\".format(mq_mode.name, cpu_mask))\n\n        return compute_cpu_mask\n\n    @staticmethod\n    def irqs_cpu_mask_for_mode(mq_mode, cpu_mask):\n        mq_mode = PerfTunerBase.SupportedModes(mq_mode)\n        irqs_cpu_mask = 0\n\n        if mq_mode != PerfTunerBase.SupportedModes.mq and mq_mode != PerfTunerBase.SupportedModes.no_irq_restrictions:\n            irqs_cpu_mask = run_hwloc_calc([cpu_mask, \"~{}\".format(PerfTunerBase.compute_cpu_mask_for_mode(mq_mode, cpu_mask))])\n        else: # mq_mode == PerfTunerBase.SupportedModes.mq or mq_mode == PerfTunerBase.SupportedModes.no_irq_restrictions\n            # distribute equally between all available cores\n            irqs_cpu_mask = cpu_mask\n\n        if PerfTunerBase.cpu_mask_is_zero(irqs_cpu_mask):\n            raise PerfTunerBase.CPUMaskIsZeroException(\"Bad configuration mode ({}) and cpu-mask value ({}): this results in a zero-mask for IRQs\".format(mq_mode.name, cpu_mask))\n\n        return irqs_cpu_mask\n\n    @property\n    def mode(self):\n        \"\"\"\n        Return the configuration mode\n        \"\"\"\n        return self.__mode\n\n    @mode.setter\n    def mode(self, new_mode):\n        \"\"\"\n        Set the new configuration mode and recalculate the corresponding masks.\n        \"\"\"\n        # Make sure the new_mode is of PerfTunerBase.AllowedModes type\n        self.__mode = PerfTunerBase.SupportedModes(new_mode)\n        self.__compute_cpu_mask = PerfTunerBase.compute_cpu_mask_for_mode(self.__mode, self.__args.cpu_mask)\n        self.__irq_cpu_mask = PerfTunerBase.irqs_cpu_mask_for_mode(self.__mode, self.__args.cpu_mask)\n\n    @property\n    def cpu_mask(self):\n        \"\"\"\n        Return the CPU mask we operate on (the total CPU set)\n        \"\"\"\n\n        return self.__args.cpu_mask\n\n    @property\n    def cores_per_irq_core(self):\n        \"\"\"\n        Return the number of cores we are going to allocate a single IRQ core out of when auto-detecting\n        \"\"\"\n        return self.__args.cores_per_irq_core\n\n    @staticmethod\n    def min_cores_per_irq_core():\n        \"\"\"\n        A minimum value of cores_per_irq_core.\n        We don't allocate a full IRQ core if total number of CPU cores is less or equal to 4.\n        \"\"\"\n        return 5\n\n    @property\n    def compute_cpu_mask(self):\n        \"\"\"\n        Return the CPU mask to use for seastar application binding.\n        \"\"\"\n        return self.__compute_cpu_mask\n\n    @property\n    def irqs_cpu_mask(self):\n        \"\"\"\n        Return the mask of CPUs used for IRQs distribution.\n        \"\"\"\n        return self.__irq_cpu_mask\n\n    @irqs_cpu_mask.setter\n    def irqs_cpu_mask(self, new_irq_cpu_mask):\n        self.__irq_cpu_mask = new_irq_cpu_mask\n\n        # Sanity check\n        if PerfTunerBase.cpu_mask_is_zero(self.__irq_cpu_mask):\n            raise PerfTunerBase.CPUMaskIsZeroException(\"Bad configuration: zero IRQ CPU mask is given\")\n\n        if run_hwloc_calc([self.__irq_cpu_mask]) == run_hwloc_calc([self.cpu_mask]):\n            # Special case: if IRQ CPU mask is the same as total CPU mask - set a Compute CPU mask to cpu_mask\n            self.__compute_cpu_mask = self.cpu_mask\n        else:\n            # Otherwise, a Compute CPU mask is a CPU mask without IRQ CPU mask bits\n            self.__compute_cpu_mask = run_hwloc_calc([self.cpu_mask, f\"~{self.__irq_cpu_mask}\"])\n\n        # Sanity check\n        if PerfTunerBase.cpu_mask_is_zero(self.__compute_cpu_mask):\n            raise PerfTunerBase.CPUMaskIsZeroException(\n                f\"Bad configuration: cpu_maks:{self.cpu_mask}, irq_cpu_mask:{self.__irq_cpu_mask}: \"\n                f\"results in a zero-mask for compute\")\n\n    @property\n    def is_aws_i3_non_metal_instance(self):\n        \"\"\"\n        :return: True if we are running on the AWS i3.nonmetal instance, e.g. i3.4xlarge\n        \"\"\"\n        if self.__is_aws_i3_nonmetal_instance is None:\n            self.__check_host_type()\n\n        return self.__is_aws_i3_nonmetal_instance\n\n    @property\n    def args(self):\n        return self.__args\n\n    @property\n    def irqs(self):\n        return self._get_irqs()\n\n#### \"Protected\"/Public (pure virtual) methods ###########\n    @abc.abstractmethod\n    def tune(self):\n        pass\n\n\n    @abc.abstractmethod\n    def _get_irqs(self):\n        \"\"\"\n        Return the iteratable value with all IRQs to be configured.\n        \"\"\"\n        pass\n\n#### Private methods ############################\n    @property\n    def __ec2_metadata_base_url(self):\n        return \"http://169.254.169.254/latest/\"\n\n    @property\n    def __metadata_token(self):\n        \"\"\"\n        Refresh IMDSv2 session token if it necessary, and return current token\n        :return: current session token\n        \"\"\"\n        token_ttl = 21600\n        update_token = False\n        if not self.__metadata_token_value:\n            update_token = True\n        else:\n            time_diff = datetime.datetime.now() - self.__metadata_token_time\n            time_diff_sec = int(time_diff.total_seconds())\n            if time_diff_sec >= token_ttl - 120:\n                update_token = True\n        if update_token:\n            self.__metadata_token_time = datetime.datetime.now()\n            req = urllib.request.Request(self.__ec2_metadata_base_url + \"api/token\", headers={\"X-aws-ec2-metadata-token-ttl-seconds\": token_ttl}, method=\"PUT\")\n            with urllib.request.urlopen(req, timeout=0.1) as res:\n                self.__metadata_token_value = res.read().decode()\n        return self.__metadata_token_value\n\n    def __get_instance_metadata(self, path):\n        \"\"\"\n        Get a parameter from EC2 Metadata server\n        :param path: metadata path to access for\n        :return: received metadata as a string\n        \"\"\"\n        req = urllib.request.Request(self.__ec2_metadata_base_url + 'meta-data/' + path, headers={\"X-aws-ec2-metadata-token\": self.__metadata_token})\n        with urllib.request.urlopen(req, timeout=0.1) as res:\n            return res.read().decode()\n\n    def __check_host_type(self):\n        \"\"\"\n        Check if we are running on the AWS i3 nonmetal instance.\n        If yes, set self.__is_aws_i3_nonmetal_instance to True, and to False otherwise.\n        \"\"\"\n        try:\n            aws_instance_type = self.__get_instance_metadata('instance-type')\n            if re.match(r'^i3\\.((?!metal)\\w)+$', aws_instance_type):\n                self.__is_aws_i3_nonmetal_instance = True\n            else:\n                self.__is_aws_i3_nonmetal_instance = False\n\n            return\n        except (urllib.error.URLError, ConnectionError, TimeoutError):\n            # Non-AWS case\n            pass\n        except:\n            logging.warning(\"Unexpected exception while attempting to access AWS meta server: {}\".format(sys.exc_info()[0]))\n\n        self.__is_aws_i3_nonmetal_instance = False\n\n#################################################\nclass NetPerfTuner(PerfTunerBase):\n    def __init__(self, args):\n        super().__init__(args)\n\n        self.nics=args.nics\n\n        self.__nic_is_bond_iface_dict = NetPerfTuner.__get_bond_ifaces()\n        self.__nic_is_vlan_iface_dict = NetPerfTuner.__get_vlan_ifaces()\n        self.__slaves_dict = self.__learn_slaves()\n\n        # check that self.nics contain a HW device or a supported composite interface\n        self.__check_nics()\n\n        # Fetch IRQs related info\n        self.__get_irqs_info()\n\n\n#### Public methods ############################\n    def tune(self):\n        \"\"\"\n        Tune the networking server configuration.\n        \"\"\"\n        for nic in self.nics:\n            if self.__nic_is_tunable(nic):\n                perftune_print(\"Setting a tunable interface {}...\".format(nic))\n                self.__setup_one_tunable_iface(nic)\n\n            if self.__nic_has_slaves(nic):\n                nic_type = \"virtual\"\n                if self.__nic_is_bond_iface(nic):\n                    nic_type = \"bond\"\n                elif self.__nic_is_vlan_iface(nic):\n                    nic_type = \"VLAN\"\n                perftune_print(f\"Setting a {nic} {nic_type} interface...\")\n                self.__setup_virtual_iface(nic)\n\n        # Increase the socket listen() backlog\n        fwriteln_and_log('/proc/sys/net/core/somaxconn', '4096')\n\n        # Increase the maximum number of remembered connection requests, which are still\n        # did not receive an acknowledgment from connecting client.\n        fwriteln_and_log('/proc/sys/net/ipv4/tcp_max_syn_backlog', '4096')\n\n        self.__tune_tcp_mem()\n\n#### Protected methods ##########################\n    def _get_irqs(self):\n        \"\"\"\n        Returns the iterator for all IRQs that are going to be configured (according to args.nics parameter).\n        For instance, for a bonding interface that's going to include IRQs of all its slaves.\n        \"\"\"\n        return itertools.chain.from_iterable(self.__nic2irqs.values())\n\n#### Private methods ############################\n    def __tune_tcp_mem(self):\n        page_size = mmap.PAGESIZE\n        total_mem = psutil.virtual_memory().total\n        # We only tune for physical memory since tcp_mem is virtualized\n        def to_pages(bytes):\n            return math.ceil(bytes / page_size)\n        max = total_mem * self.args.tcp_mem_fraction\n        fwriteln_and_log('/proc/sys/net/ipv4/tcp_mem', f\"{to_pages(max / 2)} {to_pages(max * 2/3)} {to_pages(max)}\")\n\n    def __nic_is_bond_iface(self, nic):\n        return self.__nic_is_bond_iface_dict.get(nic, False)\n\n    def __nic_is_vlan_iface(self, nic):\n        return self.__nic_is_vlan_iface_dict.get(nic, False)\n\n    def __nic_has_slaves(self, nic):\n        return nic in self.__slaves_dict and len(self.__slaves_dict[nic]) > 0\n\n    def __nic_exists(self, nic):\n        return self.__iface_exists(nic)\n\n    def __nic_is_tunable(self, nic):\n        return self.__dev_is_tunalbe_iface(nic)\n\n    def __slaves(self, nic):\n        \"\"\"\n        Returns an iterator for all slaves of the nic.\n        If agrs.nic is not a composite interface an attempt to use the returned iterator\n        will immediately raise a StopIteration exception - use __nic_has_slaves(nic) check to avoid this.\n        \"\"\"\n        return iter(self.__slaves_dict[nic])\n\n    def __get_irqs_info(self):\n        self.__irqs2procline = get_irqs2procline_map()\n        self.__nic2irqs = self.__learn_irqs()\n\n    @property\n    def __rfs_table_size(self):\n        return 32768\n\n    def __check_nics(self):\n        \"\"\"\n        Checks that self.nics are supported interfaces\n        \"\"\"\n        for nic in self.nics:\n            if not self.__nic_exists(nic):\n                raise Exception(\"Device {} does not exist\".format(nic))\n            if not self.__nic_is_tunable(nic) and not self.__nic_has_slaves(nic):\n                raise Exception(\"Not supported virtual device {}\".format(nic))\n\n    def __get_irqs_one(self, iface):\n        \"\"\"\n        Returns the list of IRQ numbers for the given interface.\n        \"\"\"\n        return self.__nic2irqs.get(iface, [])\n\n    def __setup_rfs(self, iface):\n        rps_limits = glob.glob(\"/sys/class/net/{}/queues/*/rps_flow_cnt\".format(iface))\n        one_q_limit = int(self.__rfs_table_size / len(rps_limits))\n\n        # If RFS feature is not present - get out\n        try:\n            run_one_command(['sysctl', 'net.core.rps_sock_flow_entries'])\n        except:\n            return\n\n        # Enable RFS\n        perftune_print(\"Setting net.core.rps_sock_flow_entries to {}\".format(self.__rfs_table_size))\n        run_one_command(['sysctl', '-w', 'net.core.rps_sock_flow_entries={}'.format(self.__rfs_table_size)])\n\n        # Set each RPS queue limit\n        for rfs_limit_cnt in rps_limits:\n            msg = \"Setting limit {} in {}\".format(one_q_limit, rfs_limit_cnt)\n            fwriteln(rfs_limit_cnt, \"{}\".format(one_q_limit), log_message=msg)\n\n        # Enable/Disable ntuple filtering HW offload on the NIC. This is going to enable/disable aRFS on NICs supporting\n        # aRFS since ntuple is pre-requisite for an aRFS feature.\n        # If no explicit configuration has been requested enable ntuple (and thereby aRFS) only in MQ mode.\n        #\n        # aRFS acts similar to (SW) RFS: it places a TCP packet on a HW queue that it supposed to be \"close\" to an\n        # application thread that sent a packet on the same TCP stream.\n        #\n        # For instance if a given TCP stream was sent from CPU3 then the next Rx packet is going to be placed in an Rx\n        # HW queue which IRQ affinity is set to CPU3 or otherwise to the one with affinity close enough to CPU3.\n        #\n        # Read more here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/network-acc-rfs\n        #\n        # Obviously it would achieve the best result if there is at least one Rx HW queue with an affinity set to each\n        # application threads that handle TCP.\n        #\n        # And, similarly, if we know in advance that there won't be any such HW queue (sq and sq_split modes) - there is\n        # no sense enabling aRFS.\n        op = \"Enable\"\n        value = 'on'\n\n        if (self.args.enable_arfs is None and self.irqs_cpu_mask != self.cpu_mask) or self.args.enable_arfs is False:\n            op = \"Disable\"\n            value = 'off'\n\n        ethtool_msg = \"{} ntuple filtering HW offload for {}...\".format(op, iface)\n\n        if dry_run_mode:\n            perftune_print(ethtool_msg)\n            run_one_command(['ethtool','-K', iface, 'ntuple', value], stderr=subprocess.DEVNULL)\n        else:\n            try:\n                print(\"Trying to {} ntuple filtering HW offload for {}...\".format(op.lower(), iface), end='')\n                run_one_command(['ethtool','-K', iface, 'ntuple', value], stderr=subprocess.DEVNULL)\n                print(\"ok\")\n            except:\n                print(\"not supported\")\n\n    def __setup_rps(self, iface, mask):\n        for one_rps_cpus in self.__get_rps_cpus(iface):\n            set_one_mask(one_rps_cpus, mask)\n\n        self.__setup_rfs(iface)\n\n    def __setup_xps(self, iface):\n        xps_cpus_list = glob.glob(\"/sys/class/net/{}/queues/*/xps_cpus\".format(iface))\n        masks = run_hwloc_distrib([\"{}\".format(len(xps_cpus_list))])\n\n        for i, mask in enumerate(masks):\n            set_one_mask(xps_cpus_list[i], mask)\n\n    def __iface_exists(self, iface):\n        if len(iface) == 0:\n            return False\n        return os.path.exists(\"/sys/class/net/{}\".format(iface))\n\n    def __dev_is_tunalbe_iface(self, iface):\n        return os.path.exists(\"/sys/class/net/{}/device\".format(iface))\n\n    @staticmethod\n    def __get_bond_ifaces():\n        if not os.path.exists('/sys/class/net/bonding_masters'):\n            return {}\n\n        bond_dict = {}\n        for line in open('/sys/class/net/bonding_masters', 'r').readlines():\n            for nic in line.split():\n                bond_dict[nic] = True\n\n        return bond_dict\n\n    @staticmethod\n    def __get_vlan_ifaces():\n        # Each VLAN interface is going to have a corresponding entry in /proc/net/vlan/ directory\n        return {pathlib.PurePath(pathlib.Path(f)).name: True\n                for f in filter(lambda vlan_name: vlan_name != \"/proc/net/vlan/config\", glob.glob(\"/proc/net/vlan/*\"))}\n\n    def __learn_slaves_one(self, nic):\n        \"\"\"\n        Learn underlying physical devices a given NIC\n\n        :param nic: An interface to search slaves for\n        \"\"\"\n        slaves_list = set()\n\n        if self.__nic_is_bond_iface(nic):\n            top_slaves_list = set(itertools.chain.from_iterable(\n                [line.split() for line in open(\"/sys/class/net/{}/bonding/slaves\".format(nic), 'r').readlines()]))\n        else:\n            # Some virtual interfaces (e.g. VLANs) have a symbolic link 'lower_<parent_interface_name>' under\n            # /sys/class/net/<interface name> representing a lower level dependent interface.\n            #\n            # For example:\n            #\n            # lrwxrwxrwx  1 root root    0 Jul  5 18:38 lower_eno1 -> ../../../pci0000:00/0000:00:1f.6/net/eno1/\n            #\n            top_slaves_list = set([pathlib.PurePath(pathlib.Path(f).resolve()).name\n                                   for f in glob.glob(f\"/sys/class/net/{nic}/lower_*\")])\n\n        # Slaves can themselves have slaves of their own: let's descend (DFS) all the way down to get physical devices.\n        # We don't want to include not-tunable interfaces in the resulting slaves list.\n        # We do want to check if any (tunable or not) slave has slaves of their own that might be tunable.\n        for s in top_slaves_list:\n            # Avoid cycles - should not happen but just in case\n            if s in slaves_list:\n                continue\n\n            if self.__nic_is_tunable(s):\n                slaves_list.add(s)\n\n            slaves_list |= self.__learn_slaves_one(s)\n\n        return slaves_list\n\n    def __learn_slaves(self):\n        \"\"\"\n        Resolve underlying physical devices for interfaces we are requested to configure\n        \"\"\"\n        slaves_list_per_nic = {}\n        for nic in self.nics:\n            current_slaves = self.__learn_slaves_one(nic)\n            if current_slaves:\n                slaves_list_per_nic[nic] = list(current_slaves)\n\n        return slaves_list_per_nic\n\n    def __intel_irq_to_queue_idx(self, irq):\n        \"\"\"\n        Return the HW queue index for a given IRQ for Intel NICs in order to sort the IRQs' list by this index.\n\n        Intel's fast path IRQs have the following name convention:\n             <bla-bla>-TxRx-<queue index>\n\n        Intel NICs also have the IRQ for Flow Director (which is not a regular fast path IRQ) whose name looks like\n        this:\n             <bla-bla>:fdir-TxRx-<index>\n\n        We want to put the Flow Director's IRQ at the end of the sorted list of IRQs.\n\n        :param irq: IRQ number\n        :return: HW queue index for Intel NICs and sys.maxsize for all other NICs\n        \"\"\"\n        intel_fp_irq_re = re.compile(r\"-TxRx-(\\d+)\")\n        fdir_re = re.compile(r\"fdir-TxRx-\\d+\")\n\n        m = intel_fp_irq_re.search(self.__irqs2procline[irq])\n        m1 = fdir_re.search(self.__irqs2procline[irq])\n        if m and not m1:\n            return int(m.group(1))\n        else:\n            return sys.maxsize\n\n    def __mlx_irq_to_queue_idx(self, irq):\n        \"\"\"\n        Return the HW queue index for a given IRQ for Mellanox NICs in order to sort the IRQs' list by this index.\n\n        Mellanox NICs have the IRQ which name looks like\n        this:\n        mlx5_comp23\n             mlx5_comp<index>\n        or this:\n        mlx4-6\n             mlx4-<index>\n\n        :param irq: IRQ number\n        :return: HW queue index for Mellanox NICs and sys.maxsize for all other NICs\n        \"\"\"\n        mlx5_fp_irq_re = re.compile(r\"mlx5_comp(\\d+)\")\n        mlx4_fp_irq_re = re.compile(r\"mlx4-(\\d+)\")\n\n        m5 = mlx5_fp_irq_re.search(self.__irqs2procline[irq])\n        if m5:\n            return int(m5.group(1))\n        else:\n            m4 = mlx4_fp_irq_re.search(self.__irqs2procline[irq])\n            if m4:\n                return int(m4.group(1))\n\n        return sys.maxsize\n\n    def __mana_irq_to_queue_idx(self, irq):\n        \"\"\"\n        Return the HW queue index for a given IRQ for Microsoft Azure Network Adapter (MANA) NICs in order to sort the\n        IRQs' list by this index.\n\n        MANA NICs have the IRQ which name looks like this:\n             mana_hwc@...\n             mana_q<index>@...\n\n        We don't care much about 'mana_hwc' IRQs since they are a slow path event IRQs.\n        mana_q<index> IRQs are the fast path queues IRQs.\n        Therefore, we will order HWC IRQs to always be the last in the sorted IRQs' list.\n\n        :param irq: IRQ number\n        :return: HW queue index for MANA NICs and sys.maxsize for all other NICs\n        \"\"\"\n        mana_fp_irq_re = re.compile(r\"\\s+mana_q(\\d+)\")\n\n        m = mana_fp_irq_re.search(self.__irqs2procline[irq])\n        if m:\n            return int(m.group(1))\n\n        return sys.maxsize\n\n    def __virtio_irq_to_queue_idx(self, irq):\n        \"\"\"\n        Return the HW queue index for a given IRQ for VIRTIO in order to sort the IRQs' list by this index.\n\n        VIRTIO NICs have the IRQ's name that looks like this:\n        Queue K of a device virtioY, where Y is some integer is comprised of 2 IRQs\n        with following names:\n          * Tx IRQ:\n               virtioY-output.K\n          * Rx IRQ:\n               virtioY-input.K\n\n        :param irq: IRQ number\n        :return: HW queue index for VIRTIO fast path IRQ and sys.maxsize for all other IRQs\n        \"\"\"\n        virtio_fp_re = re.compile(r\"virtio\\d+-(input|output)\\.(\\d+)$\")\n\n        virtio_fp_irq = virtio_fp_re.search(self.__irqs2procline[irq])\n        if virtio_fp_irq:\n            return int(virtio_fp_irq.group(2))\n\n        return sys.maxsize\n\n\n    def __get_driver_name(self, iface):\n        \"\"\"\n        :param iface: Interface to check\n        :return: driver name from ethtool\n        \"\"\"\n\n        driver_name = ''\n        ethtool_i_lines = run_ethtool(['-i', iface])\n        driver_re = re.compile(\"driver:\")\n        driver_lines = list(filter(lambda one_line: driver_re.search(one_line), ethtool_i_lines))\n\n        if driver_lines:\n            if len(driver_lines) > 1:\n                raise Exception(\"More than one 'driver:' entries in the 'ethtool -i {}' output. Unable to continue.\".format(iface))\n\n            driver_name = driver_lines[0].split()[1].strip()\n\n        return driver_name\n\n    def __learn_irqs_one(self, iface):\n        \"\"\"\n        This is a slow method that is going to read from the system files. Never\n        use it outside the initialization code. Use __get_irqs_one() instead.\n\n        Filter the fast path queues IRQs from the __get_all_irqs_one() result according to the known\n        patterns.\n        Right now we know about the following naming convention of the fast path queues vectors:\n          - Intel:    <bla-bla>-TxRx-<bla-bla>\n          - Broadcom: <bla-bla>-fp-<bla-bla>\n          - ena:      <bla-bla>-Tx-Rx-<bla-bla>\n          - Mellanox: for mlx4\n                      mlx4-<queue idx>@<bla-bla>\n                      or for mlx5\n                      mlx5_comp<queue idx>@<bla-bla>\n          - VIRTIO: virtioN-[input|output].D\n          - MANA: mana_q<queue idx>@<bla-bla>\n\n        So, we will try to filter the etries in /proc/interrupts for IRQs we've got from get_all_irqs_one()\n        according to the patterns above.\n\n        If as a result all IRQs are filtered out (if there are no IRQs with the names from the patterns above) then\n        this means that the given NIC uses a different IRQs naming pattern. In this case we won't filter any IRQ.\n\n        Otherwise, we will use only IRQs which names fit one of the patterns above.\n\n        For NICs with a limited number of Rx queues the IRQs that handle Rx are going to be at the beginning of the\n        list.\n        \"\"\"\n        # filter 'all_irqs' to only reference valid keys from 'irqs2procline' and avoid an IndexError on the 'irqs' search below\n        all_irqs = set(learn_all_irqs_one(\"/sys/class/net/{}/device\".format(iface), self.__irqs2procline, iface)).intersection(self.__irqs2procline.keys())\n        fp_irqs_re = re.compile(r\"-TxRx-|-fp-|-Tx-Rx-|mlx4-\\d+@|mlx5_comp\\d+@|virtio\\d+-(input|output)|mana_q\\d+@\")\n        irqs = sorted(list(filter(lambda irq : fp_irqs_re.search(self.__irqs2procline[irq]), all_irqs)))\n        if irqs:\n            irqs.sort(key=self.__get_irq_to_queue_idx_functor(iface))\n            return irqs\n        else:\n            return list(all_irqs)\n\n    def __get_irq_to_queue_idx_functor(self, iface):\n        \"\"\"\n        Get a functor returning a queue index for a given IRQ.\n        This functor is needed for NICs that are known to not release IRQs when the number of Rx\n        channels is reduced or have extra IRQs for non-RSS channels.\n\n        Therefore, for these NICs we need a functor that would allow us to pick IRQs that belong to channels that are\n        going to handle TCP traffic: first X channels, where the value of X depends on the NIC's type and configuration.\n\n        For others, e.g. ENA, or Broadcom, which are only going to allocate IRQs that belong to TCP handling channels,\n        we don't really need to sort them as long as we filter fast path IRQs and distribute them evenly among IRQ CPUs.\n\n        :param iface: NIC's interface name, e.g. eth19\n        :return: A functor that returns a queue index for a given IRQ if a mapping is known\n                 or a constant big integer value if mapping is unknown.\n        \"\"\"\n        # There are a few known drivers for which we know how to get a queue index from an IRQ name in /proc/interrupts\n        driver_name = self.__get_driver_name(iface)\n\n        # Every functor returns a sys.maxsize for an unknown driver IRQs.\n        # So, choosing Intel's as a default is as good as any other.\n        irq_to_idx_func = self.__intel_irq_to_queue_idx\n        if driver_name.startswith(\"mlx\"):\n            irq_to_idx_func = self.__mlx_irq_to_queue_idx\n        elif driver_name.startswith(\"virtio\"):\n            irq_to_idx_func = self.__virtio_irq_to_queue_idx\n        elif driver_name.startswith(\"mana\"):\n            irq_to_idx_func = self.__mana_irq_to_queue_idx\n\n        return irq_to_idx_func\n\n    def __irq_lower_bound_by_queue(self, iface, irqs, queue_idx):\n        \"\"\"\n        Get the index of the first element in irqs array which queue is greater or equal to a given index.\n        IRQs array is supposed to be sorted by queues numbers IRQs belong to.\n\n        There are additional assumptions:\n          * IRQs array items queue numbers are monotonically not decreasing, and if it increases then it increases by\n            one.\n          * Queue indexes are numbered starting from zero.\n\n        :param irqs: IRQs array sorted by queues numbers IRQs belong to\n        :param queue_idx: Queue index to partition by\n        :return: The first index in the IRQs array that corresponds to a queue number greater or equal to a given index\n                 which is at least queue_idx. If there is no such IRQ - returns len(irqs).\n        \"\"\"\n        irq_to_idx_func = self.__get_irq_to_queue_idx_functor(iface)\n\n        if queue_idx < len(irqs):\n            for idx in range(queue_idx, len(irqs)):\n                if irq_to_idx_func(irqs[idx]) >= queue_idx:\n                    return idx\n\n        return len(irqs)\n\n    def __learn_irqs(self):\n        \"\"\"\n        This is a slow method that is going to read from the system files. Never\n        use it outside the initialization code.\n        \"\"\"\n        nic_irq_dict={}\n        for nic in self.nics:\n            if self.__nic_has_slaves(nic):\n                # Slaves should not include not-tunable interfaces but just in case let's filter them out for safety\n                for slave in filter(self.__dev_is_tunalbe_iface, self.__slaves(nic)):\n                    nic_irq_dict[slave] = self.__learn_irqs_one(slave)\n            else:\n                nic_irq_dict[nic] = self.__learn_irqs_one(nic)\n        return nic_irq_dict\n\n    def __get_rps_cpus(self, iface):\n        \"\"\"\n        Prints all rps_cpus files names for the given HW interface.\n\n        There is a single rps_cpus file for each RPS queue and there is a single RPS\n        queue for each HW Rx queue. Each HW Rx queue should have an IRQ.\n        Therefore the number of these files is equal to the number of fast path Rx IRQs for this interface.\n        \"\"\"\n        return glob.glob(\"/sys/class/net/{}/queues/*/rps_cpus\".format(iface))\n\n    def __set_rx_channels_count(self, iface, count):\n        \"\"\"\n        Try to set the number of Rx channels of a given interface to a given value.\n\n        Rx channels of any NIC can be configured using 'ethtool -L' command using one of the following semantics:\n\n        ethtool -L <iface> rx <count>\n        or\n        ethtool -L <iface> combined <count>\n\n        If a specific semantics is not supported by a given NIC or if changing the number of channels is not supported\n        ethtool is going to return an error.\n\n        Instead of parsing and trying to detect which one of the following semantics a given interface supports we will\n        simply try to use both semantics till either one of them succeeds or both fail.\n\n\n        :param iface: NIC interface name, e.g. eth4\n        :param count: number of Rx channels we want to configure\n        :return: True if configuration was successful, False otherwise\n        \"\"\"\n        options = [\"rx\", \"combined\"]\n        for o in options:\n            try:\n                cmd = ['ethtool', '-L', iface, o, f\"{count}\"]\n                perftune_print(f\"Executing: {' '.join(cmd)}\")\n                run_one_command(cmd, stderr=subprocess.DEVNULL)\n                return True\n            except subprocess.CalledProcessError:\n                pass\n\n        return False\n\n    def __setup_one_tunable_iface(self, iface):\n        # Set Rx channels count to a number of IRQ CPUs unless an explicit count is given\n        if self.args.num_rx_queues is not None:\n            num_rx_channels = self.args.num_rx_queues\n        else:\n            num_rx_channels = 0\n\n            # If a mask is wider than 32 bits it's going to be presented as a comma-separated list of 32-bit masks\n            # with possibly omitted zero components, e.g. 0x01,0x100,,0x12122\n            for m in self.irqs_cpu_mask.split(\",\"):\n                if m:\n                    num_rx_channels += bin(int(m, 16)).count('1')\n\n        # Let's try setting the number of Rx channels to the number of IRQ CPUs.\n        #\n        # If we were able to change the number of Rx channels the number of IRQs could have changed.\n        # In this case let's refresh IRQs info.\n        rx_channels_set = self.__set_rx_channels_count(iface, num_rx_channels)\n        if rx_channels_set:\n            self.__get_irqs_info()\n\n        max_num_rx_queues = self.__max_rx_queue_count(iface)\n        all_irqs = self.__get_irqs_one(iface)\n\n        # Bind the NIC's IRQs according to the configuration mode\n        #\n        # If this NIC has a limited number of Rx queues then we want to distribute their IRQs separately.\n        # For such NICs we've sorted IRQs list so that IRQs that handle Rx are all at the head of the list.\n        if rx_channels_set or max_num_rx_queues < len(all_irqs):\n            num_rx_queues = self.__get_rx_queue_count(iface)\n            # Let's be optimistic during a dry run and assume that the RX channels setting was successful\n            if dry_run_mode:\n                num_rx_queues = num_rx_channels\n\n            tcp_irqs_lower_bound = self.__irq_lower_bound_by_queue(iface, all_irqs, num_rx_queues)\n            perftune_print(f\"Distributing IRQs handling Rx and Tx for first {num_rx_queues} channels:\")\n            distribute_irqs(all_irqs[0:tcp_irqs_lower_bound], self.irqs_cpu_mask)\n            perftune_print(\"Distributing the rest of IRQs\")\n            distribute_irqs(all_irqs[tcp_irqs_lower_bound:], self.irqs_cpu_mask)\n        else:\n            perftune_print(\"Distributing all IRQs\")\n            distribute_irqs(all_irqs, self.irqs_cpu_mask)\n\n        self.__setup_rps(iface, self.cpu_mask)\n        self.__setup_xps(iface)\n\n    def __setup_virtual_iface(self, nic):\n        \"\"\"\n        Set up the interface which is a bond or a VLAN interface\n        :param nic: name of a composite interface to set up\n        \"\"\"\n        for slave in self.__slaves(nic):\n            if self.__dev_is_tunalbe_iface(slave):\n                perftune_print(\"Setting up {}...\".format(slave))\n                self.__setup_one_tunable_iface(slave)\n            else:\n                perftune_print(\"Skipping {} (not a physical slave device?)\".format(slave))\n\n    def __max_rx_queue_count(self, iface):\n        \"\"\"\n        :param iface: Interface to check\n        :return: The maximum number of RSS queues for the given interface if there is known limitation and sys.maxsize\n        otherwise.\n\n        Networking drivers serving HW with the known maximum RSS queue limitation (due to lack of RSS bits):\n\n        ixgbe:   PF NICs support up to 16 RSS queues.\n        ixgbevf: VF NICs support up to 4 RSS queues.\n        i40e:    PF NICs support up to 64 RSS queues.\n        i40evf:  VF NICs support up to 16 RSS queues.\n\n        \"\"\"\n        driver_to_max_rss = {'ixgbe': 16, 'ixgbevf': 4, 'i40e': 64, 'i40evf': 16}\n\n        driver_name = self.__get_driver_name(iface)\n        return driver_to_max_rss.get(driver_name, sys.maxsize)\n\n    def __get_rx_queue_count(self, iface):\n        \"\"\"\n        :return: the RSS Rx queues count for the given interface.\n        \"\"\"\n        num_irqs = len(self.__get_irqs_one(iface))\n        rx_queues_count = len(self.__get_rps_cpus(iface))\n\n        if rx_queues_count == 0:\n            rx_queues_count = num_irqs\n\n        return min(self.__max_rx_queue_count(iface), rx_queues_count)\n\n\n\nclass ClocksourceManager:\n    class PreferredClockSourceNotAvailableException(Exception):\n        pass\n\n    def __init__(self, args):\n        self.__args = args\n        self._preferred = {\"x86_64\": \"tsc\", \"kvm\": \"kvm-clock\"}\n        self._arch = self._get_arch()\n        self._available_clocksources_file = \"/sys/devices/system/clocksource/clocksource0/available_clocksource\"\n        self._current_clocksource_file = \"/sys/devices/system/clocksource/clocksource0/current_clocksource\"\n        self._recommendation_if_unavailable = { \"x86_64\": \"The tsc clocksource is not available. Consider using a hardware platform where the tsc clocksource is available, or try forcing it withe the tsc=reliable boot option\", \"kvm\": \"kvm-clock is not available\" }\n\n    def _available_clocksources(self):\n        return open(self._available_clocksources_file).readline().split()\n\n    def _current_clocksource(self):\n        return open(self._current_clocksource_file).readline().strip()\n\n    def _get_arch(self):\n        try:\n            virt = run_read_only_command(['systemd-detect-virt']).strip()\n            # According to https://www.freedesktop.org/software/systemd/man/latest/systemd-detect-virt.html\n            # 'amazon' and 'google' are returned for KVM guests.\n            if virt in [\"kvm\", \"amazon\", \"google\"]:\n                return \"kvm\"\n        except:\n            pass\n        return platform.machine()\n\n    def enforce_preferred_clocksource(self):\n        fwriteln(self._current_clocksource_file, self._preferred[self._arch], \"Setting clocksource to {}\".format(self._preferred[self._arch]))\n\n    def preferred(self):\n        return self._preferred[self._arch]\n\n    def setting_available(self):\n        return self._arch in self._preferred\n\n    def preferred_clocksource_available(self):\n        return self._preferred[self._arch] in self._available_clocksources()\n\n    def recommendation_if_unavailable(self):\n        return self._recommendation_if_unavailable[self._arch]\n\nclass SystemPerfTuner(PerfTunerBase):\n    def __init__(self, args):\n        super().__init__(args)\n        self._clocksource_manager = ClocksourceManager(args)\n\n    def tune(self):\n        if self.args.tune_clock:\n            if not self._clocksource_manager.setting_available():\n                perftune_print(\"Clocksource setting not available or not needed for this architecture. Not tuning\")\n            elif not self._clocksource_manager.preferred_clocksource_available():\n                perftune_print(self._clocksource_manager.recommendation_if_unavailable())\n            else:\n                self._clocksource_manager.enforce_preferred_clocksource()\n\n#### Protected methods ##########################\n    def _get_irqs(self):\n        return []\n\n\n#################################################\nclass DiskPerfTuner(PerfTunerBase):\n    class SupportedDiskTypes(enum.IntEnum):\n        nvme = 0\n        non_nvme = 1\n\n    def __init__(self, args):\n        super().__init__(args)\n\n        if not (self.args.dirs or self.args.devs):\n            raise Exception(\"'disks' tuning was requested but neither directories nor storage devices were given\")\n\n        self.__pyudev_ctx = pyudev.Context()\n        self.__dir2disks = self.__learn_directories()\n        self.__irqs2procline = get_irqs2procline_map()\n        self.__disk2irqs = self.__learn_irqs()\n        self.__type2diskinfo = self.__group_disks_info_by_type()\n\n        # sets of devices that have already been tuned\n        self.__io_scheduler_tuned_devs = set()\n        self.__nomerges_tuned_devs = set()\n        self.__write_back_cache_tuned_devs = set()\n\n#### Public methods #############################\n    def tune(self):\n        \"\"\"\n        Distribute IRQs according to the requested mode (args.mode):\n           - Distribute NVMe disks' IRQs equally among all available CPUs.\n           - Distribute non-NVMe disks' IRQs equally among designated CPUs or among\n             all available CPUs in the 'mq' mode.\n        \"\"\"\n        non_nvme_disks, non_nvme_irqs = self.__disks_info_by_type(DiskPerfTuner.SupportedDiskTypes.non_nvme)\n        if non_nvme_disks:\n            perftune_print(\"Setting non-NVMe disks: {}...\".format(\", \".join(non_nvme_disks)))\n            distribute_irqs(non_nvme_irqs, self.irqs_cpu_mask)\n            self.__tune_disks(non_nvme_disks)\n        else:\n            perftune_print(\"No non-NVMe disks to tune\")\n\n        nvme_disks, nvme_irqs = self.__disks_info_by_type(DiskPerfTuner.SupportedDiskTypes.nvme)\n        if nvme_disks:\n            # Linux kernel is going to use IRQD_AFFINITY_MANAGED mode for NVMe IRQs\n            # on most systems (currently only AWS i3 non-metal are known to have a\n            # different configuration). SMP affinity of an IRQ in this mode may not be\n            # changed and an attempt to modify it is going to fail. However right now\n            # the only way to determine that IRQD_AFFINITY_MANAGED mode has been used\n            # is to attempt to modify IRQ SMP affinity (and fail) therefore we prefer\n            # to always do it.\n            #\n            # What we don't want however is to see annoying errors every time we\n            # detect that IRQD_AFFINITY_MANAGED was actually used. Therefore we will only log\n            # them in the \"verbose\" mode or when we run on an i3.nonmetal AWS instance.\n            perftune_print(\"Setting NVMe disks: {}...\".format(\", \".join(nvme_disks)))\n            distribute_irqs(nvme_irqs, self.args.cpu_mask,\n                            log_errors=(self.is_aws_i3_non_metal_instance or self.args.verbose))\n            self.__tune_disks(nvme_disks)\n        else:\n            perftune_print(\"No NVMe disks to tune\")\n\n#### Protected methods ##########################\n    def _get_irqs(self):\n        return itertools.chain.from_iterable(irqs for disks, irqs in self.__type2diskinfo.values())\n\n#### Private methods ############################\n    @property\n    def __io_schedulers(self):\n        \"\"\"\n        :return: An ordered list of IO schedulers that we want to configure. Schedulers are ordered by their priority\n        from the highest (left most) to the lowest.\n        \"\"\"\n        return [\"none\", \"noop\"]\n\n    @property\n    def __nomerges(self):\n        return '2'\n\n    @property\n    def __write_cache_config(self):\n        \"\"\"\n        :return: None - if write cache mode configuration is not requested or the corresponding write cache\n        configuration value string\n        \"\"\"\n        if self.args.set_write_back is None:\n            return None\n\n        return \"write back\" if self.args.set_write_back else \"write through\"\n\n    def __disks_info_by_type(self, disks_type):\n        \"\"\"\n        Returns a tuple ( [<disks>], [<irqs>] ) for the given disks type.\n        IRQs numbers in the second list are promised to be unique.\n        \"\"\"\n        return self.__type2diskinfo[DiskPerfTuner.SupportedDiskTypes(disks_type)]\n\n    def __nvme_fast_path_irq_filter(self, irq):\n        \"\"\"\n        Return True for fast path NVMe IRQs.\n        For NVMe device only queues 1-<number of CPUs> are going to do fast path work.\n\n        NVMe IRQs have the following name convention:\n             nvme<device index>q<queue index>, e.g. nvme0q7\n\n        :param irq: IRQ number\n        :return: True if this IRQ is an IRQ of a FP NVMe queue.\n        \"\"\"\n        nvme_irq_re = re.compile(r'(\\s|^)nvme\\d+q(\\d+)(\\s|$)')\n\n        # There may be more than an single HW queue bound to the same IRQ. In this case queue names are going to be\n        # comma separated\n        split_line = self.__irqs2procline[irq].split(\",\")\n\n        for line in split_line:\n            m = nvme_irq_re.search(line)\n            if m and 0 < int(m.group(2)) <= multiprocessing.cpu_count():\n                return True\n\n        return False\n\n    def __group_disks_info_by_type(self):\n        \"\"\"\n        Return a map of tuples ( [<disks>], [<irqs>] ), where \"disks\" are all disks of the specific type\n        and \"irqs\" are the corresponding IRQs.\n\n        It's promised that every element is \"disks\" and \"irqs\" is unique.\n\n        The disk types are 'nvme' and 'non-nvme'\n        \"\"\"\n        disks_info_by_type = {}\n        nvme_disks = set()\n        nvme_irqs = set()\n        non_nvme_disks = set()\n        non_nvme_irqs = set()\n        nvme_disk_name_pattern = re.compile('^nvme')\n\n        for disk, irqs in self.__disk2irqs.items():\n            if nvme_disk_name_pattern.search(disk):\n                nvme_disks.add(disk)\n                for irq in irqs:\n                    nvme_irqs.add(irq)\n            else:\n                non_nvme_disks.add(disk)\n                for irq in irqs:\n                    non_nvme_irqs.add(irq)\n\n        if not (nvme_disks or non_nvme_disks):\n            raise Exception(\"'disks' tuning was requested but no disks were found\")\n\n        nvme_irqs = list(nvme_irqs)\n\n        # There is a known issue with Xen hypervisor that exposes itself on AWS i3 instances where nvme module\n        # over-allocates HW queues and uses only queues 1,2,3,..., <up to number of CPUs> for data transfer.\n        # On these instances we will distribute only these queues.\n\n        if self.is_aws_i3_non_metal_instance:\n            nvme_irqs = list(filter(self.__nvme_fast_path_irq_filter, nvme_irqs))\n\n        # Sort IRQs for easier verification\n        nvme_irqs.sort(key=lambda irq_num_str: int(irq_num_str))\n\n        disks_info_by_type[DiskPerfTuner.SupportedDiskTypes.nvme] = (list(nvme_disks), nvme_irqs)\n        disks_info_by_type[DiskPerfTuner.SupportedDiskTypes.non_nvme] = ( list(non_nvme_disks), list(non_nvme_irqs) )\n\n        return disks_info_by_type\n\n    def __learn_directories(self):\n        return { directory : self.__learn_directory(directory) for directory in self.args.dirs }\n\n    def __learn_directory(self, directory, recur=False):\n        \"\"\"\n        Returns a list of disks the given directory is mounted on (there will be more than one if\n        the mount point is on the RAID volume)\n        \"\"\"\n        if not os.path.exists(directory):\n            if not recur:\n                perftune_print(\"{} doesn't exist - skipping\".format(directory))\n\n            return []\n\n        try:\n            udev_obj = pyudev.Devices.from_device_number(self.__pyudev_ctx, 'block', os.stat(directory).st_dev)\n            return self.__get_phys_devices(udev_obj)\n        except:\n            # handle cases like ecryptfs where the directory is mounted to another directory and not to some block device\n            filesystem = run_read_only_command(['df', '-P', directory]).splitlines()[-1].split()[0].strip()\n            if not re.search(r'^/dev/', filesystem):\n                devs = self.__learn_directory(filesystem, True)\n            else:\n                raise Exception(\"Logic error: failed to create a udev device while 'df -P' {} returns a {}\".format(directory, filesystem))\n\n            # log error only for the original directory\n            if not recur and not devs:\n                perftune_print(\"Can't get a block device for {} - skipping\".format(directory))\n\n            return devs\n\n    def __get_phys_devices(self, udev_obj):\n        # if device is a virtual device - the underlying physical devices are going to be its slaves\n        if re.search(r'virtual', udev_obj.sys_path):\n            slaves = os.listdir(os.path.join(udev_obj.sys_path, 'slaves'))\n            # If the device is virtual but doesn't have slaves (e.g. as nvm-subsystem virtual devices) handle it\n            # as a regular device.\n            if slaves:\n                return list(itertools.chain.from_iterable([ self.__get_phys_devices(pyudev.Devices.from_device_file(self.__pyudev_ctx, \"/dev/{}\".format(slave))) for slave in slaves ]))\n\n        # device node is something like /dev/sda1 - we need only the part without /dev/\n        return [ re.match(r'/dev/(\\S+\\d*)', udev_obj.device_node).group(1) ]\n\n    def __learn_irqs(self):\n        disk2irqs = {}\n\n        for devices in list(self.__dir2disks.values()) + [ self.args.devs ]:\n            for device in devices:\n                # There could be that some of the given directories are on the same disk.\n                # There is no need to rediscover IRQs of the disk we've already handled.\n                if device in disk2irqs.keys():\n                    continue\n\n                udev_obj = pyudev.Devices.from_device_file(self.__pyudev_ctx, \"/dev/{}\".format(device))\n                dev_sys_path = udev_obj.sys_path\n\n                # If the device is iSCSI disk, we should skip discovering IRQs\n                # since it's virtual device\n                if re.search(r'^/sys/devices/platform/host[0-9]+/session[0-9]+', udev_obj.sys_path):\n                    disk2irqs[device] = []\n                    continue\n\n                # If the device is a virtual NVMe device it's sys file name goes as follows:\n                # /sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1\n                #\n                # and then there is this symlink:\n                # /sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1/device/nvme0 -> ../../../pci0000:85/0000:85:01.0/0000:87:00.0/nvme/nvme0\n                #\n                # So, the \"main device\" is a \"nvme\\d+\" prefix of the actual device name.\n                if re.search(r'virtual', udev_obj.sys_path):\n                    m = re.match(r'(nvme\\d+)\\S*', device)\n                    if m:\n                        dev_sys_path = \"{}/device/{}\".format(udev_obj.sys_path, m.group(1))\n\n                split_sys_path = list(pathlib.PurePath(pathlib.Path(dev_sys_path).resolve()).parts)\n\n                # first part is always /sys/devices/pciXXX ...\n                controller_path_parts = split_sys_path[0:4]\n\n                # ...then there is a chain of one or more \"domain:bus:device.function\" followed by the storage device enumeration crap\n                # e.g. /sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sda/sda3 or\n                #      /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/host6/target6:2:0/6:2:0:0/block/sda/sda1\n                # We want only the path till the last BDF including - it contains the IRQs information.\n\n                patt = re.compile(r\"^[0-9ABCDEFabcdef]{4}:[0-9ABCDEFabcdef]{2}:[0-9ABCDEFabcdef]{2}\\.[0-9ABCDEFabcdef]$\")\n                for split_sys_path_branch in split_sys_path[4:]:\n                    if patt.search(split_sys_path_branch):\n                        controller_path_parts.append(split_sys_path_branch)\n                    else:\n                        break\n\n                controler_path_str = functools.reduce(lambda x, y : os.path.join(x, y), controller_path_parts)\n                disk2irqs[device] = learn_all_irqs_one(controler_path_str, self.__irqs2procline, 'blkif')\n\n        return disk2irqs\n\n    def __get_feature_file(self, dev_node, path_creator):\n        \"\"\"\n        Find the closest ancestor with the given feature and return its ('feature file', 'device node') tuple.\n\n        If there isn't such an ancestor - return (None, None) tuple.\n\n        :param dev_node Device node file name, e.g. /dev/sda1\n        :param path_creator A functor that creates a feature file name given a device system file name\n        \"\"\"\n        # Sanity check\n        if dev_node is None or path_creator is None:\n            return None, None\n\n        udev = pyudev.Devices.from_device_file(pyudev.Context(), dev_node)\n        feature_file = path_creator(udev.sys_path)\n\n        if os.path.exists(feature_file):\n            return feature_file, dev_node\n        elif udev.parent is not None:\n            return self.__get_feature_file(udev.parent.device_node, path_creator)\n        else:\n            return None, None\n\n    def __tune_one_feature(self, dev_node, path_creator, value, tuned_devs_set):\n        \"\"\"\n        Find the closest ancestor that has the given feature, configure it and\n        return True.\n\n        If there isn't such ancestor - return False.\n\n        :param dev_node Device node file name, e.g. /dev/sda1\n        :param path_creator A functor that creates a feature file name given a device system file name\n        \"\"\"\n        feature_file, feature_node = self.__get_feature_file(dev_node, path_creator)\n\n        if feature_file is None:\n            return False\n\n        if feature_node not in tuned_devs_set:\n            fwriteln_and_log(feature_file, value)\n            tuned_devs_set.add(feature_node)\n\n        return True\n\n    def __tune_io_scheduler(self, dev_node, io_scheduler):\n        return self.__tune_one_feature(dev_node, lambda p : os.path.join(p, 'queue', 'scheduler'), io_scheduler, self.__io_scheduler_tuned_devs)\n\n    def __tune_nomerges(self, dev_node):\n        return self.__tune_one_feature(dev_node, lambda p : os.path.join(p, 'queue', 'nomerges'), self.__nomerges, self.__nomerges_tuned_devs)\n\n    # If write cache configuration is not requested - return True immediately\n    def __tune_write_back_cache(self, dev_node):\n        if self.__write_cache_config is None:\n            return True\n\n        return self.__tune_one_feature(dev_node, lambda p : os.path.join(p, 'queue', 'write_cache'), self.__write_cache_config, self.__write_back_cache_tuned_devs)\n\n    def __get_io_scheduler(self, dev_node):\n        \"\"\"\n        Return a supported scheduler that is also present in the required schedulers list (__io_schedulers).\n\n        If there isn't such a supported scheduler - return None.\n        \"\"\"\n        feature_file, feature_node = self.__get_feature_file(dev_node, lambda p : os.path.join(p, 'queue', 'scheduler'))\n\n        lines = readlines(feature_file)\n        if not lines:\n            return None\n\n        # Supported schedulers appear in the config file as a single line as follows:\n        #\n        # sched1 [sched2] sched3\n        #\n        # ...with one or more schedulers where currently selected scheduler is the one in brackets.\n        #\n        # Return the scheduler with the highest priority among those that are supported for the current device.\n        supported_schedulers = frozenset([scheduler.lstrip(\"[\").rstrip(\"]\").rstrip(\"\\n\") for scheduler in lines[0].split(\" \")])\n        return next((scheduler for scheduler in self.__io_schedulers if scheduler in supported_schedulers), None)\n\n    def __tune_disk(self, device):\n        dev_node = \"/dev/{}\".format(device)\n        io_scheduler = self.__get_io_scheduler(dev_node)\n\n        if not io_scheduler:\n            perftune_print(\"Not setting I/O Scheduler for {} - required schedulers ({}) are not supported\".format(device, list(self.__io_schedulers)))\n        elif not self.__tune_io_scheduler(dev_node, io_scheduler):\n            perftune_print(\"Not setting I/O Scheduler for {} - feature not present\".format(device))\n\n        if not self.__tune_nomerges(dev_node):\n            perftune_print(\"Not setting 'nomerges' for {} - feature not present\".format(device))\n\n        if not self.__tune_write_back_cache(dev_node):\n                perftune_print(\"Not setting 'write_cache' for {} - feature not present\".format(device))\n\n    def __tune_disks(self, disks):\n        for disk in disks:\n            self.__tune_disk(disk)\n\n################################################################################\nclass TuneModes(enum.Enum):\n    disks = 0\n    net = 1\n    system = 2\n\n    @staticmethod\n    def names():\n        return list(TuneModes.__members__.keys())\n\n# Seastar defaults to allocating 93% of physical memory. The kernel's default allocation for TCP is ~9%. This adds up\n# to 102%. Reduce the TCP allocation to 3% to avoid OOM.\ndefault_tcp_mem_fraction = 0.03\n\nargp = argparse.ArgumentParser(description = 'Configure various system parameters in order to improve the seastar application performance.', formatter_class=argparse.RawDescriptionHelpFormatter,\n                               epilog=\n'''\nThis script will:\n\n    - Ban relevant IRQs from being moved by irqbalance.\n    - Configure various system parameters in /proc/sys.\n    - Distribute the IRQs (using SMP affinity configuration) among CPUs according to the configuration mode (see below)\n      or an 'irq_cpu_mask' value.\n\nAs a result some of the CPUs may be destined to only handle the IRQs and taken out of the CPU set\nthat should be used to run the seastar application (\"compute CPU set\").\n\nModes description:\n\n sq - set all IRQs of a given NIC to CPU0 and configure RPS\n      to spreads NAPIs' handling between other CPUs.\n\n sq_split - divide all IRQs of a given NIC between CPU0 and its HT siblings and configure RPS\n      to spreads NAPIs' handling between other CPUs.\n\n mq - distribute NIC's IRQs among all CPUs instead of binding\n      them all to CPU0. In this mode RPS is always enabled to\n      spreads NAPIs' handling between all CPUs.\n\n If there isn't any mode given script will use a default mode:\n    - If number of CPU cores is greater than 16, allocate a single IRQ CPU core for each 16 CPU cores in 'cpu_mask'.\n      IRQ cores are going to be allocated evenly on available NUMA nodes according to 'cpu_mask' value.\n    - If number of physical CPU cores per Rx HW queue is greater than 4 and less than 16 - use the 'sq-split' mode.\n    - Otherwise, if number of hyper-threads per Rx HW queue is greater than 4 - use the 'sq' mode.\n    - Otherwise use the 'mq' mode.\n\nDefault values:\n\n --nic NIC       - default: eth0\n --cpu-mask MASK - default: all available cores mask\n --tune-clock    - default: false\n''')\nargp.add_argument('--mode', choices=PerfTunerBase.SupportedModes.names(), help='configuration mode (deprecated, use --irq-cpu-mask instead)')\nargp.add_argument('--nic', action='append', help='network interface name(s), by default uses \\'eth0\\' (may appear more than once)', dest='nics', default=[])\nargp.add_argument('--tune-clock', action='store_true', help='Force tuning of the system clocksource')\nargp.add_argument('--get-cpu-mask', action='store_true', help=\"print the CPU mask to be used for compute\")\nargp.add_argument('--get-cpu-mask-quiet', action='store_true', help=\"print the CPU mask to be used for compute, print the zero CPU set if that's what it turns out to be\")\nargp.add_argument('--get-irq-cpu-mask', action='store_true', help=\"print the CPU mask to be used for IRQs binding\")\nargp.add_argument('--verbose', action='store_true', help=\"be more verbose about operations and their result\")\nargp.add_argument('--tune', choices=TuneModes.names(), help=\"components to configure (may be given more than once)\", action='append', default=[])\nargp.add_argument('--cpu-mask', help=\"mask of cores to use, by default use all available cores\", metavar='MASK')\nargp.add_argument('--irq-cpu-mask', help=\"mask of cores to use for IRQs binding\", metavar='MASK')\nargp.add_argument('--dir', help=\"directory to optimize (may appear more than once)\", action='append', dest='dirs', default=[])\nargp.add_argument('--dev', help=\"device to optimize (may appear more than once), e.g. sda1\", action='append', dest='devs', default=[])\nargp.add_argument('--options-file', help=\"configuration YAML file\")\nargp.add_argument('--dump-options-file', action='store_true', help=\"Print the configuration YAML file containing the current configuration\")\nargp.add_argument('--dry-run', action='store_true', help=\"Don't take any action, just recommend what to do.\")\nargp.add_argument('--write-back-cache', help=\"Enable/Disable \\'write back\\' write cache mode.\", dest=\"set_write_back\")\nargp.add_argument('--arfs', help=\"Enable/Disable aRFS\", dest=\"enable_arfs\")\nargp.add_argument('--num-rx-queues', help=\"Set a given number of Rx queues\", type=int)\nargp.add_argument('--irq-core-auto-detection-ratio', help=\"Use a given ratio for IRQ mask auto-detection. For \"\n                                                          \"instance, if 8 is given and auto-detection is requested, a \"\n                                                          \"single IRQ CPU core is going to be allocated for every 8 \"\n                                                          \"CPU cores out of available according to a 'cpu_mask' value.\"\n                                                          \"Default is 16\",\n                  type=int, default=16, dest='cores_per_irq_core')\nargp.add_argument('--tcp-mem-fraction', default=default_tcp_mem_fraction, type=float, help=\"Fraction of total memory to allocate for TCP buffers\")\n\ndef parse_cpu_mask_from_yaml(y, field_name, fname):\n    hex_32bit_pattern='0x[0-9a-fA-F]{1,8}'\n    mask_pattern = re.compile('^{}((,({})?)*,{})*$'.format(hex_32bit_pattern, hex_32bit_pattern, hex_32bit_pattern))\n\n    if mask_pattern.match(str(y[field_name])):\n        return y[field_name]\n    else:\n        raise Exception(\"Bad '{}' value in {}: {}\".format(field_name, fname, str(y[field_name])))\n\ndef extend_and_unique(orig_list, iterable):\n    \"\"\"\n    Extend items to a list, and make the list items unique\n    \"\"\"\n    assert(isinstance(orig_list, list))\n    assert(isinstance(iterable, list))\n    orig_list.extend(iterable)\n    return list(set(orig_list))\n\ndef parse_tri_state_arg(value, arg_name):\n    try:\n        if value is not None:\n            return distutils.util.strtobool(value)\n        else:\n            return None\n    except:\n        sys.exit(\"Invalid {} value: should be boolean but given: {}\".format(arg_name, value))\n\ndef parse_options_file(prog_args):\n    if not prog_args.options_file:\n        return\n\n    y = yaml.safe_load(open(prog_args.options_file))\n    if y is None:\n        return\n\n    if 'mode' in y and not prog_args.mode:\n        if not y['mode'] in PerfTunerBase.SupportedModes.names():\n            raise Exception(\"Bad 'mode' value in {}: {}\".format(prog_args.options_file, y['mode']))\n        prog_args.mode = y['mode']\n\n    if 'nic' in y:\n        # Multiple nics was supported by commit a2fc9d72c31b97840bc75ae49dbd6f4b6d394e25\n        # `nic' option dumped to config file will be list after this change, but the `nic'\n        # option in old config file is still string, which was generated before this change.\n        # So here convert the string option to list.\n        if not isinstance(y['nic'], list):\n            y['nic'] = [y['nic']]\n        prog_args.nics = extend_and_unique(prog_args.nics, y['nic'])\n\n    if 'tune_clock' in y and not prog_args.tune_clock:\n        prog_args.tune_clock= y['tune_clock']\n\n    if 'tune' in y:\n        if set(y['tune']) <= set(TuneModes.names()):\n            prog_args.tune = extend_and_unique(prog_args.tune, y['tune'])\n        else:\n            raise Exception(\"Bad 'tune' value in {}: {}\".format(prog_args.options_file, y['tune']))\n\n    if 'cpu_mask' in y and not prog_args.cpu_mask:\n        prog_args.cpu_mask = parse_cpu_mask_from_yaml(y, 'cpu_mask', prog_args.options_file)\n\n    if 'irq_cpu_mask' in y and not prog_args.irq_cpu_mask:\n        prog_args.irq_cpu_mask = parse_cpu_mask_from_yaml(y, 'irq_cpu_mask', prog_args.options_file)\n\n    if 'dir' in y:\n        prog_args.dirs = extend_and_unique(prog_args.dirs, y['dir'])\n\n    if 'dev' in y:\n        prog_args.devs = extend_and_unique(prog_args.devs, y['dev'])\n\n    if 'write_back_cache' in y:\n        prog_args.set_write_back = distutils.util.strtobool(\"{}\".format(y['write_back_cache']))\n\n    if 'arfs' in y:\n        prog_args.enable_arfs = distutils.util.strtobool(\"{}\".format(y['arfs']))\n\n    if 'num_rx_queues' in y:\n        prog_args.num_rx_queues = int(y['num_rx_queues'])\n\n    # prog_options['irq_core_auto_detection_ratio'] = prog_args.cores_per_irq_core\n    if 'irq_core_auto_detection_ratio' in y:\n        prog_args.cores_per_irq_core = int(y['irq_core_auto_detection_ratio'])\n\ndef dump_config(prog_args):\n    prog_options = {}\n\n    if prog_args.mode:\n        assert prog_args.cpu_mask, \"cpu_mask has to always be set. Something is terribly wrong (a bug in perftune.py?)\"\n        mode = PerfTunerBase.SupportedModes[prog_args.mode]\n        prog_options['irq_cpu_mask'] = PerfTunerBase.irqs_cpu_mask_for_mode(mode, prog_args.cpu_mask)\n\n    if prog_args.nics:\n        prog_options['nic'] = list(set(prog_args.nics))\n\n    if prog_args.tune_clock:\n        prog_options['tune_clock'] = prog_args.tune_clock\n\n    if prog_args.tune:\n        prog_options['tune'] = list(set(prog_args.tune))\n\n    if prog_args.cpu_mask:\n        prog_options['cpu_mask'] = prog_args.cpu_mask\n\n    if prog_args.irq_cpu_mask:\n        prog_options['irq_cpu_mask'] = prog_args.irq_cpu_mask\n\n    if prog_args.dirs:\n        prog_options['dir'] = list(set(prog_args.dirs))\n\n    if prog_args.devs:\n        prog_options['dev'] = list(set(prog_args.devs))\n\n    if prog_args.set_write_back is not None:\n        prog_options['write_back_cache'] = prog_args.set_write_back\n\n    if prog_args.enable_arfs is not None:\n        prog_options['arfs'] = prog_args.enable_arfs\n\n    if prog_args.num_rx_queues is not None:\n        prog_options['num_rx_queues'] = f\"{prog_args.num_rx_queues}\"\n\n    prog_options['irq_core_auto_detection_ratio'] = prog_args.cores_per_irq_core\n\n    perftune_print(yaml.dump(prog_options, default_flow_style=False))\n################################################################################\n\nargs = argp.parse_args()\n\n# Sanity check\nargs.set_write_back = parse_tri_state_arg(args.set_write_back, \"--write-back-cache/write_back_cache\")\nargs.enable_arfs = parse_tri_state_arg(args.enable_arfs, \"--arfs/arfs\")\n\ndry_run_mode = args.dry_run\nparse_options_file(args)\n\n# if nothing needs to be configured - quit\nif not args.tune:\n    sys.exit(\"ERROR: At least one tune mode MUST be given.\")\n\n# The must be either 'mode' or an explicit 'irq_cpu_mask' given - not both\nif args.mode and args.irq_cpu_mask:\n    sys.exit(\"ERROR: Provide either tune mode or IRQs CPU mask - not both.\")\n\n# Sanity check\nif args.cores_per_irq_core < PerfTunerBase.min_cores_per_irq_core():\n    sys.exit(f\"ERROR: irq_core_auto_detection_ratio value must be greater or equal than \"\n             f\"{PerfTunerBase.min_cores_per_irq_core()}\")\n\n# set default values #####################\nif not args.nics:\n    args.nics = ['eth0']\n\nif not args.cpu_mask:\n    args.cpu_mask = run_hwloc_calc(['all'])\n##########################################\n\n# Sanity: irq_cpu_mask should be a subset of cpu_mask\nif args.irq_cpu_mask and run_hwloc_calc([args.cpu_mask]) != run_hwloc_calc([args.cpu_mask, args.irq_cpu_mask]):\n    sys.exit(\"ERROR: IRQ CPU mask({}) must be a subset of CPU mask({})\".format(args.irq_cpu_mask, args.cpu_mask))\n\nif args.dump_options_file:\n    dump_config(args)\n    sys.exit(0)\n\ntry:\n    tuners = []\n\n    if TuneModes.disks.name in args.tune:\n        tuners.append(DiskPerfTuner(args))\n\n    if TuneModes.net.name in args.tune:\n        tuners.append(NetPerfTuner(args))\n\n    if TuneModes.system.name in args.tune:\n        tuners.append(SystemPerfTuner(args))\n\n    if args.get_cpu_mask or args.get_cpu_mask_quiet:\n        # Print the compute mask from the first tuner - it's going to be the same in all of them\n        perftune_print(tuners[0].compute_cpu_mask)\n    elif args.get_irq_cpu_mask:\n        perftune_print(tuners[0].irqs_cpu_mask)\n    else:\n        # Tune the system\n        restart_irqbalance(itertools.chain.from_iterable([ tuner.irqs for tuner in tuners ]))\n\n        for tuner in tuners:\n            tuner.tune()\nexcept PerfTunerBase.CPUMaskIsZeroException as e:\n    # Print a zero CPU set if --get-cpu-mask-quiet was requested.\n    if args.get_cpu_mask_quiet:\n        perftune_print(\"0x0\")\n    else:\n        sys.exit(\"ERROR: {}. Your system can't be tuned until the issue is fixed.\".format(e))\nexcept PerfTunerBase.InvalidNUMATopologyException as e:\n    print(\"ERROR: {}. Your system can't be tuned until the issue is fixed.\".format(e), file=sys.stderr)\n    # set special exit code to handle InvalidNUMATopologyException from the caller script\n    sys.exit(3)\nexcept Exception as e:\n    sys.exit(\"ERROR: {}. Your system can't be tuned until the issue is fixed.\".format(e))\n\n"
  },
  {
    "path": "scripts/perftune.yaml",
    "content": "# Mode is one of the following values:\n#   - 'mq'\n#   - 'sq'\n#   - 'sq_split'\n#mode: 'sq_split'\n\n# Name of the NIC to tune, e.g. eth7.\n# By default would use 'eth0'.\n#nic: eth7\n\n# If 'true' the script will the CPU mask to be used for compute.\n#get_cpu_mask: true\n\n# Define what to tune: could be any combination of the values from {'net', 'disks'} set.\n#tune:\n#  - net\n#  - disks\n\n# Mask of cores to use, by default use all available cores.\n#cpu_mask: '0x00f,,,0x0,,0x00f'\n\n# Set of directories to optimize.\n#dir:\n#  - /root\n#  - /home\n\n# Set of disk devices to optimize\n#dev:\n#  - /dev/sda2\n#  - /dev/md0\n\n# write_back_cache: false\n\n"
  },
  {
    "path": "scripts/posix_net_conf.sh",
    "content": "#!/bin/bash\n# !\n# !  Usage: posix_net_conf.sh [iface name, eth0 by default] [-mq|-sq] [--cpu-mask] [-h|--help] [--use-cpu-mask <mask>]\n# !\n# !  Ban NIC IRQs from being moved by irqbalance.\n# !\n# !  -sq - set all IRQs of a given NIC to CPU0 and configure RPS\n# !        to spreads NAPIs' handling between other CPUs.\n# !\n# !  -mq - distribute NIC's IRQs among all CPUs instead of binding\n# !        them all to CPU0. In this mode RPS is always enabled to\n# !        spreads NAPIs' handling between all CPUs.\n# !\n# !   --options-file <YAML file> - YAML file with perftune.py options\n# !\n# !  If there isn't any mode given script will use a default mode:\n# !     - If number of physical CPU cores per Rx HW queue is greater than 4 - use the '-sq' mode.\n# !     - Otherwise use the '-mq' mode.\n# !\n# !  --use-cpu-mask <mask> - mask of cores to use, by default use all available cores\n# !\n# !  --cpu-mask - Print out RPS CPU assignments. On MQ NIC, just print all cpus.\n# !\n# !  -h|--help - print this help information\n# !\n# !  Enable XPS, increase the default values of somaxconn and tcp_max_syn_backlog.\n# !\n\nusage()\n{\n    cat $0 | grep ^\"# !\" | cut -d\"!\" -f2-\n}\n\nparse_args()\n{\n    local i\n    local arg\n\n    until [ -z \"$1\" ]\n    do\n        arg=$1\n        case \"$arg\" in\n            \"-mq\")\n                MQ_MODE=\"--mode mq\"\n                ;;\n            \"-sq\")\n                MQ_MODE=\"--mode sq\"\n                ;;\n            \"--cpu-mask\")\n                CPU_MASK=\"--get-cpu-mask\"\n                ;;\n            \"--use-cpu-mask\")\n                CPU_FILTER_MASK=\"--cpu-mask $2\"\n                shift\n                ;;\n            \"--options-file\")\n                OPTIONS_FILE=\"--options-file $2\"\n                shift\n                ;;\n            \"-h\"|\"--help\")\n                usage\n                exit 0\n                ;;\n            *)\n                IFACE=$arg\n                ;;\n            esac\n            shift\n    done\n}\n\nIFACE=\"eth0\"\nMQ_MODE=\"\"\nCPU_FILTER_MASK=\"\"\nCPU_MASK=\"\"\nMY_DIR=`dirname $0`\nOPTIONS_FILE=\"\"\n\nparse_args $@\n\n$MY_DIR/perftune.py --nic $IFACE $MQ_MODE $CPU_FILTER_MASK $CPU_MASK $OPTIONS_FILE --tune net\n"
  },
  {
    "path": "scripts/pyproject.toml",
    "content": "[tool.black]\nline-length = 100\nskip-string-normalization = true\n"
  },
  {
    "path": "scripts/run_with_dpdk.sh",
    "content": "#!/bin/bash\n# !\n# ! Usage: ./prepare_dpdk_env.sh <NIC to use> <number of huge pages per NUMA Node> <command to execute> [command parameters]\n# !\n# ! Prepares the DPDK environment (binds a given NIC to UIO, allocates the required\n# ! number of huge pages) and executes the given command in it.\n# ! After the command terminates the original environment is restored apart from\n# ! huge pages, that remain allocated.\n# !\n\nusage()\n{\n    cat $0 | grep ^\"# !\" | cut -d\"!\" -f2-\n}\n\n#\n#  check_stat_and_exit <error message>\n#\ncheck_stat_and_exit()\n{\n    if [[ $? -ne 0 ]]; then\n        echo $@\n        exit 1\n    fi\n}\n\nrollback()\n{\n    echo \"Binding $NIC($BDF) back to $DRIVER...\"\n    $SCRIPTS_DIR/dpdk_nic_bind.py -u $BDF\n    $SCRIPTS_DIR/dpdk_nic_bind.py -b $DRIVER $BDF\n}\n\ncheck_stat_and_rollback()\n{\n    if [[ $? -ne 0 ]]; then\n        echo $@\n        rollback\n        exit 1\n    fi\n}\n\n# Check number of parameters\nif [[ $# -lt 3 ]]; then\n    usage\n    exit 1\nfi\n\nNIC=$1\nshift\nNUM_HUGE_PAGES_PER_NODE=$1\nshift\nSCRIPTS_DIR=`dirname $0`\n\n\nifconfig $NIC down\ncheck_stat_and_exit \"Failed to shut down $NIC. Is $NIC present? Are your permissions sufficient?\"\n\nDRIVER=`ethtool -i $NIC | grep driver | cut -d\":\" -f2- | tr -d ' '`\nBDF=`ethtool -i $NIC | grep bus-info | cut -d\":\" -f2- |  tr -d ' '`\n\n# command to execute\nCMD=$@\n\necho \"Binding $NIC($BDF) to uio_pci_generic...\"\n$SCRIPTS_DIR/dpdk_nic_bind.py -u $BDF\ncheck_stat_and_exit\n$SCRIPTS_DIR/dpdk_nic_bind.py -b uio_pci_generic $BDF\ncheck_stat_and_rollback\n\necho \"Allocating $NUM_HUGE_PAGES_PER_NODE 2MB huge pages on each NUMA Node:\"\nfor d in /sys/devices/system/node/node? ; do\n    echo $NUM_HUGE_PAGES_PER_NODE > $d/hugepages/hugepages-2048kB/nr_hugepages\n    check_stat_and_rollback\n    cur_node=`basename $d`\n    echo \"...$cur_node done...\"\ndone\n\nmkdir -p /mnt/huge\ncheck_stat_and_rollback\n\ngrep -s '/mnt/huge' /proc/mounts > /dev/null\nif [[ $? -ne 0 ]] ; then\n    echo \"Mounting hugetlbfs at /mnt/huge...\"\n    mount -t hugetlbfs nodev /mnt/huge\n    check_stat_and_rollback\nfi\n\n# Run scylla\necho \"Running: $CMD\"\n$CMD\nret=$?\n\n# Revert the NIC binding\nrollback\n\nexit $ret\n\n"
  },
  {
    "path": "scripts/seastar-addr2line",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n# Copyright (C) 2017 ScyllaDB\n\nimport argparse\nfrom typing import Any, Optional, Sequence, TextIO\nimport unittest\nimport sys\n\nfrom addr2line import BacktraceResolver\n\n\ndef read_backtrace(stdin: TextIO):\n    \"\"\"\n    Read stdin char-by-char and stop when when user pressed Ctrl+D or the\n    Enter twice. Altough reading char-by-char is slow this won't be a\n    problem here as backtraces shouldn't be huge.\n    \"\"\"\n    linefeeds = 0\n    line: list[str] = []\n\n    while True:\n        char = stdin.read(1)\n\n        if char == '\\n':\n            linefeeds += 1\n\n            if len(line) > 0:\n                yield ''.join(line)\n                line = []\n        else:\n            line.append(char)\n            linefeeds = 0\n\n        if char == '' or linefeeds > 1:\n            break\n\n\nTestResult = dict[str, Any]\nMaybeTestResult = Optional[TestResult]\nTestList = Sequence[tuple[str, MaybeTestResult]]\n\n\nclass TestStringMethods(unittest.TestCase):\n\n    def setUp(self):\n        self.parser = BacktraceResolver.BacktraceParser()\n\n    def _test(self, cases: TestList):\n        for line, expected in cases:\n            res = self.parser(line.strip() + '\\n')\n            self.assertEqual(res, expected, f\"failed to parse {line}\")\n\n    def test_separator(self):\n        data = [('---', {'type': BacktraceResolver.BacktraceParser.Type.SEPARATOR})]\n        self._test(data)\n\n    def test_addresses_only(self):\n        data: TestList = [\n            (\n                '0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': None, 'addr': '0x12f34'}],\n                },\n            ),\n            (\n                '0xa1234 0xb4567',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [\n                        {'path': None, 'addr': '0xa1234'},\n                        {'path': None, 'addr': '0xb4567'},\n                    ],\n                },\n            ),\n            (\n                '\t0xa1234 /my/path+0xb4567',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [\n                        {'path': None, 'addr': '0xa1234'},\n                        {'path': '/my/path', 'addr': '0xb4567'},\n                    ],\n                },\n            ),\n            (\n                '/my/path+0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/my/path', 'addr': '0x12f34'}],\n                },\n            ),\n            (\n                ' /my/path+0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/my/path', 'addr': '0x12f34'}],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_without_prefix(self):\n        data: TestList = [\n            (\n                'Some prefix 0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': None, 'addr': '0x12f34'}],\n                },\n            ),\n            (\n                'Some prefix /my/path+0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/my/path', 'addr': '0x12f34'}],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_reactor_stall_reports(self):\n        data: TestList = [\n            (\n                'Reactor stalled on shard 1. Backtrace: 0x12f34',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'Reactor stalled on shard 1. Backtrace:',\n                    'addresses': [{'path': None, 'addr': '0x12f34'}],\n                },\n            ),\n            (\n                'Reactor stalled on shard 1. Backtrace: 0xa1234 0xb5678',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'Reactor stalled on shard 1. Backtrace:',\n                    'addresses': [\n                        {'path': None, 'addr': '0xa1234'},\n                        {'path': None, 'addr': '0xb5678'},\n                    ],\n                },\n            ),\n            (\n                'Reactor stalled on shard 1. Backtrace: /my/path+0xabcd',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'Reactor stalled on shard 1. Backtrace:',\n                    'addresses': [{'path': '/my/path', 'addr': '0xabcd'}],\n                },\n            ),\n            (\n                'Apr 28 11:42:58 ip-172-31-2-154.ec2.internal scylla[10612]: Reactor stalled for 260 ms on shard 20.',\n                None,\n            ),\n            ('Apr 28 11:42:58 ip-172-31-2-154.ec2.internal scylla[10612]: Backtrace:', None),\n            (\n                'Apr 28 11:42:58 ip-172-31-2-154.ec2.internal scylla[10612]: 0x0000000003163dc2',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'Apr 28 11:42:58 ip-172-31-2-154.ec2.internal scylla[10612]:',\n                    'addresses': [{'path': None, 'addr': '0x0000000003163dc2'}],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_boost_failure(self):\n        data: TestList = [\n            (\n                'Expected partition_end(), but got end of stream, at 0xa1234 0xb5678 /my/path+0xabcd',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'Expected partition_end(), but got end of stream, at',\n                    'addresses': [\n                        {'path': None, 'addr': '0xa1234'},\n                        {'path': None, 'addr': '0xb5678'},\n                        {'path': '/my/path', 'addr': '0xabcd'},\n                    ],\n                },\n            )\n        ]\n        self._test(data)\n\n    def test_asan_failure(self):\n        data: TestList = [\n            (\n                '==16118==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700019c710 at pc 0x000014d24643 bp 0x7ffc51f72220 sp 0x7ffc51f72218',\n                None,\n            ),\n            ('READ of size 8 at 0x60700019c710 thread T0', None),\n            (\n                '#0 0x14d24642  (/jenkins/workspace/scylla-enterprise/dtest-debug/scylla/.ccm/scylla-repository/1a5173bd45d01697d98ba2a7645f5d86afb2d0be/scylla/libexec/scylla+0x14d24642)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [\n                        {\n                            'path': '/jenkins/workspace/scylla-enterprise/dtest-debug/scylla/.ccm/scylla-repository/1a5173bd45d01697d98ba2a7645f5d86afb2d0be/scylla/libexec/scylla',\n                            'addr': '0x14d24642',\n                        }\n                    ],\n                },\n            ),\n            (\n                '    #1 0xd8d910f  (/home/myhome/.dtest/dtest-84j9064d/test/node1/bin/scylla+0xd8d910f) (BuildId: 05a1d3d58d2b07e526decdad717e71a4590be2e0)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [\n                        {\n                            'path': '/home/myhome/.dtest/dtest-84j9064d/test/node1/bin/scylla',\n                            'addr': '0xd8d910f',\n                        }\n                    ],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_thrown_exception(self):\n        data: TestList = [\n            (\n                'seastar::internal::backtraced<std::runtime_error> (throw_with_backtrace_exception_logging Backtrace: 0x42bc95 /lib64/libc.so.6+0x281e1 0x412cfd)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'seastar::internal::backtraced<std::runtime_error> (throw_with_backtrace_exception_logging Backtrace:',\n                    'addresses': [\n                        {'path': None, 'addr': '0x42bc95'},\n                        {'path': '/lib64/libc.so.6', 'addr': '0x281e1'},\n                        {'path': None, 'addr': '0x412cfd'},\n                    ],\n                },\n            ),\n            (\n                'seastar::nested_exception: seastar::internal::backtraced<std::runtime_error> (inner Backtrace: 0x42bc95 /lib64/libc.so.6+0x281e1 0x412cfd) (while cleaning up after unknown_obj)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'seastar::nested_exception: seastar::internal::backtraced<std::runtime_error> (inner Backtrace:',\n                    'addresses': [\n                        {'path': None, 'addr': '0x42bc95'},\n                        {'path': '/lib64/libc.so.6', 'addr': '0x281e1'},\n                        {'path': None, 'addr': '0x412cfd'},\n                    ],\n                },\n            ),\n            (\n                'seastar::nested_exception: seastar::internal::backtraced<std::runtime_error> (inner Backtrace: 0x42bc95 /lib64/libc.so.6+0x281e1 0x412cfd) '\n                '(while cleaning up after seastar::internal::backtraced<std::runtime_error> (outer Backtrace: 0x1234 /lib64/libc.so.6+0x5678 0xabcd))',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'seastar::nested_exception: seastar::internal::backtraced<std::runtime_error> (inner Backtrace: 0x42bc95 /lib64/libc.so.6+0x281e1 0x412cfd)'\n                    ' (while cleaning up after seastar::internal::backtraced<std::runtime_error> (outer Backtrace:',\n                    'addresses': [\n                        {'path': None, 'addr': '0x1234'},\n                        {'path': '/lib64/libc.so.6', 'addr': '0x5678'},\n                        {'path': None, 'addr': '0xabcd'},\n                    ],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_assertion_failure(self):\n        data: TestList = [\n            (\n                \"2022-04-15T06:19:24+00:00 gemini-1tb-10h-master-db-node-c0c7fc43-4 !    INFO | \"\n                \"scylla: sstables/consumer.hh:610: future<> data_consumer::continuous_data_consumer\"\n                \"<sstables::index_consume_entry_context<sstables::index_consumer>>::fast_forward_to\"\n                \"(size_t, size_t) [StateProcessor = sstables::index_consume_entry_context\"\n                \"<sstables::index_consumer>]: Assertion `begin >= _stream_position.position' failed.\",\n                None,\n            ),\n            (\n                '2022-04-15T06:19:24+00:00 gemini-1tb-10h-master-db-node-c0c7fc43-4 !    INFO |   Backtrace:',\n                None,\n            ),\n            (\n                '2022-04-15T06:19:24+00:00 gemini-1tb-10h-master-db-node-c0c7fc43-4 !    INFO |   0x199d312',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': None, 'addr': '0x199d312'}],\n                },\n            ),\n            (\n                '2022-04-15T06:19:24+00:00 gemini-1tb-10h-master-db-node-c0c7fc43-4 !    INFO |   (throw_with_backtrace_exception_logging Backtrace: 0x42bc95 /lib64/libc.so.6+0x281e1 0x412cfd)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': '2022-04-15T06:19:24+00:00 gemini-1tb-10h-master-db-node-c0c7fc43-4 !    INFO |   (throw_with_backtrace_exception_logging Backtrace:',\n                    'addresses': [\n                        {'path': None, 'addr': '0x42bc95'},\n                        {'path': '/lib64/libc.so.6', 'addr': '0x281e1'},\n                        {'path': None, 'addr': '0x412cfd'},\n                    ],\n                },\n            ),\n        ]\n        self._test(data)\n\n    def test_segfault_failure(self):\n        data: TestList = [\n            ('[2022-04-19T23:09:28.311Z] Segmentation fault on shard 1.', None),\n            ('[2022-04-19T23:09:28.311Z] Backtrace:', None),\n            (\n                '[2022-04-19T23:09:28.311Z]   0x461bbb8',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': None, 'addr': '0x461bbb8'}],\n                },\n            ),\n            (\n                '[2022-04-19T23:09:28.311Z]   /lib64/libpthread.so.0+0x92a4',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/lib64/libpthread.so.0', 'addr': '0x92a4'}],\n                },\n            ),\n            (\n                '#0 0x19c01681 (/path/to/scylla+0xdeadbeef)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/path/to/scylla', 'addr': '0x19c01681'}],\n                },\n            ),\n            (\n                '#1 0x00007fd2dab4f950 abort (libc.so.6 + 0x26950)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': 'libc.so.6', 'addr': '0x00007fd2dab4f950'}],\n                },\n            ),\n            (\n                '#2 0x00000000015c4cd3  n/a (/path/to/scylla + 0x15c4cd3)',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': None,\n                    'addresses': [{'path': '/path/to/scylla', 'addr': '0x00000000015c4cd3'}],\n                },\n            ),\n            ('kernel callstack: ', None),\n            (\n                'kernel callstack: 0xffffffffffffff80',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'kernel callstack: ',\n                    'addresses': [{'path': '<kernel>', 'addr': '0xffffffffffffff80'}],\n                },\n            ),\n            (\n                'kernel callstack: 0xffffffffffffff80 0xffffffffaf15ccca',\n                {\n                    'type': BacktraceResolver.BacktraceParser.Type.ADDRESS,\n                    'prefix': 'kernel callstack: ',\n                    'addresses': [\n                        {'path': '<kernel>', 'addr': '0xffffffffffffff80'},\n                        {'path': '<kernel>', 'addr': '0xffffffffaf15ccca'},\n                    ],\n                },\n            ),\n        ]\n        self._test(data)\n\n\ndef main():\n    description = 'Massage and pass addresses to the real addr2line for symbol lookup.'\n    epilog = '''\nThere are three operational modes:\n  1) If -f is specified input will be read from FILE\n  2) If -f is omitted and there are ADDRESS args they will be read as input\n  3) If -f is omitted and there are no ADDRESS args input will be read from stdin\n'''\n\n    cmdline_parser = argparse.ArgumentParser(\n        description=description,\n        epilog=epilog,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n    )\n\n    cmdline_parser.add_argument(\n        '-e',\n        '--executable',\n        type=str,\n        required=True,\n        metavar='EXECUTABLE',\n        dest='executable',\n        help='The executable where the addresses originate from',\n    )\n\n    cmdline_parser.add_argument(\n        '-f',\n        '--file',\n        type=str,\n        required=False,\n        metavar='FILE',\n        dest='file',\n        help='The file containing the addresses',\n    )\n\n    cmdline_parser.add_argument(\n        '-b',\n        '--before',\n        type=int,\n        metavar='BEFORE',\n        default=1,\n        help='Non-backtrace lines to print before resolved backtraces for context.'\n        ' Set to 0 to print only resolved backtraces.'\n        ' Set to -1 to print all non-backtrace lines. Default is 1.',\n    )\n\n    cmdline_parser.add_argument(\n        '-m',\n        '--match',\n        type=str,\n        metavar='MATCH',\n        help='Only resolve backtraces whose non-backtrace lines match the regular-expression.'\n        ' The amount of non-backtrace lines considered can be set with --before.'\n        ' By default no matching is performed.',\n    )\n\n    cmdline_parser.add_argument(\n        '-a',\n        '--addr2line',\n        type=str,\n        metavar='CMD_PATH',\n        default='llvm-addr2line',\n        help='The path or name of the addr2line command, which should behave as and '\n        'accept the same options as binutils addr2line or llvm-addr2line (the default).',\n    )\n\n    cmdline_parser.add_argument(\n        '-v',\n        '--verbose',\n        action='store_true',\n        default=False,\n        help='Make resolved backtraces verbose, prepend to each line the module'\n        ' it originates from, as well as the address being resolved',\n    )\n\n    cmdline_parser.add_argument(\n        '-d', '--debug', action='store_true', help='Emit debug logging to stderr.'\n    )\n\n    cmdline_parser.add_argument(\n        '--timing', action='store_true', help='Emit timing information to stderr.'\n    )\n\n    cmdline_parser.add_argument(\n        '-t', '--test', action='store_true', default=False, help='Self-test'\n    )\n\n    cmdline_parser.add_argument(\n        'addresses', type=str, metavar='ADDRESS', nargs='*', help='Addresses to parse'\n    )\n\n    cmdline_parser.add_argument(\n        '--kallsyms',\n        type=str,\n        metavar='KALLSYMS',\n        default='/proc/kallsyms',\n        help='Alternative path for kallsyms file to resolve kernel addresses',\n    )\n\n    args = cmdline_parser.parse_args()\n\n    if args.addresses and args.file:\n        print(\"Cannot use both -f and ADDRESS\")\n        cmdline_parser.print_help()\n\n    if args.file:\n        lines = open(args.file, 'r')\n    elif args.addresses:\n        lines = args.addresses\n    else:\n        if sys.stdin.isatty():\n            lines = list(read_backtrace(sys.stdin))\n        else:\n            lines = sys.stdin\n\n    with BacktraceResolver(\n        executable=args.executable,\n        kallsyms=args.kallsyms,\n        before_lines=args.before,\n        context_re=args.match,\n        verbose=args.verbose,\n        cmd_path=args.addr2line,\n        debug=args.debug,\n        timing=args.timing,\n    ) as resolve:\n        resolve_start = resolve.timing_now()\n        for line in lines:\n            resolve(line.strip() + '\\n')\n        resolve.timing_print_from_start(resolve_start, 'full resolve loop')\n        resolve.print_resolve_time()\n\n\nif __name__ == '__main__':\n    cmdline_parser = argparse.ArgumentParser(add_help=False)\n    cmdline_parser.add_argument(\n        '-t', '--test', action='store_true', default=False, help='Self-test'\n    )\n    args, unrecognized = cmdline_parser.parse_known_args()\n\n    if args.test:\n        unittest.main(argv=[sys.argv[0]] + unrecognized)\n    else:\n        main()\n"
  },
  {
    "path": "scripts/seastar-cpu-map.sh",
    "content": "#!/bin/bash\n# !\n# ! Usage: ./seastar-cpu-map.sh -p <process_PID> -n <process_Name> -s (optional) <shard>\n# !\n# ! List CPU affinity for a particular running process\n# !  providing a map of threads -> shard, for any seastar apps.\n# ! Ex.: ./seastar-cpu-map.sh -n scylla\n# !      ./seastar-cpu-map.sh -n scylla -s 0\n# !      ./seastar-cpu-map.sh -p 34\n# !      ./seastar-cpu-map.sh -p 32 -s 12\n\nusage() {\n    cat $0 | grep ^\"# !\" | cut -d\"!\" -f2-\n}\n\nwhile getopts 'p:n:s:' option; do\n  case \"$option\" in\n    p) PID=$OPTARG\n        ;;\n    n) PID=`pgrep --newest --exact $OPTARG`\n        ;;\n    s) SHARD=$OPTARG\n        ;;\n    :) printf \"missing argument for -%s\\n\" \"$OPTARG\" >&2\n       usage >&2\n       exit 1\n       ;;\n    \\?) printf \"illegal option: -%s\\n\" \"$OPTARG\" >&2\n       usage >&2\n       exit 1\n       ;;\n  esac\ndone\n\nif [ $# -eq 0 ]; then usage >&2; exit 1; fi\n\nif [ -e \"/proc/$PID/task\" ]; then\n  # get list of threads for given PID\n  THREADS=`ls /proc/$PID/task`\n  for i in $THREADS; do\n    # get shards from threads\n    # there were three options here to get the shard number:\n    # reactor-xx, syscall-xx and timer-xx\n    # syscall was preferred because reactor as a special case (reactor-0 is called scylla)\n    SYSCALL=`grep syscall /proc/$i/comm | cut -d\"-\" -f2`\n    if [ -n \"$SYSCALL\" ] && [ \"$SYSCALL\" = \"$SHARD\" ]; then\n      echo -e \"shard: $SYSCALL, cpu:$(taskset -c -p $i | cut -d\":\" -f2)\"\n    elif [ -n \"$SYSCALL\" ] && [ -z \"$SHARD\" ]; then\n      echo -e \"shard: $SYSCALL, cpu:$(taskset -c -p $i | cut -d\":\" -f2)\"\n    fi\n  done\nelse\n  echo \"Process does not exist\"\nfi\n"
  },
  {
    "path": "scripts/seastar-json2code.py",
    "content": "#!/usr/bin/env python3\n\n# C++ Code generation utility from Swagger definitions.\n# This utility support Both the swagger 1.2 format\n#    https://github.com/OAI/OpenAPI-Specification/blob/master/versions/1.2.md\n# And the 2.0 format\n#    https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md\n#\n# Swagger 2.0 is not only different in its structure (apis have moved, and\n# models are now under definitions) It also moved from multiple file structure\n# to a single file.\n# To keep the multiple file support, each group of APIs will be placed in a single file\n# Each group can have a .def.json file with its definitions (What used to be models)\n# Because the APIs in definitions are snippets, they are not legal json objects\n# and need to be formated as such so that a json parser would work.\n\nimport json\nimport sys\nimport re\nimport glob\nimport argparse\nimport os\nimport textwrap\nfrom string import Template\n\nparser = argparse.ArgumentParser(description=\"\"\"Generate C++ class for json\nhandling from swagger definition\"\"\")\n\nparser.add_argument('--outdir', help='the output directory', default='autogen')\nparser.add_argument('-o', help='Output file', default='')\nparser.add_argument('-f', help='input file', default='api-java.json')\nparser.add_argument('-ns', help=\"\"\"namespace when set struct will be created\nunder the namespace\"\"\", default='')\nparser.add_argument('-jsoninc', help='relative path to the jsaon include',\n                    default='json/')\nparser.add_argument('-jsonns', help='set the json namespace', default='json')\nparser.add_argument('-indir', help=\"\"\"when set all json file in the given\ndirectory will be parsed, do not use with -f\"\"\", default='')\nparser.add_argument('-debug', help='debug level 0 -quite,1-error,2-verbose',\n                    default='1', type=int)\nparser.add_argument('-combined', help='set the name of the combined file',\n                    default='autogen/pathautogen.ee')\nparser.add_argument('--create-cc', dest='create_cc', action='store_true', default=False,\n                    help='Put global variables in a .cc file')\nconfig = parser.parse_args()\n\n\nvalid_vars = {'string': 'sstring', 'int': 'int', 'integer': 'int', 'double': 'double',\n             'float': 'float', 'long': 'long', 'boolean': 'bool', 'char': 'char',\n             'datetime': 'json::date_time'}\n\ncurrent_file = ''\n\nspacing = \"    \"\ndef getitem(d, key, name):\n    item = d.get(key)\n    if item is None:\n        raise Exception(f\"'{key}' not found in {name}\")\n    return item\n\ndef fprint(f, *args):\n    for arg in args:\n        f.write(arg)\n\ndef fprintln(f, *args):\n    for arg in args:\n        f.write(arg)\n    f.write('\\n')\n\n\ndef open_namespace(f, ns=config.ns):\n    fprintln(f, \"namespace \", ns , ' {\\n')\n\n\ndef close_namespace(f):\n    fprintln(f, '}')\n\n\ndef add_include(f, includes):\n    for include in includes:\n        fprintln(f, '#include ', include)\n    fprintln(f, \"\")\n\ndef trace_verbose(*params):\n    if config.debug > 1:\n        print(''.join(params))\n\n\ndef trace_err(*params):\n    if config.debug > 0:\n        print(current_file + ':' + ''.join(params))\n\n\ndef valid_type(param):\n    if param in valid_vars:\n        return valid_vars[param]\n    trace_err(\"Type [\", param, \"] not defined\")\n    return param\n\n# Map from json list types to the C++ implementing type\nLIST_TO_IMPL = {\"array\": \"json_list\", \"chunked_array\": \"json_chunked_list\"}\n\ndef is_array_type(type: str):\n    return type in LIST_TO_IMPL\n\ndef type_change(param: str, member):\n    if is_array_type(param):\n        if \"items\" not in member:\n            trace_err(\"array without item declaration in \", param)\n            return \"\"\n        item = member[\"items\"]\n        if \"type\" in item:\n            t = item[\"type\"]\n        elif \"$ref\" in item:\n            t = item[\"$ref\"]\n        else:\n            trace_err(\"array items with no type or ref declaration \", param)\n            return \"\"\n        return LIST_TO_IMPL[param] + \"< \" + valid_type(t) + \" >\"\n\n    return \"json_element< \" + valid_type(param) + \" >\"\n\n\n\ndef print_ind_comment(f, ind, comment):\n    fprintln(f, ind, \"/**\")\n    fprintln(f, ind, \" * \", comment)\n    fprintln(f, ind, \" */\")\n\ndef print_comment(f, comment):\n    print_ind_comment(f, spacing, comment)\n\ndef print_copyrights(f):\n    fprintln(f, \"/*\")\n    fprintln(f, \"* Copyright (C) 2014 Cloudius Systems, Ltd.\")\n    fprintln(f, \"*\")\n    fprintln(f, \"* This work is open source software, licensed under the\",\n           \" terms of the\")\n    fprintln(f, \"* BSD license as described in the LICENSE f in the top-\",\n           \"level directory.\")\n    fprintln(f, \"*\")\n    fprintln(f, \"*  This is an Auto-Generated-code  \")\n    fprintln(f, \"*  Changes you do in this file will be erased on next\",\n           \" code generation\")\n    fprintln(f, \"*/\\n\")\n\n\ndef print_h_file_headers(f, name):\n    print_copyrights(f)\n    fprintln(f, \"#ifndef __JSON_AUTO_GENERATED_\" + name)\n    fprintln(f, \"#define __JSON_AUTO_GENERATED_\" + name + \"\\n\")\n\n\ndef clean_param(param):\n    match = re.match(r\"^\\{\\s*([^\\}]+)\\s*}\", param)\n    if match:\n        return [match.group(1), False]\n    return [param, True]\n\n\ndef get_parameter_by_name(obj, name):\n    for p in obj[\"parameters\"]:\n        if p[\"name\"] == name:\n            return p\n    trace_err (\"No Parameter declaration found for \", name)\n\n\ndef clear_path_ending(path):\n    if not path or path[-1] != '/':\n        return path\n    return path[0:-1]\n\n\nclass Parameter:\n    '''represents a parameter\n\n    TODO: return an instance of Parameter from get_parameter_by_name()\n    '''\n    def __init__(self, definition):\n        self.definition = definition\n\n    @property\n    def name(self):\n        return self.definition['name']\n\n    @property\n    def is_required(self):\n        # check if a parameter is query required.\n        # It will return true if the required flag is set\n        # and if it is a query parameter, both swagger 1.2 'paramType' and\n        # swagger 2.0 'in' attributes are supported\n        if \"required\" not in self.definition:\n            return False\n        if not self.definition[\"required\"]:\n            return False\n        if self.definition.get(\"paramType\") == \"query\":\n            return True\n        if self.definition.get(\"in\") == \"query\":\n            return True\n        return False\n\n    @property\n    def enum(self):\n        return self.definition.get('enum')\n\n\ndef add_path(f, path, details):\n    if \"summary\" in details:\n        print_comment(f, details[\"summary\"])\n    param_starts = path.find(\"{\")\n    if param_starts >= 0:\n        path_reminder = path[param_starts:]\n        vals = path.split(\"/\")\n        vals.reverse()\n        fprintln(f, spacing, 'path_description::add_path(\"', clear_path_ending(vals.pop()),\n           '\",', details[\"method\"], ',\"', details[\"nickname\"], '\")')\n        while vals:\n            param, is_url = clean_param(vals.pop())\n            if is_url:\n                fprintln(f, spacing, '  ->pushurl(\"', param, '\")')\n            else:\n                param_type = get_parameter_by_name(details, param)\n                if (\"allowMultiple\" in param_type and\n                    param_type[\"allowMultiple\"]):\n                    fprintln(f, spacing, '  ->pushparam(\"', param, '\",true)')\n                else:\n                    fprintln(f, spacing, '  ->pushparam(\"', param, '\")')\n    else:\n        fprintln(f, spacing, 'path_description::add_path(\"', clear_path_ending(path), '\",',\n           details[\"method\"], ',\"', details[\"nickname\"], '\")')\n    if \"parameters\" in details:\n        for param_definition in details[\"parameters\"]:\n            param = Parameter(param_definition)\n            if param.is_required:\n                fprintln(f, spacing, '  ->pushmandatory_param(\"', param.name, '\")')\n    fprintln(f, spacing, \";\")\n\n\ndef not_first():\n    '''\n    Returns True when gets called for the first time, False otherwise\n\n    used as the predicate parameter of textwrap.indent(), so the first\n    line is not indented. this helps to preserve the Python code's logical\n    indention in the template, and allows us to put something like::\n\n      blah = textwrap.indent(\"\"\"\\\n           foo bar\n               blah blah\n           foo bar\n    \"\"\"\n    '''\n    _is_first = True\n\n    def should_indent(_):\n        nonlocal _is_first\n        first = _is_first\n        _is_first = False\n        return not first\n    return should_indent\n\n\ndef generate_code_from_enum(nickname, type_name, enums):\n    def indent_body(s, level):\n        return textwrap.indent(s, level * '    ', not_first())\n\n    enum_list = ',\\n'.join(enums + ['NUM_ITEMS'])\n    decl = Template('''\\\n    namespace ns_$nickname {\n        enum class $type_name {\n            $enum_list\n        };\n        $type_name str2$type_name(const sstring& str);\n   }\n   ''').substitute(nickname=nickname,\n                   type_name=type_name,\n                   enum_list=indent_body(enum_list, 3))\n\n    name_list = ',\\n'.join(f'\"{enum}\"' for enum in enums)\n    parse_func = Template('''\\\n    $type_name str2$type_name(const sstring& str) {\n        static const std::string_view arr[] = {\n            $name_list\n        };\n        int i;\n        for (i = 0; i < $num_enums; i++) {\n            if (arr[i] == str) {\n                return ($type_name)i;\n            }\n        }\n        return ($type_name)i;\n    }\n    ''').substitute(type_name=type_name,\n                    name_list=indent_body(name_list, 3),\n                    num_enums=len(enums))\n\n    return decl, parse_func\n\n\ndef add_operation(hfile, ccfile, path, oper):\n    if \"summary\" in oper:\n        print_ind_comment(hfile, '', oper[\"summary\"])\n\n    param_starts = path.find(\"{\")\n    base_url = path\n    vals = []\n    if param_starts >= 0:\n        vals = path[param_starts:].split(\"/\")\n        vals.reverse()\n        base_url = path[:param_starts]\n\n    nickname = getitem(oper, \"nickname\", oper)\n    if config.create_cc:\n        fprintln(hfile, f'extern const path_description {nickname};')\n        maybe_static = ''\n    else:\n        maybe_static = 'static '\n    normalized_path = clear_path_ending(base_url)\n    http_method = oper[\"method\"]\n    fprintln(ccfile, f'{maybe_static}const path_description {nickname}(\"{normalized_path}\",{http_method},\"{nickname}\",')\n    fprint(ccfile, '{')\n    first = True\n    while vals:\n        path_param, is_url = clean_param(vals.pop())\n        if path_param == \"\":\n            continue\n        if first:\n            first = False\n        else:\n            fprint(ccfile, \"\\n,\")\n        if is_url:\n            path_param = f\"/{path_param}\"\n            component_type = 'FIXED_STRING'\n        elif get_parameter_by_name(oper, path_param).get(\"allowMultiple\",\n                                                         False):\n            component_type = 'PARAM_UNTIL_END_OF_PATH'\n        else:\n            component_type = 'PARAM'\n        fprint(ccfile, f'{{\"{path_param}\", path_description::url_component_type::{component_type}}}')\n    fprint(ccfile, '}')\n    fprint(ccfile, ',{')\n    enum_definitions = \"\"\n    if \"enum\" in oper:\n        enum_wrapper = create_enum_wrapper(nickname, \"return_type\", oper[\"enum\"])\n        enum_definitions = Template('''\nnamespace ns_$nickname {\n$enum_wrapper\n}\n''').substitute(nickname=nickname, enum_wrapper=enum_wrapper.rstrip())\n    funcs = \"\"\n    required_query_params = []\n    for param in oper.get(\"parameters\", []):\n        query_param = Parameter(param)\n        if query_param.is_required:\n            required_query_params.append(query_param)\n        if query_param.enum is not None:\n            enum_decl, parse_func = generate_code_from_enum(nickname,\n                                                            query_param.name,\n                                                            query_param.enum)\n            enum_definitions += enum_decl\n            funcs += parse_func\n    fprint(ccfile, '\\n,'.join(f'\"{param.name}\"' for param in required_query_params))\n    fprintln(ccfile, '});')\n    fprintln(hfile, enum_definitions)\n    open_namespace(ccfile, f'ns_{nickname}')\n    fprintln(ccfile, funcs)\n    close_namespace(ccfile)\n\n\ndef get_base_name(param):\n    return os.path.basename(param)\n\n\ndef is_model_valid(name, model):\n    if name in valid_vars:\n        return \"\"\n    properties = getitem(model[name], \"properties\", name)\n    for var in properties:\n        type = getitem(properties[var], \"type\", name + \":\" + var)\n        if is_array_type(type):\n            items = getitem(properties[var], \"items\", name + \":\" + var)\n            try:\n                type = getitem(items, \"type\", name + \":\" + var + \":items\")\n            except Exception as e:\n                try:\n                    type = getitem(items, \"$ref\", name + \":\" + var + \":items\")\n                except:\n                    raise e\n        if type not in valid_vars:\n            if type not in model:\n                raise Exception(\"Unknown type '\" + type + \"' in Model '\" + name + \"'\")\n            return type\n    valid_vars[name] = name\n    return \"\"\n\ndef resolve_model_order(data):\n    res = []\n    models = set()\n    for model_name in data:\n        visited = set(model_name)\n        missing = is_model_valid(model_name, data)\n        resolved = missing == ''\n        if not resolved:\n            stack = [model_name]\n            while not resolved:\n                if missing in visited:\n                    raise Exception(\"Cyclic dependency found: \" + missing)\n                missing_depends = is_model_valid(missing, data)\n                if missing_depends == '':\n                    if missing not in models:\n                        res.append(missing)\n                        models.add(missing)\n                    resolved = len(stack) == 0\n                    if not resolved:\n                        missing = stack.pop()\n                else:\n                    stack.append(missing)\n                    missing = missing_depends\n        elif model_name not in models:\n            res.append(model_name)\n            models.add(model_name)\n    return res\n\n\ndef create_enum_wrapper(model_name, name, values):\n    enum_name = f\"{model_name}_{name}\"\n    wrapper = name + \"_wrapper\"\n\n    def indent_body(s, level):\n        return textwrap.indent(s, level * '    ', not_first())\n\n    case_clauses = \"\\n\".join(\n        Template('''case $enum_name::$enum_entry: return \"\\\\\"$enum_entry\\\\\"\";''').substitute(\n            enum_name=enum_name, enum_entry=enum_entry) for enum_entry in values)\n    res = Template(\"\"\"\\\n    virtual std::string to_json() const {\n        switch(v) {\n        $case_clauses\n        default: return \"\\\\\"Unknown\\\\\"\";\n        }\n    }\"\"\").substitute(wrapper=wrapper,\n                     case_clauses=indent_body(case_clauses, 2))\n\n    case_clauses = \"\\n\".join(\n        Template(\"\"\"case T::$enum_entry: v = $enum_name::$enum_entry; break;\"\"\").substitute(\n            enum_name=enum_name, enum_entry=enum_entry) for enum_entry in values)\n    res += Template(\"\"\"\n    template<class T>\n    $wrapper(const T& _v) {\n        switch(_v) {\n        $case_clauses\n        default: v = $enum_name::NUM_ITEMS;\n        }\n    }\"\"\").substitute(wrapper=wrapper,\n                     case_clauses=indent_body(case_clauses, 2),\n                     enum_name=enum_name)\n\n    case_clauses = \"\\n\".join(\n        Template('''case $enum_name::$enum_entry: return T::$enum_entry;''').substitute(\n            enum_name=enum_name, enum_entry=enum_entry) for enum_entry in values)\n    res += Template(\"\"\"\n    template<class T>\n    operator T() const {\n        switch(v) {\n        $case_clauses\n        default: return T::$value;\n        }\n    }\"\"\").substitute(case_clauses=indent_body(case_clauses, 2),\n                     enum_name=enum_name,\n                     value=values[0])\n\n    res += Template(\"\"\"\n    typedef typename std::underlying_type<$enum_name>::type pos_type;\n    $wrapper& operator++() {\n        v = static_cast<$enum_name>(static_cast<pos_type>(v) + 1);\n        return *this;\n    }\n    $wrapper & operator++(int) {\n        return ++(*this);\n    }\n    bool operator==(const  $wrapper& c) const {\n        return v == c.v;\n    }\n    bool operator!=(const $wrapper& c) const {\n        return v != c.v;\n    }\n    bool operator<=(const $wrapper& c) const {\n        return static_cast<pos_type>(v) <= static_cast<pos_type>(c.v);\n    }\n    std::make_signed_t<pos_type> operator-(const $wrapper& c) const {\n        return static_cast<pos_type>(v) - static_cast<pos_type>(c.v);\n    }\n    static $wrapper begin() {\n        return $wrapper($enum_name::$value);\n    }\n    static $wrapper end() {\n        return $wrapper($enum_name::NUM_ITEMS);\n    }\n    static auto /* iota_view */ all_items() {\n        return std::ranges::iota_view<$wrapper, $wrapper>(begin(), end());\n    }\n    $enum_name v;\"\"\").substitute(enum_name=enum_name,\n                                 wrapper=wrapper,\n                                 value=values[0])\n    enum_wrapper = Template(\"\"\"\n    enum class $enum_name { $enumerators, NUM_ITEMS };\n    struct $wrapper : public json::jsonable {\n        $wrapper() = default;\n        $body\n    };\n    \"\"\").substitute(enum_name=enum_name,\n                    enumerators=\", \".join(values),\n                    wrapper=wrapper,\n                    body=indent_body(res, 1).lstrip())\n    return enum_wrapper.lstrip('\\n')\n\n\ndef to_operation(opr, data):\n    data[\"method\"] = opr.upper()\n    data[\"nickname\"] = data[\"operationId\"]\n    return data\n\ndef to_path(path, data):\n    data[\"operations\"] = [to_operation(k, data[k]) for k in data]\n    data[\"path\"] = path\n\n    return data\n\ndef create_h_file(data, hfile_name, api_name, init_method, base_api):\n    if config.o != '':\n        final_hfile_name = config.o\n    else:\n        final_hfile_name = config.outdir + \"/\" + hfile_name\n    hfile = open(final_hfile_name, \"w\")\n\n    if config.create_cc:\n        ccfile = open(final_hfile_name.rstrip('.hh') + \".cc\", \"w\")\n        add_include(ccfile, ['\"{}\"'.format(final_hfile_name)])\n        open_namespace(ccfile, \"seastar\")\n        open_namespace(ccfile, \"httpd\")\n        open_namespace(ccfile, api_name)\n    else:\n        ccfile = hfile\n    print_h_file_headers(hfile, api_name)\n    add_include(hfile, ['<seastar/core/sstring.hh>',\n                        '<seastar/json/json_elements.hh>',\n                        '<seastar/http/json_path.hh>'])\n\n    add_include(hfile, ['<iostream>', '<ranges>'])\n    open_namespace(hfile, \"seastar\")\n    open_namespace(hfile, \"httpd\")\n    open_namespace(hfile, api_name)\n\n    def indent(s):\n        return textwrap.indent(s.rstrip(), '        ', not_first())\n\n    if \"models\" in data:\n        models_order = resolve_model_order(data[\"models\"])\n        for model_name in models_order:\n            model = data[\"models\"][model_name]\n            if 'description' in model:\n                print_ind_comment(hfile, \"\", model[\"description\"])\n            fprintln(hfile, \"struct \", model_name, \" : public json::json_base {\")\n            member_init = ''\n            member_assignment = ''\n            member_move_assignment = ''\n            member_copy = ''\n            for member_name in model[\"properties\"]:\n                member = model[\"properties\"][member_name]\n                if \"description\" in member:\n                    print_comment(hfile, member[\"description\"])\n                if \"enum\" in member:\n                    fprintln(hfile, create_enum_wrapper(model_name, member_name, member[\"enum\"]))\n                    fprintln(hfile, f\"    {config.jsonns}::json_element<{member_name}_wrapper> {member_name};\\n\")\n                else:\n                    type_name = type_change(member[\"type\"], member)\n                    fprintln(hfile, f\"    {config.jsonns}::{type_name} {member_name};\\n\")\n                member_init += f'add(&{member_name}, \"{member_name}\");\\n'\n                member_assignment += f'{member_name} = e.{member_name};\\n'\n                member_move_assignment += f'{member_name} = std::move(e.{member_name});\\n'\n                member_copy += f'e.{member_name} = {member_name} ;\\n'\n\n            functions = Template('''\n    void register_params() {\n        $member_init\n    }\n    $model_name() {\n        register_params();\n    }\n    $model_name(const $model_name& e) {\n        register_params();\n        $member_assignment\n    }\n    $model_name($model_name&& e) {\n        register_params();\n        $member_move_assignment\n    }\n    template<class T>\n    $model_name& operator=(const T& e) {\n        $member_assignment\n        return *this;\n    }\n    $model_name& operator=(const $model_name& e) {\n        $member_assignment\n        return *this;\n    }\n    $model_name& operator=($model_name&& e) {\n        $member_move_assignment\n        return *this;\n    }\n    template<class T>\n    $model_name& update(T& e) {\n        $member_copy\n        return *this;\n    }''').substitute(model_name=model_name,\n                     member_init=indent(member_init),\n                     member_assignment=indent(member_assignment),\n                     member_move_assignment=indent(member_move_assignment),\n                     member_copy=indent(member_copy))\n            fprintln(hfile, functions.lstrip('\\n'))\n            fprintln(hfile, \"};\\n\\n\")\n\n #   print_ind_comment(hfile, \"\", \"Initialize the path\")\n#    fprintln(hfile, init_method + \"(const std::string& description);\")\n    fprintln(hfile, 'static const sstring name = \"', base_api, '\";')\n    for item in data[\"apis\"]:\n        path = item[\"path\"]\n        for oper in item.get(\"operations\", []):\n            add_operation(hfile, ccfile, path, oper)\n\n    close_namespace(hfile)\n    close_namespace(hfile)\n    close_namespace(hfile)\n    if config.create_cc:\n        close_namespace(ccfile)\n        close_namespace(ccfile)\n        close_namespace(ccfile)\n\n    hfile.write(\"#endif //__JSON_AUTO_GENERATED_HEADERS\\n\")\n    hfile.close()\n\ndef remove_leading_comma(data):\n    return re.sub(r'^\\s*,', '', data)\n\ndef format_as_json_object(data):\n    return \"{\" + remove_leading_comma(data) + \"}\"\n\ndef check_for_models(data, param):\n    model_name = param.replace(\".json\", \".def.json\")\n    if not os.path.isfile(model_name):\n        return\n    try:\n        with open(model_name) as myfile:\n            json_data = myfile.read()\n            def_data = json.loads(format_as_json_object(json_data))\n            data[\"models\"] = def_data\n    except Exception as e:\n        type, value, tb = sys.exc_info()\n        print(\"Bad formatted JSON definition file '\" + model_name + \"' error \", value.message)\n        sys.exit(-1)\n\ndef set_apis(data):\n    return {\"apis\": [to_path(p, data[p]) for p in data]}\n\ndef parse_file(param, combined):\n    global current_file\n    trace_verbose(\"parsing \", param, \" file\")\n    with open(param) as myfile:\n        json_data = myfile.read()\n    try:\n        data = json.loads(json_data)\n    except Exception as e:\n        try:\n            # the passed data is not a valid json, so maybe its a swagger 2.0\n            # snippet, format it as json and try again\n            # set_apis and check_for_models will create an object with a similiar format\n            # to a swagger 1.2 so the code generation would work\n            data = set_apis(json.loads(format_as_json_object(json_data)))\n            check_for_models(data, param)\n        except:\n            # The problem is with the file,\n            # just report the error and exit.\n            type, value, tb = sys.exc_info()\n            print(\"Bad formatted JSON file '\" + param + \"' error \", value.message)\n            sys.exit(-1)\n    try:\n        base_file_name = get_base_name(param)\n        current_file = base_file_name\n        hfile_name = base_file_name + \".hh\"\n        api_name = base_file_name.replace('.', '_')\n        base_api = base_file_name.replace('.json', '')\n        init_method = \"void \" + api_name + \"_init_path\"\n        trace_verbose(\"creating \", hfile_name)\n        if (combined):\n            fprintln(combined, '#include \"', base_file_name, \".cc\", '\"')\n        create_h_file(data, hfile_name, api_name, init_method, base_api)\n    except Exception as e:\n        print(\"Error while parsing JSON file '\" + param + \"' error \" + str(e))\n        sys.exit(-1)\n\nif \"indir\" in config and config.indir != '':\n    combined = open(config.combined, \"w\")\n    for f in glob.glob(os.path.join(config.indir, \"*.json\")):\n        parse_file(f, combined)\nelse:\n    parse_file(config.f, None)\n"
  },
  {
    "path": "scripts/stall-analyser.py",
    "content": "#!/usr/bin/env python\n\nimport argparse\nimport sys\nimport re\n\nimport addr2line\nfrom collections import defaultdict\nfrom itertools import chain, dropwhile\nfrom typing import Iterator, Self\n\n\ndef get_command_line_parser():\n    parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,\n        description='A reactor stall backtrace graph analyser.',\n        epilog=\"\"\"\n    stall-analyser helps analyze a series of reactor-stall backtraces using a graph.\n    Each node in the graph includes:\n      `addr` - a program address\n    Each link in the graph includes:\n      `total` - the total sum of stalls, in milliseconds\n                of all reactor stalls that pass via this caller/callee link.\n      `count` - number of backtraces going through the link.\n\n    When printed, the graph is traversed in descending `total` order\n    to emphasize stall backtraces that are frequent and long.\n\n    Each node in the printed output is preceded with [level#index pct%],\n    where `level` is the level of that node in the graph (0 are root nodes),\n    `index` is the index in the parent node's list of callers/callees, and\n    `pct` is the percantage of this link's `total` time relative to\n    its siblings.\n\n    When given an executable, addresses are decoding using `addr2line`\n    \"\"\")\n    parser.add_argument('--address-threshold', default='0x100000000',\n                        help='Skip common backtrace prefix terminated by one or more addresses greater or equal to the threshold (0=disabled)')\n    parser.add_argument('-e', '--executable',\n                        help='Decode addresses to lines using given executable')\n    parser.add_argument('-f', '--full-function-names', action='store_const', const=True, default=False,\n                        help=\"When demangling C++ function names, display all information, including the type of the function's parameters. Otherwise, they are omitted (see `c++filt(1) -p`).\")\n    parser.add_argument('-w', '--width', type=int, default=0,\n                        help='Smart trim of long lines to width characters (0=disabled)')\n    parser.add_argument('-d', '--direction', choices=['bottom-up', 'top-down'], default='bottom-up',\n                        help='Print graph bottom-up (default, callees first) or top-down (callers first)')\n    parser.add_argument('-m', '--minimum', type=int, dest='tmin', default=0,\n                        help='Process only stalls lasting the given time, in milliseconds, or longer')\n    parser.add_argument('-b', '--branch-threshold', type=float, default=0.03,\n                        help='Drop branches responsible for less than this threshold relative to the previous level, not global. (default 3%%)')\n    parser.add_argument('--format', choices=['graph', 'trace'], default='graph',\n                        help='The output format, default is %(default)s. `trace` is suitable as input for flamegraph.pl')\n    parser.add_argument('-a', '--addr2line', default='llvm-addr2line',\n                        help='The path or name of the addr2line command, which should behave as and '\n                            'accept the same options as binutils addr2line or llvm-addr2line (the default).')\n    parser.add_argument('file', nargs='?',\n                        type=argparse.FileType('r'),\n                        default=sys.stdin,\n                        help='File containing reactor stall backtraces. Read from stdin if missing.')\n    return parser\n\n\nclass Node:\n    def __init__(self, addr: str):\n        self.addr = addr\n        x = addr.find('+')\n        if x >= 0:\n            self.addr_only = addr[x + 1:]\n            self.module = addr[:x]\n        else:\n            self.addr_only = addr\n            self.module = None\n        self.callers = {}\n        self.callees = {}\n        self.printed = False\n\n    def __repr__(self):\n        return f\"Node({self.addr})\"\n\n    class Link:\n        def __init__(self, node: \"Node\", t: int) -> None:\n            self.node = node\n            self.total = t\n            self.count = 1\n\n        def __eq__(self, other: Self) -> bool:\n            return self.total == other.total and self.count == other.count\n\n        def __ne__(self, other: Self):\n            return not (self == other)\n\n        def __lt__(self, other: Self) -> bool:\n            return self.total < other.total or self.total == other.total and self.count < other.count\n\n        def add(self, t: int) -> bool:\n            self.total += t\n            self.count += 1\n\n    def link_caller(self, t: int, n: Self) -> Self:\n        if n.addr in self.callers:\n            link = self.callers[n.addr]\n            link.add(t)\n            n.callees[self.addr].add(t)\n        else:\n            self.callers[n.addr] = self.Link(n, t)\n            n.callees[self.addr] = self.Link(self, t)\n        return n\n\n    def unlink_caller(self, addr: str) -> None:\n        link = self.callers.pop(addr)\n        link.node.callees.pop(self.addr)\n\n    def link_callee(self, t: int, n) -> Self:\n        if n.addr in self.callees:\n            link = self.callees[n.addr]\n            link.add(t)\n            n.callers[self.addr].add(t)\n        else:\n            self.callees[n.addr] = self.Link(n, t)\n            n.callers[self.addr] = self.Link(self, t)\n        return n\n\n    def unlink_callee(self, addr: str) -> None:\n        link = self.callees.pop(addr)\n        link.node.callers.pop(self.addr)\n\n    def sorted_links(self, links: list, descending=True) -> list:\n        return sorted([l for l in links if l.node.addr], reverse=descending)\n\n    def sorted_callers(self, descending=True) -> list:\n        return self.sorted_links(self.callers.values(), descending)\n\n    def sorted_callees(self, descending=True) -> list:\n        return self.sorted_links(self.callees.values(), descending)\n\n\nclass Graph:\n    def __init__(self, resolver: addr2line.BacktraceResolver):\n        self.resolver = resolver\n        # Each node in the tree contains:\n        self.count = 0\n        self.total = 0\n        self.nodes = dict[str, Node]()\n        self.tail = Node('')\n        self.head = Node('')\n\n    def empty(self):\n        return not self.nodes\n\n    def __bool__(self):\n        return not self.empty()\n\n    def process_trace(self, trace: list[str], t: int) -> None:\n        # process each backtrace and insert it to the tree\n        #\n        # The backtraces are assumed to be in bottom-up order, i.e.\n        # the first address indicates the innermost frame and the last\n        # address is in the outermost, in calling order.\n        #\n        # This helps identifying closely related reactor stalls\n        # where a code path that stalls may be called from multiple\n        # call sites.\n        node = None\n        for addr in trace:\n            node = self.add(node, t, addr)\n        self.add_head(t, node)\n\n    def add(self, prev: Node, t: int, addr: str):\n        if addr in self.nodes:\n            n = self.nodes[addr]\n        else:\n            n = Node(addr)\n            self.nodes[addr] = n\n        if prev:\n            if prev.addr in self.head.callees:\n                self.head.unlink_callee(prev.addr)\n            prev.link_caller(t, n)\n            if addr in self.tail.callers:\n                self.tail.unlink_caller(addr)\n        elif not n.callees or addr in self.tail.callers:\n            self.tail.link_caller(t, n)\n        return n\n\n    def add_head(self, t: int, n: Node):\n        self.head.link_callee(t, n)\n\n    def smart_print(self, lines: str, width: int):\n        def _print(l: str, width: int):\n            if not width or len(l) <= width:\n                print(l)\n                return\n            i = l.rfind(\" at \")\n            if i < 0:\n                print(l[:width])\n                return\n            sfx = l[i:]\n            w = width - len(sfx) - 3\n            if w > 0:\n                pfx = l[:w]\n            else:\n                pfx = \"\"\n            print(f\"{pfx}...{sfx}\")\n        for l in lines.splitlines():\n            if l:\n                _print(l, width)\n\n    def print_graph(self, direction: str, width: int, branch_threshold: float):\n        top_down = (direction == 'top-down')\n        print(f\"\"\"\nThis graph is printed in {direction} order, where {'callers' if top_down else 'callees'} are printed first.\nUse --direction={'bottom-up' if top_down else 'top-down'} to print {'callees' if top_down else 'callers'} first.\n\n[level#index/out_of pct%] below denotes:\n  level  - nesting level in the graph\n  index  - index of node among to its siblings\n  out_of - number of siblings\n  pct    - percentage of total stall time of this call relative to its siblings\n\"\"\")\n\n        def _prefix(prefix_list: list):\n            prefix = ''\n            for p in prefix_list:\n                prefix += p\n            return prefix\n\n        def _recursive_print_graph(n: Node, total: int = 0, count: int = 0, level: int = -1, idx: int = 0, out_of: int = 0, rel: float = 1.0, prefix_list: list[str] = [], skip_stats: bool = False):\n            nonlocal top_down\n            if level >= 0:\n                avg = round(total / count) if count else 0\n                prefix = _prefix(prefix_list)\n                p = '+' if idx == 1 or idx == out_of else '|'\n                p += '+'\n                l = f\"[{level}#{idx}/{out_of} {round(100*rel)}%]\"\n                cont_indent = len(l) + 1\n                if skip_stats:\n                    l = f\"{' ' * (len(l)-2)} -\"\n                    stats = ''\n                else:\n                    stats = f\" total={total} count={count} avg={avg}\"\n                l = f\"{prefix}{p}{l} addr={n.addr}{stats}\"\n                p = \"| \"\n                if self.resolver:\n                    lines = self.resolver.resolve_address(n.addr_only, module=n.module).splitlines()\n                    if len(lines) == 1:\n                        li = lines[0]\n                        if li.startswith(\"??\"):\n                            l += f\": {lines[0]}\"\n                        else:\n                            l += f\":\\n{prefix}{p}{' '*cont_indent}{li.strip()}\"\n                    else:\n                        l += \":\\n\"\n                        if top_down:\n                            lines.reverse()\n                        for li in lines:\n                            l += f\"{prefix}{p}{' '*cont_indent}{li.strip()}\\n\"\n                self.smart_print(l, width)\n                if n.printed:\n                    print(f\"{prefix}-> continued at addr={n.addr} above\")\n                    return\n                n.printed = True\n            next = n.sorted_callees() if top_down else n.sorted_callers()\n            if not next:\n                return\n            link = next[0]\n            if level >= 0 and len(next) == 1 and link.total == total and link.count == count:\n                _recursive_print_graph(link.node, link.total, link.count, level, idx, out_of, rel, prefix_list, skip_stats=True)\n            else:\n                total = sum(link.total for link in next)\n                next_prefix_list = prefix_list + [\"| \" if idx < out_of else \"  \"] if level >= 0 else []\n                i = 1\n                last_idx = len(next)\n                omitted_idx = 0\n                omitted_total = 0\n                omitted_count = 0\n                for link in next:\n                    rel = link.total / total\n                    if rel < branch_threshold:\n                        if not omitted_idx:\n                            omitted_idx = i\n                        omitted_total += link.total\n                        omitted_count += link.count\n                    else:\n                        _recursive_print_graph(link.node, link.total, link.count, level + 1, i, last_idx, rel, next_prefix_list)\n                    i += 1\n                if omitted_idx:\n                    prefix = _prefix(next_prefix_list)\n                    p = '++'\n                    rel = omitted_total / total\n                    avg = round(omitted_total / omitted_count) if count else 0\n                    l = f\"[{level+1}#{omitted_idx}/{last_idx} {round(100*rel)}%]\"\n                    print(f\"{prefix}{p}{l} {last_idx - omitted_idx + 1} more branches total={omitted_total} count={omitted_count} avg={avg}\")\n\n        r = self.head if top_down else self.tail\n        _recursive_print_graph(r)\n\n\nclass StackCollapse:\n    # collapse stall backtraces into single lines\n    def __init__(self, resolver: addr2line.BacktraceResolver) -> None:\n        self.resolver = resolver\n        # track every stack and the its total sample counts\n        self.collapsed = defaultdict(int)\n        # to match lines like\n        # (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:485\n        self.pattern = re.compile(r'''(.+)\\s                 # function signature\n                                      at\\s                   #\n                                      [^:]+:(?:\\?|\\d+)       # <source>:<line number>''', re.X)\n\n    def process_trace(self, frames: list[str], count: int) -> None:\n        # each stall report is mapped to a line of perf samples\n        # so the output looks like:\n        # row_cache::update;row_cache::do_update;row_cache::upgrade_entry 42\n        # from outer-most caller to the inner-most callee, and 42 is the time\n        # in ms, but we use it for the count of samples.\n        self.collapsed[';'.join(frames)] += count\n\n    def _annotate_func(self, line: str) -> str:\n        # sample input:\n        #   (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:485\n        # sample output:\n        #   position_in_partition::tri_compare::operator()_[i]\n        if line.startswith(\"??\"):\n            return \"\"\n\n        inlined_prefix = ' (inlined by) '\n        inlined = line.startswith(inlined_prefix)\n        if inlined:\n            line = line[len(inlined_prefix):]\n\n        matched = self.pattern.match(line)\n        assert matched, f\"bad line: {line}\"\n        func = matched.groups()[0]\n        # annotations\n        if inlined:\n            func += \"_[i]\"\n        return func\n\n    def _resolve(self, addr: str) -> Iterator[str]:\n        lines = self.resolver.resolve_address(addr).splitlines()\n        return (self._annotate_func(line) for line in lines)\n\n    def print_graph(self, *_) -> None:\n        for stack, count in self.collapsed.items():\n            frames = filter(lambda frame: frame,\n                            chain.from_iterable(self._resolve(addr) for addr in stack.split(';')))\n            print(';'.join(reversed(list(frames))), count)\n\n\ndef print_stats(tally: dict, tmin: int) -> None:\n    data = []\n    total_time = 0\n    total_count = 0\n    processed_count = 0\n    min_time = 1000000\n    max_time = 0\n    median = None\n    p95 = None\n    p99 = None\n    p999 = None\n    for t in sorted(tally.keys()):\n        count = tally[t]\n        data.append((t, count))\n        total_time += t * count\n        if t < min_time:\n            min_time = t\n        if t > max_time:\n            max_time = t\n        total_count += count\n        if t >= tmin:\n            processed_count += count\n    running_count = 0\n    for (t, count) in data:\n        running_count += count\n        if median is None and running_count >= total_count / 2:\n            median = t\n        elif p95 is None and running_count >= (total_count * 95) / 100:\n            p95 = t\n        elif p99 is None and running_count >= (total_count * 99) / 100:\n            p99 = t\n        elif p999 is None and running_count >= (total_count * 999) / 1000:\n            p999 = t\n    print(f\"Processed {total_count} stalls lasting a total of {total_time} milliseconds.\")\n    if tmin:\n        print(f\"Of which, {processed_count} lasted {tmin} milliseconds or longer.\")\n    avg_time = total_time / total_count if total_count else 0\n    print(f\"min={min_time} avg={avg_time:.1f} median={median} p95={p95} p99={p99} p999={p999} max={max_time}\")\n\n\ndef print_command_line_options(args):\n    varargs = vars(args)\n    clopts = \"\"\n    for k in varargs.keys():\n        val = varargs[k]\n        opt = re.sub('_', '-', k)\n        if val is None:\n            continue\n        elif not isinstance(val, bool):\n            clopts += f\" --{opt}={val}\"\n        elif val:\n            clopts += f\" --{opt}\"\n    print(f\"Command line options:{clopts}\\n\")\n\n\ndef main():\n    args = get_command_line_parser().parse_args()\n    comment = re.compile(r'^\\s*#')\n    pattern = re.compile(r\"Reactor stalled for (?P<stall>\\d+) ms on shard (?P<shard>\\d+).*Backtrace:\")\n    expected_input_format = \"Expected one or more lines ending with: 'Reactor stalled for <n> ms on shard <i>. Backtrace: <addr> [<addr> ...]'\"\n    address_threshold = int(args.address_threshold, 0)\n    # map from stall time in ms to the count of the stall time\n    tally = {}\n    resolver = None\n    if args.executable:\n        resolver = addr2line.BacktraceResolver(executable=args.executable,\n                                               concise=not args.full_function_names,\n                                               cmd_path=args.addr2line)\n    if args.format == 'graph':\n        render = Graph(resolver)\n    else:\n        render = StackCollapse(resolver)\n\n    for s in args.file:\n        if comment.search(s):\n            continue\n        # parse log line like:\n        # ... scylla[7795]: Reactor stalled for 860 ms on shard 4. Backtrace: 0x4f002d2 0x4efef30 0x4f001e0\n        m = pattern.search(s)\n        if not m:\n            continue\n        # extract the time in ms\n        trace = s[m.span()[1]:].split()\n        t = int(m.group(\"stall\"))\n        # and the addresses after \"Backtrace:\"\n        tally[t] = tally.pop(t, 0) + 1\n        # The address_threshold typically indicates a library call\n        # and the backtrace up-to and including it are usually of\n        # no interest as they all contain the stall backtrace geneneration code, e.g.:\n        #  seastar::internal::cpu_stall_detector::generate_trace\n        # void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backt>\n        #  (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:771\n        #  (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc>\n        # seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1257\n        # seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1103\n        #  (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1117\n        #  (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1240\n        # ?? ??:0\n        if address_threshold:\n            trace = list(dropwhile(lambda addr: int(addr, 0) >= address_threshold, trace))\n        if t >= args.tmin:\n            if not trace:\n                print(f\"\"\"Invalid input line: '{s.strip()}'\n{expected_input_format}\nPlease run `stall-analyser.py --help` for usage instruction\"\"\", file=sys.stderr)\n                sys.exit(1)\n            render.process_trace(trace, t)\n\n    try:\n        if not render:\n            print(f\"\"\"No input data found.\n{expected_input_format}\nPlease run `stall-analyser.py --help` for usage instruction\"\"\", file=sys.stderr)\n            sys.exit(1)\n        if args.format == 'graph':\n            print_command_line_options(args)\n            print_stats(tally, args.tmin)\n        render.print_graph(args.direction, args.width, args.branch_threshold)\n    except BrokenPipeError:\n        pass\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "scripts/tap.sh",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n### Set up a tap device for seastar\ntap=tap0\nbridge=virbr0\nuser=`whoami`\nsudo ip link del $tap\nsudo ip tuntap add mode tap dev $tap user $user one_queue vnet_hdr\nsudo ifconfig $tap up\nsudo brctl addif $bridge $tap\nsudo brctl stp $bridge off\nsudo modprobe vhost-net\nsudo chown $user.$user /dev/vhost-net\nsudo brctl show $bridge\nsudo ifconfig $bridge\n"
  },
  {
    "path": "seastar_cmake.py",
    "content": "# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nimport os\n\nSUPPORTED_MODES = ['release', 'debug', 'dev', 'sanitize', 'fuzz']\n\nROOT_PATH = os.path.realpath(os.path.dirname(__file__))\n\nDEFAULT_BUILD_ROOT = 'build'\n\nCOOKING_BASIC_ARGS = ['./cooking.sh']\n\ndef build_path(mode, build_root):\n    \"\"\"Return the absolute path to the build directory for the given mode,\n    i.e., seastar_dir/<build_root>/<mode>\"\"\"\n    assert mode in SUPPORTED_MODES, f'Unsupported build mode: {mode}'\n    return os.path.join(ROOT_PATH, build_root, mode)\n\ndef is_release_mode(mode):\n    return mode == 'release'\n\ndef convert_strings_to_cmake_list(*args):\n    \"\"\"Converts a sequence of whitespace-separated strings of tokens into a semicolon-separated\n    string of tokens for CMake.\n\n    \"\"\"\n    return ';'.join(' '.join(args).split())\n\ndef translate_arg(arg, new_name, value_when_none='no'):\n    \"\"\"\n    Translate a value populated from the command-line into a name to pass to the invocation of CMake.\n    \"\"\"\n    if arg is None:\n        value = value_when_none\n    elif type(arg) is bool:\n        value = 'yes' if arg else 'no'\n    else:\n        value = arg\n\n    if value is None:\n        return ''\n    else:\n        return '-DSeastar_{}={}'.format(new_name, value)\n"
  },
  {
    "path": "src/core/alien.cc",
    "content": "// -*- mode:C++; tab-width:4; c-basic-offset:4; indent-tabs-mode:nil -*-\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 Red Hat\n */\n\n\n#include <compare>\n#include <atomic>\n#include <iterator>\n#include <memory>\n#include <vector>\n\n#include <seastar/core/alien.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/prefetch.hh>\n\nnamespace seastar {\nnamespace alien {\n\nmessage_queue::message_queue(reactor *to)\n  : _pending(to)\n{}\n\nvoid message_queue::stop() {\n    _metrics.clear();\n}\n\nvoid\nmessage_queue::lf_queue::maybe_wakeup() {\n    // see also smp_message_queue::lf_queue::maybe_wakeup()\n    std::atomic_signal_fence(std::memory_order_seq_cst);\n    remote->wakeup();\n}\n\nvoid message_queue::submit_item(std::unique_ptr<message_queue::work_item> item) {\n    if (!_pending.push(item.get())) {\n        throw std::bad_alloc();\n    }\n    item.release();\n    _pending.maybe_wakeup();\n    ++_sent.value;\n}\n\nbool message_queue::pure_poll_rx() const {\n    return !_pending.empty();\n}\n\ntemplate<typename Func>\nsize_t message_queue::process_queue(lf_queue& q, Func process) {\n    // copy batch to local memory in order to minimize\n    // time in which cross-cpu data is accessed\n    work_item* wi;\n    if (!q.pop(wi)) {\n        return 0;\n    }\n    work_item* items[batch_size + prefetch_cnt];\n    // start prefetching first item before popping the rest to overlap memory\n    // access with potential cache miss the second pop may cause\n    prefetch<2>(wi);\n    size_t nr = 0;\n    while (nr < batch_size && q.pop(items[nr])) {\n        ++nr;\n    }\n    std::fill(std::begin(items) + nr, std::begin(items) + nr + prefetch_cnt, nr ? items[nr - 1] : wi);\n    unsigned i = 0;\n    do {\n        prefetch_n<2>(std::begin(items) + i, std::begin(items) + i + prefetch_cnt);\n        process(wi);\n        wi = items[i++];\n    } while (i <= nr);\n\n    return nr + 1;\n}\n\nsize_t message_queue::process_incoming() {\n    if (_pending.empty()) {\n        return 0;\n    }\n    auto nr = process_queue(_pending, [] (work_item* wi) {\n        wi->process();\n        delete wi;\n    });\n    _received += nr;\n    _last_rcv_batch = nr;\n    return nr;\n}\n\nvoid message_queue::start() {\n    namespace sm = seastar::metrics;\n    char instance[10];\n    std::snprintf(instance, sizeof(instance), \"%u\", this_shard_id());\n    _metrics.add_group(\"alien\", {\n        // Absolute value of num packets in last tx batch.\n        sm::make_queue_length(\"receive_batch_queue_length\", _last_rcv_batch, sm::description(\"Current receive batch queue length\")),\n        // total_operations value:DERIVE:0:U\n        sm::make_counter(\"total_received_messages\", _received, sm::description(\"Total number of received messages\")),\n        // total_operations value:DERIVE:0:U\n        sm::make_counter(\"total_sent_messages\", [this] { return _sent.value.load(); }, sm::description(\"Total number of sent messages\")),\n    });\n}\n\n\nvoid internal::qs_deleter::operator()(alien::message_queue* qs) const {\n    for (unsigned i = 0; i < count; i++) {\n        qs[i].~message_queue();\n    }\n    ::operator delete[](qs);\n}\n\ninstance::qs instance::create_qs(const std::vector<reactor*>& reactors) {\n    auto queues = reinterpret_cast<alien::message_queue*>(operator new[] (sizeof(alien::message_queue) * reactors.size()));\n    for (unsigned i = 0; i < reactors.size(); i++) {\n        new (&queues[i]) alien::message_queue(reactors[i]);\n    }\n    return qs{queues, internal::qs_deleter{static_cast<unsigned>(reactors.size())}};\n}\n\nbool instance::poll_queues() {\n    auto& queue = _qs[this_shard_id()];\n    return queue.process_incoming() != 0;\n}\n\nbool instance::pure_poll_queues() {\n    auto& queue = _qs[this_shard_id()];\n    return queue.pure_poll_rx();\n}\n\ninstance* internal::default_instance;\n\n}\n}\n"
  },
  {
    "path": "src/core/app-template.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <fstream>\n#include <cstdlib>\n#include <chrono>\n#include <iostream>\n#include <fmt/format.h>\n#include <boost/program_options.hpp>\n#include <boost/make_shared.hpp>\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/alien.hh>\n#include <seastar/core/scollectd.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/log-cli.hh>\n#include <seastar/net/native-stack.hh>\n#include \"program_options.hh\"\n\nnamespace seastar {\n\nnamespace bpo = boost::program_options;\n\nusing namespace std::chrono_literals;\n\nstatic\napp_template::seastar_options\nseastar_options_from_config(app_template::config cfg) {\n    app_template::seastar_options opts;\n    opts.name = std::move(cfg.name);\n    opts.description = std::move(cfg.description);\n    opts.auto_handle_sigint_sigterm = std::move(cfg.auto_handle_sigint_sigterm);\n    opts.reactor_opts.task_quota_ms.set_default_value(cfg.default_task_quota / 1ms);\n    opts.reactor_opts.max_networking_io_control_blocks.set_default_value(cfg.max_networking_aio_io_control_blocks);\n    opts.smp_opts.reserve_additional_memory_per_shard = cfg.reserve_additional_memory_per_shard;\n    return opts;\n}\n\napp_template::seastar_options::seastar_options()\n    : program_options::option_group(nullptr, \"seastar\")\n    , reactor_opts(this)\n    , metrics_opts(this)\n    , smp_opts(this)\n    , scollectd_opts(this)\n    , log_opts(this)\n{\n}\n\napp_template::app_template(app_template::seastar_options opts)\n    : _alien(std::make_unique<alien::instance>())\n    , _smp(std::make_shared<smp>(*_alien))\n    , _opts(std::move(opts))\n    , _app_opts(_opts.name + \" options\")\n    , _conf_reader(get_default_configuration_reader()) {\n\n        if (!alien::internal::default_instance) {\n            alien::internal::default_instance = _alien.get();\n        }\n        _app_opts.add_options()\n                (\"help,h\", \"show help message\")\n                ;\n        _app_opts.add_options()\n                (\"help-seastar\", \"show help message about seastar options\")\n                ;\n        _app_opts.add_options()\n                (\"help-loggers\", \"print a list of logger names and exit\")\n                ;\n\n        {\n            program_options::options_description_building_visitor visitor;\n            _opts.describe(visitor);\n            _opts_conf_file.add(std::move(visitor).get_options_description());\n        }\n\n        _seastar_opts.add(_opts_conf_file);\n}\n\napp_template::app_template(app_template::config cfg)\n    : app_template(seastar_options_from_config(std::move(cfg)))\n{\n}\n\napp_template::~app_template() = default;\n\nconst app_template::seastar_options& app_template::options() const {\n    return _opts;\n}\n\napp_template::configuration_reader app_template::get_default_configuration_reader() {\n    return [this] (bpo::variables_map& configuration) {\n        auto home = std::getenv(\"HOME\");\n        if (home) {\n            std::ifstream ifs(std::string(home) + \"/.config/seastar/seastar.conf\");\n            if (ifs) {\n                bpo::store(bpo::parse_config_file(ifs, _opts_conf_file), configuration);\n            }\n            std::ifstream ifs_io(std::string(home) + \"/.config/seastar/io.conf\");\n            if (ifs_io) {\n                bpo::store(bpo::parse_config_file(ifs_io, _opts_conf_file), configuration);\n            }\n        }\n    };\n}\n\nvoid app_template::set_configuration_reader(configuration_reader conf_reader) {\n    _conf_reader = conf_reader;\n}\n\nboost::program_options::options_description& app_template::get_options_description() {\n    return _app_opts;\n}\n\nboost::program_options::options_description& app_template::get_conf_file_options_description() {\n    return _opts_conf_file;\n}\n\nboost::program_options::options_description_easy_init\napp_template::add_options() {\n    return _app_opts.add_options();\n}\n\nvoid\napp_template::add_positional_options(std::initializer_list<positional_option> options) {\n    for (auto&& o : options) {\n        _app_opts.add(boost::make_shared<bpo::option_description>(o.name, o.value_semantic, o.help));\n        _pos_opts.add(o.name, o.max_count);\n    }\n}\n\n\nbpo::variables_map&\napp_template::configuration() {\n    return *_configuration;\n}\n\nint\napp_template::run(int ac, char ** av, std::function<future<int> ()>&& func) noexcept {\n    return run_deprecated(ac, av, [func = std::move(func)] () mutable {\n        auto func_done = make_lw_shared<promise<>>();\n        internal::at_exit([func_done] { return func_done->get_future(); });\n        // No need to wait for this future.\n        // func's returned exit_code is communicated via engine().exit()\n        (void)futurize_invoke(func).finally([func_done] {\n            func_done->set_value();\n        }).then([] (int exit_code) {\n            return engine().exit(exit_code);\n        }).or_terminate();\n    });\n}\n\nint\napp_template::run(int ac, char ** av, std::function<future<> ()>&& func) noexcept {\n    return run(ac, av, [func = std::move(func)] {\n        return func().then([] () {\n            return 0;\n        });\n    });\n}\n\nint\napp_template::run_deprecated(int ac, char ** av, std::function<void ()>&& func) noexcept {\n    boost::program_options::options_description all_opts;\n    all_opts.add(_app_opts);\n    all_opts.add(_seastar_opts);\n\n    bpo::variables_map configuration;\n    try {\n        bpo::store(bpo::command_line_parser(ac, av)\n                    .options(all_opts)\n                    .positional(_pos_opts)\n                    .run()\n            , configuration);\n        _conf_reader(configuration);\n    } catch (bpo::error& e) {\n        fmt::print(\"error: {}\\n\\nTry --help.\\n\", e.what());\n        return 2;\n    }\n    if (configuration.count(\"help\")) {\n        if (!_opts.description.empty()) {\n            std::cout << _opts.description << \"\\n\";\n        }\n        std::cout << _app_opts << \"\\n\";\n        return 0;\n    }\n    if (configuration.count(\"help-seastar\")) {\n        std::cout << _seastar_opts << \"\\n\";\n        return 0;\n    }\n    if (configuration.count(\"help-loggers\")) {\n        log_cli::print_available_loggers(std::cout);\n        return 0;\n    }\n\n    try {\n        bpo::notify(configuration);\n    } catch (const bpo::error& ex) {\n        std::cout << ex.what() << std::endl;\n        return 1;\n    }\n\n    {\n        program_options::variables_map_extracting_visitor visitor(configuration);\n        _opts.mutate(visitor);\n    }\n    _opts.reactor_opts._argv0 = std::string(av[0]);\n    _opts.reactor_opts._auto_handle_sigint_sigterm = _opts.auto_handle_sigint_sigterm;\n    if (auto* native_stack = dynamic_cast<net::native_stack_options*>(_opts.reactor_opts.network_stack.get_selected_candidate_opts())) {\n        native_stack->_hugepages = _opts.smp_opts.hugepages;\n    }\n\n    // Needs to be before `smp::configure()`.\n    try {\n        apply_logging_settings(log_cli::extract_settings(_opts.log_opts));\n    } catch (const std::runtime_error& exn) {\n        std::cout << \"logging configuration error: \" << exn.what() << '\\n';\n        return 1;\n    }\n\n    try {\n        _smp->configure(_opts.smp_opts, _opts.reactor_opts);\n    } catch (...) {\n        std::cerr << \"Could not initialize seastar: \" << std::current_exception() << std::endl;\n        return 1;\n    }\n    _configuration = {std::move(configuration)};\n    // No need to wait for this future.\n    // func is waited on via engine().run()\n    (void)engine().when_started().then([this] {\n        return seastar::metrics::configure(_opts.metrics_opts).then([this] {\n            // set scollectd use the metrics configuration, so the later\n            // need to be set first\n            scollectd::configure( _opts.scollectd_opts);\n        });\n    }).then(\n        std::move(func)\n    ).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n        } catch (std::exception& ex) {\n            std::cout << \"program failed with uncaught exception: \" << ex.what() << \"\\n\";\n            engine().exit(1);\n        }\n    });\n    auto exit_code = engine().run();\n    _smp->cleanup();\n    return exit_code;\n}\n\n}\n"
  },
  {
    "path": "src/core/cgroup.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <string>\n#include <seastar/util/std-compat.hh>\n#include <set>\n\nnamespace seastar {\n\nnamespace cgroup {\n\nusing std::optional;\nusing cpuset = std::set<unsigned>;\n\noptional<cpuset> cpu_set();\nsize_t memory_limit();\n\ntemplate <typename T>\noptional<T> read_setting_as(std::string path);\n\ntemplate <typename T>\noptional<T> read_setting_V1V2_as(std::string cg1_path, std::string cg2_fname);\n}\n\n}\n"
  },
  {
    "path": "src/core/condition-variable.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB, Ltd.\n */\n#include <seastar/core/condition-variable.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nconst char* broken_condition_variable::what() const noexcept {\n    return \"Condition variable is broken\";\n}\n\nconst char* condition_variable_timed_out::what() const noexcept {\n    return \"Condition variable timed out\";\n}\n\ncondition_variable::~condition_variable() {\n    broken();\n}\n\nvoid condition_variable::add_waiter(waiter& w) noexcept {\n    SEASTAR_ASSERT(!_signalled); // should not have snuck between\n    if (_ex) {\n        w.set_exception(_ex);\n        return;\n    }\n    _waiters.push_back(w);\n}\n\nvoid condition_variable::waiter::timeout() noexcept {\n    this->unlink();\n    this->set_exception(std::make_exception_ptr(condition_variable_timed_out()));\n}\n\nbool condition_variable::wakeup_first() noexcept {\n    if (_waiters.empty()) {\n        return false;\n    }\n    auto& w = _waiters.front();\n    _waiters.pop_front();\n    if (_ex) {\n        w.set_exception(_ex);\n    } else {\n        w.signal();\n    }\n    return true;\n}\n\nbool condition_variable::check_and_consume_signal() noexcept {\n    return std::exchange(_signalled, false);\n}\n\nvoid condition_variable::signal() noexcept {\n    if (!wakeup_first()) {\n        _signalled = true;\n    }\n}\n\n/// Notify variable and wake up all waiter\nvoid condition_variable::broadcast() noexcept {\n    auto tmp(std::move(_waiters));\n    while (!tmp.empty()) {\n        auto& w = tmp.front();\n        tmp.pop_front();\n        if (_ex) {\n            w.set_exception(_ex);\n        } else {\n            w.signal();\n        }\n    }\n}\n\n/// Signal to waiters that an error occurred.  \\ref wait() will see\n/// an exceptional future<> containing the provided exception parameter.\n/// The future is made available immediately.\nvoid condition_variable::broken() noexcept {\n    broken(std::make_exception_ptr(broken_condition_variable()));\n}\n\nvoid condition_variable::broken(std::exception_ptr ep) noexcept {\n    _ex = ep;\n    broadcast();\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "src/core/disk_params.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2025 ScyllaDB\n */\n\n\n\n#include <chrono>\n#include <unordered_set>\n#include <fmt/format.h>\n#include <fmt/ostream.h>\n#include <fmt/ranges.h>\n#include <sys/stat.h>\n#include <yaml-cpp/yaml.h>\n\n#include <seastar/core/disk_params.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/smp_options.hh>\n#include <seastar/util/conversions.hh>\n\n\nusing namespace std::chrono_literals;\n\nnamespace YAML {\ntemplate<>\nstruct convert<seastar::internal::disk_params> {\n    static bool decode(const Node& node, seastar::internal::disk_params& mp) {\n        using namespace seastar;\n        if (node[\"mountpoints\"]) {\n            mp.mountpoints = node[\"mountpoints\"].as<std::vector<std::string>>();\n        } else {\n            mp.mountpoints.push_back(node[\"mountpoint\"].as<std::string>());\n        }\n        mp.read_bytes_rate = parse_memory_size(node[\"read_bandwidth\"].as<std::string>());\n        mp.read_req_rate = parse_memory_size(node[\"read_iops\"].as<std::string>());\n        mp.write_bytes_rate = parse_memory_size(node[\"write_bandwidth\"].as<std::string>());\n        mp.write_req_rate = parse_memory_size(node[\"write_iops\"].as<std::string>());\n        if (node[\"read_saturation_length\"]) {\n            mp.read_saturation_length = parse_memory_size(node[\"read_saturation_length\"].as<std::string>());\n        }\n        if (node[\"write_saturation_length\"]) {\n            mp.write_saturation_length = parse_memory_size(node[\"write_saturation_length\"].as<std::string>());\n        }\n        if (node[\"physical_block_size\"]) {\n            mp.physical_block_size = parse_memory_size(node[\"physical_block_size\"].as<std::string>());\n        }\n        if (node[\"duplex\"]) {\n            mp.duplex = node[\"duplex\"].as<bool>();\n        }\n        if (node[\"rate_factor\"]) {\n            mp.rate_factor = node[\"rate_factor\"].as<float>();\n        }\n        return true;\n    }\n};\n}\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\nnamespace internal {\n\nvoid disk_config_params::parse_config(const smp_options& smp_opts, const reactor_options& reactor_opts) {\n    seastar_logger.debug(\"smp::count: {}\", smp::count);\n    _latency_goal = std::chrono::duration_cast<std::chrono::duration<double>>(latency_goal_opt(reactor_opts) * 1ms);\n    seastar_logger.debug(\"latency_goal: {}\", latency_goal().count());\n    _flow_ratio_backpressure_threshold = reactor_opts.io_flow_ratio_threshold.get_value();\n    seastar_logger.debug(\"flow-ratio threshold: {}\", _flow_ratio_backpressure_threshold);\n    _stall_threshold = reactor_opts.io_completion_notify_ms.defaulted() ? std::chrono::milliseconds::max() : reactor_opts.io_completion_notify_ms.get_value() * 1ms;\n\n    if (smp_opts.num_io_groups) {\n        _num_io_groups = smp_opts.num_io_groups.get_value();\n        if (!_num_io_groups) {\n            throw std::runtime_error(\"num-io-groups must be greater than zero\");\n        }\n    }\n    if (smp_opts.io_properties_file && smp_opts.io_properties) {\n        throw std::runtime_error(\"Both io-properties and io-properties-file specified. Don't know which to trust!\");\n    }\n\n    std::optional<YAML::Node> doc;\n    if (smp_opts.io_properties_file) {\n        doc = YAML::LoadFile(smp_opts.io_properties_file.get_value());\n    } else if (smp_opts.io_properties) {\n        doc = YAML::Load(smp_opts.io_properties.get_value());\n    }\n\n    if (doc) {\n        if (!doc->IsMap()) {\n            throw std::runtime_error(\"Bogus io-properties (did you mix up --io-properties and --io-properties-file?)\");\n        }\n        for (auto&& section : *doc) {\n            auto sec_name = section.first.as<std::string>();\n            if (sec_name != \"disks\") {\n                throw std::runtime_error(fmt::format(\"While parsing I/O options: section {} currently unsupported.\", sec_name));\n            }\n            auto disks = section.second.as<std::vector<disk_params>>();\n            unsigned queue_id_gen = 1;\n            std::unordered_set<dev_t> devices;\n            for (auto& d : disks) {\n                for (auto mp : d.mountpoints) {\n                    struct ::stat buf;\n                    auto ret = stat(mp.c_str(), &buf);\n                    if (ret < 0) {\n                        throw std::runtime_error(fmt::format(\"Couldn't stat {}\", mp));\n                    }\n\n                    auto st_dev = S_ISBLK(buf.st_mode) ? buf.st_rdev : buf.st_dev;\n                    d.devices.push_back(st_dev);\n                    auto [ it, inserted ] = devices.insert(st_dev);\n                    if (!inserted) {\n                        throw std::runtime_error(fmt::format(\"Mountpoint {}, device {} already configured\", mp, st_dev));\n                    }\n                }\n\n                if (_disks.size() >= _max_queues) {\n                    throw std::runtime_error(fmt::format(\"Configured number of queues {} is larger than the maximum {}\",\n                                                _disks.size(), _max_queues));\n                }\n\n                d.read_bytes_rate *= d.rate_factor;\n                d.write_bytes_rate *= d.rate_factor;\n                d.read_req_rate *= d.rate_factor;\n                d.write_req_rate *= d.rate_factor;\n\n                if (d.read_bytes_rate == 0 || d.write_bytes_rate == 0 ||\n                        d.read_req_rate == 0 || d.write_req_rate == 0) {\n                    throw std::runtime_error(fmt::format(\"R/W bytes and req rates must not be zero\"));\n                }\n\n                unsigned q = queue_id_gen++;\n                seastar_logger.debug(\"queue-id: {} mountpoints: {} devices: {}\", q, d.mountpoints, d.devices);\n                _disks.emplace(q, d);\n            }\n        }\n    }\n\n    // Placeholder for unconfigured disks.\n    disk_params d = {};\n    d.devices.push_back(0);\n    _disks.emplace(0, d);\n}\n\nstruct io_queue::config disk_config_params::generate_config(unsigned q, unsigned nr_groups) const {\n    auto it = _disks.find(q);\n    if (it == _disks.end()) {\n        throw std::runtime_error(fmt::format(\"No disk configuration for queue-id {}\", q));\n    }\n    return generate_config(it->second, q, nr_groups);\n}\n\nstruct io_queue::config disk_config_params::generate_config(const disk_params& p, unsigned q, unsigned nr_groups) const {\n    seastar_logger.debug(\"generate_config queue-id: {}\", q);\n    struct io_queue::config cfg;\n\n    cfg.id = q;\n\n    if (p.read_bytes_rate != std::numeric_limits<uint64_t>::max()) {\n        cfg.blocks_count_rate = (io_queue::read_request_base_count * (unsigned long)per_io_group(p.read_bytes_rate, nr_groups)) >> io_queue::block_size_shift;\n        cfg.disk_blocks_write_to_read_multiplier = (io_queue::read_request_base_count * p.read_bytes_rate) / p.write_bytes_rate;\n    }\n    if (p.read_req_rate != std::numeric_limits<uint64_t>::max()) {\n        cfg.req_count_rate = io_queue::read_request_base_count * (unsigned long)per_io_group(p.read_req_rate, nr_groups);\n        cfg.disk_req_write_to_read_multiplier = (io_queue::read_request_base_count * p.read_req_rate) / p.write_req_rate;\n    }\n    if (p.read_saturation_length != std::numeric_limits<uint64_t>::max()) {\n        cfg.disk_read_saturation_length = p.read_saturation_length;\n    }\n    if (p.write_saturation_length != std::numeric_limits<uint64_t>::max()) {\n        cfg.disk_write_saturation_length = p.write_saturation_length;\n    }\n    cfg.mountpoint = fmt::to_string(fmt::join(p.mountpoints, \":\"));\n    cfg.duplex = p.duplex;\n    cfg.rate_limit_duration = latency_goal();\n    cfg.flow_ratio_backpressure_threshold = _flow_ratio_backpressure_threshold;\n    // Block count limit should not be less than the minimal IO size on the device\n    // On the other hand, even this is not good enough -- in the worst case the\n    // scheduler will self-tune to allow for the single 64k request, while it would\n    // be better to sacrifice some IO latency, but allow for larger concurrency\n    cfg.block_count_limit_min = (64 << 10) >> io_queue::block_size_shift;\n    cfg.stall_threshold = stall_threshold();\n    cfg.physical_block_size = p.physical_block_size;\n\n    return cfg;\n}\n\n} // namespace internal\n\n} // namespace seastar\n"
  },
  {
    "path": "src/core/dpdk_rte.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n#ifdef SEASTAR_HAVE_DPDK\n\n#include <cinttypes>\n#include <seastar/net/dpdk.hh>\n#include <seastar/core/dpdk_rte.hh>\n#include <seastar/util/conversions.hh>\n#include <seastar/util/std-compat.hh>\n#include <rte_pci.h>\n\nnamespace seastar {\n\nnamespace dpdk {\n\nbool eal::initialized = false;\n\nvoid eal::init(cpuset cpus, const std::string& argv0, const std::optional<std::string>& hugepages_path, bool dpdk_pmd)\n{\n    if (initialized) {\n        return;\n    }\n\n    size_t cpu_count = cpus.count();\n    std::stringstream mask;\n    cpuset nibble = 0xF;\n    while (cpus.any()) {\n        mask << std::hex << (cpus & nibble).to_ulong();\n        cpus >>= 4;\n    }\n\n    std::string mask_str = mask.str();\n    std::reverse(mask_str.begin(), mask_str.end());\n\n    std::vector<std::vector<char>> args {\n        string2vector(argv0),\n        string2vector(\"-c\"), string2vector(mask_str),\n        string2vector(\"-n\"), string2vector(\"1\")\n    };\n\n    // If \"hugepages\" is not provided and DPDK PMD drivers mode is requested -\n    // use the default DPDK huge tables configuration.\n    if (hugepages_path) {\n        args.push_back(string2vector(\"--huge-dir\"));\n        args.push_back(string2vector(hugepages_path.value()));\n\n        //\n        // We don't know what is going to be our networking configuration so we\n        // assume there is going to be a queue per-CPU. Plus we'll give a DPDK\n        // 64MB for \"other stuff\".\n        //\n        size_t size_MB = mem_size(cpu_count) >> 20;\n        std::stringstream size_MB_str;\n        size_MB_str << size_MB;\n\n        args.push_back(string2vector(\"-m\"));\n        args.push_back(string2vector(size_MB_str.str()));\n    } else if (!dpdk_pmd) {\n        args.push_back(string2vector(\"--no-huge\"));\n    }\n\n    std::vector<char*> cargs;\n\n    for (auto&& a: args) {\n        cargs.push_back(a.data());\n    }\n    /* initialise the EAL for all */\n    int ret = rte_eal_init(cargs.size(), cargs.data());\n    if (ret < 0) {\n        rte_exit(EXIT_FAILURE, \"Cannot init EAL\\n\");\n    }\n\n    initialized = true;\n}\n\nuint32_t __attribute__((weak)) qp_mempool_obj_size(bool hugetlbfs_membackend)\n{\n    return 0;\n}\n\nsize_t eal::mem_size(int num_cpus, bool hugetlbfs_membackend)\n{\n    size_t memsize = 0;\n    //\n    // PMD mempool memory:\n    //\n    // We don't know what is going to be our networking configuration so we\n    // assume there is going to be a queue per-CPU.\n    //\n    memsize += num_cpus * qp_mempool_obj_size(hugetlbfs_membackend);\n\n    // Plus we'll give a DPDK 64MB for \"other stuff\".\n    memsize += (64UL << 20);\n\n    return memsize;\n}\n\n} // namespace dpdk\n\n}\n\n#endif // SEASTAR_HAVE_DPDK\n"
  },
  {
    "path": "src/core/exception_hacks.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n// The purpose of the hacks here is to workaround C++ exception scalability problem\n// with gcc and glibc. For the best result gcc-7 is required.\n//\n// To summarize all the locks we have now and their purpose:\n// 1. There is a lock in _Unwind_Find_FDE (libgcc) that protects\n//    list of \"Frame description entries\" registered with __register_frame*\n//    functions. The catch is that dynamically linked binary do not do that,\n//    so all that it protects is checking that a certain list is empty.\n//    This lock no longer relevant in gcc-7 since there is a patch there\n//    that checks that the list is empty outside of the lock and it will be\n//    always true for us.\n// 2. The lock in dl_iterate_phdr (glibc) that protects loaded object\n//    list against runtime object loading/unloading.\n//\n// To get rid of the first lock using gcc-7 is required.\n//\n// To get rid of the second one we can use the fact that we do not\n// load/unload objects dynamically (at least for now). To do that we\n// can mirror all elf header information in seastar and provide our\n// own dl_iterate_phdr symbol which uses this mirror without locking.\n//\n// Unfortunately there is another gotcha in this approach: dl_iterate_phdr\n// supplied by glibc never calls more then one callback simultaneously as an\n// unintended consequences of the lock there, but unfortunately libgcc relies\n// on that to maintain small cache of translations. The access to the cache is\n// not protected by any lock since up until now only one callback could have\n// run at a time. But luckily libgcc cannot use the cache if older version\n// of dl_phdr_info is provided to the callback because the older version\n// did not have an indication that loaded object list may have changed,\n// so libgcc does not know when cache should be invalidated and disables it\n// entirely. By calling the callback with old version of dl_phdr_info from\n// our dl_iterate_phdr we can effectively make libgcc callback thread safe.\n\n\n#include <link.h>\n#include <dlfcn.h>\n#include <vector>\n#include <cstddef>\n\n#include <seastar/core/exception_hacks.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\nusing dl_iterate_fn = int (*) (int (*callback) (struct dl_phdr_info *info, size_t size, void *data), void *data);\n\n[[gnu::no_sanitize_address]]\nstatic dl_iterate_fn dl_iterate_phdr_org() {\n    // #2670 - Do the check and resolve manually, to avoid\n    // init order fiasco iff we're called before dl_init of this\n    // module.\n    // During initialization, ASan verifies it's the first DSO in the library list\n    // by calling dl_iterate_phdr(3). This is necessary for ASan to properly\n    // intercept memory operations.\n    //\n    // However, Seastar also provides the dl_iterate_phdr symbol, and its\n    // implementation depends on a static local variable `dl_iterate_fn`. This\n    // creates a circular dependency:\n    //\n    // 1. ASan loads first (as required)\n    // 2. ASan calls dl_iterate_phdr\n    // 3. The linker resolves this to Seastar's implementation\n    // 4. But Seastar's shared library hasn't been fully initialized via dl_init()\n    // 5. The function local static variable `dl_iterate_fn` that Seastar's implementation requires\n    //    isn't initialized to _be_ initialized - i.e. the lambda it required to run for init\n    //    is not in fact runnable.\n    // 6. This results in a segmentation fault\n    //\n    // So, we need to manually handle this initialization sequence to ensure both ASan\n    // and Seastar can properly initialize without conflicts.\n    static dl_iterate_fn org = nullptr;\n    if (org == nullptr) {\n        org = (dl_iterate_fn)dlsym (RTLD_NEXT, \"dl_iterate_phdr\");\n    }\n    return org;\n}\n\n// phdrs_cache has to remain valid until very late in the process\n// life, and that time is not a mirror image of when it is first used.\n// Given that, we avoid a static constructor/destructor pair and just\n// never destroy it.\nstatic std::vector<dl_phdr_info> *phdrs_cache = nullptr;\n\nvoid init_phdr_cache() {\n    // this process started reactor before, no need to fill the cache\n    if (phdrs_cache) {\n        return;\n    }\n    // Fill out elf header cache for access without locking.\n    // This assumes no dynamic object loading/unloading after this point\n    phdrs_cache = new std::vector<dl_phdr_info>();\n    dl_iterate_phdr_org()([] (struct dl_phdr_info *info, size_t size, void *data) {\n        phdrs_cache->push_back(*info);\n        return 0;\n    }, nullptr);\n}\n\nvoid internal::increase_thrown_exceptions_counter() noexcept {\n    seastar::engine()._cxx_exceptions++;\n}\n\n#ifndef NO_EXCEPTION_INTERCEPT\nseastar::logger exception_logger(\"exception\");\n\nvoid log_exception_trace(seastar::log_level level) noexcept {\n    static thread_local bool nested = false;\n    if (!nested && exception_logger.is_enabled(level)) {\n        nested = true;\n        exception_logger.log(level, \"Throw exception at:\\n{}\", current_backtrace());\n        nested = false;\n    }\n}\n#else\nvoid log_exception_trace(seastar::log_level) noexcept {}\n#endif\n\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\n[[gnu::no_sanitize_address]]\nint dl_iterate_phdr(int (*callback) (struct dl_phdr_info *info, size_t size, void *data), void *data) {\n    if (!seastar::phdrs_cache || !seastar::local_engine) {\n        // Cache is not yet populated, pass through to original function\n        return seastar::dl_iterate_phdr_org()(callback, data);\n    }\n    int r = 0;\n    for (auto h : *seastar::phdrs_cache) {\n        // Pass dl_phdr_info size that does not include dlpi_adds and dlpi_subs.\n        // This forces libgcc to disable caching which is not thread safe and\n        // requires dl_iterate_phdr to serialize calls to callback. Since we do\n        // not serialize here we have to disable caching.\n        r = callback(&h, offsetof(dl_phdr_info, dlpi_adds), data);\n        if (r) {\n            break;\n        }\n    }\n    return r;\n}\n\n// We disable interception of _Unwind_RaiseException when ASAN is enabled\n// since it can produce a large number of stack-related false positives when\n// it is enabled and exceptions are thrown.\n#if !defined(NO_EXCEPTION_INTERCEPT) && !defined(SEASTAR_ASAN_ENABLED)\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nint _Unwind_RaiseException(struct ::_Unwind_Exception *h) {\n    using throw_fn =  int (*)(void *);\n    static throw_fn org = nullptr;\n\n    if (!org) {\n        org = (throw_fn)dlsym (RTLD_NEXT, \"_Unwind_RaiseException\");\n    }\n    if (seastar::local_engine) {\n        seastar::internal::increase_thrown_exceptions_counter();\n        seastar::log_exception_trace(seastar::log_level::debug);\n    }\n    return org(h);\n}\n#endif\n"
  },
  {
    "path": "src/core/execution_stage.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/make_task.hh>\n#include <seastar/util/defer.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\nvoid execution_stage_manager::register_execution_stage(execution_stage& stage) {\n    auto ret = _stages_by_name.emplace(stage.name(), &stage);\n    if (!ret.second) {\n        throw std::invalid_argument(format(\"Execution stage {} already exists.\", stage.name()));\n    }\n    try {\n        _execution_stages.push_back(&stage);\n    } catch (...) {\n        _stages_by_name.erase(stage.name());\n        throw;\n    }\n}\n\nvoid execution_stage_manager::unregister_execution_stage(execution_stage& stage) noexcept {\n    auto it = std::find(_execution_stages.begin(), _execution_stages.end(), &stage);\n    if (it == _execution_stages.end()) {\n        return; // was changed by update_execution_stage_registration\n    }\n    _execution_stages.erase(it);\n    _stages_by_name.erase(stage.name());\n}\n\nvoid execution_stage_manager::update_execution_stage_registration(execution_stage& old_es, execution_stage& new_es) noexcept {\n    auto it = std::find(_execution_stages.begin(), _execution_stages.end(), &old_es);\n    *it = &new_es;\n    _stages_by_name.find(new_es.name())->second = &new_es;\n}\n\nexecution_stage* execution_stage_manager::get_stage(const sstring& name) {\n    return _stages_by_name[name];\n}\n\nvoid execution_stage_manager::update_scheduling_group_name(scheduling_group sg) noexcept {\n    for (auto&& stage : _execution_stages) {\n        if (stage->get_scheduling_group() == sg) {\n            auto old_name = stage->name();\n            stage->update_name_and_metric_group();\n            _stages_by_name.erase(old_name);\n            _stages_by_name.emplace(stage->name(), stage);\n        }\n    }\n}\n\nbool execution_stage_manager::flush() noexcept {\n    bool did_work = false;\n    for (auto&& stage : _execution_stages) {\n        did_work |= stage->flush();\n    }\n    return did_work;\n}\n\nbool execution_stage_manager::poll() const noexcept {\n    for (auto&& stage : _execution_stages) {\n        if (stage->poll()) {\n            return true;\n        }\n    }\n    return false;\n}\n\nexecution_stage_manager& execution_stage_manager::get() noexcept {\n    static thread_local execution_stage_manager instance;\n    return instance;\n}\n\n}\n\nexecution_stage::~execution_stage()\n{\n    internal::execution_stage_manager::get().unregister_execution_stage(*this);\n}\n\nexecution_stage::execution_stage(execution_stage&& other)\n    : _sg(other._sg)\n    , _stats(other._stats)\n    , _name(std::move(other._name))\n    , _metric_group(std::move(other._metric_group))\n{\n    internal::execution_stage_manager::get().update_execution_stage_registration(other, *this);\n}\n\nexecution_stage::execution_stage(const sstring& name, scheduling_group sg)\n    : _sg(sg)\n    , _name(name)\n{\n    internal::execution_stage_manager::get().register_execution_stage(*this);\n    auto undo = defer([&] () noexcept { internal::execution_stage_manager::get().unregister_execution_stage(*this); });\n    update_metric_group();\n    undo.cancel();\n}\n\nvoid execution_stage::update_metric_group() {\n    _metric_group.clear();\n    _metric_group = metrics::metric_group(\"execution_stages\", {\n             metrics::make_counter(\"tasks_scheduled\",\n                                  metrics::description(\"Counts tasks scheduled by execution stages\"),\n                                  { metrics::label_instance(\"execution_stage\", _name), },\n                                  [name = _name, &esm = internal::execution_stage_manager::get()] {\n                                      return esm.get_stage(name)->get_stats().tasks_scheduled;\n                                  }),\n             metrics::make_counter(\"tasks_preempted\",\n                                  metrics::description(\"Counts tasks which were preempted before execution all queued operations\"),\n                                  { metrics::label_instance(\"execution_stage\", _name), },\n                                  [name = _name, &esm = internal::execution_stage_manager::get()] {\n                                      return esm.get_stage(name)->get_stats().tasks_preempted;\n                                  }),\n             metrics::make_counter(\"function_calls_enqueued\",\n                                  metrics::description(\"Counts function calls added to execution stages queues\"),\n                                  { metrics::label_instance(\"execution_stage\", _name), },\n                                  [name = _name, &esm = internal::execution_stage_manager::get()] {\n                                      return esm.get_stage(name)->get_stats().function_calls_enqueued;\n                                  }),\n             metrics::make_counter(\"function_calls_executed\",\n                                  metrics::description(\"Counts function calls executed by execution stages\"),\n                                  { metrics::label_instance(\"execution_stage\", _name), },\n                                  [name = _name, &esm = internal::execution_stage_manager::get()] {\n                                      return esm.get_stage(name)->get_stats().function_calls_executed;\n                                  }),\n           });\n}\n\nbool execution_stage::flush() noexcept {\n    if (_empty || _flush_scheduled) {\n        return false;\n    }\n    _stats.tasks_scheduled++;\n    schedule(make_task(_sg, [this] {\n        do_flush();\n        _flush_scheduled = false;\n    }));\n    _flush_scheduled = true;\n    return true;\n};\n\n}\n"
  },
  {
    "path": "src/core/fair_queue.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n\n#include <chrono>\n#include <functional>\n#include <utility>\n#include <boost/container/small_vector.hpp>\n#include <boost/intrusive/parent_from_member.hpp>\n\n#include <seastar/core/fair_queue.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nstatic_assert(sizeof(fair_queue_ticket) == sizeof(uint64_t), \"unexpected fair_queue_ticket size\");\nstatic_assert(sizeof(fair_queue_entry) <= 3 * sizeof(void*), \"unexpected fair_queue_entry::_hook size\");\nstatic_assert(sizeof(fair_queue_entry::container_list_t) == 2 * sizeof(void*), \"unexpected priority_class::_queue size\");\n\nfair_queue_ticket::fair_queue_ticket(uint32_t weight, uint32_t size) noexcept\n    : _weight(weight)\n    , _size(size)\n{}\n\nfloat fair_queue_ticket::normalize(fair_queue_ticket denominator) const noexcept {\n    return float(_weight) / denominator._weight + float(_size) / denominator._size;\n}\n\nfair_queue_ticket fair_queue_ticket::operator+(fair_queue_ticket desc) const noexcept {\n    return fair_queue_ticket(_weight + desc._weight, _size + desc._size);\n}\n\nfair_queue_ticket& fair_queue_ticket::operator+=(fair_queue_ticket desc) noexcept {\n    _weight += desc._weight;\n    _size += desc._size;\n    return *this;\n}\n\nfair_queue_ticket fair_queue_ticket::operator-(fair_queue_ticket desc) const noexcept {\n    return fair_queue_ticket(_weight - desc._weight, _size - desc._size);\n}\n\nfair_queue_ticket& fair_queue_ticket::operator-=(fair_queue_ticket desc) noexcept {\n    _weight -= desc._weight;\n    _size -= desc._size;\n    return *this;\n}\n\nfair_queue_ticket::operator bool() const noexcept {\n    return (_weight > 0) || (_size > 0);\n}\n\nbool fair_queue_ticket::is_non_zero() const noexcept {\n    return (_weight > 0) && (_size > 0);\n}\n\nbool fair_queue_ticket::operator==(const fair_queue_ticket& o) const noexcept {\n    return _weight == o._weight && _size == o._size;\n}\n\nstd::ostream& operator<<(std::ostream& os, fair_queue_ticket t) {\n    return os << t._weight << \":\" << t._size;\n}\n\nfair_queue_ticket wrapping_difference(const fair_queue_ticket& a, const fair_queue_ticket& b) noexcept {\n    return fair_queue_ticket(std::max<int32_t>(a._weight - b._weight, 0),\n            std::max<int32_t>(a._size - b._size, 0));\n}\n\n// Priority class, to be used with a given fair_queue\nclass fair_queue::priority_class_data final : public priority_entry {\n    friend class fair_queue;\n    capacity_t _pure_accumulated = 0;\n    fair_queue_entry::container_list_t _queue;\n\n    bool plug() noexcept;\n\npublic:\n    explicit priority_class_data(uint32_t shares, priority_class_group_data* p) noexcept : priority_entry(shares, p) {}\n    priority_class_data(const priority_class_data&) = delete;\n    priority_class_data(priority_class_data&&) = delete;\n\n    fair_queue_entry* top() override;\n    std::pair<bool, capacity_t> pop_front() override;\n};\n\nfair_queue_entry* fair_queue::priority_class_data::top() {\n    return (_plugged && !_queue.empty()) ? &_queue.front() : nullptr;\n}\n\nstd::pair<bool, fair_queue_entry::capacity_t> fair_queue::priority_class_data::pop_front() {\n    auto req_cap = _queue.front()._capacity;\n    _pure_accumulated += req_cap;\n    _queue.pop_front();\n    return std::make_pair(_queue.empty(), req_cap);\n}\n\nbool fair_queue::class_compare::operator() (const priority_entry_ptr& lhs, const priority_entry_ptr& rhs) const noexcept {\n    return lhs->_accumulated > rhs->_accumulated;\n}\n\nfair_queue::fair_queue(config cfg)\n    : _config(std::move(cfg))\n    , _root(0, nullptr)\n{\n}\n\nfair_queue::~fair_queue() {\n    for (const auto& fq : _priority_classes) {\n        SEASTAR_ASSERT(!fq);\n    }\n}\n\nvoid fair_queue::priority_entry::wakeup(const fair_queue::config& cfg) noexcept {\n    if (!_queued) {\n        _parent->push_from_idle(*this, cfg);\n    }\n}\n\nvoid fair_queue::priority_class_group_data::push_from_idle(priority_entry& pc, const fair_queue::config& cfg) noexcept {\n    // Don't let the newcomer monopolize the disk for more than tau\n    // duration. For this estimate how many capacity units can be\n    // accumulated with the current class shares per rate resulution\n    // and scale it up to tau.\n    // On start this deviation can go to negative values, so not to\n    // introduce extra if's for that short corner case, use signed\n    // arithmetics and make sure the _accumulated value doesn't grow\n    // over signed maximum (see overflow check below)\n    pc._accumulated = std::max<signed_capacity_t>(_last_accumulated - cfg.forgiving_factor / pc._shares, pc._accumulated);\n    _children.assert_enough_capacity();\n    _children.push(&pc);\n    pc._queued = true;\n    pc._activations++;\n    wakeup(cfg);\n}\n\nbool fair_queue::priority_class_data::plug() noexcept {\n    SEASTAR_ASSERT(!_plugged);\n    _plugged = true;\n    return !_queue.empty();\n}\n\nbool fair_queue::priority_class_group_data::plug() noexcept {\n    SEASTAR_ASSERT(!_plugged);\n    _plugged = true;\n    return !_children.empty();\n}\n\nvoid fair_queue::plug_class(class_id cid) noexcept {\n    auto& pc = *_priority_classes[cid];\n    if (pc.plug()) {\n        pc.wakeup(_config);\n    }\n}\n\nvoid fair_queue::plug_class_group(unsigned index) noexcept {\n    auto& pg = *_priority_groups[index];\n    if (pg.plug()) {\n        pg.wakeup(_config);\n    }\n}\n\nvoid fair_queue::priority_entry::unplug() noexcept {\n    SEASTAR_ASSERT(_plugged);\n    _plugged = false;\n}\n\nvoid fair_queue::unplug_class(class_id cid) noexcept {\n    _priority_classes[cid]->unplug();\n}\n\nvoid fair_queue::unplug_class_group(unsigned group) noexcept {\n    _priority_groups[group]->unplug();\n}\n\nfair_queue::capacity_t fair_queue::accumulated(class_id cid) const noexcept {\n    return _priority_classes[cid]->_accumulated;\n}\n\nfair_queue::capacity_t fair_queue::pure_accumulated(class_id cid) const noexcept {\n    return _priority_classes[cid]->_pure_accumulated;\n}\n\nunsigned fair_queue::activations(class_id cid) const noexcept {\n    return _priority_classes[cid]->_activations;\n}\n\nvoid fair_queue::register_priority_class(class_id id, uint32_t shares, std::optional<unsigned> group) {\n    if (id >= _priority_classes.size()) {\n        _priority_classes.resize(id + 1);\n    } else {\n        SEASTAR_ASSERT(!_priority_classes[id]);\n    }\n\n    priority_class_group_data* pg = !group.has_value() ? &_root : _priority_groups[*group].get();\n\n    pg->reserve();\n    _priority_classes[id] = std::make_unique<priority_class_data>(shares, pg);\n    pg->_nr_children++;\n}\n\nvoid fair_queue::ensure_priority_group(unsigned index, uint32_t shares) {\n    if (index >= _priority_groups.size()) {\n        _priority_groups.resize(index + 1);\n    }\n\n    if (!_priority_groups[index]) {\n        _root.reserve();\n        _priority_groups[index] = std::make_unique<priority_class_group_data>(shares, &_root);\n        _root._nr_children++;\n    } else {\n        _priority_groups[index]->update_shares(shares);\n    }\n}\n\nvoid fair_queue::unregister_priority_class(class_id id) {\n    auto& pclass = _priority_classes[id];\n    SEASTAR_ASSERT(pclass);\n    pclass->_parent->_nr_children--;\n    pclass.reset();\n}\n\nvoid fair_queue::update_shares_for_class(class_id id, uint32_t shares) {\n    SEASTAR_ASSERT(id < _priority_classes.size());\n    auto& pc = _priority_classes[id];\n    SEASTAR_ASSERT(pc);\n    pc->update_shares(shares);\n}\n\nvoid fair_queue::queue(class_id id, fair_queue_entry& ent) noexcept {\n    priority_class_data& pc = *_priority_classes[id];\n    // We need to return a future in this function on which the caller can wait.\n    // Since we don't know which queue we will use to execute the next request - if ours or\n    // someone else's, we need a separate promise at this point.\n    if (pc._plugged) {\n        pc.wakeup(_config);\n    }\n    pc._queue.push_back(ent);\n    _queued_capacity += ent.capacity();\n}\n\nvoid fair_queue::notify_request_finished(fair_queue_entry::capacity_t cap) noexcept {\n}\n\nvoid fair_queue::notify_request_cancelled(fair_queue_entry& ent) noexcept {\n    _queued_capacity -= ent._capacity;\n    ent._capacity = 0;\n}\n\nfair_queue_entry* fair_queue::top() {\n    return _root.top();\n}\n\nfair_queue_entry* fair_queue::priority_class_group_data::top() {\n    if (!_plugged) {\n        return nullptr;\n    }\n\n    while (!_children.empty()) {\n        priority_entry& h = *_children.top();\n        auto* ent = h.top();\n        if (ent == nullptr) {\n            SEASTAR_ASSERT(h._queued);\n            h._queued = false;\n            _children.pop();\n            continue;\n        }\n\n        return ent;\n    }\n\n    return nullptr;\n}\n\nvoid fair_queue::pop_front() {\n    auto [empty, req_cap] = _root.pop_front();\n    _queued_capacity -= req_cap;\n}\n\nstd::pair<bool, fair_queue_entry::capacity_t> fair_queue::priority_class_group_data::pop_front() {\n    auto& h = *_children.top();\n    _children.pop();\n\n    auto [empty, req_cap] = h.pop_front();\n\n    _last_accumulated = std::max(h._accumulated, _last_accumulated);\n\n    // Usually the cost of request is tens to hundreeds of thousands. However, for\n    // unrestricted queue it can be as low as 2k. With large enough shares this\n    // has chances to be translated into zero cost which, in turn, will make the\n    // class show no progress and monopolize the queue.\n    auto req_cost  = std::max(req_cap / h._shares, (capacity_t)1);\n    h._accumulated += req_cost;\n\n    // signed overflow check to make push_priority_class_from_idle math work\n    SEASTAR_ASSERT(h._accumulated < std::numeric_limits<signed_capacity_t>::max() - req_cost);\n\n    if (empty) {\n        h._queued = false;\n    } else {\n        _children.push(&h);\n    }\n\n    return std::make_pair(_children.empty(), req_cap);\n}\n\n}\n"
  },
  {
    "path": "src/core/file-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/file.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <atomic>\n#include <deque>\n#include <functional>\n#include <memory>\n#include <vector>\n#include <sys/uio.h>\n\nnamespace seastar {\nclass io_queue;\n\nnamespace internal {\n\nstruct fs_info;\nclass io_sink;\n\n}\n\ntemplate <typename FileImpl>\nclass posix_file_handle_impl : public seastar::file_handle_impl {\n    int _fd;\n    std::atomic<unsigned>* _refcount;\n    dev_t _device_id;\n    open_flags _open_flags;\n    uint32_t _memory_dma_alignment;\n    uint32_t _disk_read_dma_alignment;\n    uint32_t _disk_write_dma_alignment;\n    uint32_t _disk_overwrite_dma_alignment;\n    bool _nowait_works;\n    bool _durable;\n    bool _aio_fdatasync;\npublic:\n    posix_file_handle_impl(int fd, open_flags f, std::atomic<unsigned>* refcount, dev_t device_id,\n            uint32_t memory_dma_alignment,\n            uint32_t disk_read_dma_alignment,\n            uint32_t disk_write_dma_alignment,\n            uint32_t disk_overwrite_dma_alignment,\n            bool nowait_works, bool durable, bool aio_fdatasync)\n            : _fd(fd), _refcount(refcount), _device_id(device_id), _open_flags(f)\n            , _memory_dma_alignment(memory_dma_alignment)\n            , _disk_read_dma_alignment(disk_read_dma_alignment)\n            , _disk_write_dma_alignment(disk_write_dma_alignment)\n            , _disk_overwrite_dma_alignment(disk_overwrite_dma_alignment)\n            , _nowait_works(nowait_works)\n            , _durable(durable)\n            , _aio_fdatasync(aio_fdatasync)\n    {\n    }\n    virtual ~posix_file_handle_impl();\n    posix_file_handle_impl(const posix_file_handle_impl&) = delete;\n    posix_file_handle_impl(posix_file_handle_impl&&) = delete;\n    virtual shared_ptr<file_impl> to_file() && override;\n    virtual std::unique_ptr<seastar::file_handle_impl> clone() const override;\n};\n\nclass posix_file_impl : public file_impl {\n    std::atomic<unsigned>* _refcount = nullptr;\n    const bool _nowait_works;\n    const bool _durable;\n    const bool _aio_fdatasync;\n    const dev_t _device_id;\n    io_queue& _io_queue;\n    const open_flags _open_flags;\nprotected:\n    int _fd;\n\n    posix_file_impl(int fd, open_flags, file_open_options options, dev_t device_id, const internal::fs_info& fsi);\n    posix_file_impl(int fd, open_flags, std::atomic<unsigned>* refcount, dev_t device_id,\n            uint32_t memory_dma_alignment,\n            uint32_t disk_read_dma_alignment,\n            uint32_t disk_write_dma_alignment,\n            uint32_t disk_overwrite_dma_alignment,\n            bool nowait_works, bool durable, bool aio_fdatasync);\npublic:\n    virtual ~posix_file_impl() override;\n    future<> flush() noexcept override;\n    future<struct stat> stat() noexcept override;\n    future<struct stat> statat(std::string_view name, int flags = 0) noexcept override;\n    future<> truncate(uint64_t length) noexcept override;\n    future<> discard(uint64_t offset, uint64_t length) noexcept override;\n    future<int> ioctl(uint64_t cmd, void* argp) noexcept override;\n    future<int> ioctl_short(uint64_t cmd, void* argp) noexcept override;\n    future<int> fcntl(int op, uintptr_t arg) noexcept override;\n    future<int> fcntl_short(int op, uintptr_t arg) noexcept override;\n    virtual future<> allocate(uint64_t position, uint64_t length) noexcept override;\n    future<uint64_t> size() noexcept override;\n    // close() never fails. It just reports errors and swallows them.\n    // The user must call flush() first if they care aout stable storage semantics.\n    virtual future<> close() noexcept override;\n    virtual std::unique_ptr<seastar::file_handle_impl> dup() override = 0;\n    virtual subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)> next) override;\n    virtual list_directory_generator_type experimental_list_directory() override;\n\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept override = 0;\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override = 0;\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept override = 0;\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override = 0;\n    virtual future<temporary_buffer<uint8_t>> dma_read_bulk(uint64_t offset, size_t range_size, io_intent* intent) noexcept final override;\n\n    open_flags flags() const {\n        return _open_flags;\n    }\n\n    // can be moved to private once reactor::read_directory is removed\n    static future<size_t> read_directory(int fd, char* buffer, size_t buffer_size);\n    // can be moved to private once reactor::fdatasync is removed\n    static future<> fdatasync(bool with_aio, int fd, internal::io_sink& sink);\n\nprivate:\n    void configure_io_lengths() noexcept;\n\n    /**\n     * Try to read from the given position where the previous short read has\n     * stopped. Check the EOF condition.\n     *\n     * The below code assumes the following: short reads due to I/O errors\n     * always end at address aligned to HW block boundary. Therefore if we issue\n     * a new read operation from the next position we are promised to get an\n     * error (different from EINVAL). If we've got a short read because we have\n     * reached EOF then the above read would either return a zero-length success\n     * (if the file size is aligned to HW block size) or an EINVAL error (if\n     * file length is not aligned to HW block size).\n     *\n     * @param pos offset to read from\n     * @param len number of bytes to read\n     * @param pc the IO priority class under which to queue this operation\n     *\n     * @return temporary buffer with read data or zero-sized temporary buffer if\n     *         pos is at or beyond EOF.\n     * @throw appropriate exception in case of I/O error.\n     */\n    future<temporary_buffer<uint8_t>> read_maybe_eof(uint64_t pos, size_t len, io_intent* intent);\n\n    future<size_t> read_dma_one(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept {\n        return read_dma(pos, buffer, len, intent);\n    }\nprotected:\n    future<size_t> do_write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept;\n    future<size_t> do_write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept;\n    future<size_t> do_read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept;\n    future<size_t> do_read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept;\n    template <typename FileImpl>\n    std::unique_ptr<seastar::file_handle_impl> do_dup();\n};\n\nclass posix_file_real_impl final : public posix_file_impl {\npublic:\n    posix_file_real_impl(int fd, open_flags of, file_open_options options, const internal::fs_info& fsi, dev_t device_id)\n        : posix_file_impl(fd, of, std::move(options), device_id, fsi) {}\n    posix_file_real_impl(int fd, open_flags of, std::atomic<unsigned>* refcount, dev_t device_id,\n            uint32_t memory_dma_alignment, uint32_t disk_read_dma_alignment, uint32_t disk_write_dma_alignment, uint32_t disk_overwrite_dma_alignment, bool nowait_works, bool durable, bool aio_fdatasync)\n        : posix_file_impl(fd, of, refcount, device_id, memory_dma_alignment, disk_read_dma_alignment, disk_write_dma_alignment, disk_overwrite_dma_alignment, nowait_works, durable, aio_fdatasync) {}\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual std::unique_ptr<seastar::file_handle_impl> dup() override;\n};\n\nnamespace testing {\nclass append_challenged_posix_file_test;\n}\n\n// The Linux XFS implementation is challenged wrt. append: a write that changes\n// eof will be blocked by any other concurrent AIO operation to the same file, whether\n// it changes file size or not. Furthermore, ftruncate() will also block and be blocked\n// by AIO, so attempts to game the system and call ftruncate() have to be done very carefully.\n//\n// Other Linux filesystems may have different locking rules, so this may need to be\n// adjusted for them.\nclass append_challenged_posix_file_impl final : public posix_file_impl, public enable_shared_from_this<append_challenged_posix_file_impl> {\n    // File size as a result of completed kernel operations (writes and truncates)\n    uint64_t _committed_size;\n    // File size as a result of seastar API calls\n    uint64_t _logical_size;\n    // Pending operations\n    enum class opcode {\n        invalid,\n        read,\n        write,\n        truncate,\n        flush,\n        allocate,\n    };\n    struct op {\n        opcode type;\n        uint64_t pos;\n        size_t len;\n        noncopyable_function<future<> ()> run;\n    };\n    // Queue of pending operations; processed from front to end to avoid\n    // starvation, but can issue concurrent operations.\n    std::deque<op> _q;\n    const unsigned _max_size_changing_ops = 0;\n    unsigned _current_non_size_changing_ops = 0;\n    unsigned _current_size_changing_ops = 0;\n    const bool _fsync_is_exclusive = true;\n\n    // Set when the user is closing the file\n    enum class state { open, draining, closing, closed };\n    state _closing_state = state::open;\n\n    const bool _sloppy_size = false;\n    const uint64_t _sloppy_size_hint;\n    // Fulfiled when _done and I/O is complete\n    promise<> _completed;\nprivate:\n    void commit_size(uint64_t size) noexcept;\n    bool must_run_alone(const op& candidate) const noexcept;\n    bool appending_write(const op& candidate) const noexcept;\n    bool size_changing(const op& candidate) const noexcept;\n    bool may_dispatch(const op& candidate) const noexcept;\n    void dispatch(op& candidate) noexcept;\n    void optimize_queue() noexcept;\n    void process_queue() noexcept;\n    bool may_quit() const noexcept;\n    void enqueue_op(op&& op);\n    int truncate_sync(uint64_t len) noexcept;\n    void truncate_to_logical_size();\n    template <typename... T, typename Func>\n    future<T...> enqueue(opcode type, uint64_t pos, size_t len, Func&& func) noexcept {\n        try {\n            auto pr = make_lw_shared(promise<T...>());\n            auto fut = pr->get_future();\n            auto op_func = [func = std::move(func), pr = std::move(pr)] () mutable {\n                return futurize_invoke(std::move(func)).then_wrapped([pr = std::move(pr)] (future<T...> f) mutable {\n                    f.forward_to(std::move(*pr));\n                });\n            };\n            try {\n                enqueue_op({type, pos, len, std::move(op_func)});\n                return fut;\n            } catch (...) {\n                fut.ignore_ready_future();\n                return current_exception_as_future<T...>();\n            }\n        } catch (...) {\n            return make_exception_future<T...>(std::current_exception());\n        }\n    }\npublic:\n    append_challenged_posix_file_impl(int fd, open_flags, file_open_options options, const internal::fs_info& fsi, dev_t device_id);\n    ~append_challenged_posix_file_impl() override;\n\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual std::unique_ptr<seastar::file_handle_impl> dup() override;\n\n    future<> flush() noexcept override;\n    future<struct stat> stat() noexcept override;\n    future<> truncate(uint64_t length) noexcept override;\n    future<uint64_t> size() noexcept override;\n    virtual future<> allocate(uint64_t position, uint64_t length) noexcept override;\n    future<> close() noexcept override;\n\n    friend class testing::append_challenged_posix_file_test;\n};\n\nclass file_desc;\nnamespace testing {\nshared_ptr<append_challenged_posix_file_impl>\nmake_append_challenged_posix_file(file_desc& fd, unsigned concurrency, bool fsync_is_exclusive, std::optional<size_t> sloppy_size);\n}\n\nclass blockdev_file_impl final : public posix_file_impl {\npublic:\n    blockdev_file_impl(int fd, open_flags f, file_open_options options, const internal::fs_info& fsi, dev_t device_id)\n        : posix_file_impl(fd, f, options, device_id, fsi) {}\n    blockdev_file_impl(int fd, open_flags of, std::atomic<unsigned>* refcount, dev_t device_id,\n            uint32_t memory_dma_alignment, uint32_t disk_read_dma_alignment, uint32_t disk_write_dma_alignment, uint32_t disk_overwrite_dma_alignment, bool nowait_works, bool durable, bool aio_fdatasync)\n        : posix_file_impl(fd, of, refcount, device_id, memory_dma_alignment, disk_read_dma_alignment, disk_write_dma_alignment, disk_overwrite_dma_alignment, nowait_works, durable, aio_fdatasync) {}\n\n    future<> truncate(uint64_t length) noexcept override;\n    future<> discard(uint64_t offset, uint64_t length) noexcept override;\n    future<uint64_t> size() noexcept override;\n    virtual future<> allocate(uint64_t position, uint64_t length) noexcept override;\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept override;\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept override;\n    virtual std::unique_ptr<seastar::file_handle_impl> dup() override;\n};\n\n}\n"
  },
  {
    "path": "src/core/file.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n\n#include <algorithm>\n#include <atomic>\n#include <coroutine>\n#include <deque>\n#include <functional>\n#include <memory>\n#include <optional>\n#include <vector>\n#include <seastar/util/assert.hh>\n\n#define __user /* empty */  // for xfs includes, below\n\n// Define STATX_DIOALIGN and related fields if not available from system headers (Linux < 6.1)\n#ifndef STATX_DIOALIGN\n#define STATX_DIOALIGN 0x00002000U\n// Check if struct statx has the stx_dio_mem_align field (added in Linux 6.1)\n// We need to extend the structure for older kernels/headers\n#define SEASTAR_STATX_NEEDS_DIO_FIELDS 1\n#endif\n\n#include <sys/syscall.h>\n#include <dirent.h>\n#include <linux/types.h> // for xfs, below\n#include <linux/fs.h> // BLKBSZGET\n#include <linux/major.h>\n#include <sys/ioctl.h>\n#include <unistd.h>\n#include <fcntl.h>\n#include <xfs/linux.h>\n/*\n * With package xfsprogs-devel >= 5.14.1, `fallthrough` has defined to\n * fix compilation warning in header <xfs/linux.h>,\n * (see: https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/commit/?id=df9c7d8d8f3ed0785ed83e7fd0c7ddc92cbfbe15)\n * There is a confliction with c++ keyword `fallthrough`, so undefine fallthrough here.\n */\n#undef fallthrough\n#define min min    /* prevent xfs.h from defining min() as a macro */\n#include <xfs/xfs.h>\n#undef min\n\n#include <seastar/core/align.hh>\n#include <seastar/core/internal/uname.hh>\n#include <seastar/core/internal/io_intent.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/report_exception.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/internal/magic.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/coroutine/as_future.hh>\n#include \"core/file-impl.hh\"\n#include \"core/syscall_result.hh\"\n#include \"core/thread_pool.hh\"\n\nnamespace seastar {\n\nnamespace internal {\n\nstruct alignments {\n    unsigned memory;\n    unsigned disk_read;\n    unsigned disk_write;\n    unsigned disk_overwrite;\n};\n\n// Minimum memory alignment required by posix_memalign\n// Must be power of 2 and multiple of sizeof(void*) (8 bytes on 64-bit, 4 on 32-bit)\n// Since Linux 6.1, O_DIRECT buffer addresses can use DMA alignment (as low as 4 bytes for NVMe)\n// instead of logical block size (typically 512 bytes), but we must satisfy posix_memalign's\n// requirement which is stricter on 64-bit systems (sizeof(void*) = 8 > 4)\nstatic constexpr unsigned min_memory_alignment = sizeof(void*);\n\nstruct fs_info {\n    uint32_t block_size;\n    bool append_challenged;\n    unsigned append_concurrency;\n    bool fsync_is_exclusive;\n    bool nowait_works;\n    std::optional<alignments> align;\n};\n\n};\n\nusing namespace internal::linux_abi;\n\nfile_handle::file_handle(const file_handle& x)\n        : _impl(x._impl ? x._impl->clone() : std::unique_ptr<file_handle_impl>()) {\n}\n\nfile_handle::file_handle(file_handle&& x) noexcept = default;\n\nfile_handle&\nfile_handle::operator=(const file_handle& x) {\n    return operator=(file_handle(x));\n}\n\nfile_handle&\nfile_handle::operator=(file_handle&&) noexcept = default;\n\nfile\nfile_handle::to_file() const & {\n    return file_handle(*this).to_file();\n}\n\nfile\nfile_handle::to_file() && {\n    return file(std::move(*_impl).to_file());\n}\n\nposix_file_impl::posix_file_impl(int fd, open_flags f, file_open_options options, dev_t device_id, const internal::fs_info& fsi)\n        : _nowait_works(fsi.nowait_works)\n        , _durable(options.durable)\n        , _aio_fdatasync(engine().have_aio_fdatasync())\n        , _device_id(device_id)\n        , _io_queue(engine().get_io_queue(_device_id))\n        , _open_flags(f)\n        , _fd(fd)\n{\n    if (fsi.align.has_value()) {\n        _memory_dma_alignment = fsi.align->memory;\n        _disk_read_dma_alignment = fsi.align->disk_read;\n        _disk_write_dma_alignment = fsi.align->disk_write;\n        _disk_overwrite_dma_alignment = fsi.align->disk_overwrite;\n    }\n    configure_io_lengths();\n}\n\nposix_file_impl::~posix_file_impl() {\n    if (_refcount && _refcount->fetch_add(-1, std::memory_order_relaxed) != 1) {\n        return;\n    }\n    delete _refcount;\n    if (_fd != -1) {\n        // Note: close() can be a blocking operation on NFS\n        ::close(_fd);\n    }\n}\n\nvoid posix_file_impl::configure_io_lengths() noexcept {\n    auto limits = _io_queue.get_request_limits();\n    _read_max_length = std::min<size_t>(_read_max_length, limits.max_read);\n    _write_max_length = std::min<size_t>(_write_max_length, limits.max_write);\n}\n\ntemplate <typename FileImpl>\nstd::unique_ptr<seastar::file_handle_impl>\nposix_file_impl::do_dup() {\n    if ((_open_flags & open_flags_mode_mask) != open_flags::ro) {\n        throw std::runtime_error(\"File is not read-only\");\n    }\n\n    if (!_refcount) {\n        _refcount = new std::atomic<unsigned>(1u);\n    }\n    auto ret = std::make_unique<posix_file_handle_impl<FileImpl>>(_fd, _open_flags, _refcount, _device_id,\n            _memory_dma_alignment, _disk_read_dma_alignment, _disk_write_dma_alignment, _disk_overwrite_dma_alignment,\n            _nowait_works, _durable, _aio_fdatasync);\n    _refcount->fetch_add(1, std::memory_order_relaxed);\n    return ret;\n}\n\nposix_file_impl::posix_file_impl(int fd, open_flags f, std::atomic<unsigned>* refcount, dev_t device_id,\n        uint32_t memory_dma_alignment,\n        uint32_t disk_read_dma_alignment,\n        uint32_t disk_write_dma_alignment,\n        uint32_t disk_overwrite_dma_alignment,\n        bool nowait_works, bool durable, bool aio_fdatasync)\n        : _refcount(refcount)\n        , _nowait_works(nowait_works)\n        , _durable(durable)\n        , _aio_fdatasync(aio_fdatasync)\n        , _device_id(device_id)\n        , _io_queue(engine().get_io_queue(_device_id))\n        , _open_flags(f)\n        , _fd(fd) {\n    _memory_dma_alignment = memory_dma_alignment;\n    _disk_read_dma_alignment = disk_read_dma_alignment;\n    _disk_write_dma_alignment = disk_write_dma_alignment;\n    _disk_overwrite_dma_alignment = disk_overwrite_dma_alignment;\n    configure_io_lengths();\n}\n\nfuture<>\nposix_file_impl::flush() noexcept {\n    if (!_durable || ((_open_flags & open_flags::dsync) != open_flags{})) {\n        // If the file is opened with open_flags::dsync, all writes are already\n        // fdatasync-ed to the disk, so we don't need to fdatasync again.\n        return make_ready_future<>();\n    }\n\n    reactor::io_stats::local().fsyncs++;\n    return fdatasync(_aio_fdatasync, _fd, _io_queue.sink());\n}\n\nfuture<>\nreactor::fdatasync(int fd) noexcept {\n    if (_cfg.bypass_fsync) {\n        return make_ready_future<>();\n    }\n    _io_stats.fsyncs++;\n    return posix_file_impl::fdatasync(_cfg.have_aio_fsync, fd, _io_sink);\n}\n\nfuture<> posix_file_impl::fdatasync(bool with_aio, int fd, internal::io_sink& sink) {\n    if (with_aio) {\n        // Does not go through the I/O queue, but has to be deleted\n        struct fsync_io_desc final : public io_completion {\n            promise<> _pr;\n        public:\n            virtual void complete(size_t res) noexcept override {\n                _pr.set_value();\n                delete this;\n            }\n\n            virtual void set_exception(std::exception_ptr eptr) noexcept override {\n                _pr.set_exception(std::move(eptr));\n                delete this;\n            }\n\n            future<> get_future() {\n                return _pr.get_future();\n            }\n        };\n\n        auto desc = new fsync_io_desc;\n        auto fut = desc->get_future();\n        auto req = internal::io_request::make_fdatasync(fd);\n        sink.submit(desc, std::move(req));\n        co_await std::move(fut);\n        co_return;\n    }\n    syscall_result<int> sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [fd] {\n        return wrap_syscall<int>(::fdatasync(fd));\n    });\n    sr.throw_if_error();\n}\n\nfuture<struct stat>\nposix_file_impl::stat() noexcept {\n    auto ret = co_await engine()._thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [fd = _fd] {\n        struct stat st;\n        auto ret = ::fstat(fd, &st);\n        return wrap_syscall(ret, st);\n    });\n    ret.throw_if_error();\n    co_return ret.extra;\n}\n\nfuture<struct stat>\nposix_file_impl::statat(std::string_view name, int flags) noexcept {\n    auto ret = co_await engine()._thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [fd = _fd, name = sstring(name), flags] {\n        struct stat st;\n        auto ret = ::fstatat(fd, name.data(), &st, flags);\n        return wrap_syscall(ret, st);\n    });\n    ret.throw_if_error();\n    co_return ret.extra;\n}\n\nfuture<>\nposix_file_impl::truncate(uint64_t length) noexcept {\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, length] {\n        return wrap_syscall<int>(::ftruncate(_fd, length));\n    });\n    sr.throw_if_error();\n}\n\nfuture<int>\nposix_file_impl::ioctl(uint64_t cmd, void* argp) noexcept {\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, cmd, argp] () mutable {\n        return wrap_syscall<int>(::ioctl(_fd, cmd, argp));\n    });\n    sr.throw_if_error();\n    // Some ioctls require to return a positive integer back.\n    co_return sr.result;\n}\n\nfuture<int>\nposix_file_impl::ioctl_short(uint64_t cmd, void* argp) noexcept {\n    int ret = ::ioctl(_fd, cmd, argp);\n    if (ret == -1) {\n        return make_exception_future<int>(\n                std::system_error(errno, std::system_category(), \"ioctl failed\"));\n    }\n    return make_ready_future<int>(ret);\n}\n\nfuture<int>\nposix_file_impl::fcntl(int op, uintptr_t arg) noexcept {\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, op, arg] () mutable {\n        return wrap_syscall<int>(::fcntl(_fd, op, arg));\n    });\n    sr.throw_if_error();\n    // Some fcntls require to return a positive integer back.\n    co_return sr.result;\n}\n\nfuture<int>\nposix_file_impl::fcntl_short(int op, uintptr_t arg) noexcept {\n    int ret = ::fcntl(_fd, op, arg);\n    if (ret == -1) {\n        return make_exception_future<int>(\n                std::system_error(errno, std::system_category(), \"fcntl failed\"));\n    }\n    return make_ready_future<int>(ret);\n}\n\nfuture<>\nposix_file_impl::discard(uint64_t offset, uint64_t length) noexcept {\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, offset, length] () mutable {\n        return wrap_syscall<int>(::fallocate(_fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,\n            offset, length));\n    });\n    sr.throw_if_error();\n}\n\nfuture<>\nposix_file_impl::allocate(uint64_t position, uint64_t length) noexcept {\n#ifdef FALLOC_FL_ZERO_RANGE\n    // FALLOC_FL_ZERO_RANGE is fairly new, so don't fail if it's not supported.\n    static bool supported = true;\n    if (!supported) {\n        co_return;\n    }\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, position, length] () mutable {\n        auto ret = ::fallocate(_fd, FALLOC_FL_ZERO_RANGE|FALLOC_FL_KEEP_SIZE, position, length);\n        if (ret == -1 && errno == EOPNOTSUPP) {\n            ret = 0;\n            supported = false; // Racy, but harmless.  At most we issue an extra call or two.\n        }\n        return wrap_syscall<int>(ret);\n    });\n    sr.throw_if_error();\n#else\n    return make_ready_future<>();\n#endif\n}\n\nfuture<uint64_t>\nposix_file_impl::size() noexcept {\n    auto r = ::lseek(_fd, 0, SEEK_END);\n    if (r == -1) {\n        return make_exception_future<uint64_t>(std::system_error(errno, std::system_category()));\n    }\n    return make_ready_future<uint64_t>(r);\n}\n\nfuture<>\nposix_file_impl::close() noexcept {\n    if (_fd == -1) {\n        seastar_logger.warn(\"double close() detected, contact support\");\n        return make_ready_future<>();\n    }\n    auto fd = _fd;\n    _fd = -1;  // Prevent a concurrent close (which is illegal) from closing another file's fd\n    if (_refcount && _refcount->fetch_add(-1, std::memory_order_relaxed) != 1) {\n        _refcount = nullptr;\n        return make_ready_future<>();\n    }\n    delete _refcount;\n    _refcount = nullptr;\n    auto closed = make_ready_future<syscall_result<int>>(0, 0);\n    if ((_open_flags & open_flags_mode_mask) == open_flags::ro) {\n        closed = futurize_invoke([fd] () noexcept {\n            return wrap_syscall<int>(::close(fd));\n        });\n    } else {\n        closed = std::invoke([fd] () noexcept {\n            try {\n                return engine()._thread_pool->submit<syscall_result<int>>(\n                        internal::thread_pool_submit_reason::file_operation, [fd] {\n                    return wrap_syscall<int>(::close(fd));\n                });\n            } catch (...) {\n                report_exception(\"Running ::close() in reactor thread, submission failed with exception\", std::current_exception());\n                return make_ready_future<syscall_result<int>>(wrap_syscall<int>(::close(fd)));\n            }\n        });\n    }\n    return closed.then([] (syscall_result<int> sr) {\n      try {\n        sr.throw_if_error();\n      } catch (...) {\n        report_exception(\"close() syscall failed\", std::current_exception());\n      }\n    });\n}\n\nfuture<uint64_t>\nblockdev_file_impl::size() noexcept {\n    auto ret = co_await engine()._thread_pool->submit<syscall_result_extra<size_t>>(\n            internal::thread_pool_submit_reason::file_operation, [this] {\n        uint64_t size;\n        int ret = ::ioctl(_fd, BLKGETSIZE64, &size);\n        return wrap_syscall(ret, size);\n    });\n    ret.throw_if_error();\n    co_return ret.extra;\n}\n\nstatic std::optional<directory_entry_type> dirent_type(const linux_dirent64& de) {\n    std::optional<directory_entry_type> type;\n    switch (de.d_type) {\n    case DT_BLK:\n        type = directory_entry_type::block_device;\n        break;\n    case DT_CHR:\n        type = directory_entry_type::char_device;\n        break;\n    case DT_DIR:\n        type = directory_entry_type::directory;\n        break;\n    case DT_FIFO:\n        type = directory_entry_type::fifo;\n        break;\n    case DT_REG:\n        type = directory_entry_type::regular;\n        break;\n    case DT_LNK:\n        type = directory_entry_type::link;\n        break;\n    case DT_SOCK:\n        type = directory_entry_type::socket;\n        break;\n    default:\n        // unknown, ignore\n        ;\n    }\n    return type;\n}\n\nfuture<size_t> posix_file_impl::read_directory(int fd, char* buffer, size_t buffer_size) {\n    syscall_result<long> ret = co_await engine()._thread_pool->submit<syscall_result<long>>(\n            internal::thread_pool_submit_reason::file_operation, [fd, buffer, buffer_size] () {\n        auto ret = ::syscall(__NR_getdents64, fd, reinterpret_cast<linux_dirent64*>(buffer), buffer_size);\n        return wrap_syscall(ret);\n    });\n    ret.throw_if_error();\n    co_return ret.result;\n}\n\nfuture<size_t> reactor::read_directory(int fd, char* buffer, size_t buffer_size) {\n    return posix_file_impl::read_directory(fd, buffer, buffer_size);\n}\n\nlist_directory_generator_type make_list_directory_generator(int fd) {\n    temporary_buffer<char> buf(8192);\n\n    while (true) {\n        auto size = co_await posix_file_impl::read_directory(fd, buf.get_write(), buf.size());\n        if (size == 0) {\n            co_return;\n        }\n\n        for (const char* b = buf.get(); b < buf.get() + size; ) {\n            const auto de = reinterpret_cast<const linux_dirent64*>(b);\n            b += de->d_reclen;\n            sstring name(de->d_name);\n            if (name == \".\" || name == \"..\") {\n                continue;\n            }\n            std::optional<directory_entry_type> type = dirent_type(*de);\n            directory_entry ret{std::move(name), type};\n            co_yield ret;\n        }\n    }\n}\n\nlist_directory_generator_type posix_file_impl::experimental_list_directory() {\n    return make_list_directory_generator(_fd);\n}\n\nsubscription<directory_entry>\nposix_file_impl::list_directory(std::function<future<> (directory_entry de)> next) {\n    static constexpr size_t buffer_size = 8192;\n    struct work {\n        stream<directory_entry> s;\n        unsigned current = 0;\n        unsigned total = 0;\n        bool eof = false;\n        int error = 0;\n        char buffer[buffer_size];\n    };\n\n    // While it would be natural to use fdopendir()/readdir(),\n    // our syscall thread pool doesn't support malloc(), which is\n    // required for this to work.  So resort to using getdents()\n    // instead.\n\n    auto w = make_lw_shared<work>();\n    auto ret = w->s.listen(std::move(next));\n    // List the directory asynchronously in the background.\n    // Caller synchronizes using the returned subscription.\n    (void)w->s.started().then([w, this] {\n        auto eofcond = [w] { return w->eof; };\n        return do_until(eofcond, [w, this] {\n            if (w->current == w->total) {\n                return read_directory(_fd, w->buffer, buffer_size).then([w] (auto size) {\n                    if (size == 0) {\n                        w->eof = true;\n                    } else {\n                        w->current = 0;\n                        w->total = size;\n                    }\n                });\n            }\n            auto start = w->buffer + w->current;\n            auto de = reinterpret_cast<linux_dirent64*>(start);\n            w->current += de->d_reclen;\n            sstring name = de->d_name;\n            if (name == \".\" || name == \"..\") {\n                return make_ready_future<>();\n            }\n            std::optional<directory_entry_type> type = dirent_type(*de);\n            return w->s.produce({std::move(name), type});\n        });\n    }).then([w] {\n        w->s.close();\n    }).handle_exception([] (std::exception_ptr ignored) {});\n    return ret;\n}\n\nfuture<size_t>\nposix_file_impl::do_write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept {\n    auto req = internal::io_request::make_write(_fd, pos, buffer, len, _nowait_works);\n    return _io_queue.submit_io_write(len, std::move(req), intent);\n}\n\nfuture<size_t>\nposix_file_impl::do_write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    auto len = internal::sanitize_iovecs(iov, _disk_write_dma_alignment);\n    auto req = internal::io_request::make_writev(_fd, pos, iov, _nowait_works);\n    return _io_queue.submit_io_write(len, std::move(req), intent, std::move(iov));\n}\n\nfuture<size_t>\nposix_file_impl::do_read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept {\n    auto req = internal::io_request::make_read(_fd, pos, buffer, len, _nowait_works);\n    return _io_queue.submit_io_read(len, std::move(req), intent);\n}\n\nfuture<size_t>\nposix_file_impl::do_read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    auto len = internal::sanitize_iovecs(iov, _disk_read_dma_alignment);\n    auto req = internal::io_request::make_readv(_fd, pos, iov, _nowait_works);\n    return _io_queue.submit_io_read(len, std::move(req), intent, std::move(iov));\n}\n\nfuture<size_t>\nposix_file_real_impl::write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept {\n    return posix_file_impl::do_write_dma(pos, buffer, len, intent);\n}\n\nfuture<size_t>\nposix_file_real_impl::write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    return posix_file_impl::do_write_dma(pos, std::move(iov), intent);\n}\n\nfuture<size_t>\nposix_file_real_impl::read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept {\n    return posix_file_impl::do_read_dma(pos, buffer, len, intent);\n}\n\nfuture<size_t>\nposix_file_real_impl::read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    return posix_file_impl::do_read_dma(pos, std::move(iov), intent);\n}\n\nstd::unique_ptr<seastar::file_handle_impl>\nposix_file_real_impl::dup() {\n    return posix_file_impl::do_dup<posix_file_real_impl>();\n}\n\nfuture<temporary_buffer<uint8_t>>\nposix_file_impl::dma_read_bulk(uint64_t offset, size_t range_size, io_intent* intent) noexcept {\n    auto front = offset & (_disk_read_dma_alignment - 1);\n    offset -= front;\n    range_size += front;\n\n    temporary_buffer<uint8_t> buf = temporary_buffer<uint8_t>::aligned(\n            _memory_dma_alignment, align_up(range_size, size_t(_disk_read_dma_alignment)));\n\n    //\n    // First, try to read directly into the buffer. Most of the reads will\n    // end here.\n    //\n    size_t pos = co_await read_dma_one(offset, buf.get_write(), buf.size(), intent);\n\n    //\n    // If we haven't read all required data at once -\n    // start read-copy sequence. We can't continue with direct reads\n    // into the previously allocated buffer here since we have to ensure\n    // the aligned read length and thus the aligned destination buffer\n    // size.\n    //\n    // The copying will actually take place only if there was a HW glitch.\n    // In EOF case or in case of a persistent I/O error the only overhead is\n    // an extra allocation.\n    //\n    while (pos < range_size) {\n        auto buf1 = co_await read_maybe_eof(offset + pos, range_size - pos, intent);\n        if (!buf1.size()) {\n            break;\n        }\n\n        auto to_copy = std::min(buf.size() - pos, buf1.size());\n        std::memcpy(buf.get_write() + pos, buf1.get(), to_copy);\n        pos += to_copy;\n    }\n    //\n    // If we are here we are promised to have read some bytes beyond\n    // \"front\" so we may trim straight away.\n    //\n    if (pos > front) {\n        buf.trim(pos);\n        buf.trim_front(front);\n    } else {\n        buf.trim(0);\n    }\n    co_return std::move(buf);\n}\n\nfuture<temporary_buffer<uint8_t>>\nposix_file_impl::read_maybe_eof(uint64_t pos, size_t len, io_intent* intent) {\n    //\n    // We have to allocate a new aligned buffer to make sure we don't get\n    // an EINVAL error due to unaligned destination buffer.\n    //\n    temporary_buffer<uint8_t> buf = temporary_buffer<uint8_t>::aligned(\n               _memory_dma_alignment, align_up(len, size_t(_disk_read_dma_alignment)));\n\n    // try to read a single bulk from the given position\n    auto dst = buf.get_write();\n    auto buf_size = buf.size();\n    try {\n        size_t size = co_await read_dma_one(pos, dst, buf_size, intent);\n        buf.trim(size);\n\n        co_return std::move(buf);\n    } catch (std::system_error& e) {\n        //\n        // TODO: implement a non-trowing file_impl::dma_read() interface to\n        //       avoid the exceptions throwing in a good flow completely.\n        //       Otherwise for users that don't want to care about the\n        //       underlying file size and preventing the attempts to read\n        //       bytes beyond EOF there will always be at least one\n        //       exception throwing at the file end for files with unaligned\n        //       length.\n        //\n        if (e.code().value() == EINVAL) {\n            buf.trim(0);\n            co_return std::move(buf);\n        } else {\n            throw;\n        }\n    }\n}\n\nstatic bool blockdev_gen_nowait_works = internal::kernel_uname().whitelisted({\"4.13\"});\nstatic bool blockdev_md_nowait_works = internal::kernel_uname().whitelisted({\"5.17\"});\n\nstatic bool blockdev_nowait_works(dev_t device_id) {\n    if (major(device_id) == MD_MAJOR) {\n        return blockdev_md_nowait_works;\n    }\n\n    return blockdev_gen_nowait_works;\n}\n\nfuture<>\nblockdev_file_impl::truncate(uint64_t length) noexcept {\n    return make_ready_future<>();\n}\n\nfuture<>\nblockdev_file_impl::discard(uint64_t offset, uint64_t length) noexcept {\n    auto sr = co_await engine()._thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [this, offset, length] () mutable {\n        uint64_t range[2] { offset, length };\n        return wrap_syscall<int>(::ioctl(_fd, BLKDISCARD, &range));\n    });\n    sr.throw_if_error();\n}\n\nfuture<>\nblockdev_file_impl::allocate(uint64_t position, uint64_t length) noexcept {\n    // nothing to do for block device\n    return make_ready_future<>();\n}\n\nfuture<size_t>\nblockdev_file_impl::write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept {\n    return posix_file_impl::do_write_dma(pos, buffer, len, intent);\n}\n\nfuture<size_t>\nblockdev_file_impl::write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    return posix_file_impl::do_write_dma(pos, std::move(iov), intent);\n}\n\nfuture<size_t>\nblockdev_file_impl::read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept {\n    return posix_file_impl::do_read_dma(pos, buffer, len, intent);\n}\n\nfuture<size_t>\nblockdev_file_impl::read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    return posix_file_impl::do_read_dma(pos, std::move(iov), intent);\n}\n\nstd::unique_ptr<seastar::file_handle_impl>\nblockdev_file_impl::dup() {\n    return posix_file_impl::do_dup<blockdev_file_impl>();\n}\n\nappend_challenged_posix_file_impl::append_challenged_posix_file_impl(int fd, open_flags f, file_open_options options, const internal::fs_info& fsi, dev_t device_id)\n        : posix_file_impl(fd, f, options, device_id, fsi)\n        , _max_size_changing_ops(fsi.append_concurrency)\n        , _fsync_is_exclusive(fsi.fsync_is_exclusive)\n        , _sloppy_size(options.sloppy_size)\n        , _sloppy_size_hint(align_up<uint64_t>(options.sloppy_size_hint, _disk_write_dma_alignment))\n{\n    auto r = ::lseek(fd, 0, SEEK_END);\n    throw_system_error_on(r == -1);\n    _committed_size = _logical_size = r;\n}\n\nappend_challenged_posix_file_impl::~append_challenged_posix_file_impl() {\n    // If the file has not been closed we risk having running tasks\n    // that will try to access freed memory.\n    //\n    // It is safe to destory it if nothing is queued.\n    // Note that posix_file_impl::~posix_file_impl auto-closes the file descriptor.\n    SEASTAR_ASSERT(_q.empty() && (_logical_size == _committed_size || _closing_state == state::closed));\n}\n\nbool\nappend_challenged_posix_file_impl::must_run_alone(const op& candidate) const noexcept {\n    // checks if candidate is a non-write, size-changing operation.\n    return (candidate.type == opcode::truncate)\n            || (candidate.type == opcode::allocate)\n            || (candidate.type == opcode::flush && (_fsync_is_exclusive || _sloppy_size));\n}\n\nbool\nappend_challenged_posix_file_impl::appending_write(const op& candidate) const noexcept {\n    return (candidate.type == opcode::write) &&\n            (candidate.pos + candidate.len > _committed_size);\n}\n\nbool\nappend_challenged_posix_file_impl::size_changing(const op& candidate) const noexcept {\n    return appending_write(candidate) || must_run_alone(candidate);\n}\n\nbool\nappend_challenged_posix_file_impl::may_dispatch(const op& candidate) const noexcept {\n    if (size_changing(candidate)) {\n        return !_current_size_changing_ops && !_current_non_size_changing_ops;\n    } else {\n        return !_current_size_changing_ops;\n    }\n}\n\nvoid\nappend_challenged_posix_file_impl::dispatch(op& candidate) noexcept {\n    unsigned* op_counter = size_changing(candidate)\n            ? &_current_size_changing_ops : &_current_non_size_changing_ops;\n    ++*op_counter;\n    // FIXME: future is discarded\n    (void)candidate.run().then([me = shared_from_this(), op_counter] {\n        --*op_counter;\n        me->process_queue();\n    });\n}\n\nint append_challenged_posix_file_impl::truncate_sync(uint64_t length) noexcept {\n    int r = ::ftruncate(_fd, length);\n    if (r != -1) {\n        _committed_size = length;\n    }\n    return r;\n}\n\nvoid append_challenged_posix_file_impl::truncate_to_logical_size() {\n    auto r = truncate_sync(_logical_size);\n    if (r == -1) {\n        throw std::system_error(errno, std::system_category(), \"truncate\");\n    }\n}\n\n// If we have a bunch of size-extending writes in the queue,\n// issue an ftruncate() extending the file size, so they can\n// be issued concurrently.\nvoid\nappend_challenged_posix_file_impl::optimize_queue() noexcept {\n    if (_current_non_size_changing_ops || _current_size_changing_ops) {\n        // Can't issue an ftruncate() if something is going on\n        return;\n    }\n    auto speculative_size = _committed_size;\n    unsigned n_appending_writes = 0;\n    for (const auto& op : _q) {\n        // stop calculating speculative size after a non-write, size-changing\n        // operation is found to prevent an useless truncate from being issued.\n        if (must_run_alone(op)) {\n            break;\n        }\n\n        if (appending_write(op)) {\n            speculative_size = std::max(speculative_size, op.pos + op.len);\n            ++n_appending_writes;\n        }\n    }\n    if (n_appending_writes > _max_size_changing_ops\n            || (n_appending_writes && _sloppy_size)) {\n        if (_sloppy_size) {\n          if (!_committed_size) {\n            speculative_size = std::max(speculative_size, _sloppy_size_hint);\n          } else if (speculative_size < 2 * _committed_size) {\n            speculative_size = align_up<uint64_t>(2 * _committed_size, _disk_write_dma_alignment);\n          }\n        }\n        // We're all alone, so issuing the ftruncate() in the reactor\n        // thread won't block us.\n        //\n        // Issuing it in the syscall thread is too slow; this can happen\n        // every several ops, and the syscall thread latency can be very\n        // high.\n\n        truncate_sync(speculative_size);\n        // If we failed, the next write will pick it up.\n    }\n}\n\nvoid\nappend_challenged_posix_file_impl::process_queue() noexcept {\n    optimize_queue();\n    while (!_q.empty() && may_dispatch(_q.front())) {\n        op candidate = std::move(_q.front());\n        _q.pop_front();\n        dispatch(candidate);\n    }\n    if (may_quit()) {\n        _completed.set_value();\n        _closing_state = state::closing; // prevents _completed to be signaled again in case of recursion\n    }\n}\n\nvoid\nappend_challenged_posix_file_impl::enqueue_op(op&& op) {\n    _q.push_back(std::move(op));\n    process_queue();\n}\n\nbool\nappend_challenged_posix_file_impl::may_quit() const noexcept {\n    return _closing_state == state::draining && _q.empty() && !_current_non_size_changing_ops &&\n           !_current_size_changing_ops;\n}\n\nvoid\nappend_challenged_posix_file_impl::commit_size(uint64_t size) noexcept {\n    _committed_size = std::max(size, _committed_size);\n    _logical_size = std::max(size, _logical_size);\n}\n\nfuture<size_t>\nappend_challenged_posix_file_impl::read_dma(uint64_t pos, void* buffer, size_t len, io_intent* intent) noexcept {\n    if (pos >= _logical_size) {\n        // yield() avoids tail recursion\n        return yield().then([] {\n            return size_t(0);\n        });\n    }\n    len = std::min(pos + len, align_up<uint64_t>(_logical_size, _disk_read_dma_alignment)) - pos;\n    internal::intent_reference iref(intent);\n    return enqueue<size_t>(\n        opcode::read,\n        pos,\n        len,\n        [this, pos, buffer, len, iref = std::move(iref)] () mutable {\n            return posix_file_impl::do_read_dma(pos, buffer, len, iref.retrieve());\n        }\n    );\n}\n\nfuture<size_t>\nappend_challenged_posix_file_impl::read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    if (pos >= _logical_size) {\n        // yield() avoids tail recursion\n        return yield().then([] {\n            return size_t(0);\n        });\n    }\n    size_t len = 0;\n    auto i = iov.begin();\n    auto aligned_logical_size = align_up<uint64_t>(_logical_size, _disk_read_dma_alignment);\n    while (i != iov.end() && pos + len + i->iov_len <= aligned_logical_size) {\n        len += i++->iov_len;\n    }\n    if (i != iov.end()) {\n        auto last_len = pos + len + i->iov_len - aligned_logical_size;\n        if (last_len) {\n            i++->iov_len = last_len;\n        }\n        iov.erase(i, iov.end());\n    }\n    internal::intent_reference iref(intent);\n    return enqueue<size_t>(\n        opcode::read,\n        pos,\n        len,\n        [this, pos, iov = std::move(iov), iref = std::move(iref)] () mutable {\n            return posix_file_impl::do_read_dma(pos, std::move(iov), iref.retrieve());\n        }\n    );\n}\n\nfuture<size_t>\nappend_challenged_posix_file_impl::write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* intent) noexcept {\n    internal::intent_reference iref(intent);\n    return enqueue<size_t>(\n        opcode::write,\n        pos,\n        len,\n        [this, pos, buffer, len, iref = std::move(iref)] {\n            return posix_file_impl::do_write_dma(pos, buffer, len, iref.retrieve()).then([this, pos] (size_t ret) {\n                commit_size(pos + ret);\n                return make_ready_future<size_t>(ret);\n            });\n        }\n    );\n}\n\nfuture<size_t>\nappend_challenged_posix_file_impl::write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n    auto len = internal::iovec_len(iov);\n    internal::intent_reference iref(intent);\n    return enqueue<size_t>(\n        opcode::write,\n        pos,\n        len,\n        [this, pos, iov = std::move(iov), iref = std::move(iref)] () mutable {\n            return posix_file_impl::do_write_dma(pos, std::move(iov), iref.retrieve()).then([this, pos] (size_t ret) {\n                commit_size(pos + ret);\n                return make_ready_future<size_t>(ret);\n            });\n        }\n    );\n}\n\nfuture<>\nappend_challenged_posix_file_impl::flush() noexcept {\n    if ((!_sloppy_size || _logical_size == _committed_size) && !_fsync_is_exclusive) {\n        // FIXME: determine if flush can block concurrent reads or writes\n        return posix_file_impl::flush();\n    } else {\n        return enqueue<>(\n            opcode::flush,\n            0,\n            0,\n            [this] () {\n                if (_logical_size != _committed_size) {\n                    // We're all alone, so can truncate in reactor thread\n                    truncate_to_logical_size();\n                }\n                return posix_file_impl::flush();\n            }\n        );\n    }\n}\n\nfuture<>\nappend_challenged_posix_file_impl::allocate(uint64_t position, uint64_t length) noexcept {\n    return enqueue<>(\n            opcode::allocate,\n            position,\n            length,\n            [this, position, length] () {\n                return posix_file_impl::allocate(position, length);\n            }\n        );\n}\n\nfuture<struct stat>\nappend_challenged_posix_file_impl::stat() noexcept {\n    // FIXME: can this conflict with anything?\n    auto stat = co_await posix_file_impl::stat();\n    stat.st_size = _logical_size;\n    co_return stat;\n}\n\nfuture<>\nappend_challenged_posix_file_impl::truncate(uint64_t length) noexcept {\n    return enqueue<>(\n        opcode::truncate,\n        length,\n        0,\n        [this, length] () mutable {\n            return posix_file_impl::truncate(length).then([this, length] () mutable {\n                _committed_size = _logical_size = length;\n            });\n        }\n    );\n}\n\nfuture<uint64_t>\nappend_challenged_posix_file_impl::size() noexcept {\n    return make_ready_future<size_t>(_logical_size);\n}\n\nstd::unique_ptr<seastar::file_handle_impl>\nappend_challenged_posix_file_impl::dup() {\n    throw std::runtime_error(\"File is not read-only\");\n    return nullptr;\n}\n\nfuture<>\nappend_challenged_posix_file_impl::close() noexcept {\n    // Caller should have drained all pending I/O\n    _closing_state = state::draining;\n    process_queue();\n    return _completed.get_future().then([this] {\n        if (_logical_size != _committed_size) {\n            truncate_to_logical_size();\n        }\n    }).then_wrapped([this] (future<> f) {\n        if (f.failed()) {\n            report_exception(\"Closing append_challenged_posix_file_impl failed.\", f.get_exception());\n        }\n        return posix_file_impl::close().finally([this] { _closing_state = state::closed; });\n    });\n}\n\ntemplate <typename FileImpl>\nposix_file_handle_impl<FileImpl>::~posix_file_handle_impl() {\n    if (_refcount && _refcount->fetch_add(-1, std::memory_order_relaxed) == 1) {\n        ::close(_fd);\n        delete _refcount;\n    }\n}\n\ntemplate <typename FileImpl>\nstd::unique_ptr<seastar::file_handle_impl>\nposix_file_handle_impl<FileImpl>::clone() const {\n    auto ret = std::make_unique<posix_file_handle_impl<FileImpl>>(_fd, _open_flags, _refcount, _device_id,\n            _memory_dma_alignment, _disk_read_dma_alignment, _disk_write_dma_alignment, _disk_overwrite_dma_alignment, _nowait_works, _durable, _aio_fdatasync);\n    if (_refcount) {\n        _refcount->fetch_add(1, std::memory_order_relaxed);\n    }\n    return ret;\n}\n\ntemplate <typename FileImpl>\nshared_ptr<file_impl>\nposix_file_handle_impl<FileImpl>::to_file() && {\n    auto ret = ::seastar::make_shared<FileImpl>(_fd, _open_flags, _refcount, _device_id,\n            _memory_dma_alignment, _disk_read_dma_alignment, _disk_write_dma_alignment, _disk_overwrite_dma_alignment, _nowait_works, _durable, _aio_fdatasync);\n    _fd = -1;\n    _refcount = nullptr;\n    return ret;\n}\n\n// Some kernels can append to xfs filesystems, some cannot; determine\n// from kernel version.\nstatic\nunsigned\nxfs_concurrency_from_kernel_version() {\n    // try to see if this is a mainline kernel with xfs append fixed (3.15+)\n    // or a RHEL kernel with the backported fix (3.10.0-325.el7+)\n    if (internal::kernel_uname().whitelisted({\"3.15\", \"3.10.0-325.el7\"})) {\n            // Can append, but not concurrently\n            return 1;\n    }\n    // Cannot append at all; need ftrucnate().\n    return 0;\n}\n\nnamespace {\n\n// Query DIO memory alignment using statx (kernel 6.1+)\n// Returns the optimal memory buffer alignment for this file descriptor,\n// or std::nullopt if statx doesn't support STATX_DIOALIGN\nstd::optional<size_t>\nquery_statx_mem_align(int fd) {\n#ifndef SEASTAR_STATX_NEEDS_DIO_FIELDS\n    // Only call statx if we have the required struct fields\n    struct statx stx;\n    if (syscall(__NR_statx, fd, \"\", AT_EMPTY_PATH, STATX_DIOALIGN, &stx) == 0) {\n        if (stx.stx_mask & STATX_DIOALIGN) {\n            // Use kernel-reported memory alignment (stx_dio_mem_align)\n            // Field is available in Linux 6.1+ headers\n            return stx.stx_dio_mem_align;\n        }\n    }\n#else\n    // Headers don't have the stx_dio_mem_align field, so we can't use it\n    // even if the kernel supports it. We'll rely on other fallbacks.\n#endif\n    return std::nullopt;\n}\n\n// Query device-level alignment properties (common to all filesystems)\nstruct device_alignment_info {\n    std::optional<unsigned> memory_alignment;      // from statx\n    std::optional<unsigned> physical_block_size;   // from io_queue override\n};\n\ndevice_alignment_info query_device_alignment_info(int fd, dev_t device_id) {\n    device_alignment_info info;\n\n    // Query statx for DIO memory alignment (kernel 6.1+)\n    info.memory_alignment = query_statx_mem_align(fd);\n\n    // Check for physical_block_size override from io_properties.yaml\n    // Note: I/O queues may not be registered for all devices (e.g., tmpfs, test environments)\n    if (auto* io_queue = engine().try_get_io_queue(device_id)) {\n        info.physical_block_size = io_queue->physical_block_size();\n    }\n\n    return info;\n}\n\n// Unified function for all filesystem alignment calculations\ninternal::alignments filesystem_alignments(\n    int fd,\n    dev_t device_id,\n    unsigned block_size,\n    unsigned long fs_type  // from statfs.f_type\n) {\n    auto device_info = query_device_alignment_info(fd, device_id);\n\n    // Initialize with generic defaults for all filesystems\n    internal::alignments align = {\n        .memory = std::max(device_info.memory_alignment.value_or(4096), internal::min_memory_alignment),\n        .disk_read = 512,  // Linux O_DIRECT minimum\n        .disk_write = block_size,\n        .disk_overwrite = block_size,\n    };\n\n    // Override with filesystem-specific alignments if available\n    if (fs_type == internal::fs_magic::xfs) {\n        dioattr da;\n        if (::ioctl(fd, XFS_IOC_DIOINFO, &da) == 0) {\n            static bool xfs_with_relaxed_overwrite_alignment = internal::kernel_uname().whitelisted({\"5.12\"});\n\n            // Override with XFS-specific values\n            align.memory = std::max(device_info.memory_alignment.value_or(da.d_mem), internal::min_memory_alignment);\n            align.disk_read = da.d_miniosz;\n            align.disk_write = std::max<unsigned>(da.d_miniosz, block_size);\n            align.disk_overwrite = xfs_with_relaxed_overwrite_alignment ? da.d_miniosz : align.disk_write;\n        }\n    }\n\n    // Common: apply physical_block_size override for all filesystems\n    // This ensures we avoid hardware read-modify-write regardless of filesystem\n    if (device_info.physical_block_size) {\n        align.disk_write = std::max<unsigned>(align.disk_write, *device_info.physical_block_size);\n        align.disk_overwrite = std::max<unsigned>(align.disk_overwrite, *device_info.physical_block_size);\n    }\n\n    return align;\n}\n\n// Query block device alignment properties using ioctl and statx\ninternal::alignments\nblkdev_alignments(int fd, dev_t device_id) {\n    // Query logical block size (smallest addressable unit from hardware)\n    // Use BLKSSZGET (not BLKBSZGET) - BLKBSZGET returns the filesystem's\n    // buffered I/O block size, while BLKSSZGET returns the actual hardware\n    // sector size that the kernel enforces for O_DIRECT alignment\n    size_t logical_block_size;\n    if (auto ret = ::ioctl(fd, BLKSSZGET, &logical_block_size); ret == -1) {\n        throw std::system_error(errno, std::system_category(), \"ioctl(BLKSSZGET) failed\");\n    }\n\n    // Query physical block size (smallest atomic write unit)\n    size_t physical_block_size;\n    if (auto ret = ::ioctl(fd, BLKPBSZGET, &physical_block_size); ret == -1) {\n        throw std::system_error(errno, std::system_category(), \"ioctl(BLKPBSZGET) failed\");\n    }\n\n    // Query device alignment info (statx memory alignment and physical_block_size override)\n    auto device_info = query_device_alignment_info(fd, device_id);\n\n    // Apply physical_block_size override from io_properties.yaml if present\n    // This is needed because some devices lie about their physical block size\n    if (device_info.physical_block_size) {\n        physical_block_size = *device_info.physical_block_size;\n    }\n\n    // For writes: use physical_block_size to avoid hardware-level read-modify-write\n    // - physical_block_size: smallest unit a physical storage device can write atomically (e.g., 4096 bytes for Advanced Format disks)\n    // - logical_block_size: smallest unit the storage device can address (typically 512 bytes)\n    //\n    // The Linux kernel only enforces logical_block_size alignment for O_DIRECT (see block/fops.c:blkdev_dio_invalid).\n    // Using physical_block_size avoids RMW at the hardware level.\n    return {\n        .memory = std::max(static_cast<unsigned>(device_info.memory_alignment.value_or(physical_block_size)), internal::min_memory_alignment),\n        .disk_read = static_cast<unsigned>(logical_block_size),  // For reads: use logical_block_size (no performance penalty for reading 512-byte blocks from 4K sector disks)\n        .disk_write = static_cast<unsigned>(physical_block_size),\n        .disk_overwrite = static_cast<unsigned>(physical_block_size),\n    };\n}\n\n} // anonymous namespace\n\nfuture<shared_ptr<file_impl>>\nmake_file_impl(int fd, file_open_options options, int flags, struct stat st) noexcept {\n    if (S_ISBLK(st.st_mode)) {\n        try {\n            auto align = blkdev_alignments(fd, st.st_rdev);\n            internal::fs_info fsi;\n            fsi.block_size = align.disk_read; // use logical_block_size for block_size\n            fsi.nowait_works = blockdev_nowait_works(st.st_rdev);\n            fsi.align = align;\n            return make_ready_future<shared_ptr<file_impl>>(make_shared<blockdev_file_impl>(fd, open_flags(flags), options, fsi, st.st_rdev));\n        } catch (...) {\n            return make_exception_future<shared_ptr<file_impl>>(std::current_exception());\n        }\n    }\n\n    if (S_ISDIR(st.st_mode)) {\n        // Directories don't care about block size, so we need not\n        // query it here. Just provide something reasonable.\n        internal::fs_info fsi;\n        fsi.block_size = 4096;\n        fsi.nowait_works = false;\n        return make_ready_future<shared_ptr<file_impl>>(make_shared<posix_file_real_impl>(fd, open_flags(flags), options, fsi, st.st_dev));\n    }\n\n    auto st_dev = st.st_dev;\n    static thread_local std::unordered_map<decltype(st_dev), internal::fs_info> s_fstype;\n\n    auto i = s_fstype.find(st_dev);\n    if (i == s_fstype.end()) [[unlikely]] {\n        return engine().fstatfs(fd).then([fd, options = std::move(options), flags, st = std::move(st)] (struct statfs sfs) {\n            internal::fs_info fsi;\n            fsi.block_size = sfs.f_bsize;\n            switch (sfs.f_type) {\n            case internal::fs_magic::xfs:\n                fsi.append_challenged = true;\n                static auto xc = xfs_concurrency_from_kernel_version();\n                fsi.append_concurrency = xc;\n                fsi.fsync_is_exclusive = true;\n                fsi.nowait_works = internal::kernel_uname().whitelisted({\"4.13\"});\n                break;\n            case internal::fs_magic::nfs:\n                fsi.append_challenged = false;\n                fsi.append_concurrency = 0;\n                fsi.fsync_is_exclusive = false;\n                fsi.nowait_works = internal::kernel_uname().whitelisted({\"4.13\"});\n                break;\n            case internal::fs_magic::ext4:\n                fsi.append_challenged = true;\n                fsi.append_concurrency = 0;\n                fsi.fsync_is_exclusive = false;\n                fsi.nowait_works = internal::kernel_uname().whitelisted({\"5.5\"});\n                break;\n            case internal::fs_magic::btrfs:\n                fsi.append_challenged = true;\n                fsi.append_concurrency = 0;\n                fsi.fsync_is_exclusive = true;\n                fsi.nowait_works = internal::kernel_uname().whitelisted({\"5.9\"});\n                break;\n            case internal::fs_magic::tmpfs:\n            case internal::fs_magic::fuse:\n                fsi.append_challenged = false;\n                fsi.append_concurrency = 999;\n                fsi.fsync_is_exclusive = false;\n                fsi.nowait_works = false;\n                break;\n            default:\n                fsi.append_challenged = true;\n                fsi.append_concurrency = 0;\n                fsi.fsync_is_exclusive = true;\n                fsi.nowait_works = false;\n            }\n            fsi.nowait_works &= engine()._cfg.aio_nowait_works;\n            fsi.align = filesystem_alignments(fd, st.st_dev, fsi.block_size, sfs.f_type);\n            s_fstype.insert(std::make_pair(st.st_dev, std::move(fsi)));\n            return make_file_impl(fd, std::move(options), flags, std::move(st));\n        });\n    }\n\n    try {\n        const internal::fs_info& fsi = i->second;\n        if (!fsi.append_challenged || options.append_is_unlikely || ((flags & O_ACCMODE) == O_RDONLY)) {\n            return make_ready_future<shared_ptr<file_impl>>(make_shared<posix_file_real_impl>(fd, open_flags(flags), std::move(options), fsi, st_dev));\n        }\n        return make_ready_future<shared_ptr<file_impl>>(make_shared<append_challenged_posix_file_impl>(fd, open_flags(flags), std::move(options), fsi, st_dev));\n    } catch(...) {\n        return current_exception_as_future<shared_ptr<file_impl>>();\n    }\n}\n\nfile::file(seastar::file_handle&& handle) noexcept\n        : _file_impl(std::move(std::move(handle).to_file()._file_impl)) {\n}\n\nfuture<uint64_t> file::size() const noexcept {\n  try {\n    return _file_impl->size();\n  } catch (...) {\n    return current_exception_as_future<uint64_t>();\n  }\n}\n\nfuture<> file::close() noexcept {\n    auto f = std::move(_file_impl);\n    return f->close().handle_exception([f = std::move(f)] (std::exception_ptr ex) {\n        report_exception(\"Closing the file failed unexpectedly\", std::move(ex));\n    });\n}\n\nsubscription<directory_entry>\nfile::list_directory(std::function<future<>(directory_entry de)> next) {\n    return _file_impl->list_directory(std::move(next));\n}\n\nlist_directory_generator_type file::experimental_list_directory() {\n    return _file_impl->experimental_list_directory();\n}\n\nfuture<int> file::ioctl(uint64_t cmd, void* argp) noexcept {\n    return _file_impl->ioctl(cmd, argp);\n}\n\nfuture<int> file::ioctl_short(uint64_t cmd, void* argp) noexcept {\n    return _file_impl->ioctl_short(cmd, argp);\n}\n\nfuture<int> file::fcntl(int op, uintptr_t arg) noexcept {\n    return _file_impl->fcntl(op, arg);\n}\n\nfuture<int> file::fcntl_short(int op, uintptr_t arg) noexcept {\n    return _file_impl->fcntl_short(op, arg);\n}\n\nfuture<> file::set_lifetime_hint_impl(int op, uint64_t hint) noexcept {\n    uint64_t arg = hint;\n    future<int> f = co_await coroutine::as_future(_file_impl->fcntl(op, (uintptr_t)&arg));\n    // Need to handle return value differently from that of fcntl\n    if (f.failed()) {\n        co_await coroutine::exception(f.get_exception());\n    }\n}\n\nfuture<> file::set_inode_lifetime_hint(uint64_t hint) noexcept {\n    return set_lifetime_hint_impl(F_SET_RW_HINT, hint);\n}\n\nfuture<uint64_t> file::get_lifetime_hint_impl(int op) noexcept {\n    uint64_t arg;\n    future<int> f = co_await coroutine::as_future(_file_impl->fcntl(op, (uintptr_t)&arg));\n    if (f.failed()) {\n        co_return coroutine::exception(f.get_exception());\n    }\n    co_return arg;\n}\n\nfuture<uint64_t> file::get_inode_lifetime_hint() noexcept {\n    return get_lifetime_hint_impl(F_GET_RW_HINT);\n}\n\nfuture<temporary_buffer<uint8_t>>\nfile::dma_read_bulk_impl(uint64_t offset, size_t range_size, io_intent* intent) noexcept {\n  try {\n    return _file_impl->dma_read_bulk(offset, range_size, intent);\n  } catch (...) {\n    return current_exception_as_future<temporary_buffer<uint8_t>>();\n  }\n}\n\nfuture<> file::discard(uint64_t offset, uint64_t length) noexcept {\n  try {\n    return _file_impl->discard(offset, length);\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\nfuture<> file::allocate(uint64_t position, uint64_t length) noexcept {\n  try {\n    return _file_impl->allocate(position, length);\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\nfuture<> file::truncate(uint64_t length) noexcept {\n  try {\n    return _file_impl->truncate(length);\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\nfuture<struct stat> file::stat() noexcept {\n  try {\n    return _file_impl->stat();\n  } catch (...) {\n    return current_exception_as_future<struct stat>();\n  }\n}\n\nfuture<struct stat> file::statat(std::string_view name, int flags) noexcept {\n    try {\n        return _file_impl->statat(name, flags);\n    } catch (...) {\n        return current_exception_as_future<struct stat>();\n    }\n}\n\nfuture<> file::flush() noexcept {\n  try {\n    return _file_impl->flush();\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\nfuture<size_t> file::dma_write_impl(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n  try {\n    return _file_impl->write_dma(pos, std::move(iov), intent);\n  } catch (...) {\n    return current_exception_as_future<size_t>();\n  }\n}\n\nfuture<size_t>\nfile::dma_write_impl(uint64_t pos, const uint8_t* buffer, size_t len, io_intent* intent) noexcept {\n  try {\n    return _file_impl->write_dma(pos, buffer, len, intent);\n  } catch (...) {\n    return current_exception_as_future<size_t>();\n  }\n}\n\nfuture<size_t> file::dma_read_impl(uint64_t pos, std::vector<iovec> iov, io_intent* intent) noexcept {\n  try {\n    return _file_impl->read_dma(pos, std::move(iov), intent);\n  } catch (...) {\n    return current_exception_as_future<size_t>();\n  }\n}\n\nfuture<temporary_buffer<uint8_t>>\nfile::dma_read_exactly_impl(uint64_t pos, size_t len, io_intent* intent) noexcept {\n    auto buf = co_await dma_read_impl(pos, len, intent);\n    if (buf.size() < len) {\n        throw eof_error();\n    }\n\n    co_return std::move(buf);\n}\n\nfuture<temporary_buffer<uint8_t>>\nfile::dma_read_impl(uint64_t pos, size_t len, io_intent* intent) noexcept {\n    temporary_buffer<uint8_t> buf = co_await dma_read_bulk_impl(pos, len, intent);\n    if (len < buf.size()) {\n        buf.trim(len);\n    }\n\n    co_return std::move(buf);\n}\n\nfuture<size_t>\nfile::dma_read_impl(uint64_t aligned_pos, uint8_t* aligned_buffer, size_t aligned_len, io_intent* intent) noexcept {\n  try {\n    return _file_impl->read_dma(aligned_pos, aligned_buffer, aligned_len, intent);\n  } catch (...) {\n    return current_exception_as_future<size_t>();\n  }\n}\n\nseastar::file_handle\nfile::dup() {\n    return seastar::file_handle(_file_impl->dup());\n}\n\nfile_impl* file_impl::get_file_impl(file& f) {\n    return f._file_impl.get();\n}\n\nstd::unique_ptr<seastar::file_handle_impl>\nfile_impl::dup() {\n    throw std::runtime_error(\"this file type cannot be duplicated\");\n}\n\nstatic list_directory_generator_type make_list_directory_fallback_generator(file_impl& me) {\n    auto ents = make_lw_shared<queue<std::optional<directory_entry>>>(list_directory_generator_buffer_size);\n    auto lister = me.list_directory([ents] (directory_entry de) {\n        return ents->push_eventually(std::move(de));\n    });\n    auto done = lister.done().finally([ents] {\n        return ents->push_eventually(std::nullopt);\n    });\n    auto abort = defer([ents, &done] () mutable noexcept {\n        ents->abort(std::make_exception_ptr(std::runtime_error(\"generator abandoned\")));\n        engine().run_in_background(std::move(done).handle_exception([] (std::exception_ptr ignored) {}));\n    });\n\n    while (true) {\n        auto de = co_await ents->pop_eventually();\n        if (!de) {\n            break;\n        }\n        co_yield *de;\n    }\n\n    abort.cancel();\n    co_await std::move(done);\n}\n\nlist_directory_generator_type file_impl::experimental_list_directory() {\n    return make_list_directory_fallback_generator(*this);\n}\n\nfuture<int> file_impl::ioctl(uint64_t cmd, void* argp) noexcept {\n    return make_exception_future<int>(std::runtime_error(\"this file type does not support ioctl\"));\n}\n\nfuture<int> file_impl::ioctl_short(uint64_t cmd, void* argp) noexcept {\n    return make_exception_future<int>(std::runtime_error(\"this file type does not support ioctl_short\"));\n}\n\nfuture<int> file_impl::fcntl(int op, uintptr_t arg) noexcept {\n    return make_exception_future<int>(std::runtime_error(\"this file type does not support fcntl\"));\n}\n\nfuture<int> file_impl::fcntl_short(int op, uintptr_t arg) noexcept {\n    return make_exception_future<int>(std::runtime_error(\"this file type does not support fcntl_short\"));\n}\n\nfuture<struct stat> file_impl::statat(std::string_view name, int flags) {\n    return make_exception_future<struct stat>(std::runtime_error(\"this file type does not support statat\"));\n}\n\nfuture<file> open_file_dma(std::string_view name, open_flags flags) noexcept {\n    return engine().open_file_dma(name, flags, file_open_options());\n}\n\nfuture<file> open_file_dma(std::string_view name, open_flags flags, file_open_options options) noexcept {\n    return engine().open_file_dma(name, flags, std::move(options));\n}\n\nfuture<file> open_directory(std::string_view name) noexcept {\n    return engine().open_directory(name);\n}\n\nnamespace testing {\n\nshared_ptr<append_challenged_posix_file_impl>\nmake_append_challenged_posix_file(file_desc& fd, unsigned concurrency, bool fsync_is_exclusive, std::optional<size_t> sloppy_size) {\n    internal::fs_info fsi {\n        .block_size = 4096,\n        .append_challenged = true,\n        .append_concurrency = concurrency,\n        .fsync_is_exclusive = fsync_is_exclusive,\n        .nowait_works = true,\n        .align = std::nullopt,\n    };\n    // device number can be any value, reactor would just pick \"fallback\" queue\n    // that will be unused anyway, because no real IO is supposed to happen\n    constexpr dev_t dev = 0;\n    return make_shared<append_challenged_posix_file_impl>(fd.get(), open_flags::rw, file_open_options{}, fsi, dev);\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/core/fsnotify.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB Ltd.\n */\n\n#include <seastar/core/internal/pollable_fd.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/fsnotify.hh>\n\nnamespace seastar::experimental {\n\nclass fsnotifier::impl : public enable_shared_from_this<impl> {\n    class my_poll_fd : public pollable_fd {\n    public:\n        using pollable_fd::pollable_fd;\n        using pollable_fd::get_fd;\n\n        operator int() const {\n            return get_fd();\n        }\n    };\n    my_poll_fd _fd;\n    watch_token _close_dummy = -1;\npublic:\n    impl()\n        : _fd(file_desc::inotify_init(IN_NONBLOCK | IN_CLOEXEC))\n    {}\n    void remove_watch(watch_token);\n    future<watch_token> create_watch(const sstring& path, flags events);\n    future<std::vector<event>> wait();\n    void shutdown();\n    bool active() const {\n        return bool(_fd);\n    }\n};\n\nvoid fsnotifier::impl::remove_watch(watch_token token) {\n    if (active()) {\n        auto res = ::inotify_rm_watch(_fd, token);\n        // throw if any other error than EINVAL.\n        throw_system_error_on(res == -1 && errno != EINVAL, \"could not remove inotify watch\");\n    }\n}\n\nfuture<fsnotifier::watch_token> fsnotifier::impl::create_watch(const sstring& path, flags events) {\n    if (!active()) {\n        throw std::runtime_error(\"attempting to use closed notifier\");\n    }\n    return engine().inotify_add_watch(_fd, path, uint32_t(events));\n}\n\nfuture<std::vector<fsnotifier::event>> fsnotifier::impl::wait() {\n    // be paranoid about buffer alignment\n    auto buf = temporary_buffer<char>::aligned(std::max(alignof(::inotify_event), alignof(int64_t)), 4096);\n    auto f = _fd.read_some(buf.get_write(), buf.size());\n    return f.then([me = shared_from_this(), buf = std::move(buf)](size_t n) {\n        auto p = buf.get();\n        auto e = buf.get() + n;\n\n        std::vector<event> events;\n\n        while (p < e) {\n            auto ev = reinterpret_cast<const ::inotify_event*>(p);\n            if (ev->wd == me->_close_dummy && me->_close_dummy != -1) {\n                me->_fd.close();\n            } else {\n                events.emplace_back(event {\n                    ev->wd, flags(ev->mask), ev->cookie,\n                    ev->len != 0 ? sstring(ev->name) : sstring{}\n                });\n            }\n            p += sizeof(::inotify_event) + ev->len;\n        }\n\n        return events;\n    });\n}\n\nvoid fsnotifier::impl::shutdown() {\n    // reactor does not yet have\n    // any means of \"shutting down\" a non-socket read,\n    // so we work around this by creating a watch for something ubiquitous,\n    // then removing the watch while adding a mark.\n    // This will cause any event waiter to wake up, but ignore the event for our\n    // dummy.\n    (void)create_watch(\"/\", flags::delete_self).then([me = shared_from_this()](watch_token t) {\n        me->_close_dummy = t;\n        me->remove_watch(t);\n    });\n}\n\nfsnotifier::watch::~watch() {\n    if (_impl) {\n        _impl->remove_watch(_token);\n    }\n}\n\nfsnotifier::watch::watch(watch&&) noexcept = default;\nfsnotifier::watch& fsnotifier::watch::operator=(watch&&) noexcept = default;\n\nfsnotifier::watch_token fsnotifier::watch::release() {\n    _impl = {};\n    return _token;\n}\n\nfsnotifier::watch::watch(shared_ptr<impl> impl, watch_token token)\n    : _token(token)\n    , _impl(std::move(impl))\n{}\n\nfsnotifier::fsnotifier()\n    : _impl(make_shared<impl>())\n{}\n\nfsnotifier::~fsnotifier() = default;\n\nfsnotifier::fsnotifier(fsnotifier&&) = default;\nfsnotifier& fsnotifier::operator=(fsnotifier&&) = default;\n\nfuture<fsnotifier::watch> fsnotifier::create_watch(const sstring& path, flags events) {\n    return _impl->create_watch(path, events).then([this](watch_token token) {\n        return watch(_impl, token);\n    });\n}\n\nfuture<std::vector<fsnotifier::event>> fsnotifier::wait() const {\n    return _impl->wait();\n}\n\nvoid fsnotifier::shutdown() {\n    _impl->shutdown();\n}\n\nbool fsnotifier::active() const {\n    return _impl->active();\n}\n\n}\n"
  },
  {
    "path": "src/core/fsqual.cc",
    "content": "/*\n * Copyright 2016 ScyllaDB\n */\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <seastar/core/posix.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/core/internal/linux-aio.hh>\n#include <sys/time.h>\n#include <sys/resource.h>\n#include <fcntl.h>\n#include <iostream>\n#include <unistd.h>\n#include <cstdlib>\n#include <type_traits>\n#include <seastar/core/fsqual.hh>\n\nnamespace seastar {\n\nusing namespace seastar::internal;\nusing namespace seastar::internal::linux_abi;\n\n// Runs func(), and also adds the number of context switches\n// that happened during func() to counter.\ntemplate <typename Counter, typename Func>\ntypename std::invoke_result_t<Func>\nwith_ctxsw_counting(Counter& counter, Func&& func) {\n    struct count_guard {\n        Counter& counter;\n        count_guard(Counter& counter) : counter(counter) {\n            counter -= nvcsw();\n        }\n        ~count_guard() {\n            counter += nvcsw();\n        }\n        static Counter nvcsw() {\n            struct rusage usage;\n            getrusage(RUSAGE_THREAD, &usage);\n            return usage.ru_nvcsw;\n        }\n    };\n    count_guard g(counter);\n    return func();\n}\n\nbool filesystem_has_good_aio_support(sstring directory, bool verbose) {\n    aio_context_t ioctx = {};\n    auto r = io_setup(1, &ioctx);\n    throw_system_error_on(r == -1, \"io_setup\");\n    auto cleanup = defer([&] () noexcept { io_destroy(ioctx); });\n    auto fname = directory + \"/fsqual.tmp\";\n    auto fd = file_desc::open(fname, O_CREAT|O_EXCL|O_RDWR|O_DIRECT, 0600);\n    unlink(fname.c_str());\n    auto nr = 1000;\n    fd.truncate(nr * 4096);\n    auto bufsize = 4096;\n    auto ctxsw = 0;\n    auto buf = aligned_alloc(4096, 4096);\n    auto del = defer([&] () noexcept { ::free(buf); });\n    for (int i = 0; i < nr; ++i) {\n        struct iocb cmd;\n        cmd = make_write_iocb(fd.get(), bufsize*i, buf, bufsize);\n        struct iocb* cmds[1] = { &cmd };\n        with_ctxsw_counting(ctxsw, [&] {\n            auto r = io_submit(ioctx, 1, cmds);\n            throw_system_error_on(r == -1, \"io_submit\");\n            SEASTAR_ASSERT(r == 1);\n        });\n        struct io_event ioev;\n        int n = -1;\n        do {\n            n = io_getevents(ioctx, 1, 1, &ioev, nullptr);\n            throw_system_error_on((n == -1) && (errno != EINTR) , \"io_getevents\");\n        } while (n == -1);\n        SEASTAR_ASSERT(n == 1);\n        throw_kernel_error(long(ioev.res));\n        SEASTAR_ASSERT(long(ioev.res) == bufsize);\n    }\n    auto rate = float(ctxsw) / nr;\n    bool ok = rate < 0.1;\n    if (verbose) {\n        auto verdict = ok ? \"GOOD\" : \"BAD\";\n        std::cout << \"context switch per appending io: \" << rate\n                  << \" (\" << verdict << \")\\n\";\n    }\n    return ok;\n}\n\n}\n"
  },
  {
    "path": "src/core/fstream.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n\n#include <fmt/format.h>\n#include <fmt/ostream.h>\n#include <malloc.h>\n#include <string.h>\n#include <ratio>\n#include <optional>\n#include <utility>\n#include <seastar/util/assert.hh>\n\n#include <seastar/core/fstream.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/io_intent.hh>\n\nnamespace seastar {\n\nstatic_assert(std::is_nothrow_constructible_v<data_source>);\nstatic_assert(std::is_nothrow_move_constructible_v<data_source>);\n\nstatic_assert(std::is_nothrow_constructible_v<data_sink>);\nstatic_assert(std::is_nothrow_move_constructible_v<data_sink>);\n\nstatic_assert(std::is_nothrow_constructible_v<temporary_buffer<char>>);\nstatic_assert(std::is_nothrow_move_constructible_v<temporary_buffer<char>>);\n\nstatic_assert(std::is_nothrow_constructible_v<input_stream<char>>);\nstatic_assert(std::is_nothrow_move_constructible_v<input_stream<char>>);\n\nstatic_assert(std::is_nothrow_constructible_v<output_stream<char>>);\nstatic_assert(std::is_nothrow_move_constructible_v<output_stream<char>>);\n\n// The buffers size must not be greater than the limit, but when capping\n// it we make it 2^n to better utilize the memory allocated for buffers\ntemplate <typename T>\nstatic inline T select_buffer_size(T configured_value, T maximum_value) noexcept {\n    if (configured_value <= maximum_value) {\n        return configured_value;\n    } else {\n        return T(1) << log2floor(maximum_value);\n    }\n}\n\nclass file_data_source_impl : public data_source_impl {\n    struct issued_read {\n        uint64_t _pos;\n        uint64_t _size;\n        future<temporary_buffer<char>> _ready;\n\n        issued_read(uint64_t pos, uint64_t size, future<temporary_buffer<char>> f)\n            : _pos(pos), _size(size), _ready(std::move(f)) { }\n    };\n\n    reactor::io_stats& _stats = reactor::io_stats::local();\n    file _file;\n    file_input_stream_options _options;\n    uint64_t _pos;\n    uint64_t _remain;\n    circular_buffer<issued_read> _read_buffers;\n    unsigned _reads_in_progress = 0;\n    unsigned _current_read_ahead;\n    future<> _dropped_reads = make_ready_future<>();\n    std::optional<promise<>> _done;\n    size_t _current_buffer_size;\n    bool _in_slow_start = false;\n    io_intent _intent;\n    using unused_ratio_target = std::ratio<25, 100>;\nprivate:\n    size_t minimal_buffer_size() const {\n        return std::min(std::max(_options.buffer_size / 4, size_t(8192)), _options.buffer_size);\n    }\n\n    void try_increase_read_ahead() {\n        // Read-ahead can be increased up to user-specified limit if the\n        // consumer has to wait for a buffer and we are not in a slow start\n        // phase.\n        if (_current_read_ahead < _options.read_ahead && !_in_slow_start) {\n            _current_read_ahead++;\n            if (_options.dynamic_adjustments) {\n                auto& h = *_options.dynamic_adjustments;\n                h.read_ahead = std::max(h.read_ahead, _current_read_ahead);\n            }\n        }\n    }\n    unsigned get_initial_read_ahead() const {\n        return _options.dynamic_adjustments\n               ? std::min(_options.dynamic_adjustments->read_ahead, _options.read_ahead)\n               : !!_options.read_ahead;\n    }\n\n    void update_history(uint64_t unused, uint64_t total) {\n        // We are maintaining two windows each no larger than window_size.\n        // Dynamic adjustment logic uses data from both of them, which\n        // essentially means that the actual window size is variable and\n        // in the range [window_size, 2*window_size].\n        auto& h = *_options.dynamic_adjustments;\n        h.current_window.total_read += total;\n        h.current_window.unused_read += unused;\n        if (h.current_window.total_read >= h.window_size) {\n            h.previous_window = h.current_window;\n            h.current_window = { };\n        }\n    }\n    static bool below_target(uint64_t unused, uint64_t total) {\n        return unused * unused_ratio_target::den < total * unused_ratio_target::num;\n    }\n    void update_history_consumed(uint64_t bytes) {\n        if (!_options.dynamic_adjustments) {\n            return;\n        }\n        update_history(0, bytes);\n        if (!_in_slow_start) {\n            return;\n        }\n        unsigned new_size = std::min(_current_buffer_size * 2, _options.buffer_size);\n        auto& h = *_options.dynamic_adjustments;\n        auto total = h.current_window.total_read + h.previous_window.total_read + new_size;\n        auto unused = h.current_window.unused_read + h.previous_window.unused_read + new_size;\n        // Check whether we can safely increase the buffer size to new_size\n        // and still be below unused_ratio_target even if it is entirely\n        // dropped.\n        if (below_target(unused, total)) {\n            _current_buffer_size = new_size;\n            _in_slow_start = _current_buffer_size < _options.buffer_size;\n        }\n    }\n\n    using after_skip = bool_class<class after_skip_tag>;\n    void set_new_buffer_size(after_skip skip) {\n        if (!_options.dynamic_adjustments) {\n            return;\n        }\n        auto& h = *_options.dynamic_adjustments;\n        int64_t total = h.current_window.total_read + h.previous_window.total_read;\n        int64_t unused = h.current_window.unused_read + h.previous_window.unused_read;\n        if (skip == after_skip::yes && below_target(unused, total)) {\n            // Do not attempt to shrink buffer size if we are still below the\n            // target. Otherwise, we could get a bad interaction with\n            // update_history_consumed() which tries to increase the buffer\n            // size as much as possible so that after a single drop we are\n            // still below the target.\n            return;\n        }\n        // Calculate the maximum buffer size that would guarantee that we are\n        // still below unused_ratio_target even if the subsequent reads are\n        // dropped. If it is larger than or equal to the current buffer size do\n        // nothing. If it is smaller then we are back in the slow start phase.\n        auto new_target = (unused_ratio_target::num * total - unused_ratio_target::den * unused) / (unused_ratio_target::den - unused_ratio_target::num);\n        uint64_t new_size = std::max(new_target, int64_t(minimal_buffer_size()));\n        new_size = std::max(uint64_t(1) << log2floor(new_size), uint64_t(minimal_buffer_size()));\n        if (new_size >= _current_buffer_size) {\n            return;\n        }\n        _in_slow_start = true;\n        _current_read_ahead = std::min(_current_read_ahead, 1u);\n        _current_buffer_size = new_size;\n    }\n    void update_history_unused(uint64_t bytes) {\n        if (!_options.dynamic_adjustments) {\n            return;\n        }\n        update_history(bytes, bytes);\n        set_new_buffer_size(after_skip::yes);\n    }\n    // Safely ignores read future even if it is not resolved yet.\n    void ignore_read_future(future<temporary_buffer<char>> read_future) {\n        if (read_future.available()) {\n            read_future.ignore_ready_future();\n            return;\n        }\n        auto f = read_future.then_wrapped([] (auto f) { f.ignore_ready_future(); });\n        _dropped_reads = _dropped_reads.then([f = std::move(f)] () mutable { return std::move(f); });\n    }\npublic:\n    file_data_source_impl(file f, uint64_t offset, uint64_t len, file_input_stream_options options)\n            : _file(std::move(f)), _options(options), _pos(offset), _remain(len), _current_read_ahead(get_initial_read_ahead())\n    {\n        _options.buffer_size = select_buffer_size(_options.buffer_size, _file.disk_read_max_length());\n        _current_buffer_size = _options.buffer_size;\n        // prevent wraparounds\n        set_new_buffer_size(after_skip::no);\n        _remain = std::min(std::numeric_limits<uint64_t>::max() - _pos, _remain);\n    }\n    virtual ~file_data_source_impl() override {\n        // If the data source hasn't been closed, we risk having reads in progress\n        // that will try to access freed memory.\n        SEASTAR_ASSERT(_reads_in_progress == 0);\n    }\n    virtual future<temporary_buffer<char>> get() override {\n        if (!_read_buffers.empty() && !_read_buffers.front()._ready.available()) {\n            try_increase_read_ahead();\n        }\n        issue_read_aheads(1);\n        auto ret = std::move(_read_buffers.front());\n        _read_buffers.pop_front();\n        update_history_consumed(ret._size);\n        _stats.fstream_reads += 1;\n        _stats.fstream_read_bytes += ret._size;\n        if (!ret._ready.available()) {\n            _stats.fstream_reads_blocked += 1;\n            _stats.fstream_read_bytes_blocked += ret._size;\n        }\n        return std::move(ret._ready);\n    }\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        uint64_t dropped = 0;\n        while (n) {\n            if (_read_buffers.empty()) {\n                SEASTAR_ASSERT(n <= _remain);\n                _pos += n;\n                _remain -= n;\n                break;\n            }\n            auto& front = _read_buffers.front();\n            if (n < front._size) {\n                front._size -= n;\n                front._pos += n;\n                front._ready = front._ready.then([n] (temporary_buffer<char> buf) {\n                    buf.trim_front(n);\n                    return buf;\n                });\n                break;\n            } else {\n                ignore_read_future(std::move(front._ready));\n                n -= front._size;\n                dropped += front._size;\n                _stats.fstream_read_aheads_discarded += 1;\n                _stats.fstream_read_ahead_discarded_bytes += front._size;\n                _read_buffers.pop_front();\n            }\n        }\n        update_history_unused(dropped);\n        return make_ready_future<temporary_buffer<char>>();\n    }\n    virtual future<> close() override {\n        _done.emplace();\n        if (!_reads_in_progress) {\n            _done->set_value();\n        }\n        _intent.cancel();\n        return _done->get_future().then([this] {\n            uint64_t dropped = 0;\n            for (auto&& c : _read_buffers) {\n                _stats.fstream_read_aheads_discarded += 1;\n                _stats.fstream_read_ahead_discarded_bytes += c._size;\n                dropped += c._size;\n                ignore_read_future(std::move(c._ready));\n            }\n            update_history_unused(dropped);\n            return std::move(_dropped_reads);\n        });\n    }\nprivate:\n    void issue_read_aheads(unsigned additional = 0) {\n        if (_done) {\n            return;\n        }\n        auto ra = _current_read_ahead + additional;\n        _read_buffers.reserve(ra); // prevent push_back() failure\n        while (_read_buffers.size() < ra) {\n            if (!_remain) {\n                if (_read_buffers.size() >= additional) {\n                    return;\n                }\n                _read_buffers.emplace_back(_pos, 0, make_ready_future<temporary_buffer<char>>());\n                continue;\n            }\n            ++_reads_in_progress;\n            // if _pos is not dma-aligned, we'll get a short read.  Account for that.\n            // Also avoid reading beyond _remain.\n            uint64_t align = _file.disk_read_dma_alignment();\n            auto start = align_down(_pos, align);\n            auto end = std::min(align_up(start + _current_buffer_size, align), _pos + _remain);\n            auto len = end - start;\n            auto actual_size = std::min(end - _pos, _remain);\n            _read_buffers.emplace_back(_pos, actual_size, futurize_invoke([&] {\n                    return _file.dma_read_bulk_impl(start, len, &_intent);\n            }).then_wrapped(\n                    [this, start, pos = _pos, remain = _remain] (future<temporary_buffer<uint8_t>> ret) {\n                --_reads_in_progress;\n                if (_done && !_reads_in_progress) {\n                    _done->set_value();\n                }\n                if (ret.failed()) {\n                    // no games needed\n                    return make_exception_future<temporary_buffer<char>>(ret.get_exception());\n                } else {\n                    // first or last buffer, need trimming\n                    auto tmp = ret.get();\n                    auto real_end = start + tmp.size();\n                    if (real_end <= pos) {\n                        return make_ready_future<temporary_buffer<char>>();\n                    }\n                    if (real_end > pos + remain) {\n                        tmp.trim(pos + remain - start);\n                    }\n                    if (start < pos) {\n                        tmp.trim_front(pos - start);\n                    }\n                    return make_ready_future<temporary_buffer<char>>(temporary_buffer<char>(reinterpret_cast<char*>(tmp.get_write()), tmp.size(), tmp.release()));\n                }\n            }));\n            _remain -= end - _pos;\n            _pos = end;\n        };\n    }\n};\n\ndata_source make_file_data_source(file f, uint64_t offset, uint64_t len, file_input_stream_options opt) {\n    return data_source(std::make_unique<file_data_source_impl>(std::move(f), offset, len, std::move(opt)));\n}\n\ndata_source make_file_data_source(file f, file_input_stream_options opt) {\n    return make_file_data_source(std::move(f), 0, std::numeric_limits<uint64_t>::max(), std::move(opt));\n}\n\ninput_stream<char> make_file_input_stream(\n        file f, uint64_t offset, uint64_t len, file_input_stream_options options) {\n    return input_stream<char>(make_file_data_source(std::move(f), offset, len, std::move(options)));\n}\n\ninput_stream<char> make_file_input_stream(\n        file f, uint64_t offset, file_input_stream_options options) {\n    return make_file_input_stream(std::move(f), offset, std::numeric_limits<uint64_t>::max(), std::move(options));\n}\n\ninput_stream<char> make_file_input_stream(\n        file f, file_input_stream_options options) {\n    return make_file_input_stream(std::move(f), 0, std::move(options));\n}\n\n\nclass file_data_sink_impl : public data_sink_impl {\n    file _file;\n    file_output_stream_options _options;\n    uint64_t _pos = 0;\n    semaphore _write_behind_sem = { _options.write_behind };\n    future<> _background_writes_done = make_ready_future<>();\n    bool _failed = false;\npublic:\n    file_data_sink_impl(file f, file_output_stream_options options)\n            : _file(std::move(f)), _options(options) {\n        _options.buffer_size = select_buffer_size<unsigned>(_options.buffer_size, _file.disk_write_max_length());\n\tif (_options.write_behind) {\n            _write_behind_sem.ensure_space_for_waiters(1); // So that wait() doesn't throw\n\t}\n    }\n    virtual temporary_buffer<char> allocate_buffer(size_t size) override {\n        return temporary_buffer<char>::aligned(_file.memory_dma_alignment(), size);\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        return data_sink_impl::fallback_put(bufs, [this] (temporary_buffer<char>&& buf) {\n            return do_put(std::move(buf));\n        });\n    }\n#else\n    using data_sink_impl::put;\n    future<> put(net::packet data) override { abort(); }\n    virtual future<> put(temporary_buffer<char> buf) override {\n        return do_put(std::move(buf));\n    }\n#endif\nprivate:\n    future<> do_put(temporary_buffer<char> buf) {\n        uint64_t pos = _pos;\n        _pos += buf.size();\n        if (!_options.write_behind) {\n            return do_put(pos, std::move(buf));\n        }\n        // Write behind strategy:\n        //\n        // 1. Issue N writes in parallel, using a semaphore to limit to N\n        // 2. Collect results in _background_writes_done, merging exception futures\n        // 3. If we've already seen a failure, don't issue more writes.\n        return _write_behind_sem.wait().then([this, pos, buf = std::move(buf)] () mutable {\n            if (_failed) {\n                _write_behind_sem.signal();\n                auto ret = std::move(_background_writes_done);\n                _background_writes_done = make_ready_future<>();\n                return ret;\n            }\n            auto this_write_done = do_put(pos, std::move(buf)).finally([this] {\n                _write_behind_sem.signal();\n            });\n            _background_writes_done = when_all(std::move(_background_writes_done), std::move(this_write_done))\n                    .then([this] (std::tuple<future<>, future<>> possible_errors) {\n                // merge the two errors, preferring the first\n                auto& e1 = std::get<0>(possible_errors);\n                auto& e2 = std::get<1>(possible_errors);\n                if (e1.failed()) {\n                    e2.ignore_ready_future();\n                    return std::move(e1);\n                } else {\n                    if (e2.failed()) {\n                        _failed = true;\n                    }\n                    return std::move(e2);\n                }\n            });\n            return make_ready_future<>();\n        });\n    }\n\n    future<> do_put(uint64_t pos, temporary_buffer<char> buf) noexcept {\n      try {\n        // put() must usually be of chunks multiple of file::dma_alignment.\n        // Only the last part can have an unaligned length. If put() was\n        // called again with an unaligned pos, we have a bug in the caller.\n        SEASTAR_ASSERT(!(pos & (_file.disk_write_dma_alignment() - 1)));\n        bool truncate = false;\n        auto p = static_cast<const char*>(buf.get());\n        size_t buf_size = buf.size();\n\n        if ((buf.size() & (_file.disk_write_dma_alignment() - 1)) != 0) {\n            // If buf size isn't aligned, copy its content into a new aligned buf.\n            // This should only happen when the user calls output_stream::flush().\n            auto tmp = allocate_buffer(align_up(buf.size(), _file.disk_write_dma_alignment()));\n            ::memcpy(tmp.get_write(), buf.get(), buf.size());\n            ::memset(tmp.get_write() + buf.size(), 0, tmp.size() - buf.size());\n            buf = std::move(tmp);\n            p = buf.get();\n            buf_size = buf.size();\n            truncate = true;\n        }\n\n        return _file.dma_write_impl(pos, reinterpret_cast<const uint8_t*>(p), buf_size, nullptr).then(\n                [this, pos, buf = std::move(buf), truncate, buf_size] (size_t size) mutable {\n            // short write handling\n            if (size < buf_size) {\n                buf.trim_front(size);\n                return do_put(pos + size, std::move(buf)).then([this, truncate] {\n                    if (truncate) {\n                        return _file.truncate(_pos);\n                    }\n                    return make_ready_future<>();\n                });\n            }\n            if (truncate) {\n                return _file.truncate(_pos);\n            }\n            return make_ready_future<>();\n        });\n      } catch (...) {\n          return make_exception_future<>(std::current_exception());\n      }\n    }\n    future<> wait() noexcept {\n        // restore to pristine state; for flush() + close() sequence\n        // (we allow either flush, or close, or both)\n        return _write_behind_sem.wait(_options.write_behind).then([this] {\n            return std::exchange(_background_writes_done, make_ready_future<>());\n        }).finally([this] {\n            _write_behind_sem.signal(_options.write_behind);\n        });\n    }\npublic:\n    virtual future<> flush() override {\n        return wait().then([this] {\n            return _file.flush();\n        });\n    }\n    virtual future<> close() noexcept override {\n        return wait().finally([this] {\n            return _file.close();\n        });\n    }\n    virtual size_t buffer_size() const noexcept override { return _options.buffer_size; }\n};\n\nfuture<data_sink> make_file_data_sink(file f, file_output_stream_options options) noexcept {\n    try {\n        return make_ready_future<data_sink>(std::make_unique<file_data_sink_impl>(f, options));\n    } catch (...) {\n        return f.close().then_wrapped([ex = std::current_exception(), f] (future<> fut) mutable {\n            if (fut.failed()) {\n                try {\n                    std::rethrow_exception(std::move(ex));\n                } catch (...) {\n                    std::throw_with_nested(std::runtime_error(fmt::format(\"While handling failed construction of data_sink, caught exception: {}\",\n                                fut.get_exception())));\n                }\n            }\n            return make_exception_future<data_sink>(std::move(ex));\n        });\n    }\n}\n\nfuture<output_stream<char>> make_file_output_stream(file f, size_t buffer_size) noexcept {\n    file_output_stream_options options;\n    options.buffer_size = buffer_size;\n    return make_file_output_stream(std::move(f), options);\n}\n\nfuture<output_stream<char>> make_file_output_stream(file f, file_output_stream_options options) noexcept {\n    return make_file_data_sink(std::move(f), options).then([] (data_sink&& ds) {\n        return output_stream<char>(std::move(ds));\n    });\n}\n\n/*\n * template initialization, definition in iostream-impl.hh\n */\ntemplate struct internal::stream_copy_consumer<char>;\ntemplate future<> copy<char>(input_stream<char>&, output_stream<char>&);\n\n}\n\n"
  },
  {
    "path": "src/core/future-util.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n#include <seastar/core/future-util.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/semaphore.hh>\n\nnamespace seastar {\n\nparallel_for_each_state::parallel_for_each_state(size_t n) {\n    _incomplete.reserve(n);\n}\n\nfuture<> parallel_for_each_state::get_future() {\n    auto ret = _result.get_future();\n    wait_for_one();\n    return ret;\n}\n\nvoid parallel_for_each_state::add_future(future<>&& f) {\n    _incomplete.push_back(std::move(f));\n}\n\nvoid parallel_for_each_state::wait_for_one() noexcept {\n    // Process from back to front, on the assumption that the front\n    // futures are likely to complete earlier than the back futures.\n    // If that's indeed the case, then the front futures will be\n    // available and we won't have to wait for them.\n\n    // Skip over futures that happen to be complete already.\n    while (!_incomplete.empty() && _incomplete.back().available()) {\n        if (_incomplete.back().failed()) {\n            _ex = _incomplete.back().get_exception();\n        }\n        _incomplete.pop_back();\n    }\n\n    // If there's an incompelete future, wait for it.\n    if (!_incomplete.empty()) {\n        internal::set_callback(std::move(_incomplete.back()), static_cast<continuation_base<>*>(this));\n        // This future's state will be collected in run_and_dispose(), so we can drop it.\n        _incomplete.pop_back();\n        return;\n    }\n\n    // Everything completed, report a result.\n    if (__builtin_expect(bool(_ex), false)) {\n        _result.set_exception(std::move(_ex));\n    } else {\n        _result.set_value();\n    }\n    delete this;\n}\n\nvoid parallel_for_each_state::run_and_dispose() noexcept {\n    if (_state.failed()) {\n        _ex = std::move(_state).get_exception();\n    }\n    _state = {};\n    wait_for_one();\n}\n\ntemplate <typename Clock>\nfuture<> sleep_abortable(typename Clock::duration dur) {\n    return engine().wait_for_stop(dur).then([] {\n        throw sleep_aborted();\n    }).handle_exception([] (std::exception_ptr ep) {\n        try {\n            std::rethrow_exception(ep);\n        } catch(condition_variable_timed_out&) {};\n    });\n}\n\ntemplate future<> sleep_abortable<steady_clock_type>(typename steady_clock_type::duration);\ntemplate future<> sleep_abortable<lowres_clock>(typename lowres_clock::duration);\n\ntemplate <typename Clock>\nfuture<> sleep_abortable(typename Clock::duration dur, abort_source& as) {\n    struct sleeper {\n        promise<> done;\n        timer<Clock> tmr;\n        abort_source::subscription sc;\n\n        sleeper(typename Clock::duration dur, abort_source& as)\n                : tmr([this] { done.set_value(); }) {\n            auto sc_opt = as.subscribe([this] (const std::optional<std::exception_ptr>& opt_ex) noexcept {\n                if (tmr.cancel()) {\n                    done.set_exception(opt_ex.value_or(std::make_exception_ptr(sleep_aborted())));\n                }\n            });\n            if (sc_opt) {\n                sc = std::move(*sc_opt);\n                tmr.arm(dur);\n            } else {\n                done.set_exception(sleep_aborted());\n            }\n        }\n    };\n    //FIXME: Use do_with() after #373\n    auto s = std::make_unique<sleeper>(dur, as);\n    auto fut = s->done.get_future();\n    return fut.finally([s = std::move(s)] { });\n}\n\ntemplate future<> sleep_abortable<steady_clock_type>(typename steady_clock_type::duration, abort_source&);\ntemplate future<> sleep_abortable<lowres_clock>(typename lowres_clock::duration, abort_source&);\ntemplate future<> sleep_abortable<manual_clock>(typename manual_clock::duration, abort_source&);\n\n}\n"
  },
  {
    "path": "src/core/future.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB\n */\n\n#include <seastar/core/future.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/report_exception.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\n// We can't test future_state_base directly because its private\n// destructor is protected.\nstatic_assert(std::is_nothrow_move_constructible_v<future_state<std::tuple<int>>>,\n              \"future_state's move constructor must not throw\");\n\nstatic_assert(sizeof(future_state<std::tuple<>>) <= 8, \"future_state<std::tuple<>> is too large\");\nstatic_assert(sizeof(future_state<std::tuple<long>>) <= 16, \"future_state<std::tuple<long>> is too large\");\nstatic_assert(future_state<std::tuple<>>::has_trivial_move_and_destroy, \"future_state<std::tuple<>> not trivial\");\nstatic_assert(future_state<long>::has_trivial_move_and_destroy, \"future_state<long> not trivial\");\n\n// We need to be able to move and copy std::exception_ptr in and out\n// of future/promise/continuations without that producing a new\n// exception.\nstatic_assert(std::is_nothrow_copy_constructible_v<std::exception_ptr>,\n    \"std::exception_ptr's copy constructor must not throw\");\nstatic_assert(std::is_nothrow_move_constructible_v<std::exception_ptr>,\n    \"std::exception_ptr's move constructor must not throw\");\n\nnamespace internal {\n\nstatic_assert(std::is_empty_v<uninitialized_wrapper<std::tuple<>>>, \"This should still be empty\");\n\nvoid promise_base::move_it(promise_base&& x) noexcept {\n    // Don't use std::exchange to make sure x's values are nulled even\n    // if &x == this.\n    _task = x._task;\n    x._task = nullptr;\n#ifdef SEASTAR_DEBUG_PROMISE\n    _task_shard = x._task_shard;\n#endif\n    _state = x._state;\n    x._state = nullptr;\n    _future = x._future;\n    if (auto* fut = _future) {\n        fut->detach_promise();\n        fut->_promise = this;\n    }\n}\n\nstatic void set_to_broken_promise(future_state_base& state) noexcept {\n    try {\n        // Constructing broken_promise may throw (std::logic_error ctor is not noexcept).\n        state.set_exception(std::make_exception_ptr(broken_promise{}));\n    } catch (...) {\n        state.set_exception(std::current_exception());\n    }\n}\n\npromise_base::promise_base(promise_base&& x) noexcept {\n    move_it(std::move(x));\n}\n\nvoid promise_base::clear() noexcept {\n    if (__builtin_expect(bool(_task), false)) {\n        SEASTAR_ASSERT(_state && !_state->available());\n        set_to_broken_promise(*_state);\n        ::seastar::schedule(std::exchange(_task, nullptr));\n    }\n    if (_future) {\n        SEASTAR_ASSERT(_state);\n        if (!_state->available()) {\n            set_to_broken_promise(*_state);\n        }\n        _future->detach_promise();\n    }\n}\n\npromise_base& promise_base::operator=(promise_base&& x) noexcept {\n    clear();\n    move_it(std::move(x));\n    return *this;\n}\n\nvoid promise_base::set_to_current_exception() noexcept {\n    set_exception(std::current_exception());\n}\n\n#ifdef SEASTAR_DEBUG_PROMISE\n\nvoid promise_base::assert_task_shard() const noexcept {\n    if (_task_shard >= 0 && static_cast<shard_id>(_task_shard) != this_shard_id()) {\n        on_fatal_internal_error(seastar_logger, format(\"Promise task was set on shard {} but made ready on shard {}\", _task_shard, this_shard_id()));\n    }\n}\n\n#endif\n\ntemplate <promise_base::urgent Urgent>\nvoid promise_base::make_ready() noexcept {\n    if (_task) {\n        assert_task_shard();\n        if (Urgent == urgent::yes) {\n            ::seastar::schedule_urgent(std::exchange(_task, nullptr));\n        } else {\n            ::seastar::schedule(std::exchange(_task, nullptr));\n        }\n    }\n}\n\ntemplate void promise_base::make_ready<promise_base::urgent::no>() noexcept;\ntemplate void promise_base::make_ready<promise_base::urgent::yes>() noexcept;\n}\n\ntemplate\nfuture<void> current_exception_as_future() noexcept;\n\n/**\n * engine_exit() exits the reactor. It should be given a pointer to the\n * exception which prompted this exit - or a null pointer if the exit\n * request was not caused by any exception.\n */\nvoid engine_exit(std::exception_ptr eptr) {\n    if (!eptr) {\n        engine().exit(0);\n        return;\n    }\n    report_exception(\"Exiting on unhandled exception\", eptr);\n    engine().exit(1);\n}\n\nbroken_promise::broken_promise() : logic_error(\"broken promise\") { }\n\nfuture_state_base::future_state_base(current_exception_future_marker) noexcept\n    : future_state_base(std::current_exception()) { }\n\nvoid future_state_base::ignore() noexcept {\n    switch (_u.st) {\n    case state::invalid:\n    case state::future:\n    case state::result_unavailable:\n        SEASTAR_ASSERT(0 && \"invalid state for ignore\");\n    case state::result:\n        _u.st = state::result_unavailable;\n        break;\n    default:\n        // Ignore the exception\n        _u.take_exception();\n    }\n}\n\nnested_exception::nested_exception(std::exception_ptr inner, std::exception_ptr outer) noexcept\n    : inner(std::move(inner)), outer(std::move(outer)) {}\n\nnested_exception::nested_exception(nested_exception&&) noexcept = default;\n\nnested_exception::nested_exception(const nested_exception&) noexcept = default;\n\nconst char* nested_exception::what() const noexcept {\n    return \"seastar::nested_exception\";\n}\n\n[[noreturn]] void nested_exception::rethrow_nested() const {\n    std::rethrow_exception(outer);\n}\n\nstatic std::exception_ptr make_nested(std::exception_ptr&& inner, future_state_base&& old) noexcept {\n    std::exception_ptr outer = std::move(old).get_exception();\n    nested_exception nested{std::move(inner), std::move(outer)};\n    return std::make_exception_ptr<nested_exception>(std::move(nested));\n}\n\nfuture_state_base::future_state_base(nested_exception_marker, future_state_base&& n, future_state_base&& old) noexcept {\n    std::exception_ptr inner = std::move(n).get_exception();\n    if (!old.failed()) {\n        new (this) future_state_base(std::move(inner));\n    } else {\n        new (this) future_state_base(make_nested(std::move(inner), std::move(old)));\n    }\n}\n\nfuture_state_base::future_state_base(nested_exception_marker, future_state_base&& old) noexcept {\n    if (!old.failed()) {\n        new (this) future_state_base(current_exception_future_marker());\n        return;\n    } else {\n        new (this) future_state_base(make_nested(std::current_exception(), std::move(old)));\n    }\n}\n\nvoid future_state_base::rethrow_exception() && {\n    // Move ex out so future::~future() knows we've handled it\n    std::rethrow_exception(std::move(*this).get_exception());\n}\n\nvoid future_state_base::rethrow_exception() const& {\n    std::rethrow_exception(_u.ex);\n}\n\nnamespace internal {\n\nvoid report_failed_future(const std::exception_ptr& eptr) noexcept {\n    ++engine()._abandoned_failed_futures;\n    seastar_logger.warn(\"Exceptional future ignored: {}, backtrace: {}\", eptr, current_backtrace());\n}\n\nvoid report_failed_future(const future_state_base& state) noexcept {\n    report_failed_future(state._u.ex);\n}\n\nvoid report_failed_future(future_state_base::any&& state) noexcept {\n    report_failed_future(std::move(state).take_exception());\n}\n\n} // internal namespace\n\nvoid reactor::test::with_allow_abandoned_failed_futures(unsigned count, noncopyable_function<void ()> func) {\n    auto before = engine()._abandoned_failed_futures;\n    auto old_level = seastar_logger.level();\n    seastar_logger.set_level(log_level::error);\n    func();\n    auto after = engine()._abandoned_failed_futures;\n    SEASTAR_ASSERT(after - before == count);\n    engine()._abandoned_failed_futures = before;\n    seastar_logger.set_level(old_level);\n}\n\nnamespace {\nclass thread_wake_task final : public task {\n    thread_context* _thread;\npublic:\n    thread_wake_task(thread_context* thread) noexcept : _thread(thread) {}\n    virtual void run_and_dispose() noexcept override {\n        thread_impl::switch_in(_thread);\n        // no need to delete, since this is always allocated on\n        // _thread's stack.\n    }\n    /// Returns the task which is waiting for this thread to be done, or nullptr.\n    virtual task* waiting_task() noexcept override {\n        return _thread->waiting_task();\n    }\n};\n}\n\nvoid internal::future_base::do_wait() noexcept {\n    auto thread = thread_impl::get();\n    SEASTAR_ASSERT(thread);\n    thread_wake_task wake_task{thread};\n    wake_task.make_backtrace();\n    _promise->set_task(&wake_task);\n    thread_impl::switch_out(thread);\n}\n\nvoid internal::future_base::set_coroutine(task& coroutine) noexcept {\n    SEASTAR_ASSERT(_promise);\n    _promise->set_task(&coroutine);\n}\n\n}\n"
  },
  {
    "path": "src/core/io_queue.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n\n#include <array>\n#include <chrono>\n#include <cstdint>\n#include <mutex>\n#include <utility>\n#include <fmt/format.h>\n#include <fmt/ostream.h>\n#include <boost/intrusive/parent_from_member.hpp>\n#include <boost/container/small_vector.hpp>\n#include <sys/uio.h>\n#include <seastar/util/assert.hh>\n#include <seastar/util/integrated-length.hh>\n\n#include <seastar/core/file.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/internal/io_desc.hh>\n#include <seastar/core/internal/io_sink.hh>\n#include <seastar/core/io_priority_class.hh>\n#include <seastar/util/log.hh>\n\nnamespace seastar {\n\nlogger io_log(\"io\");\n\nusing namespace std::chrono_literals;\nusing io_direction_and_length = internal::io_direction_and_length;\nstatic constexpr auto io_direction_read = io_direction_and_length::read_idx;\nstatic constexpr auto io_direction_write = io_direction_and_length::write_idx;\n\nstruct default_io_exception_factory {\n    static auto cancelled() {\n        return cancelled_error();\n    }\n};\n\nio_throttler::io_throttler(config cfg, unsigned nr_queues)\n        : _token_bucket(fixed_point_factor,\n                        std::max<capacity_t>(fixed_point_factor * token_bucket_t::rate_cast(cfg.rate_limit_duration).count(), tokens_capacity(cfg.limit_min_tokens)),\n                        tokens_capacity(cfg.min_tokens)\n                       )\n        , _per_tick_threshold(_token_bucket.limit() / nr_queues)\n{\n    if (tokens_capacity(cfg.min_tokens) > _token_bucket.threshold()) {\n        throw std::runtime_error(\"Fair-group replenisher limit is lower than threshold\");\n    }\n}\n\nauto io_throttler::grab_capacity(capacity_t cap) noexcept -> capacity_t {\n    SEASTAR_ASSERT(cap <= _token_bucket.limit());\n    return _token_bucket.grab(cap);\n}\n\nvoid io_throttler::replenish_capacity(clock_type::time_point now) noexcept {\n    _token_bucket.replenish(now);\n}\n\nvoid io_throttler::refund_tokens(capacity_t cap) noexcept {\n    _token_bucket.refund(cap);\n}\n\nvoid io_throttler::maybe_replenish_capacity(clock_type::time_point& local_ts) noexcept {\n    auto now = clock_type::now();\n    auto extra = _token_bucket.accumulated_in(now - local_ts);\n\n    if (extra >= _token_bucket.threshold()) {\n        local_ts = now;\n        replenish_capacity(now);\n    }\n}\n\nauto io_throttler::capacity_deficiency(capacity_t from) const noexcept -> capacity_t {\n    return _token_bucket.deficiency(from);\n}\n\nstruct io_group::priority_class_data {\n    priority_class_group_data* parent;\n\n    using token_bucket_t = internal::shared_token_bucket<uint64_t, std::ratio<1>, internal::capped_release::no>;\n\n    static constexpr uint64_t bandwidth_burst_in_blocks = 10 << (20 - io_queue::block_size_shift); // 10MB\n    static constexpr uint64_t bandwidth_threshold_in_blocks = 128 << (10 - io_queue::block_size_shift); // 128kB\n    token_bucket_t tb;\n\n    static uint64_t tokens(size_t length) noexcept {\n        return length >> io_queue::block_size_shift;\n    }\n\n    void update_bandwidth(uint64_t bandwidth) {\n        if (bandwidth >> io_queue::block_size_shift > tb.max_rate) {\n            // It's ... tooooo big value indeed\n            throw std::runtime_error(format(\"Too large rate, maximum is {}MB/s\", tb.max_rate >> (20 - io_queue::block_size_shift)));\n        }\n\n        tb.update_rate(tokens(bandwidth));\n    }\n\n    priority_class_data(priority_class_group_data* p) noexcept\n            : parent(p)\n            , tb(std::numeric_limits<uint64_t>::max(), bandwidth_burst_in_blocks, bandwidth_threshold_in_blocks)\n    {\n    }\n};\n\nclass io_queue::priority_class_data {\n    io_queue& _queue;\n    const internal::priority_class _pc;\n    uint32_t _shares;\n    struct {\n        size_t bytes = 0;\n        uint64_t ops = 0;\n\n        void add(size_t len) noexcept {\n            ops++;\n            bytes += len;\n        }\n    } _rwstat[2] = {}, _splits = {};\n    util::integrated_length<unsigned short, lowres_clock> _nr_queued;\n    util::integrated_length<unsigned short, lowres_clock> _nr_executing;\n    std::chrono::duration<double> _queue_time;\n    std::chrono::duration<double> _total_queue_time;\n    std::chrono::duration<double> _total_execution_time;\n    std::chrono::duration<double> _starvation_time;\n    io_queue::clock_type::time_point _activated;\n\n    class bandwidth_throttler {\n        io_group::priority_class_data::token_bucket_t& _tb;\n        uint64_t _replenish_head;\n        priority_class_data& _pc;\n        std::optional<unsigned> _group;\n        timer<lowres_clock> _replenish;\n\n        void throttle() noexcept {\n            if (_group) {\n                _pc._queue.throttle_priority_class_group(*_group);\n            } else {\n                _pc._queue.throttle_priority_class(_pc);\n            }\n        }\n\n        void unthrottle() noexcept {\n            if (_group) {\n                _pc._queue.unthrottle_priority_class_group(*_group);\n            } else {\n                _pc._queue.unthrottle_priority_class(_pc);\n            }\n        }\n\n        void try_to_replenish() noexcept {\n            _tb.replenish(io_queue::clock_type::now());\n            auto delta = _tb.deficiency(_replenish_head);\n            if (delta > 0) {\n                _replenish.arm(std::chrono::duration_cast<std::chrono::microseconds>(_tb.duration_for(delta)));\n            } else {\n                unthrottle();\n            }\n        }\n\n    public:\n        bandwidth_throttler(io_group::priority_class_data& pg, priority_class_data& pc, std::optional<unsigned> g) noexcept\n                : _tb(pg.tb)\n                , _pc(pc)\n                , _group(g)\n                , _replenish([this] { try_to_replenish(); })\n        {}\n\n        void grab(uint64_t tokens) noexcept {\n            auto ph = _tb.grab(tokens);\n            auto delta = _tb.deficiency(ph);\n            if (delta > 0) {\n                throttle();\n                _replenish_head = ph;\n                _replenish.arm(std::chrono::duration_cast<std::chrono::microseconds>(_tb.duration_for(delta)));\n            }\n        }\n    };\n\n    boost::container::static_vector<bandwidth_throttler, 2> _bw;\n\npublic:\n    void update_shares(uint32_t shares) noexcept {\n        _shares = std::max(shares, 1u);\n    }\n\n    priority_class_data(internal::priority_class pc, uint32_t shares, io_queue& q, io_group::priority_class_data& pg, std::optional<unsigned> group_index)\n        : _queue(q)\n        , _pc(pc)\n        , _shares(shares)\n        , _nr_queued(0)\n        , _nr_executing(0)\n        , _queue_time(0)\n        , _total_queue_time(0)\n        , _total_execution_time(0)\n        , _starvation_time(0)\n    {\n        _bw.emplace_back(pg, *this, std::nullopt);\n        if (pg.parent != nullptr) {\n            _bw.emplace_back(*pg.parent, *this, group_index);\n        }\n    }\n    priority_class_data(const priority_class_data&) = delete;\n    priority_class_data(priority_class_data&&) = delete;\n\n    ~priority_class_data() {\n        SEASTAR_ASSERT(_nr_queued == 0);\n        SEASTAR_ASSERT(_nr_executing == 0);\n    }\n\n    void on_queue() noexcept {\n        _nr_queued++;\n        if (_nr_executing == 0 && _nr_queued == 1) {\n            _activated = io_queue::clock_type::now();\n        }\n    }\n\n    void on_dispatch(io_direction_and_length dnl, std::chrono::duration<double> lat) noexcept {\n        _rwstat[dnl.rw_idx()].add(dnl.length());\n        _queue_time = lat;\n        _total_queue_time += lat;\n        _nr_queued--;\n        _nr_executing++;\n        if (_nr_executing == 1) {\n            _starvation_time += io_queue::clock_type::now() - _activated;\n        }\n\n        auto tokens = io_group::priority_class_data::tokens(dnl.length());\n        for (auto& bw : _bw) {\n            bw.grab(tokens);\n        }\n    }\n\n    void on_cancel() noexcept {\n        _nr_queued--;\n    }\n\n    void on_complete(std::chrono::duration<double> lat) noexcept {\n        _total_execution_time += lat;\n        _nr_executing--;\n        if (_nr_executing == 0 && _nr_queued != 0) {\n            _activated = io_queue::clock_type::now();\n        }\n    }\n\n    void on_error() noexcept {\n        _nr_executing--;\n        if (_nr_executing == 0 && _nr_queued != 0) {\n            _activated = io_queue::clock_type::now();\n        }\n    }\n\n    void on_split(io_direction_and_length dnl) noexcept {\n        _splits.add(dnl.length());\n    }\n\n    void on_before_dispatch() noexcept {\n        _nr_queued.checkpoint();\n    }\n\n    void on_after_dispatch() noexcept {\n        _nr_executing.checkpoint();\n    }\n\n    fair_queue::class_id fq_class() const noexcept { return _pc.id(); }\n\n    std::vector<seastar::metrics::impl::metric_definition_impl> metrics();\n    metrics::metric_groups metric_groups;\n};\n\nclass io_desc_read_write final : public io_completion {\n    io_queue& _ioq;\n    io_queue::priority_class_data& _pclass;\n    io_queue::clock_type::time_point _ts;\n    const stream_id _stream;\n    const io_direction_and_length _dnl;\n    const fair_queue_entry::capacity_t _fq_capacity;\n    promise<size_t> _pr;\n    iovec_keeper _iovs;\n    uint64_t _dispatched_polls;\n\npublic:\n    io_desc_read_write(io_queue& ioq, io_queue::priority_class_data& pc, stream_id stream, io_direction_and_length dnl, fair_queue_entry::capacity_t cap, iovec_keeper iovs)\n        : _ioq(ioq)\n        , _pclass(pc)\n        , _ts(io_queue::clock_type::now())\n        , _stream(stream)\n        , _dnl(dnl)\n        , _fq_capacity(cap)\n        , _iovs(std::move(iovs))\n    {\n        io_log.trace(\"dev {} : req {} queue  len {} capacity {}\", _ioq.id(), fmt::ptr(this), _dnl.length(), _fq_capacity);\n    }\n\n    virtual void set_exception(std::exception_ptr eptr) noexcept override {\n        io_log.trace(\"dev {} : req {} error\", _ioq.id(), fmt::ptr(this));\n        _pclass.on_error();\n        _ioq.complete_request(*this, std::chrono::duration<double>(0.0));\n        _pr.set_exception(eptr);\n        delete this;\n    }\n\n    virtual void complete(size_t res) noexcept override {\n        io_log.trace(\"dev {} : req {} complete\", _ioq.id(), fmt::ptr(this));\n        auto now = io_queue::clock_type::now();\n        auto delay = std::chrono::duration_cast<std::chrono::duration<double>>(now - _ts);\n        _pclass.on_complete(delay);\n        _ioq.complete_request(*this, delay);\n        _pr.set_value(res);\n        delete this;\n    }\n\n    void cancel() noexcept {\n        _pclass.on_cancel();\n        _pr.set_exception(std::make_exception_ptr(default_io_exception_factory::cancelled()));\n        delete this;\n    }\n\n    void dispatch() noexcept {\n        io_log.trace(\"dev {} : req {} submit\", _ioq.id(), fmt::ptr(this));\n        auto now = io_queue::clock_type::now();\n        _pclass.on_dispatch(_dnl, std::chrono::duration_cast<std::chrono::duration<double>>(now - _ts));\n        _ts = now;\n        _dispatched_polls = engine().polls();\n    }\n\n    future<size_t> get_future() {\n        return _pr.get_future();\n    }\n\n    fair_queue_entry::capacity_t capacity() const noexcept { return _fq_capacity; }\n    stream_id stream() const noexcept { return _stream; }\n    uint64_t polls() const noexcept { return _dispatched_polls; }\n};\n\nclass queued_io_request : private internal::io_request {\n    io_queue& _ioq;\n    const stream_id _stream;\n    fair_queue_entry _fq_entry;\n    internal::cancellable_queue::link _intent;\n    std::unique_ptr<io_desc_read_write> _desc;\n\n    bool is_cancelled() const noexcept { return !_desc; }\n\npublic:\n    queued_io_request(internal::io_request req, io_queue& q, fair_queue_entry::capacity_t cap, io_queue::priority_class_data& pc, io_direction_and_length dnl, iovec_keeper iovs)\n        : io_request(std::move(req))\n        , _ioq(q)\n        , _stream(_ioq.request_stream(dnl))\n        , _fq_entry(cap)\n        , _desc(std::make_unique<io_desc_read_write>(_ioq, pc, _stream, dnl, cap, std::move(iovs)))\n    {\n    }\n\n    queued_io_request(queued_io_request&&) = delete;\n\n    void dispatch() noexcept {\n        if (is_cancelled()) {\n            _ioq.complete_cancelled_request(*this);\n            delete this;\n            return;\n        }\n\n        _intent.maybe_dequeue();\n        _desc->dispatch();\n        _ioq.submit_request(_desc.release(), std::move(*this));\n        delete this;\n    }\n\n    void cancel() noexcept {\n        _ioq.cancel_request(*this);\n        _desc.release()->cancel();\n    }\n\n    void set_intent(internal::cancellable_queue& cq) noexcept {\n        _intent.enqueue(cq);\n    }\n\n    future<size_t> get_future() noexcept { return _desc->get_future(); }\n    fair_queue_entry& queue_entry() noexcept { return _fq_entry; }\n    stream_id stream() const noexcept { return _stream; }\n\n    static queued_io_request& from_fq_entry(fair_queue_entry& ent) noexcept {\n        return *boost::intrusive::get_parent_from_member(&ent, &queued_io_request::_fq_entry);\n    }\n\n    static queued_io_request& from_cq_link(internal::cancellable_queue::link& link) noexcept {\n        return *boost::intrusive::get_parent_from_member(&link, &queued_io_request::_intent);\n    }\n};\n\nnamespace internal {\n\npriority_class::priority_class(const scheduling_group& sg) noexcept : _id(internal::scheduling_group_index(sg))\n{ }\n\ncancellable_queue::cancellable_queue(cancellable_queue&& o) noexcept\n        : _first(std::exchange(o._first, nullptr))\n        , _rest(std::move(o._rest)) {\n    if (_first != nullptr) {\n        _first->_ref = this;\n    }\n}\n\ncancellable_queue& cancellable_queue::operator=(cancellable_queue&& o) noexcept {\n    if (this != &o) {\n        _first = std::exchange(o._first, nullptr);\n        _rest = std::move(o._rest);\n        if (_first != nullptr) {\n            _first->_ref = this;\n        }\n    }\n    return *this;\n}\n\ncancellable_queue::~cancellable_queue() {\n    while (_first != nullptr) {\n        queued_io_request::from_cq_link(*_first).cancel();\n        pop_front();\n    }\n}\n\nvoid cancellable_queue::push_back(link& il) noexcept {\n    if (_first == nullptr) {\n        _first = &il;\n        il._ref = this;\n    } else {\n        new (&il._hook) bi::slist_member_hook<>();\n        _rest.push_back(il);\n    }\n}\n\nvoid cancellable_queue::pop_front() noexcept {\n    _first->_ref = nullptr;\n    if (_rest.empty()) {\n        _first = nullptr;\n    } else {\n        _first = &_rest.front();\n        _rest.pop_front();\n        _first->_hook.~slist_member_hook<>();\n        _first->_ref = this;\n    }\n}\n\nintent_reference::intent_reference(io_intent* intent) noexcept : _intent(intent) {\n    if (_intent != nullptr) {\n        intent->_refs.bind(*this);\n    }\n}\n\nio_intent* intent_reference::retrieve() const {\n    if (is_cancelled()) {\n        throw default_io_exception_factory::cancelled();\n    }\n\n    return _intent;\n}\n\nvoid io_sink::submit(io_completion* desc, io_request req) noexcept {\n    try {\n        _pending_io.emplace_back(std::move(req), desc);\n    } catch (...) {\n        desc->set_exception(std::current_exception());\n    }\n}\n\nstd::vector<io_request::part> io_request::split(size_t max_length) {\n    auto op = opcode();\n    if (op == operation::read || op == operation::write) {\n        return split_buffer(max_length);\n    }\n    if (op == operation::readv || op == operation::writev) {\n        return split_iovec(max_length);\n    }\n\n    io_log.error(\"Invalid operation for split: {}\", static_cast<int>(op));\n    std::abort();\n}\n\nstd::vector<io_request::part> io_request::split_buffer(size_t max_length) {\n    std::vector<part> ret;\n    // the layout of _read and _write should be identical, otherwise we need to\n    // have two different implementations for each of them\n    static_assert(std::is_same_v<decltype(_read), decltype(_write)>);\n    const auto& op = _read;\n    ret.reserve((op.size + max_length - 1) / max_length);\n\n    size_t off = 0;\n    do {\n        size_t len = std::min(op.size - off, max_length);\n        ret.push_back({ sub_req_buffer(off, len), len, {} });\n        off += len;\n    } while (off < op.size);\n\n    return ret;\n}\n\nstd::vector<io_request::part> io_request::split_iovec(size_t max_length) {\n    std::vector<part> parts;\n    std::vector<::iovec> vecs;\n    // the layout of _readv and _writev should be identical, otherwise we need to\n    // have two different implementations for each of them\n    static_assert(std::is_same_v<decltype(_readv), decltype(_writev)>);\n    const auto& op = _readv;\n    ::iovec* cur = op.iovec;\n    size_t pos = 0;\n    size_t off = 0;\n    ::iovec* end = cur + op.iov_len;\n    size_t remaining = max_length;\n\n    while (cur != end) {\n        ::iovec iov;\n        iov.iov_base = reinterpret_cast<char*>(cur->iov_base) + off;\n        iov.iov_len = cur->iov_len - off;\n\n        if (iov.iov_len <= remaining) {\n            remaining -= iov.iov_len;\n            vecs.push_back(std::move(iov));\n            cur++;\n            off = 0;\n            continue;\n        }\n\n        if (remaining > 0) {\n            iov.iov_len = remaining;\n            off += remaining;\n            vecs.push_back(std::move(iov));\n        }\n\n        auto req = sub_req_iovec(pos, vecs);\n        parts.push_back({ std::move(req), max_length, std::move(vecs) });\n        pos += max_length;\n        remaining = max_length;\n    }\n\n    if (vecs.size() > 0) {\n        SEASTAR_ASSERT(remaining < max_length);\n        auto req = sub_req_iovec(pos, vecs);\n        parts.push_back({ std::move(req), max_length - remaining, std::move(vecs) });\n    }\n\n    return parts;\n}\n\nsstring io_request::opname() const {\n    switch (opcode()) {\n    case io_request::operation::fdatasync:\n        return \"fdatasync\";\n    case io_request::operation::write:\n        return \"write\";\n    case io_request::operation::writev:\n        return \"vectored write\";\n    case io_request::operation::read:\n        return \"read\";\n    case io_request::operation::readv:\n        return \"vectored read\";\n    case io_request::operation::recv:\n        return \"recv\";\n    case io_request::operation::recvmsg:\n        return \"recvmsg\";\n    case io_request::operation::send:\n        return \"send\";\n    case io_request::operation::sendmsg:\n        return \"sendmsg\";\n    case io_request::operation::accept:\n        return \"accept\";\n    case io_request::operation::connect:\n        return \"connect\";\n    case io_request::operation::poll_add:\n        return \"poll add\";\n    case io_request::operation::poll_remove:\n        return \"poll remove\";\n    case io_request::operation::cancel:\n        return \"cancel\";\n    }\n    std::abort();\n}\n\nconst io_throttler& get_throttler(const io_queue& ioq, unsigned stream) {\n    return ioq._group->_fgs[stream];\n}\n\n} // internal namespace\n\ntemplate <typename T>\nvoid update_moving_average(T& result, T value, double factor) noexcept {\n    result = result * factor + value * (1.0 - factor);\n}\n\nvoid io_queue::update_flow_ratio() noexcept {\n    if (_requests_completed > _prev_completed) {\n        auto instant = double(_requests_dispatched - _prev_dispatched) / double(_requests_completed - _prev_completed);\n        update_moving_average(_flow_ratio, instant, get_config().flow_ratio_ema_factor);\n        _prev_dispatched = _requests_dispatched;\n        _prev_completed = _requests_completed;\n    }\n}\n\nvoid io_queue::lower_stall_threshold() noexcept {\n    auto new_threshold = _stall_threshold - std::chrono::milliseconds(1);\n    _stall_threshold = std::max(_stall_threshold_min, new_threshold);\n}\n\nvoid\nio_queue::complete_request(io_desc_read_write& desc, std::chrono::duration<double> delay) noexcept {\n    _requests_executing--;\n    _requests_completed++;\n    _streams[desc.stream()].fq.notify_request_finished(desc.capacity());\n\n    if (delay > _stall_threshold) {\n        _stall_threshold *= 2;\n        io_log.warn(\"Request took {:.3f}ms ({} polls) to execute, queued {} executing {}\",\n            std::chrono::duration_cast<std::chrono::duration<double, std::milli>>(delay).count(),\n            engine().polls() - desc.polls(), _queued_requests, _requests_executing);\n    }\n}\n\nfair_queue::config io_queue::make_fair_queue_config(const config& iocfg, sstring label) {\n    fair_queue::config cfg;\n    cfg.label = label;\n    cfg.forgiving_factor = io_throttler::fixed_point_factor * io_throttler::token_bucket_t::rate_cast(iocfg.tau).count();\n    return cfg;\n}\n\nio_queue::io_queue(io_group_ptr group, internal::io_sink& sink)\n    : _priority_classes()\n    , _group(std::move(group))\n    , _id(_group->_config.id)\n    , _sink(sink)\n    , _averaging_decay_timer([this] {\n        update_flow_ratio();\n        lower_stall_threshold();\n    })\n    , _stall_threshold_min(std::max(get_config().stall_threshold, 1ms))\n    , _stall_threshold(_stall_threshold_min)\n    , _physical_block_size(get_config().physical_block_size)\n{\n    auto& cfg = get_config();\n    if (cfg.duplex) {\n        static_assert(internal::io_direction_and_length::write_idx == 0);\n        _streams.emplace_back(_group->_fgs[0], make_fair_queue_config(cfg, \"write\"));\n        static_assert(internal::io_direction_and_length::read_idx == 1);\n        _streams.emplace_back(_group->_fgs[1], make_fair_queue_config(cfg, \"read\"));\n    } else {\n        _streams.emplace_back(_group->_fgs[0], make_fair_queue_config(cfg, \"rw\"));\n    }\n    _averaging_decay_timer.arm_periodic(std::chrono::duration_cast<std::chrono::milliseconds>(_group->io_latency_goal() * cfg.averaging_decay_ticks));\n\n    namespace sm = seastar::metrics;\n    auto owner_l = sm::shard_label(this_shard_id());\n    auto mnt_l = sm::label(\"mountpoint\")(mountpoint());\n    auto group_l = sm::label(\"iogroup\")(to_sstring(_group->_allocated_on));\n    _metric_groups.add_group(\"io_queue\", {\n        sm::make_gauge(\"flow_ratio\", [this] { return _flow_ratio; },\n                sm::description(\"Ratio of dispatch rate to completion rate. Is expected to be 1.0+ growing larger on reactor stalls or (!) disk problems\"),\n                { owner_l, mnt_l, group_l }),\n    });\n}\n\nio_throttler::config io_group::configure_throttler(const io_queue::config& qcfg) noexcept {\n    io_throttler::config cfg;\n    cfg.label = fmt::format(\"io-queue-{}\", qcfg.id);\n    double min_weight = std::min(io_queue::read_request_base_count, qcfg.disk_req_write_to_read_multiplier);\n    double min_size = std::min(io_queue::read_request_base_count, qcfg.disk_blocks_write_to_read_multiplier);\n    cfg.min_tokens = min_weight / qcfg.req_count_rate + min_size / qcfg.blocks_count_rate;\n    double limit_min_weight = std::max(io_queue::read_request_base_count, qcfg.disk_req_write_to_read_multiplier);\n    double limit_min_size = std::max(io_queue::read_request_base_count, qcfg.disk_blocks_write_to_read_multiplier) * qcfg.block_count_limit_min;\n    cfg.limit_min_tokens = limit_min_weight / qcfg.req_count_rate + limit_min_size / qcfg.blocks_count_rate;\n    cfg.rate_limit_duration = qcfg.rate_limit_duration;\n    return cfg;\n}\n\nstd::chrono::duration<double> io_group::io_latency_goal() const noexcept {\n    return _fgs.front().rate_limit_duration();\n}\n\nio_group::io_group(io_queue::config io_cfg, unsigned nr_queues)\n    : _config(std::move(io_cfg))\n    , _allocated_on(this_shard_id())\n{\n    auto throttler_config = configure_throttler(_config);\n    _fgs.emplace_back(throttler_config, nr_queues);\n    if (_config.duplex) {\n        _fgs.emplace_back(throttler_config, nr_queues);\n    }\n\n    auto goal = io_latency_goal();\n    auto lvl = goal > 1.1 * _config.rate_limit_duration ? log_level::warn : log_level::debug;\n    io_log.log(lvl, \"IO queue uses {:.2f}ms latency goal for {}\", goal.count() * 1000, _config.mountpoint);\n\n    /*\n     * The maximum request size shouldn't result in the capacity that would\n     * be larger than the group's replenisher limit.\n     *\n     * To get the correct value check the 2^N sizes and find the largest one\n     * with little enough capacity. Actually (FIXME) requests should calculate\n     * capacities instead of request_fq_ticket() so this math would go away.\n     */\n    auto update_max_size = [this] (unsigned idx) {\n        auto g_idx = _config.duplex ? idx : 0;\n        const auto& fg = _fgs[g_idx];\n        auto max_cap = fg.maximum_capacity();\n        for (unsigned shift = 0; ; shift++) {\n            unsigned long req_length = 1ul << (shift + io_queue::block_size_shift);\n\n            auto tokens = internal::request_tokens(io_direction_and_length(idx, req_length), _config);\n            auto cap = fg.tokens_capacity(tokens);\n            if (cap > max_cap) {\n                if (shift == 0) {\n                    throw std::runtime_error(\"IO-group limits are too low\");\n                }\n\n                // The current shift will give us the biggest power of 2 request length for which\n                // capacity is still less than max_cap, but request_length_limit might be a better\n                // fit if tokens_capacity for that is closer to max_cap.\n                auto fitting_req_length = 1ul << (shift - 1 + io_queue::block_size_shift);\n                auto fitting_cap = fg.tokens_capacity(internal::request_tokens(io_direction_and_length(idx, fitting_req_length), _config));\n                auto cap_limit = fg.tokens_capacity(internal::request_tokens(io_direction_and_length(idx, io_group::request_length_limit), _config));\n                if (fitting_cap < cap_limit && cap_limit < max_cap) {\n                    _max_request_length[idx] = io_group::request_length_limit;\n                } else {\n                    _max_request_length[idx] = fitting_req_length;\n                }\n                break;\n            }\n\n            // Break this loop if the calculated request length gets larger than the\n            // request length limit. This avoids an infinite loop if the\n            // configured io_properties are huge and capacity ends up being always 0.\n            // _max_request_length will hold default values of io_group::request_length_limit\n            if (req_length > io_group::request_length_limit) {\n                io_log.info(\"IO queue was unable to find a suitable maximum request length, the search was cut-off early at: {}MB\", io_group::request_length_limit >> 20);\n                break;\n            }\n        };\n    };\n\n    update_max_size(io_direction_write);\n    update_max_size(io_direction_read);\n}\n\nio_group::~io_group() {\n}\n\nio_queue::~io_queue() {\n    // It is illegal to stop the I/O queue with pending requests.\n    // Technically we would use a gate to guarantee that. But here, it is not\n    // needed since this is expected to be destroyed only after the reactor is destroyed.\n    //\n    // And that will happen only when there are no more fibers to run. If we ever change\n    // that, then this has to change.\n    SEASTAR_ASSERT(_queued_requests == 0);\n    for (auto&& pc_data : _priority_classes) {\n        if (pc_data) {\n            for (auto&& s : _streams) {\n                s.fq.unregister_priority_class(pc_data->fq_class());\n            }\n        }\n    }\n}\n\nstd::vector<seastar::metrics::impl::metric_definition_impl> io_queue::priority_class_data::metrics() {\n    namespace sm = seastar::metrics;\n    return std::vector<sm::impl::metric_definition_impl>({\n            sm::make_counter(\"total_bytes\", [this] {\n                    return _rwstat[io_direction_read].bytes + _rwstat[io_direction_write].bytes;\n                }, sm::description(\"Total bytes passed in the queue\")),\n            sm::make_counter(\"total_operations\", [this] {\n                    return _rwstat[io_direction_read].ops + _rwstat[io_direction_write].ops;\n                }, sm::description(\"Total operations passed in the queue\")),\n            sm::make_counter(\"total_read_bytes\", _rwstat[io_direction_read].bytes,\n                    sm::description(\"Total read bytes passed in the queue\")),\n            sm::make_counter(\"total_read_ops\", _rwstat[io_direction_read].ops,\n                    sm::description(\"Total read operations passed in the queue\")),\n            sm::make_counter(\"total_write_bytes\", _rwstat[io_direction_write].bytes,\n                    sm::description(\"Total write bytes passed in the queue\")),\n            sm::make_counter(\"total_write_ops\", _rwstat[io_direction_write].ops,\n                    sm::description(\"Total write operations passed in the queue\")),\n            sm::make_counter(\"total_split_ops\", _splits.ops,\n                    sm::description(\"Total number of requests split\")),\n            sm::make_counter(\"total_split_bytes\", _splits.bytes,\n                    sm::description(\"Total number of bytes split\")),\n            sm::make_counter(\"total_delay_sec\", [this] {\n                    return _total_queue_time.count();\n                }, sm::description(\"Total time spent in the queue\")),\n            sm::make_counter(\"total_exec_sec\", [this] {\n                    return _total_execution_time.count();\n                }, sm::description(\"Total time spent in disk\")),\n            sm::make_counter(\"starvation_time_sec\", [this] {\n                auto st = _starvation_time;\n                if (_nr_queued != 0 && _nr_executing == 0) {\n                    st += io_queue::clock_type::now() - _activated;\n                }\n                return st.count();\n            }, sm::description(\"Total time spent starving for disk\")),\n\n            // Note: The counter below is not the same as reactor's queued-io-requests\n            // queued-io-requests shows us how many requests in total exist in this I/O Queue.\n            //\n            // This counter lives in the priority class, so it will count only queued requests\n            // that belong to that class.\n            //\n            // In other words: the new counter tells you how busy a class is, and the\n            // old counter tells you how busy the system is.\n\n            sm::make_queue_length(\"queue_length\", [this] { return _nr_queued.value(); }, sm::description(\"Number of requests in the queue\")),\n            sm::make_queue_length(\"disk_queue_length\", [this] { return _nr_executing.value(); }, sm::description(\"Number of requests in the disk\")),\n            sm::make_gauge(\"delay\", [this] {\n                return _queue_time.count();\n            }, sm::description(\"random delay time in the queue\")),\n            sm::make_gauge(\"shares\", _shares, sm::description(\"current amount of shares\")),\n\n            sm::make_counter(\"integrated_queue_length\", [this] { return _nr_queued.integral(); }, sm::description(\"Integrated queue length\")),\n            sm::make_counter(\"integrated_disk_queue_length\", [this] { return _nr_executing.integral(); }, sm::description(\"Integrated disk queue length\")),\n    });\n}\n\nvoid io_queue::register_stats(sstring name, priority_class_data& pc) {\n    namespace sm = seastar::metrics;\n    seastar::metrics::metric_groups new_metrics;\n\n    auto owner_l = sm::shard_label(this_shard_id());\n    auto mnt_l = sm::label(\"mountpoint\")(mountpoint());\n    auto class_l = sm::label(\"class\")(name);\n    auto group_l = sm::label(\"iogroup\")(to_sstring(_group->_allocated_on));\n\n    std::vector<sm::metric_definition> metrics;\n    for (auto&& m : pc.metrics()) {\n        m(owner_l)(mnt_l)(class_l)(group_l);\n        metrics.emplace_back(std::move(m));\n    }\n\n    for (auto&& s : _streams) {\n        for (auto&& m : s.metrics(pc)) {\n            m(owner_l)(mnt_l)(class_l)(group_l)(sm::label(\"stream\")(s.fq.label()));\n            metrics.emplace_back(std::move(m));\n        }\n    }\n\n    new_metrics.add_group(\"io_queue\", std::move(metrics));\n    pc.metric_groups = std::exchange(new_metrics, {});\n}\n\nio_queue::priority_class_data& io_queue::find_or_create_class(internal::priority_class pc) {\n    auto id = pc.id();\n    if (id >= _priority_classes.size()) {\n        _priority_classes.resize(id + 1);\n    }\n    if (!_priority_classes[id]) {\n        auto sg = internal::scheduling_group_from_index(id);\n        auto ssg = internal::scheduling_supergroup_for(sg);\n\n        // A note on naming:\n        //\n        // We could just add the owner as the instance id and have something like:\n        //  io_queue-<class_owner>-<counter>-<class_name>\n        //\n        // However, when there are more than one shard per I/O queue, it is very useful\n        // to know which shards are being served by the same queue. Therefore, a better name\n        // scheme is:\n        //\n        //  io_queue-<queue_owner>-<counter>-<class_name>, shard=<class_owner>\n        //  using the shard label to hold the owner number\n        //\n        // This conveys all the information we need and allows one to easily group all classes from\n        // the same I/O queue (by filtering by shard)\n\n        std::optional<unsigned> group_index;\n        if (!ssg.is_root()) {\n            group_index = ssg.index();\n            for (auto&& s : _streams) {\n                s.fq.ensure_priority_group(*group_index, ssg.get_shares());\n            }\n        }\n\n        auto& pg = _group->find_or_create_class(pc, group_index);\n\n        auto shares = sg.get_shares();\n        auto pc_data = std::make_unique<priority_class_data>(pc, shares, *this, pg, group_index);\n        for (auto&& s : _streams) {\n            s.fq.register_priority_class(pc_data->fq_class(), shares, group_index);\n        }\n        register_stats(sg.name(), *pc_data);\n\n        _priority_classes[id] = std::move(pc_data);\n    }\n    return *_priority_classes[id];\n}\n\nio_group::priority_class_group_data& io_group::find_or_create_class_group(unsigned group_index) {\n    std::lock_guard _(_lock);\n    return find_or_create_class_group_locked(group_index);\n}\n\nio_group::priority_class_group_data& io_group::find_or_create_class_group_locked(unsigned id) {\n    if (id >= _priority_groups.size()) {\n        _priority_groups.resize(id + 1);\n    }\n    if (!_priority_groups[id]) {\n        auto pg = std::make_unique<priority_class_group_data>(nullptr);\n        _priority_groups[id] = std::move(pg);\n    }\n    return *_priority_groups[id];\n}\n\nio_group::priority_class_data& io_group::find_or_create_class(internal::priority_class pc) {\n    auto sg = internal::scheduling_group_from_index(pc.id());\n    auto ssg = internal::scheduling_supergroup_for(sg);\n    std::optional<unsigned> group_index;\n    if (!ssg.is_root()) {\n        group_index = ssg.index();\n    }\n    return find_or_create_class(pc, group_index);\n}\n\nio_group::priority_class_data& io_group::find_or_create_class(internal::priority_class pc, std::optional<unsigned> group_index) {\n    std::lock_guard _(_lock);\n\n    priority_class_group_data* parent = nullptr;\n    if (group_index.has_value()) {\n        parent = &find_or_create_class_group_locked(*group_index);\n    }\n\n    auto id = pc.id();\n    if (id >= _priority_classes.size()) {\n        _priority_classes.resize(id + 1);\n    }\n    if (!_priority_classes[id]) {\n        auto pg = std::make_unique<priority_class_data>(parent);\n        _priority_classes[id] = std::move(pg);\n    }\n\n    return *_priority_classes[id];\n}\n\nstream_id io_queue::request_stream(io_direction_and_length dnl) const noexcept {\n    return get_config().duplex ? dnl.rw_idx() : 0;\n}\n\ndouble internal::request_tokens(io_direction_and_length dnl, const io_queue::config& cfg) noexcept {\n    struct {\n        unsigned weight;\n        unsigned size;\n    } mult[2];\n\n    mult[io_direction_write] = {\n        cfg.disk_req_write_to_read_multiplier,\n        cfg.disk_blocks_write_to_read_multiplier,\n    };\n    mult[io_direction_read] = {\n        io_queue::read_request_base_count,\n        io_queue::read_request_base_count,\n    };\n\n    const auto& m = mult[dnl.rw_idx()];\n\n    return double(m.weight) / cfg.req_count_rate + double(m.size) * (dnl.length() >> io_queue::block_size_shift) / cfg.blocks_count_rate;\n}\n\nfair_queue_entry::capacity_t io_queue::request_capacity(io_direction_and_length dnl) const noexcept {\n    const auto& cfg = get_config();\n    auto tokens = internal::request_tokens(dnl, cfg);\n    if (_flow_ratio <= cfg.flow_ratio_backpressure_threshold) {\n        return _streams[request_stream(dnl)].out.tokens_capacity(tokens);\n    }\n\n    auto stream = request_stream(dnl);\n    auto cap = _streams[stream].out.tokens_capacity(tokens * _flow_ratio);\n    auto max_cap = _streams[stream].out.maximum_capacity();\n    return std::min(cap, max_cap);\n}\n\nio_queue::request_limits io_queue::get_request_limits() const noexcept {\n    request_limits l;\n    l.max_read = align_down<size_t>(std::min<size_t>(get_config().disk_read_saturation_length, _group->_max_request_length[io_direction_read]), 1 << block_size_shift);\n    l.max_write = align_down<size_t>(std::min<size_t>(get_config().disk_write_saturation_length, _group->_max_request_length[io_direction_write]), 1 << block_size_shift);\n    return l;\n}\n\nstd::chrono::duration<double> io_queue::get_io_latency_goal() const noexcept {\n    return _group->io_latency_goal();\n}\n\nfuture<size_t> io_queue::queue_one_request(internal::priority_class pc, io_direction_and_length dnl, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept {\n    return futurize_invoke([pc = std::move(pc), dnl = std::move(dnl), req = std::move(req), this, intent, iovs = std::move(iovs)] () mutable {\n        // First time will hit here, and then we create the class. It is important\n        // that we create the shared pointer in the same shard it will be used at later.\n        auto& pclass = find_or_create_class(pc);\n        auto cap = request_capacity(dnl);\n        auto queued_req = std::make_unique<queued_io_request>(std::move(req), *this, cap, pclass, std::move(dnl), std::move(iovs));\n        auto fut = queued_req->get_future();\n        if (intent != nullptr) {\n            auto& cq = intent->find_or_create_cancellable_queue(_id, pc.id());\n            queued_req->set_intent(cq);\n        }\n\n        _streams[queued_req->stream()].fq.queue(pclass.fq_class(), queued_req->queue_entry());\n        queued_req.release();\n        pclass.on_queue();\n        _queued_requests++;\n        return fut;\n    });\n}\n\nfuture<size_t> io_queue::queue_request(internal::priority_class pc, io_direction_and_length dnl, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept {\n    size_t max_length = _group->_max_request_length[dnl.rw_idx()];\n\n    if (__builtin_expect(dnl.length() <= max_length, true)) {\n        return queue_one_request(std::move(pc), dnl, std::move(req), intent, std::move(iovs));\n    }\n\n    std::vector<internal::io_request::part> parts;\n    lw_shared_ptr<std::vector<future<size_t>>> p;\n\n    try {\n        parts = req.split(max_length);\n        p = make_lw_shared<std::vector<future<size_t>>>();\n        p->reserve(parts.size());\n        find_or_create_class(pc).on_split(dnl);\n        reactor::io_stats::local().aio_outsizes++;\n    } catch (...) {\n        return current_exception_as_future<size_t>();\n    }\n\n    // No exceptions from now on. If queue_one_request fails it will resolve\n    // into exceptional future which will be picked up by when_all() below\n    for (auto&& part : parts) {\n        auto f = queue_one_request(pc, io_direction_and_length(dnl.rw_idx(), part.size), std::move(part.req), intent, std::move(part.iovecs));\n        p->push_back(std::move(f));\n    }\n\n    return when_all(p->begin(), p->end()).then([p, max_length] (auto results) {\n        bool prev_ok = true;\n        size_t total = 0;\n        std::exception_ptr ex;\n\n        for (auto&& res : results) {\n            if (!res.failed()) {\n                if (prev_ok) {\n                    size_t sz = res.get();\n                    total += sz;\n                    prev_ok &= (sz == max_length);\n                }\n            } else {\n                if (!ex) {\n                    ex = res.get_exception();\n                } else {\n                    res.ignore_ready_future();\n                }\n                prev_ok = false;\n            }\n        }\n\n        if (total > 0) {\n            return make_ready_future<size_t>(total);\n        } else if (ex) {\n            return make_exception_future<size_t>(std::move(ex));\n        } else {\n            return make_ready_future<size_t>(0);\n        }\n    });\n}\n\nfuture<size_t> io_queue::submit_io_read(size_t len, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept {\n    internal::priority_class pc = internal::priority_class(current_scheduling_group());\n    auto& io_stats = reactor::io_stats::local();\n    ++io_stats.aio_reads;\n    io_stats.aio_read_bytes += len;\n    return queue_request(std::move(pc), io_direction_and_length(io_direction_read, len), std::move(req), intent, std::move(iovs));\n}\n\nfuture<size_t> io_queue::submit_io_write(size_t len, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept {\n    internal::priority_class pc = internal::priority_class(current_scheduling_group());\n    auto& io_stats = reactor::io_stats::local();\n    ++io_stats.aio_writes;\n    io_stats.aio_write_bytes += len;\n    return queue_request(std::move(pc), io_direction_and_length(io_direction_write, len), std::move(req), intent, std::move(iovs));\n}\n\n// This function is called by the shard on every poll.\n// It picks up tokens granted by the group, spends available tokens on IO dispatches,\n// and makes a reservation for more tokens, if needed.\n//\n// Reservations are done in batches of size `_group.per_tick_grab_threshold()`.\n// During contention, in an average moment in time each contending shard can be expected to\n// be holding a reservation of such size after the current head of the token bucket.\n//\n// A shard which is currently calling `dispatch_requests()` can expect a latency\n// of at most `nr_contenders * (_group.per_tick_grab_threshold() + max_request_cap)` before its next reservation is fulfilled.\n// If a shard calls `dispatch_requests()` at least once per X total tokens, it should receive bandwidth\n// of at least `_group.per_tick_grab_threshold() / (X + nr_contenders * (_group.per_tick_grab_threshold() + max_request_cap))`.\n//\n// A shard which is polling continuously should be able to grab its fair share of the disk for itself.\n//\n// Given a task quota of 500us and IO latency goal of 750 us,\n// a CPU-starved shard should still be able to grab at least ~30% of its fair share in the worst case.\n// This is far from ideal, but it's something.\n\nvoid io_queue::poll_io_queue() {\n    for (auto& pc : _priority_classes) {\n        if (pc) {\n            pc->on_before_dispatch();\n        }\n    }\n\n    for (auto&& st : _streams) {\n        st.out.maybe_replenish_capacity(st.replenish);\n        auto available = st.reap_pending_capacity();\n\n        while (true) {\n            auto* ent = st.fq.top();\n            if (ent == nullptr) {\n                available.ready_tokens = 0;\n                break;\n            }\n\n            auto result = st.grab_capacity(ent->capacity(), available);\n            if (result == stream::grab_result::stop) {\n                break;\n            }\n            if (result == stream::grab_result::again) {\n                continue;\n            }\n\n            st.fq.pop_front();\n            queued_io_request::from_fq_entry(*ent).dispatch();\n        }\n\n        SEASTAR_ASSERT(available.ready_tokens == 0);\n        // Note: if IO cancellation happens, it's possible that we are still holding some tokens in `ready` here.\n        //\n        // We could refund them to the bucket, but permanently refunding tokens (as opposed to only\n        // \"rotating\" the bucket like the earlier refund() calls in this function do) is theoretically\n        // unpleasant (it can bloat the bucket beyond its size limit, and its hard to write a correct\n        // countermeasure for that), so we just discard the tokens. There's no harm in it, IO cancellation\n        // can't have resource-saving guarantees anyway.\n    }\n\n    for (auto& pc : _priority_classes) {\n        if (pc) {\n            pc->on_after_dispatch();\n        }\n    }\n}\n\nvoid io_queue::submit_request(io_desc_read_write* desc, internal::io_request req) noexcept {\n    _queued_requests--;\n    _requests_executing++;\n    _requests_dispatched++;\n    _sink.submit(desc, std::move(req));\n}\n\nvoid io_queue::cancel_request(queued_io_request& req) noexcept {\n    _queued_requests--;\n    _streams[req.stream()].fq.notify_request_cancelled(req.queue_entry());\n}\n\nvoid io_queue::complete_cancelled_request(queued_io_request& req) noexcept {\n    _streams[req.stream()].fq.notify_request_finished(req.queue_entry().capacity());\n}\n\nio_queue::clock_type::time_point io_queue::next_pending_aio() const noexcept {\n    clock_type::time_point next = clock_type::time_point::max();\n\n    for (const auto& s : _streams) {\n        clock_type::time_point n = s.next_pending_aio();\n        if (n < next) {\n            next = std::move(n);\n        }\n    }\n\n    return next;\n}\n\nvoid\nio_queue::update_shares_for_class(internal::priority_class pc, size_t new_shares) {\n    auto& pclass = find_or_create_class(pc);\n    pclass.update_shares(new_shares);\n    for (auto&& s : _streams) {\n        s.fq.update_shares_for_class(pclass.fq_class(), new_shares);\n    }\n}\n\n\nvoid io_queue::update_shares_for_class_group(unsigned index, size_t new_shares) {\n    for (auto&& s : _streams) {\n        s.fq.ensure_priority_group(index, new_shares);\n    }\n}\n\nfuture<> io_queue::update_bandwidth_for_class(internal::priority_class pc, uint64_t new_bandwidth) {\n    return futurize_invoke([this, pc, new_bandwidth] {\n        if (_group->_allocated_on == this_shard_id()) {\n            auto& pclass = _group->find_or_create_class(pc);\n            pclass.update_bandwidth(new_bandwidth);\n            io_log.debug(\"Updated {} class bandwidth to {}MB/s\", pc.id(), new_bandwidth >> 20);\n        }\n    });\n}\n\nfuture<> io_queue::update_bandwidth_for_class_group(unsigned group_index, uint64_t new_bandwidth) {\n    return futurize_invoke([this, group_index, new_bandwidth] {\n        if (_group->_allocated_on == this_shard_id()) {\n            auto& pclass = _group->find_or_create_class_group(group_index);\n            pclass.update_bandwidth(new_bandwidth);\n            io_log.debug(\"Updated {} class group bandwidth to {}MB/s\", group_index, new_bandwidth >> 20);\n        }\n    });\n}\n\nvoid\nio_queue::rename_priority_class(internal::priority_class pc, sstring new_name) {\n    if (_priority_classes.size() > pc.id() &&\n            _priority_classes[pc.id()]) {\n        try {\n            register_stats(new_name, *_priority_classes[pc.id()]);\n        } catch (metrics::double_registration &e) {\n            // we need to ignore this exception, since it can happen that\n            // a class that was already created with the new name will be\n            // renamed again (this will cause a double registration exception\n            // to be thrown).\n        }\n    }\n}\n\nvoid io_queue::destroy_priority_class(internal::priority_class pc) noexcept {\n    if (_priority_classes.size() > pc.id() && _priority_classes[pc.id()]) {\n        auto& pc_ptr = _priority_classes[pc.id()];\n        for (auto&& s : _streams) {\n            s.fq.unregister_priority_class(pc_ptr->fq_class());\n        }\n        pc_ptr.reset();\n    }\n}\n\nvoid io_queue::throttle_priority_class(const priority_class_data& pc) noexcept {\n    for (auto&& s : _streams) {\n        s.fq.unplug_class(pc.fq_class());\n    }\n}\n\nvoid io_queue::unthrottle_priority_class(const priority_class_data& pc) noexcept {\n    for (auto&& s : _streams) {\n        s.fq.plug_class(pc.fq_class());\n    }\n}\n\nvoid io_queue::throttle_priority_class_group(unsigned group) noexcept {\n    for (auto&& s : _streams) {\n        s.fq.unplug_class_group(group);\n    }\n}\n\nvoid io_queue::unthrottle_priority_class_group(unsigned group) noexcept {\n    for (auto&& s : _streams) {\n        s.fq.plug_class_group(group);\n    }\n}\n\nauto io_queue::stream::reap_pending_capacity() noexcept -> reap_result {\n    auto result = reap_result{.ready_tokens = 0, .our_turn_has_come = true};\n    if (_pending.cap) {\n        capacity_t deficiency = out.capacity_deficiency(_pending.head);\n        result.our_turn_has_come = deficiency <= _pending.cap;\n        if (result.our_turn_has_come) {\n            result.ready_tokens = _pending.cap - deficiency;\n            _pending.cap = deficiency;\n        }\n    }\n    return result;\n}\n\nio_queue::clock_type::time_point io_queue::stream::next_pending_aio() const noexcept {\n    if (_pending.cap) {\n        /*\n         * We expect the disk to release the ticket within some time,\n         * but it's ... OK if it doesn't -- the pending wait still\n         * needs the head rover value to be ahead of the needed value.\n         *\n         * It may happen that the capacity gets released before we think\n         * it will, in this case we will wait for the full value again,\n         * which's sub-optimal. The expectation is that we think disk\n         * works faster, than it really does.\n         */\n        auto over = out.capacity_deficiency(_pending.head);\n        auto ticks = out.capacity_duration(over);\n        return std::chrono::steady_clock::now() + std::chrono::duration_cast<std::chrono::microseconds>(ticks);\n    }\n\n    return std::chrono::steady_clock::time_point::max();\n}\n\nauto io_queue::stream::grab_capacity(capacity_t cap, reap_result& available) -> grab_result {\n    const uint64_t max_unamortized_reservation = out.per_tick_grab_threshold();\n\n    if (cap <= available.ready_tokens) {\n        // We can dispatch the request immediately.\n        // We do that after the if-else.\n        available.ready_tokens -= cap;\n        return grab_result::ok;\n    } else if (cap <= available.ready_tokens + _pending.cap || _pending.cap >= max_unamortized_reservation) {\n        // We can't dispatch the request yet, but we already have a pending reservation\n        // which will provide us with enough tokens for it eventually,\n        // or our reservation is already max-size and we can't reserve more tokens until we reap some.\n        // So we should just wait.\n        // We return any immediately-available tokens back to `_pending`\n        // and we bail. The next `dispatch_request` will again take those tokens\n        // (possibly joined by some newly-granted tokens) and retry.\n        _pending.cap += available.ready_tokens;\n        available.ready_tokens = 0;\n        return grab_result::stop;\n    } else if (available.our_turn_has_come) {\n        // The current reservation isn't enough to fulfill the next request,\n        // and we can cancel it (because `our_turn_has_come == true`) and make a bigger one\n        // (because `_pending.cap < can_grab_this_tick`).\n        // So we cancel it and do a bigger one.\n\n        // We do token recycling here: we return the tokens which we have available, and the tokens we have reserved\n        // immediately after the group head, and we return them to the bucket, immediately grabbing the same amount from the tail.\n        // This is neutral to fairness. The bandwidth we consume is still influenced only by the\n        // `max_unarmortized_reservation` portions.\n        auto recycled = available.ready_tokens + _pending.cap;\n        capacity_t grab_amount = std::min<capacity_t>(recycled + max_unamortized_reservation, fq.queued_capacity());\n        // There's technically nothing wrong with grabbing more than `out.maximum_capacity()`,\n        // but the token bucket has an assert for that, and its a reasonable expectation, so let's respect that limit.\n        // It shouldn't matter in practice.\n        grab_amount = std::min<capacity_t>(grab_amount, out.maximum_capacity());\n        out.refund_tokens(recycled);\n        // Replace _pending with a new reservation starting at the current\n        // group bucket tail.\n        capacity_t want_head = out.grab_capacity(grab_amount);\n        _pending = pending{want_head, grab_amount};\n        available = reap_pending_capacity();\n        return grab_result::again;\n    } else {\n        // We can already see that our current reservation is going to be insufficient\n        // for the highest-priority request as of now. But since group head didn't touch\n        // it yet, there's no good way to cancel it, so we have no choice but to wait\n        // until the touch time.\n        SEASTAR_ASSERT(available.ready_tokens == 0);\n        return grab_result::stop;\n    }\n}\n\nstd::vector<seastar::metrics::impl::metric_definition_impl> io_queue::stream::metrics(const priority_class_data& pc) {\n    namespace sm = seastar::metrics;\n    auto c = pc.fq_class();\n    return std::vector<sm::impl::metric_definition_impl>({\n            sm::make_counter(\"consumption\",\n                    [this, c] { return io_throttler::capacity_tokens(fq.pure_accumulated(c)); },\n                    sm::description(\"Accumulated disk capacity units consumed by this class; an increment per-second rate indicates full utilization\")),\n            sm::make_counter(\"adjusted_consumption\",\n                    [this, c] { return io_throttler::capacity_tokens(fq.accumulated(c)); },\n                    sm::description(\"Consumed disk capacity units adjusted for class shares and idling preemption\")),\n            sm::make_counter(\"activations\",\n                    [this, c] { return fq.activations(c); },\n                    sm::description(\"The number of times the class was woken up from idle\")),\n    });\n}\n\n} // seastar namespace\n"
  },
  {
    "path": "src/core/linux-aio.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n\n#include <atomic>\n#include <algorithm>\n#include <cerrno>\n#include <cstring>\n#include <stdexcept>\n#include <fmt/format.h>\n#include <unistd.h>\n#include <sys/syscall.h>\n#include <valgrind/valgrind.h>\n\n#include <seastar/core/internal/linux-aio.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/read_first_line.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\nnamespace linux_abi {\n\nstruct linux_aio_ring {\n    uint32_t id;\n    uint32_t nr;\n    std::atomic<uint32_t> head;\n    std::atomic<uint32_t> tail;\n    uint32_t magic;\n    uint32_t compat_features;\n    uint32_t incompat_features;\n    uint32_t header_length;\n};\n\n}\n\nusing namespace linux_abi;\n\nstatic linux_aio_ring* to_ring(aio_context_t io_context) {\n    return reinterpret_cast<linux_aio_ring*>(uintptr_t(io_context));\n}\n\nstatic bool usable(const linux_aio_ring* ring) {\n    return ring->magic == 0xa10a10a1 && ring->incompat_features == 0 && !RUNNING_ON_VALGRIND;\n}\n\nint io_setup(int nr_events, aio_context_t* io_context) {\n    return ::syscall(SYS_io_setup, nr_events, io_context);\n}\n\nint io_destroy(aio_context_t io_context) noexcept {\n   return ::syscall(SYS_io_destroy, io_context);\n}\n\nint io_submit(aio_context_t io_context, long nr, iocb** iocbs) {\n    return ::syscall(SYS_io_submit, io_context, nr, iocbs);\n}\n\nint io_cancel(aio_context_t io_context, iocb* iocb, io_event* result) {\n    return ::syscall(SYS_io_cancel, io_context, iocb, result);\n}\n\nstatic int try_reap_events(aio_context_t io_context, long min_nr, long nr, io_event* events, const ::timespec* timeout,\n        bool force_syscall) {\n    auto ring = to_ring(io_context);\n    if (usable(ring) && !force_syscall) {\n        // Try to complete in userspace, if enough available events,\n        // or if the timeout is zero\n\n        // We're the only writer to ->head, so we can load with memory_order_relaxed (assuming\n        // only a single thread calls io_getevents()).\n        auto head = ring->head.load(std::memory_order_relaxed);\n        // The kernel will write to the ring from an interrupt and then release with a write\n        // to ring->tail, so we must memory_order_acquire here.\n        auto tail = ring->tail.load(std::memory_order_acquire); // kernel writes from interrupts\n        auto available = tail - head;\n        if (tail < head) {\n            available += ring->nr;\n        }\n        if (available >= uint32_t(min_nr)\n                || (timeout && timeout->tv_sec == 0 && timeout->tv_nsec == 0)) {\n            if (!available) {\n                return 0;\n            }\n            auto ring_events = reinterpret_cast<const io_event*>(uintptr_t(io_context) + ring->header_length);\n            auto now = std::min<uint32_t>(nr, available);\n            auto start = ring_events + head;\n            head += now;\n            if (head < ring->nr) {\n                std::copy(start, start + now, events);\n            } else {\n                head -= ring->nr;\n                auto p = std::copy(start, ring_events + ring->nr, events);\n                std::copy(ring_events, ring_events + head, p);\n            }\n            // The kernel will read ring->head and update its view of how many entries\n            // in the ring are available, so memory_order_release to make sure any ring\n            // accesses are completed before the update to ring->head is visible.\n            ring->head.store(head, std::memory_order_release);\n            return now;\n        }\n    }\n    return -1;\n}\n\nint io_getevents(aio_context_t io_context, long min_nr, long nr, io_event* events, const ::timespec* timeout,\n        bool force_syscall) {\n    auto r = try_reap_events(io_context, min_nr, nr, events, timeout, force_syscall);\n    if (r >= 0) {\n        return r;\n    }\n    return ::syscall(SYS_io_getevents, io_context, min_nr, nr, events, timeout);\n}\n\n\n#ifndef __NR_io_pgetevents\n\n#  if defined(__x86_64__)\n#    define __NR_io_pgetevents 333\n#  elif defined(__i386__)\n#    define __NR_io_pgetevents 385\n#  endif\n\n#endif\n\nint io_pgetevents(aio_context_t io_context, long min_nr, long nr, io_event* events, const ::timespec* timeout, const sigset_t* sigmask,\n        bool force_syscall) {\n#ifdef __NR_io_pgetevents\n    auto r = try_reap_events(io_context, min_nr, nr, events, timeout, force_syscall);\n    if (r >= 0) {\n        return r;\n    }\n    aio_sigset as;\n    as.sigmask = sigmask;\n    as.sigsetsize = 8;  // Can't use sizeof(*sigmask) because user and kernel sigset_t are inconsistent\n    return ::syscall(__NR_io_pgetevents, io_context, min_nr, nr, events, timeout, &as);\n#else\n    errno = ENOSYS;\n    return -1;\n#endif\n}\n\nvoid setup_aio_context(size_t nr, linux_abi::aio_context_t* io_context) {\n    auto r = io_setup(nr, io_context);\n    if (r < 0) {\n        char buf[1024];\n#ifdef SEASTAR_STRERROR_R_CHAR_P\n        const char *msg = strerror_r(errno, buf, sizeof(buf));\n#else\n        const char *msg = strerror_r(errno, buf, sizeof(buf)) ? \"unknown error\" : buf;\n#endif\n        if (errno == EAGAIN) {\n            auto aio_max_nr = read_first_line_as<unsigned>(\"/proc/sys/fs/aio-max-nr\");\n            throw std::runtime_error(\n                fmt::format(\"Could not setup Async I/O: {}. \"\n                            \"The required nr_events {} exceeds the capacity in /proc/sys/fs/aio-max-nr {}. \"\n                            \"Set /proc/sys/fs/aio-max-nr to at least {}.\",\n                            msg, nr, aio_max_nr, nr));\n        } else {\n            throw std::runtime_error(fmt::format(\"Could not setup Async I/O: {}\", msg));\n        }\n    }\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/core/memory.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n/// \\cond internal\n\n//\n// Seastar memory allocator\n//\n// This is a share-nothing allocator (memory allocated on one cpu must\n// be freed on the same cpu).\n//\n// Inspired by gperftools' tcmalloc.\n//\n// Memory map:\n//\n// 0x0000'sccc'vvvv'vvvv\n//\n// 0000 - required by architecture (only 48 bits of address space)\n// s    - chosen to satisfy system allocator (1-7)\n// ccc  - cpu number (0-12 bits allocated vary according to system)\n// v    - virtual address within cpu (32-44 bits, according to how much ccc\n//        leaves us\n//\n// Each page has a page structure that describes it.  Within a cpu's\n// memory pool, the page array starts at offset 0, describing all pages\n// within that pool.  Page 0 does not describe a valid page.\n//\n// Each pool can contain at most 2^32 pages (or 44 address bits), so we can\n// use a 32-bit integer to identify a page.\n//\n// Runs of pages are organized into spans.  Free spans are organized into lists,\n// by size.  When spans are broken up or coalesced, they may move into new lists.\n// Spans have a size that is a power-of-two and are naturally aligned (aka buddy\n// allocator)\n//\n// If compiled with SEASTAR_HEAPPROF seastar features a sampling memory\n// profiler. Allocations are sampled at random (see `sampler` for the sampling\n// logic) and tracked. The sampled live set can be retrieved with\n// `sampled_memory_profile()`. Sampled allocations carry an extra\n// allocation_site pointer with them which is used on free to remove them from\n// the sampled live set.\n//\n// Large allocations are tracked via a pointer to the allocation_site which is\n// stored on the page structure. To check whether an allocation was sampled or\n// not this pointer is being looked at on free.\n//\n// Small allocations store an extra 8 bytes at the end of their allocation.\n// Sampled allocations are allocated in a separate set of small pools. Hence, to\n// check whether an allocation was sampled or not one only has to look at the\n// tag in pool.\n//\n\n\n#include <concepts>\n#include <unordered_set>\n#include <iostream>\n#include <optional>\n#include <memory_resource>\n#include <thread>\n\n#include <seastar/util/assert.hh>\n\n#include <fmt/format.h>\n#include <fmt/ostream.h>\n\n#include <boost/container/static_vector.hpp>\n\n#include <dlfcn.h>\n\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n#include <new>\n#include <cstdint>\n#include <algorithm>\n#include <limits>\n#include <atomic>\n#include <mutex>\n#include <functional>\n#include <cstring>\n#include <utility>\n#include <boost/intrusive/list.hpp>\n#include <sys/mman.h>\n#include <sys/syscall.h>\n#include <linux/mempolicy.h>\n\n#endif // !defined(SEASTAR_DEFAULT_ALLOCATOR)\n\n#include <seastar/core/cacheline.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/alloc_failure_injector.hh>\n#include <seastar/util/memory_diagnostics.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/sampler.hh>\n#include <seastar/util/log.hh>\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/align.hh>\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n#include <seastar/core/bitops.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/util/backtrace.hh>\n#endif\n\n#ifdef SEASTAR_DEBUG\n#define dassert(expr) SEASTAR_ASSERT(expr)\n#else\n#define dassert(expr) do {} while(false)\n#endif\n\nnamespace seastar {\n\nextern seastar::logger seastar_logger;\n\nvoid* internal::allocate_aligned_buffer_impl(size_t size, size_t align) {\n    void *ret;\n    auto r = posix_memalign(&ret, align, size);\n    if (r == ENOMEM) {\n        throw std::bad_alloc();\n    } else if (r == EINVAL) {\n        throw std::runtime_error(format(\"Invalid alignment of {:d}; allocating {:d} bytes\", align, size));\n    } else {\n        SEASTAR_ASSERT(r == 0);\n        return ret;\n    }\n}\n\nnamespace memory {\n\n// We always create the logger object for memory disagnostics, even in\n// in SEASTAR_DEFAULT_ALLOCATOR builds, though it only logs when the\n// seastar allocator is enabled.\nseastar::logger seastar_memory_logger(\"seastar_memory\");\n\nnamespace internal {\n\nthread_local constinit int abort_on_alloc_failure_suppressed = 0;\n\n}\n\nstatic std::pmr::polymorphic_allocator<char> static_malloc_allocator{std::pmr::get_default_resource()};;\nstd::pmr::polymorphic_allocator<char>* malloc_allocator{&static_malloc_allocator};\n\nnamespace internal {\n\n#ifdef __cpp_constinit\n#define SEASTAR_CONSTINIT constinit\n#else\n#define SEASTAR_CONSTINIT\n#endif\n\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\n#ifdef __cpp_constinit\nthread_local constinit volatile int critical_alloc_section = 0;\n#else\n__thread volatile int critical_alloc_section = 0;\n#endif\n\n#endif  // SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\nnuma_layout\nmerge(numa_layout one, numa_layout two) {\n    // There's no chance to merge, so just concatenate\n    one.ranges.insert(one.ranges.end(), two.ranges.begin(), two.ranges.end());\n    return one;\n}\n\n} // namespace internal\n\n}\n\n}\n\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n\nnamespace seastar::memory {\nstruct human_readable_value;\n}\ntemplate <> struct fmt::formatter<struct seastar::memory::human_readable_value> : fmt::ostream_formatter {};\n\nnamespace seastar {\n\nusing allocation_site_ptr = const memory::allocation_site*;\n\nnamespace memory {\n\n[[gnu::unused]]\nstatic allocation_site_ptr get_allocation_site();\n\n[[gnu::noinline]]\nstatic void on_allocation_failure(size_t size);\n\nstatic constexpr unsigned bits_for_cpu_id_and_memory = 44; // 3 reserved for memory area prefix, 47 total address space\n\nstatic constexpr unsigned cpu_id_shift_initial = bits_for_cpu_id_and_memory - 8;\nstatic constinit unsigned cpu_id_shift = cpu_id_shift_initial; // Will be adjusted later after we know how many cpus we have\nstatic constexpr unsigned max_cpus = 4096;\nstatic constinit uintptr_t cpu_id_and_mem_base_mask = ~((uintptr_t(1) << cpu_id_shift_initial) - 1);\n\nusing pageidx = uint32_t;\n\nstruct page;\nclass page_list;\n\nstatic std::atomic<bool> live_cpus[max_cpus];\n\nusing std::optional;\n\n// is_reactor_thread gets set to true when memory::configure() gets called\n// it is used to identify seastar threads and hence use system memory allocator\n// for those threads\nstatic thread_local bool is_reactor_thread = false;\n\n// We default transparent hugepages to true since we prefer to transiently\n// use a transparent hugepage and then break it, to having the kernel\n// work to rearrange a broken transparent hugepage.\nstd::atomic<bool> use_transparent_hugepages = true;\n\nnamespace alloc_stats {\n\nenum class types { allocs, frees, cross_cpu_frees, total_bytes_allocated, reclaims, large_allocs, failed_allocs,\n    foreign_mallocs, foreign_frees, foreign_cross_frees, enum_size };\n\nusing stats_array = std::array<uint64_t, static_cast<std::size_t>(types::enum_size)>;\nusing stats_atomic_array = std::array<std::atomic_uint64_t, static_cast<std::size_t>(types::enum_size)>;\n\nstatic thread_local SEASTAR_CONSTINIT stats_array stats{};\nstd::array<stats_atomic_array, max_cpus> alien_stats{};\n\nstatic void increment_local(types stat_type, uint64_t size = 1) {\n    stats[static_cast<std::size_t>(stat_type)] += size;\n}\n\nstatic void increment(types stat_type, uint64_t size=1)\n{\n    // fast path, reactor threads takes thread local statistics\n    if (is_reactor_thread) {\n        increment_local(stat_type, size);\n    } else {\n        auto hash = std::hash<std::thread::id>()(std::this_thread::get_id());\n        auto i = static_cast<std::size_t>(stat_type);\n        alien_stats[hash % alien_stats.size()][i].fetch_add(size, std::memory_order_relaxed);\n    }\n}\n\nstatic uint64_t get(types stat_type)\n{\n    auto i = static_cast<std::size_t>(stat_type);\n    // fast path, reactor threads takes thread local statistics\n    if (is_reactor_thread) {\n        return stats[i];\n    } else {\n        auto hash = std::hash<std::thread::id>()(std::this_thread::get_id());\n        return alien_stats[hash % alien_stats.size()][i].load();\n    }\n}\n\n}\n\n// original memory allocator support\n// note: allocations before calling the constructor would use seastar allocator\nusing malloc_func_type = void * (*)(size_t);\nusing free_func_type = void * (*)(void *);\nusing realloc_func_type = void * (*)(void *, size_t);\nusing aligned_alloc_type = void * (*)(size_t alignment, size_t size);\nusing malloc_trim_type = int (*)(size_t);\nusing malloc_usable_size_type = size_t (*)(void *);\n\nmalloc_func_type original_malloc_func = reinterpret_cast<malloc_func_type>(dlsym(RTLD_NEXT, \"malloc\"));\nfree_func_type original_free_func = reinterpret_cast<free_func_type>(dlsym(RTLD_NEXT, \"free\"));\nrealloc_func_type original_realloc_func = reinterpret_cast<realloc_func_type>(dlsym(RTLD_NEXT, \"realloc\"));\naligned_alloc_type original_aligned_alloc_func = reinterpret_cast<aligned_alloc_type>(dlsym(RTLD_NEXT, \"aligned_alloc\"));\nmalloc_trim_type original_malloc_trim_func = reinterpret_cast<malloc_trim_type>(dlsym(RTLD_NEXT, \"malloc_trim\"));\nmalloc_usable_size_type original_malloc_usable_size_func = reinterpret_cast<malloc_usable_size_type>(dlsym(RTLD_NEXT, \"malloc_usable_size\"));\n\nusing allocate_system_memory_fn\n        = std::function<mmap_area (void* where, size_t how_much)>;\n\nnamespace bi = boost::intrusive;\n\nstatic thread_local uintptr_t local_expected_cpu_id = std::numeric_limits<uintptr_t>::max();\n\n\ninline\nunsigned object_cpu_id(const void* ptr) {\n    auto uptr = reinterpret_cast<uintptr_t>(ptr);\n    auto mask = (size_t(1) << bits_for_cpu_id_and_memory) - 1;\n    return (uptr & mask) >> cpu_id_shift;\n}\n\nclass page_list_link {\n    uint32_t _prev;\n    uint32_t _next;\n    friend class page_list;\n    friend seastar::internal::log_buf::inserter_iterator do_dump_memory_diagnostics(seastar::internal::log_buf::inserter_iterator);\n};\n\nconstexpr size_t mem_base_alloc = size_t(1) << bits_for_cpu_id_and_memory;\n\nstatic char* mem_base() {\n    static char* known;\n    static std::once_flag flag;\n    std::call_once(flag, [] {\n        auto r = ::mmap(NULL, 2 * mem_base_alloc,\n                    PROT_NONE,\n                    MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,\n                    -1, 0);\n        if (r == MAP_FAILED) {\n            abort();\n        }\n        ::madvise(r, 2 * mem_base_alloc, MADV_DONTDUMP);\n        auto cr = reinterpret_cast<char*>(r);\n        known = align_up(cr, mem_base_alloc);\n        ::munmap(cr, known - cr);\n        ::munmap(known + mem_base_alloc, cr + 2 * mem_base_alloc - (known + mem_base_alloc));\n        // extremely unlikely for mmap to return a mapping at 0, but our detection of free(null)\n        // depends on it not doing that so check it\n        SEASTAR_ASSERT(known != nullptr);\n        SEASTAR_ASSERT(reinterpret_cast<uintptr_t>(known) != 0);\n    });\n    return known;\n}\n\nbool is_seastar_memory(void * ptr)\n{\n    auto begin = mem_base();\n    auto end = begin + mem_base_alloc;\n    return ptr >= begin && ptr < end;\n}\n\nconstexpr bool is_page_aligned(size_t size) {\n    return (size & (page_size - 1)) == 0;\n}\n\nconstexpr size_t next_page_aligned(size_t size) {\n    return (size + (page_size - 1)) & ~(page_size - 1);\n}\n\nclass small_pool;\n\nstruct free_object {\n    free_object* next;\n};\n\nstruct page {\n    bool free;\n    uint8_t offset_in_span;\n    uint16_t nr_small_alloc;\n    uint32_t span_size; // in pages, if we're the head or the tail\n    page_list_link link;\n    small_pool* pool;  // if used in a small_pool\n    free_object* freelist;\n#ifdef SEASTAR_HEAPPROF\n    allocation_site_ptr alloc_site; // for objects whose size is multiple of page size, valid for head only\n#endif\n};\n\nclass page_list {\n    uint32_t _front = 0;\n    uint32_t _back = 0;\npublic:\n    page& front(page* ary) { return ary[_front]; }\n    page& back(page* ary) { return ary[_back]; }\n    bool empty() const { return !_front; }\n    void erase(page* ary, page& span) {\n        if (span.link._next) {\n            ary[span.link._next].link._prev = span.link._prev;\n        } else {\n            _back = span.link._prev;\n        }\n        if (span.link._prev) {\n            ary[span.link._prev].link._next = span.link._next;\n        } else {\n            _front = span.link._next;\n        }\n    }\n    void push_front(page* ary, page& span) {\n        auto idx = &span - ary;\n        if (_front) {\n            ary[_front].link._prev = idx;\n        } else {\n            _back = idx;\n        }\n        span.link._next = _front;\n        span.link._prev = 0;\n        _front = idx;\n    }\n    void pop_front(page* ary) {\n        if (ary[_front].link._next) {\n            ary[ary[_front].link._next].link._prev = 0;\n        } else {\n            _back = 0;\n        }\n        _front = ary[_front].link._next;\n    }\n    friend seastar::internal::log_buf::inserter_iterator do_dump_memory_diagnostics(seastar::internal::log_buf::inserter_iterator);\n};\n\nclass small_pool {\n    struct span_sizes {\n        uint8_t preferred;\n        uint8_t fallback;\n    };\n    free_object* _free = nullptr;\n    unsigned _object_size;\n    span_sizes _span_sizes;\n    unsigned _free_count = 0;\n    unsigned _min_free;\n    unsigned _max_free;\n    unsigned _pages_in_use = 0;\n    // Flag to indicate whether this pool stores sampled allocations.\n    // When freeing small allocations this flag is checked to see whether an\n    // allocation site pointer is part of the object and the allocation needs\n    // removal from the allocation_site tracking\n#ifdef SEASTAR_HEAPPROF\n    bool _sampled_pool = false;\n#endif\n    page_list _span_list;\n    static constexpr unsigned idx_frac_bits = 2;\npublic:\n    explicit small_pool(unsigned object_size, bool is_sampled) noexcept;\n    ~small_pool();\n    inline void* allocate();\n    void deallocate(void* object);\n    unsigned object_size() const { return _object_size; }\n    /// See _sampled_pool\n    bool is_sampled_pool() const {\n#ifdef SEASTAR_HEAPPROF\n        return _sampled_pool;\n#else\n        return false;\n#endif\n    }\n    bool objects_page_aligned() const { return is_page_aligned(_object_size); }\n    static constexpr unsigned size_to_idx(unsigned size);\n    static constexpr unsigned idx_to_size(unsigned idx);\n    allocation_site_ptr& alloc_site_holder(void* ptr);\nprivate:\n    inline void* pop_free();\n    [[gnu::noinline]] void* add_more_objects();\n    void trim_free_list();\n    friend seastar::internal::log_buf::inserter_iterator do_dump_memory_diagnostics(seastar::internal::log_buf::inserter_iterator);\n};\n\n// index 0b0001'1100 -> size (1 << 4) + 0b11 << (4 - 2)\n\nconstexpr unsigned\nsmall_pool::idx_to_size(unsigned idx) {\n    size_t s = (((1 << idx_frac_bits) | (idx & ((1 << idx_frac_bits) - 1)))\n              << (idx >> idx_frac_bits))\n                  >> idx_frac_bits;\n    // If size is larger than max_align_t, force it to be a multiple of\n    // max_align_t. Clang relies in this property to use aligned mov\n    // instructions (e.g. movaps)\n    //\n    // Note this function is used at initialization time only, so it doesn't\n    // need to be especially fast.\n    if (s > alignof(std::max_align_t)) {\n\ts = align_up(s, alignof(std::max_align_t));\n    }\n    return s;\n}\n\nconstexpr unsigned\nsmall_pool::size_to_idx(unsigned size) {\n    return ((log2floor(size) << idx_frac_bits) - ((1 << idx_frac_bits) - 1))\n            + ((size - 1) >> (log2floor(size) - idx_frac_bits));\n}\n\ntemplate<bool sampled> // tag the pools in this array as sampled, see small_pool._sampled_pool\nclass small_pool_array {\npublic:\n    static constexpr unsigned nr_small_pools = small_pool::size_to_idx(4 * page_size) + 1;\nprivate:\n    union u {\n        small_pool a[nr_small_pools];\n        u() {\n            for (unsigned i = 0; i < nr_small_pools; ++i) {\n                new (&a[i]) small_pool(small_pool::idx_to_size(i), sampled);\n            }\n        }\n        ~u() {\n            // cannot really call destructor, since other\n            // objects may be freed after we are gone.\n        }\n    } _u;\npublic:\n    small_pool& operator[](unsigned idx) { return _u.a[idx]; }\n};\n\nstatic constexpr size_t max_small_allocation\n    = small_pool::idx_to_size(small_pool_array<false>::nr_small_pools - 1);\n\n#ifdef SEASTAR_HEAPPROF\nconstexpr size_t object_size_with_alloc_site(size_t size) {\n    // For page-aligned sizes, allocation_site* lives in page::alloc_site, not with the object.\n    static_assert(is_page_aligned(max_small_allocation), \"assuming that max_small_allocation is page aligned so that we\"\n            \" don't need to add allocation_site_ptr to objects of size close to it\");\n    size_t next_page_aligned_size = next_page_aligned(size);\n    if (next_page_aligned_size - size > sizeof(allocation_site_ptr)) {\n        size += sizeof(allocation_site_ptr);\n    } else {\n        return next_page_aligned_size;\n    }\n    return size;\n}\n\n// Ensure that object_size_with_alloc_site() does not exceed max_small_allocation\nstatic_assert(object_size_with_alloc_site(max_small_allocation) == max_small_allocation, \"\");\nstatic_assert(object_size_with_alloc_site(max_small_allocation - 1) == max_small_allocation, \"\");\nstatic_assert(object_size_with_alloc_site(max_small_allocation - sizeof(allocation_site_ptr) + 1) == max_small_allocation, \"\");\nstatic_assert(object_size_with_alloc_site(max_small_allocation - sizeof(allocation_site_ptr)) == max_small_allocation, \"\");\nstatic_assert(object_size_with_alloc_site(max_small_allocation - sizeof(allocation_site_ptr) - 1) == max_small_allocation - 1, \"\");\nstatic_assert(object_size_with_alloc_site(max_small_allocation - sizeof(allocation_site_ptr) - 2) == max_small_allocation - 2, \"\");\n#endif\n\nstruct cross_cpu_free_item {\n    cross_cpu_free_item* next;\n};\n\nstruct cpu_pages {\n    small_pool_array<false> small_pools;\n    uint32_t min_free_pages = 20000000 / page_size;\n    char* memory;\n    page* pages;\n    uint32_t nr_pages;\n    uint32_t nr_free_pages;\n    uint32_t current_min_free_pages = 0;\n    struct {\n        size_t warn = std::numeric_limits<size_t>::max();\n        size_t check = std::numeric_limits<size_t>::max();\n        unsigned descend_attempt = 0;\n\n        void set(size_t value) noexcept {\n            warn = value;\n            check = value;\n            descend_attempt = 0;\n        }\n\n        void ascend() noexcept {\n            warn *= 1.618;\n            descend_attempt = 0;\n        }\n\n        void maybe_descend() noexcept {\n            // Large allocation rate per second is much smaller than one\n            // So in the worst case the warn threshold should be descended after 20+ seconds\n            if (++descend_attempt >= 20) {\n                if (warn > check + page_size) {\n                    warn -= page_size;\n                } else {\n                    warn = check;\n                }\n                descend_attempt = 0;\n            }\n        }\n    } large_allocation_warning_threshold = {};\n    unsigned cpu_id = -1U;\n    std::function<void (std::function<void ()>)> reclaim_hook;\n    std::vector<reclaimer*> reclaimers;\n    static constexpr unsigned nr_span_lists = 32;\n    page_list free_spans[nr_span_lists];  // contains aligned spans with span_size == 2^idx\n    alignas(seastar::cache_line_size) std::atomic<cross_cpu_free_item*> xcpu_freelist;\n    static std::atomic<unsigned> cpu_id_gen;\n    static cpu_pages* all_cpus[max_cpus];\n    union asu {\n        using alloc_sites_type = std::unordered_set<allocation_site>;\n        asu() : alloc_sites{} {\n        }\n        ~asu() {} // alloc_sites live forever\n        alloc_sites_type alloc_sites;\n    } asu;\n    allocation_site_ptr alloc_site_list_head = nullptr; // For easy traversal of asu.alloc_sites from scylla-gdb.py\n    sampler heap_prof_sampler;\n    small_pool_array<true> sampled_small_pools;\n\n    char* mem() { return memory; }\n\n    void link(page_list& list, page* span);\n    void unlink(page_list& list, page* span);\n    struct trim {\n        unsigned offset;\n        unsigned nr_pages;\n    };\n    void maybe_reclaim();\n    void* allocate_large_and_trim(unsigned nr_pages, bool should_sample);\n    void* allocate_large(unsigned nr_pages, bool should_sample);\n    void* allocate_large_aligned(unsigned align_pages, unsigned nr_pages, bool should_sample);\n    page* find_and_unlink_span(unsigned nr_pages);\n    page* find_and_unlink_span_reclaiming(unsigned n_pages);\n    void free_large(void* ptr);\n    bool grow_span(pageidx& start, uint32_t& nr_pages, unsigned idx);\n    void free_span(pageidx start, uint32_t nr_pages);\n    void free_span_no_merge(pageidx start, uint32_t nr_pages);\n    void free_span_unaligned(pageidx start, uint32_t nr_pages);\n    void free(void* ptr);\n    void free(void* ptr, size_t size);\n    static bool try_free_fastpath(void* ptr);\n    static bool is_local_pointer(void* ptr);\n    static void do_foreign_free(void* ptr);\n    void shrink(void* ptr, size_t new_size);\n    static void free_cross_cpu(unsigned cpu_id, void* ptr);\n    bool drain_cross_cpu_freelist();\n    size_t object_size(void* ptr);\n\n    page* to_page(void* p) {\n        size_t page_idx = ((uintptr_t)p) << (64 - cpu_id_shift) >> (64 - cpu_id_shift + page_bits);\n        return &pages[page_idx];\n    }\n\n    bool is_initialized() const;\n    bool initialize();\n    reclaiming_result run_reclaimers(reclaimer_scope, size_t pages_to_reclaim);\n    void schedule_reclaim();\n    void set_reclaim_hook(std::function<void (std::function<void ()>)> hook);\n    void set_min_free_pages(size_t pages);\n    void resize(size_t new_size, allocate_system_memory_fn alloc_sys_mem);\n    void do_resize(size_t new_size, allocate_system_memory_fn alloc_sys_mem);\n    void replace_memory_backing(allocate_system_memory_fn alloc_sys_mem);\n    void check_large_allocation(size_t size);\n    void warn_large_allocation(size_t size);\n    allocation_site_ptr add_alloc_site(size_t allocated_size);\n    void remove_alloc_site(allocation_site_ptr alloc_site, size_t deallocated_size);\n    bool maybe_sample(size_t size);\n    bool definitely_sample(size_t size);\n    memory::memory_layout memory_layout();\n    ~cpu_pages();\n};\n\nstatic thread_local cpu_pages cpu_mem;\nstd::atomic<unsigned> cpu_pages::cpu_id_gen;\ncpu_pages* cpu_pages::all_cpus[max_cpus];\n\nstatic cpu_pages& get_cpu_mem();\n\n#ifdef SEASTAR_HEAPPROF\n\nvoid set_heap_profiling_sampling_rate(size_t sample_rate) {\n    bool current_sample_rate = get_cpu_mem().heap_prof_sampler.sampling_interval();\n    if (sample_rate) {\n        if (!current_sample_rate) {\n            seastar_logger.info(\"Enabling heap profiler - using {} bytes sampling rate\", sample_rate);\n        } else {\n            seastar_logger.warn(\"Ignoring change to heap profiler sample rate as heap profiling is already turned on\");\n            return;\n        }\n    } else {\n        if (current_sample_rate) {\n            seastar_logger.info(\"Disabling heap profiler\");\n        }\n    }\n    get_cpu_mem().heap_prof_sampler.set_sampling_interval(sample_rate);\n}\n\nsize_t get_heap_profiling_sample_rate() {\n    return get_cpu_mem().heap_prof_sampler.sampling_interval();\n}\n\nstatic thread_local int64_t scoped_heap_profiling_embed_count = 0;\n\nscoped_heap_profiling::scoped_heap_profiling(size_t sample_rate) noexcept {\n    ++scoped_heap_profiling_embed_count;\n    set_heap_profiling_sampling_rate(sample_rate);\n}\n\nscoped_heap_profiling::~scoped_heap_profiling() {\n    if (!--scoped_heap_profiling_embed_count) {\n        set_heap_profiling_sampling_rate(0);\n    }\n}\n\n#else\n\nvoid set_heap_profiling_sampling_rate(size_t enable) {\n    seastar_logger.warn(\"Seastar compiled without heap profiling support, heap profiler not supported;\"\n            \" compile with the Seastar_HEAP_PROFILING=ON CMake option to add heap profiling support\");\n}\n\nsize_t get_heap_profiling_sample_rate() {\n    // don't log here, called on all paths\n    return 0;\n}\n\nscoped_heap_profiling::scoped_heap_profiling(size_t sample_rate) noexcept {\n    set_heap_profiling_sampling_rate(sample_rate); // let it print the warning\n}\n\nscoped_heap_profiling::~scoped_heap_profiling() {\n}\n\n#endif\n\n// Smallest index i such that all spans stored in the index are >= pages.\nstatic inline\nunsigned index_of(unsigned pages) {\n    if (pages == 1) {\n        return 0;\n    }\n    return std::numeric_limits<unsigned>::digits - count_leading_zeros(pages - 1);\n}\n\nvoid\ncpu_pages::unlink(page_list& list, page* span) {\n    list.erase(pages, *span);\n}\n\nvoid\ncpu_pages::link(page_list& list, page* span) {\n    list.push_front(pages, *span);\n}\n\nvoid cpu_pages::free_span_no_merge(uint32_t span_start, uint32_t nr_pages) {\n    SEASTAR_ASSERT(nr_pages);\n    nr_free_pages += nr_pages;\n    auto span = &pages[span_start];\n    auto span_end = &pages[span_start + nr_pages - 1];\n    span->free = span_end->free = true;\n    span->span_size = span_end->span_size = nr_pages;\n    auto idx = index_of(nr_pages);\n    link(free_spans[idx], span);\n}\n\nbool cpu_pages::grow_span(uint32_t& span_start, uint32_t& nr_pages, unsigned idx) {\n    auto which = (span_start >> idx) & 1; // 0=lower, 1=upper\n    // locate first page of upper buddy or last page of lower buddy\n    // examples: span_start = 0x10 nr_pages = 0x08 -> buddy = 0x18  (which = 0)\n    //           span_start = 0x18 nr_pages = 0x08 -> buddy = 0x17  (which = 1)\n    // delta = which ? -1u : nr_pages\n    auto delta = ((which ^ 1) << idx) | -which;\n    auto buddy = span_start + delta;\n    if (pages[buddy].free && pages[buddy].span_size == nr_pages) {\n        unlink(free_spans[idx], &pages[span_start ^ nr_pages]);\n        nr_free_pages -= nr_pages; // free_span_no_merge() will restore\n        span_start &= ~nr_pages;\n        nr_pages *= 2;\n        return true;\n    }\n    return false;\n}\n\nvoid cpu_pages::free_span(uint32_t span_start, uint32_t nr_pages) {\n    auto idx = index_of(nr_pages);\n    while (grow_span(span_start, nr_pages, idx)) {\n        ++idx;\n    }\n    free_span_no_merge(span_start, nr_pages);\n}\n\n// Internal, used during startup. Span is not aligned so needs to be broken up\nvoid cpu_pages::free_span_unaligned(uint32_t span_start, uint32_t nr_pages) {\n    while (nr_pages) {\n        auto start_nr_bits = span_start ? count_trailing_zeros(span_start) : 32;\n        auto size_nr_bits = count_trailing_zeros(nr_pages);\n        auto now = 1u << std::min(start_nr_bits, size_nr_bits);\n        free_span(span_start, now);\n        span_start += now;\n        nr_pages -= now;\n    }\n}\n\npage*\ncpu_pages::find_and_unlink_span(unsigned n_pages) {\n    auto idx = index_of(n_pages);\n    if (n_pages >= (2u << idx)) {\n        return nullptr;\n    }\n    while (idx < nr_span_lists && free_spans[idx].empty()) {\n        ++idx;\n    }\n    if (idx == nr_span_lists) {\n        if (initialize()) {\n            return find_and_unlink_span(n_pages);\n        }\n        return nullptr;\n    }\n    auto& list = free_spans[idx];\n    page* span = &list.front(pages);\n    unlink(list, span);\n    return span;\n}\n\npage*\ncpu_pages::find_and_unlink_span_reclaiming(unsigned n_pages) {\n    while (true) {\n        auto span = find_and_unlink_span(n_pages);\n        if (span) {\n            return span;\n        }\n        if (run_reclaimers(reclaimer_scope::sync, n_pages) == reclaiming_result::reclaimed_nothing) {\n            return nullptr;\n        }\n    }\n}\n\nvoid cpu_pages::maybe_reclaim() {\n    if (nr_free_pages < current_min_free_pages) {\n        drain_cross_cpu_freelist();\n        if (nr_free_pages < current_min_free_pages) {\n            run_reclaimers(reclaimer_scope::sync, current_min_free_pages - nr_free_pages);\n        }\n        if (nr_free_pages < current_min_free_pages) {\n            schedule_reclaim();\n        }\n    }\n}\n\nvoid*\ncpu_pages::allocate_large_and_trim(unsigned n_pages, bool should_sample) {\n    // Avoid exercising the reclaimers for requests we'll not be able to satisfy\n    // nr_pages might be zero during startup, so check for that too\n    if (nr_pages && n_pages >= nr_pages) {\n        return nullptr;\n    }\n    page* span = find_and_unlink_span_reclaiming(n_pages);\n    if (!span) {\n        return nullptr;\n    }\n    auto span_size = span->span_size;\n    auto span_idx = span - pages;\n    nr_free_pages -= span->span_size;\n    while (span_size >= n_pages * 2) {\n        span_size /= 2;\n        auto other_span_idx = span_idx + span_size;\n        free_span_no_merge(other_span_idx, span_size);\n    }\n    auto span_end = &pages[span_idx + span_size - 1];\n    span->free = span_end->free = false;\n    span->span_size = span_end->span_size = span_size;\n    span->pool = nullptr;\n#ifdef SEASTAR_HEAPPROF\n    if (should_sample) {\n        auto alloc_site = add_alloc_site(span->span_size * page_size);\n        span->alloc_site = alloc_site;\n    }\n    else {\n        span->alloc_site = nullptr;\n    }\n#endif\n    maybe_reclaim();\n    return mem() + span_idx * page_size;\n}\n\nvoid\ncpu_pages::warn_large_allocation(size_t size) {\n    alloc_stats::increment_local(alloc_stats::types::large_allocs);\n    seastar_memory_logger.warn(\"oversized allocation: {} bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at {}\", size, current_backtrace());\n}\n\nallocation_site_ptr\ncpu_pages::add_alloc_site(size_t allocated_size) {\n    allocation_site_ptr alloc_site = get_allocation_site();\n    if (alloc_site) {\n        ++alloc_site->count;\n        alloc_site->size += heap_prof_sampler.sample_size(allocated_size);\n    }\n\n    return alloc_site;\n}\n\nvoid\ncpu_pages::remove_alloc_site(allocation_site_ptr alloc_site, size_t deallocated_size) {\n    if (alloc_site) {\n        --alloc_site->count;\n        auto sample_size = heap_prof_sampler.sample_size(deallocated_size);\n        // prevent underflow in case sample rate changed\n        alloc_site->size -= alloc_site->size < sample_size ? alloc_site->size : sample_size;\n        if (alloc_site->count == 0) {\n            if (alloc_site->prev) {\n                alloc_site->prev->next = alloc_site->next;\n            }\n            if (alloc_site->next) {\n                alloc_site->next->prev = alloc_site->prev;\n            }\n            if (alloc_site_list_head == alloc_site) {\n                alloc_site_list_head = alloc_site->next;\n            }\n\n            asu.alloc_sites.erase(*alloc_site);\n        }\n    }\n}\n\n[[gnu::always_inline]]\ninline bool\ncpu_pages::maybe_sample(size_t size) {\n#ifdef SEASTAR_HEAPPROF\n    return heap_prof_sampler.maybe_sample(size);\n#else\n    return false;\n#endif\n}\n\n[[gnu::always_inline]]\ninline bool\ncpu_pages::definitely_sample(size_t size) {\n#ifdef SEASTAR_HEAPPROF\n    return heap_prof_sampler.definitely_sample(size);\n#else\n    return false;\n#endif\n}\n\n[[gnu::always_inline]]\nvoid\ninline\ncpu_pages::check_large_allocation(size_t size) {\n    if (size >= large_allocation_warning_threshold.check) [[unlikely]] {\n        if (size >= large_allocation_warning_threshold.warn) {\n            warn_large_allocation(size);\n            large_allocation_warning_threshold.ascend(); // prevent spam;\n        } else {\n            large_allocation_warning_threshold.maybe_descend(); // possibly re-encourage spam;\n        }\n    }\n}\n\n[[gnu::always_inline]]\ninline void*\ncpu_pages::allocate_large(unsigned n_pages, bool should_sample) {\n    check_large_allocation(n_pages * page_size);\n    return allocate_large_and_trim(n_pages, should_sample);\n}\n\nvoid*\ncpu_pages::allocate_large_aligned(unsigned align_pages, unsigned n_pages, bool should_sample) {\n    check_large_allocation(n_pages * page_size);\n    // buddy allocation is always aligned\n    return allocate_large_and_trim(n_pages, should_sample);\n}\n\ndisable_backtrace_temporarily::disable_backtrace_temporarily()\n    : _disable_sampling(cpu_mem.heap_prof_sampler.pause_sampling()) {\n}\n\nstatic\nsimple_backtrace get_backtrace() noexcept {\n    disable_backtrace_temporarily dbt;\n    return current_backtrace_tasklocal();\n}\n\nstatic\nallocation_site_ptr get_allocation_site() {\n    if (!cpu_mem.is_initialized() || !cpu_mem.heap_prof_sampler.sampling_interval()) {\n        return nullptr;\n    }\n    disable_backtrace_temporarily dbt;\n    allocation_site new_alloc_site;\n    new_alloc_site.backtrace = get_backtrace();\n    if (cpu_mem.asu.alloc_sites.size() >= 1000\n        && cpu_mem.asu.alloc_sites.find(new_alloc_site) == cpu_mem.asu.alloc_sites.end()) {\n        // Drop sample for now. Could do something smarter like dropping a\n        // current one at random but needs more work in remove_alloc_site as we\n        // might then have allocations for which the allocsite is no longer\n        // alive\n        return nullptr;\n    }\n    auto insert_result = cpu_mem.asu.alloc_sites.insert(std::move(new_alloc_site));\n    allocation_site_ptr alloc_site = &*insert_result.first;\n    if (insert_result.second) {\n        alloc_site->next = cpu_mem.alloc_site_list_head;\n        if (cpu_mem.alloc_site_list_head) {\n            cpu_mem.alloc_site_list_head->prev = alloc_site;\n        }\n        cpu_mem.alloc_site_list_head = alloc_site;\n    }\n    return alloc_site;\n}\n\n#ifdef SEASTAR_HEAPPROF\n\nallocation_site_ptr&\nsmall_pool::alloc_site_holder(void* ptr) {\n    if (objects_page_aligned()) {\n        return get_cpu_mem().to_page(ptr)->alloc_site;\n    } else {\n        return *reinterpret_cast<allocation_site_ptr*>(reinterpret_cast<char*>(ptr) + _object_size - sizeof(allocation_site_ptr));\n    }\n}\n\n#endif\n\nstatic\nvoid\nmaybe_enable_transparent_hugepages(void* addr, size_t len) {\n    if (use_transparent_hugepages.load(std::memory_order_relaxed)) {\n        ::madvise(addr, len, MADV_HUGEPAGE);\n    }\n}\n\nstatic\nvoid\nmaybe_disable_transparent_hugepages(void* addr, size_t len) {\n    if (!use_transparent_hugepages.load(std::memory_order_relaxed)) {\n        ::madvise(addr, len, MADV_NOHUGEPAGE);\n    }\n}\n\nvoid cpu_pages::free_large(void* ptr) {\n    pageidx idx = (reinterpret_cast<char*>(ptr) - mem()) / page_size;\n    page* span = &pages[idx];\n#ifdef SEASTAR_HEAPPROF\n    if (span->alloc_site) {\n        auto alloc_site = span->alloc_site;\n        remove_alloc_site(alloc_site, span->span_size * page_size);\n    }\n#endif\n    free_span(idx, span->span_size);\n}\n\nsize_t cpu_pages::object_size(void* ptr) {\n    page* span = to_page(ptr);\n    if (span->pool) {\n        auto s = span->pool->object_size();\n#ifdef SEASTAR_HEAPPROF\n        if (span->pool->is_sampled_pool()) {\n            // We must not allow the object to be extended onto the allocation_site_ptr field.\n            if (!span->pool->objects_page_aligned()) {\n                s -= sizeof(allocation_site_ptr);\n            }\n        }\n#endif\n        return s;\n    } else {\n        return size_t(span->span_size) * page_size;\n    }\n}\n\nvoid cpu_pages::free_cross_cpu(unsigned cpu_id, void* ptr) {\n    if (!live_cpus[cpu_id].load(std::memory_order_relaxed)) {\n        // Thread was destroyed; leak object\n        // should only happen for boost unit-tests.\n        return;\n    }\n    auto p = reinterpret_cast<cross_cpu_free_item*>(ptr);\n    auto& list = all_cpus[cpu_id]->xcpu_freelist;\n    auto old = list.load(std::memory_order_relaxed);\n    do {\n        p->next = old;\n    } while (!list.compare_exchange_weak(old, p, std::memory_order_release, std::memory_order_relaxed));\n    alloc_stats::increment(alloc_stats::types::cross_cpu_frees);\n}\n\nbool cpu_pages::drain_cross_cpu_freelist() {\n    if (!xcpu_freelist.load(std::memory_order_relaxed)) {\n        return false;\n    }\n    auto p = xcpu_freelist.exchange(nullptr, std::memory_order_acquire);\n    while (p) {\n        auto n = p->next;\n        alloc_stats::increment_local(alloc_stats::types::frees);\n        free(p);\n        p = n;\n    }\n    return true;\n}\n\n[[gnu::always_inline]]\ninline void cpu_pages::free(void* ptr) {\n    page* span = to_page(ptr);\n    if (span->pool) {\n        small_pool& pool = *span->pool;\n#ifdef SEASTAR_HEAPPROF\n        if (pool.is_sampled_pool()) {\n            allocation_site_ptr alloc_site = pool.alloc_site_holder(ptr);\n            remove_alloc_site(alloc_site, pool.object_size());\n        }\n#endif\n        pool.deallocate(ptr);\n    } else {\n        free_large(ptr);\n    }\n}\n\nvoid cpu_pages::free(void* ptr, size_t size) {\n#ifdef SEASTAR_HEAPPROF\n    // sized free can avoid accessing the `page` structure as an optimization.\n    // With memory sampling we always need to check the pool though to see\n    // whether this allocation was sampled. Hence, just defer to the non-sized\n    // implementation\n    (void) size;\n    free(ptr);\n#else\n    // match action on allocate() so hit the right pool\n    if (size <= sizeof(free_object)) {\n        size = sizeof(free_object);\n    }\n    if (size <= max_small_allocation) {\n        auto pool = &small_pools[small_pool::size_to_idx(size)];\n        pool->deallocate(ptr);\n    } else {\n        free_large(ptr);\n    }\n#endif\n}\n\n// Is the passed pointer a local pointer, i.e., allocated on the current shard from the\n// per-shard allocator.\n[[gnu::always_inline]]\ninline bool\ncpu_pages::is_local_pointer(void* ptr) {\n    return (reinterpret_cast<uintptr_t>(ptr) & cpu_id_and_mem_base_mask) == local_expected_cpu_id;\n}\n\n// Try to execute free on the fast path, which succeeds if:\n//\n// 1) The pointer is local to this shard\n// 2) The pointer is from a small pool\n// 3) The small pool is not sampled\n//\n// In this case, complete the de-allocation and return true.\n// Otherwise, modify nothing and return false.\n[[gnu::always_inline]]\ninline bool\ncpu_pages::try_free_fastpath(void* ptr) {\n    if (__builtin_expect(is_local_pointer(ptr), true)) {\n        auto pool = get_cpu_mem().to_page(ptr)->pool;\n        if (__builtin_expect(pool && !pool->is_sampled_pool(), true)) {\n            alloc_stats::increment_local(alloc_stats::types::frees);\n            pool->deallocate(ptr);\n            return true;\n        }\n    }\n    return false;\n}\n\n/// Helper to allow a single implementation for sized and non-sized functions.\n/// Indicator to allow a single implementation for sized and non-sized functions.\n/// The size parameter will be either no_size tag type or size_t, and most\n/// of the implementation can be shared, using constexpr if or other dispatch\n/// in the places where there should be a difference of behavior.\nstruct no_size {};\n\ntemplate <typename S>\nrequires std::same_as<S, size_t> || std::same_as<S, no_size>\n[[gnu::noinline]]\nstatic void free_slowpath(void* obj, S size) {\n    if (cpu_pages::is_local_pointer(obj)) {\n        alloc_stats::increment_local(alloc_stats::types::frees);\n        if constexpr (std::is_same_v<decltype(size), no_size>) {\n            get_cpu_mem().free(obj);\n        } else {\n            get_cpu_mem().free(obj, size);\n        }\n    } else {\n        cpu_pages::do_foreign_free(obj);\n    }\n}\n\n[[gnu::noinline]]\nvoid\ncpu_pages::do_foreign_free(void* ptr) {\n    // handles:\n    // 1) non-seastar pointers\n    // 2) cross-shard frees\n    // 3) null pointer\n\n    if (!ptr) {\n        return;\n    }\n\n    if (!is_seastar_memory(ptr)) {\n        if (is_reactor_thread) {\n            alloc_stats::increment_local(alloc_stats::types::foreign_cross_frees);\n        } else {\n            alloc_stats::increment(alloc_stats::types::foreign_frees);\n        }\n        original_free_func(ptr);\n        return;\n    }\n    free_cross_cpu(object_cpu_id(ptr), ptr);\n}\n\nvoid cpu_pages::shrink(void* ptr, size_t new_size) {\n    SEASTAR_ASSERT(object_cpu_id(ptr) == cpu_id);\n    page* span = to_page(ptr);\n    if (span->pool) {\n        return;\n    }\n    auto old_size_pages = span->span_size;\n    size_t new_size_pages = old_size_pages;\n    while (new_size_pages / 2 * page_size >= new_size) {\n        new_size_pages /= 2;\n    }\n    if (new_size_pages == old_size_pages) {\n        return;\n    }\n#ifdef SEASTAR_HEAPPROF\n    auto alloc_site = span->alloc_site;\n    if (alloc_site) {\n        alloc_site->size -= span->span_size * page_size;\n        alloc_site->size += new_size_pages * page_size;\n    }\n#endif\n    span->span_size = new_size_pages;\n    span[new_size_pages - 1].free = false;\n    span[new_size_pages - 1].span_size = new_size_pages;\n    pageidx idx = span - pages;\n    free_span_unaligned(idx + new_size_pages, old_size_pages - new_size_pages);\n}\n\ncpu_pages::~cpu_pages() {\n    if (is_initialized()) {\n        live_cpus[cpu_id].store(false, std::memory_order_relaxed);\n    }\n}\n\nbool cpu_pages::is_initialized() const {\n    return bool(nr_pages);\n}\n\nbool cpu_pages::initialize() {\n    if (is_initialized()) {\n        return false;\n    }\n    cpu_id = cpu_id_gen.fetch_add(1, std::memory_order_relaxed);\n    local_expected_cpu_id = (static_cast<uint64_t>(cpu_id) << cpu_id_shift)\n\t        | reinterpret_cast<uintptr_t>(mem_base());\n    SEASTAR_ASSERT(cpu_id < max_cpus);\n    all_cpus[cpu_id] = this;\n    auto base = mem_base() + (size_t(cpu_id) << cpu_id_shift);\n    auto size = 32 << 20;  // Small size for bootstrap\n    auto r = ::mmap(base, size,\n            PROT_READ | PROT_WRITE,\n            MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,\n            -1, 0);\n    if (r == MAP_FAILED) {\n        abort();\n    }\n    maybe_enable_transparent_hugepages(base, size);\n    pages = reinterpret_cast<page*>(base);\n    memory = base;\n    nr_pages = size / page_size;\n    // we reserve the end page so we don't have to special case\n    // the last span.\n    auto reserved = align_up(sizeof(page) * (nr_pages + 1), page_size) / page_size;\n    reserved = 1u << log2ceil(reserved);\n    for (pageidx i = 0; i < reserved; ++i) {\n        pages[i].free = false;\n    }\n    pages[nr_pages].free = false;\n    free_span_unaligned(reserved, nr_pages - reserved);\n    live_cpus[cpu_id].store(true, std::memory_order_relaxed);\n    return true;\n}\n\nmmap_area\nstatic allocate_anonymous_memory(void* where, size_t how_much) {\n    return mmap_anonymous(where,\n            how_much,\n            PROT_READ | PROT_WRITE,\n            MAP_PRIVATE | MAP_FIXED);\n}\n\nmmap_area\nallocate_hugetlbfs_memory(file_desc& fd, void* where, size_t how_much) {\n    auto pos = fd.size();\n    fd.truncate(pos + how_much);\n    auto ret = fd.map(\n            how_much,\n            PROT_READ | PROT_WRITE,\n            MAP_SHARED | MAP_POPULATE | (where ? MAP_FIXED : 0),\n            pos,\n            where);\n    return ret;\n}\n\nvoid cpu_pages::replace_memory_backing(allocate_system_memory_fn alloc_sys_mem) {\n    // We would like to use ::mremap() to atomically replace the old anonymous\n    // memory with hugetlbfs backed memory, but mremap() does not support hugetlbfs\n    // (for no reason at all).  So we must copy the anonymous memory to some other\n    // place, map hugetlbfs in place, and copy it back, without modifying it during\n    // the operation.\n    auto bytes = nr_pages * page_size;\n    auto old_mem = mem();\n    auto relocated_old_mem = mmap_anonymous(nullptr, bytes, PROT_READ|PROT_WRITE, MAP_PRIVATE);\n    std::memcpy(relocated_old_mem.get(), old_mem, bytes);\n    alloc_sys_mem(old_mem, bytes).release();\n    std::memcpy(old_mem, relocated_old_mem.get(), bytes);\n}\n\nvoid cpu_pages::do_resize(size_t new_size, allocate_system_memory_fn alloc_sys_mem) {\n    auto new_pages = new_size / page_size;\n    if (new_pages <= nr_pages) {\n        return;\n    }\n    auto old_size = nr_pages * page_size;\n    auto mmap_start = memory + old_size;\n    auto mmap_size = new_size - old_size;\n    auto mem = alloc_sys_mem(mmap_start, mmap_size);\n    mem.release();\n    maybe_enable_transparent_hugepages(mmap_start, mmap_size);\n    // one past last page structure is a sentinel\n    auto new_page_array_pages = align_up(sizeof(page) * (new_pages + 1), page_size) / page_size;\n    auto new_page_array\n        = reinterpret_cast<page*>(allocate_large(new_page_array_pages, false));\n    if (!new_page_array) {\n        throw std::bad_alloc();\n    }\n    std::copy(pages, pages + nr_pages, new_page_array);\n    // mark new one-past-last page as taken to avoid boundary conditions\n    new_page_array[new_pages].free = false;\n    auto old_pages = reinterpret_cast<char*>(pages);\n    auto old_nr_pages = nr_pages;\n    auto old_pages_size = align_up(sizeof(page) * (nr_pages + 1), page_size);\n    old_pages_size = size_t(1) << log2ceil(old_pages_size);\n    pages = new_page_array;\n    nr_pages = new_pages;\n    auto old_pages_start = (old_pages - memory) / page_size;\n    if (old_pages_start == 0) {\n        // keep page 0 allocated\n        old_pages_start = 1;\n        old_pages_size -= page_size;\n    }\n    if (old_pages_size != 0) {\n        free_span_unaligned(old_pages_start, old_pages_size / page_size);\n    }\n    free_span_unaligned(old_nr_pages, new_pages - old_nr_pages);\n}\n\nvoid cpu_pages::resize(size_t new_size, allocate_system_memory_fn alloc_memory) {\n    new_size = align_down(new_size, huge_page_size);\n    while (nr_pages * page_size < new_size) {\n        // don't reallocate all at once, since there might not\n        // be enough free memory available to relocate the pages array\n        auto tmp_size = std::min(new_size, 4 * nr_pages * page_size);\n        do_resize(tmp_size, alloc_memory);\n    }\n}\n\nreclaiming_result cpu_pages::run_reclaimers(reclaimer_scope scope, size_t n_pages) {\n    auto target = std::max<size_t>(nr_free_pages + n_pages, min_free_pages);\n    reclaiming_result result = reclaiming_result::reclaimed_nothing;\n    while (nr_free_pages < target) {\n        bool made_progress = false;\n        alloc_stats::increment_local(alloc_stats::types::reclaims);\n        for (auto&& r : reclaimers) {\n            if (r->scope() >= scope) {\n                made_progress |= r->do_reclaim((target - nr_free_pages) * page_size) == reclaiming_result::reclaimed_something;\n            }\n        }\n        if (!made_progress) {\n            return result;\n        }\n        result = reclaiming_result::reclaimed_something;\n    }\n    return result;\n}\n\nvoid cpu_pages::schedule_reclaim() {\n    current_min_free_pages = 0;\n    reclaim_hook([this] {\n        if (nr_free_pages < min_free_pages) {\n            try {\n                run_reclaimers(reclaimer_scope::async, min_free_pages - nr_free_pages);\n            } catch (...) {\n                current_min_free_pages = min_free_pages;\n                throw;\n            }\n        }\n        current_min_free_pages = min_free_pages;\n    });\n}\n\nmemory::memory_layout cpu_pages::memory_layout() {\n    SEASTAR_ASSERT(is_initialized());\n    return {\n        reinterpret_cast<uintptr_t>(memory),\n        reinterpret_cast<uintptr_t>(memory) + nr_pages * page_size\n    };\n}\n\nvoid cpu_pages::set_reclaim_hook(std::function<void (std::function<void ()>)> hook) {\n    reclaim_hook = hook;\n    current_min_free_pages = min_free_pages;\n}\n\nvoid cpu_pages::set_min_free_pages(size_t pages) {\n    if (pages > std::numeric_limits<decltype(min_free_pages)>::max()) {\n        throw std::runtime_error(\"Number of pages too large\");\n    }\n    min_free_pages = pages;\n    maybe_reclaim();\n}\n\nsmall_pool::small_pool(unsigned object_size, bool is_sampled) noexcept\n    : _object_size(object_size)\n#ifdef SEASTAR_HEAPPROF\n    , _sampled_pool(is_sampled)\n#endif\n    {\n    unsigned span_size = 1;\n    auto span_bytes = [&] { return span_size * page_size; };\n    auto waste = [&] { return (span_bytes() % _object_size) / (1.0 * span_bytes()); };\n    while (object_size > span_bytes()) {\n        ++span_size;\n    }\n    _span_sizes.fallback = span_size;\n\n    // Choose a preferred span size which keeps waste (internal fragmentation) below\n    // 5% and fits at least 4 objects. If there is no span size (up to 32 pages) that\n    // satisfies this, just go with the minimum waste out of the checked span sizes.\n    float min_waste = std::numeric_limits<float>::max();\n    unsigned min_waste_span_size = 0;\n    for (span_size = 1; span_size <= 32; span_size *= 2) {\n        if (span_bytes() / object_size >= 4) {\n            auto w = waste();\n            if (w < min_waste) {\n                min_waste = w;\n                min_waste_span_size = span_size;\n                if (w < 0.05) {\n                    break;\n                }\n            }\n        }\n    }\n    _span_sizes.preferred = min_waste_span_size ? min_waste_span_size : _span_sizes.fallback;\n\n    _max_free = std::max<unsigned>(100, span_bytes() * 2 / _object_size);\n    _min_free = _max_free / 2;\n    _free = nullptr;\n}\n\nsmall_pool::~small_pool() {\n    _min_free = _max_free = 0;\n    trim_free_list();\n}\n\n/**\n * Remove the next object from the freelist and return it.\n * It must exist (caller must check) or UB.\n */\nvoid *\nsmall_pool::pop_free() {\n    auto* obj = _free;\n    _free = _free->next;\n    --_free_count;\n    return obj;\n}\n\n// Should not throw in case of running out of memory to avoid infinite recursion,\n// becaue throwing std::bad_alloc requires allocation. __cxa_allocate_exception\n// falls back to the emergency pool in case malloc() returns nullptr.\nvoid*\nsmall_pool::allocate() {\n    return __builtin_expect((bool)_free, true) ? pop_free() : add_more_objects();\n}\n\nvoid\nsmall_pool::deallocate(void* object) {\n    auto o = reinterpret_cast<free_object*>(object);\n    o->next = _free;\n    _free = o;\n    ++_free_count;\n    if (_free_count >= _max_free) {\n        trim_free_list();\n    }\n}\n\nvoid*\nsmall_pool::add_more_objects() {\n    auto goal = (_min_free + _max_free) / 2;\n    while (!_span_list.empty() && _free_count < goal) {\n        page& span = _span_list.front(get_cpu_mem().pages);\n        _span_list.pop_front(get_cpu_mem().pages);\n        while (span.freelist) {\n            auto obj = span.freelist;\n            span.freelist = span.freelist->next;\n            obj->next = _free;\n            _free = obj;\n            ++_free_count;\n            ++span.nr_small_alloc;\n        }\n    }\n    while (_free_count < goal) {\n        auto span_size = _span_sizes.preferred;\n        auto data = reinterpret_cast<char*>(get_cpu_mem().allocate_large(span_size, false));\n        if (!data) {\n            span_size = _span_sizes.fallback;\n            data = reinterpret_cast<char*>(get_cpu_mem().allocate_large(span_size, false));\n            if (!data) {\n                break;\n            }\n        }\n        auto span = get_cpu_mem().to_page(data);\n        span_size = span->span_size;\n        _pages_in_use += span_size;\n        for (unsigned i = 0; i < span_size; ++i) {\n            span[i].offset_in_span = i;\n            span[i].pool = this;\n        }\n        span->nr_small_alloc = 0;\n        span->freelist = nullptr;\n        for (size_t offset = 0; offset <= span_size * page_size - _object_size; offset += _object_size) {\n            auto h = reinterpret_cast<free_object*>(data + offset);\n            h->next = _free;\n            _free = h;\n            ++_free_count;\n            ++span->nr_small_alloc;\n        }\n    }\n\n    return _free ? pop_free() : nullptr;\n}\n\nvoid\nsmall_pool::trim_free_list() {\n    auto goal = (_min_free + _max_free) / 2;\n    while (_free && _free_count > goal) {\n        auto obj = _free;\n        _free = _free->next;\n        --_free_count;\n        page* span = get_cpu_mem().to_page(obj);\n        span -= span->offset_in_span;\n        if (!span->freelist) {\n            new (&span->link) page_list_link();\n            _span_list.push_front(get_cpu_mem().pages, *span);\n        }\n        obj->next = span->freelist;\n        span->freelist = obj;\n        if (--span->nr_small_alloc == 0) {\n            _pages_in_use -= span->span_size;\n            _span_list.erase(get_cpu_mem().pages, *span);\n            get_cpu_mem().free_span(span - get_cpu_mem().pages, span->span_size);\n        }\n    }\n}\n\nvoid\nabort_on_underflow(size_t size) {\n    if (std::make_signed_t<size_t>(size) < 0) {\n        // probably a logic error, stop hard\n        abort();\n    }\n}\n\n[[gnu::always_inline]]\ninline void* allocate_large(size_t size, bool should_sample) {\n    abort_on_underflow(size);\n    unsigned size_in_pages = (size + page_size - 1) >> page_bits;\n    if ((size_t(size_in_pages) << page_bits) < size) {\n        return nullptr; // (size + page_size - 1) caused an overflow\n    }\n    return get_cpu_mem().allocate_large(size_in_pages, should_sample);\n\n}\n\nvoid* allocate_large_aligned(size_t align, size_t size, bool should_sample) {\n    abort_on_underflow(size);\n    unsigned size_in_pages = (size + page_size - 1) >> page_bits;\n    unsigned align_in_pages = std::max(align, page_size) >> page_bits;\n    return get_cpu_mem().allocate_large_aligned(align_in_pages, size_in_pages, should_sample);\n}\n\nvoid free_large(void* ptr) {\n    return get_cpu_mem().free_large(ptr);\n}\n\nsize_t object_size(void* ptr) {\n    return cpu_pages::all_cpus[object_cpu_id(ptr)]->object_size(ptr);\n}\n\nstatic thread_local cpu_pages* cpu_mem_ptr = nullptr;\n\n// Mark as cold so that GCC8+ can move to .text.unlikely.\n[[gnu::cold]]\nstatic void init_cpu_mem() {\n    cpu_mem_ptr = &cpu_mem;\n    cpu_mem.initialize();\n}\n\n[[gnu::always_inline]]\nstatic inline cpu_pages& get_cpu_mem()\n{\n    // cpu_pages has a non-trivial constructor which means that the compiler\n    // must make sure the instance local to the current thread has been\n    // constructed before each access. So instead we access cpu_mem_ptr\n    // which has been initialized by calls to init_cpu_mem() before it is\n    // accessed.\n    return *cpu_mem_ptr;\n}\n\n#ifdef SEASTAR_DEBUG_ALLOCATIONS\nstatic constexpr int debug_allocation_pattern = 0xab;\n#endif\n\nenum class alignment_t { aligned, unaligned };\n\n#ifdef SEASTAR_HEAPPROF\ntemplate<alignment_t alignment>\nvoid* allocate_from_sampled_small_pool(size_t size) {\n    size = object_size_with_alloc_site(size);\n    if constexpr (alignment == alignment_t::aligned) {\n        size = 1 << log2ceil(size);\n    }\n    auto idx = small_pool::size_to_idx(size);\n    auto& pool = get_cpu_mem().sampled_small_pools[idx];\n    dassert(size <= pool.object_size());\n    void* ptr = pool.allocate();\n    if (__builtin_expect(ptr != nullptr, true)) {\n        // we failed to allocate, so we won't sample either\n        auto alloc_site = get_cpu_mem().add_alloc_site(pool.object_size());\n        new (&pool.alloc_site_holder(ptr)) allocation_site_ptr{alloc_site};\n    }\n    return ptr;\n}\n\n#endif\n\ntemplate<alignment_t alignment>\nvoid* allocate_from_small_pool(size_t size)\n{\n    if constexpr (alignment == alignment_t::aligned) {\n        size = 1 << log2ceil(size);\n    }\n    auto idx = small_pool::size_to_idx(size);\n    auto& pool = get_cpu_mem().small_pools[idx];\n    dassert(size <= pool.object_size());\n    return pool.allocate();\n}\n\n/**\n * Common handling code when a pointer (possibly null) has\n * been returned by any allocation path.\n */\n[[gnu::always_inline]]\nstatic inline void* finish_allocation(void* ptr, size_t size) {\n    alloc_stats::increment_local(alloc_stats::types::allocs);\n    if (__builtin_expect(!ptr, false)) {\n        on_allocation_failure(size);\n    } else {\n        alloc_stats::increment_local(alloc_stats::types::total_bytes_allocated, size);\n#ifdef SEASTAR_DEBUG_ALLOCATIONS\n        std::memset(ptr, debug_allocation_pattern, size);\n#endif\n    }\n    return ptr;\n}\n\nvoid *allocate_slowpath(size_t size) {\n    if (!is_reactor_thread) {\n        if (original_malloc_func) {\n            alloc_stats::increment(alloc_stats::types::foreign_mallocs);\n            return original_malloc_func(size);\n        }\n\n        // original_malloc_func might be null for allocations before main\n        // in constructors before original_malloc_func ctor is called\n        // Note on #2137: Moved to here, because there is lots of code\n        // that implicitly relies on the static init fiasco below to have occurred, and thus\n        // cpu_mem_ptr being available and inited. This is not great.\n        init_cpu_mem();\n\n        // #2137 - static init fiasco for fallback functions.\n        // If dependent libraries do malloc _before_ the above declaration inits are run,\n        // we end up here with nowhere to go. Add a second check and attempt the full init\n        // already. If we find the functions, all is good. Otherwise, trudge along and probably\n        // crash terribly later.\n        // Note: this is only relevant if we're in dlinit phase, in which case we are single\n        // threaded. Or there is no external func to find. No need for atomics or fences.\n        static bool double_checked = false;\n        if (!double_checked) {\n            double_checked = true;\n\n            original_malloc_func = reinterpret_cast<malloc_func_type>(dlsym(RTLD_NEXT, \"malloc\"));\n            original_free_func = reinterpret_cast<free_func_type>(dlsym(RTLD_NEXT, \"free\"));\n            original_realloc_func = reinterpret_cast<realloc_func_type>(dlsym(RTLD_NEXT, \"realloc\"));\n            original_aligned_alloc_func = reinterpret_cast<aligned_alloc_type>(dlsym(RTLD_NEXT, \"aligned_alloc\"));\n            original_malloc_trim_func = reinterpret_cast<malloc_trim_type>(dlsym(RTLD_NEXT, \"malloc_trim\"));\n            original_malloc_usable_size_func = reinterpret_cast<malloc_usable_size_type>(dlsym(RTLD_NEXT, \"malloc_usable_size\"));\n\n            if (original_malloc_func) {\n                return allocate_slowpath(size);\n            }\n        }\n    }\n    // On the fast path we've already called maybe_sample, except in the case\n    // of !is_reactor_thread (we don't sample such alloctions).\n    bool should_sample = get_cpu_mem().definitely_sample(size);\n    void* ptr;\n    if (size <= max_small_allocation) {\n#ifdef SEASTAR_HEAPPROF\n        if (should_sample) {\n            ptr = allocate_from_sampled_small_pool<alignment_t::unaligned>(size);\n        } else\n#endif\n        {\n            ptr = allocate_from_small_pool<alignment_t::unaligned>(size);\n        }\n    } else {\n        ptr = allocate_large(size, should_sample);\n    }\n    return finish_allocation(ptr, size);\n}\n\n/// The main entry point for allocation.\n///\n/// The idea is that function is inlined into malloc and all the variations of new\n/// so that the body of each of those functions will contain the full fast path\n/// fully inlined, and all of these variations will share the allocate_slowpath\n/// fallback, which is not inlined.\n[[gnu::always_inline]]\ninline void* allocate(size_t size) {\n    size = std::max(size, sizeof(free_object));\n    if (__builtin_expect(is_reactor_thread && !get_cpu_mem().maybe_sample(size) && size <= max_small_allocation, true)) {\n        auto ptr = allocate_from_small_pool<alignment_t::unaligned>(size);\n        return finish_allocation(ptr, size);\n    }\n\n    return allocate_slowpath(size);\n}\n\nvoid* allocate_aligned(size_t align, size_t size) {\n    if (!is_reactor_thread) {\n        if (original_aligned_alloc_func) {\n            alloc_stats::increment(alloc_stats::types::foreign_mallocs);\n            return original_aligned_alloc_func(align, size);\n        }\n        // original_realloc_func might be null for allocations before main\n        // in constructors before original_realloc_func ctor is called\n        init_cpu_mem();\n    }\n    if (size <= sizeof(free_object)) {\n        size = std::max(sizeof(free_object), align);\n    }\n#ifdef SEASTAR_HEAPPROF\n    auto& mem = get_cpu_mem();\n    bool should_sample = mem.maybe_sample(size) && mem.definitely_sample(size);\n#else\n    bool should_sample = false;\n#endif\n    void* ptr;\n    if (size <= max_small_allocation && align <= page_size) {\n        // Our small allocator only guarantees alignment for power-of-two\n        // allocations which are not larger than a page.\n#ifdef SEASTAR_HEAPPROF\n        if (should_sample) {\n            ptr = allocate_from_sampled_small_pool<alignment_t::aligned>(size);\n        } else\n#endif\n        {\n            ptr = allocate_from_small_pool<alignment_t::aligned>(size);\n        }\n    } else {\n        ptr = allocate_large_aligned(align, size, should_sample);\n    }\n    return finish_allocation(ptr, size);\n}\n\n\n/// Similarly to memory::allocate() we expect to inline the whole fast path\n/// of free into all the variations of the top level dallocation functions like\n/// free, delete, sized delete, etc. The slow path is not inline and is shared\n/// by all implementations.\n///\n/// The S template object allows us to handle both sized and unsized allocations\n/// in the same code path.\ntemplate <typename S = no_size>\n[[gnu::always_inline]]\ninline void free(void* obj, S size = {}) {\n    if (!__builtin_expect(cpu_pages::try_free_fastpath(obj), true)) {\n        free_slowpath(obj, size);\n    }\n}\n\nvoid free_aligned(void* obj, size_t align, size_t size) {\n    if (size <= sizeof(free_object)) {\n        size = sizeof(free_object);\n    }\n#ifndef SEASTAR_HEAPPROF\n    // See comment in the sized-free implementation below. With heap profiling we\n    // can skip this as sized-free defers to non-sized-free\n    if (size <= max_small_allocation && align <= page_size) {\n        // Same adjustment as allocate_aligned()\n        size = 1 << log2ceil(size);\n    }\n#endif\n    free(obj, size);\n}\n\nvoid shrink(void* obj, size_t new_size) {\n    alloc_stats::increment_local(alloc_stats::types::frees);\n    alloc_stats::increment_local(alloc_stats::types::allocs); // keep them balanced\n    get_cpu_mem().shrink(obj, new_size);\n}\n\nvoid set_reclaim_hook(std::function<void (std::function<void ()>)> hook) {\n    // in general, we are using Seastar allocator here. but the Seastar application\n    // can still configure smp_opts.memory_allocator with memory_allocator::standard.\n    // in that case, the memory::configure() is not called, hence cpu_mem_ptr is not\n    // set.\n    if (cpu_mem_ptr) {\n        cpu_mem_ptr->set_reclaim_hook(hook);\n    }\n}\n\nreclaimer::reclaimer(std::function<reclaiming_result ()> reclaim, reclaimer_scope scope)\n    : reclaimer([reclaim = std::move(reclaim)] (request) {\n        return reclaim();\n    }, scope) {\n}\n\nreclaimer::reclaimer(std::function<reclaiming_result (request)> reclaim, reclaimer_scope scope)\n    : _reclaim(std::move(reclaim))\n    , _scope(scope) {\n    get_cpu_mem().reclaimers.push_back(this);\n}\n\nreclaimer::~reclaimer() {\n    auto& r = get_cpu_mem().reclaimers;\n    r.erase(std::find(r.begin(), r.end(), this));\n}\n\nvoid set_large_allocation_warning_threshold(size_t threshold) {\n    get_cpu_mem().large_allocation_warning_threshold.set(threshold);\n}\n\nsize_t get_large_allocation_warning_threshold() {\n    return get_cpu_mem().large_allocation_warning_threshold.warn;\n}\n\nvoid disable_large_allocation_warning() {\n    get_cpu_mem().large_allocation_warning_threshold.set(std::numeric_limits<size_t>::max());\n}\n\nvoid configure_minimal() {\n    init_cpu_mem();\n}\n\nstatic long mbind(void *addr,\n                  unsigned long len,\n                  int mode,\n                  const unsigned long *nodemask,\n                  unsigned long maxnode,\n                  unsigned flags) {\n    return syscall(\n        SYS_mbind,\n        addr,\n        len,\n        mode,\n        nodemask,\n        maxnode,\n        flags\n    );\n}\n\nstatic\nunsigned\ncpu_id_bits(unsigned nr_shards) {\n    int w = std::bit_width(nr_shards - 1);\n    // Preserve at least 8 bits for CPU ID to have a traditional memory map\n    return std::max(w, 8);\n}\n\nsize_t\ninternal::per_shard_memory(size_t mem, unsigned procs) {\n    // limit memory address to fit in 36-bit, see Memory map\n    size_t max_mem_per_proc = 1UL << cpu_id_shift;\n    auto mem_per_proc = std::min(align_down<size_t>(mem / procs, 2 << 20), max_mem_per_proc);\n    return mem_per_proc;\n}\n\nvoid\ninternal::global_setup(unsigned nr_shards) {\n    cpu_id_shift = bits_for_cpu_id_and_memory - cpu_id_bits(nr_shards);\n    cpu_id_and_mem_base_mask = ~((uintptr_t(1) << cpu_id_shift) - 1);\n}\n\ninternal::numa_layout\nconfigure(std::vector<resource::memory> m, bool mbind,\n        bool transparent_hugepages,\n        optional<std::string> hugetlbfs_path) {\n    // we need to make sure cpu_mem is initialize since configure calls cpu_mem.resize\n    // and we might reach configure without ever allocating, hence without ever calling\n    // cpu_pages::initialize.\n    // The correct solution is to add a condition inside cpu_mem.resize, but since all\n    // other paths to cpu_pages::resize are already verifying initialize was called, we\n    // verify that here.\n    use_transparent_hugepages.store(transparent_hugepages, std::memory_order_relaxed);\n    init_cpu_mem();\n    // init_cpu_mem() could have been called very early, see call site in allocate().\n    // In that case we don't know about the transparent_hugepages parameter and conservatively\n    // assume it's true. Undo that here if it turned out to be false.\n    maybe_disable_transparent_hugepages(cpu_mem.memory, cpu_mem.nr_pages * page_size);\n    is_reactor_thread = true;\n    internal::numa_layout ret_layout;\n    size_t total = 0;\n    for (auto&& x : m) {\n        total += x.bytes;\n    }\n    allocate_system_memory_fn sys_alloc = allocate_anonymous_memory;\n    if (hugetlbfs_path) {\n        // std::function is copyable, but file_desc is not, so we must use\n        // a shared_ptr to allow sys_alloc to be copied around\n        auto fdp = make_lw_shared<file_desc>(file_desc::temporary(*hugetlbfs_path));\n        sys_alloc = [fdp] (void* where, size_t how_much) {\n            return allocate_hugetlbfs_memory(*fdp, where, how_much);\n        };\n        get_cpu_mem().replace_memory_backing(sys_alloc);\n    }\n    get_cpu_mem().resize(total, sys_alloc);\n    size_t pos = 0;\n    for (auto&& x : m) {\n        unsigned long nodemask = 1UL << x.nodeid;\n        if (mbind) {\n            auto start = get_cpu_mem().mem() + pos;\n            auto r = seastar::memory::mbind(\n                            start, x.bytes,\n                            MPOL_PREFERRED,\n                            &nodemask, std::numeric_limits<unsigned long>::digits,\n                            MPOL_MF_MOVE);\n\n            if (r == -1) {\n                char err[1000] = {};\n#ifdef SEASTAR_STRERROR_R_CHAR_P\n                const char *msg = strerror_r(errno, err, sizeof(err));\n#else\n                const char *msg = strerror_r(errno, err, sizeof(err)) ? \"unknown error\" : err;\n#endif\n                std::cerr << \"WARNING: unable to mbind shard memory; performance may suffer: \"\n                        << msg << std::endl;\n            }\n            ret_layout.ranges.push_back({.start = start, .end = start + x.bytes, .numa_node_id = x.nodeid});\n        }\n        pos += x.bytes;\n    }\n    return ret_layout;\n}\n\nstatistics stats() {\n    return statistics{alloc_stats::get(alloc_stats::types::allocs), alloc_stats::get(alloc_stats::types::frees), alloc_stats::get(alloc_stats::types::cross_cpu_frees),\n        cpu_mem.nr_pages * page_size, cpu_mem.nr_free_pages * page_size, alloc_stats::get(alloc_stats::types::total_bytes_allocated), alloc_stats::get(alloc_stats::types::reclaims), alloc_stats::get(alloc_stats::types::large_allocs),\n        alloc_stats::get(alloc_stats::types::failed_allocs), alloc_stats::get(alloc_stats::types::foreign_mallocs), alloc_stats::get(alloc_stats::types::foreign_frees),\n        alloc_stats::get(alloc_stats::types::foreign_cross_frees)};\n}\n\nsize_t free_memory() {\n    return get_cpu_mem().nr_free_pages * page_size;\n}\n\nbool drain_cross_cpu_freelist() {\n    return get_cpu_mem().drain_cross_cpu_freelist();\n}\n\nmemory_layout get_memory_layout() {\n    return get_cpu_mem().memory_layout();\n}\n\nsize_t min_free_memory() {\n    return get_cpu_mem().min_free_pages * page_size;\n}\n\nvoid set_min_free_pages(size_t pages) {\n    get_cpu_mem().set_min_free_pages(pages);\n}\n\nstatic thread_local int report_on_alloc_failure_suppressed = 0;\n\nclass disable_report_on_alloc_failure_temporarily {\npublic:\n    disable_report_on_alloc_failure_temporarily() {\n        ++report_on_alloc_failure_suppressed;\n    };\n    ~disable_report_on_alloc_failure_temporarily() noexcept {\n        --report_on_alloc_failure_suppressed;\n    }\n};\n\nstatic std::atomic<bool> abort_on_allocation_failure{false};\nstatic std::atomic<alloc_failure_kind> dump_diagnostics_on_alloc_failure_kind{alloc_failure_kind::critical};\n\nvoid set_abort_on_allocation_failure(bool enabled) {\n    abort_on_allocation_failure.store(enabled, std::memory_order_seq_cst);\n}\n\nbool is_abort_on_allocation_failure() {\n    return abort_on_allocation_failure;\n}\n\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind kind) {\n    dump_diagnostics_on_alloc_failure_kind.store(kind, std::memory_order_seq_cst);\n}\n\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(std::string_view str) {\n    if (str == \"none\") {\n        set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind::none);\n    } else if (str == \"critical\") {\n        set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind::critical);\n    } else if (str == \"all\") {\n        set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind::all);\n    } else {\n        seastar_logger.error(\"Ignoring invalid option '{}' for the allocation failure kind to dump seastar memory diagnostics for, valid options are: none, critical and all\", str);\n    }\n}\n\nstatic thread_local noncopyable_function<void(memory_diagnostics_writer)> additional_diagnostics_producer;\n\nvoid set_additional_diagnostics_producer(noncopyable_function<void(memory_diagnostics_writer)> producer) {\n    additional_diagnostics_producer = std::move(producer);\n}\n\nstruct human_readable_value {\n    uint16_t value;  // [0, 16k)\n    char suffix; // 0 -> no suffix\n    constexpr bool operator==(const human_readable_value& o) const = default;\n};\n\nstd::ostream& operator<<(std::ostream& os, const human_readable_value& val) {\n    os << val.value;\n    if (val.suffix) {\n        os << val.suffix;\n    }\n    return os;\n}\n\nstatic constexpr human_readable_value to_human_readable_value(uint64_t value, uint64_t step, uint64_t precision, const std::array<char, 6>& suffixes) {\n    if (!value) {\n        return {0, suffixes[0]};\n    }\n\n    uint64_t result = value;\n    uint64_t remainder = 0;\n    unsigned i = 0;\n    // If there is no remainder we go below precision because we don't loose any.\n    while (((!remainder && result >= step) || result >= precision)) {\n        if (i + 1 == suffixes.size()) {\n            break;\n        }\n        remainder = result % step;\n        result /= step;\n        ++i;\n    }\n    return {uint16_t(remainder < (step / 2) ? result : result + 1), suffixes[i]};\n}\n\nstatic constexpr human_readable_value to_hr_size(uint64_t size) {\n    const std::array<char, 6> suffixes = {'B', 'K', 'M', 'G', 'T', 'P'};\n    return to_human_readable_value(size, 1024, 8192, suffixes);\n}\n\nstatic_assert(to_hr_size(1ull) == human_readable_value{1, 'B'});\nstatic_assert(to_hr_size(1ull << 10) == human_readable_value{1, 'K'});\nstatic_assert(to_hr_size(1ull << 20) == human_readable_value{1, 'M'});\nstatic_assert(to_hr_size(1ull << 30) == human_readable_value{1, 'G'});\nstatic_assert(to_hr_size(1ull << 40) == human_readable_value{1, 'T'});\nstatic_assert(to_hr_size(1ull << 50) == human_readable_value{1, 'P'});\nstatic_assert(to_hr_size(1ull << 60) == human_readable_value{1024, 'P'});\nstatic_assert(to_hr_size(std::numeric_limits<uint64_t>::max()) == human_readable_value{16 << 10, 'P'});\n\nstatic constexpr human_readable_value to_hr_number(uint64_t number) {\n    const std::array<char, 6> suffixes = {'\\0', 'k', 'm', 'b', 't', 'p'};\n    return to_human_readable_value(number, 1000, 10000, suffixes);\n}\n\nseastar::internal::log_buf::inserter_iterator do_dump_memory_diagnostics(seastar::internal::log_buf::inserter_iterator it) {\n    auto free_mem = get_cpu_mem().nr_free_pages * page_size;\n    auto total_mem = get_cpu_mem().nr_pages * page_size;\n    it = fmt::format_to(it, \"Dumping seastar memory diagnostics\\n\");\n\n    it = fmt::format_to(it, \"Used memory:   {}\\n\", to_hr_size(total_mem - free_mem));\n    it = fmt::format_to(it, \"Free memory:   {}\\n\", to_hr_size(free_mem));\n    it = fmt::format_to(it, \"Total memory:  {}\\n\", to_hr_size(total_mem));\n    it = fmt::format_to(it, \"Hard failures: {}\\n\\n\", alloc_stats::get(alloc_stats::types::failed_allocs));\n\n    if (additional_diagnostics_producer) {\n        additional_diagnostics_producer([&it] (std::string_view v) mutable {\n            it = fmt::format_to(it, fmt::runtime(v));\n        });\n    }\n\n    it = fmt::format_to(it, \"Small pools:\\n\");\n    it = fmt::format_to(it, \"objsz spansz usedobj memory unused wst%\\n\");\n    for (unsigned i = 0; i < get_cpu_mem().small_pools.nr_small_pools; i++) {\n        auto& sp = get_cpu_mem().small_pools[i];\n        // We don't use pools too small to fit a free_object, so skip these, they\n        // are always empty.\n        if (sp.object_size() < sizeof(free_object)) {\n            continue;\n        }\n\n        // For the small pools, there are two types of free objects:\n        // Pool freelist objects are poitned to by sp._free and their count is sp._free_count\n        // Span freelist objects are those removed from the pool freelist when that list\n        // becomes too large: they are instead attached to the spans allocated to this\n        // pool. To count this second category, we iterate over the spans below.\n        uint32_t span_freelist_objs = 0;\n        auto front = sp._span_list._front;\n        while (front) {\n            auto& span = get_cpu_mem().pages[front];\n            auto capacity_in_objects = span.span_size * page_size / sp.object_size();\n            span_freelist_objs += capacity_in_objects - span.nr_small_alloc;\n            front = span.link._next;\n        }\n        const auto free_objs = sp._free_count + span_freelist_objs; // pool + span free objects\n        const auto use_count = sp._pages_in_use * page_size / sp.object_size() - free_objs;\n        auto memory = sp._pages_in_use * page_size;\n        const auto unused = free_objs * sp.object_size();\n        const auto wasted_percent = memory ? unused * 100 / memory : 0;\n        it = fmt::format_to(it,\n                \"{:>5}  {:>5}   {:>5}  {:>5}  {:>5} {:>4}\\n\",\n                sp.object_size(),\n                to_hr_size(sp._span_sizes.preferred * page_size),\n                to_hr_number(use_count),\n                to_hr_size(memory),\n                to_hr_size(unused),\n                unsigned(wasted_percent));\n    }\n    it = fmt::format_to(it, \"\\nPage spans:\\n\");\n    it = fmt::format_to(it, \"index  size  free  used spans\\n\");\n\n    std::array<uint32_t, cpu_pages::nr_span_lists> span_size_histogram;\n    span_size_histogram.fill(0);\n\n    for (unsigned i = 0; i < get_cpu_mem().nr_pages;) {\n        const auto span_size = get_cpu_mem().pages[i].span_size;\n        if (!span_size) {\n            ++i;\n            continue;\n        }\n        ++span_size_histogram[log2ceil(span_size)];\n        i += span_size;\n    }\n\n    for (unsigned i = 0; i< get_cpu_mem().nr_span_lists; i++) {\n        auto& span_list = get_cpu_mem().free_spans[i];\n        auto front = span_list._front;\n        uint32_t free_pages = 0;\n        while (front) {\n            auto& span = get_cpu_mem().pages[front];\n            free_pages += span.span_size;\n            front = span.link._next;\n        }\n        const auto total_spans = span_size_histogram[i];\n        const auto total_pages = total_spans * (1 << i);\n        it = fmt::format_to(it,\n                \"{:>5} {:>5} {:>5} {:>5} {:>5}\\n\",\n                i,\n                to_hr_size((uint64_t(1) << i) * page_size),\n                to_hr_size(free_pages * page_size),\n                to_hr_size((total_pages - free_pages) * page_size),\n                to_hr_number(total_spans));\n    }\n\n    return it;\n}\n\nvoid dump_memory_diagnostics(log_level lvl, logger::rate_limit& rate_limit) {\n    logger::lambda_log_writer writer([] (seastar::internal::log_buf::inserter_iterator it) {\n        return do_dump_memory_diagnostics(it);\n    });\n    seastar_memory_logger.log(lvl, rate_limit, writer);\n}\n\nnamespace internal {\nvoid log_memory_diagnostics_report(log_level lvl) {\n    logger::rate_limit rl{std::chrono::seconds(0)}; // never limit for explicit dump requests\n    dump_memory_diagnostics(lvl, rl);\n}\n}\n\nvoid maybe_dump_memory_diagnostics(size_t size, bool is_aborting) {\n    if (report_on_alloc_failure_suppressed) {\n        return;\n    }\n\n    disable_report_on_alloc_failure_temporarily guard;\n    if (seastar_memory_logger.is_enabled(log_level::debug)) {\n        seastar_memory_logger.debug(\"Failed to allocate {} bytes at {}\", size, current_backtrace());\n    }\n\n    auto lvl = log_level::debug;\n    switch (dump_diagnostics_on_alloc_failure_kind.load(std::memory_order_relaxed)) {\n        case alloc_failure_kind::none:\n            lvl = log_level::debug;\n            break;\n        case alloc_failure_kind::critical:\n            lvl = is_critical_alloc_section() ? log_level::error : log_level::debug;\n            break;\n        case alloc_failure_kind::all:\n            lvl = log_level::error;\n            break;\n    }\n\n    if (is_aborting) {\n        // if we are about to abort, always report the memory diagnositics at error level\n        lvl = log_level::error;\n    }\n\n    static thread_local logger::rate_limit rate_limit(std::chrono::seconds(10));\n    dump_memory_diagnostics(lvl, rate_limit);\n\n\n}\n\nvoid on_allocation_failure(size_t size) {\n    alloc_stats::increment(alloc_stats::types::failed_allocs);\n\n    bool will_abort = !internal::abort_on_alloc_failure_suppressed\n            && abort_on_allocation_failure.load(std::memory_order_relaxed);\n\n    maybe_dump_memory_diagnostics(size, will_abort);\n\n    if (will_abort) {\n        seastar_logger.error(\"Failed to allocate {} bytes\", size);\n        abort();\n    }\n}\n\nsstring generate_memory_diagnostics_report() {\n    seastar::internal::log_buf buf;\n    auto it = buf.back_insert_begin();\n    do_dump_memory_diagnostics(it);\n    return sstring(buf.data(), buf.size());\n}\n\nstatic void trigger_error_injector() {\n    on_alloc_point();\n}\n\nstatic bool try_trigger_error_injector() {\n    try {\n        on_alloc_point();\n        return false;\n    } catch (...) {\n        return true;\n    }\n}\n\nstd::vector<allocation_site> sampled_memory_profile() {\n    disable_backtrace_temporarily dbt;\n    std::vector<allocation_site> ret(get_cpu_mem().asu.alloc_sites.begin(), get_cpu_mem().asu.alloc_sites.end());\n    return ret;\n}\n\nsize_t sampled_memory_profile(allocation_site* output, size_t size) {\n    auto to_copy = std::min(size, get_cpu_mem().asu.alloc_sites.size());\n    std::copy_n(get_cpu_mem().asu.alloc_sites.begin(), to_copy, output);\n    return to_copy;\n}\n\n}\n\n}\n\nusing namespace seastar::memory;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid* malloc(size_t n) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    return allocate(n);\n}\n\nextern \"C\"\n[[gnu::alias(\"malloc\")]]\n[[gnu::visibility(\"default\")]]\n[[gnu::malloc]]\n[[gnu::alloc_size(1)]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\nvoid* __libc_malloc(size_t n) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid free(void* ptr) {\n    seastar::memory::free(ptr);\n}\n\nextern \"C\"\n[[gnu::alias(\"free\")]]\n[[gnu::visibility(\"default\")]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\nvoid __libc_free(void* obj) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid free_sized(void* ptr, size_t size) {\n    seastar::memory::free(ptr, size);\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid free_aligned_sized(void* ptr, size_t alignment, size_t size) {\n    seastar::memory::free_aligned(ptr, alignment, size);\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nvoid* calloc(size_t nmemb, size_t size) {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    auto s1 = __int128(nmemb) * __int128(size);\n    SEASTAR_ASSERT(s1 == size_t(s1));\n    size_t s = s1;\n    auto p = malloc(s);\n    if (p) {\n        std::memset(p, 0, s);\n    }\n    return p;\n}\n\nextern \"C\"\n[[gnu::alias(\"calloc\")]]\n[[gnu::visibility(\"default\")]]\n[[gnu::alloc_size(1, 2)]]\n[[gnu::malloc]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\nvoid* __libc_calloc(size_t n, size_t m) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nvoid* realloc(void* ptr, size_t size) {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    if (ptr == nullptr) {\n        // https://en.cppreference.com/w/cpp/memory/c/realloc\n        // If ptr is a null pointer, the behavior is the same as calling std::malloc(new_size).\n        return malloc(size);\n    } else if (!is_seastar_memory(ptr)) {\n        // original_realloc_func might be null when previous ctor allocates\n        if (original_realloc_func) {\n            return original_realloc_func(ptr, size);\n        }\n        // we can't realloc foreign memory without the original libc function\n        abort();\n    }\n    // if we're here, it's a non-null seastar memory ptr\n    auto old_size = ptr ? object_size(ptr) : 0;\n    if (size == old_size) {\n        return ptr;\n    }\n    if (size == 0) {\n        ::free(ptr);\n        return nullptr;\n    }\n    if (size < old_size && cpu_pages::is_local_pointer(ptr)) {\n        // local pointers can sometimes be shrunk by returning freed\n        // pages to the buddy allocator\n        seastar::memory::shrink(ptr, size);\n        return ptr;\n    }\n\n    // either a request to realloc larger than the existing allocation size\n    // or a cross-shard pointer: in either case we allocate a new local\n    // pointer and copy the contents\n    auto nptr = malloc(size);\n    if (!nptr) {\n        return nptr;\n    }\n    if (ptr) {\n        std::memcpy(nptr, ptr, std::min(size, old_size));\n        ::free(ptr);\n    }\n    return nptr;\n}\n\nextern \"C\"\n[[gnu::alias(\"realloc\")]]\n[[gnu::visibility(\"default\")]]\n[[gnu::alloc_size(2)]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\nvoid* __libc_realloc(void* obj, size_t size) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\n[[gnu::nonnull(1)]]\nint posix_memalign(void** ptr, size_t align, size_t size) noexcept {\n    if (try_trigger_error_injector()) {\n        return ENOMEM;\n    }\n    *ptr = allocate_aligned(align, size);\n    if (!*ptr) {\n        return ENOMEM;\n    }\n    return 0;\n}\n\nextern \"C\"\n[[gnu::alias(\"posix_memalign\")]]\n[[gnu::visibility(\"default\")]]\n#ifndef __clang__\n[[gnu::leaf]]\n#endif\n[[gnu::nonnull(1)]]\nint __libc_posix_memalign(void** ptr, size_t align, size_t size) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::malloc]]\n#if defined(__GLIBC__)\n#if __GLIBC_PREREQ(2, 30)\n[[gnu::alloc_size(2)]]\n#endif\n#if __GLIBC_PREREQ(2, 35)\n[[gnu::alloc_align(1)]]\n#endif\n#endif\nvoid* memalign(size_t align, size_t size) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    size = seastar::align_up(size, align);\n    return allocate_aligned(align, size);\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nvoid *aligned_alloc(size_t align, size_t size) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    return allocate_aligned(align, size);\n}\n\nextern \"C\"\n[[gnu::alias(\"memalign\")]]\n[[gnu::visibility(\"default\")]]\n[[gnu::malloc]]\n#if defined(__GLIBC__)\n#if __GLIBC_PREREQ(2, 30)\n[[gnu::alloc_size(2)]]\n#endif\n#if __GLIBC_PREREQ(2, 35)\n[[gnu::alloc_align(1)]]\n#endif\n#endif\nvoid* __libc_memalign(size_t align, size_t size) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nvoid cfree(void* obj) noexcept {\n    return ::free(obj);\n}\n\nextern \"C\"\n[[gnu::alias(\"cfree\")]]\n[[gnu::visibility(\"default\")]]\nvoid __libc_cfree(void* obj) noexcept;\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nsize_t malloc_usable_size(void* obj) {\n    if (!is_seastar_memory(obj)) {\n        return original_malloc_usable_size_func(obj);\n    }\n    return object_size(obj);\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\nint malloc_trim(size_t pad) {\n    if (!is_reactor_thread) {\n        return original_malloc_trim_func(pad);\n    }\n    return 0;\n}\n\nstatic inline\nvoid* throw_if_null(void* ptr) {\n    if (!ptr) {\n        throw std::bad_alloc();\n    }\n    return ptr;\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new(size_t size) {\n    trigger_error_injector();\n    return throw_if_null(allocate(size));\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new[](size_t size) {\n    trigger_error_injector();\n    return throw_if_null(allocate(size));\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, size_t size) noexcept {\n    seastar::memory::free(ptr, size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, size_t size) noexcept {\n    seastar::memory::free(ptr, size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new(size_t size, std::nothrow_t) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    return allocate(size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new[](size_t size, std::nothrow_t) noexcept {\n    return allocate(size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, std::nothrow_t) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, std::nothrow_t) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, size_t size, std::nothrow_t) noexcept {\n    seastar::memory::free(ptr, size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, size_t size, std::nothrow_t) noexcept {\n    seastar::memory::free(ptr, size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new(size_t size, std::align_val_t a) {\n    trigger_error_injector();\n    auto ptr = allocate_aligned(size_t(a), size);\n    return throw_if_null(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new[](size_t size, std::align_val_t a) {\n    trigger_error_injector();\n    auto ptr = allocate_aligned(size_t(a), size);\n    return throw_if_null(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new(size_t size, std::align_val_t a, const std::nothrow_t&) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    return allocate_aligned(size_t(a), size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid* operator new[](size_t size, std::align_val_t a, const std::nothrow_t&) noexcept {\n    if (try_trigger_error_injector()) {\n        return nullptr;\n    }\n    return allocate_aligned(size_t(a), size);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, std::align_val_t a) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, std::align_val_t a) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, size_t size, std::align_val_t a) noexcept {\n    if (ptr) {\n        seastar::memory::free_aligned(ptr, size_t(a), size);\n    }\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, size_t size, std::align_val_t a) noexcept {\n    if (ptr) {\n        seastar::memory::free_aligned(ptr, size_t(a), size);\n    }\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete(void* ptr, std::align_val_t a, const std::nothrow_t&) noexcept {\n    seastar::memory::free(ptr);\n}\n\nextern \"C++\"\n[[gnu::visibility(\"default\")]]\nvoid operator delete[](void* ptr, std::align_val_t a, const std::nothrow_t&) noexcept {\n    seastar::memory::free(ptr);\n}\n\nnamespace seastar {\n\n#else\n\nnamespace seastar {\n\nnamespace memory {\n\ndisable_backtrace_temporarily::disable_backtrace_temporarily() {\n    (void)_disable_sampling;\n}\n\nvoid set_heap_profiling_sampling_rate(size_t) {\n    seastar_logger.warn(\"Seastar compiled with default allocator, heap profiler not supported\");\n}\n\nsize_t get_heap_profiling_sample_rate() {\n    return 0;\n}\n\nstd::vector<allocation_site> sampled_memory_profile() {\n    return {};\n}\n\nsize_t sampled_memory_profile(allocation_site* output, size_t size) {\n    return 0;\n}\n\nscoped_heap_profiling::scoped_heap_profiling(size_t sample_rate) noexcept {\n    set_heap_profiling_sampling_rate(sample_rate); // let it print the warning\n}\n\nscoped_heap_profiling::~scoped_heap_profiling() {\n}\n\nvoid set_abort_on_allocation_failure(bool enabled) {\n    if (enabled) {\n        seastar_logger.warn(\"Seastar compiled with default allocator, will not abort on bad_alloc\");\n    }\n}\n\nbool is_abort_on_allocation_failure() {\n    return false;\n}\n\nreclaimer::reclaimer(std::function<reclaiming_result ()> reclaim, reclaimer_scope) {\n}\n\nreclaimer::reclaimer(std::function<reclaiming_result (request)> reclaim, reclaimer_scope) {\n}\n\nreclaimer::~reclaimer() {\n}\n\nvoid set_reclaim_hook(std::function<void (std::function<void ()>)> hook) {\n}\n\ninternal::numa_layout\nconfigure(std::vector<resource::memory> m, bool mbind,\n        bool transparent_hugepages,\n        std::optional<std::string> hugepages_path) {\n    return {};\n}\n\nvoid configure_minimal()\n{}\n\nstatistics stats() {\n    return statistics{0, 0, 0, 1 << 30, 1 << 30, 0, 0, 0, 0, 0, 0, 0};\n}\n\nsize_t free_memory() {\n    return stats().free_memory();\n}\n\nbool drain_cross_cpu_freelist() {\n    return false;\n}\n\nmemory_layout get_memory_layout() {\n    throw std::runtime_error(\"get_memory_layout() not supported\");\n}\n\nsize_t min_free_memory() {\n    return 0;\n}\n\nvoid set_min_free_pages(size_t pages) {\n    // Ignore, reclaiming not supported for default allocator.\n}\n\nvoid set_large_allocation_warning_threshold(size_t) {\n    // Ignore, not supported for default allocator.\n}\n\nsize_t get_large_allocation_warning_threshold() {\n    // Ignore, not supported for default allocator.\n    return std::numeric_limits<size_t>::max();\n}\n\nvoid disable_large_allocation_warning() {\n    // Ignore, not supported for default allocator.\n}\n\n\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(alloc_failure_kind) {\n    // Ignore, not supported for default allocator.\n}\n\nvoid set_dump_memory_diagnostics_on_alloc_failure_kind(std::string_view) {\n    // Ignore, not supported for default allocator.\n}\n\nvoid set_additional_diagnostics_producer(noncopyable_function<void(memory_diagnostics_writer)>) {\n    // Ignore, not supported for default allocator.\n}\n\nsstring generate_memory_diagnostics_report() {\n    // Ignore, not supported for default allocator.\n    return {};\n}\n\nsize_t\ninternal::per_shard_memory(size_t total_memory, unsigned nr_shards) {\n    return align_down<size_t>(total_memory / nr_shards, 2 << 20);\n}\n\nvoid\ninternal::global_setup(unsigned nr_shards) {\n}\n\nvoid free(void* ptr, size_t size) {\n    ::free(ptr);\n}\n\n}\n\n}\n\n#if !(__GLIBC__ == 2 && __GLIBC_MINOR__ >= 43) && !(__GLIBC__ > 2)\n\n// glibc 2.43 or later defines free_sized and free_aligned_sized, while we want to use\n// it even earlier\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid free_sized(void* ptr, size_t size) {\n    ::free(ptr);\n}\n\nextern \"C\"\n[[gnu::visibility(\"default\")]]\n[[gnu::used]]\nvoid free_aligned_sized(void* ptr, size_t alignment, size_t size) {\n    ::free(ptr);\n}\n\n#endif\n\nnamespace seastar {\n\n#endif\n\n/// \\endcond\n\n}\n"
  },
  {
    "path": "src/core/metrics.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n\n#include <memory>\n#include <regex>\n#include <random>\n#include <variant>\n#include <boost/range/algorithm.hpp>\n#include <boost/algorithm/string.hpp>\n#include <boost/algorithm/string/replace.hpp>\n#include <boost/range/algorithm_ext/erase.hpp>\n#include <fmt/ranges.h>\n\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/relabel_config.hh>\n#include <seastar/core/reactor.hh>\n\nnamespace seastar {\nextern seastar::logger seastar_logger;\nnamespace metrics {\n\ndouble_registration::double_registration(std::string what): std::runtime_error(what) {}\n\nmetric_groups::metric_groups() noexcept : _impl(impl::create_metric_groups()) {\n}\n\nvoid metric_groups::clear() {\n    _impl = impl::create_metric_groups();\n}\n\nmetric_groups::metric_groups(std::initializer_list<metric_group_definition> mg) : _impl(impl::create_metric_groups()) {\n    for (auto&& i : mg) {\n        add_group(i.name, i.metrics);\n    }\n}\nmetric_groups& metric_groups::add_group(const group_name_type& name, const std::initializer_list<metric_definition>& l) {\n    _impl->add_group(name, l);\n    return *this;\n}\nmetric_groups& metric_groups::add_group(const group_name_type& name, const std::vector<metric_definition>& l) {\n    _impl->add_group(name, l);\n    return *this;\n}\nmetric_group::metric_group() noexcept = default;\nmetric_group::~metric_group() = default;\nmetric_group::metric_group(const group_name_type& name, std::initializer_list<metric_definition> l) {\n    add_group(name, l);\n}\n\nmetric_group_definition::metric_group_definition(const group_name_type& name, std::initializer_list<metric_definition> l) : name(name), metrics(l) {\n}\n\nmetric_group_definition::~metric_group_definition() = default;\n\nmetric_groups::~metric_groups() = default;\nmetric_definition::metric_definition(metric_definition&& m) noexcept : _impl(std::move(m._impl)) {\n}\n\nmetric_definition::~metric_definition()  = default;\n\nmetric_definition::metric_definition(impl::metric_definition_impl const& m) noexcept :\n    _impl(std::make_unique<impl::metric_definition_impl>(m)) {\n}\n\nbool label_instance::operator<(const label_instance& id2) const {\n    auto& id1 = *this;\n    return std::tie(id1.key(), id1.value())\n                < std::tie(id2.key(), id2.value());\n}\n\nbool label_instance::operator==(const label_instance& id2) const {\n    auto& id1 = *this;\n    return std::tie(id1.key(), id1.value())\n                    == std::tie(id2.key(), id2.value());\n}\n\n\nstatic std::string get_hostname() {\n    char hostname[PATH_MAX];\n    gethostname(hostname, sizeof(hostname));\n    hostname[PATH_MAX-1] = '\\0';\n    return hostname;\n}\n\n\noptions::options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"Metrics options\")\n    , metrics_hostname(*this, \"metrics-hostname\", get_hostname(),\n            \"set the hostname used by the metrics, if not set, the local hostname will be used\")\n{\n}\n\nfuture<> configure(const options& opts) {\n    impl::config c;\n    c.hostname = opts.metrics_hostname.get_value();\n    return smp::invoke_on_all([c] {\n        impl::get_local_impl()->set_config(c);\n    });\n}\n\nfuture<metric_relabeling_result> set_relabel_configs(const std::vector<relabel_config>& relabel_configs) {\n    return impl::get_local_impl()->set_relabel_configs(relabel_configs);\n}\n\nconst std::vector<relabel_config>& get_relabel_configs() {\n    return impl::get_local_impl()->get_relabel_configs();\n}\nvoid impl::impl::update_aggregate(metric_family_info& mf) const noexcept {\n    for (const auto& fc : _metric_family_configs) {\n        if (fc.name == mf.name || fc.regex_name.match(mf.name)) {\n            mf.aggregate_labels = fc.aggregate_labels;\n        }\n    }\n}\nvoid set_metric_family_configs(const std::vector<metric_family_config>& family_config) {\n    impl::get_local_impl()->set_metric_family_configs(family_config);\n}\n\nconst std::vector<metric_family_config>& get_metric_family_configs() {\n    return impl::get_local_impl()->get_metric_family_configs();\n}\n\nbool impl::impl::apply_relabeling(const relabel_config& rc, metric_info& info) {\n    std::stringstream s;\n    bool first = true;\n    for (auto&& l: rc.source_labels) {\n        auto val = info.id.labels().find(l);\n        if (l != \"__name__\" && val == info.id.labels().end()) {\n            //If not all the labels are found nothing todo\n            return false;\n        }\n        if (first) {\n            first = false;\n        } else {\n            s << rc.separator;\n        }\n        s << ((l == \"__name__\") ? info.id.full_name() : val->second);\n    }\n    std::smatch match;\n    // regex_search forbid temporary strings\n    std::string tmps = s.str();\n    if (!std::regex_search(tmps, match, rc.expr.regex())) {\n        return false;\n    }\n\n    switch (rc.action) {\n        case relabel_config::relabel_action::drop:\n        case relabel_config::relabel_action::keep: {\n            info.enabled = rc.action == relabel_config::relabel_action::keep;\n            return true;\n        }\n        case relabel_config::relabel_action::report_when_empty:\n        case relabel_config::relabel_action::skip_when_empty: {\n            info.should_skip_when_empty = (rc.action == relabel_config::relabel_action::skip_when_empty) ? skip_when_empty::yes : skip_when_empty::no;\n            return false;\n        }\n        case relabel_config::relabel_action::drop_label: {\n            if (info.id.labels().find(rc.target_label) != info.id.labels().end()) {\n                auto new_labels = info.id.labels();\n                new_labels.erase(rc.target_label);\n                info.id.update_labels(internalize_labels(std::move(new_labels)));\n            }\n            return true;\n        };\n        case relabel_config::relabel_action::replace: {\n            if (!rc.target_label.empty()) {\n                std::string fmt_s = match.format(rc.replacement);\n                auto new_labels = info.id.labels();\n                new_labels[rc.target_label] = fmt_s;\n                info.id.update_labels(internalize_labels(std::move(new_labels)));\n            }\n            return true;\n        }\n        default:\n            break;\n    }\n    return true;\n}\n\nbool label_instance::operator!=(const label_instance& id2) const {\n    auto& id1 = *this;\n    return !(id1 == id2);\n}\n\n/*!\n * \\brief get_unique_id generate a random id\n */\nstatic std::string get_unique_id() {\n    std::random_device rd;\n    return std::to_string(rd()) + \"-\" + std::to_string(rd()) + \"-\" + std::to_string(rd()) + \"-\" + std::to_string(rd());\n}\n\nlabel shard_label(\"shard\");\nnamespace impl {\n\nnamespace {\n/*\n * true if a label value needs escaping under prometheus rules, invalid characters in\n * prometheus are also invalid in scollectd\n */\ninline bool label_needs_escaping(std::string_view value) {\n    // newline, \" and \\ need to be escaped\n    static constexpr std::string_view disallowed = \"\\n\\\"\\\\\";\n    return value.find_first_of(disallowed) != std::string_view::npos;\n}\n\n/*\n * Sanitizes the label value as per the line format rules\n * > label_value can be any sequence of UTF-8 characters, but the backslash (\\), double-quote (\"), and\n * > line feed (\\n) characters have to be escaped as \\\\, \\\", and \\n, respectively.\n */\ninline sstring escape_label_value(sstring label_value) {\n    if (!label_needs_escaping(label_value)) {\n        return label_value;\n    }\n    std::string escaped;\n    escaped.reserve(label_value.size() + 1); // some extra space\n    for (char c : label_value) {\n        switch (c) {\n            case '\\\\': escaped += R\"(\\\\)\"; break;\n            case '\\\"': escaped += R\"(\\\")\"; break;\n            case '\\n': escaped += R\"(\\n)\"; break;\n            default:   escaped += c;\n        }\n    }\n    return escaped;\n}\n\n}\n\nescaped_string::escaped_string(sstring v) : _value(escape_label_value(std::move(v))) {}\n\nescaped_string& escaped_string::operator=(const sstring& v) {\n    _value = escape_label_value(v);\n    return *this;\n}\n\nregistered_metric::registered_metric(metric_id id, metric_function f, bool enabled, skip_when_empty skip) :\n        _f(f) {\n    _info.enabled = enabled;\n    _info.should_skip_when_empty = skip;\n    _info.id = id;\n    _info.original_labels = id.internalized_labels();\n}\n\nmetric_value& metric_value::operator+=(const metric_value& c) {\n    switch (_type) {\n    case data_type::HISTOGRAM:\n    case data_type::SUMMARY:\n        std::get<histogram>(u) += std::get<histogram>(c.u);\n        break;\n    default:\n        std::get<double>(u) += std::get<double>(c.u);\n        break;\n    }\n    return *this;\n}\n\nvoid metric_value::ulong_conversion_error(double d) {\n    throw std::range_error(format(\"cannot convert double value {} to unsigned long\", d));\n}\n\nmetric_definition_impl::metric_definition_impl(\n        metric_name_type name,\n        metric_type type,\n        metric_function f,\n        description d,\n        std::vector<label_instance> _labels,\n        std::vector<label> _aggregate_labels)\n        : name(name), type(type), f(f)\n        , d(d), enabled(true) {\n    for (auto i: _labels) {\n        labels[i.key()] = i.value();\n    }\n    if (labels.find(shard_label.name()) == labels.end()) {\n        labels[shard_label.name()] = shard();\n    }\n    aggregate(_aggregate_labels);\n}\n\nmetric_definition_impl& metric_definition_impl::operator ()(bool _enabled) {\n    enabled = _enabled;\n    return *this;\n}\n\nmetric_definition_impl& metric_definition_impl::operator ()(const label_instance& label) {\n    labels[label.key()] = label.value();\n    return *this;\n}\n\nmetric_definition_impl& metric_definition_impl::operator ()(skip_when_empty skip) noexcept {\n    _skip_when_empty = skip;\n    return *this;\n}\n\nmetric_definition_impl& metric_definition_impl::set_type(const sstring& type_name) {\n    type.type_name = type_name;\n    return *this;\n}\n\nmetric_definition_impl& metric_definition_impl::aggregate(const std::vector<label>& _labels) noexcept {\n    aggregate_labels.reserve(_labels.size());\n    std::transform(_labels.begin(), _labels.end(),std::back_inserter(aggregate_labels),\n            [](const label& l) { return l.name(); });\n    return *this;\n}\n\nmetric_definition_impl& metric_definition_impl::set_skip_when_empty(bool skip) noexcept {\n    _skip_when_empty = skip_when_empty(skip);\n    return *this;\n}\n\nstd::unique_ptr<metric_groups_def> create_metric_groups() {\n    return  std::make_unique<metric_groups_impl>();\n}\n\nmetric_groups_impl::metric_groups_impl() {}\n\nmetric_groups_impl::~metric_groups_impl() {\n    for (const auto& i : _registration) {\n        unregister_metric(i->info().id);\n    }\n}\n\ninternalized_labels_ref impl::internalize_labels(labels_type labels) {\n    auto it = _internalized_labels.find(labels);\n    if (it == _internalized_labels.end()) {\n        it = _internalized_labels.emplace(labels).first;\n    }\n    return it->labels_ref();\n}\n\nmetric_groups_impl& metric_groups_impl::add_metric(group_name_type name, const metric_definition& md)  {\n    // We could just do this in the constructor but some metric groups (like the\n    // smp queue ones) are actually constructed on a different shard originally\n    // than where the actual metrics are added.\n    // Hence, the shared_ptr owning shard check would fail so we do it only here.\n    if (_impl == nullptr) {\n        _impl = get_local_impl();\n    }\n\n    auto internalized_labels = get_local_impl()->internalize_labels(md._impl->labels);\n\n    metric_id id(name, md._impl->name, internalized_labels);\n\n    auto reg = get_local_impl()->add_registration(id, md._impl->type, md._impl->f, md._impl->d, md._impl->enabled, md._impl->_skip_when_empty, md._impl->aggregate_labels);\n\n    _registration.push_back(std::move(reg));\n    return *this;\n}\n\nmetric_groups_impl& metric_groups_impl::add_group(group_name_type name, const std::vector<metric_definition>& l) {\n    for (auto i = l.begin(); i != l.end(); ++i) {\n        add_metric(name, *(i->_impl.get()));\n    }\n    return *this;\n}\n\nmetric_groups_impl& metric_groups_impl::add_group(group_name_type name, const std::initializer_list<metric_definition>& l) {\n    for (auto i = l.begin(); i != l.end(); ++i) {\n        add_metric(name, *i);\n    }\n    return *this;\n}\n\nbool metric_id::operator<(\n        const metric_id& id2) const {\n    return as_tuple() < id2.as_tuple();\n}\n\nstatic std::string safe_name(const sstring& name) {\n    auto rep = boost::replace_all_copy(boost::replace_all_copy(name, \"-\", \"_\"), \" \", \"_\");\n    boost::remove_erase_if(rep, boost::is_any_of(\"+()\"));\n    return rep;\n}\n\nsstring metric_id::full_name() const {\n    return safe_name(_group + \"_\" + _name);\n}\n\nbool metric_id::operator==(\n        const metric_id & id2) const {\n    return as_tuple() == id2.as_tuple();\n}\n\n// Unfortunately, metrics_impl can not be shared because it\n// need to be available before the first users (reactor) will call it\n\nshared_ptr<impl>  get_local_impl() {\n    static thread_local auto the_impl = ::seastar::make_shared<impl>();\n    return the_impl;\n}\nvoid impl::remove_registration(const metric_id& id) {\n    auto i = get_value_map().find(id.full_name());\n    if (i != get_value_map().end()) {\n        auto j = i->second.find(id.labels());\n        if (j != i->second.end()) {\n            j->second = nullptr;\n            i->second.erase(j);\n        }\n        if (i->second.empty()) {\n            get_value_map().erase(i);\n        }\n        dirty();\n    }\n}\n\nvoid unregister_metric(const metric_id & id) {\n    get_local_impl()->remove_registration(id);\n}\n\nconst value_map& get_value_map() {\n    return get_local_impl()->get_value_map();\n}\n\nforeign_ptr<values_reference> get_values() {\n    shared_ptr<values_copy> res_ref = ::seastar::make_shared<values_copy>();\n    auto& res = *(res_ref.get());\n    auto& mv = res.values;\n    res.metadata = get_local_impl()->metadata();\n    auto & functions = get_local_impl()->functions();\n    for (auto&& i : functions) {\n        value_vector values;\n        for (auto&& v : i) {\n            values.emplace_back(v());\n        }\n        mv.emplace_back(std::move(values));\n    }\n    return res_ref;\n}\n\n\nescaped_string shard() {\n    return escaped_string{to_sstring(this_shard_id())};\n}\n\nvoid impl::gc_internalized_labels() {\n    for (auto it = _internalized_labels.begin(); it != _internalized_labels.end();) {\n        if (!it->has_users()) {\n            it = _internalized_labels.erase(it);\n        } else {\n            ++it;\n        }\n    }\n}\n\nvoid impl::update_metrics_if_needed() {\n    if (_dirty) {\n        // Forcing the metadata to an empty initialization\n        // Will prevent using corrupted data if an exception is thrown\n        _metadata = ::seastar::make_shared<metric_metadata>();\n\n        auto mt_ref = ::seastar::make_shared<metric_metadata>();\n        auto &mt = *(mt_ref.get());\n        mt.reserve(_value_map.size());\n        _current_metrics.resize(_value_map.size());\n        size_t i = 0;\n        for (auto&& mf : _value_map) {\n            metric_metadata_fifo metrics;\n            _current_metrics[i].clear();\n            for (auto&& m : mf.second) {\n                if (m.second && m.second->is_enabled()) {\n                    metrics.emplace_back(m.second->info().id, m.second->info().should_skip_when_empty);\n                    _current_metrics[i].emplace_back(m.second->get_function());\n                }\n            }\n            if (!metrics.empty()) {\n                // If nothing was added, no need to add the metric_family\n                // and no need to progress\n                mt.emplace_back(mf.second.info(), std::move(metrics));\n                i++;\n            }\n        }\n        // Maybe we didn't use all the original size\n        _current_metrics.resize(i);\n        _metadata = mt_ref;\n        _dirty = false;\n\n        gc_internalized_labels();\n    }\n}\n\nshared_ptr<metric_metadata> impl::metadata() {\n    update_metrics_if_needed();\n    return _metadata;\n}\n\nstd::vector<std::deque<metric_function>>& impl::functions() {\n    update_metrics_if_needed();\n    return _current_metrics;\n}\n\nregister_ref impl::add_registration(const metric_id& id, const metric_type& type, metric_function f, const description& d, bool enabled, skip_when_empty skip, const std::vector<std::string>& aggregate_labels) {\n    auto rm = ::seastar::make_shared<registered_metric>(id, f, enabled, skip);\n    for (auto&& rl : _relabel_configs) {\n        apply_relabeling(rl, rm->info());\n    }\n\n    sstring name = id.full_name();\n    if (_value_map.find(name) != _value_map.end()) {\n        auto& metric = _value_map[name];\n        if (metric.find(rm->info().id.labels()) != metric.end()) {\n            throw double_registration(fmt::format(\"registering metrics twice for metrics: {} with labels {}\", name, rm->info().id.labels()));\n        }\n        if (metric.info().type != type.base_type) {\n            throw std::runtime_error(\"registering metrics \" + name + \" registered with different type.\");\n        }\n        metric[rm->info().id.internalized_labels()] = rm;\n        for (auto&& i : rm->info().id.labels()) {\n            _labels.insert(i.first);\n        }\n    } else {\n        _value_map[name].info().type = type.base_type;\n        _value_map[name].info().d = d;\n        _value_map[name].info().inherit_type = type.type_name;\n        _value_map[name].info().name = id.full_name();\n        _value_map[name].info().aggregate_labels = aggregate_labels;\n        impl::update_aggregate(_value_map[name].info());\n        _value_map[name][rm->info().id.internalized_labels()] = rm;\n    }\n    dirty();\n\n    return rm;\n}\n\nfuture<metric_relabeling_result> impl::set_relabel_configs(const std::vector<relabel_config>& relabel_configs) {\n    _relabel_configs = relabel_configs;\n    metric_relabeling_result conflicts{0};\n    for (auto&& family : _value_map) {\n        std::vector<shared_ptr<registered_metric>> rms;\n        for (auto&& metric = family.second.begin(); metric != family.second.end();) {\n            metric->second->info().id.update_labels(metric->second->info().original_labels);\n            for (auto rl : _relabel_configs) {\n                if (apply_relabeling(rl, metric->second->info())) {\n                    dirty();\n                }\n            }\n            if (metric->first.labels() != metric->second->info().id.labels()) {\n                // If a metric labels were changed, we should remove it from the map, and place it back again\n                rms.push_back(metric->second);\n                family.second.erase(metric++);\n                dirty();\n            } else {\n                ++metric;\n            }\n        }\n        for (auto rm : rms) {\n            auto ilb = rm->info().id.internalized_labels();\n            if (family.second.find(rm->info().id.labels()) != family.second.end()) {\n                /*\n                 You can not have a two metrics with the same name and label, there is a problem with the configuration.\n                 On startup we would have throw an exception and stop.\n                 But during normal run, we don't want to crash the server just because a metric reconfiguration.\n                 We cannot throw away the metric.\n                 Instead we add it with an extra label, allowing the user to reconfigure.\n                */\n                seastar_logger.error(\"Metrics: After relabeling, registering metrics twice for metrics : {}\", family.first);\n                auto id = get_unique_id();\n                auto new_labels = *ilb;\n                new_labels[\"err\"] = id;\n                ilb = internalize_labels(new_labels);\n                conflicts.metrics_relabeled_due_to_collision++;\n                rm->info().id.update_labels(ilb);\n            }\n\n            family.second[ilb] = rm;\n        }\n    }\n    return make_ready_future<metric_relabeling_result>(conflicts);\n}\n\nvoid impl::set_metric_family_configs(const std::vector<metric_family_config>& family_config) {\n    _metric_family_configs = family_config;\n    for (auto& [name, family] : _value_map) {\n        for  (const auto& fc : family_config) {\n            if (fc.name == name || fc.regex_name.match(name)) {\n                family.info().aggregate_labels = fc.aggregate_labels;\n            }\n        }\n    }\n\n    if (!_metadata) {\n        // The metadata structure may not have been built yet,\n        // or it may rebuild right now, in this case just return\n        // and it will be updated the next time, setting the dirty to true\n        // is just in case we are somehow in the middle of rebuilding the\n        // metadata.\n        dirty();\n        return;\n    }\n    for (auto& mf_metadata: *_metadata) {\n        for  (const auto& fc : family_config) {\n            if (fc.name == mf_metadata.mf.name || fc.regex_name.match(mf_metadata.mf.name)) {\n                mf_metadata.mf.aggregate_labels = fc.aggregate_labels;\n            }\n        }\n    }\n}\n}\n\nconst bool metric_disabled = false;\n\nrelabel_config::relabel_action relabel_config_action(const std::string& action) {\n    if (action == \"skip_when_empty\") {\n        return relabel_config::relabel_action::skip_when_empty;\n    }\n    if (action == \"report_when_empty\") {\n        return relabel_config::relabel_action::report_when_empty;\n    }\n    if (action == \"keep\") {\n        return relabel_config::relabel_action::keep;\n    }\n    if (action == \"drop\") {\n        return relabel_config::relabel_action::drop;\n    } if (action == \"drop_label\") {\n        return relabel_config::relabel_action::drop_label;\n    }\n    return relabel_config::relabel_action::replace;\n}\n\nhistogram& histogram::operator+=(const histogram& c) {\n    if (c.sample_count == 0) {\n        return *this;\n    }\n    for (size_t i = 0; i < c.buckets.size(); i++) {\n        if (buckets.size() <= i) {\n            buckets.push_back(c.buckets[i]);\n        } else {\n            if (buckets[i].upper_bound != c.buckets[i].upper_bound) {\n                throw std::out_of_range(\"Trying to add histogram with different bucket limits\");\n            }\n            buckets[i].count += c.buckets[i].count;\n        }\n    }\n    sample_count += c.sample_count;\n    sample_sum += c.sample_sum;\n    return *this;\n}\n\nhistogram histogram::operator+(const histogram& c) const {\n    histogram res = *this;\n    res += c;\n    return res;\n}\n\nhistogram histogram::operator+(histogram&& c) const {\n    c += *this;\n    return std::move(c);\n}\n\n}\n}\n"
  },
  {
    "path": "src/core/on_internal_error.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2020 ScyllaDB\n */\n\n\n#include <atomic>\n#include <exception>\n#include <stdexcept>\n#include <string_view>\n#include <cstdlib>\n\n#include <seastar/core/on_internal_error.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/log.hh>\n\nstatic std::atomic<bool> abort_on_internal_error{false};\n\nusing namespace seastar;\n\nbool seastar::set_abort_on_internal_error(bool do_abort) noexcept {\n    return abort_on_internal_error.exchange(do_abort);\n}\n\nnamespace seastar::internal {\nthread_local uint64_t internal_errors = 0;\nvoid increase_internal_errors_counter() noexcept {\n    internal_errors++;\n}\n}\n\ntemplate <typename Message>\nstatic void log_error_and_backtrace(logger& logger, const Message& msg) noexcept {\n    internal::increase_internal_errors_counter();\n    logger.error(\"{}, at: {}\", msg, current_backtrace());\n}\n\nvoid seastar::on_internal_error(logger& logger, std::string_view msg) {\n    log_error_and_backtrace(logger, msg);\n    if (abort_on_internal_error.load()) {\n        abort();\n    } else {\n        throw_with_backtrace<std::runtime_error>(std::string(msg));\n    }\n}\n\nvoid seastar::on_internal_error(logger& logger, std::exception_ptr ex) {\n    log_error_and_backtrace(logger, ex);\n    if (abort_on_internal_error.load()) {\n        abort();\n    } else {\n        std::rethrow_exception(std::move(ex));\n    }\n}\n\nvoid seastar::on_internal_error_noexcept(logger& logger, std::string_view msg) noexcept {\n    log_error_and_backtrace(logger, msg);\n    if (abort_on_internal_error.load()) {\n        abort();\n    }\n}\n\nvoid seastar::on_fatal_internal_error(logger& logger, std::string_view msg) noexcept {\n    log_error_and_backtrace(logger, msg);\n    abort();\n}\n"
  },
  {
    "path": "src/core/posix.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <memory>\n#include <set>\n#include <vector>\n#include <functional>\n#include <fmt/format.h>\n#include <stdlib.h>\n#include <unistd.h>\n#include <sys/mman.h>\n#include <sys/inotify.h>\n\n#include <seastar/core/posix.hh>\n#include <seastar/core/align.hh>\n#include <seastar/util/critical_alloc_section.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nfile_desc\nfile_desc::temporary(sstring directory) {\n    // FIXME: add O_TMPFILE support one day\n    directory += \"/XXXXXX\";\n    std::vector<char> templat(directory.c_str(), directory.c_str() + directory.size() + 1);\n    int fd = ::mkstemp(templat.data());\n    throw_system_error_on(fd == -1);\n    int r = ::unlink(templat.data());\n    throw_system_error_on(r == -1); // leaks created file, but what can we do?\n    return file_desc(fd);\n}\n\nfile_desc\nfile_desc::inotify_init(int flags) {\n    int fd = ::inotify_init1(flags);\n    throw_system_error_on(fd == -1, \"could not create inotify instance\");\n    return file_desc(fd);\n}\n\nsstring file_desc::fdinfo() const noexcept {\n    memory::scoped_critical_alloc_section _;\n    auto path = fmt::format(\"/proc/self/fd/{}\", _fd);\n    temporary_buffer<char> buf(64);\n    auto ret = ::readlink(path.c_str(), buf.get_write(), buf.size());\n    if (ret > 0) {\n        return sstring(buf.get(), ret);\n    } else {\n        return fmt::format(\"error({})\", errno);\n    }\n}\n\nvoid mmap_deleter::operator()(void* ptr) const {\n    ::munmap(ptr, _size);\n}\n\nmmap_area mmap_anonymous(void* addr, size_t length, int prot, int flags) {\n    auto ret = ::mmap(addr, length, prot, flags | MAP_ANONYMOUS, -1, 0);\n    throw_system_error_on(ret == MAP_FAILED);\n    return mmap_area(reinterpret_cast<char*>(ret), mmap_deleter{length});\n}\n\nvoid* posix_thread::start_routine(void* arg) noexcept {\n    auto pfunc = reinterpret_cast<std::function<void ()>*>(arg);\n    (*pfunc)();\n    return nullptr;\n}\n\nposix_thread::posix_thread(std::function<void ()> func)\n    : posix_thread(attr{}, std::move(func)) {\n}\n\nposix_thread::posix_thread(attr a, std::function<void ()> func)\n    : _func(std::make_unique<std::function<void ()>>(std::move(func))) {\n    pthread_attr_t pa;\n    auto r = pthread_attr_init(&pa);\n    if (r) {\n        throw std::system_error(r, std::system_category());\n    }\n\n#ifndef SEASTAR_ASAN_ENABLED\n    auto stack_size = a._stack_size.size;\n    if (!stack_size) {\n        stack_size = 2 << 20;\n    }\n    // allocate guard area as well\n    _stack = mmap_anonymous(nullptr, stack_size + (4 << 20),\n            PROT_NONE, MAP_PRIVATE | MAP_NORESERVE);\n    auto stack_start = align_up(_stack.get() + 1, 2 << 20);\n    mmap_area real_stack = mmap_anonymous(stack_start, stack_size,\n            PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FIXED | MAP_STACK);\n    real_stack.release(); // protected by @_stack\n    ::madvise(stack_start, stack_size, MADV_HUGEPAGE);\n    r = pthread_attr_setstack(&pa, stack_start, stack_size);\n    if (r) {\n        throw std::system_error(r, std::system_category());\n    }\n#endif\n\n#ifdef SEASTAR_PTHREAD_ATTR_SETAFFINITY_NP\n    if (a._affinity) {\n        auto& cpuset = *a._affinity;\n        pthread_attr_setaffinity_np(&pa, sizeof(cpuset), &cpuset);\n    }\n#endif\n\n    r = pthread_create(&_pthread, &pa,\n                &posix_thread::start_routine, _func.get());\n    if (r) {\n        throw std::system_error(r, std::system_category());\n    }\n\n#ifndef SEASTAR_PTHREAD_ATTR_SETAFFINITY_NP\n    if (a._affinity) {\n        auto& cpuset = *a._affinity;\n        pthread_setaffinity_np(_pthread, sizeof(cpuset), &cpuset);\n    }\n#endif\n}\n\nposix_thread::posix_thread(posix_thread&& x)\n    : _func(std::move(x._func)), _pthread(x._pthread), _valid(x._valid)\n    , _stack(std::move(x._stack)) {\n    x._valid = false;\n}\n\nposix_thread::~posix_thread() {\n    SEASTAR_ASSERT(!_valid);\n}\n\nvoid posix_thread::join() {\n    SEASTAR_ASSERT(_valid);\n    pthread_join(_pthread, NULL);\n    _valid = false;\n}\n\nstd::set<unsigned> get_current_cpuset() {\n    cpu_set_t cs;\n    auto r = pthread_getaffinity_np(pthread_self(), sizeof(cs), &cs);\n    SEASTAR_ASSERT(r == 0);\n    std::set<unsigned> ret;\n    unsigned nr = CPU_COUNT(&cs);\n    for (int cpu = 0; cpu < CPU_SETSIZE && ret.size() < nr; cpu++) {\n        if (CPU_ISSET(cpu, &cs)) {\n            ret.insert(cpu);\n        }\n    }\n    return ret;\n}\n\n}\n\n"
  },
  {
    "path": "src/core/prefault.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n// Copyright 2023 ScyllaDB\n\n#pragma once\n\n#include <atomic>\n#include <optional>\n#include <unordered_map>\n#include <vector>\n\n#include <seastar/core/posix.hh>\n#include <seastar/core/resource.hh>\n#include <seastar/core/task.hh>\n#include <seastar/core/memory.hh>\n\nnamespace seastar::alien {\n\nclass instance;\n\n};\n\nnamespace seastar::internal {\n\n// Responsible for pre-faulting in memory so soft page fault latency doesn't impact applications\nclass memory_prefaulter {\n    std::atomic<bool> _stop_request = false;\n    std::vector<posix_thread> _worker_threads;\n    // Keep this in object scope to avoid allocating in worker thread\n    std::unordered_map<unsigned, std::vector<memory::internal::memory_range>> _layout_by_node_id;\n    std::atomic<unsigned> _active_threads = 0;\npublic:\n    explicit memory_prefaulter(alien::instance& alien, const resource::resources& res, memory::internal::numa_layout layout);\n    ~memory_prefaulter();\nprivate:\n    void work(std::vector<memory::internal::memory_range>& ranges, size_t page_size, std::optional<size_t> huge_page_size_opt);\n    void join_threads() noexcept;\n};\n\n\n}\n\n"
  },
  {
    "path": "src/core/program_options.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 Cloudius Systems, Ltd.\n */\n\n#include <boost/type.hpp>\n\n#include \"core/program_options.hh\"\n\n#include <seastar/util/log-cli.hh>\n#include <seastar/util/memory_diagnostics.hh>\n#include <seastar/core/reactor_config.hh>\n#include <seastar/core/resource.hh>\n\nnamespace seastar::program_options {\n\nnamespace {\n\nconst char* to_string(memory::alloc_failure_kind val) {\n    switch (val) {\n        case memory::alloc_failure_kind::none: return \"none\";\n        case memory::alloc_failure_kind::critical: return \"critical\";\n        case memory::alloc_failure_kind::all: return \"all\";\n    }\n    std::abort();\n}\n\nconst char* to_string(log_level val) {\n    switch (val) {\n        case log_level::error: return \"error\";\n        case log_level::warn: return \"warn\";\n        case log_level::info: return \"info\";\n        case log_level::debug: return \"debug\";\n        case log_level::trace: return \"trace\";\n    }\n    std::abort();\n}\n\nconst char* to_string(logger_timestamp_style val) {\n    switch (val) {\n        case logger_timestamp_style::none: return \"none\";\n        case logger_timestamp_style::boot: return \"boot\";\n        case logger_timestamp_style::real: return \"real\";\n    }\n    std::abort();\n}\n\nconst char* to_string(logger_ostream_type val) {\n    switch (val) {\n        case logger_ostream_type::none: return \"none\";\n        case logger_ostream_type::cout: return \"stdout\";\n        case logger_ostream_type::cerr: return \"stderr\";\n    }\n    std::abort();\n}\n\nmemory::alloc_failure_kind from_string(const std::string& val, boost::type<memory::alloc_failure_kind>) {\n    if (val == \"none\") {\n        return memory::alloc_failure_kind::none;\n    } else if (val == \"critical\") {\n        return memory::alloc_failure_kind::critical;\n    } else if (val == \"all\") {\n        return memory::alloc_failure_kind::all;\n    }\n    throw std::runtime_error(fmt::format(\"Invalid value for enum memory::alloc_failure_kind: {}\", val));\n}\n\nlog_level from_string(const std::string& val, boost::type<log_level>) {\n    if (val == \"error\") {\n        return log_level::error;\n    } else if (val == \"warn\") {\n        return log_level::warn;\n    } else if (val == \"info\") {\n        return log_level::info;\n    } else if (val == \"debug\") {\n        return log_level::debug;\n    } else if (val == \"trace\") {\n        return log_level::trace;\n    }\n    throw std::runtime_error(fmt::format(\"Invalid value for enum log_level: {}\", val));\n}\n\nlogger_timestamp_style from_string(const std::string& val, boost::type<logger_timestamp_style>) {\n    if (val == \"none\") {\n        return logger_timestamp_style::none;\n    } else if (val == \"boot\") {\n        return logger_timestamp_style::boot;\n    } else if (val == \"real\") {\n        return logger_timestamp_style::real;\n    }\n    throw std::runtime_error(fmt::format(\"Invalid value for enum logger_timestamp_style: {}\", val));\n}\n\nlogger_ostream_type from_string(const std::string& val, boost::type<logger_ostream_type>) {\n    if (val == \"none\") {\n        return logger_ostream_type::none;\n    } else if (val == \"stdout\") {\n        return logger_ostream_type::cout;\n    } else if (val == \"stderr\") {\n        return logger_ostream_type::cerr;\n    }\n    throw std::runtime_error(fmt::format(\"Invalid value for enum logger_ostream_type: {}\", val));\n}\n\ntemplate <typename Type>\nvoid describe_value(bpo::options_description& opts, const std::string& name, const std::string& description, const Type& default_value) {\n    opts.add_options()(name.c_str(), boost::program_options::value<Type>()->default_value(default_value), description.c_str());\n}\n\ntemplate <typename Type>\nvoid describe_value(bpo::options_description& opts, const options_description_building_visitor::value_metadata& d, const Type& default_value) {\n    describe_value(opts, d.name, d.description, default_value);\n}\n\ntemplate <typename Type>\nvoid describe_value(bpo::options_description& opts, const std::string& name, const std::string& description) {\n    opts.add_options()(name.c_str(), boost::program_options::value<Type>(), description.c_str());\n}\n\ntemplate <typename Type>\nvoid describe_value(bpo::options_description& opts, const options_description_building_visitor::value_metadata& d) {\n    describe_value<Type>(opts, d.name, d.description);\n}\n\ntemplate <typename Type>\nvoid describe_value_maybe_default(bpo::options_description& opts, const std::string& name, const std::string& description, const Type* default_value) {\n    if (default_value) {\n        describe_value(opts, name, description, *default_value);\n    } else {\n        describe_value<Type>(opts, name, description);\n    }\n}\n\ntemplate <typename Type>\nvoid describe_value_maybe_default(bpo::options_description& opts, const options_description_building_visitor::value_metadata& d, const Type* default_value) {\n    describe_value_maybe_default(opts, d.name, d.description, default_value);\n}\n\ntemplate <typename Enum>\nvoid describe_enum_value(bpo::options_description& opts, const options_description_building_visitor::value_metadata& d, const Enum* default_value) {\n    if (default_value) {\n        opts.add_options()(d.name.c_str(), boost::program_options::value<std::string>()->default_value(to_string(*default_value)), d.description.c_str());\n    } else {\n        opts.add_options()(d.name.c_str(), boost::program_options::value<std::string>(), d.description.c_str());\n    }\n}\n\ntemplate <typename T>\nbool extract_value(const bpo::variables_map& values, const std::string& current_name, T& val) {\n    auto it = values.find(current_name);\n    if (it == values.end() || it->second.defaulted()) {\n        return false;\n    }\n    val = it->second.as<T>();\n    return true;\n}\n\ntemplate <typename T>\nbool extract_enum_value(const bpo::variables_map& values, const std::string& current_name, T& val) {\n    auto it = values.find(current_name);\n    if (it == values.end() || it->second.defaulted()) {\n        return false;\n    }\n    val = from_string(it->second.as<std::string>(), boost::type<T>{});\n    return true;\n}\n\n} // anonymous namespace\n\nbool options_description_building_visitor::visit_group_start(const std::string& name, bool used) {\n    _groups.push({name, bpo::options_description(name.c_str()), used});\n    return used;\n}\nvoid options_description_building_visitor::visit_group_end() {\n    if (_groups.size() == 1) {\n        return;\n    }\n    auto grp = std::move(_groups.top());\n    _groups.pop();\n    if (grp.used && grp.values) {\n        _groups.top().description.add(std::move(grp.description));\n    }\n}\n\nbool options_description_building_visitor::visit_value_metadata(const std::string& name, const std::string& description, bool used) {\n    if (!used) {\n        return false;\n    }\n    ++_groups.top().values;\n    _current_metadata.emplace(value_metadata{name, description});\n    return true;\n}\n\nvoid options_description_building_visitor::visit_value() {\n    _groups.top().description.add_options()(_current_metadata->name.c_str(), _current_metadata->description.c_str());\n}\n\nvoid options_description_building_visitor::visit_value(const bool* default_value) {\n    describe_value_maybe_default(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const int* default_value) {\n    describe_value_maybe_default(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const unsigned* default_value) {\n    auto name = _current_metadata->name;\n    if (_current_metadata->name == \"smp\") {\n        name = \"smp,c\";\n    }\n    describe_value_maybe_default(_groups.top().description, name, _current_metadata->description, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const float* default_value) {\n    describe_value_maybe_default(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const double* default_value) {\n    describe_value_maybe_default(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const std::string* default_value) {\n    auto name = _current_metadata->name;\n    if (_current_metadata->name == \"memory\") {\n        name = \"memory,m\";\n    }\n    describe_value_maybe_default(_groups.top().description, name, _current_metadata->description, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const std::set<unsigned>*) {\n    describe_value<std::string>(_groups.top().description, *_current_metadata);\n}\n\nvoid options_description_building_visitor::visit_value(const memory::alloc_failure_kind* default_value) {\n    describe_enum_value(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const log_level* default_value) {\n    describe_enum_value(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const logger_timestamp_style* default_value) {\n    describe_enum_value(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const logger_ostream_type* default_value) {\n    describe_enum_value(_groups.top().description, *_current_metadata, default_value);\n}\n\nvoid options_description_building_visitor::visit_value(const std::unordered_map<sstring, log_level>*) {\n    describe_value<std::vector<std::string>>(_groups.top().description, *_current_metadata);\n}\n\nvoid options_description_building_visitor::visit_selection_value(const std::vector<std::string>& candidates, const std::size_t* selected_candidate) {\n    if (selected_candidate) {\n        describe_value<std::string>(_groups.top().description, *_current_metadata, candidates.at(*selected_candidate));\n    } else {\n        describe_value<std::string>(_groups.top().description, *_current_metadata);\n    }\n}\n\nvariables_map_extracting_visitor::variables_map_extracting_visitor(const bpo::variables_map& values) : _values(values) {\n}\n\nbool variables_map_extracting_visitor::visit_group_start(const std::string& name, bool used) {\n    return used;\n}\n\nvoid variables_map_extracting_visitor::visit_group_end() {\n}\n\nbool variables_map_extracting_visitor::visit_value_metadata(const std::string& name, bool used) {\n    if (used) {\n        _current_name = &name;\n        return true;\n    } else {\n        _current_name = nullptr;\n        return false;\n    }\n}\n\nbool variables_map_extracting_visitor::visit_value() {\n    return _values.count(*_current_name);\n}\n\nbool variables_map_extracting_visitor::visit_value(bool& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(int& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(unsigned& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(float& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(double& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(std::string& val) {\n    return extract_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(std::set<unsigned>& val) {\n    std::string raw_val;\n    if (!extract_value(_values, *_current_name, raw_val)) {\n        return false;\n    }\n    if (auto parsed_cpu_set = resource::parse_cpuset(raw_val)) {\n        val = std::move(*parsed_cpu_set);\n        return true;\n    }\n    throw std::invalid_argument(fmt::format(\"invalid value for option {}: failed to parse cpuset: {}\", *_current_name, raw_val));\n}\n\nbool variables_map_extracting_visitor::visit_value(log_level& val) {\n    return extract_enum_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(logger_timestamp_style& val) {\n    return extract_enum_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(logger_ostream_type& val) {\n    return extract_enum_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(memory::alloc_failure_kind& val) {\n    return extract_enum_value(_values, *_current_name, val);\n}\n\nbool variables_map_extracting_visitor::visit_value(std::unordered_map<sstring, log_level>& val) {\n    std::vector<std::string> raw_val;\n    if (!extract_value(_values, *_current_name, raw_val)) {\n        return false;\n    }\n    for (const auto& e : raw_val) {\n        log_cli::parse_map_associations(e, [&val] (std::string k, std::string v) { val[std::move(k)] = log_cli::parse_log_level(v); });\n    }\n    return !val.empty();\n}\n\nbool variables_map_extracting_visitor::visit_selection_value(const std::vector<std::string>& candidates, std::size_t& selected_candidate) {\n    std::string candidate_name;\n    if (!extract_value(_values, *_current_name, candidate_name)) {\n        return false;\n    }\n    auto it = std::find(candidates.begin(), candidates.end(), candidate_name);\n    if (it == candidates.end()) {\n        throw std::invalid_argument(fmt::format(\"invalid value for option {}: selected candidate doesn't exist: {}\", *_current_name, candidate_name));\n    }\n    selected_candidate = it - candidates.begin();\n    return true;\n}\n\n} // namespace seastar::program_options\n"
  },
  {
    "path": "src/core/program_options.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <boost/program_options.hpp>\n#include <optional>\n#include <stack>\n\n#include <seastar/util/program-options.hh>\n\nnamespace bpo = boost::program_options;\n\nnamespace seastar::program_options {\n\n/// \\cond internal\n\nclass options_description_building_visitor : public options_descriptor {\npublic:\n    struct group_metadata {\n        const std::string& name;\n        bpo::options_description description;\n        bool used;\n        size_t values = 0;\n    };\n    struct value_metadata {\n        const std::string& name;\n        const std::string& description;\n    };\n\nprivate:\n    std::stack<group_metadata> _groups;\n    std::optional<value_metadata> _current_metadata;\n\npublic:\n    virtual bool visit_group_start(const std::string& name, bool used) override;\n    virtual void visit_group_end() override;\n\n    virtual bool visit_value_metadata(const std::string& name, const std::string& description, bool used) override;\n\n    virtual void visit_value() override;\n    virtual void visit_value(const bool* default_value) override;\n    virtual void visit_value(const int* default_value) override;\n    virtual void visit_value(const unsigned* default_value) override;\n    virtual void visit_value(const float* default_value) override;\n    virtual void visit_value(const double* default_value) override;\n    virtual void visit_value(const std::string* default_value) override;\n    virtual void visit_value(const std::set<unsigned>*) override;\n    virtual void visit_value(const memory::alloc_failure_kind* default_value) override;\n    virtual void visit_value(const log_level* default_value) override;\n    virtual void visit_value(const logger_timestamp_style* default_value) override;\n    virtual void visit_value(const logger_ostream_type* default_value) override;\n    virtual void visit_value(const std::unordered_map<sstring, log_level>*) override;\n    virtual void visit_selection_value(const std::vector<std::string>&, const std::size_t*) override;\n\n    bpo::options_description get_options_description() && { return std::move(_groups.top().description); }\n};\n\nclass variables_map_extracting_visitor : public options_mutator {\n    const bpo::variables_map& _values;\n    const std::string* _current_name = nullptr;\npublic:\n    explicit variables_map_extracting_visitor(const bpo::variables_map& values);\n\n    virtual bool visit_group_start(const std::string& name, bool used) override;\n    virtual void visit_group_end() override;\n\n    virtual bool visit_value_metadata(const std::string& name, bool used) override;\n\n    virtual bool visit_value() override;\n    virtual bool visit_value(bool&) override;\n    virtual bool visit_value(int&) override;\n    virtual bool visit_value(unsigned&) override;\n    virtual bool visit_value(float&) override;\n    virtual bool visit_value(double&) override;\n    virtual bool visit_value(std::string&) override;\n    virtual bool visit_value(std::set<unsigned>&) override;\n    virtual bool visit_value(log_level&) override;\n    virtual bool visit_value(logger_timestamp_style&) override;\n    virtual bool visit_value(logger_ostream_type&) override;\n    virtual bool visit_value(memory::alloc_failure_kind&) override;\n    virtual bool visit_value(std::unordered_map<sstring, log_level>&) override;\n\n    virtual bool visit_selection_value(const std::vector<std::string>&, std::size_t& selected_candidate) override;\n};\n\n/// \\endcond\n\n} // namespace seastar::program_options\n"
  },
  {
    "path": "src/core/prometheus-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n\n\n#include <boost/container_hash/hash_fwd.hpp>\n#include <ranges>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <string_view>\n\n#include <unordered_map>\n#include <vector>\n#include <string>\n\nnamespace seastar::prometheus {\n\n\nnamespace internal {\n// return true if label_name appears in aggr_labels\ninline bool is_aggregated(const std::vector<std::string>& aggr_labels, std::string_view label_name) {\n    return std::find_if(aggr_labels.begin(), aggr_labels.end(), [&](const std::string_view& lhs) {\n        return lhs == label_name;\n    }) != aggr_labels.end();\n}\n}\n\nstruct label_key {\n    using label_list = std::vector<std::string>;\n\n    label_key() {\n        // only the scratch key is created with this constructor, we want to reserve\n        // some reasonable amount to avoid a bunch of small allocations for the first\n        // key (which is sometimes the only key)\n        key.reserve(200);\n    }\n\n    label_key(const metrics::impl::labels_type& labels, const label_list& aggr_labels) {\n        construct(labels, aggr_labels);\n    }\n\n    void construct(const metrics::impl::labels_type& labels, const label_list& aggr_labels) {\n        key.clear();\n        for (auto& [lkey, lvalue] : labels) {\n            std::string_view key_view = lkey;\n            if (!internal::is_aggregated(aggr_labels, key_view)) {\n                key += key_view;\n                key += '\\n';\n                key += lvalue.value();\n                key += '\\n';\n            }\n        }\n\n        hash = std::hash<std::string>{}(key);\n    }\n\n    bool operator==(const label_key& o) const noexcept {\n        return hash == o.hash && key == o.key;\n    }\n\n    std::string key;\n    size_t hash = 0;\n};\n\n} // namespace seastar::prometheus\n\nnamespace std {\ntemplate <>\nstruct hash<seastar::prometheus::label_key> {\n    size_t operator()(const seastar::prometheus::label_key& lk) const noexcept {\n        return lk.hash;\n    }\n};\n}\n\n\nnamespace seastar::prometheus {\n\n/*!\n * \\brief a helper class to aggregate metrics over labels\n *\n * This class sum multiple metrics based on a list of labels.\n * It returns one or more metrics each aggregated by the aggregate_by labels.\n *\n * To use it, you define what labels it should aggregate by and then pass to\n * it metrics with their labels.\n * For example if a metrics has a 'shard' and 'name' labels and you aggregate by 'shard'\n * it would return a map of metrics each with only the 'name' label\n *\n */\nclass metric_aggregate_by_labels {\n    using labels_type = metrics::impl::labels_type;\n\n    struct labels_value {\n        labels_type labels;\n        metrics::impl::metric_value m;\n    };\n\npublic:\n    using label_list_type = std::vector<std::string>;\n    using map_type = std::unordered_map<label_key, labels_value>;\n    const label_list_type& _labels_to_aggregate_by;\n    label_key scratch_key;\n    map_type _values;\npublic:\n    metric_aggregate_by_labels(const label_list_type& labels) : _labels_to_aggregate_by(labels) {\n    }\n    /*!\n     * \\brief add a metric\n     *\n     * This method gets a metric and its labels and adds it to the aggregated metric.\n     * For example, if a metric has the labels {'shard':'0', 'name':'myhist'} and we are aggregating\n     * over 'shard'\n     * The metric would be added to the aggregated metric with labels {'name':'myhist'}.\n     *\n     */\n    void add(const seastar::metrics::impl::metric_value& m, const labels_type& input_labels) noexcept {\n        scratch_key.construct(input_labels, _labels_to_aggregate_by);\n        auto i = _values.find(scratch_key);\n        if (i == _values.end()) {\n            labels_type labels;\n            for (auto& l : input_labels) {\n                if (!internal::is_aggregated(_labels_to_aggregate_by, l.first)) {\n                    labels.insert(l);\n                }\n            }\n            _values.emplace(scratch_key, labels_value{std::move(labels), m});\n        } else {\n            i->second.m += m;\n        }\n    }\n    const auto& get_values() const noexcept {\n        return _values;\n    }\n    bool empty() const noexcept {\n        return _values.empty();\n    }\n};\n\n}\n"
  },
  {
    "path": "src/core/prometheus.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#include <fmt/core.h>\n#include <fmt/compile.h>\n#include <google/protobuf/io/coded_stream.h>\n#include <google/protobuf/io/zero_copy_stream_impl_lite.h>\n#include \"proto/metrics2.pb.h\"\n\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/core/scollectd.hh>\n#include <seastar/http/function_handlers.hh>\n\n#include \"prometheus-impl.hh\"\n\n#include <boost/algorithm/string/replace.hpp>\n#include <boost/range/algorithm_ext/erase.hpp>\n#include <boost/algorithm/string.hpp>\n#include <boost/range/algorithm.hpp>\n#include <boost/range/combine.hpp>\n#include <seastar/core/thread.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/assert.hh>\n#include <ranges>\n#include <regex>\n#include <string_view>\n#include <type_traits>\n\nusing namespace std::literals;\n\ntemplate<>\nstruct fmt::formatter<seastar::metrics::impl::metric_value> {\n    constexpr auto parse (format_parse_context& ctx) {\n        return ctx.begin();\n    }\n\n    constexpr auto format(const seastar::metrics::impl::metric_value& v, auto& ctx) const {\n        switch (v.type()) {\n        case seastar::metrics::impl::data_type::GAUGE:\n        case seastar::metrics::impl::data_type::REAL_COUNTER:\n            return format_to(ctx.out(), FMT_COMPILE(\"{:.6f}\"), v.d());\n            break;\n        case seastar::metrics::impl::data_type::COUNTER:\n            return format_to(ctx.out(), FMT_COMPILE(\"{}\"), v.i());\n            break;\n        case seastar::metrics::impl::data_type::HISTOGRAM:\n        case seastar::metrics::impl::data_type::SUMMARY:\n            // handled with a different code path\n            return ctx.out();\n        }\n        assert(false);\n        __builtin_unreachable();\n    }\n};\n\nnamespace seastar {\n\nextern seastar::logger seastar_logger;\n\nconstexpr std::string_view to_string(seastar::metrics::impl::data_type t) {\n    switch (t) {\n    case seastar::metrics::impl::data_type::GAUGE:\n        return \"gauge\";\n    case seastar::metrics::impl::data_type::COUNTER:\n    case seastar::metrics::impl::data_type::REAL_COUNTER:\n        return \"counter\";\n    case seastar::metrics::impl::data_type::HISTOGRAM:\n        return \"histogram\";\n    case seastar::metrics::impl::data_type::SUMMARY:\n        return \"summary\";\n    };\n    return \"untyped\";\n};\n\nnamespace prometheus {\nnamespace pm = io::prometheus::client;\n\nnamespace mi = metrics::impl;\n\nusing mi::labels_type;\nusing write_body_args = details::write_body_args;\n\nnamespace {\n\n// A wrapper around a std::vector for efficient formatting and appending\n// of text data. Just using a std::vector is much slower.\n// Inspired by log_buf.\nclass fmt_buf {\n    char* _current;\n    std::vector<char> buffer;\nprivate:\n\n    const char* buf_end() const noexcept {\n        return buffer.data() + buffer.size();\n    }\n\n    auto& realloc_buffer_and_append(char c) {\n        auto current_size = size();  // Save size before resizing\n        buffer.resize(buffer.size() * 2);\n        _current = buffer.data() + current_size;\n        *_current++ = c;\n        return *this;\n    }\n\npublic:\n    static constexpr size_t initial_size = 1024;\n\n    fmt_buf() {\n        buffer.resize(initial_size);\n        _current = buffer.data();\n    }\n\n    fmt_buf(const fmt_buf&) = delete;\n    fmt_buf(fmt_buf&&) = delete;\n    fmt_buf& operator=(const fmt_buf&) = delete;\n    fmt_buf& operator=(fmt_buf&&) = delete;\n\n    /// Clear the buffer, setting its position back to the start, but does not\n    /// free any buffers (after this called, size is zero, capacity is unchanged).\n    /// Any existing iterators are invalidated.\n    void clear() { _current = data(); }\n\n    /// The amount of data written so far.\n    size_t size() const noexcept { return _current - data(); }\n\n    /// The remaining space\n    size_t remaining() const noexcept { return buf_end() - _current; }\n\n    const char* data() const noexcept { return buffer.data(); }\n    char* data() noexcept { return buffer.data(); }\n\n    auto& append(char c) noexcept {\n        if (_current == buf_end()) [[unlikely]] {\n            return realloc_buffer_and_append(c);\n        }\n        *_current++ = c;\n        return *this;\n    }\n\n    fmt_buf& append(std::string_view str) {\n        if (str.size() <= remaining()) [[likely]] {\n            std::copy(str.begin(), str.end(), _current);\n            _current += str.size();\n        } else {\n            append_slowpath(str);\n        }\n        return *this;\n    }\n\n    fmt_buf& operator<<(std::string_view str) {\n        return append(str);\n    }\n\n    void append_slowpath(std::string_view sv) {\n        // This is taken very infrequently, as we (a) size the buffer generously\n        // to start and (b) keep using the same buffer for the entire request\n        // but flush it every metric, so so once it grows larger (if it needs\n        // to) that space can be reused by subsequent metrics, so the total number\n        // of appends is effectively capped per request.\n        // Therefore, we just do the simplest thing here which is to append char\n        // by char (which internally handles the resize).\n        for (auto c : sv) {\n            append(c);\n        }\n    }\n\n    // removes last character added (buffer must not be empty)\n    void pop_back() {\n        --_current;\n    }\n\n    // the last character added (buffer must not be empty)\n    char back() {\n        return *(_current - 1);\n    }\n\n    std::string_view str() const {\n        return {data(), size()};\n    }\n\n    class inserter_iterator {\n        fmt_buf* _buf;\n    public:\n        using iterator_category = std::output_iterator_tag;\n        using difference_type = std::ptrdiff_t;\n        using value_type = void;\n        using pointer = void;\n        using reference = void;\n\n        explicit inserter_iterator(fmt_buf& buf) noexcept : _buf(&buf) { }\n\n        void operator=(char c) noexcept {\n            _buf->append(c);\n        }\n\n        inserter_iterator& operator*() noexcept { return *this; }\n        inserter_iterator& operator++() noexcept { return *this; }\n        inserter_iterator operator++(int) noexcept { return *this; }\n    };\n\n    /// Create an output iterator which allows writing into the buffer.\n    inserter_iterator back_insert_begin() noexcept { return inserter_iterator(*this); }\n};\n}\n\nusing buf_t = fmt_buf;\n/**\n * Taken from an answer in stackoverflow:\n * http://stackoverflow.com/questions/2340730/are-there-c-equivalents-for-the-protocol-buffers-delimited-i-o-functions-in-ja\n */\nstatic bool write_delimited_to(const google::protobuf::MessageLite& message,\n        google::protobuf::io::ZeroCopyOutputStream* rawOutput) {\n    google::protobuf::io::CodedOutputStream output(rawOutput);\n\n#if GOOGLE_PROTOBUF_VERSION >= 3004000\n    const size_t size = message.ByteSizeLong();\n    output.WriteVarint64(size);\n#else\n    const int size = message.ByteSize();\n    output.WriteVarint32(size);\n#endif\n\n    uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);\n    if (buffer != nullptr) {\n        message.SerializeWithCachedSizesToArray(buffer);\n    } else {\n        message.SerializeWithCachedSizes(&output);\n        if (output.HadError()) {\n            return false;\n        }\n    }\n\n    return true;\n}\n\nstatic pm::Metric* add_label(pm::Metric* mt, const metrics::impl::labels_type & id, const config& ctx) {\n    mt->mutable_label()->Reserve(id.size() + 1);\n    if (ctx.label) {\n        auto label = mt->add_label();\n        label->set_name(std::string(ctx.label->key()));\n        label->set_value(std::string(ctx.label->value()));\n    }\n    for (auto && [name, value] : id) {\n        auto label = mt->add_label();\n        label->set_name(std::string(name));\n        label->set_value(std::string(value.value()));\n    }\n    return mt;\n}\n\nstatic void fill_old_type_histogram(const metrics::histogram& h, ::io::prometheus::client::Histogram* mh) {\n    mh->set_sample_count(h.sample_count);\n    mh->set_sample_sum(h.sample_sum);\n    for (auto b : h.buckets) {\n        auto bc = mh->add_bucket();\n        bc->set_cumulative_count(b.count);\n        bc->set_upper_bound(b.upper_bound);\n    }\n}\n/*!\n * Fill a histogram using the Prometheus native histogram representation.\n *\n * Prometheus Native histogram (also known as sparse histograms)\n * uses an exponential bucket size with a coefficient equal to 2^(2^-schema).\n * Besides the schema, a native histogram has a list of BucketSpan and a list of deltas.\n * Each entry in the list of deltas represents a nonempty bucket,\n * the bucket value is stored as a delta from the previous nonempty bucket.\n * The bucket-spans list describes the buckets ids.\n * Each back span represents multiple consecutive nonempty buckets.\n * It holds the id of the first bucket in the span of buckets and the length (number of nonempty consecutive buckets).\n */\nstatic void fill_native_type_histogram(const metrics::histogram& h, ::io::prometheus::client::Histogram* mh) {\n    mh->set_sample_count(h.sample_count);\n    mh->set_sample_sum(h.sample_sum);\n    size_t id = h.native_histogram.value().min_id;\n\n    mh->set_schema(h.native_histogram.value().schema);\n    double last_bucket = 0;\n    double count = 0;\n\n    size_t length = 0;\n    size_t last_bucket_id = 0;\n    ::io::prometheus::client::BucketSpan* bucket_span = nullptr;\n    for (auto b : h.buckets) {\n        // Metrics histograms are aggregated histograms\n        // A non empty bucket is bigger than the previous one\n        if (count < b.count) {\n            // If we are not part of an existing bucket-span, create one\n            if (!bucket_span) {\n                bucket_span = mh->add_positive_span();\n                bucket_span->set_offset(id - last_bucket_id);\n                length = 0;\n            }\n            length++;\n            mh->add_positive_delta(b.count - count - last_bucket);\n            last_bucket = b.count - count;\n        } else {\n            // The current bucket is empty, if there is an existing bucket-span\n            // set its length\n            if (bucket_span) {\n                bucket_span->set_length(length);\n                bucket_span = nullptr;\n                last_bucket_id = id;\n            }\n        }\n        count = b.count;\n        id++;\n    }\n    // maybe there is an open bucket span (the last bucket was part of a bucket-span)\n    // set its length\n    if (bucket_span) {\n        bucket_span->set_length(length);\n    }\n}\n\nstatic void fill_metric(pm::MetricFamily& mf, const metrics::impl::metric_value& c,\n        const metrics::impl::labels_type & id, const config& ctx) {\n    switch (c.type()) {\n    case scollectd::data_type::GAUGE:\n        add_label(mf.add_metric(), id, ctx)->mutable_gauge()->set_value(c.d());\n        mf.set_type(pm::MetricType::GAUGE);\n        break;\n    case scollectd::data_type::SUMMARY: {\n        auto& h = c.get_histogram();\n        auto mh = add_label(mf.add_metric(), id,ctx)->mutable_summary();\n        mh->set_sample_count(h.sample_count);\n        mh->set_sample_sum(h.sample_sum);\n        for (auto b : h.buckets) {\n            auto bc = mh->add_quantile();\n            bc->set_value(b.count);\n            bc->set_quantile(b.upper_bound);\n        }\n        mf.set_type(pm::MetricType::SUMMARY);\n        break;\n    }\n    case scollectd::data_type::HISTOGRAM:\n    {\n        auto& h = c.get_histogram();\n        auto mh = add_label(mf.add_metric(), id,ctx)->mutable_histogram();\n        mh->set_sample_count(h.sample_count);\n        mh->set_sample_sum(h.sample_sum);\n        if (h.native_histogram) {\n            fill_native_type_histogram(h, mh);\n        } else {\n            fill_old_type_histogram(h, mh);\n        }\n        mf.set_type(pm::MetricType::HISTOGRAM);\n        break;\n    }\n    case scollectd::data_type::REAL_COUNTER:\n        [[fallthrough]];\n    case scollectd::data_type::COUNTER:\n        add_label(mf.add_metric(), id, ctx)->mutable_counter()->set_value(c.d());\n        mf.set_type(pm::MetricType::COUNTER);\n        break;\n    }\n}\n\nstatic inline bool is_internal(const sstring& name) {\n    if (auto cstr = name.c_str(); cstr[0] == '_' && cstr[1] == '_') [[unlikely]] {\n        return true;\n    }\n    return false;\n}\n\n[[gnu::always_inline]]\nstatic inline void write_label(buf_t& s, std::string_view key, std::string_view value) {\n    s << key << \"=\\\"\";\n    s << value;\n    s << \"\\\",\";\n};\n\nstruct no_label {};\n\ntemplate <typename Extra = no_label>\nstatic void write_name_and_labels(buf_t& buf, std::string_view name, auto suffix, const labels_type& labels, const config& ctx, Extra extra = {}) {\n\n    // extra label can be injected in by the caller, unfortunately prometheus\n    // requires labels to be sorted by name, so we have to jump through some\n    // hoops to do that.\n    constexpr bool has_extra = !std::is_same_v<Extra, no_label>;\n\n    buf << name << suffix << \"{\";\n    if (ctx.label) [[unlikely]] {\n        write_label(buf, ctx.label->key(), ctx.label->value());\n    }\n\n    bool wrote_extra = false;\n\n    for (auto& l : labels) {\n        std::string_view label_name = l.first;\n        if constexpr (has_extra) {\n            if (!wrote_extra && extra.name < label_name) {\n                write_label(buf, extra.name, extra.value);\n                wrote_extra = true;\n            }\n        }\n        if (!is_internal(l.first)) [[likely]] {\n            write_label(buf, label_name, l.second.value());\n        }\n    }\n\n    if constexpr (has_extra) {\n        if (!wrote_extra) {\n            write_label(buf, extra.name, extra.value);\n        }\n    }\n\n    if (buf.back() == ',') {\n        buf.pop_back();\n    }\n\n    buf.append(\"} \");\n}\n\n/*!\n * \\brief iterator for metric family\n *\n * In prometheus, a single shard collecct all the data from the other\n * shards and report it.\n *\n * Each shard returns a value_copy struct that has a vector of vector values (a vector per metric family)\n * and a vector of metadata (and insdie it a vector of metric metadata)\n *\n * The metrics are sorted by the metric family name.\n *\n * In prometheus, all the metrics that belongs to the same metric family are reported together.\n *\n * For efficiency the results from the metrics layer are kept in a vector.\n *\n * So we have a vector of shards of a vector of metric families of a vector of values.\n *\n * To produce the result, we use the metric_family_iterator that is created by metric_family_range.\n *\n * When iterating over the metrics we use two helper structure.\n *\n * 1. A map between metric family name and the total number of values (combine on all shards) and\n *    pointer to the metric family metadata.\n * 2. A vector of positions to the current metric family for each shard.\n *\n * The metric_family_range returns a metric_family_iterator that goes over all the families.\n *\n * The iterator returns a metric_family object, that can report the metric_family name, the size (how many\n * metrics in total belongs to the metric family) and a a foreach_metric method.\n *\n * The foreach_metric method can be used to perform an action on each of the metric that belongs to\n * that metric family\n *\n * Iterating over the metrics is done:\n * - go over each of the shard and each of the entry in the position vector:\n *   - if the current family (the metric family that we get from the shard and position) has the current name:\n *     - iterate over each of the metrics belong to that metric family:\n *\n * for example, if m is a metric_family_range\n *\n * for (auto&& i : m) {\n *   std::cout << i.name() << std::endl;\n *   i.foreach_metric([](const mi::metric_value& value, const mi::metric_info& value_info) {\n *     std::cout << value_info.id.labels().size() <<std::cout;\n *   });\n * }\n *\n * Will print all the metric family names followed by the number of labels each metric has.\n */\nclass metric_family_iterator;\n\nclass metric_family_range;\n\nclass metrics_families_per_shard {\n    using metrics_family_per_shard_data_container = std::vector<foreign_ptr<mi::values_reference>>;\n    metrics_family_per_shard_data_container _data;\n    using comp_function = std::function<bool(const sstring&, const mi::metric_family_metadata&)>;\n    /*!\n     * \\brief find the last item in a range of metric family based on a comparator function\n     *\n     */\n    metric_family_iterator find_bound(const sstring& family_name, comp_function comp) const;\n\npublic:\n\n    using const_iterator = metrics_family_per_shard_data_container::const_iterator;\n    using iterator = metrics_family_per_shard_data_container::iterator;\n    using reference = metrics_family_per_shard_data_container::reference;\n    using const_reference = metrics_family_per_shard_data_container::const_reference;\n\n    /*!\n     * \\brief find the first item following a metric family range.\n     * metric family are sorted, this will return the first item that is outside\n     * of the range\n     */\n    metric_family_iterator upper_bound(const sstring& family_name) const;\n\n    /*!\n     * \\brief find the first item in a range of metric family.\n     * metric family are sorted, the first item, is the first to match the\n     * criteria.\n     */\n    metric_family_iterator lower_bound(const sstring& family_name) const;\n\n    /**\n     * \\defgroup Variables Global variables\n     */\n\n    /*\n     * @defgroup Vector properties\n     * The following methods making metrics_families_per_shard act as\n     * a vector of foreign_ptr<mi::values_reference>\n     * @{\n     *\n     *\n     */\n    iterator begin() {\n        return _data.begin();\n    }\n\n    iterator end() {\n        return _data.end();\n    }\n\n    const_iterator begin() const {\n        return _data.begin();\n    }\n\n    const_iterator end() const {\n        return _data.end();\n    }\n\n    void resize(size_t new_size) {\n        _data.resize(new_size);\n    }\n\n    reference& operator[](size_t n) {\n        return _data[n];\n    }\n\n    const_reference& operator[](size_t n) const {\n        return _data[n];\n    }\n    /** @} */\n};\n\nstatic future<metrics_families_per_shard> get_map_value() {\n    metrics_families_per_shard vec;\n    vec.resize(smp::count);\n    co_await parallel_for_each(std::views::iota(0u, smp::count), [&vec] (auto cpu) {\n        return smp::submit_to(cpu, [] {\n            return mi::get_values();\n        }).then([&vec, cpu] (auto res) {\n            vec[cpu] = std::move(res);\n        });\n    });\n    co_return vec;\n}\n\n/*!\n * \\brief a facade class for metric family\n */\nclass metric_family {\n    const sstring* _name = nullptr;\n    uint32_t _size = 0;\n    const mi::metric_family_info* _family_info = nullptr;\n    metric_family_iterator& _iterator_state;\n    metric_family(metric_family_iterator& state) : _iterator_state(state) {\n    }\n    metric_family(const sstring* name , uint32_t size, const mi::metric_family_info* family_info, metric_family_iterator& state) :\n        _name(name), _size(size), _family_info(family_info), _iterator_state(state) {\n    }\n    metric_family(const metric_family& info, metric_family_iterator& state) :\n        metric_family(info._name, info._size, info._family_info, state) {\n    }\npublic:\n    metric_family(const metric_family&) = delete;\n    metric_family(metric_family&&) = delete;\n\n    const sstring& name() const {\n        return *_name;\n    }\n\n    uint32_t size() const {\n        return _size;\n    }\n\n    const mi::metric_family_info& metadata() const {\n        return *_family_info;\n    }\n\n    void foreach_metric(std::function<void(const mi::metric_value&, const mi::metric_series_metadata&)>&& f);\n\n    bool end() const {\n        return !_name || !_family_info;\n    }\n    friend class metric_family_iterator;\n};\n\nclass metric_family_iterator {\n    const metrics_families_per_shard& _families;\n    std::vector<size_t> _positions;\n    metric_family _info;\n\n    void next() {\n        if (_positions.empty()) {\n            return;\n        }\n        const sstring *new_name = nullptr;\n        const mi::metric_family_info* new_family_info = nullptr;\n        _info._size = 0;\n        for (auto&& i : boost::combine(_positions, _families)) {\n            auto& pos_in_metric_per_shard = boost::get<0>(i);\n            auto& metric_family = boost::get<1>(i);\n            if (_info._name &&  pos_in_metric_per_shard < metric_family->metadata->size() &&\n                    metric_family->metadata->at(pos_in_metric_per_shard).mf.name.compare(*_info._name) <= 0) {\n                pos_in_metric_per_shard++;\n            }\n            if (pos_in_metric_per_shard >= metric_family->metadata->size()) {\n                // no more metric family in this shard\n                continue;\n            }\n            auto& metadata = metric_family->metadata->at(pos_in_metric_per_shard);\n            int cmp = (!new_name) ? -1 : metadata.mf.name.compare(*new_name);\n            if (cmp < 0) {\n                new_name = &metadata.mf.name;\n                new_family_info = &metadata.mf;\n                _info._size = 0;\n            }\n            if (cmp <= 0) {\n                _info._size += metadata.metrics.size();\n            }\n        }\n        _info._name = new_name;\n        _info._family_info = new_family_info;\n    }\n\npublic:\n    metric_family_iterator() = delete;\n    metric_family_iterator(const metric_family_iterator& o) : _families(o._families), _positions(o._positions), _info(*this) {\n        next();\n    }\n\n    metric_family_iterator(metric_family_iterator&& o) : _families(o._families), _positions(std::move(o._positions)),\n            _info(*this) {\n        next();\n    }\n\n    metric_family_iterator(const metrics_families_per_shard& families,\n            unsigned shards)\n        : _families(families), _positions(shards, 0), _info(*this) {\n        next();\n    }\n\n    metric_family_iterator(const metrics_families_per_shard& families,\n            std::vector<size_t>&& positions)\n        : _families(families), _positions(std::move(positions)), _info(*this) {\n        next();\n    }\n\n    metric_family_iterator& operator++() {\n        next();\n        return *this;\n    }\n\n    metric_family_iterator operator++(int) {\n        metric_family_iterator previous(*this);\n        next();\n        return previous;\n    }\n\n    bool operator!=(const metric_family_iterator& o) const {\n        return !(*this == o);\n    }\n\n    bool operator==(const metric_family_iterator& o) const {\n        if (end()) {\n            return o.end();\n        }\n        if (o.end()) {\n            return false;\n        }\n        return name() == o.name();\n    }\n\n    metric_family& operator*() {\n        return _info;\n    }\n\n    metric_family* operator->() {\n        return &_info;\n    }\n    const sstring& name() const {\n        return *_info._name;\n    }\n\n    uint32_t size() const {\n        return _info._size;\n    }\n\n    const mi::metric_family_info& metadata() const {\n        return *_info._family_info;\n    }\n\n    bool end() const {\n        return _positions.empty() || _info.end();\n    }\n\n    void foreach_metric(std::function<void(const mi::metric_value&, const mi::metric_series_metadata&)>&& f) {\n        // iterating over the shard vector and the position vector\n        for (auto&& i : boost::combine(_positions, _families)) {\n            auto& pos_in_metric_per_shard = boost::get<0>(i);\n            auto& metric_family = boost::get<1>(i);\n            if (pos_in_metric_per_shard >= metric_family->metadata->size()) {\n                // no more metric family in this shard\n                continue;\n            }\n            auto& metadata = metric_family->metadata->at(pos_in_metric_per_shard);\n            // the the name is different, that means that on this shard, the metric family\n            // does not exist, because everything is sorted by metric family name, this is fine.\n            if (metadata.mf.name == name()) {\n                const mi::value_vector& values = metric_family->values[pos_in_metric_per_shard];\n                const mi::metric_metadata_fifo& metrics_metadata = metadata.metrics;\n                for (auto&& vm : boost::combine(values, metrics_metadata)) {\n                    auto& value = boost::get<0>(vm);\n                    auto& metric_metadata = boost::get<1>(vm);\n                    f(value, metric_metadata);\n                }\n            }\n        }\n    }\n\n};\n\nvoid metric_family::foreach_metric(std::function<void(const mi::metric_value&, const mi::metric_series_metadata&)>&& f) {\n    _iterator_state.foreach_metric(std::move(f));\n}\n\nclass metric_family_range {\n    metric_family_iterator _begin;\n    metric_family_iterator _end;\npublic:\n    metric_family_range(const metrics_families_per_shard& families) : _begin(families, smp::count),\n        _end(metric_family_iterator(families, 0))\n    {\n    }\n\n    metric_family_range(const metric_family_iterator& b, const metric_family_iterator& e) : _begin(b), _end(e)\n    {\n    }\n\n    metric_family_iterator begin() const {\n        return _begin;\n    }\n\n    metric_family_iterator end() const {\n        return _end;\n    }\n};\n\nmetric_family_iterator metrics_families_per_shard::find_bound(const sstring& family_name, comp_function comp) const {\n    std::vector<size_t> positions;\n    positions.reserve(smp::count);\n\n    for (auto& shard_info : _data) {\n        std::vector<mi::metric_family_metadata>& metadata = *(shard_info->metadata);\n        std::vector<mi::metric_family_metadata>::iterator it_b = boost::range::upper_bound(metadata, family_name, comp);\n        positions.emplace_back(it_b - metadata.begin());\n    }\n    return metric_family_iterator(*this, std::move(positions));\n\n}\n\nmetric_family_iterator metrics_families_per_shard::lower_bound(const sstring& family_name) const {\n    return find_bound(family_name, [](const sstring& a, const mi::metric_family_metadata& b) {\n        //sstring doesn't have a <= operator\n        return a < b.mf.name || a == b.mf.name;\n    });\n}\n\nmetric_family_iterator metrics_families_per_shard::upper_bound(const sstring& family_name) const {\n    return find_bound(family_name, [](const sstring& a, const mi::metric_family_metadata& b) {\n        return a < b.mf.name;\n    });\n}\n\n/*!\n * \\brief a helper function to get metric family range\n * if metric_family_name is empty will return everything, if not, it will return\n * the range of metric family that match the metric_family_name.\n *\n * if prefix is true the match will be based on prefix\n */\nmetric_family_range get_range(const metrics_families_per_shard& mf, const sstring& metric_family_name, bool prefix) {\n    if (metric_family_name == \"\") {\n        return metric_family_range(mf);\n    }\n    auto upper_bount_prefix = metric_family_name;\n    ++upper_bount_prefix.back();\n    if (prefix) {\n        return metric_family_range(mf.lower_bound(metric_family_name), mf.lower_bound(upper_bount_prefix));\n    }\n    auto lb = mf.lower_bound(metric_family_name);\n    if (lb.end() || lb->name() != metric_family_name) {\n        return metric_family_range(lb, lb); // just return an empty range\n    }\n    auto up = lb;\n    ++up;\n    return metric_family_range(lb, up);\n\n}\n\n\ntemplate <typename Extra = no_label>\nstatic inline void write_series(buf_t& buf, std::string_view name, const labels_type& labels, const config& ctx, auto v, std::string_view suffix, Extra e = {}) {\n    write_name_and_labels(buf, name, suffix, labels, ctx, e);\n    static constexpr auto format = std::is_floating_point_v<decltype(v)> ? \"{:g}\\n\" : \"{}\\n\";\n    fmt::format_to(buf.back_insert_begin(), FMT_COMPILE(format), v);\n};\n\n\nstruct extra_label {\n    std::string_view name, value;\n};\n\nvoid write_histogram(buf_t& buf, const config& ctx, std::string_view name, const seastar::metrics::histogram& h, const labels_type& labels) noexcept {\n\n    auto write_one = [&] (auto v, std::string_view suffix) {\n        write_series(buf, name, labels, ctx, v, suffix);\n    };\n\n    write_one(h.sample_sum, \"_sum\");\n    write_one(h.sample_count, \"_count\");\n\n    for (auto  i : h.buckets) {\n        write_series(buf, name, labels, ctx, i.count, \"_bucket\",\n            extra_label{\"le\", fmt::format(FMT_COMPILE(\"{:f}\"), i.upper_bound)} );\n    }\n    write_series(buf, name, labels, ctx, h.sample_count, \"_bucket\", extra_label{\"le\", \"+Inf\"} );\n}\n\nvoid write_summary(buf_t& buf, const config& ctx, std::string_view name, const seastar::metrics::histogram& h, const labels_type& labels) noexcept {\n\n    auto write_one = [&] (auto v, std::string_view suffix) {\n        write_series(buf, name, labels, ctx, v, suffix);\n    };\n\n    if (h.sample_sum) {\n        write_one(h.sample_sum, \"_sum\");\n    }\n    if (h.sample_count) {\n        write_one(h.sample_count, \"_count\");\n    }\n\n    for (auto  i : h.buckets) {\n        write_series(buf, name, labels, ctx, i.count, \"\",\n            extra_label{\"quantile\", fmt::format(FMT_COMPILE(\"{:f}\"), i.upper_bound)} );\n    }\n}\n\nvoid write_value_as_string(buf_t& s, const mi::metric_value& value) noexcept {\n    try {\n        fmt::format_to(s.back_insert_begin(), FMT_COMPILE(\"{}\\n\"), value);\n    } catch (const std::range_error& e) {\n        seastar_logger.debug(\"prometheus: write_value_as_string: {}: {}\", s.str(), e.what());\n        s << \"NaN\\n\";\n    } catch (...) {\n        auto ex = std::current_exception();\n        // print this error as it's ignored later on by `connection::start_response`\n        seastar_logger.error(\"prometheus: write_value_as_string: {}: {}\", s.str(), ex);\n        std::rethrow_exception(std::move(ex));\n    }\n}\n\ndetails::family_filter_t details::make_family_filter(std::vector<details::name_filter> filters, std::string_view prefix) {\n    if (filters.empty()) {\n        return [](std::string_view) { return true; };\n    }\n    // Strip prefix from filter names if present\n    if (!prefix.empty()) {\n        auto prefix_with_underscore = sstring(prefix) + \"_\";\n        for (auto& f : filters) {\n            if (f.name.starts_with(prefix_with_underscore)) {\n                f.name = f.name.substr(prefix_with_underscore.size());\n            }\n        }\n    }\n    return [filters = std::move(filters)](std::string_view family_name) {\n        for (const auto& f : filters) {\n            bool match = f.is_prefix ? family_name.starts_with(f.name) : family_name == f.name;\n            if (match) {\n                return true;\n            }\n        }\n        return false;\n    };\n}\n\nstruct write_context {\n    output_stream<char>& out;\n    const config& ctx;\n    const metric_family_range m;\n    const write_body_args args;\n\n    future<> write_text_representation();\n    future<> write_protobuf_representation();\n};\n\nfuture<> write_context::write_text_representation() {\n    return seastar::async([this] {\n        buf_t s;\n        for (metric_family& metric_family : m) {\n            if (!args.family_filter(metric_family.name())) {\n                continue;\n            }\n            auto name = ctx.prefix + \"_\" + metric_family.name();\n            bool found = false;\n            metric_aggregate_by_labels aggregated_values(metric_family.metadata().aggregate_labels);\n            bool should_aggregate = args.enable_aggregation && !metric_family.metadata().aggregate_labels.empty();\n            metric_family.foreach_metric([this, &s, &found, &name, &metric_family, &aggregated_values, should_aggregate](const mi::metric_value& value, const mi::metric_series_metadata& value_info) mutable {\n                s.clear();\n                if ((value_info.should_skip_when_empty() && value.is_empty()) || !args.filter(value_info.labels())) {\n                    return;\n                }\n                if (!found) {\n                    if (args.show_help && metric_family.metadata().d.str() != \"\") {\n                        s << \"# HELP \" << name << \" \" <<  metric_family.metadata().d.str() << \"\\n\";\n                    }\n                    s << \"# TYPE \" << name << \" \" << to_string(metric_family.metadata().type) << \"\\n\";\n                    found = true;\n                }\n                if (should_aggregate) {\n                    aggregated_values.add(value, value_info.labels());\n                } else if (value.type() == mi::data_type::SUMMARY) {\n                    write_summary(s, ctx, name, value.get_histogram(), value_info.labels());\n                } else if (value.type() == mi::data_type::HISTOGRAM) {\n                    write_histogram(s, ctx, name, value.get_histogram(), value_info.labels());\n                } else {\n                    write_name_and_labels(s, name, \"\", value_info.labels(), ctx);\n                    write_value_as_string(s, value);\n                }\n                out.write(s.data(), s.size()).get();\n                thread::maybe_yield();\n            });\n            if (!aggregated_values.empty()) {\n                for (auto&& h : aggregated_values.get_values()) {\n                    s.clear();\n                    // Labels are already filtered (aggregated labels removed)\n                    auto& labels = h.second.labels;\n                    auto& value = h.second.m;\n                    if (value.type() == mi::data_type::HISTOGRAM) {\n                        write_histogram(s, ctx, name, value.get_histogram(), labels);\n                    } else {\n                        write_name_and_labels(s, name, \"\", labels, ctx);\n                        write_value_as_string(s, value);\n                    }\n                    out.write(s.data(), s.size()).get();\n                    thread::maybe_yield();\n                }\n            }\n        }\n    });\n}\n\nfuture<> write_context::write_protobuf_representation() {\n    return do_for_each(m, [this](metric_family& metric_family) mutable {\n        if (!args.family_filter(metric_family.name())) {\n            return make_ready_future<>();\n        }\n        std::string s;\n        google::protobuf::io::StringOutputStream os(&s);\n        metric_aggregate_by_labels aggregated_values(metric_family.metadata().aggregate_labels);\n        bool should_aggregate = args.enable_aggregation && !metric_family.metadata().aggregate_labels.empty();\n        auto& name = metric_family.name();\n        pm::MetricFamily mtf;\n        bool empty_metric = true;\n        mtf.set_name(fmt::format(\"{}_{}\", ctx.prefix, name));\n        mtf.mutable_metric()->Reserve(metric_family.size());\n        metric_family.foreach_metric([this, &mtf, &aggregated_values, &empty_metric, should_aggregate](const auto& value, const auto& value_info) {\n            if ((value_info.should_skip_when_empty() && value.is_empty()) || !args.filter(value_info.labels())) {\n                return;\n            }\n            if (should_aggregate) {\n                aggregated_values.add(value, value_info.labels());\n            } else {\n                fill_metric(mtf, value, value_info.labels(), ctx);\n                empty_metric = false;\n            }\n        });\n        for (auto& [_, value] : aggregated_values.get_values()) {\n            fill_metric(mtf, value.m, value.labels, ctx);\n            empty_metric = false;\n        }\n        if (empty_metric) {\n            return make_ready_future<>();\n        }\n        if (!write_delimited_to(mtf, &os)) {\n            seastar_logger.warn(\"Failed to write protobuf metrics\");\n        }\n        return out.write(s);\n    });\n}\n\nbool is_accept_protobuf(const std::string& accept) {\n    std::vector<std::string> strs;\n    boost::split(strs, accept, boost::is_any_of(\",\"));\n    for (auto i : strs) {\n        boost::trim(i);\n        if (boost::starts_with(i, \"application/vnd.google.protobuf;\")) {\n            return true;\n        }\n    }\n    return false;\n}\n\nclass metrics_handler : public httpd::handler_base  {\n    sstring _prefix;\n    config _ctx;\n    static std::function<bool(const mi::labels_type&)> _true_function;\n\n    /*!\n     * \\brief tries to trim an asterisk from the end of the string\n     * return true if an asterisk exists.\n     */\n    bool trim_asterisk(sstring& name) {\n        if (name.size() && name.back() == '*') {\n            name.resize(name.length() - 1);\n            return true;\n        }\n        // Prometheus uses url encoding for the path so '*' is encoded as '%2A'\n        if (boost::algorithm::ends_with(name, \"%2A\")) {\n            // This assert is obviously true. It is in here just to\n            // silence a bogus gcc warning:\n            // https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89337\n            SEASTAR_ASSERT(name.length() >= 3);\n            name.resize(name.length() - 3);\n            return true;\n        }\n        return false;\n    }\n    /*!\n     * \\brief Return a filter function, based on the request\n     *\n     * A filter function filter what metrics should be included.\n     * It returns true if a metric should be included, or false otherwise.\n     * The filters are created from the request query parameters.\n     */\n    std::function<bool(const mi::labels_type&)> make_filter(const http::request& req) {\n        std::unordered_map<sstring, std::regex> matcher;\n        auto labels = mi::get_local_impl()->get_labels();\n        for (auto&& qp : req.get_query_params()) {\n            if (labels.find(qp.first) != labels.end()) {\n                matcher.emplace(qp.first, std::regex(qp.second.back().c_str()));\n            }\n        }\n        return (matcher.empty()) ? _true_function : [matcher](const mi::labels_type& labels) {\n            for (auto&& m : matcher) {\n                auto l = labels.find(m.first);\n                if (!std::regex_match((l == labels.end())? \"\" : l->second.value().c_str(), m.second)) {\n                    return false;\n                }\n            }\n            return true;\n        };\n    }\n\npublic:\n    metrics_handler(config ctx) : _ctx(ctx) {}\n\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n        std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        // Build name filters from all __name__ query parameters\n        std::vector<details::name_filter> name_filters;\n        for (auto name : req->get_query_param_array(\"__name__\")) {\n            if (!name.empty()) {\n                bool is_prefix = trim_asterisk(name);\n                name_filters.push_back({std::move(name), is_prefix});\n            }\n        }\n        write_body_args args{\n            .filter = make_filter(*req),\n            .family_filter = details::make_family_filter(std::move(name_filters), _ctx.prefix),\n            .use_protobuf_format = _ctx.allow_protobuf && is_accept_protobuf(req->get_header(\"Accept\")),\n            .show_help = req->get_query_param(\"__help__\") != \"false\",\n            .enable_aggregation = req->get_query_param(\"__aggregate__\") != \"false\"\n        };\n        rep->write_body(args.use_protobuf_format ? \"proto\" : \"txt\", [this, args = std::move(args)](output_stream<char>&& s) {\n            return write_body(std::move(args), std::move(s));\n        });\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n\nprivate:\n\n    future<> write_body(write_body_args args, output_stream<char>&& out_stream) {\n        auto s = std::move(out_stream);\n        auto families = co_await get_map_value();\n        bool use_protobuf = args.use_protobuf_format;\n\n        write_context context{\n            .out = s,\n            .ctx = _ctx,\n            .m = metric_family_range(families),\n            .args = std::move(args)\n        };\n\n        co_return co_await (use_protobuf ? context.write_protobuf_representation() : context.write_text_representation())\n                .finally([&s] { return s.close(); });\n    }\n\n    friend details::test_access;\n};\n\nfuture<> details::test_access::write_body(config cfg, write_body_args args, output_stream<char>&& s) {\n    metrics_handler handler(std::move(cfg));\n    co_return co_await handler.write_body(std::move(args), std::move(s));\n}\n\nstd::function<bool(const mi::labels_type&)> metrics_handler::_true_function = [](const mi::labels_type&) {\n    return true;\n};\n\nfuture<> add_prometheus_routes(httpd::http_server& server, config ctx) {\n    server._routes.put(httpd::GET, \"/metrics\", new metrics_handler(ctx));\n    return make_ready_future<>();\n}\n\nfuture<> add_prometheus_routes(sharded<httpd::http_server>& server, config ctx) {\n    return server.invoke_on_all([ctx](httpd::http_server& s) {\n        return add_prometheus_routes(s, ctx);\n    });\n}\n\nfuture<> start(httpd::http_server_control& http_server, config ctx) {\n    return add_prometheus_routes(http_server.server(), ctx);\n}\n\n}\n}\n"
  },
  {
    "path": "src/core/reactor.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#include <atomic>\n#include <chrono>\n#include <cmath>\n#include <coroutine>\n#include <exception>\n#include <filesystem>\n#include <fstream>\n#include <ranges>\n#include <regex>\n#include <thread>\n#include <unordered_set>\n#include <barrier>\n\n#include <grp.h>\n#include <spawn.h>\n#include <sys/syscall.h>\n#include <sys/vfs.h>\n#include <sys/statfs.h>\n#include <sys/statvfs.h>\n#include <sys/time.h>\n#include <sys/resource.h>\n#include <sys/socket.h>\n#include <sys/inotify.h>\n#include <sys/wait.h>\n#include <unistd.h>\n#include <fcntl.h>\n#include <sys/eventfd.h>\n#include <poll.h>\n#include <netinet/in.h>\n#include <boost/lexical_cast.hpp>\n#include <boost/algorithm/string/classification.hpp>\n#include <boost/algorithm/string/constants.hpp>\n#include <boost/algorithm/string/find_iterator.hpp>\n#include <boost/algorithm/string/finder.hpp>\n#include <boost/algorithm/string/split.hpp>\n#include <boost/container/small_vector.hpp>\n#include <boost/iterator/counting_iterator.hpp>\n#include <boost/intrusive/list.hpp>\n#include <boost/range/adaptor/transformed.hpp>\n#include <boost/range/numeric.hpp>\n#include <boost/range/algorithm/sort.hpp>\n#include <boost/range/algorithm/remove_if.hpp>\n#include <boost/range/algorithm/find_if.hpp>\n#include <boost/algorithm/clamp.hpp>\n#include <boost/version.hpp>\n#define __user /* empty */  // for xfs includes, below\n#include <linux/types.h> // for xfs, below\n#include <sys/ioctl.h>\n#include <linux/perf_event.h>\n#include <xfs/linux.h>\n/*\n * With package xfsprogs-devel >= 5.14.1, `fallthrough` has defined to\n * fix compilation warning in header <xfs/linux.h>,\n * (see: https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/commit/?id=df9c7d8d8f3ed0785ed83e7fd0c7ddc92cbfbe15)\n * There is a confliction with c++ keyword `fallthrough`, so undefine fallthrough here.\n */\n#undef fallthrough\n#define min min    /* prevent xfs.h from defining min() as a macro */\n#include <xfs/xfs.h>\n#undef min\n#include <fmt/ostream.h>\n#include <fmt/ranges.h>\n\n#ifdef SEASTAR_SHUFFLE_TASK_QUEUE\n#include <random>\n#endif\n\n#include <sys/mman.h>\n#include <sys/utsname.h>\n#include <linux/falloc.h>\n#ifdef SEASTAR_HAVE_SYSTEMTAP_SDT\n#include <sys/sdt.h>\n#else\n#define STAP_PROBE(provider, name)\n#endif\n\n#if defined(__x86_64__) || defined(__i386__)\n#include <xmmintrin.h>\n#endif\n\n#ifdef SEASTAR_HAVE_DPDK\n#include <rte_lcore.h>\n#include <rte_launch.h>\n#endif\n#ifdef __GNUC__\n#include <iostream>\n#include <system_error>\n#include <cxxabi.h>\n#endif\n\n#include <yaml-cpp/yaml.h>\n\n#ifdef SEASTAR_TASK_HISTOGRAM\n#include <typeinfo>\n#endif\n\n#include <seastar/core/abort_on_ebadf.hh>\n#include <seastar/core/alien.hh>\n#include <seastar/core/exception_hacks.hh>\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/make_task.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/prefetch.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/report_exception.hh>\n#include <seastar/core/resource.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/scheduling_specific.hh>\n#include <seastar/core/signal.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/smp_options.hh>\n#include <seastar/core/stall_sampler.hh>\n#include <seastar/core/task.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/thread_cputime_clock.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/internal/buffer_allocator.hh>\n#include <seastar/core/disk_params.hh>\n#include <seastar/core/internal/io_desc.hh>\n#include <seastar/core/internal/uname.hh>\n#include <seastar/core/internal/stall_detector.hh>\n#include <seastar/core/internal/run_in_background.hh>\n#include <seastar/coroutine/all.hh>\n#include <seastar/net/native-stack.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/net/posix-stack.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/conversions.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/memory_diagnostics.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/print_safe.hh>\n#include <seastar/util/process.hh>\n#include <seastar/util/read_first_line.hh>\n#include <seastar/util/spinlock.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n#include <seastar/util/internal/magic.hh>\n#include \"core/reactor_backend.hh\"\n#include \"core/syscall_result.hh\"\n#include \"core/thread_pool.hh\"\n#include \"syscall_work_queue.hh\"\n#include \"cgroup.hh\"\n#ifdef SEASTAR_HAVE_DPDK\n#include <seastar/core/dpdk_rte.hh>\n#endif\n#include <seastar/util/assert.hh>\n#include <seastar/core/internal/systemwide_memory_barrier.hh>\n\nnamespace std {\n    template <> struct hash<std::pair<std::string_view, int>> {\n        size_t operator () (std::pair<std::string_view, int> v) const {\n            return std::hash<std::string_view>()(v.first) * 1009 + v.second;\n        }\n    };\n}\nnamespace seastar {\n\nstatic_assert(posix::shutdown_mask(SHUT_RD) == posix::rcv_shutdown);\nstatic_assert(posix::shutdown_mask(SHUT_WR) == posix::snd_shutdown);\nstatic_assert(posix::shutdown_mask(SHUT_RDWR) == (posix::snd_shutdown | posix::rcv_shutdown));\n\n}\n\nnamespace seastar {\n\nseastar::logger seastar_logger(\"seastar\");\n\nshard_id reactor::cpu_id() const {\n    SEASTAR_ASSERT(_id == this_shard_id());\n    return _id;\n}\n\nvoid reactor::update_shares_for_queues(internal::priority_class pc, uint32_t shares) {\n    for (auto&& q : _io_queues) {\n        q.second->update_shares_for_class(pc, shares);\n    }\n}\n\nvoid reactor::update_group_shares_for_queues(unsigned index, uint32_t shares) {\n    for (auto&& q : _io_queues) {\n        q.second->update_shares_for_class_group(index, shares);\n    }\n}\n\nfuture<> reactor::update_bandwidth_for_queues(internal::priority_class pc, uint64_t bandwidth) {\n    return smp::invoke_on_all([pc, bandwidth = bandwidth / _num_io_groups] {\n        return parallel_for_each(engine()._io_queues, [pc, bandwidth] (auto& queue) {\n            return queue.second->update_bandwidth_for_class(pc, bandwidth);\n        });\n    });\n}\n\nfuture<> reactor::update_bandwidth_for_queues(unsigned group_index, uint64_t bandwidth) {\n    return smp::invoke_on_all([group_index, bandwidth = bandwidth / _num_io_groups] {\n        return parallel_for_each(engine()._io_queues, [group_index, bandwidth] (auto& queue) {\n            return queue.second->update_bandwidth_for_class_group(group_index, bandwidth);\n        });\n    });\n}\n\nvoid reactor::rename_queues(internal::priority_class pc, sstring new_name) {\n    for (auto&& queue : _io_queues) {\n        queue.second->rename_priority_class(pc, new_name);\n    }\n}\n\nfuture<std::tuple<pollable_fd, socket_address>>\nreactor::do_accept(pollable_fd_state& listenfd) {\n    return readable_or_writeable(listenfd).then([this, &listenfd] () mutable {\n        socket_address sa;\n        listenfd.maybe_no_more_recv();\n        auto maybe_fd = listenfd.fd.try_accept(sa, SOCK_NONBLOCK | SOCK_CLOEXEC);\n        if (!maybe_fd) {\n            // We speculated that we will have an another connection, but got a false\n            // positive. Try again without speculation.\n            return do_accept(listenfd);\n        }\n        // Speculate that there is another connection on this listening socket, to avoid\n        // a task-quota delay. Usually this will fail, but accept is a rare-enough operation\n        // that it is worth the false positive in order to withstand a connection storm\n        // without having to accept at a rate of 1 per task quota.\n        listenfd.speculate_epoll(EPOLLIN);\n        pollable_fd pfd(std::move(*maybe_fd), pollable_fd::speculation(EPOLLOUT));\n        return make_ready_future<std::tuple<pollable_fd, socket_address>>(std::make_tuple(std::move(pfd), std::move(sa)));\n    });\n}\n\nfuture<> reactor::do_connect(pollable_fd_state& pfd, socket_address& sa) {\n    pfd.fd.connect(sa.u.sa, sa.length());\n    return pfd.writeable().then([&pfd]() mutable {\n        auto err = pfd.fd.getsockopt<int>(SOL_SOCKET, SO_ERROR);\n        if (err != 0) {\n            throw std::system_error(err, std::system_category());\n        }\n        return make_ready_future<>();\n    });\n}\n\nfuture<size_t>\nreactor::do_read(pollable_fd_state& fd, void* buffer, size_t len) {\n    return readable(fd).then([this, &fd, buffer, len] () mutable {\n        auto r = fd.fd.read(buffer, len);\n        if (!r) {\n            return do_read(fd, buffer, len);\n        }\n        if (size_t(*r) == len) {\n            fd.speculate_epoll(EPOLLIN);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n\nfuture<temporary_buffer<char>>\nreactor::do_read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return fd.readable().then([this, &fd, ba] {\n        auto buffer = ba->allocate_buffer();\n        auto r = fd.fd.read(buffer.get_write(), buffer.size());\n        if (!r) {\n            // Speculation failure, try again with real polling this time\n            // Note we release the buffer and will reallocate it when poll\n            // completes.\n            return do_read_some(fd, ba);\n        }\n        if (size_t(*r) == buffer.size()) {\n            fd.speculate_epoll(EPOLLIN);\n        }\n        buffer.trim(*r);\n        return make_ready_future<temporary_buffer<char>>(std::move(buffer));\n    });\n}\n\nfuture<size_t>\nreactor::do_recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) {\n    return readable(fd).then([this, &fd, iov = iov] () mutable {\n        ::msghdr mh = {};\n        mh.msg_iov = &iov[0];\n        mh.msg_iovlen = iov.size();\n        auto r = fd.fd.recvmsg(&mh, 0);\n        if (!r) {\n            return do_recvmsg(fd, iov);\n        }\n        if (size_t(*r) == internal::iovec_len(iov)) {\n            fd.speculate_epoll(EPOLLIN);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n\n#if SEASTAR_API_LEVEL < 9\nfuture<size_t>\nreactor::do_send(pollable_fd_state& fd, const void* buffer, size_t len) {\n    return writeable(fd).then([this, &fd, buffer, len] () mutable {\n        auto r = fd.fd.send(buffer, len, MSG_NOSIGNAL);\n        if (!r) {\n            return do_send(fd, buffer, len);\n        }\n        if (size_t(*r) == len) {\n            fd.speculate_epoll(EPOLLOUT);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n#endif\n\nfuture<size_t>\nreactor::do_sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) {\n    return writeable(fd).then([this, &fd, iovs, len] () mutable {\n        msghdr mh = {};\n        mh.msg_iov = iovs.data();\n        mh.msg_iovlen = std::min<size_t>(iovs.size(), IOV_MAX);\n        auto r = fd.fd.sendmsg(&mh, MSG_NOSIGNAL);\n        if (!r) {\n            return do_sendmsg(fd, iovs, len);\n        }\n        if (size_t(*r) == len) {\n            fd.speculate_epoll(EPOLLOUT);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n\n#if SEASTAR_API_LEVEL < 9\nfuture<>\nreactor::send_all_part(pollable_fd_state& fd, const void* buffer, size_t len, size_t completed) {\n    if (completed == len) {\n        return make_ready_future<>();\n    } else {\n        return _backend->send(fd, static_cast<const char*>(buffer) + completed, len - completed).then(\n                [&fd, buffer, len, completed, this] (size_t part) mutable {\n            return send_all_part(fd, buffer, len, completed + part);\n        });\n    }\n}\n#endif\n\nfuture<temporary_buffer<char>>\nreactor::do_recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return fd.readable().then([this, &fd, ba] {\n        auto buffer = ba->allocate_buffer();\n        auto r = fd.fd.recv(buffer.get_write(), buffer.size(), MSG_DONTWAIT);\n        if (!r) {\n            return do_recv_some(fd, ba);\n        }\n        if (size_t(*r) == buffer.size()) {\n            fd.speculate_epoll(EPOLLIN);\n        }\n        buffer.trim(*r);\n        return make_ready_future<temporary_buffer<char>>(std::move(buffer));\n    });\n}\n\n#if SEASTAR_API_LEVEL < 9\nfuture<>\nreactor::send_all(pollable_fd_state& fd, const void* buffer, size_t len) {\n    SEASTAR_ASSERT(len);\n    return send_all_part(fd, buffer, len, 0);\n}\n#endif\n\nfuture<size_t> pollable_fd_state::read_some(char* buffer, size_t size) {\n    return engine()._backend->read(*this, buffer, size);\n}\n\nfuture<size_t> pollable_fd_state::read_some(uint8_t* buffer, size_t size) {\n    return engine()._backend->read(*this, buffer, size);\n}\n\nfuture<size_t> pollable_fd_state::read_some(const std::vector<iovec>& iov) {\n    return engine()._backend->recvmsg(*this, iov);\n}\n\nfuture<temporary_buffer<char>> pollable_fd_state::read_some(internal::buffer_allocator* ba) {\n    return engine()._backend->read_some(*this, ba);\n}\n\n#if SEASTAR_API_LEVEL >= 9\nfuture<size_t> pollable_fd_state::write_some(std::span<iovec> iovs) {\n    return engine()._backend->sendmsg(*this, iovs, internal::iovec_len(iovs));\n}\n#else\nfuture<size_t> pollable_fd_state::write_some(net::packet& p) {\n    static_assert(offsetof(iovec, iov_base) == offsetof(net::fragment, base) &&\n        sizeof(iovec::iov_base) == sizeof(net::fragment::base) &&\n        offsetof(iovec, iov_len) == offsetof(net::fragment, size) &&\n        sizeof(iovec::iov_len) == sizeof(net::fragment::size) &&\n        alignof(iovec) == alignof(net::fragment) &&\n        sizeof(iovec) == sizeof(net::fragment)\n        , \"net::fragment and iovec should be equivalent\");\n\n    auto fragments = p.fragments();\n    auto iovecs = std::span(reinterpret_cast<iovec*>(fragments._start), fragments._finish - fragments._start);\n    return engine()._backend->sendmsg(*this, iovecs, p.len());\n}\n#endif\n\n#if SEASTAR_API_LEVEL >= 9\nfuture<> pollable_fd_state::write_all(std::span<iovec> iovs) {\n    return write_some(iovs).then([this, iovs] (size_t size) {\n        auto niovs = internal::iovec_trim_front(iovs, size);\n        return niovs.empty() ? make_ready_future<>() : write_all(niovs);\n    });\n}\n#else\nfuture<> pollable_fd_state::write_all(net::packet& p) {\n    return write_some(p).then([this, &p] (size_t size) {\n        if (p.len() == size) {\n            return make_ready_future<>();\n        }\n        p.trim_front(size);\n        return write_all(p);\n    });\n}\n\nfuture<> pollable_fd_state::write_all(const char* buffer, size_t size) {\n    return engine().send_all(*this, buffer, size);\n}\n\nfuture<> pollable_fd_state::write_all(const uint8_t* buffer, size_t size) {\n    return engine().send_all(*this, buffer, size);\n}\n#endif\n\nfuture<> pollable_fd_state::readable() {\n    return engine().readable(*this);\n}\n\nfuture<> pollable_fd_state::writeable() {\n    return engine().writeable(*this);\n}\n\nfuture<> pollable_fd_state::poll_rdhup() {\n    return engine().poll_rdhup(*this);\n}\n\nfuture<> pollable_fd_state::readable_or_writeable() {\n    return engine().readable_or_writeable(*this);\n}\n\nfuture<std::tuple<pollable_fd, socket_address>> pollable_fd_state::accept() {\n    return engine()._backend->accept(*this);\n}\n\nfuture<> pollable_fd_state::connect(socket_address& sa) {\n    return engine()._backend->connect(*this, sa);\n}\n\nfuture<temporary_buffer<char>> pollable_fd_state::recv_some(internal::buffer_allocator* ba) {\n    maybe_no_more_recv();\n    return engine()._backend->recv_some(*this, ba);\n}\n\nfuture<size_t> pollable_fd_state::recvmsg(struct msghdr *msg) {\n    maybe_no_more_recv();\n    return engine().readable(*this).then([this, msg] {\n        auto r = fd.recvmsg(msg, 0);\n        if (!r) {\n            return recvmsg(msg);\n        }\n        // We always speculate here to optimize for throughput in a workload\n        // with multiple outstanding requests. This way the caller can consume\n        // all messages without resorting to epoll. However this adds extra\n        // recvmsg() call when we hit the empty queue condition, so it may\n        // hurt request-response workload in which the queue is empty when we\n        // initially enter recvmsg(). If that turns out to be a problem, we can\n        // improve speculation by using recvmmsg().\n        speculate_epoll(EPOLLIN);\n        return make_ready_future<size_t>(*r);\n    });\n}\n\nfuture<size_t> pollable_fd_state::sendmsg(struct msghdr* msg) {\n    maybe_no_more_send();\n    return engine().writeable(*this).then([this, msg] () mutable {\n        auto r = fd.sendmsg(msg, 0);\n        if (!r) {\n            return sendmsg(msg);\n        }\n        // For UDP this will always speculate. We can't know if there's room\n        // or not, but most of the time there should be so the cost of mis-\n        // speculation is amortized.\n        if (size_t(*r) == internal::iovec_len(msg->msg_iov, msg->msg_iovlen)) {\n            speculate_epoll(EPOLLOUT);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n\nfuture<size_t> pollable_fd_state::sendto(socket_address addr, const void* buf, size_t len) {\n    maybe_no_more_send();\n    return engine().writeable(*this).then([this, buf, len, addr] () mutable {\n        auto r = fd.sendto(addr, buf, len, 0);\n        if (!r) {\n            return sendto(std::move(addr), buf, len);\n        }\n        // See the comment about speculation in sendmsg().\n        if (size_t(*r) == len) {\n            speculate_epoll(EPOLLOUT);\n        }\n        return make_ready_future<size_t>(*r);\n    });\n}\n\nnamespace internal {\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nconst preemption_monitor*& get_need_preempt_var() {\n    static preemption_monitor bootstrap_preemption_monitor;\n    static thread_local const preemption_monitor* g_need_preempt = &bootstrap_preemption_monitor;\n    return g_need_preempt;\n}\n#endif\n\nvoid set_need_preempt_var(const preemption_monitor* np) {\n    get_need_preempt_var() = np;\n}\n\n#ifdef SEASTAR_TASK_HISTOGRAM\n\nclass task_histogram {\n    static constexpr unsigned max_countdown = 1'000'000;\n    std::unordered_map<std::type_index, uint64_t> _histogram;\n    unsigned _countdown_to_print = max_countdown;\npublic:\n    void add(const task& t) {\n        ++_histogram[std::type_index(typeid(t))];\n        if (!--_countdown_to_print) {\n            print();\n            _countdown_to_print = max_countdown;\n            _histogram.clear();\n        }\n    }\n    void print() const {\n        seastar::fmt::print(\"task histogram, {:d} task types {:d} tasks\\n\", _histogram.size(), max_countdown - _countdown_to_print);\n        for (auto&& type_count : _histogram) {\n            auto&& type = type_count.first;\n            auto&& count = type_count.second;\n            seastar::fmt::print(\"  {:10d} {}\\n\", count, type.name());\n        }\n    }\n};\n\nthread_local task_histogram this_thread_task_histogram;\n\nvoid task_histogram_add_task(const task& t) {\n    this_thread_task_histogram.add(t);\n}\n#else\nvoid task_histogram_add_task(const task& t) {\n}\n#endif\n\nscheduling_supergroup scheduling_supergroup_for(scheduling_group sg) noexcept {\n    auto& q = engine()._task_queues[internal::scheduling_group_index(sg)];\n    auto& supergroups = engine()._supergroups;\n    for (unsigned i = 0; i < supergroups.size(); i++) {\n        if (supergroups[i].get() == q->_parent) {\n            return scheduling_supergroup(i);\n        }\n    }\n\n    return scheduling_supergroup();\n}\n\n}\n\nusing namespace std::chrono_literals;\nnamespace fs = std::filesystem;\n\nusing namespace net;\n\nusing namespace internal::linux_abi;\n\nstd::atomic<manual_clock::rep> manual_clock::_now;\n\nstatic std::atomic<bool> abort_on_ebadf = { false };\n\nvoid set_abort_on_ebadf(bool do_abort) {\n    abort_on_ebadf.store(do_abort);\n}\n\nbool is_abort_on_ebadf_enabled() {\n    return abort_on_ebadf.load();\n}\n\nvoid lowres_clock::update() noexcept {\n    lowres_clock::_now = lowres_clock::time_point(std::chrono::steady_clock::now().time_since_epoch());\n    lowres_system_clock::_now = lowres_system_clock::time_point(std::chrono::system_clock::now().time_since_epoch());\n}\n\ntemplate <typename Clock>\ninline\ntimer<Clock>::~timer() {\n    if (_queued) {\n        engine().del_timer(this);\n    }\n}\n\ntemplate <typename Clock>\ninline\nvoid timer<Clock>::arm(time_point until, std::optional<duration> period) noexcept {\n    arm_state(until, period);\n    engine().add_timer(this);\n}\n\ntemplate <typename Clock>\ninline\nvoid timer<Clock>::readd_periodic() noexcept {\n    arm_state(Clock::now() + _period.value(), {_period.value()});\n    engine().queue_timer(this);\n}\n\ntemplate <typename Clock>\ninline\nbool timer<Clock>::cancel() noexcept {\n    if (!_armed) {\n        return false;\n    }\n    _armed = false;\n    if (_queued) {\n        engine().del_timer(this);\n        _queued = false;\n    }\n    return true;\n}\n\ntemplate class timer<steady_clock_type>;\ntemplate class timer<lowres_clock>;\ntemplate class timer<manual_clock>;\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nthread_local lowres_clock::time_point lowres_clock::_now;\nthread_local lowres_system_clock::time_point lowres_system_clock::_now;\n#endif\n\nreactor::signals::signals() : _pending_signals(0) {\n}\n\nreactor::signals::~signals() {\n    sigset_t mask;\n    sigfillset(&mask);\n    ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n}\n\nreactor::signals::signal_handler::signal_handler(int signo, noncopyable_function<void ()>&& handler)\n        : _handler(std::move(handler)) {\n}\n\nvoid\nreactor::signals::handle_signal(int signo, noncopyable_function<void ()>&& handler) {\n    signal_handler h(signo, std::move(handler));\n    auto [_, inserted] =  _signal_handlers.insert_or_assign(signo, std::move(h));\n    if (!inserted) {\n        // since we register the same handler to OS for all signals, we could\n        // skip sigaction when a handler has already been registered before.\n        return;\n    }\n\n    struct sigaction sa;\n    sa.sa_sigaction = [](int sig, siginfo_t *info, void *p) {\n        engine()._backend->signal_received(sig, info, p);\n    };\n    sa.sa_mask = make_empty_sigset_mask();\n    sa.sa_flags = SA_SIGINFO | SA_RESTART;\n    auto r = ::sigaction(signo, &sa, nullptr);\n    throw_system_error_on(r == -1);\n    auto mask = make_sigset_mask(signo);\n    r = ::pthread_sigmask(SIG_UNBLOCK, &mask, NULL);\n    throw_pthread_error(r);\n}\n\nvoid\nreactor::signals::handle_signal_once(int signo, noncopyable_function<void ()>&& handler) {\n    return handle_signal(signo, [fired = false, handler = std::move(handler)] () mutable {\n        if (!fired) {\n            fired = true;\n            handler();\n        }\n    });\n}\n\nbool reactor::signals::poll_signal() {\n    auto signals = _pending_signals.load(std::memory_order_relaxed);\n    if (signals) {\n        _pending_signals.fetch_and(~signals, std::memory_order_relaxed);\n        for (size_t i = 0; i < sizeof(signals)*8; i++) {\n            if (signals & (1ull << i)) {\n               _signal_handlers.at(i)._handler();\n            }\n        }\n    }\n    return signals;\n}\n\nbool reactor::signals::pure_poll_signal() const {\n    return _pending_signals.load(std::memory_order_relaxed);\n}\n\nvoid reactor::signals::action(int signo, siginfo_t* siginfo, void* ignore) {\n    engine().start_handling_signal();\n    engine()._signals._pending_signals.fetch_or(1ull << signo, std::memory_order_relaxed);\n}\n\nvoid reactor::signals::failed_to_handle(int signo) {\n    char tname[64];\n    pthread_getname_np(pthread_self(), tname, sizeof(tname));\n    auto tid = syscall(SYS_gettid);\n    seastar_logger.error(\"Failed to handle signal {} on thread {} ({}): engine not ready\", signo, tid, tname);\n}\n\nvoid reactor::handle_signal(int signo, noncopyable_function<void ()>&& handler) {\n    _signals.handle_signal(signo, std::move(handler));\n}\n\n// Accumulates an in-memory backtrace and flush to stderr eventually.\n// Async-signal safe.\nclass backtrace_buffer {\n    static constexpr unsigned _max_size = 8 << 10;\n    unsigned _pos = 0;\n    char _buf[_max_size];\n    bool _immediate;\npublic:\n    backtrace_buffer(bool immediate = false)\n        : _immediate{immediate} {}\n\n    ~backtrace_buffer() {\n        flush();\n    }\n\n    // copying / moving 2^13 bytes not useful, compiler will help us here\n    backtrace_buffer(const backtrace_buffer &) = delete;\n    backtrace_buffer(backtrace_buffer &&) = delete;\n    backtrace_buffer &operator = (const backtrace_buffer &) = delete;\n    backtrace_buffer &operator = (backtrace_buffer &&) = delete;\n\n    void flush() noexcept {\n        if (!_immediate && _pos > 0) {\n            print_safe(_buf, _pos);\n            _pos = 0;\n        }\n    }\n\n    void reserve(size_t len) noexcept {\n        if (_immediate) {\n            return;\n        }\n        SEASTAR_ASSERT(len < _max_size);\n        if (_pos + len >= _max_size) {\n            flush();\n        }\n    }\n\n    void append(const char* str, size_t len) noexcept {\n        if (_immediate) {\n            print_safe(str, len);\n        } else {\n            reserve(len);\n            memcpy(_buf + _pos, str, len);\n            _pos += len;\n        }\n    }\n\n    void append(const char* str) noexcept { append(str, strlen(str)); }\n\n    template <typename Integral>\n    void append_decimal(Integral n) noexcept {\n        char buf[sizeof(n) * 3];\n        auto len = convert_decimal_safe(buf, sizeof(buf), n);\n        append(buf, len);\n    }\n\n    template <typename Integral>\n    void append_hex(Integral ptr) noexcept {\n        char buf[sizeof(ptr) * 2];\n        auto p = convert_hex_safe(buf, sizeof(buf), ptr);\n        append(p, (buf + sizeof(buf)) - p);\n    }\n\n    void append_backtrace() noexcept {\n        backtrace([this] (frame f) {\n            append(\"  \");\n            if (!f.so->name.empty()) {\n                append(f.so->name.c_str(), f.so->name.size());\n                append(\"+\");\n            }\n\n            append(\"0x\");\n            append_hex(f.addr);\n            append(\"\\n\");\n        }, _immediate);\n    }\n\n    void append_backtrace_oneline() noexcept {\n        backtrace([this] (frame f) noexcept {\n            reserve(3 + sizeof(f.addr) * 2 + (f.so->name.empty() ? 0 : f.so->name.size() + 1));\n            append(\" \");\n            if (!f.so->name.empty()) {\n                append(f.so->name.c_str(), f.so->name.size());\n                append(\"+\");\n            }\n            append(\"0x\");\n            append_hex(f.addr);\n        }, _immediate);\n    }\n};\n\nstatic void print_with_backtrace(backtrace_buffer& buf, bool oneline) noexcept {\n    if (local_engine) {\n        buf.append(\" on shard \");\n        buf.append_decimal(this_shard_id());\n\n        buf.append(\", in scheduling group \");\n        buf.append(current_scheduling_group().name().c_str());\n    }\n\n  if (!oneline) {\n    buf.append(\".\\nBacktrace:\\n\");\n    buf.append_backtrace();\n  } else {\n    buf.append(\". Backtrace:\");\n    buf.append_backtrace_oneline();\n    buf.append(\"\\n\");\n  }\n}\n\n// Print the current backtrace to stdout with the given cause.\n// If oneline is true, backtrace is printed entirely on one line,\n// otherwise it is printed with 1 line per frame.\n// If immediate is true, the backtrace is printed frame by frame\n// with a call to write(2), otherwise it is printed in a single\n// call to write(2). The former strategy is more robust in cases\n// where the backtrace itself may crash at some point down the stack\n// while the latter is more efficient and avoids splitting output\n// in the face of concurrent logging by other shards.\nstatic void print_with_backtrace(const char* cause, bool oneline = false, bool immediate = false) noexcept {\n    backtrace_buffer buf(immediate);\n    buf.append(cause);\n    print_with_backtrace(buf, oneline);\n}\n\n#ifndef SEASTAR_ASAN_ENABLED\n// Installs signal handler stack for current thread.\n// The stack remains installed as long as the returned object is kept alive.\n// When it goes out of scope the previous handler is restored.\nstatic decltype(auto) install_signal_handler_stack() {\n    size_t size = SIGSTKSZ;\n    auto mem = std::make_unique<char[]>(size);\n    stack_t stack;\n    stack_t prev_stack;\n    stack.ss_sp = mem.get();\n    stack.ss_flags = 0;\n    stack.ss_size = size;\n    auto r = sigaltstack(&stack, &prev_stack);\n    throw_system_error_on(r == -1);\n    return defer([mem = std::move(mem), prev_stack] () mutable noexcept {\n        try {\n            auto r = sigaltstack(&prev_stack, NULL);\n            throw_system_error_on(r == -1);\n        } catch (...) {\n            mem.release(); // We failed to restore previous stack, must leak it.\n            seastar_logger.error(\"Failed to restore signal stack: {}\", std::current_exception());\n        }\n    });\n}\n#else\n// SIGSTKSZ is too small when using asan. We also don't need to\n// handle SIGSEGV ourselves when using asan, so just don't install\n// a signal handler stack.\nauto install_signal_handler_stack() {\n    struct nothing { ~nothing() {} };\n    return nothing{};\n}\n#endif\n\nstatic sstring shorten_name(const sstring& name, size_t length) {\n    SEASTAR_ASSERT(!name.empty());\n    SEASTAR_ASSERT(length > 0);\n\n    namespace ba = boost::algorithm;\n    using split_iter_t = ba::split_iterator<sstring::const_iterator>;\n    static constexpr auto delimiter = \"_\";\n\n    auto shortname = uninitialized_string(length);\n    auto output = shortname.begin();\n    auto last = shortname.end();\n    if (name.find(delimiter) == name.npos) {\n        // use the prefix as the fallback, if the name is not underscore\n        // delimited\n        size_t n = std::min(length, name.size());\n        output = std::copy_n(name.begin(), n, output);\n    } else {\n        for (split_iter_t split_it = ba::make_split_iterator(\n                 name,\n                 ba::token_finder(ba::is_any_of(\"_\"),\n                                  ba::token_compress_on)), split_last{};\n             output != last && split_it != split_last;\n             ++split_it) {\n            auto& part = *split_it;\n            SEASTAR_ASSERT(part.size() > 0);\n            // convert \"hello_world\" to \"hw\"\n            *output++ = part[0];\n        }\n    }\n    // pad the remaining part with spaces, so the shortened name is always\n    // of length long.\n    std::fill(output, last, ' ');\n    return shortname;\n}\n\nreactor::task_queue_group::task_queue_group(task_queue_group* p, float shares)\n        : sched_entity(p, shares)\n{\n    if (p == nullptr) {\n        sched_entity::_active = true;\n    }\n}\n\nreactor::sched_entity::sched_entity(task_queue_group* p, float shares)\n        : _shares(std::max(shares, 1.0f))\n        , _reciprocal_shares_times_2_power_32((uint64_t(1) << 32) / _shares)\n        , _parent(p)\n        , _ts(now())\n{\n    if (_parent != nullptr) {\n        if (_parent->_nr_children >= max_scheduling_groups()) {\n            on_fatal_internal_error(seastar_logger, \"Attempted to create too many supergroups\");\n        }\n        _parent->_nr_children++;\n    }\n}\n\nreactor::sched_entity::~sched_entity() {\n    if (_parent != nullptr) {\n        _parent->_nr_children--;\n    }\n}\n\nreactor::task_queue::task_queue(task_queue_group* parent, unsigned id, sstring name, sstring shortname, float shares)\n        : sched_entity(parent, shares)\n        , _id(id)\n{\n    rename(name, shortname);\n}\n\nvoid\nreactor::task_queue::register_stats() {\n    seastar::metrics::metric_groups new_metrics;\n    namespace sm = seastar::metrics;\n    static auto group = sm::label(\"group\");\n    auto group_label = group(_name);\n    new_metrics.add_group(\"scheduler\", {\n        sm::make_counter(\"runtime_ms\", [this] {\n            return std::chrono::duration_cast<std::chrono::milliseconds>(_runtime).count();\n        }, sm::description(\"Accumulated runtime of this task queue; an increment rate of 1000ms per second indicates full utilization\"),\n            {group_label}),\n        sm::make_counter(\"waittime_ms\", [this] {\n            return std::chrono::duration_cast<std::chrono::milliseconds>(_waittime).count();\n        }, sm::description(\"Accumulated waittime of this task queue; an increment rate of 1000ms per second indicates queue is waiting for something (e.g. IO)\"),\n            {group_label}),\n        sm::make_counter(\"starvetime_ms\", [this] {\n            return std::chrono::duration_cast<std::chrono::milliseconds>(_starvetime).count();\n        }, sm::description(\"Accumulated starvation time of this task queue; an increment rate of 1000ms per second indicates the scheduler feels really bad\"),\n            {group_label}),\n        sm::make_counter(\"tasks_processed\", _tasks_processed,\n                sm::description(\"Count of tasks executing on this queue; indicates together with runtime_ms indicates length of tasks\"),\n                {group_label}),\n        sm::make_gauge(\"queue_length\", [this] { return _q.size(); },\n                sm::description(\"Size of backlog on this queue, in tasks; indicates whether the queue is busy and/or contended\"),\n                {group_label}),\n        sm::make_gauge(\"shares\", [this] { return _shares; },\n                sm::description(\"Shares allocated to this queue\"),\n                {group_label}),\n        sm::make_counter(\"time_spent_on_task_quota_violations_ms\", [this] {\n                return _time_spent_on_task_quota_violations / 1ms;\n        }, sm::description(\"Total amount in milliseconds we were in violation of the task quota\"),\n           {group_label}),\n    });\n\n    register_net_metrics_for_scheduling_group(new_metrics, _id, group_label);\n\n    _metrics = std::exchange(new_metrics, {});\n}\n\nvoid\nreactor::task_queue::rename(sstring new_name, sstring new_shortname) {\n    SEASTAR_ASSERT(!new_name.empty());\n    if (_name != new_name) {\n        _name = new_name;\n        if (new_shortname.empty()) {\n            _shortname = shorten_name(_name, shortname_size);\n        } else {\n            _shortname = fmt::format(\"{:>{}}\", new_shortname, shortname_size);\n        }\n        register_stats();\n    }\n}\n\n#ifdef __clang__\n__attribute__((no_sanitize(\"undefined\"))) // multiplication below may overflow; we check for that\n#elif defined(__GNUC__)\n[[gnu::no_sanitize_undefined]]\n#endif\ninline\nint64_t\nreactor::sched_entity::to_vruntime(sched_clock::duration runtime) const {\n    auto scaled = (runtime.count() * _reciprocal_shares_times_2_power_32) >> 32;\n    // Prevent overflow from returning ridiculous values\n    return std::max<int64_t>(scaled, 0);\n}\n\nvoid reactor::sched_entity::set_shares(float shares) noexcept {\n    _shares = std::max(shares, 1.0f);\n    _reciprocal_shares_times_2_power_32 = (uint64_t(1) << 32) / _shares;\n}\n\nvoid\nreactor::task_queue_group::account_runtime(reactor& r, sched_entity& tq, sched_clock::duration runtime) {\n    if (runtime > (2 * r._cfg.task_quota)) {\n        r._stalls_histogram.add(runtime);\n        tq._time_spent_on_task_quota_violations += runtime - r._cfg.task_quota;\n    }\n    tq._vruntime += tq.to_vruntime(runtime);\n    tq._runtime += runtime;\n}\n\nstruct reactor::task_queue::indirect_compare {\n    bool operator()(const sched_entity* tq1, const sched_entity* tq2) const {\n        return tq1->_vruntime < tq2->_vruntime;\n    }\n};\n\nreactor::reactor(std::shared_ptr<seastar::smp> smp, alien::instance& alien, unsigned id, reactor_backend_selector rbs, reactor_config cfg)\n    : _smp(std::move(smp))\n    , _alien(alien)\n    , _cfg(std::move(cfg))\n    , _notify_eventfd(file_desc::eventfd(0, EFD_CLOEXEC))\n    , _task_quota_timer(file_desc::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC))\n    , _id(id)\n    , _cpu_stall_detector(internal::make_cpu_stall_detector())\n    , _cpu_sched(nullptr, 0)\n    , _thread_pool(std::make_unique<thread_pool>(seastar::format(\"syscall-{}\", id), _notify_eventfd)) {\n    /*\n     * The _backend assignment is here, not on the initialization list as\n     * the chosen backend constructor may want to handle signals and thus\n     * needs the _signals._signal_handlers map to be initialized.\n     */\n    _backend = rbs.create(*this);\n    *internal::get_scheduling_group_specific_thread_local_data_ptr() = &_scheduling_group_specific_data;\n    _task_queues[0] = std::make_unique<task_queue>(&_cpu_sched, 0, \"main\", \"main\", 1000);\n    _task_queues[1] = std::make_unique<task_queue>(&_cpu_sched, 1, \"atexit\", \"exit\", 1000);\n    _at_destroy_tasks = _task_queues[1].get();\n    set_need_preempt_var(&_preemption_monitor);\n    seastar::thread_impl::init();\n    _backend->start_tick();\n\n    sigset_t mask;\n    sigemptyset(&mask);\n    sigaddset(&mask, internal::cpu_stall_detector::signal_number());\n    auto r = ::pthread_sigmask(SIG_UNBLOCK, &mask, NULL);\n    SEASTAR_ASSERT(r == 0);\n    memory::set_reclaim_hook([this] (std::function<void ()> reclaim_fn) {\n        add_high_priority_task(make_task(default_scheduling_group(), [fn = std::move(reclaim_fn)] {\n            fn();\n        }));\n    });\n\n    _loads.reserve(loads_size);\n    for (unsigned i = 0; i < loads_size; i++) {\n        _loads.push_back(0.0);\n    }\n}\n\nreactor::~reactor() {\n    sigset_t mask;\n    sigemptyset(&mask);\n    sigaddset(&mask, internal::cpu_stall_detector::signal_number());\n    auto r = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n    SEASTAR_ASSERT(r == 0);\n\n    _backend->stop_tick();\n    auto eraser = [](auto& list) {\n        while (!list.empty()) {\n            auto& timer = *list.begin();\n            timer.cancel();\n        }\n    };\n    eraser(_expired_timers);\n    eraser(_expired_lowres_timers);\n    eraser(_expired_manual_timers);\n    for (auto&& tq : _task_queues) {\n        if (tq) {\n            // The following line will preserve the convention that constructor and destructor functions\n            // for the per sg values are called in the context of the containing scheduling group.\n            auto sg = scheduling_group(tq->_id);\n            *internal::current_scheduling_group_ptr() = sg;\n            get_sg_data(sg).specific_vals.clear();\n        }\n    }\n}\n\nstd::string_view reactor::get_backend_name() const {\n    return _backend->get_backend_name();\n}\n\nreactor::sched_stats\nreactor::get_sched_stats() const {\n    sched_stats ret;\n    ret.tasks_processed = tasks_processed();\n    return ret;\n}\n\nfuture<> reactor::readable(pollable_fd_state& fd) {\n    return _backend->readable(fd);\n}\n\nfuture<> reactor::writeable(pollable_fd_state& fd) {\n    return _backend->writeable(fd);\n}\n\nfuture<> reactor::readable_or_writeable(pollable_fd_state& fd) {\n    return _backend->readable_or_writeable(fd);\n}\n\nfuture<> reactor::poll_rdhup(pollable_fd_state& fd) {\n    return _backend->poll_rdhup(fd);\n}\n\nvoid reactor::set_strict_dma(bool value) {\n    _cfg.strict_o_direct = value;\n}\n\nvoid reactor::set_bypass_fsync(bool value) {\n    _cfg.bypass_fsync = value;\n}\n\nvoid\nreactor::reset_preemption_monitor() {\n    return _backend->reset_preemption_monitor();\n}\n\nvoid\nreactor::request_preemption() {\n    return _backend->request_preemption();\n}\n\nvoid reactor::start_handling_signal() {\n    return _backend->start_handling_signal();\n}\n\nnamespace internal {\n\ncpu_stall_detector::cpu_stall_detector(cpu_stall_detector_config cfg)\n        : _shard_id(this_shard_id()) {\n    // glib's backtrace() calls dlopen(\"libgcc_s.so.1\") once to resolve unwind related symbols.\n    // If first stall detector invocation happens during another dlopen() call the calling thread\n    // will deadlock. The dummy call here makes sure that backtrace's initialization happens in\n    // a safe place.\n    backtrace([] (frame) {});\n    update_config(cfg);\n\n    namespace sm = seastar::metrics;\n\n    _metrics.add_group(\"stall_detector\", {\n            sm::make_counter(\"reported\", _total_reported, sm::description(\"Total number of reported stalls, look in the traces for the exact reason\"))});\n\n    // note: if something is added here that can, it should take care to destroy _timer.\n}\n\ncpu_stall_detector_posix_timer::cpu_stall_detector_posix_timer(cpu_stall_detector_config cfg) : cpu_stall_detector(cfg) {\n    struct sigevent sev = {};\n    sev.sigev_notify = SIGEV_THREAD_ID;\n    sev.sigev_signo = signal_number();\n#ifndef sigev_notify_thread_id\n#define sigev_notify_thread_id _sigev_un._tid\n#endif\n    sev.sigev_notify_thread_id = syscall(SYS_gettid);\n    int err = timer_create(CLOCK_THREAD_CPUTIME_ID, &sev, &_timer);\n    if (err) {\n        throw std::system_error(std::error_code(err, std::system_category()));\n    }\n}\n\ncpu_stall_detector_posix_timer::~cpu_stall_detector_posix_timer() {\n    timer_delete(_timer);\n}\n\ncpu_stall_detector_config\ncpu_stall_detector::get_config() const {\n    return _config;\n}\n\nvoid cpu_stall_detector::update_config(cpu_stall_detector_config cfg) {\n    _config = cfg;\n    _threshold = std::chrono::duration_cast<sched_clock::duration>(cfg.threshold);\n    _slack = std::chrono::duration_cast<sched_clock::duration>(cfg.threshold * cfg.slack);\n    _max_reports_per_minute = cfg.stall_detector_reports_per_minute;\n    _rearm_timer_at = reactor::now();\n}\n\nvoid cpu_stall_detector::maybe_report() {\n    if (_reported++ < _max_reports_per_minute) {\n        generate_trace();\n    }\n}\n// We use a tick at every timer firing so we can report suppressed backtraces.\n// Best case it's a correctly predicted branch. If a backtrace had happened in\n// the near past it's an increment and two branches.\n//\n// We can do it a cheaper if we don't report suppressed backtraces.\nvoid cpu_stall_detector::on_signal() {\n    auto tasks_processed = engine().tasks_processed();\n    auto last_seen = _last_tasks_processed_seen.load(std::memory_order_relaxed);\n    if (!last_seen) {\n        return; // stall detector in not active\n    } else if (last_seen == tasks_processed) {\n        // we defer to the check of spuriousness to inside this unlikely condition\n        // since the check itself can be costly, involving a syscall (reactor::now())\n        if (is_spurious_signal()) {\n            return;\n        }\n        // no task was processed - report unless supressed\n        maybe_report();\n        _report_at <<= 1;\n    } else {\n        _last_tasks_processed_seen.store(tasks_processed, std::memory_order_relaxed);\n    }\n    arm_timer();\n}\n\nvoid cpu_stall_detector::report_suppressions(sched_clock::time_point now) {\n    if (now > _minute_mark + 60s) {\n        if (_reported > _max_reports_per_minute) {\n            auto suppressed = _reported - _max_reports_per_minute;\n            backtrace_buffer buf;\n            // Reuse backtrace buffer infrastructure so we don't have to allocate here\n            buf.append(\"Rate-limit: suppressed \");\n            buf.append_decimal(suppressed);\n            suppressed == 1 ? buf.append(\" backtrace\") : buf.append(\" backtraces\");\n            buf.append(\" on shard \");\n            buf.append_decimal(_shard_id);\n            buf.append(\"\\n\");\n        }\n        reset_suppression_state(now);\n    }\n}\n\nvoid\ncpu_stall_detector::reset_suppression_state(sched_clock::time_point now) {\n    _reported = 0;\n    _minute_mark = now;\n}\n\nvoid cpu_stall_detector_posix_timer::arm_timer() {\n    auto its = posix::to_relative_itimerspec(_threshold * _report_at + _slack, 0s);\n    timer_settime(_timer, 0, &its, nullptr);\n}\n\nvoid cpu_stall_detector::start_task_run(sched_clock::time_point now) {\n    if (now > _rearm_timer_at) {\n        report_suppressions(now);\n        _report_at = 1;\n        _run_started_at = now;\n        _rearm_timer_at = now + _threshold * _report_at;\n        arm_timer();\n    }\n    _last_tasks_processed_seen.store(engine().tasks_processed(), std::memory_order_relaxed);\n    std::atomic_signal_fence(std::memory_order_release); // Don't delay this write, so the signal handler can see it\n}\n\nvoid cpu_stall_detector::end_task_run(sched_clock::time_point now) {\n    std::atomic_signal_fence(std::memory_order_acquire); // Don't hoist this write, so the signal handler can see it\n    _last_tasks_processed_seen.store(0, std::memory_order_relaxed);\n}\n\nvoid cpu_stall_detector_posix_timer::start_sleep() {\n    auto its = posix::to_relative_itimerspec(0s,  0s);\n    timer_settime(_timer, 0, &its, nullptr);\n    _rearm_timer_at = reactor::now();\n}\n\nvoid cpu_stall_detector::end_sleep() {\n}\n\nstatic long\nperf_event_open(struct perf_event_attr* hw_event, pid_t pid, int cpu, int group_fd, unsigned long flags) {\n    return syscall(__NR_perf_event_open, hw_event, pid, cpu, group_fd, flags);\n}\n\ncpu_stall_detector_linux_perf_event::cpu_stall_detector_linux_perf_event(file_desc fd, cpu_stall_detector_config cfg)\n        : cpu_stall_detector(cfg), _fd(std::move(fd)) {\n    void* ret = ::mmap(nullptr, 2*getpagesize(), PROT_READ|PROT_WRITE, MAP_SHARED, _fd.get(), 0);\n    if (ret == MAP_FAILED) {\n        abort();\n    }\n    _mmap = static_cast<struct ::perf_event_mmap_page*>(ret);\n    _data_area = reinterpret_cast<char*>(_mmap) + getpagesize();\n    _data_area_mask = getpagesize() - 1;\n}\n\ncpu_stall_detector_linux_perf_event::~cpu_stall_detector_linux_perf_event() {\n    ::munmap(_mmap, 2*getpagesize());\n}\n\nvoid\ncpu_stall_detector_linux_perf_event::arm_timer() {\n    auto period = _threshold * _report_at + _slack;\n    uint64_t ns =  period / 1ns;\n    _next_signal_time = reactor::now() + period;\n\n    // clear out any existing records in the ring buffer, so when we get interrupted next time\n    // we have only the stack associated with that interrupt, and so we don't overflow.\n    data_area_reader(*this).skip_all();\n    if (__builtin_expect(_enabled && _current_period == ns, 1)) {\n        // Common case - we're re-arming with the same period, the counter\n        // is already enabled.\n\n        // We want to set the next interrupt to ns from now, and somewhat oddly the\n        // way to do this is PERF_EVENT_IOC_PERIOD, even with the same period as\n        // already configured, see the code at:\n        //\n        // https://elixir.bootlin.com/linux/v5.15.86/source/kernel/events/core.c#L5636\n        //\n        // Ths change is intentional: kernel commit bad7192b842c83e580747ca57104dd51fe08c223\n        // so we can resumably rely on it.\n        _fd.ioctl(PERF_EVENT_IOC_PERIOD, ns);\n\n    } else {\n        // Uncommon case - we're moving from disabled to enabled, or changing\n        // the period. Issue more calls and be careful.\n        _fd.ioctl(PERF_EVENT_IOC_DISABLE, 0); // avoid false alarms while we modify stuff\n        _fd.ioctl(PERF_EVENT_IOC_PERIOD, ns);\n        _fd.ioctl(PERF_EVENT_IOC_RESET, 0);\n        _fd.ioctl(PERF_EVENT_IOC_ENABLE, 0);\n        _enabled = true;\n        _current_period = ns;\n    }\n}\n\nvoid\ncpu_stall_detector_linux_perf_event::start_sleep() {\n    _fd.ioctl(PERF_EVENT_IOC_DISABLE, 0);\n    _enabled = false;\n}\n\nbool\ncpu_stall_detector_linux_perf_event::is_spurious_signal() {\n    // If the current time is before the expected signal time, it is\n    // probably a spurious signal. One reason this could occur is that\n    // PERF_EVENT_IOC_PERIOD does not reset the current overflow point\n    // on kernels prior to 3.14 (or 3.7 on Arm).\n    return reactor::now() < _next_signal_time;\n}\n\nvoid\ncpu_stall_detector_linux_perf_event::maybe_report_kernel_trace(backtrace_buffer& buf) {\n    data_area_reader reader(*this);\n    auto current_record = [&] () -> ::perf_event_header {\n        return reader.read_struct<perf_event_header>();\n    };\n\n    while (reader.have_data()) {\n        auto record = current_record();\n\n        if (record.type != PERF_RECORD_SAMPLE) {\n            reader.skip(record.size - sizeof(record));\n            continue;\n        }\n\n        auto nr = reader.read_u64();\n        if (nr > 0) {\n            buf.append(\"kernel callstack:\");\n            for (uint64_t i = 0; i < nr; ++i) {\n                buf.append(\" 0x\");\n                buf.append_hex(uintptr_t(reader.read_u64()));\n            }\n            buf.append(\"\\n\");\n        }\n    };\n}\n\nstd::unique_ptr<cpu_stall_detector_linux_perf_event>\ncpu_stall_detector_linux_perf_event::try_make(cpu_stall_detector_config cfg) {\n    ::perf_event_attr pea = {\n        .type = PERF_TYPE_SOFTWARE,\n        .size = sizeof(pea),\n        .config = PERF_COUNT_SW_TASK_CLOCK, // more likely to work on virtual machines than hardware events\n        .sample_period = 1'000'000'000, // Needs non-zero value or PERF_IOC_PERIOD gets confused\n        .sample_type = PERF_SAMPLE_CALLCHAIN,\n        .disabled = 1,\n        .exclude_callchain_user = 1,  // we're using backtrace() to capture the user callchain\n        .wakeup_events = 1,\n    };\n    unsigned long flags = 0;\n    if (internal::kernel_uname().whitelisted({\"3.14\"})) {\n        flags |= PERF_FLAG_FD_CLOEXEC;\n    }\n    int fd = perf_event_open(&pea, 0, -1, -1, flags);\n    if (fd == -1) {\n        throw std::system_error(errno, std::system_category(), \"perf_event_open() failed\");\n    }\n    auto desc = file_desc::from_fd(fd);\n    struct f_owner_ex sig_owner = {\n        .type = F_OWNER_TID,\n        .pid = static_cast<pid_t>(syscall(SYS_gettid)),\n    };\n    auto ret1 = ::fcntl(fd, F_SETOWN_EX, &sig_owner);\n    if (ret1 == -1) {\n        abort();\n    }\n    auto ret2 = ::fcntl(fd, F_SETSIG, signal_number());\n    if (ret2 == -1) {\n        abort();\n    }\n    auto fd_flags = ::fcntl(fd, F_GETFL);\n    if (fd_flags == -1) {\n        abort();\n    }\n    auto ret3 = ::fcntl(fd, F_SETFL, fd_flags | O_ASYNC);\n    if (ret3 == -1) {\n        abort();\n    }\n    return std::make_unique<cpu_stall_detector_linux_perf_event>(std::move(desc), std::move(cfg));\n}\n\n\nstd::unique_ptr<cpu_stall_detector> make_cpu_stall_detector(cpu_stall_detector_config cfg) {\n    bool was_eaccess_failure = false;\n    try {\n        try {\n            return cpu_stall_detector_linux_perf_event::try_make(cfg);\n        } catch (std::system_error& e) {\n            // This failure occurs when /proc/sys/kernel/perf_event_paranoid is set\n            // to 2 or higher, and is expected since most distributions set it to that\n            // way as of 2023. In this case we log a different message and only at INFO\n            // level on shard 0.\n            was_eaccess_failure = e.code() == std::error_code(EACCES, std::system_category());\n            throw;\n        }\n    } catch (...) {\n        if (was_eaccess_failure) {\n            seastar_logger.info0(\"Perf-based stall detector creation failed (EACCESS), \"\n                    \"try setting /proc/sys/kernel/perf_event_paranoid to 1 or less to \"\n                    \"enable kernel backtraces: falling back to posix timer.\");\n        } else {\n            seastar_logger.warn(\"Creation of perf_event based stall detector failed: falling back to posix timer: {}\", std::current_exception());\n        }\n        return std::make_unique<cpu_stall_detector_posix_timer>(cfg);\n    }\n}\n\nvoid cpu_stall_detector::generate_trace() {\n    auto delta = reactor::now() - _run_started_at;\n\n    _total_reported++;\n    if (_config.report) {\n        _config.report();\n        return;\n    }\n\n    backtrace_buffer buf;\n    buf.append(\"Reactor stalled for \");\n    buf.append_decimal(uint64_t(delta / 1ms));\n    buf.append(\" ms\");\n    if (std::uncaught_exceptions() > 0) {\n        buf.append(\", backtrace omitted (uncaught exception in progress)\\n\");\n    } else {\n        print_with_backtrace(buf, _config.oneline);\n    }\n    maybe_report_kernel_trace(buf);\n}\n\n} // internal namespace\n\nvoid\nreactor::update_blocked_reactor_notify_ms(std::chrono::milliseconds ms) {\n    auto cfg = _cpu_stall_detector->get_config();\n    if (ms != cfg.threshold) {\n        cfg.threshold = ms;\n        _cpu_stall_detector->update_config(cfg);\n        seastar_logger.info(\"updated: blocked-reactor-notify-ms={}\", ms.count());\n    }\n}\n\nstd::chrono::milliseconds\nreactor::get_blocked_reactor_notify_ms() const {\n    auto d = _cpu_stall_detector->get_config().threshold;\n    return std::chrono::duration_cast<std::chrono::milliseconds>(d);\n}\n\nvoid\nreactor::test::set_stall_detector_report_function(std::function<void ()> report) {\n    auto& r = engine();\n    auto cfg = r._cpu_stall_detector->get_config();\n    cfg.report = std::move(report);\n    r._cpu_stall_detector->update_config(std::move(cfg));\n    r._cpu_stall_detector->reset_suppression_state(reactor::now());\n}\n\nstd::function<void ()>\nreactor::test::get_stall_detector_report_function() {\n    return engine()._cpu_stall_detector->get_config().report;\n}\n\nbool reactor::test::linux_aio_nowait() {\n    return engine()._cfg.aio_nowait_works;\n}\n\nreactor::test::long_task_queue_state\nreactor::test::get_long_task_queue_state() noexcept {\n    auto& r = engine();\n    return long_task_queue_state{\n        .abort_on_too_long_task_queue = r._cfg.abort_on_too_long_task_queue,\n        .max_task_backlog = r._cfg.max_task_backlog,\n    };\n}\n\nfuture<> reactor::test::restore_long_task_queue_state(const long_task_queue_state& state) noexcept {\n    return smp::invoke_on_all([&state] {\n        reactor::test::set_abort_on_too_long_task_queue(state.abort_on_too_long_task_queue);\n        reactor::test::set_max_task_backlog(state.max_task_backlog);\n    });\n}\n\nvoid reactor::test::set_abort_on_too_long_task_queue(bool value) noexcept {\n    auto& r = engine();\n    r._cfg.abort_on_too_long_task_queue = value;\n}\n\nvoid reactor::test::set_max_task_backlog(unsigned value) noexcept {\n    auto& r = engine();\n    r._cfg.max_task_backlog = value;\n}\n\nvoid\nreactor::block_notifier(int) {\n    engine()._cpu_stall_detector->on_signal();\n}\n\nclass network_stack_factory {\n    network_stack_entry::factory_func _func;\n\npublic:\n    network_stack_factory(noncopyable_function<future<std::unique_ptr<network_stack>> (const program_options::option_group&)> func)\n        : _func(std::move(func)) { }\n    future<std::unique_ptr<network_stack>> operator()(const program_options::option_group& opts) { return _func(opts); }\n};\n\nvoid reactor::configure(const reactor_options& opts) {\n    _network_stack_ready = opts.network_stack.get_selected_candidate()(*opts.network_stack.get_selected_candidate_opts());\n\n    auto blocked_time = opts.blocked_reactor_notify_ms.get_value() * 1ms;\n    internal::cpu_stall_detector_config csdc;\n    csdc.threshold = blocked_time;\n    csdc.stall_detector_reports_per_minute = opts.blocked_reactor_reports_per_minute.get_value();\n    csdc.oneline = opts.blocked_reactor_report_format_oneline.get_value();\n    _cpu_stall_detector->update_config(csdc);\n\n    if (_cfg.no_poll_aio) {\n        _aio_eventfd = pollable_fd(file_desc::eventfd(0, 0));\n    }\n}\n\nnamespace internal {\n\npollable_fd posix_listen(socket_address sa, listen_options opts) {\n    auto specific_protocol = (int)(opts.proto);\n    if (sa.is_af_unix()) {\n        // no type-safe way to create listen_opts with proto=0\n        specific_protocol = 0;\n    }\n    static auto somaxconn = [] {\n        std::optional<int> result;\n        std::ifstream ifs(\"/proc/sys/net/core/somaxconn\");\n        if (ifs) {\n            result = 0;\n            ifs >> *result;\n        }\n        return result;\n    }();\n    if (somaxconn && *somaxconn < opts.listen_backlog) {\n        fmt::print(\n            \"Warning: /proc/sys/net/core/somaxconn is set to {:d} \"\n            \"which is lower than the backlog parameter {:d} used for listen(), \"\n            \"please change it with `sysctl -w net.core.somaxconn={:d}`\\n\",\n            *somaxconn, opts.listen_backlog, opts.listen_backlog);\n    }\n\n    file_desc fd = file_desc::socket(sa.u.sa.sa_family, SOCK_STREAM | SOCK_NONBLOCK | SOCK_CLOEXEC, specific_protocol);\n    if (opts.reuse_address) {\n        fd.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1);\n    }\n\n    if (opts.so_sndbuf) {\n        fd.setsockopt(SOL_SOCKET, SO_SNDBUF, *opts.so_sndbuf);\n    }\n\n    if (opts.so_rcvbuf) {\n        fd.setsockopt(SOL_SOCKET, SO_RCVBUF, *opts.so_rcvbuf);\n    }\n\n    try {\n        fd.bind(sa.u.sa, sa.length());\n\n        if (sa.is_af_unix() && opts.unix_domain_socket_permissions) {\n            // After bind the socket is created in the file system.\n            mode_t mode = static_cast<mode_t>(opts.unix_domain_socket_permissions.value());\n            auto result = ::chmod(sa.u.un.sun_path, mode);\n            if (result < 0) {\n                auto errno_copy = errno;\n                ::unlink(sa.u.un.sun_path);\n                throw std::system_error(errno_copy, std::system_category(), \"chmod failed\");\n            }\n        }\n\n        fd.listen(opts.listen_backlog);\n    } catch (const std::system_error& s) {\n        throw std::system_error(s.code(), fmt::format(\"posix_listen failed for address {}\", sa));\n    }\n\n    return pollable_fd(std::move(fd));\n}\n\n} // internel namespace\n\nvoid pollable_fd_state::maybe_no_more_recv() {\n    if (shutdown_mask & posix::rcv_shutdown) {\n        throw std::system_error(std::error_code(ECONNABORTED, std::system_category()));\n    }\n}\n\nvoid pollable_fd_state::maybe_no_more_send() {\n    if (shutdown_mask & posix::snd_shutdown) {\n        throw std::system_error(std::error_code(ECONNABORTED, std::system_category()));\n    }\n}\n\nvoid pollable_fd_state::forget() {\n    engine()._backend->forget(*this);\n}\n\nvoid pollable_fd_state::shutdown(int how) {\n    fd.shutdown(how);\n}\n\nvoid intrusive_ptr_release(pollable_fd_state* fd) {\n    if (!--fd->_refs) {\n        fd->forget();\n    }\n}\n\npollable_fd_state_ptr pollable_fd_state::make(file_desc fd, speculation speculate) {\n    return engine()._backend->make_pollable_fd_state(std::move(fd), speculate);\n}\n\nvoid pollable_fd::shutdown(int how, shutdown_kernel_only kernel_only) {\n    if (!kernel_only) {\n        // TCP will respond to shutdown() by returning ECONNABORT on the next IO,\n        // but UDP responds by returning AGAIN. The shutdown_mask tells us to convert\n        // EAGAIN to ECONNABORT in that case.\n        _s->shutdown_mask |= posix::shutdown_mask(how);\n    }\n    _s->shutdown(how);\n}\n\npollable_fd\nreactor::make_pollable_fd(socket_address sa, int proto) {\n    int maybe_nonblock = _backend->do_blocking_io() ? 0 : SOCK_NONBLOCK;\n    file_desc fd = file_desc::socket(sa.u.sa.sa_family, SOCK_STREAM | maybe_nonblock | SOCK_CLOEXEC, proto);\n    return pollable_fd(std::move(fd));\n}\n\nbool reactor::posix_sock_need_nonblock() const {\n    return !_backend->do_blocking_io();\n}\n\nbool reactor::have_aio_fdatasync() const {\n    return _cfg.have_aio_fsync && _backend->have_aio_fdatasync();\n}\n\nnamespace internal {\n\nfuture<> posix_connect(pollable_fd pfd, socket_address sa, socket_address local) {\n  try {\n#ifdef IP_BIND_ADDRESS_NO_PORT\n    if (!sa.is_af_unix()) {\n        try {\n            // do not reserve an ephemeral port when using bind() with port number 0.\n            // connect() will handle it later. The reason for that is that bind() may fail\n            // to allocate a port while connect will success, this is because bind() does not\n            // know dst address and has to find globally unique local port.\n            pfd.get_file_desc().setsockopt(SOL_IP, IP_BIND_ADDRESS_NO_PORT, 1);\n        } catch (std::system_error& err) {\n            if (err.code() !=  std::error_code(ENOPROTOOPT, std::system_category())) {\n                throw;\n            }\n        }\n    }\n#endif\n    if (!local.is_wildcard()) {\n        // call bind() only if local address is not wildcard\n        pfd.get_file_desc().bind(local.u.sa, local.length());\n    }\n    return pfd.connect(sa).finally([pfd] {});\n  } catch (...) {\n    return current_exception_as_future();\n  }\n}\n\n} // internal namespace\n\nserver_socket\nreactor::listen(socket_address sa, listen_options opt) {\n    return server_socket(_network_stack->listen(sa, opt));\n}\n\nfuture<connected_socket>\nreactor::connect(socket_address sa) {\n    return _network_stack->connect(sa);\n}\n\nfuture<connected_socket>\nreactor::connect(socket_address sa, socket_address local, transport proto) {\n    return _network_stack->connect(sa, local, proto);\n}\n\nvoid io_completion::complete_with(ssize_t res) {\n    if (res >= 0) {\n        complete(res);\n        return;\n    }\n\n    reactor::io_stats::local().aio_errors++;\n    try {\n        throw_kernel_error(res);\n    } catch (...) {\n        set_exception(std::current_exception());\n    }\n}\n\nbool\nreactor::flush_pending_aio() {\n    for (auto& ioq : _io_queues) {\n        ioq.second->poll_io_queue();\n    }\n    return false;\n}\n\nsteady_clock_type::time_point reactor::next_pending_aio() const noexcept {\n    steady_clock_type::time_point next = steady_clock_type::time_point::max();\n\n    for (auto& ioq : _io_queues) {\n        steady_clock_type::time_point n = ioq.second->next_pending_aio();\n        if (n < next) {\n            next = std::move(n);\n        }\n    }\n\n    return next;\n}\n\nbool\nreactor::reap_kernel_completions() {\n    return _backend->reap_kernel_completions();\n}\n\nnamespace internal {\n\nsize_t sanitize_iovecs(std::vector<iovec>& iov, size_t disk_alignment) noexcept {\n    if (iov.size() > IOV_MAX) {\n        iov.resize(IOV_MAX);\n    }\n    auto length = iovec_len(iov);\n    while (auto rest = length & (disk_alignment - 1)) {\n        if (iov.back().iov_len <= rest) {\n            length -= iov.back().iov_len;\n            iov.pop_back();\n        } else {\n            iov.back().iov_len -= rest;\n            length -= rest;\n        }\n    }\n    return length;\n}\n\n}\n\nfuture<file>\nreactor::open_file_dma(std::string_view nameref, open_flags flags, file_open_options options) noexcept {\n    auto open_flags = static_cast<int>(flags);\n    sstring name(nameref);\n    syscall_result_extra<struct stat> sr = co_await _thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [this, name, &open_flags, &options, strict_o_direct = _cfg.strict_o_direct, bypass_fsync = _cfg.bypass_fsync] () mutable {\n        // We want O_DIRECT, except in three cases:\n        //   - tmpfs (which doesn't support it, but works fine anyway)\n        //   - strict_o_direct == false (where we forgive it being not supported)\n        //   - kernel_page_cache == true (where we disable it for short-lived test processes)\n        // Because open() with O_DIRECT will fail, we open it without O_DIRECT, try\n        // to update it to O_DIRECT with fcntl(), and if that fails, see if we\n        // can forgive it.\n        auto is_tmpfs = [] (int fd) {\n            struct ::statfs buf;\n            auto r = ::fstatfs(fd, &buf);\n            if (r == -1) {\n                return false;\n            }\n            return buf.f_type == internal::fs_magic::tmpfs;\n        };\n        open_flags |= O_CLOEXEC;\n        if (bypass_fsync) {\n            options.durable = false;\n        }\n        if (!options.durable) {\n            open_flags &= ~O_DSYNC;\n        }\n        struct stat st;\n        auto mode = static_cast<mode_t>(options.create_permissions);\n        int fd = ::open(name.c_str(), open_flags, mode);\n        if (fd == -1) {\n            return wrap_syscall(fd, st);\n        }\n        auto close_fd = defer([fd] () noexcept { ::close(fd); });\n        int o_direct_flag = _cfg.kernel_page_cache ? 0 : O_DIRECT;\n        int r = ::fcntl(fd, F_SETFL, open_flags | o_direct_flag);\n        if (r == -1  && strict_o_direct) {\n            auto maybe_ret = wrap_syscall(r, st);  // capture errno (should be EINVAL)\n            if (!is_tmpfs(fd)) {\n                return maybe_ret;\n            }\n        }\n        if (fd != -1 && options.extent_allocation_size_hint && !_cfg.kernel_page_cache) {\n            fsxattr attr = {};\n            int r = ::ioctl(fd, XFS_IOC_FSGETXATTR, &attr);\n            // xfs delayed allocation is disabled when extent size hints are present.\n            // This causes tons of xfs log fsyncs. Given that extent size hints are\n            // unneeded when delayed allocation is available (which is the case\n            // when not using O_DIRECT), disable them.\n            //\n            // Ignore error; may be !xfs, and just a hint anyway\n            if (r != -1) {\n                attr.fsx_xflags |= XFS_XFLAG_EXTSIZE;\n                attr.fsx_extsize = std::min(options.extent_allocation_size_hint,\n                                    file_open_options::max_extent_allocation_size_hint);\n\n                attr.fsx_extsize = align_up<uint32_t>(attr.fsx_extsize, file_open_options::min_extent_size_hint_alignment);\n\n                // Ignore error; may be !xfs, and just a hint anyway\n                ::ioctl(fd, XFS_IOC_FSSETXATTR, &attr);\n            }\n        }\n        r = ::fstat(fd, &st);\n        if (r == -1) {\n            return wrap_syscall(r, st);\n        }\n        close_fd.cancel();\n        return wrap_syscall(fd, st);\n    });\n    sr.throw_fs_exception_if_error(\"open failed\", name);\n    shared_ptr<file_impl> impl = co_await make_file_impl(sr.result, options, open_flags, sr.extra);\n    co_return file(std::move(impl));\n}\n\nfuture<>\nreactor::remove_file(std::string_view pathname_view) noexcept {\n    auto pathname = sstring(pathname_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [pathname] {\n        return wrap_syscall<int>(::remove(pathname.c_str()));\n    });\n    sr.throw_fs_exception_if_error(\"remove failed\", pathname);\n}\n\nfuture<>\nreactor::rename_file(std::string_view old_pathname_view, std::string_view new_pathname_view) noexcept {\n    auto old_pathname = sstring(old_pathname_view);\n    auto new_pathname = sstring(new_pathname_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [old_pathname, new_pathname] {\n        return wrap_syscall<int>(::rename(old_pathname.c_str(), new_pathname.c_str()));\n    });\n    sr.throw_fs_exception_if_error(\"rename failed\",  old_pathname, new_pathname);\n}\n\nfuture<>\nreactor::link_file(std::string_view oldpath_view, std::string_view newpath_view) noexcept {\n    auto oldpath = sstring(oldpath_view);\n    auto newpath = sstring(newpath_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [oldpath, newpath] {\n        return wrap_syscall<int>(::link(oldpath.c_str(), newpath.c_str()));\n    });\n    sr.throw_fs_exception_if_error(\"link failed\", oldpath, newpath);\n}\n\nfuture<>\nreactor::chmod(std::string_view name_view, file_permissions permissions) noexcept {\n    auto mode = static_cast<mode_t>(permissions);\n    auto name = sstring(name_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [name, mode] {\n        return wrap_syscall<int>(::chmod(name.c_str(), mode));\n    });\n    if (sr.result == -1) {\n        auto reason = format(\"chmod(0{:o}) failed\", mode);\n        sr.throw_fs_exception(reason, fs::path(name));\n    }\n}\n\ndirectory_entry_type stat_to_entry_type(mode_t type) {\n    if (S_ISDIR(type)) {\n        return directory_entry_type::directory;\n    }\n    if (S_ISBLK(type)) {\n        return directory_entry_type::block_device;\n    }\n    if (S_ISCHR(type)) {\n        return directory_entry_type::char_device;\n    }\n    if (S_ISFIFO(type)) {\n        return directory_entry_type::fifo;\n    }\n    if (S_ISLNK(type)) {\n        return directory_entry_type::link;\n    }\n    if (S_ISSOCK(type)) {\n        return directory_entry_type::socket;\n    }\n    if (S_ISREG(type)) {\n        return directory_entry_type::regular;\n    }\n    return directory_entry_type::unknown;\n}\n\nfuture<std::optional<directory_entry_type>>\nreactor::file_type(std::string_view name_view, follow_symlink follow) noexcept {\n    auto name = sstring(name_view);\n    syscall_result_extra<struct stat> sr = co_await _thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [name, follow] {\n        struct stat st;\n        auto stat_syscall = follow ? stat : lstat;\n        auto ret = stat_syscall(name.c_str(), &st);\n        return wrap_syscall(ret, st);\n    });\n    if (long(sr.result) == -1) {\n        if (sr.error != ENOENT && sr.error != ENOTDIR) {\n            sr.throw_fs_exception_if_error(\"stat failed\", name);\n        }\n        co_return std::optional<directory_entry_type>();\n    }\n    co_return std::optional<directory_entry_type>(stat_to_entry_type(sr.extra.st_mode));\n}\n\nfuture<std::optional<directory_entry_type>>\nfile_type(std::string_view name, follow_symlink follow) noexcept {\n    return engine().file_type(name, follow);\n}\n\nstatic std::chrono::system_clock::time_point\ntimespec_to_time_point(const timespec& ts) {\n    auto d = std::chrono::duration_cast<std::chrono::system_clock::duration>(\n            ts.tv_sec * 1s + ts.tv_nsec * 1ns);\n    return std::chrono::system_clock::time_point(d);\n}\n\nfuture<int>\nreactor::inotify_add_watch(int fd, std::string_view path_view, uint32_t flags) {\n    auto path = sstring(path_view);\n    syscall_result<int> ret = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [fd, path, flags] {\n        auto ret = ::inotify_add_watch(fd, path.c_str(), flags);\n        return wrap_syscall(ret);\n    });\n    ret.throw_if_error();\n    co_return ret.result;\n}\n\nfuture<std::tuple<file_desc, file_desc>>\nreactor::make_pipe() {\n    auto pipe = std::array<int, 2>{};\n    syscall_result<int> ret = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [&pipe] {\n        return wrap_syscall<int>(::pipe2(pipe.data(), O_NONBLOCK));\n    });\n    ret.throw_if_error();\n    co_return std::tuple<file_desc, file_desc>(file_desc::from_fd(pipe[0]),\n                                                                file_desc::from_fd(pipe[1]));\n}\n\nfuture<std::tuple<pid_t, file_desc, file_desc, file_desc>>\nreactor::spawn(std::string_view pathname_view,\n               std::vector<sstring> argv,\n               std::vector<sstring> env) {\n    auto pathname = sstring(pathname_view);\n\n    auto [cin_pipe, cout_pipe, cerr_pipe] = co_await coroutine::all(\n            std::bind(&reactor::make_pipe, this),\n            std::bind(&reactor::make_pipe, this),\n            std::bind(&reactor::make_pipe, this));\n\n    auto child_pid = pid_t{};\n    auto actions = posix_spawn_file_actions_t{};\n    auto attr = posix_spawnattr_t{};\n\n    static constexpr int pipefd_read_end = 0;\n    static constexpr int pipefd_write_end = 1;\n\n    // the args and envs parameters passed to posix_spawn() should be array of pointers, and\n    // the last one should be a null pointer.\n    std::vector<const char*> argvp;\n    std::transform(argv.cbegin(), argv.cend(), std::back_inserter(argvp),\n                    [](auto& s) { return s.c_str(); });\n    argvp.push_back(nullptr);\n\n    std::vector<const char*> envp;\n    std::transform(env.cbegin(), env.cend(), std::back_inserter(envp),\n                    [](auto& s) { return s.c_str(); });\n    envp.push_back(nullptr);\n\n    int r = 0;\n    r = ::posix_spawn_file_actions_init(&actions);\n    throw_pthread_error(r);\n    // the child process does not write to stdin\n    auto undo1 = defer([&actions] () noexcept{\n        ::posix_spawn_file_actions_destroy(&actions);\n    });\n    std::get<pipefd_write_end>(cin_pipe).spawn_actions_add_close(&actions);\n    // the child process does not read from stdout\n    std::get<pipefd_read_end>(cout_pipe).spawn_actions_add_close(&actions);\n    // the child process does not read from stderr\n    std::get<pipefd_read_end>(cerr_pipe).spawn_actions_add_close(&actions);\n    // redirect stdin, stdout and stderr to cin_pipe, cout_pipe and cerr_pipe respectively\n    std::get<pipefd_read_end>(cin_pipe).spawn_actions_add_dup2(&actions, STDIN_FILENO);\n    std::get<pipefd_write_end>(cout_pipe).spawn_actions_add_dup2(&actions, STDOUT_FILENO);\n    std::get<pipefd_write_end>(cerr_pipe).spawn_actions_add_dup2(&actions, STDERR_FILENO);\n    // after dup2() the interesting ends of pipes, close them\n    std::get<pipefd_read_end>(cin_pipe).spawn_actions_add_close(&actions);\n    std::get<pipefd_write_end>(cout_pipe).spawn_actions_add_close(&actions);\n    std::get<pipefd_write_end>(cerr_pipe).spawn_actions_add_close(&actions);\n    // tools like \"cat\" expect a fd opened in blocking mode when performing I/O\n    std::get<pipefd_read_end>(cin_pipe).template ioctl<int>(FIONBIO, 0);\n    std::get<pipefd_write_end>(cout_pipe).template ioctl<int>(FIONBIO, 0);\n    std::get<pipefd_write_end>(cerr_pipe).template ioctl<int>(FIONBIO, 0);\n\n    r = ::posix_spawnattr_init(&attr);\n    throw_pthread_error(r);\n    auto undo2 = defer([&attr] () noexcept {\n        ::posix_spawnattr_destroy(&attr);\n    });\n    // make sure the following signals are not ignored by the child process\n    sigset_t default_signals;\n    sigemptyset(&default_signals);\n    sigaddset(&default_signals, SIGINT);\n    sigaddset(&default_signals, SIGTERM);\n    r = ::posix_spawnattr_setsigdefault(&attr, &default_signals);\n    throw_pthread_error(r);\n    // make sure no signals are marked in the child process\n    sigset_t mask_signals;\n    sigemptyset(&mask_signals);\n    r = ::posix_spawnattr_setsigmask(&attr, &mask_signals);\n    throw_pthread_error(r);\n    r = ::posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF | POSIX_SPAWN_SETSIGMASK);\n    throw_pthread_error(r);\n\n\n\n    syscall_result<int> ret = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::process_operation,\n            [&child_pid, &pathname, &actions, &attr,\n                argv = std::move(argvp),\n                env =  std::move(envp)] {\n        return wrap_syscall<int>(::posix_spawn(&child_pid, pathname.c_str(), &actions, &attr,\n                                                const_cast<char* const *>(argv.data()),\n                                                const_cast<char* const *>(env.data())));\n    });\n\n    throw_pthread_error(ret.result);\n    co_return std::tuple(\n        child_pid,\n        std::get<pipefd_write_end>(std::move(cin_pipe)),\n        std::get<pipefd_read_end>(std::move(cout_pipe)),\n        std::get<pipefd_read_end>(std::move(cerr_pipe)));\n}\n\nstatic auto next_waitpid_timeout(std::chrono::milliseconds this_timeout) {\n    constexpr std::chrono::milliseconds step_timeout(20);\n    constexpr std::chrono::milliseconds max_timeout(1000);\n    if (this_timeout >= max_timeout) {\n        return max_timeout;\n    }\n    return this_timeout + step_timeout;\n}\n\n#ifndef __NR_pidfd_open\n\n#  if defined(__alpha__)\n#    define __NR_pidfd_open 544\n#  else\n#    define __NR_pidfd_open 434\n#  endif\n\n#endif\n\nfuture<int> reactor::waitpid(pid_t pid) {\n    syscall_result<int> pidfd = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::process_operation, [pid] {\n        return wrap_syscall<int>(syscall(__NR_pidfd_open, pid, O_NONBLOCK));\n    });\n    // pidfd_open() was introduced in linux 5.3, so the pidfd.error could be ENOSYS on\n    // older kernels. But it could be other error like EMFILE or ENFILE. anyway, we\n    // should always waitpid().\n    std::optional<pollable_fd> pfd;\n    if (pidfd.result != -1) {\n        pfd.emplace(file_desc::from_fd(pidfd.result));\n        co_await pfd->readable();\n    }\n\n    auto do_waitpid = [this] (pid_t pid) -> future<std::optional<int>> {\n        int wstatus;\n        auto ret = co_await _thread_pool->submit<syscall_result<pid_t>>(\n                internal::thread_pool_submit_reason::process_operation, [&] {\n            return wrap_syscall<pid_t>(::waitpid(pid, &wstatus, WNOHANG));\n        });\n        if (ret.result == 0) {\n            // Result not ready yet (with WNOHANG)\n            co_return std::nullopt;\n        } else if (ret.result > 0) {\n            // Success.  Return the waited pid status\n            co_return wstatus;\n        } else {\n            // Error.  Maybe throw exception, or return -1 status.\n            ret.throw_if_error();\n            co_return -1;\n        }\n    };\n\n    std::optional<int> ret_opt;\n    std::chrono::milliseconds wait_timeout(0);\n    while (!(ret_opt = co_await do_waitpid(pid))) {\n        wait_timeout = next_waitpid_timeout(wait_timeout);\n        co_await sleep(wait_timeout);\n    }\n    co_return *ret_opt;\n}\n\nvoid reactor::kill(pid_t pid, int sig) {\n    auto ret = wrap_syscall<int>(::kill(pid, sig));\n    ret.throw_if_error();\n}\n\nfuture<std::optional<struct group_details>> reactor::getgrnam(std::string_view name) {\n    syscall_result_extra<std::optional<struct group_details>> sr = co_await _thread_pool->submit<syscall_result_extra<std::optional<struct group_details>>>(\n        internal::thread_pool_submit_reason::file_operation, [name = sstring(name)] {\n            struct group grp;\n            struct group *result;\n            memset(&grp, 0, sizeof(struct group));\n            char buf[1024];\n            errno = 0;\n            int ret = ::getgrnam_r(name.c_str(), &grp, buf, sizeof(buf), &result);\n            if (!result) {\n                return wrap_syscall(ret, std::optional<struct group_details>(std::nullopt));\n            }\n\n            group_details gd;\n            gd.group_name = sstring(grp.gr_name);\n            gd.group_passwd = sstring(grp.gr_passwd);\n            gd.group_id = grp.gr_gid;\n            for (char **members = grp.gr_mem; *members != nullptr; ++members) {\n                gd.group_members.emplace_back(sstring(*members));\n            }\n            return wrap_syscall(ret, std::optional<struct group_details>(gd));\n        });\n\n    // Non-existent groups may return 0, or they may return ENOENT, depending on the\n    // configuration of the system, see seastar#2793. Treat both as \"not found\".\n    if (sr.result != 0 && sr.result != ENOENT) {\n        throw std::system_error(sr.ec());\n    }\n\n    co_return sr.extra;\n}\n\nfuture<> reactor::chown(std::string_view filepath, uid_t owner, gid_t group) {\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n        internal::thread_pool_submit_reason::file_operation, [filepath = sstring(filepath), owner, group] {\n            int ret = ::chown(filepath.c_str(), owner, group);\n            return wrap_syscall(ret);\n        });\n\n    sr.throw_if_error();\n    co_return;\n}\n\nstatic stat_data make_stat_data(const struct stat& st) {\n    stat_data sd;\n    sd.device_id = st.st_dev;\n    sd.inode_number = st.st_ino;\n    sd.mode = st.st_mode;\n    sd.type = stat_to_entry_type(st.st_mode);\n    sd.number_of_links = st.st_nlink;\n    sd.uid = st.st_uid;\n    sd.gid = st.st_gid;\n    sd.rdev = st.st_rdev;\n    sd.size = st.st_size;\n    sd.block_size = st.st_blksize;\n    sd.allocated_size = st.st_blocks * 512UL;\n    sd.time_accessed = timespec_to_time_point(st.st_atim);\n    sd.time_modified = timespec_to_time_point(st.st_mtim);\n    sd.time_changed = timespec_to_time_point(st.st_ctim);\n    return sd;\n}\n\nfuture<stat_data>\nreactor::file_stat(std::string_view pathname_view, follow_symlink follow) noexcept {\n    auto pathname = sstring(pathname_view);\n    syscall_result_extra<struct stat> sr = co_await _thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        struct stat st;\n        auto stat_syscall = follow ? stat : lstat;\n        auto ret = stat_syscall(pathname.c_str(), &st);\n        return wrap_syscall(ret, st);\n    });\n    sr.throw_fs_exception_if_error(\"stat failed\", pathname);\n    co_return make_stat_data(sr.extra);\n}\n\n\nfuture<stat_data>\nreactor::file_stat(file& directory, std::string_view pathname, follow_symlink follow) noexcept {\n    auto st = co_await directory.statat(pathname, follow ? 0 : AT_SYMLINK_NOFOLLOW);\n    co_return make_stat_data(st);\n}\n\nfuture<uint64_t>\nreactor::file_size(std::string_view pathname) noexcept {\n    stat_data sd = co_await file_stat(pathname, follow_symlink::yes);\n    co_return sd.size;\n}\n\nfuture<bool>\nreactor::file_accessible(std::string_view pathname_view, access_flags flags) noexcept {\n    auto pathname = sstring(pathname_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        auto aflags = std::underlying_type_t<access_flags>(flags);\n        auto ret = ::access(pathname.c_str(), aflags);\n        return wrap_syscall(ret);\n    });\n    if (sr.result < 0) {\n        if ((sr.error == ENOENT && flags == access_flags::exists) ||\n            (sr.error == EACCES && flags != access_flags::exists)) {\n            co_return false;\n        }\n        sr.throw_fs_exception(\"access failed\", fs::path(pathname));\n    }\n\n    co_return true;\n}\n\nfuture<fs_type>\nreactor::file_system_at(std::string_view pathname_view) noexcept {\n    auto pathname = sstring(pathname_view);\n    syscall_result_extra<struct statfs> sr = co_await _thread_pool->submit<syscall_result_extra<struct statfs>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        struct statfs st;\n        auto ret = statfs(pathname.c_str(), &st);\n        return wrap_syscall(ret, st);\n    });\n    static std::unordered_map<long int, fs_type> type_mapper = {\n        { internal::fs_magic::xfs, fs_type::xfs },\n        { internal::fs_magic::ext2, fs_type::ext2 },\n        { internal::fs_magic::ext3, fs_type::ext3 },\n        { internal::fs_magic::ext4, fs_type::ext4 },\n        { internal::fs_magic::btrfs, fs_type::btrfs },\n        { internal::fs_magic::hfs, fs_type::hfs },\n        { internal::fs_magic::tmpfs, fs_type::tmpfs },\n    };\n    sr.throw_fs_exception_if_error(\"statfs failed\", pathname);\n\n    fs_type ret = fs_type::other;\n    if (type_mapper.count(sr.extra.f_type) != 0) {\n        ret = type_mapper.at(sr.extra.f_type);\n    }\n    co_return ret;\n}\n\nfuture<struct statfs>\nreactor::fstatfs(int fd) noexcept {\n    syscall_result_extra<struct statfs> sr = co_await _thread_pool->submit<syscall_result_extra<struct statfs>>(\n            internal::thread_pool_submit_reason::file_operation, [fd] {\n        struct statfs st;\n        auto ret = ::fstatfs(fd, &st);\n        return wrap_syscall(ret, st);\n    });\n    sr.throw_if_error();\n    struct statfs st = sr.extra;\n    co_return st;\n}\n\nfuture<std::filesystem::space_info>\nreactor::file_system_space(std::string_view pathname) noexcept {\n    auto sr = co_await _thread_pool->submit<syscall_result_extra<std::filesystem::space_info>>(\n            internal::thread_pool_submit_reason::file_operation, [path = std::filesystem::path(pathname)] {\n        std::error_code ec;\n        auto si = std::filesystem::space(path, ec);\n        return wrap_syscall(ec.value(), si);\n    });\n    sr.throw_fs_exception_if_error(\"std::filesystem::space failed\", sstring(pathname));\n    co_return sr.extra;\n}\n\nfuture<struct statvfs>\nreactor::statvfs(std::string_view pathname_view) noexcept {\n    auto pathname = sstring(pathname_view);\n    syscall_result_extra<struct statvfs> sr = co_await _thread_pool->submit<syscall_result_extra<struct statvfs>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        struct statvfs st;\n        auto ret = ::statvfs(pathname.c_str(), &st);\n        return wrap_syscall(ret, st);\n    });\n    sr.throw_fs_exception_if_error(\"statvfs failed\", pathname);\n    struct statvfs st = sr.extra;\n    co_return st;\n}\n\nfuture<file>\nreactor::open_directory(std::string_view name_view) noexcept {\n    auto name = sstring(name_view);\n    auto oflags = O_DIRECTORY | O_CLOEXEC | O_RDONLY;\n    syscall_result_extra<struct stat> sr = co_await _thread_pool->submit<syscall_result_extra<struct stat>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        struct stat st;\n        int fd = ::open(name.c_str(), oflags);\n        if (fd != -1) {\n            int r = ::fstat(fd, &st);\n            if (r == -1) {\n                ::close(fd);\n                fd = r;\n            }\n        }\n        return wrap_syscall(fd, st);\n    });\n    sr.throw_fs_exception_if_error(\"open failed\", name);\n    shared_ptr<file_impl> file_impl = co_await make_file_impl(sr.result, file_open_options(), oflags, sr.extra);\n    co_return file(std::move(file_impl));\n}\n\nfuture<>\nreactor::make_directory(std::string_view name_view, file_permissions permissions) noexcept {\n    auto name = sstring(name_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        auto mode = static_cast<mode_t>(permissions);\n        return wrap_syscall<int>(::mkdir(name.c_str(), mode));\n    });\n    sr.throw_fs_exception_if_error(\"mkdir failed\", name);\n}\n\nfuture<>\nreactor::touch_directory(std::string_view name_view, file_permissions permissions) noexcept {\n    auto name = sstring(name_view);\n    syscall_result<int> sr = co_await _thread_pool->submit<syscall_result<int>>(\n            internal::thread_pool_submit_reason::file_operation, [&] {\n        auto mode = static_cast<mode_t>(permissions);\n        return wrap_syscall<int>(::mkdir(name.c_str(), mode));\n    });\n    if (sr.result == -1 && sr.error != EEXIST) {\n        sr.throw_fs_exception(\"mkdir failed\", fs::path(name));\n    }\n}\n\n// Note: terminate if arm_highres_timer throws\n// `when` should always be valid\nvoid reactor::enable_timer(steady_clock_type::time_point when) noexcept\n{\n    itimerspec its;\n    its.it_interval = {};\n    its.it_value = posix::to_timespec(when.time_since_epoch());\n    _backend->arm_highres_timer(its);\n}\n\n\nvoid reactor::add_timer(timer<steady_clock_type>* tmr) noexcept {\n    if (queue_timer(tmr)) {\n        enable_timer(_timers.get_next_timeout());\n    }\n}\n\nbool reactor::queue_timer(timer<steady_clock_type>* tmr) noexcept {\n    return _timers.insert(*tmr);\n}\n\nvoid reactor::del_timer(timer<steady_clock_type>* tmr) noexcept {\n    _timers.remove(*tmr, _expired_timers);\n}\n\nvoid reactor::add_timer(timer<lowres_clock>* tmr) noexcept {\n    if (queue_timer(tmr)) {\n        _lowres_next_timeout = _lowres_timers.get_next_timeout();\n    }\n}\n\nbool reactor::queue_timer(timer<lowres_clock>* tmr) noexcept {\n    return _lowres_timers.insert(*tmr);\n}\n\nvoid reactor::del_timer(timer<lowres_clock>* tmr) noexcept {\n    _lowres_timers.remove(*tmr, _expired_lowres_timers);\n}\n\nvoid reactor::add_timer(timer<manual_clock>* tmr) noexcept {\n    queue_timer(tmr);\n}\n\nbool reactor::queue_timer(timer<manual_clock>* tmr) noexcept {\n    return _manual_timers.insert(*tmr);\n}\n\nvoid reactor::del_timer(timer<manual_clock>* tmr) noexcept {\n    _manual_timers.remove(*tmr, _expired_manual_timers);\n}\n\nvoid reactor::do_at_exit(noncopyable_function<future<> ()> func) {\n    SEASTAR_ASSERT(!_stopping);\n    _exit_funcs.push_back(std::move(func));\n}\n\nvoid reactor::at_exit(noncopyable_function<future<> ()> func) {\n    do_at_exit(std::move(func));\n}\n\nvoid internal::at_exit(noncopyable_function<future<> ()> func) {\n    engine().do_at_exit(std::move(func));\n}\n\nfuture<> reactor::run_exit_tasks() {\n    _stop_requested.broadcast();\n    stop_aio_eventfd_loop();\n    return do_for_each(_exit_funcs.rbegin(), _exit_funcs.rend(), [] (auto& func) {\n        return func();\n    });\n}\n\nvoid reactor::stop() {\n    SEASTAR_ASSERT(_id == 0);\n    _smp->cleanup_cpu();\n    if (!std::exchange(_stopping, true)) {\n        // Run exit tasks locally and then stop all other engines\n        // in the background and wait on semaphore for all to complete.\n        // Finally, set _stopped on cpu 0.\n        (void)drain().then([this] {\n          return run_exit_tasks().then([this] {\n            return do_with(semaphore(0), [this] (semaphore& sem) {\n                // Stop other cpus asynchronously, signal when done.\n                (void)smp::invoke_on_others(0, [] {\n                    engine()._smp->cleanup_cpu();\n                    engine()._stopping = true;\n                    return engine().run_exit_tasks().then([] {\n                        engine()._stopped = true;\n                    });\n                }).then([&sem]() {\n                    sem.signal();\n                });\n                return sem.wait().then([this] {\n                    _stopped = true;\n                });\n            });\n          });\n        });\n    }\n}\n\nvoid reactor::exit(int ret) {\n    // Run stop() asynchronously on cpu 0.\n    (void)smp::submit_to(0, [this, ret] { _return = ret; stop(); });\n}\n\nuint64_t\nreactor::pending_task_count() const {\n    uint64_t ret = 0;\n    for (auto&& tq : _task_queues) {\n        if (tq) {\n            ret += tq->_q.size();\n        }\n    }\n    return ret;\n}\n\nuint64_t\nreactor::tasks_processed() const {\n    return _global_tasks_processed;\n}\n\nvoid reactor::register_metrics() {\n\n    namespace sm = seastar::metrics;\n\n    auto io_fallback_counter = [this](const sstring& reason_str, internal::thread_pool_submit_reason r) {\n        static auto reason_label = sm::label(\"reason\");\n        return sm::make_counter(\"io_threaded_fallbacks\", std::bind(&thread_pool::count, _thread_pool.get(), r),\n                sm::description(\"Total number of io-threaded-fallbacks operations\"), { reason_label(reason_str), });\n    };\n\n    _metric_groups.add_group(\"reactor\", {\n            sm::make_gauge(\"tasks_pending\", std::bind(&reactor::pending_task_count, this), sm::description(\"Number of pending tasks in the queue\")),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"tasks_processed\", std::bind(&reactor::tasks_processed, this), sm::description(\"Total tasks processed\")),\n            sm::make_counter(\"polls\", _polls, sm::description(\"Number of times pollers were executed\")),\n            sm::make_gauge(\"timers_pending\", std::bind(&decltype(_timers)::size, &_timers), sm::description(\"Number of tasks in the timer-pending queue\")),\n            sm::make_gauge(\"utilization\", [this] { return (1-_load)  * 100; }, sm::description(\"CPU utilization\")),\n            sm::make_counter(\"cpu_busy_ms\", [this] () -> int64_t { return total_busy_time() / 1ms; },\n                    sm::description(\"Total cpu busy time in milliseconds\")),\n            sm::make_counter(\"sleep_time_ms_total\", [this] () -> int64_t { return _total_sleep / 1ms; },\n                    sm::description(\"Total reactor sleep time (wall clock)\")),\n            sm::make_counter(\"awake_time_ms_total\", [this] () -> int64_t { return total_awake_time() / 1ms; },\n                    sm::description(\"Total reactor awake time (wall_clock)\")),\n            sm::make_counter(\"cpu_used_time_ms\", [this] () -> int64_t { return total_cpu_time() / 1ms; },\n                    sm::description(\"Total reactor thread CPU time (from CLOCK_THREAD_CPUTIME)\")),\n            sm::make_counter(\"cpu_steal_time_ms\", [this] () -> int64_t { return total_steal_time() / 1ms; },\n                    sm::description(\"Total steal time, the time in which something else was running while the reactor was runnable (not sleeping).\"\n                                     \"Because this is in userspace, some time that could be legitimally thought as steal time is not accounted as such. For example, if we are sleeping and can wake up but the kernel hasn't woken us up yet.\")),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"aio_reads\", _io_stats.aio_reads, sm::description(\"Total aio-reads operations\")),\n\n            sm::make_total_bytes(\"aio_bytes_read\", _io_stats.aio_read_bytes, sm::description(\"Total aio-reads bytes\")),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"aio_writes\", _io_stats.aio_writes, sm::description(\"Total aio-writes operations\")),\n            sm::make_total_bytes(\"aio_bytes_write\", _io_stats.aio_write_bytes, sm::description(\"Total aio-writes bytes\")),\n            sm::make_counter(\"aio_outsizes\", _io_stats.aio_outsizes, sm::description(\"Total number of aio operations that exceed IO limit\")),\n            sm::make_counter(\"aio_errors\", _io_stats.aio_errors, sm::description(\"Total aio errors\")),\n            sm::make_histogram(\"stalls\", sm::description(\"A histogram of reactor stall durations\"), [this] {return _stalls_histogram.to_metrics_histogram();}).aggregate({seastar::metrics::shard_label}).set_skip_when_empty(),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"fsyncs\", _io_stats.fsyncs, sm::description(\"Total number of fsync operations\")),\n            sm::make_counter(\"aio_retries\", _io_stats.aio_retries, sm::description(\"Total number of IOCB-s re-submitted via thread-pool\")),\n            // total_operations value:DERIVE:0:U\n            io_fallback_counter(\"aio_fallback\", internal::thread_pool_submit_reason::aio_fallback),\n            // total_operations value:DERIVE:0:U\n            io_fallback_counter(\"file_operation\", internal::thread_pool_submit_reason::file_operation),\n            // total_operations value:DERIVE:0:U\n            io_fallback_counter(\"process_operation\", internal::thread_pool_submit_reason::process_operation),\n    });\n\n    _metric_groups.add_group(\"memory\", {\n            sm::make_counter(\"malloc_operations\", [] { return memory::stats().mallocs(); },\n                    sm::description(\"Total number of malloc operations\")),\n            sm::make_counter(\"free_operations\", [] { return memory::stats().frees(); }, sm::description(\"Total number of free operations\")),\n            sm::make_counter(\"cross_cpu_free_operations\", [] { return memory::stats().cross_cpu_frees(); }, sm::description(\"Total number of cross cpu free\")),\n            sm::make_gauge(\"malloc_live_objects\", [] { return memory::stats().live_objects(); }, sm::description(\"Number of live objects\")),\n            sm::make_current_bytes(\"free_memory\", [] { return memory::stats().free_memory(); }, sm::description(\"Free memory size in bytes\")),\n            sm::make_current_bytes(\"total_memory\", [] { return memory::stats().total_memory(); }, sm::description(\"Total memory size in bytes\")),\n            sm::make_current_bytes(\"allocated_memory\", [] { return memory::stats().allocated_memory(); }, sm::description(\"Allocated memory size in bytes\")),\n            sm::make_counter(\"reclaims_operations\", [] { return memory::stats().reclaims(); }, sm::description(\"Total reclaims operations\")),\n            sm::make_counter(\"malloc_failed\", [] { return memory::stats().failed_allocations(); }, sm::description(\"Total count of failed memory allocations\")),\n            sm::make_counter(\"oversized_allocs\", [] { return memory::stats().large_allocations(); }, sm::description(\"Total count of oversized memory allocations\"))\n    });\n\n    _metric_groups.add_group(\"reactor\", {\n            sm::make_counter(\"logging_failures\", [] { return logging_failures; }, sm::description(\"Total number of logging failures\")),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"cpp_exceptions\", _cxx_exceptions, sm::description(\"Total number of C++ exceptions\")),\n            sm::make_counter(\"internal_errors\", internal::internal_errors, sm::description(\"Total number of internal errors (subset of cpp_exceptions) that usually indicate malfunction in the code\")),\n            sm::make_counter(\"abandoned_failed_futures\", _abandoned_failed_futures, sm::description(\"Total number of abandoned failed futures, futures destroyed while still containing an exception\")),\n    });\n\n    _metric_groups.add_group(\"reactor\", {\n        sm::make_counter(\"fstream_reads\", _io_stats.fstream_reads,\n                sm::description(\n                        \"Counts reads from disk file streams.  A high rate indicates high disk activity.\"\n                        \" Contrast with other fstream_read* counters to locate bottlenecks.\")),\n        sm::make_counter(\"fstream_read_bytes\", _io_stats.fstream_read_bytes,\n                sm::description(\n                        \"Counts bytes read from disk file streams.  A high rate indicates high disk activity.\"\n                        \" Divide by fstream_reads to determine average read size.\")),\n        sm::make_counter(\"fstream_reads_blocked\", _io_stats.fstream_reads_blocked,\n                sm::description(\n                        \"Counts the number of times a disk read could not be satisfied from read-ahead buffers, and had to block.\"\n                        \" Indicates short streams, or incorrect read ahead configuration.\")),\n        sm::make_counter(\"fstream_read_bytes_blocked\", _io_stats.fstream_read_bytes_blocked,\n                sm::description(\n                        \"Counts the number of bytes read from disk that could not be satisfied from read-ahead buffers, and had to block.\"\n                        \" Indicates short streams, or incorrect read ahead configuration.\")),\n        sm::make_counter(\"fstream_reads_aheads_discarded\", _io_stats.fstream_read_aheads_discarded,\n                sm::description(\n                        \"Counts the number of times a buffer that was read ahead of time and was discarded because it was not needed, wasting disk bandwidth.\"\n                        \" Indicates over-eager read ahead configuration.\")),\n        sm::make_counter(\"fstream_reads_ahead_bytes_discarded\", _io_stats.fstream_read_ahead_discarded_bytes,\n                sm::description(\n                        \"Counts the number of buffered bytes that were read ahead of time and were discarded because they were not needed, wasting disk bandwidth.\"\n                        \" Indicates over-eager read ahead configuration.\")),\n    });\n}\n\nseastar::internal::log_buf::inserter_iterator reactor::task_queue::do_dump(seastar::internal::log_buf::inserter_iterator it) const {\n    memory::scoped_critical_alloc_section _;\n    std::unordered_map<std::pair<std::string_view, int>, std::pair<unsigned, task*>> infos;\n    for (const auto& tp : _q) {\n        std::string_view name = tp->get_resume_point().file_name();\n        if (name.empty()) {\n            name = typeid(*tp).name();\n        }\n        auto& [count, task] = infos[{ name, tp->get_resume_point().line() }];\n        ++count;\n        task = tp;\n    }\n    it = fmt::format_to(it, \"Too long queue accumulated for {} ({} tasks)\\n\", _name, _q.size());\n    auto dump_task = [](auto it, task& task) {\n        const auto rp = task.get_resume_point();\n        const std::string_view file_name = rp.file_name();\n        return file_name.empty()\n            ? fmt::format_to(it, \"{}\\n\", typeid(task).name())\n            : fmt::format_to(it, \"{}:{}:{}\\n\", file_name, rp.line(), rp.column());\n    };\n    for (const auto& ti : infos) {\n        auto [ count, task ] = ti.second;\n        it = fmt::format_to(it, \" {}: \", count);\n        it = dump_task(it, *task);\n        for (auto* t = task->waiting_task(); t; t = t->waiting_task()) {\n            it = fmt::format_to(it, \"        \");\n            it = dump_task(it, *t);\n        }\n    }\n    return it;\n}\n\nbool reactor::task_queue::run_tasks() {\n    reactor& r = engine();\n\n    // Make sure new tasks will inherit our scheduling group\n    *internal::current_scheduling_group_ptr() = scheduling_group(_id);\n    while (!_q.empty()) {\n        auto tsk = _q.front();\n        _q.pop_front();\n        STAP_PROBE(seastar, reactor_run_tasks_single_start);\n        internal::task_histogram_add_task(*tsk);\n        r._current_task = tsk;\n        tsk->run_and_dispose();\n        r._current_task = nullptr;\n        STAP_PROBE(seastar, reactor_run_tasks_single_end);\n        ++_tasks_processed;\n        ++r._global_tasks_processed;\n        // check at end of loop, to allow at least one task to run\n        if (internal::scheduler_need_preempt()) {\n            if (_q.size() <= r._cfg.max_task_backlog) {\n                break;\n            } else {\n                // While need_preempt() is set, task execution is inefficient due to\n                // need_preempt() checks breaking out of loops and .then() calls. See\n                // #302.\n                r.reset_preemption_monitor();\n                lowres_clock::update();\n\n                static thread_local logger::rate_limit rate_limit(std::chrono::seconds(10));\n                logger::lambda_log_writer writer([this] (auto it) { return do_dump(it); });\n                seastar_logger.log(log_level::warn, rate_limit, writer);\n                if (r._cfg.abort_on_too_long_task_queue) {\n                    auto msg = fmt::format(\"Too long task queue: {}, max_task_backlog={}\", _q.size(), r._cfg.max_task_backlog);\n                    on_fatal_internal_error(seastar_logger, msg);\n                }\n            }\n        }\n    }\n\n    return !_q.empty();\n}\n\nnamespace {\n\n#ifdef SEASTAR_SHUFFLE_TASK_QUEUE\nvoid shuffle(task*& t, circular_buffer<task*>& q) {\n    static thread_local std::mt19937 gen = std::mt19937(std::default_random_engine()());\n    std::uniform_int_distribution<size_t> tasks_dist{0, q.size() - 1};\n    auto& to_swap = q[tasks_dist(gen)];\n    std::swap(to_swap, t);\n}\n#else\nvoid shuffle(task*&, circular_buffer<task*>&) {\n}\n#endif\n\n}\n\nvoid reactor::force_poll() {\n    request_preemption();\n}\n\nbool\nreactor::flush_tcp_batches() {\n    bool work = !_flush_batching.empty();\n    while (!_flush_batching.empty()) {\n        auto& os = _flush_batching.front();\n        _flush_batching.pop_front();\n        os.poll_flush();\n    }\n    return work;\n}\n\nbool\nreactor::do_expire_lowres_timers() noexcept {\n    auto now = lowres_clock::now();\n    if (now >= _lowres_next_timeout) {\n        _lowres_timers.complete(_expired_lowres_timers);\n        if (!_lowres_timers.empty()) {\n            _lowres_next_timeout = _lowres_timers.get_next_timeout();\n        } else {\n            _lowres_next_timeout = lowres_clock::time_point::max();\n        }\n        return true;\n    }\n    return false;\n}\n\nvoid\nreactor::expire_manual_timers() noexcept {\n    _manual_timers.complete(_expired_manual_timers);\n}\n\nvoid\nmanual_clock::expire_timers() noexcept {\n    local_engine->expire_manual_timers();\n}\n\nvoid\nmanual_clock::advance(manual_clock::duration d) noexcept {\n    _now.fetch_add(d.count());\n    if (local_engine) {\n        // Expire timers on all cores in the background.\n        local_engine->run_in_background([] {\n            schedule_urgent(make_task(default_scheduling_group(), &manual_clock::expire_timers));\n            return smp::invoke_on_all(&manual_clock::expire_timers);\n        });\n    }\n}\n\nbool\nreactor::do_check_lowres_timers() const noexcept {\n    return lowres_clock::now() > _lowres_next_timeout;\n}\n\nclass reactor::kernel_submit_work_pollfn final : public simple_pollfn<true> {\n    reactor& _r;\npublic:\n    kernel_submit_work_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() override final {\n        return _r._backend->kernel_submit_work();\n    }\n};\n\nclass reactor::signal_pollfn final : public reactor::pollfn {\n    reactor& _r;\npublic:\n    signal_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r._signals.poll_signal();\n    }\n    virtual bool pure_poll() override final {\n        return _r._signals.pure_poll_signal();\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        // Signals will interrupt our epoll_pwait() call, but\n        // disable them now to avoid a signal between this point\n        // and epoll_pwait()\n        sigset_t block_all;\n        sigfillset(&block_all);\n        ::pthread_sigmask(SIG_SETMASK, &block_all, &_r._active_sigmask);\n        if (poll()) {\n            // raced already, and lost\n            exit_interrupt_mode();\n            return false;\n        }\n        return true;\n    }\n    virtual void exit_interrupt_mode() override final {\n        ::pthread_sigmask(SIG_SETMASK, &_r._active_sigmask, nullptr);\n    }\n};\n\nclass reactor::batch_flush_pollfn final : public simple_pollfn<true> {\n    reactor& _r;\npublic:\n    batch_flush_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r.flush_tcp_batches();\n    }\n};\n\nclass reactor::reap_kernel_completions_pollfn final : public reactor::pollfn {\n    reactor& _r;\npublic:\n    reap_kernel_completions_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r.reap_kernel_completions();\n    }\n    virtual bool pure_poll() override final {\n        return poll(); // actually performs work, but triggers no user continuations, so okay\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        return _r._backend->kernel_events_can_sleep();\n    }\n    virtual void exit_interrupt_mode() override final {\n    }\n};\n\nclass reactor::io_queue_submission_pollfn final : public reactor::pollfn {\n    reactor& _r;\n    // Wake-up the reactor with highres timer when the io-queue\n    // decides to delay dispatching until some time point in\n    // the future\n    timer<> _nearest_wakeup { [this] { _armed = false; } };\n    bool _armed = false;\npublic:\n    io_queue_submission_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r.flush_pending_aio();\n    }\n    virtual bool pure_poll() override final {\n        return poll();\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        auto next = _r.next_pending_aio();\n        auto now = steady_clock_type::now();\n        if (next <= now) {\n            return false;\n        }\n        _nearest_wakeup.arm(next);\n        _armed = true;\n        return true;\n    }\n    virtual void exit_interrupt_mode() override final {\n        if (_armed) {\n            _nearest_wakeup.cancel();\n            _armed = false;\n        }\n    }\n};\n\n// Other cpus can queue items for us to free; and they won't notify\n// us about them.  But it's okay to ignore those items, freeing them\n// doesn't have any side effects.\n//\n// We'll take care of those items when we wake up for another reason.\nclass reactor::drain_cross_cpu_freelist_pollfn final : public simple_pollfn<true> {\npublic:\n    virtual bool poll() final override {\n        return memory::drain_cross_cpu_freelist();\n    }\n};\n\nclass reactor::lowres_timer_pollfn final : public reactor::pollfn {\n    reactor& _r;\n    // A highres timer is implemented as a waking  signal; so\n    // we arm one when we have a lowres timer during sleep, so\n    // it can wake us up.\n    timer<> _nearest_wakeup { [this] { _armed = false; } };\n    bool _armed = false;\npublic:\n    lowres_timer_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r.do_expire_lowres_timers();\n    }\n    virtual bool pure_poll() final override {\n        return _r.do_check_lowres_timers();\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        // arm our highres timer so a signal will wake us up\n        auto next = _r._lowres_next_timeout;\n        if (next == lowres_clock::time_point::max()) {\n            // no pending timers\n            return true;\n        }\n        auto now = lowres_clock::now();\n        if (next <= now) {\n            // whoops, go back\n            return false;\n        }\n        _nearest_wakeup.arm(next - now);\n        _armed = true;\n        return true;\n    }\n    virtual void exit_interrupt_mode() override final {\n        if (_armed) {\n            _nearest_wakeup.cancel();\n            _armed = false;\n        }\n    }\n};\n\nclass reactor::smp_pollfn final : public reactor::pollfn {\n    reactor& _r;\npublic:\n    smp_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        // Avoid short-circuiting with `||` since there are side effects\n        // we want to take place (instantiating tasks from the alien queue).\n        // Cast to int to silence gcc -Wbitwise-instead-of-logical.\n        return (int(smp::poll_queues()) |\n                int(_r._alien.poll_queues()));\n    }\n    virtual bool pure_poll() final override {\n        return (smp::pure_poll_queues() ||\n                _r._alien.pure_poll_queues());\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        // systemwide_memory_barrier() is very slow if run concurrently,\n        // so don't go to sleep if it is running now.\n        _r._sleeping.store(true, std::memory_order_relaxed);\n        bool barrier_done = internal::try_systemwide_memory_barrier();\n        if (!barrier_done) {\n            _r._sleeping.store(false, std::memory_order_relaxed);\n            return false;\n        }\n        if (poll()) {\n            // raced\n            _r._sleeping.store(false, std::memory_order_relaxed);\n            return false;\n        }\n        return true;\n    }\n    virtual void exit_interrupt_mode() override final {\n        _r._sleeping.store(false, std::memory_order_relaxed);\n    }\n};\n\nclass reactor::execution_stage_pollfn final : public reactor::pollfn {\n    internal::execution_stage_manager& _esm;\npublic:\n    execution_stage_pollfn() : _esm(internal::execution_stage_manager::get()) { }\n\n    virtual bool poll() override {\n        return _esm.flush();\n    }\n    virtual bool pure_poll() override {\n        return _esm.poll();\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        // This is a passive poller, so if a previous poll\n        // returned false (idle), there's no more work to do.\n        return true;\n    }\n    virtual void exit_interrupt_mode() override { }\n};\n\nclass reactor::syscall_pollfn final : public reactor::pollfn {\n    reactor& _r;\npublic:\n    syscall_pollfn(reactor& r) : _r(r) {}\n    virtual bool poll() final override {\n        return _r._thread_pool->complete();\n    }\n    virtual bool pure_poll() override final {\n        return poll(); // actually performs work, but triggers no user continuations, so okay\n    }\n    virtual bool try_enter_interrupt_mode() override {\n        _r._thread_pool->enter_interrupt_mode();\n        if (poll()) {\n            // raced\n            _r._thread_pool->exit_interrupt_mode();\n            return false;\n        }\n        return true;\n    }\n    virtual void exit_interrupt_mode() override final {\n        _r._thread_pool->exit_interrupt_mode();\n    }\n};\n\nvoid\nreactor::wakeup() {\n    if (!_sleeping.load(std::memory_order_relaxed)) {\n        return;\n    }\n\n    // We are free to clear it, because we're sending a signal now\n    _sleeping.store(false, std::memory_order_relaxed);\n\n    uint64_t one = 1;\n    auto res = ::write(_notify_eventfd.get(), &one, sizeof(one));\n    SEASTAR_ASSERT(res == sizeof(one) && \"write(2) failed on _reactor._notify_eventfd\");\n}\n\nvoid reactor::start_aio_eventfd_loop() {\n    if (!_aio_eventfd) {\n        return;\n    }\n    future<> loop_done = repeat([this] {\n        return _aio_eventfd->readable().then([this] {\n            char garbage[8];\n            std::ignore = ::read(_aio_eventfd->get_fd(), garbage, 8); // totally uninteresting\n            return _stopping ? stop_iteration::yes : stop_iteration::no;\n        });\n    });\n    // must use make_lw_shared, because at_exit expects a copyable function\n    do_at_exit([loop_done = make_lw_shared(std::move(loop_done))] {\n        return std::move(*loop_done);\n    });\n}\n\nvoid reactor::stop_aio_eventfd_loop() {\n    if (!_aio_eventfd) {\n        return;\n    }\n    uint64_t one = 1;\n    auto res = ::write(_aio_eventfd->get_fd(), &one, 8);\n    SEASTAR_ASSERT(res == 8 && \"write(2) failed on _reactor._aio_eventfd\");\n}\n\ninline\nbool\nreactor::have_more_tasks() const {\n    return _cpu_sched.active();\n}\n\ninline bool reactor::task_queue_group::active() const noexcept {\n    return _active.size() + _activating.size();\n}\n\nvoid reactor::task_queue_group::activate(sched_entity* tq) {\n    // If wakeup() was called, the task queue is likely network-bound or I/O bound, not CPU-bound. As\n    // such its vruntime will be low, and it will have a large advantage over other task queues. Limit\n    // the advantage so it doesn't dominate scheduling for a long time, in case it _does_ become CPU\n    // bound later.\n    //\n    // FIXME: different scheduling groups have different sensitivity to jitter, take advantage\n    tq->_vruntime = std::max(_last_vruntime, tq->_vruntime);\n    bool was_active = active();\n    _activating.push_back(tq);\n    if (!was_active) {\n        sched_entity::wakeup();\n    }\n}\n\nvoid reactor::task_queue_group::insert_active_entity(sched_entity* tq) {\n    tq->_active = true;\n    auto less = task_queue::indirect_compare();\n    if (_active.empty() || less(_active.back(), tq)) {\n        // Common case: idle->working\n        // Common case: CPU intensive task queue going to the back\n        _active.push_back(tq);\n    } else {\n        // Common case: newly activated queue preempting everything else\n        _active.push_front(tq);\n        // Less common case: newly activated queue behind something already active\n        size_t i = 0;\n        while (i + 1 != _active.size() && !less(_active[i], _active[i+1])) {\n            std::swap(_active[i], _active[i+1]);\n            ++i;\n        }\n    }\n}\n\nreactor::sched_entity* reactor::task_queue_group::pop_active_entity(sched_clock::time_point now) {\n    sched_entity* tq = _active.front();\n    _active.pop_front();\n    tq->_starvetime += now - tq->_ts;\n    return tq;\n}\n\nvoid\nreactor::task_queue_group::insert_activating_entities() {\n    // Quadratic, but since we expect the common cases in insert_active_entity() to dominate, faster\n    for (auto&& tq : _activating) {\n        insert_active_entity(tq);\n    }\n    _activating.clear();\n}\n\nvoid reactor::add_task(task* t) noexcept {\n    auto sg = t->group();\n    auto* q = _task_queues[sg._id].get();\n    bool was_empty = q->_q.empty();\n    q->_q.push_back(std::move(t));\n    shuffle(q->_q.back(), q->_q);\n    if (was_empty) {\n        q->wakeup();\n    }\n}\n\nvoid reactor::add_urgent_task(task* t) noexcept {\n    memory::scoped_critical_alloc_section _;\n    auto sg = t->group();\n    auto* q = _task_queues[sg._id].get();\n    bool was_empty = q->_q.empty();\n    q->_q.push_front(std::move(t));\n    shuffle(q->_q.front(), q->_q);\n    if (was_empty) {\n        q->wakeup();\n    }\n}\n\nvoid reactor::run_in_background(future<> f) {\n    try {\n        // _backgroud_gate closed in reactor::close()\n        (void)with_gate(_background_gate, [f = std::move(f)] () mutable {\n            return f.handle_exception([] (std::exception_ptr ex) {\n                seastar_logger.warn(\"Ignored background task failure: {}\", std::move(ex));\n            });\n        });\n    } catch (...) {\n        // Swallow gate_closed_exception in particular\n        seastar_logger.error(\"run_in_background: {}\", std::current_exception());\n    }\n}\n\nfuture<> reactor::drain() {\n    seastar_logger.debug(\"reactor::drain\");\n    return smp::invoke_on_all([] {\n        if (engine()._background_gate.is_closed()) {\n            return make_ready_future<>();\n        }\n        return engine()._background_gate.close();\n    });\n}\n\nvoid\nreactor::task_queue_group::run_some_tasks() {\n    if (!active()) {\n        return;\n    }\n\n    reactor& r = engine();\n    r.reset_preemption_monitor();\n    lowres_clock::update();\n\n    STAP_PROBE(seastar, reactor_run_tasks_start);\n    r._cpu_stall_detector->start_task_run(now());\n\n    run_tasks();\n\n    r._cpu_stall_detector->end_task_run(now());\n    STAP_PROBE(seastar, reactor_run_tasks_end);\n    *internal::current_scheduling_group_ptr() = default_scheduling_group(); // Prevent inheritance from last group run\n}\n\nbool reactor::task_queue_group::run_tasks() {\n    reactor& r = engine();\n\n    sched_clock::time_point t_run_completed = now();\n    while (active()) {\n        auto t_run_started = t_run_completed;\n        insert_activating_entities();\n        sched_entity* tq = pop_active_entity(t_run_started);\n        _last_vruntime = std::max(tq->_vruntime, _last_vruntime);\n        bool active = tq->run_tasks();\n        t_run_completed = now();\n        auto delta = t_run_completed - t_run_started;\n        account_runtime(r, *tq, delta);\n        tq->_ts = t_run_completed;\n        if (active) {\n            insert_active_entity(tq);\n        } else {\n            tq->_active = false;\n        }\n        // We must not use internal::scheduler_need_preempt() below,\n        // since in debug mode we'll never have two successive calls\n        // to internal::scheduler_need_preempt() both return true, so\n        // an infinite loop with yield() will never terminate.\n        //\n        // Settle on a regular need_preempt(), which will return true in\n        // debug mode.\n        if (need_preempt()) {\n            break;\n        }\n    }\n\n    return active();\n}\n\nvoid reactor::sched_entity::wakeup() {\n    if (_active) {\n        return;\n    }\n    auto now = reactor::now();\n    _waittime += now - _ts;\n    _ts = now;\n    _parent->activate(this);\n}\n\nvoid reactor::service_highres_timer() noexcept {\n    _timers.complete(_expired_timers);\n    if (auto next = _timers.get_next_timeout(); next != decltype(next)::time_point::max()) {\n        enable_timer(next);\n    }\n}\n\nint reactor::run() noexcept {\n    try {\n        return do_run();\n    } catch (const std::exception& e) {\n        seastar_logger.error(\"{}\", e.what());\n        print_with_backtrace(\"exception running reactor main loop\");\n        _exit(1);\n    }\n}\n\nint reactor::do_run() {\n    auto signal_stack = install_signal_handler_stack();\n\n    register_metrics();\n\n    // The order in which we execute the pollers is very important for performance.\n    //\n    // This is because events that are generated in one poller may feed work into others. If\n    // they were reversed, we'd only be able to do that work in the next task quota.\n    //\n    // One example is the relationship between the smp poller and the I/O submission poller:\n    // If the smp poller runs first, requests from remote I/O queues can be dispatched right away\n    //\n    // We will run the pollers in the following order:\n    //\n    // 1. SMP: any remote event arrives before anything else\n    // 2. reap kernel events completion: storage related completions may free up space in the I/O\n    //                                   queue.\n    // 4. I/O queue: must be after reap, to free up events. If new slots are freed may submit I/O\n    // 5. kernel submission: for I/O, will submit what was generated from last step.\n    // 6. reap kernel events completion: some of the submissions from last step may return immediately.\n    //                                   For example if we are dealing with poll() on a fd that has events.\n    poller smp_poller(std::make_unique<smp_pollfn>(*this));\n\n    poller reap_kernel_completions_poller(std::make_unique<reap_kernel_completions_pollfn>(*this));\n    poller io_queue_submission_poller(std::make_unique<io_queue_submission_pollfn>(*this));\n    poller kernel_submit_work_poller(std::make_unique<kernel_submit_work_pollfn>(*this));\n    poller final_real_kernel_completions_poller(std::make_unique<reap_kernel_completions_pollfn>(*this));\n\n    poller batch_flush_poller(std::make_unique<batch_flush_pollfn>(*this));\n    poller execution_stage_poller(std::make_unique<execution_stage_pollfn>());\n\n    start_aio_eventfd_loop();\n\n    if (_id == 0 && _cfg.auto_handle_sigint_sigterm) {\n       if (_cfg.handle_sigint) {\n          _signals.handle_signal_once(SIGINT, [this] { stop(); });\n       }\n       _signals.handle_signal_once(SIGTERM, [this] { stop(); });\n    }\n\n    if (_id == 0) {\n        // Start initialization in the background.\n        // Wait for network stack to appear on all cpus.\n        // Communicate when done using _start_promise\n        (void)smp::invoke_on_all([] {\n            return engine()._network_stack_ready->then([] (std::unique_ptr<network_stack> stack) {\n                engine()._network_stack = std::move(stack);\n            });\n        }).then([] {\n            return smp::invoke_on_all([] {\n                return engine()._network_stack->initialize().then([] {\n                    engine()._start_promise.set_value();\n                });\n            });\n        });\n    }\n\n    poller syscall_poller(std::make_unique<syscall_pollfn>(*this));\n\n    poller drain_cross_cpu_freelist(std::make_unique<drain_cross_cpu_freelist_pollfn>());\n\n    poller expire_lowres_timers(std::make_unique<lowres_timer_pollfn>(*this));\n    poller sig_poller(std::make_unique<signal_pollfn>(*this));\n\n    using namespace std::chrono_literals;\n    auto last_idle = _total_idle;\n    auto idle_start = now(), idle_end = idle_start;\n    _load_timer.set_callback([this, &last_idle, &idle_start, &idle_end] () mutable {\n        _total_idle += idle_end - idle_start;\n        auto load = double((_total_idle - last_idle).count()) / double(std::chrono::duration_cast<sched_clock::duration>(1s).count());\n        last_idle = _total_idle;\n        load = std::min(load, 1.0);\n        idle_start = idle_end;\n        _load -= _loads.back() / loads_size;\n        _loads.pop_back();\n        _loads.push_front(load);\n        _load += (load / loads_size);\n    });\n    _load_timer.arm_periodic(1s);\n\n    itimerspec its = seastar::posix::to_relative_itimerspec(_cfg.task_quota, _cfg.task_quota);\n    _task_quota_timer.timerfd_settime(0, its);\n    auto& task_quote_itimerspec = its;\n\n    struct sigaction sa_block_notifier = {};\n    sa_block_notifier.sa_handler = &reactor::block_notifier;\n    sa_block_notifier.sa_flags = SA_RESTART;\n    auto r = sigaction(internal::cpu_stall_detector::signal_number(), &sa_block_notifier, nullptr);\n    SEASTAR_ASSERT(r == 0);\n\n    bool idle = false;\n\n    auto check_for_work = [this] () {\n        return poll_once() || have_more_tasks();\n    };\n    const noncopyable_function<bool()> pure_check_for_work = [this] () {\n        return pure_poll_once() || have_more_tasks();\n    };\n    while (true) {\n        _cpu_sched.run_some_tasks();\n        if (_stopped) {\n            break;\n        }\n\n        _polls++;\n\n        lowres_clock::update(); // Don't delay expiring lowres timers\n        if (check_for_work()) {\n            if (idle) {\n                _total_idle += idle_end - idle_start;\n                idle_start = idle_end;\n                idle = false;\n            }\n        } else {\n            idle_end = now();\n            if (!idle) {\n                idle_start = idle_end;\n                idle = true;\n            }\n            bool go_to_sleep = true;\n            try {\n                // we can't run check_for_work(), because that can run tasks in the context\n                // of the idle handler which change its state, without the idle handler expecting\n                // it.  So run pure_check_for_work() instead.\n                auto handler_result = _idle_cpu_handler(pure_check_for_work);\n                go_to_sleep = handler_result == idle_cpu_handler_result::no_more_work;\n            } catch (...) {\n                report_exception(\"Exception while running idle cpu handler\", std::current_exception());\n            }\n            if (go_to_sleep) {\n                internal::cpu_relax();\n                if (idle_end - idle_start > _cfg.max_poll_time) {\n                    if (pollers_enter_interrupt_mode()) {\n                        // Turn off the task quota timer to avoid spurious wakeups\n                        struct itimerspec zero_itimerspec = {};\n                        _task_quota_timer.timerfd_settime(0, zero_itimerspec);\n                        _cpu_stall_detector->start_sleep();\n\n                        wait_and_process_events();\n                        pollers_exit_interrupt_mode();\n\n                        _cpu_stall_detector->end_sleep();\n                        // We may have slept for a while, so freshen idle_end\n                        idle_end = now();\n                        _task_quota_timer.timerfd_settime(0, task_quote_itimerspec);\n                    }\n                }\n            } else {\n                // We previously ran pure_check_for_work(), might not actually have performed\n                // any work.\n                check_for_work();\n            }\n        }\n    }\n\n    _load_timer.cancel();\n    // Final tasks may include sending the last response to cpu 0, so run them\n    while (have_more_tasks()) {\n        _cpu_sched.run_some_tasks();\n    }\n    while (_at_destroy_tasks->run_tasks()) {\n        // keep running while it's active\n    }\n    _finished_running_tasks = true;\n    _smp->arrive_at_event_loop_end();\n    if (_id == 0) {\n        _smp->join_all();\n    }\n    // To prevent ordering issues from rising, destroy the I/O queue explicitly at this point.\n    // This is needed because the reactor is destroyed from the thread_local destructors. If\n    // the I/O queue happens to use any other infrastructure that is also kept this way (for\n    // instance, collectd), we will not have any way to guarantee who is destroyed first.\n    _io_queues.clear();\n    return _return;\n}\n\nbool\nreactor::pollers_enter_interrupt_mode() {\n    for (auto i = _pollers.begin(); i != _pollers.end(); ++i) {\n        auto ok = (*i)->try_enter_interrupt_mode();\n        if (!ok) {\n            while (i != _pollers.begin()) {\n                (*--i)->exit_interrupt_mode();\n            }\n            return false;\n        }\n    }\n\n    return true;\n}\n\nvoid\nreactor::pollers_exit_interrupt_mode() {\n    for (auto i = _pollers.rbegin(); i != _pollers.rend(); ++i) {\n        (*i)->exit_interrupt_mode();\n    }\n}\n\nvoid\nreactor::wait_and_process_events() {\n    _backend->wait_and_process_events(&_active_sigmask);\n}\n\nbool\nreactor::poll_once() {\n    bool work = false;\n    for (auto c : _pollers) {\n        work |= c->poll();\n    }\n\n    return work;\n}\n\nbool\nreactor::pure_poll_once() {\n    for (auto c : _pollers) {\n        if (c->pure_poll()) {\n            return true;\n        }\n    }\n    return false;\n}\n\nnamespace internal {\n\nclass poller::registration_task final : public task {\nprivate:\n    poller* _p;\npublic:\n    explicit registration_task(poller* p) : _p(p) {}\n    virtual void run_and_dispose() noexcept override {\n        if (_p) {\n            engine().register_poller(_p->_pollfn.get());\n            _p->_registration_task = nullptr;\n        }\n        delete this;\n    }\n    task* waiting_task() noexcept override { return nullptr; }\n    void cancel() {\n        _p = nullptr;\n    }\n    void moved(poller* p) {\n        _p = p;\n    }\n};\n\nclass poller::deregistration_task final : public task {\nprivate:\n    std::unique_ptr<pollfn> _p;\npublic:\n    explicit deregistration_task(std::unique_ptr<pollfn>&& p) : _p(std::move(p)) {}\n    virtual void run_and_dispose() noexcept override {\n        engine().unregister_poller(_p.get());\n        delete this;\n    }\n    task* waiting_task() noexcept override { return nullptr; }\n};\n\n}\n\nvoid reactor::register_poller(pollfn* p) {\n    _pollers.push_back(p);\n}\n\nvoid reactor::unregister_poller(pollfn* p) {\n    _pollers.erase(std::find(_pollers.begin(), _pollers.end(), p));\n}\n\nvoid reactor::replace_poller(pollfn* old, pollfn* neww) {\n    std::replace(_pollers.begin(), _pollers.end(), old, neww);\n}\n\nnamespace internal {\n\npoller::poller(poller&& x) noexcept\n        : _pollfn(std::move(x._pollfn)), _registration_task(std::exchange(x._registration_task, nullptr)) {\n    if (_pollfn && _registration_task) {\n        _registration_task->moved(this);\n    }\n}\n\npoller&\npoller::operator=(poller&& x) noexcept {\n    if (this != &x) {\n        this->~poller();\n        new (this) poller(std::move(x));\n    }\n    return *this;\n}\n\nvoid\npoller::do_register() noexcept {\n    // We can't just insert a poller into reactor::_pollers, because we\n    // may be running inside a poller ourselves, and so in the middle of\n    // iterating reactor::_pollers itself.  So we schedule a task to add\n    // the poller instead.\n    auto task = new registration_task(this);\n    engine().add_task(task);\n    _registration_task = task;\n}\n\npoller::~poller() {\n    // We can't just remove the poller from reactor::_pollers, because we\n    // may be running inside a poller ourselves, and so in the middle of\n    // iterating reactor::_pollers itself.  So we schedule a task to remove\n    // the poller instead.\n    //\n    // Since we don't want to call the poller after we exit the destructor,\n    // we replace it atomically with another one, and schedule a task to\n    // delete the replacement.\n    if (_pollfn) {\n        if (_registration_task) {\n            // not added yet, so don't do it at all.\n            _registration_task->cancel();\n        } else if (!engine()._finished_running_tasks) {\n            // If _finished_running_tasks, the call to add_task() below will just\n            // leak it, since no one will call task::run_and_dispose(). Just leave\n            // the poller there, the reactor will never use it.\n            auto dummy = make_pollfn([] { return false; });\n            auto dummy_p = dummy.get();\n            auto task = new deregistration_task(std::move(dummy));\n            engine().add_task(task);\n            engine().replace_poller(_pollfn.get(), dummy_p);\n        }\n    }\n}\n\n}\n\nsyscall_work_queue::syscall_work_queue()\n    : _pending()\n    , _completed()\n    , _start_eventfd(0) {\n}\n\nvoid syscall_work_queue::submit_item(std::unique_ptr<syscall_work_queue::work_item> item) {\n    (void)_queue_has_room.wait().then_wrapped([this, item = std::move(item)] (future<> f) mutable {\n        // propagate wait failure via work_item\n        if (f.failed()) {\n            item->set_exception(f.get_exception());\n            return;\n        }\n        _pending.push(item.release());\n        _start_eventfd.signal(1);\n    });\n}\n\nunsigned syscall_work_queue::complete() {\n    std::array<work_item*, queue_length> tmp_buf;\n    auto end = tmp_buf.data();\n    auto nr = _completed.consume_all([&] (work_item* wi) {\n        *end++ = wi;\n    });\n    for (auto p = tmp_buf.data(); p != end; ++p) {\n        auto wi = *p;\n        wi->complete();\n        delete wi;\n    }\n    _queue_has_room.signal(nr);\n    return nr;\n}\n\n\nsmp_message_queue::smp_message_queue(reactor* from, reactor* to)\n    : _pending(to)\n    , _completed(from)\n{\n}\n\nsmp_message_queue::~smp_message_queue()\n{\n    if (_pending.remote != _completed.remote) {\n        _tx.a.~aa();\n    }\n}\n\nvoid smp_message_queue::stop() {\n    _metrics.clear();\n}\n\nvoid smp_message_queue::move_pending() {\n    auto begin = _tx.a.pending_fifo.cbegin();\n    auto end = _tx.a.pending_fifo.cend();\n    end = _pending.push(begin, end);\n    if (begin == end) {\n        return;\n    }\n    auto nr = end - begin;\n    _pending.maybe_wakeup();\n    _tx.a.pending_fifo.erase(begin, end);\n    _current_queue_length += nr;\n    _last_snt_batch = nr;\n    _sent += nr;\n}\n\nbool smp_message_queue::pure_poll_tx() const {\n    // can't use read_available(), not available on older boost\n    // empty() is not const, so need const_cast.\n    return !const_cast<lf_queue&>(_completed).empty();\n}\n\nvoid smp_message_queue::submit_item(shard_id t, smp_timeout_clock::time_point timeout, std::unique_ptr<smp_message_queue::work_item> item) {\n  // matching signal() in process_completions()\n  auto ssg_id = internal::smp_service_group_id(item->ssg);\n  auto& sem = get_smp_service_groups_semaphore(ssg_id, t);\n  // Future indirectly forwarded to `item`.\n  (void)get_units(sem, 1, timeout).then_wrapped([this, item = std::move(item)] (future<smp_service_group_semaphore_units> units_fut) mutable {\n    if (units_fut.failed()) {\n        item->fail_with(units_fut.get_exception());\n        ++_compl;\n        ++_last_cmpl_batch;\n        return;\n    }\n    _tx.a.pending_fifo.push_back(item.get());\n    // no exceptions from this point\n    item.release();\n    units_fut.get().release();\n    if (_tx.a.pending_fifo.size() >= batch_size) {\n        move_pending();\n    }\n  });\n}\n\nvoid smp_message_queue::respond(work_item* item) {\n    _completed_fifo.push_back(item);\n    if (_completed_fifo.size() >= batch_size || engine().stopped()) {\n        flush_response_batch();\n    }\n}\n\nvoid smp_message_queue::flush_response_batch() {\n    if (!_completed_fifo.empty()) {\n        auto begin = _completed_fifo.cbegin();\n        auto end = _completed_fifo.cend();\n        end = _completed.push(begin, end);\n        if (begin == end) {\n            return;\n        }\n        _completed.maybe_wakeup();\n        _completed_fifo.erase(begin, end);\n    }\n}\n\nbool smp_message_queue::has_unflushed_responses() const {\n    return !_completed_fifo.empty();\n}\n\nbool smp_message_queue::pure_poll_rx() const {\n    // can't use read_available(), not available on older boost\n    // empty() is not const, so need const_cast.\n    return !const_cast<lf_queue&>(_pending).empty();\n}\n\nvoid\nsmp_message_queue::lf_queue::maybe_wakeup() {\n    // Called after lf_queue_base::push().\n    //\n    // This is read-after-write, which wants memory_order_seq_cst,\n    // but we insert that barrier using systemwide_memory_barrier()\n    // because seq_cst is so expensive.\n    //\n    // However, we do need a compiler barrier:\n    std::atomic_signal_fence(std::memory_order_seq_cst);\n    remote->wakeup();\n}\n\nsmp_message_queue::lf_queue::~lf_queue() {\n    consume_all([] (work_item* ptr) {\n        delete ptr;\n    });\n}\n\n\ntemplate<size_t PrefetchCnt, typename Func>\nsize_t smp_message_queue::process_queue(lf_queue& q, Func process) {\n    // copy batch to local memory in order to minimize\n    // time in which cross-cpu data is accessed\n    work_item* items[queue_length + PrefetchCnt];\n    work_item* wi;\n    if (!q.pop(wi))\n        return 0;\n    // start prefetching first item before popping the rest to overlap memory\n    // access with potential cache miss the second pop may cause\n    prefetch<2>(wi);\n    auto nr = q.pop(items);\n    std::fill(std::begin(items) + nr, std::begin(items) + nr + PrefetchCnt, nr ? items[nr - 1] : wi);\n    unsigned i = 0;\n    do {\n        prefetch_n<2>(std::begin(items) + i, std::begin(items) + i + PrefetchCnt);\n        process(wi);\n        wi = items[i++];\n    } while(i <= nr);\n\n    return nr + 1;\n}\n\nsize_t smp_message_queue::process_completions(shard_id t) {\n    auto nr = process_queue<prefetch_cnt*2>(_completed, [t] (work_item* wi) {\n        wi->complete();\n        auto ssg_id = internal::smp_service_group_id(wi->ssg);\n        get_smp_service_groups_semaphore(ssg_id, t).signal();\n        delete wi;\n    });\n    _current_queue_length -= nr;\n    _compl += nr;\n    _last_cmpl_batch = nr;\n\n    return nr;\n}\n\nvoid smp_message_queue::flush_request_batch() {\n    if (!_tx.a.pending_fifo.empty()) {\n        move_pending();\n    }\n}\n\nsize_t smp_message_queue::process_incoming() {\n    auto nr = process_queue<prefetch_cnt>(_pending, [] (work_item* wi) {\n        wi->process();\n    });\n    _received += nr;\n    _last_rcv_batch = nr;\n    return nr;\n}\n\nvoid smp_message_queue::start(unsigned cpuid) {\n    _tx.init();\n    namespace sm = seastar::metrics;\n    char instance[10];\n    std::snprintf(instance, sizeof(instance), \"%u-%u\", this_shard_id(), cpuid);\n    _metrics.add_group(\"smp\", {\n            // queue_length     value:GAUGE:0:U\n            // Absolute value of num packets in last tx batch.\n            sm::make_queue_length(\"send_batch_queue_length\", _last_snt_batch, sm::description(\"Current send batch queue length\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            sm::make_queue_length(\"receive_batch_queue_length\", _last_rcv_batch, sm::description(\"Current receive batch queue length\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            sm::make_queue_length(\"complete_batch_queue_length\", _last_cmpl_batch, sm::description(\"Current complete batch queue length\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            sm::make_queue_length(\"send_queue_length\", _current_queue_length, sm::description(\"Current send queue length\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"total_received_messages\", _received, sm::description(\"Total number of received messages\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"total_sent_messages\", _sent, sm::description(\"Total number of sent messages\"), {sm::shard_label(instance)})(sm::metric_disabled),\n            // total_operations value:DERIVE:0:U\n            sm::make_counter(\"total_completed_messages\", _compl, sm::description(\"Total number of messages completed\"), {sm::shard_label(instance)})(sm::metric_disabled)\n    });\n}\n\nreadable_eventfd writeable_eventfd::read_side() {\n    return readable_eventfd(_fd.dup());\n}\n\nfile_desc writeable_eventfd::try_create_eventfd(size_t initial) {\n    SEASTAR_ASSERT(size_t(int(initial)) == initial);\n    return file_desc::eventfd(initial, EFD_CLOEXEC);\n}\n\nvoid writeable_eventfd::signal(size_t count) {\n    uint64_t c = count;\n    auto r = _fd.write(&c, sizeof(c));\n    SEASTAR_ASSERT(r == sizeof(c));\n}\n\nwriteable_eventfd readable_eventfd::write_side() {\n    return writeable_eventfd(_fd.get_file_desc().dup());\n}\n\nfile_desc readable_eventfd::try_create_eventfd(size_t initial) {\n    SEASTAR_ASSERT(size_t(int(initial)) == initial);\n    return file_desc::eventfd(initial, EFD_CLOEXEC | EFD_NONBLOCK);\n}\n\nfuture<size_t> readable_eventfd::wait() {\n    return _fd.readable().then([this] {\n        uint64_t count;\n        int r = ::read(_fd.get_fd(), &count, sizeof(count));\n        SEASTAR_ASSERT(r == sizeof(count));\n        return make_ready_future<size_t>(count);\n    });\n}\n\nvoid schedule(task* t) noexcept {\n    engine().add_task(t);\n}\n\nvoid schedule_checked(task* t) noexcept {\n    if (t->group().is_at_exit()) {\n        // trying to schedule a task in at_destroy. Not allowed\n        on_internal_error(seastar_logger, \"Cannot schedule tasks in at_destroy queue. Use reactor::at_destroy.\");\n    }\n    engine().add_task(t);\n}\n\nvoid schedule_urgent(task* t) noexcept {\n    engine().add_urgent_task(t);\n}\n\n}\n\nbool operator==(const ::sockaddr_in a, const ::sockaddr_in b) {\n    return (a.sin_addr.s_addr == b.sin_addr.s_addr) && (a.sin_port == b.sin_port);\n}\n\nnamespace seastar {\n\nstatic bool kernel_supports_aio_fsync() {\n    return internal::kernel_uname().whitelisted({\"4.18\"});\n}\n\nstatic std::tuple<std::filesystem::path, uint64_t> wakeup_granularity() {\n    auto try_read = [] (auto path) -> uint64_t {\n        try {\n            return read_first_line_as<uint64_t>(path);\n        } catch (...) {\n            return 0;\n        }\n    };\n\n    auto legacy_path = \"/proc/sys/kernel/sched_wakeup_granularity_ns\";\n    if (auto val = try_read(legacy_path); val) {\n        return {legacy_path, val};\n    }\n\n    // This will in practice almost always fail because debug fs requires root\n    // perms to read so we are out of luck\n    auto debug_fs_path = \"/sys/kernel/debug/sched/wakeup_granularity_ns\";\n    if (auto val = try_read(debug_fs_path); val) {\n        return {debug_fs_path, val};\n    }\n\n    return {\"\", 0};\n}\n\nstatic program_options::selection_value<network_stack_factory> create_network_stacks_option(reactor_options& zis) {\n    using value_type = program_options::selection_value<network_stack_factory>;\n    value_type::candidates candidates;\n    std::vector<std::string> net_stack_names;\n\n    auto deleter = [] (network_stack_factory* p) { delete p; };\n\n    std::string default_stack;\n    for (auto reg_func : {register_native_stack, register_posix_stack}) {\n        auto s = reg_func();\n        if (s.is_default) {\n            default_stack = s.name;\n        }\n        candidates.push_back({s.name, {new network_stack_factory(std::move(s.factory)), deleter}, std::move(s.opts)});\n        net_stack_names.emplace_back(s.name);\n    }\n\n    return program_options::selection_value<network_stack_factory>(zis, \"network-stack\", std::move(candidates), default_stack,\n            fmt::format(\"select network stack (valid values: {})\", fmt::join(net_stack_names, \", \")));\n}\n\nstatic program_options::selection_value<reactor_backend_selector>::candidates backend_selector_candidates() {\n    using value_type = program_options::selection_value<reactor_backend_selector>;\n    value_type::candidates candidates;\n\n    auto deleter = [] (reactor_backend_selector* p) { delete p; };\n\n    for (auto&& be : reactor_backend_selector::available()) {\n        auto name = be.name();\n        candidates.push_back({std::move(name), {new reactor_backend_selector(std::move(be)), deleter}, {}});\n    }\n    return candidates;\n}\n\nreactor_options::reactor_options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"Core options\")\n    , network_stack(create_network_stacks_option(*this))\n    , poll_mode(*this, \"poll-mode\", \"poll continuously (100% cpu use)\")\n    , idle_poll_time_us(*this, \"idle-poll-time-us\", reactor::calculate_poll_time() / 1us,\n                \"idle polling time in microseconds (reduce for overprovisioned environments or laptops)\")\n    , poll_aio(*this, \"poll-aio\", true,\n                \"busy-poll for disk I/O (reduces latency and increases throughput)\")\n    , task_quota_ms(*this, \"task-quota-ms\", 0.5, \"Max time (ms) between polls\")\n    , io_latency_goal_ms(*this, \"io-latency-goal-ms\", {}, \"Max time (ms) io operations must take (1.5 * task-quota-ms if not set)\")\n    , io_flow_ratio_threshold(*this, \"io-flow-rate-threshold\", 1.1, \"Dispatch rate to completion rate threshold\")\n    , io_completion_notify_ms(*this, \"io-completion-notify-ms\", {}, \"Threshold in milliseconds over which IO request completion is reported to logs\")\n    , max_task_backlog(*this, \"max-task-backlog\", 1000, \"Maximum number of task backlog to allow; above this we ignore I/O\")\n    , blocked_reactor_notify_ms(*this, \"blocked-reactor-notify-ms\", 25, \"threshold in miliseconds over which the reactor is considered blocked if no progress is made\")\n    , blocked_reactor_reports_per_minute(*this, \"blocked-reactor-reports-per-minute\", 5, \"Maximum number of backtraces reported by stall detector per minute\")\n    , blocked_reactor_report_format_oneline(*this, \"blocked-reactor-report-format-oneline\", true, \"Print a simplified backtrace on a single line\")\n    , relaxed_dma(*this, \"relaxed-dma\", \"allow using buffered I/O if DMA is not available (reduces performance)\")\n    , linux_aio_nowait(*this, \"linux-aio-nowait\", internal::kernel_uname().whitelisted({\"4.13\"}), // base version where this works\n                \"use the Linux NOWAIT AIO feature, which reduces reactor stalls due to aio (autodetected)\")\n    , unsafe_bypass_fsync(*this, \"unsafe-bypass-fsync\", false, \"Bypass fsync(), may result in data loss. Use for testing on consumer drives\")\n    , kernel_page_cache(*this, \"kernel-page-cache\", false,\n                \"Use the kernel page cache. This disables DMA (O_DIRECT).\"\n                \" Useful for short-lived functional tests with a small data set.\")\n    , overprovisioned(*this, \"overprovisioned\", \"run in an overprovisioned environment (such as docker or a laptop); equivalent to --idle-poll-time-us 0 --thread-affinity 0 --poll-aio 0\")\n    , abort_on_seastar_bad_alloc(*this, \"abort-on-seastar-bad-alloc\", \"abort when seastar allocator cannot allocate memory\")\n    , abort_on_too_long_task_queue(*this, \"abort-on-too-long-task-queue\", false, \"abort when the task queue is too long\")\n    , force_aio_syscalls(*this, \"force-aio-syscalls\", false,\n                \"Force io_getevents(2) to issue a system call, instead of bypassing the kernel when possible.\"\n                \" This makes strace output more useful, but slows down the application\")\n    , dump_memory_diagnostics_on_alloc_failure_kind(*this, \"dump-memory-diagnostics-on-alloc-failure-kind\", memory::alloc_failure_kind::critical,\n                \"Dump diagnostics of the seastar allocator state on allocation failure.\"\n                 \" Accepted values: none, critical (default), all. When set to critical, only allocations marked as critical will trigger diagnostics dump.\"\n                 \" The diagnostics will be written to the seastar_memory logger, with error level.\"\n                 \" Note that if the seastar_memory logger is set to debug or trace level, the diagnostics will be logged irrespective of this setting.\")\n    , reactor_backend(*this, \"reactor-backend\", backend_selector_candidates(), reactor_backend_selector::default_backend().name(),\n                fmt::format(\"Internal reactor implementation ({})\", reactor_backend_selector::available()))\n    , aio_fsync(*this, \"aio-fsync\", kernel_supports_aio_fsync(),\n                \"Use Linux aio for fsync() calls. This reduces latency; requires Linux 4.18 or later.\")\n    , max_networking_io_control_blocks(*this, \"max-networking-io-control-blocks\", 10000,\n                \"Maximum number of I/O control blocks (IOCBs) to allocate per shard. This translates to the number of sockets supported per shard.\"\n                \" Requires tuning /proc/sys/fs/aio-max-nr. Only valid for the linux-aio reactor backend (see --reactor-backend).\")\n    , reserve_io_control_blocks(*this, \"reserve-io-control-blocks\", 0,\n                \"Reserve this many IOCBs, so it is available to any side application that runs parallel to the seastar appliation.\"\n                \" Takes precedence over --max-networking-io-control-blocks. Only valid for the linux-aio reactor backend (see --reactor-backend).\")\n#ifdef SEASTAR_HEAPPROF\n    , heapprof(*this, \"heapprof\", 0, \"Enable seastar heap profiling. Sample every ARG bytes. 0 means off\")\n#else\n    , heapprof(*this, \"heapprof\", program_options::unused{})\n#endif\n    , no_handle_interrupt(*this, \"no-handle-interrupt\", \"ignore SIGINT (for gdb)\")\n{\n}\n\nsmp_options::smp_options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"SMP options\")\n    , smp(*this, \"smp\", {}, \"number of threads (default: one per CPU)\")\n    , cpuset(*this, \"cpuset\", {}, \"CPUs to use (in cpuset(7) list format (ex: 0,1-3,7); default: all))\")\n    , memory(*this, \"memory\", std::nullopt, \"memory to use, in bytes (ex: 4G) (default: all)\")\n    , reserve_memory(*this, \"reserve-memory\", {}, \"memory reserved to OS (if --memory not specified)\")\n    , hugepages(*this, \"hugepages\", {}, \"path to accessible hugetlbfs mount (typically /dev/hugepages/something)\")\n    , lock_memory(*this, \"lock-memory\", {}, \"lock all memory (prevents swapping)\")\n    , thread_affinity(*this, \"thread-affinity\", true, \"pin threads to their cpus (disable for overprovisioning)\")\n#ifdef SEASTAR_HAVE_HWLOC\n    , num_io_groups(*this, \"num-io-groups\", {}, \"Number of IO groups. Each IO group will be responsible for a fraction of the IO requests. Defaults to the number of NUMA nodes\")\n#else\n    , num_io_groups(*this, \"num-io-groups\", program_options::unused{})\n#endif\n    , io_properties_file(*this, \"io-properties-file\", {}, \"path to a YAML file describing the characteristics of the I/O Subsystem\")\n    , io_properties(*this, \"io-properties\", {}, \"a YAML string describing the characteristics of the I/O Subsystem\")\n    , mbind(*this, \"mbind\", true, \"enable mbind\")\n#ifndef SEASTAR_NO_EXCEPTION_HACK\n    , enable_glibc_exception_scaling_workaround(*this, \"enable-glibc-exception-scaling-workaround\", true, \"enable workaround for glibc/gcc c++ exception scalablity problem\")\n#else\n    , enable_glibc_exception_scaling_workaround(*this, \"enable-glibc-exception-scaling-workaround\", program_options::unused{})\n#endif\n#ifdef SEASTAR_HAVE_HWLOC\n    , allow_cpus_in_remote_numa_nodes(*this, \"allow-cpus-in-remote-numa-nodes\", true, \"if some CPUs are found not to have any local NUMA nodes, allow assigning them to remote ones\")\n#else\n    , allow_cpus_in_remote_numa_nodes(*this, \"allow-cpus-in-remote-numa-nodes\", program_options::unused{})\n#endif\n{\n}\n\nstruct reactor_deleter {\n    void operator()(reactor* p) {\n        p->~reactor();\n        free(p);\n    }\n};\n\nthread_local std::unique_ptr<reactor, reactor_deleter> reactor_holder;\n\nthread_local smp_message_queue** smp::_qs;\nthread_local std::thread::id smp::_tmain;\nunsigned smp::count = 0;\n\nvoid smp::start_all_queues()\n{\n    for (unsigned c = 0; c < count; c++) {\n        if (c != this_shard_id()) {\n            _qs[c][this_shard_id()].start(c);\n        }\n    }\n    _alien._qs[this_shard_id()].start();\n}\n\n#ifdef SEASTAR_HAVE_DPDK\n\nint dpdk_thread_adaptor(void* f)\n{\n    (*static_cast<std::function<void ()>*>(f))();\n    return 0;\n}\n\n#endif\n\nvoid smp::join_all()\n{\n#ifdef SEASTAR_HAVE_DPDK\n    if (_using_dpdk) {\n        rte_eal_mp_wait_lcore();\n        return;\n    }\n#endif\n    for (auto&& t: smp::_threads) {\n        t.join();\n    }\n}\n\nvoid smp::pin(unsigned cpu_id) {\n    if (_using_dpdk) {\n        // dpdk does its own pinning\n        return;\n    }\n    pin_this_thread(cpu_id);\n}\n\nvoid smp::arrive_at_event_loop_end() {\n    if (_all_event_loops_done) {\n        _all_event_loops_done->arrive_and_wait();\n    }\n}\n\nvoid smp::allocate_reactor(unsigned id, reactor_backend_selector rbs, reactor_config cfg) {\n    SEASTAR_ASSERT(!reactor_holder);\n\n    // we cannot just write \"local_engin = new reactor\" since reactor's constructor\n    // uses local_engine\n    void *buf;\n    int r = posix_memalign(&buf, cache_line_size, sizeof(reactor));\n    SEASTAR_ASSERT(r == 0);\n    *internal::this_shard_id_ptr() = id;\n    local_engine = new (buf) reactor(this->shared_from_this(), _alien, id, std::move(rbs), cfg);\n    reactor_holder.reset(local_engine);\n}\n\nvoid smp::cleanup() noexcept {\n    smp::_threads = std::vector<posix_thread>();\n    _thread_loops.clear();\n    _shard_to_numa_node_mapping = decltype(_shard_to_numa_node_mapping)();\n    reactor_holder.reset();\n    local_engine = nullptr;\n}\n\nvoid smp::cleanup_cpu() {\n    size_t cpuid = this_shard_id();\n\n    if (_qs) {\n        for(unsigned i = 0; i < smp::count; i++) {\n            _qs[i][cpuid].stop();\n        }\n    }\n    if (_alien._qs) {\n        _alien._qs[cpuid].stop();\n    }\n}\n\nvoid smp::create_thread(std::function<void ()> thread_loop) {\n    if (_using_dpdk) {\n        _thread_loops.push_back(std::move(thread_loop));\n    } else {\n        _threads.emplace_back(std::move(thread_loop));\n    }\n}\n\n// Installs handler for Signal which ensures that Func is invoked only once\n// in the whole program and that after it is invoked the default handler is restored.\ntemplate<int Signal, void(*Func)(siginfo_t *, ucontext_t *)>\nvoid install_oneshot_signal_handler() {\n    static bool handled = false;\n    static util::spinlock lock;\n\n    struct sigaction sa;\n    sa.sa_sigaction = [](int sig, siginfo_t *info, void *p) {\n        std::lock_guard<util::spinlock> g(lock);\n        if (!handled) {\n            handled = true;\n            signal(sig, SIG_DFL);\n            Func(info, (ucontext_t *)p);\n        }\n    };\n    sigfillset(&sa.sa_mask);\n    sa.sa_flags = SA_SIGINFO | SA_RESTART;\n    if (Signal == SIGSEGV) {\n        sa.sa_flags |= SA_ONSTACK;\n    }\n    auto r = ::sigaction(Signal, &sa, nullptr);\n    throw_system_error_on(r == -1);\n}\n\nstatic void reraise_signal(int signo) {\n    signal(signo, SIG_DFL);\n    pthread_kill(pthread_self(), signo);\n}\n\nstatic void sigsegv_action(siginfo_t *info, ucontext_t* uc) noexcept {\n    print_safe(\"Segmentation fault: si_code: \");\n    auto code = info->si_code;\n    print_decimal_safe(static_cast<unsigned>(info->si_code));\n\n    if (code == SI_USER) {\n        print_safe(\", si_pid: \");\n        // print the pid in the case the signal was sent by someone else\n        print_decimal_safe(static_cast<unsigned>(info->si_pid));\n    } else if (code == SEGV_MAPERR || code == SEGV_ACCERR || code == SEGV_BNDERR) {\n        // print the address of the data access\n        print_safe(\", si_addr: \");\n        print_zero_padded_hex_safe(reinterpret_cast<uintptr_t>(info->si_addr));\n    }\n\n\n    uintptr_t ip;\n    if (uc) {\n#if defined(__x86_64__)\n        ip = uc->uc_mcontext.gregs[REG_RIP];\n#elif defined(__aarch64__)\n        ip = uc->uc_mcontext.pc;\n#else\n        ip = 0xBAD;\n#endif\n    } else {\n        ip = 0xBAD2;\n    }\n    print_safe(\", ip: \");\n    print_zero_padded_hex_safe(ip);\n    print_safe(\"\\n\");\n\n    // Print the resolved IP, i.e., suitable for use\n    // with addr2line and other tools which expect an\n    // address without any added offset (from ASLR or\n    // because it's a relocated shared object).\n    print_safe(\"Segmentation fault: resolved ip: 0x\");\n    auto f = decorate(ip);\n    print_zero_padded_hex_safe(f.addr);\n    print_safe(\" in \");\n    print_safe(f.so->name.c_str());\n    print_safe(\"[0x\");\n    print_zero_padded_hex_safe(f.so->begin);\n    print_safe(\"+0x\");\n    print_zero_padded_hex_safe(f.so->end - f.so->begin);\n    print_safe(\"]\\n\");\n\n    // print the backtrace in immediate mode, so if we crash\n    // during the backtrace we get as much output as possible\n    print_with_backtrace(\"Segmentation fault\", false, true);\n    reraise_signal(SIGSEGV);\n}\n\nstatic void sigabrt_action(siginfo_t *info, ucontext_t* uc) noexcept {\n    print_with_backtrace(\"Aborting\");\n    reraise_signal(SIGABRT);\n}\n\nstatic void sigill_action(siginfo_t *info, ucontext_t* uc) noexcept {\n    print_with_backtrace(\"Invalid instruction\");\n    reraise_signal(SIGILL);\n}\n\n// We don't need to handle SIGSEGV when asan is enabled.\n#ifdef SEASTAR_ASAN_ENABLED\ntemplate<>\nvoid install_oneshot_signal_handler<SIGSEGV, sigsegv_action>() {\n    (void)sigsegv_action;\n}\n#endif\n\nvoid smp::qs_deleter::operator()(smp_message_queue** qs) const {\n    for (unsigned i = 0; i < smp::count; i++) {\n        for (unsigned j = 0; j < smp::count; j++) {\n            qs[i][j].~smp_message_queue();\n        }\n        ::operator delete[](qs[i], std::align_val_t(alignof(smp_message_queue))\n        );\n    }\n    delete[](qs);\n}\n\nvoid smp::log_aiocbs(log_level level, unsigned storage, unsigned preempt, unsigned network, unsigned reserve) {\n    // Each cell in the table should be\n    // - as wide as the grand total,\n    // - as wide as its containing column's header,\n    // whichever is wider.\n    std::string percpu_hdr = format(\"per cpu\");\n    std::string allcpus_hdr = format(\"all {} cpus\", smp::count);\n    unsigned percpu_total = storage + preempt + network;\n    unsigned allcpus_total = reserve + percpu_total * smp::count;\n    size_t num_width = format(\"{}\", allcpus_total).length();\n    size_t percpu_width = std::max(num_width, percpu_hdr.length());\n    size_t allcpus_width = std::max(num_width, allcpus_hdr.length());\n\n    seastar_logger.log(level, \"purpose  {:{}}  {:{}}\",     percpu_hdr,   percpu_width, allcpus_hdr,          allcpus_width);\n    seastar_logger.log(level, \"-------  {:-<{}}  {:-<{}}\", \"\",           percpu_width, \"\",                   allcpus_width);\n    seastar_logger.log(level, \"reserve  {:{}}  {:{}}\",     \"\",           percpu_width, reserve,              allcpus_width);\n    seastar_logger.log(level, \"storage  {:{}}  {:{}}\",     storage,      percpu_width, storage * smp::count, allcpus_width);\n    seastar_logger.log(level, \"preempt  {:{}}  {:{}}\",     preempt,      percpu_width, preempt * smp::count, allcpus_width);\n    seastar_logger.log(level, \"network  {:{}}  {:{}}\",     network,      percpu_width, network * smp::count, allcpus_width);\n    seastar_logger.log(level, \"-------  {:-<{}}  {:-<{}}\", \"\",           percpu_width, \"\",                   allcpus_width);\n    seastar_logger.log(level, \"total    {:{}}  {:{}}\",     percpu_total, percpu_width, allcpus_total,        allcpus_width);\n}\n\nunsigned smp::adjust_max_networking_aio_io_control_blocks(unsigned network_iocbs, unsigned reserve_iocbs)\n{\n    static unsigned constexpr storage_iocbs = reactor::max_aio;\n    static unsigned constexpr preempt_iocbs = 2;\n\n    auto aio_max_nr = read_first_line_as<unsigned>(\"/proc/sys/fs/aio-max-nr\");\n    auto aio_nr = read_first_line_as<unsigned>(\"/proc/sys/fs/aio-nr\");\n    auto available_aio = aio_max_nr - aio_nr;\n    auto requested_aio_network = network_iocbs * smp::count;\n    auto requested_aio_other = reserve_iocbs + (storage_iocbs + preempt_iocbs) * smp::count;\n    auto requested_aio = requested_aio_network + requested_aio_other;\n\n    seastar_logger.debug(\"Intended AIO control block usage:\");\n    seastar_logger.debug(\"\");\n    log_aiocbs(log_level::debug, storage_iocbs, preempt_iocbs, network_iocbs, reserve_iocbs);\n    seastar_logger.debug(\"\");\n    seastar_logger.debug(\"Available AIO control blocks = aio-max-nr - aio-nr = {} - {} = {}\", aio_max_nr, aio_nr, available_aio);\n\n    if (available_aio < requested_aio) {\n        if (available_aio >= requested_aio_other + smp::count) { // at least one queue for each shard\n            network_iocbs = (available_aio - requested_aio_other) / smp::count;\n            seastar_logger.warn(\"Your system does not have enough AIO capacity for optimal network performance; reducing `max-networking-io-control-blocks'.\");\n            seastar_logger.warn(\"Resultant AIO control block usage:\");\n            seastar_logger.warn(\"\");\n            log_aiocbs(log_level::warn, storage_iocbs, preempt_iocbs, network_iocbs, reserve_iocbs);\n            seastar_logger.warn(\"\");\n            seastar_logger.warn(\"For optimal network performance, set /proc/sys/fs/aio-max-nr to at least {}.\", aio_nr + requested_aio);\n        } else {\n            throw std::runtime_error(\n                format(\"Your system does not satisfy minimum AIO requirements. \"\n                       \"Set /proc/sys/fs/aio-max-nr to at least {} (minimum) or {} (recommended for networking performance).\",\n                       aio_nr + (requested_aio_other + smp::count), aio_nr + requested_aio));\n        }\n    }\n\n    return network_iocbs;\n}\n\nvoid smp::configure(const smp_options& smp_opts, const reactor_options& reactor_opts)\n{\n    bool use_transparent_hugepages = !reactor_opts.overprovisioned;\n\n#ifndef SEASTAR_NO_EXCEPTION_HACK\n    if (smp_opts.enable_glibc_exception_scaling_workaround.get_value()) {\n        init_phdr_cache();\n    }\n#endif\n\n    // Mask most, to prevent threads (esp. dpdk helper threads)\n    // from servicing a signal.  Individual reactors will unmask signals\n    // as they become prepared to handle them.\n    //\n    // We leave some signals unmasked since we don't handle them ourself.\n    sigset_t sigs;\n    sigfillset(&sigs);\n    for (auto sig : {SIGHUP, SIGQUIT, SIGILL, SIGABRT, SIGFPE, SIGSEGV,\n            SIGALRM, SIGCONT, SIGSTOP, SIGTSTP, SIGTTIN, SIGTTOU}) {\n        sigdelset(&sigs, sig);\n    }\n    if (!reactor_opts._auto_handle_sigint_sigterm) {\n        sigdelset(&sigs, SIGINT);\n        sigdelset(&sigs, SIGTERM);\n    }\n    pthread_sigmask(SIG_BLOCK, &sigs, nullptr);\n\n    install_oneshot_signal_handler<SIGSEGV, sigsegv_action>();\n    install_oneshot_signal_handler<SIGABRT, sigabrt_action>();\n    install_oneshot_signal_handler<SIGILL, sigill_action>();\n\n#ifdef SEASTAR_HAVE_DPDK\n    const auto* native_stack = dynamic_cast<const net::native_stack_options*>(reactor_opts.network_stack.get_selected_candidate_opts());\n    _using_dpdk = native_stack && native_stack->dpdk_pmd;\n#endif\n    auto thread_affinity = smp_opts.thread_affinity.get_value();\n    if (reactor_opts.overprovisioned\n           && smp_opts.thread_affinity.defaulted()) {\n        thread_affinity = false;\n    }\n    if (!thread_affinity && _using_dpdk) {\n        fmt::print(\"warning: --thread-affinity 0 ignored in dpdk mode\\n\");\n    }\n    auto mbind = smp_opts.mbind.get_value();\n    if (!thread_affinity) {\n        mbind = false;\n    }\n\n    resource::configuration rc;\n\n    rc.overcommit = reactor_opts.overprovisioned;\n\n    smp::_tmain = std::this_thread::get_id();\n    resource::cpuset cpu_set = get_current_cpuset();\n\n    if (smp_opts.cpuset) {\n        auto opts_cpuset = smp_opts.cpuset.get_value();\n        // CPUs that are not available are those pinned by\n        // --cpuset but not present in current task set\n        std::set<unsigned int> not_available_cpus;\n        std::set_difference(opts_cpuset.begin(), opts_cpuset.end(),\n                            cpu_set.begin(), cpu_set.end(),\n                            std::inserter(not_available_cpus, not_available_cpus.end()));\n\n        if (!not_available_cpus.empty()) {\n            std::ostringstream not_available_cpus_list;\n            for (auto cpu_id : not_available_cpus) {\n                not_available_cpus_list << \" \" << cpu_id;\n            }\n            seastar_logger.error(\"Bad value for --cpuset:{} not allowed. Shutting down.\", not_available_cpus_list.str());\n            exit(1);\n        }\n        cpu_set = opts_cpuset;\n    }\n\n    if (smp_opts.smp) {\n        smp::count = smp_opts.smp.get_value();\n    } else {\n        smp::count = cpu_set.size();\n    }\n    std::vector<reactor*> reactors(smp::count);\n    if (smp_opts.memory) {\n#ifdef SEASTAR_DEFAULT_ALLOCATOR\n        seastar_logger.warn(\"Seastar compiled with default allocator, --memory option won't take effect\");\n#endif\n        rc.total_memory = parse_memory_size(smp_opts.memory.get_value());\n#ifdef SEASTAR_HAVE_DPDK\n        if (smp_opts.hugepages &&\n            !reactor_opts.network_stack.get_selected_candidate_name().compare(\"native\") &&\n            _using_dpdk) {\n            size_t dpdk_memory = dpdk::eal::mem_size(smp::count);\n\n            if (dpdk_memory >= rc.total_memory) {\n                seastar_logger.error(\"Can't run with the given amount of memory: {}. \"\n                                     \"Consider giving more.\",\n                                     smp_opts.memory.get_value());\n                exit(1);\n            }\n\n            //\n            // Subtract the memory we are about to give to DPDK from the total\n            // amount of memory we are allowed to use.\n            //\n            rc.total_memory.value() -= dpdk_memory;\n        }\n#endif\n    }\n    if (smp_opts.reserve_memory) {\n        rc.reserve_memory = parse_memory_size(smp_opts.reserve_memory.get_value());\n    }\n    rc.reserve_additional_memory_per_shard = smp_opts.reserve_additional_memory_per_shard;\n    std::optional<std::string> hugepages_path;\n    if (smp_opts.hugepages) {\n        hugepages_path = smp_opts.hugepages.get_value();\n    }\n    auto mlock = false;\n    if (smp_opts.lock_memory) {\n        mlock = smp_opts.lock_memory.get_value();\n    }\n    if (mlock) {\n        auto extra_flags = 0;\n#ifdef MCL_ONFAULT\n        // Linux will serialize faulting in anonymous memory, and also\n        // serialize marking them as locked. This can take many minutes on\n        // terabyte class machines, so fault them in the future to spread\n        // out the cost. This isn't good since we'll see contention if\n        // multiple shards fault in memory at once, but if that work can be\n        // in parallel to regular reactor work on other shards.\n        extra_flags |= MCL_ONFAULT; // Linux 4.4+\n#endif\n        auto r = mlockall(MCL_CURRENT | MCL_FUTURE | extra_flags);\n        if (r) {\n            // Don't hard fail for now, it's hard to get the configuration right\n            fmt::print(\"warning: failed to mlockall: {}\\n\", strerror(errno));\n        }\n    }\n\n    rc.cpus = smp::count;\n    rc.cpu_set = std::move(cpu_set);\n\n    internal::disk_config_params disk_config(reactor::max_queues);\n    disk_config.parse_config(smp_opts, reactor_opts);\n    for (auto& id : disk_config.queue_ids()) {\n        rc.io_queues.push_back(id);\n    }\n    rc.num_io_groups = disk_config.num_io_groups();\n\n#ifdef SEASTAR_HAVE_HWLOC\n    if (smp_opts.allow_cpus_in_remote_numa_nodes.get_value()) {\n        rc.assign_orphan_cpus = true;\n    }\n#endif\n\n    auto resources = resource::allocate(rc);\n    logger::set_shard_field_width(std::ceil(std::log10(smp::count)));\n    std::vector<resource::cpu> allocations = std::move(resources.cpus);\n    if (thread_affinity) {\n        smp::pin(allocations[0].cpu_id);\n    }\n    std::optional<memory::internal::numa_layout> layout;\n    if (smp_opts.memory_allocator == memory_allocator::seastar) {\n        layout = memory::configure(allocations[0].mem, mbind, use_transparent_hugepages, hugepages_path);\n    } else {\n        // #2148 - if running seastar allocator but options that contradict this, we still need to\n        // init memory at least minimally, otherwise a bunch of stuff breaks.\n        // Previously, we got away wth this by accident due to #2137.\n        memory::configure_minimal();\n    }\n\n    _shard_to_numa_node_mapping.reserve(smp::count);\n    for (unsigned i = 0; i < smp::count; i++) {\n        _shard_to_numa_node_mapping.push_back(allocations[i].mem.size() > 0 ? allocations[i].mem[0].nodeid : 0);\n    }\n\n    if (reactor_opts.abort_on_seastar_bad_alloc) {\n        memory::set_abort_on_allocation_failure(true);\n    }\n\n    if (reactor_opts.dump_memory_diagnostics_on_alloc_failure_kind) {\n        memory::set_dump_memory_diagnostics_on_alloc_failure_kind(reactor_opts.dump_memory_diagnostics_on_alloc_failure_kind.get_value());\n    }\n\n    auto max_networking_aio_io_control_blocks = reactor_opts.max_networking_io_control_blocks.get_value();\n    // Prevent errors about insufficient AIO blocks, when they are not needed by the reactor backend.\n    if (reactor_opts.reactor_backend.get_selected_candidate().name() == \"linux-aio\") {\n        max_networking_aio_io_control_blocks = adjust_max_networking_aio_io_control_blocks(max_networking_aio_io_control_blocks,\n                reactor_opts.reserve_io_control_blocks.get_value());\n    }\n\n    reactor_config reactor_cfg = {\n        .task_quota = std::chrono::duration_cast<sched_clock::duration>(reactor_opts.task_quota_ms.get_value() * 1ms),\n        .max_poll_time = [&reactor_opts] () -> std::chrono::nanoseconds {\n            if (reactor_opts.poll_mode) {\n                return std::chrono::nanoseconds::max();\n            } else if (reactor_opts.overprovisioned && reactor_opts.idle_poll_time_us.defaulted()) {\n                return 0us;\n            } else {\n                return reactor_opts.idle_poll_time_us.get_value() * 1us;\n            }\n        }(),\n        .handle_sigint = !reactor_opts.no_handle_interrupt,\n        .auto_handle_sigint_sigterm = reactor_opts._auto_handle_sigint_sigterm,\n        .max_networking_aio_io_control_blocks = max_networking_aio_io_control_blocks,\n        .force_io_getevents_syscall = reactor_opts.force_aio_syscalls.get_value(),\n        .kernel_page_cache = reactor_opts.kernel_page_cache.get_value(),\n        .have_aio_fsync = reactor_opts.aio_fsync.get_value(),\n        .max_task_backlog = reactor_opts.max_task_backlog.get_value(),\n        .strict_o_direct = !reactor_opts.relaxed_dma,\n        .bypass_fsync = reactor_opts.unsafe_bypass_fsync.get_value(),\n        .no_poll_aio = !reactor_opts.poll_aio.get_value() || (reactor_opts.poll_aio.defaulted() && reactor_opts.overprovisioned),\n        .aio_nowait_works = reactor_opts.linux_aio_nowait.get_value(), // Mixed in with filesystem-provided values later\n        .abort_on_too_long_task_queue = reactor_opts.abort_on_too_long_task_queue.get_value(),\n    };\n\n    // Disable hot polling if sched wakeup granularity is too high\n    // dio thread will be starved otherwise\n    // see https://github.com/scylladb/seastar/issues/2696\n    if (!reactor_cfg.no_poll_aio || reactor_cfg.max_poll_time != 0us) {\n        auto [wakeup_file, granularity] = wakeup_granularity();\n        // 15M is chosen as it's what tuned sets. Though you probably already\n        // see an adverse effect earlier.\n        if (granularity >= 15000000) {\n            reactor_cfg.no_poll_aio = true;\n            reactor_cfg.max_poll_time = 0us;\n            seastar_logger.warn(\n                \"Setting --poll-aio=0 and --idle-poll-time-us=0 due to too high sched_wakeup_granularity of {} in {}\",\n                granularity, wakeup_file.string());\n        }\n    }\n\n    std::mutex mtx;\n\n#ifdef SEASTAR_HEAPPROF\n    size_t heapprof_sampling_rate = reactor_opts.heapprof.get_value();\n    if (heapprof_sampling_rate) {\n        memory::set_heap_profiling_sampling_rate(heapprof_sampling_rate);\n    }\n#else\n    size_t heapprof_sampling_rate = 0;\n#endif\n\n#ifdef SEASTAR_HAVE_DPDK\n    if (_using_dpdk) {\n        dpdk::eal::cpuset cpus;\n        for (auto&& a : allocations) {\n            cpus[a.cpu_id] = true;\n        }\n        dpdk::eal::init(cpus, reactor_opts._argv0, hugepages_path, native_stack ? bool(native_stack->dpdk_pmd) : false);\n    }\n#endif\n\n    // Better to put it into the smp class, but at smp construction time\n    // correct smp::count is not known.\n    std::barrier reactors_registered(smp::count);\n    std::barrier smp_queues_constructed(smp::count);\n    // We use shared_ptr since this thread can exit while other threads are still unlocking\n    auto inited = std::make_shared<std::barrier<>>(smp::count);\n\n    auto ioq_topology = std::move(resources.ioq_topology);\n\n    // ATTN: The ioq_topology value is referenced by below lambdas which are\n    // then copied to other shard's threads, so each shard has a copy of the\n    // ioq_topology on stack, but (!) still references and uses the value\n    // from shard-0. This access is race-free because\n    //  1. The .shard_to_group is not modified\n    //  2. The .queues is pre-resize()-d in advance, so the vector itself\n    //     doesn't change; existing slots are accessed by owning shards only\n    //     without interference\n    //  3. The .groups manipulations are guarded by the .lock lock (but it's\n    //     also pre-resize()-d in advance)\n\n    auto alloc_io_queues = [&ioq_topology, &disk_config] (shard_id shard) {\n        for (auto& [q, io_info] : ioq_topology) {\n            auto num_io_groups = io_info.groups.size();\n            if (engine()._num_io_groups == 0) {\n                engine()._num_io_groups = num_io_groups;\n            } else if (engine()._num_io_groups != num_io_groups) {\n                throw std::logic_error(format(\"Number of IO-groups mismatch, {} != {}\", engine()._num_io_groups, num_io_groups));\n            }\n\n            auto group_idx = io_info.shard_to_group[shard];\n            std::shared_ptr<io_group> group;\n\n            {\n                std::lock_guard _(io_info.lock);\n                auto& iog = io_info.groups[group_idx];\n                if (!iog) {\n                    struct io_queue::config qcfg = disk_config.generate_config(q, io_info.groups.size());\n                    iog = std::make_shared<io_group>(std::move(qcfg), io_info.shards_in_group[group_idx]);\n                    seastar_logger.debug(\"allocate {} IO group with {} queues, queue-id {}\", group_idx, io_info.shards_in_group[group_idx], q);\n                }\n                group = iog;\n            }\n\n            io_info.queues[shard] = seastar::make_shared<io_queue>(std::move(group), engine()._io_sink);\n            seastar_logger.debug(\"attached {} queue to {} IO group, queue-id {}\", shard, group_idx, q);\n        }\n    };\n\n    auto assign_io_queues = [&ioq_topology, &disk_config] (shard_id shard) {\n        for (auto& [q, io_info] : ioq_topology) {\n            auto queue = std::move(io_info.queues[shard]);\n            SEASTAR_ASSERT(queue);\n            for (const dev_t& dev : disk_config.queue_devices(q)) {\n                engine()._io_queues.emplace(dev, queue);\n            }\n        }\n    };\n\n    _all_event_loops_done.emplace(smp::count);\n\n    auto backend_selector = reactor_opts.reactor_backend.get_selected_candidate();\n    seastar_logger.info(\"Reactor backend: {}\", backend_selector);\n\n    _qs_owner = decltype(smp::_qs_owner){new smp_message_queue* [smp::count], qs_deleter{}};\n\n    auto allocate_qs_owner = [this] (unsigned i) {\n        // smp_message_queue has members with hefty alignment requirements.\n        // if we are reactor thread, or not running with dpdk, doing this\n        // new default aligned seemingly works, as does reordering\n        // dlinit dependencies (ugh). But we should enforce calling out to\n        // aligned_alloc, instead of pure malloc, if possible.\n        smp::_qs_owner[i] = reinterpret_cast<smp_message_queue*>(operator new[] (sizeof(smp_message_queue) * smp::count\n            , std::align_val_t(alignof(smp_message_queue))\n        ));\n    };\n\n    auto allocate_smp_queues = [this, &reactors] (unsigned i) {\n        for (unsigned j = 0; j < smp::count; ++j) {\n            new (&smp::_qs_owner[i][j]) smp_message_queue(reactors[j], reactors[i]);\n        }\n    };\n\n    unsigned i;\n    auto smp_tmain = smp::_tmain;\n    for (i = 1; i < smp::count; i++) {\n        auto allocation = allocations[i];\n        create_thread([this, smp_tmain, inited, &reactors_registered, &smp_queues_constructed, &smp_opts, &reactor_opts, &reactors, hugepages_path, i, allocation, assign_io_queues, alloc_io_queues, thread_affinity, heapprof_sampling_rate, mbind, backend_selector, reactor_cfg, &mtx, &layout, use_transparent_hugepages, allocate_qs_owner, allocate_smp_queues] {\n          try {\n            // initialize thread_locals that are equal across all reacto threads of this smp instance\n            smp::_tmain = smp_tmain;\n            auto thread_name = fmt::format(\"reactor-{}\", i);\n            pthread_setname_np(pthread_self(), thread_name.c_str());\n            if (thread_affinity) {\n                smp::pin(allocation.cpu_id);\n            }\n            if (smp_opts.memory_allocator == memory_allocator::seastar) {\n                auto another_layout = memory::configure(allocation.mem, mbind, use_transparent_hugepages, hugepages_path);\n                auto guard = std::lock_guard(mtx);\n                *layout = memory::internal::merge(std::move(*layout), std::move(another_layout));\n            } else {\n                // See comment above (shard 0)\n                memory::configure_minimal();\n            }\n            if (heapprof_sampling_rate) {\n                memory::set_heap_profiling_sampling_rate(heapprof_sampling_rate);\n            }\n            sigset_t mask;\n            sigfillset(&mask);\n            for (auto sig : { SIGSEGV, SIGILL }) {\n                sigdelset(&mask, sig);\n            }\n            auto r = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n            throw_pthread_error(r);\n            init_default_smp_service_group(i);\n            lowres_clock::update();\n            allocate_reactor(i, backend_selector, reactor_cfg);\n            reactors[i] = &engine();\n            alloc_io_queues(i);\n            allocate_qs_owner(i);\n            _qs = _qs_owner.get();\n            reactors_registered.arrive_and_wait();\n            allocate_smp_queues(i);\n            smp_queues_constructed.arrive_and_wait();\n            start_all_queues();\n            assign_io_queues(i);\n            inited->arrive_and_wait();\n            engine().configure(reactor_opts);\n            engine().do_run();\n          } catch (const std::exception& e) {\n              seastar_logger.error(\"{}\", e.what());\n              _exit(1);\n          }\n        });\n    }\n\n    init_default_smp_service_group(0);\n    lowres_clock::update();\n    try {\n        allocate_reactor(0, backend_selector, reactor_cfg);\n    } catch (const std::exception& e) {\n        seastar_logger.error(\"{}\", e.what());\n        _exit(1);\n    }\n\n    reactors[0] = &engine();\n    alloc_io_queues(0);\n\n#ifdef SEASTAR_HAVE_DPDK\n    if (_using_dpdk) {\n        auto it = _thread_loops.begin();\n        RTE_LCORE_FOREACH_WORKER(i) {\n            rte_eal_remote_launch(dpdk_thread_adaptor, static_cast<void*>(&*(it++)), i);\n        }\n    }\n#endif\n\n    allocate_qs_owner(0);\n    reactors_registered.arrive_and_wait();\n    _qs = _qs_owner.get();\n    allocate_smp_queues(0);\n    _alien._qs = alien::instance::create_qs(reactors);\n    smp_queues_constructed.arrive_and_wait();\n    start_all_queues();\n    assign_io_queues(0);\n    inited->arrive_and_wait();\n\n    engine().configure(reactor_opts);\n\n    if (smp_opts.lock_memory && smp_opts.lock_memory.get_value() && layout && !layout->ranges.empty()) {\n        smp::setup_prefaulter(resources, std::move(*layout));\n    }\n}\n\nbool smp::poll_queues() {\n    size_t got = 0;\n    for (unsigned i = 0; i < count; i++) {\n        if (this_shard_id() != i) {\n            auto& rxq = _qs[this_shard_id()][i];\n            rxq.flush_response_batch();\n            got += rxq.has_unflushed_responses();\n            got += rxq.process_incoming();\n            auto& txq = _qs[i][this_shard_id()];\n            txq.flush_request_batch();\n            got += txq.process_completions(i);\n        }\n    }\n    return got != 0;\n}\n\nbool smp::pure_poll_queues() {\n    for (unsigned i = 0; i < count; i++) {\n        if (this_shard_id() != i) {\n            auto& rxq = _qs[this_shard_id()][i];\n            rxq.flush_response_batch();\n            auto& txq = _qs[i][this_shard_id()];\n            txq.flush_request_batch();\n            if (rxq.pure_poll_rx() || txq.pure_poll_tx() || rxq.has_unflushed_responses()) {\n                return true;\n            }\n        }\n    }\n    return false;\n}\n\n__thread reactor* local_engine;\n\nvoid report_exception(std::string_view message, std::exception_ptr eptr) noexcept {\n    seastar_logger.error(\"{}: {}\", message, eptr);\n}\n\nfuture<> check_direct_io_support(std::string_view path) noexcept {\n    struct w {\n        sstring path;\n        open_flags flags;\n        std::function<future<>()> cleanup;\n\n        static w parse(sstring path, std::optional<directory_entry_type> type) {\n            if (!type) {\n                throw std::invalid_argument(format(\"Could not open file at {}. Make sure it exists\", path));\n            }\n\n            if (type == directory_entry_type::directory) {\n                auto fpath = path + \"/.o_direct_test\";\n                return w{fpath, open_flags::wo | open_flags::create | open_flags::truncate, [fpath] { return remove_file(fpath); }};\n            } else if ((type == directory_entry_type::regular) || (type == directory_entry_type::link)) {\n                return w{path, open_flags::ro, [] { return make_ready_future<>(); }};\n            } else {\n                throw std::invalid_argument(format(\"{} neither a directory nor file. Can't be opened with O_DIRECT\", path));\n            }\n        };\n    };\n\n    // Allocating memory for a sstring can throw, hence the futurize_invoke\n    return futurize_invoke([path] {\n        return engine().file_type(path).then([path = sstring(path)] (auto type) {\n            auto w = w::parse(path, type);\n            return open_file_dma(w.path, w.flags).then_wrapped([path = w.path, cleanup = std::move(w.cleanup)] (future<file> f) {\n                try {\n                    auto fd = f.get();\n                    return cleanup().finally([fd = std::move(fd)] () mutable {\n                        return fd.close();\n                    });\n                } catch (std::system_error& e) {\n                    if (e.code() == std::error_code(EINVAL, std::system_category())) {\n                        report_exception(format(\"Could not open file at {}. Does your filesystem support O_DIRECT?\", path), std::current_exception());\n                    }\n                    throw;\n                }\n            });\n        });\n    });\n}\n\nserver_socket listen(socket_address sa) {\n    return engine().listen(sa);\n}\n\nserver_socket listen(socket_address sa, listen_options opts) {\n    return engine().listen(sa, opts);\n}\n\nfuture<connected_socket> connect(socket_address sa) {\n    return engine().connect(sa);\n}\n\nfuture<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) {\n    return engine().connect(sa, local, proto);\n}\n\nsocket make_socket() {\n    return engine().net().socket();\n}\n\nnet::udp_channel make_udp_channel() {\n    return make_unbound_datagram_channel(AF_INET);\n}\n\nnet::udp_channel make_udp_channel(const socket_address& local) {\n    return make_bound_datagram_channel(local);\n}\n\nnet::datagram_channel make_unbound_datagram_channel(sa_family_t family) {\n    return engine().net().make_unbound_datagram_channel(family);\n}\n\nnet::datagram_channel make_bound_datagram_channel(const socket_address& local) {\n    return engine().net().make_bound_datagram_channel(local);\n}\n\nvoid reactor::add_high_priority_task(task* t) noexcept {\n    add_urgent_task(t);\n    // break .then() chains\n    request_preemption();\n}\n\n\nvoid set_idle_cpu_handler(idle_cpu_handler&& handler) {\n    engine().set_idle_cpu_handler(std::move(handler));\n}\n\nnamespace experimental {\nfuture<std::tuple<file_desc, file_desc>> make_pipe() {\n    return engine().make_pipe();\n}\n\nfuture<process> spawn_process(const std::filesystem::path& pathname,\n                              spawn_parameters params) {\n    return process::spawn(pathname, std::move(params));\n}\n\nfuture<process> spawn_process(const std::filesystem::path& pathname) {\n    return process::spawn(pathname);\n}\n}\n\nstatic\nbool\nvirtualized() {\n    return fs::exists(\"/sys/hypervisor/type\");\n}\n\nstd::chrono::nanoseconds\nreactor::calculate_poll_time() {\n    // In a non-virtualized environment, select a poll time\n    // that is competitive with halt/unhalt.\n    //\n    // In a virutalized environment, IPIs are slow and dominate\n    // sleep/wake (mprotect/tgkill), so increase poll time to reduce\n    // so we don't sleep in a request/reply workload\n    return virtualized() ? 2000us : 200us;\n}\n\nfuture<>\nyield() noexcept {\n    memory::scoped_critical_alloc_section _;\n    auto tsk = make_task([] {});\n    schedule(tsk);\n    return tsk->get_future();\n}\n\nfuture<> check_for_io_immediately() noexcept {\n    memory::scoped_critical_alloc_section _;\n    engine().force_poll();\n    auto tsk = make_task(default_scheduling_group(), [] {});\n    schedule(tsk);\n    return tsk->get_future();\n}\n\nfuture<> later() noexcept {\n    return check_for_io_immediately();\n}\n\nsteady_clock_type::duration reactor::total_idle_time() const {\n    return _total_idle;\n}\n\nsteady_clock_type::duration reactor::total_busy_time() const {\n    return now() - _start_time - _total_idle;\n}\n\nsteady_clock_type::duration reactor::total_awake_time() const {\n    return now() - _start_time - _total_sleep;\n}\n\nstd::chrono::nanoseconds reactor::total_cpu_time() const {\n    return thread_cputime_clock::now().time_since_epoch();\n}\n\nstd::chrono::nanoseconds reactor::total_steal_time() const {\n    // Steal time: this mimics the concept some Hypervisors have about Steal time.\n    // That is the time in which a VM has something to run, but is not running because some other\n    // process (another VM or the hypervisor itself) is in control.\n    //\n    // For us, we notice that during the time in which we were not sleeping (either running or busy\n    // polling while idle), we should be accumulating thread runtime. If we are not, that's because\n    // someone stole it from us.\n    //\n    // Because this is totally in userspace we can miss some events. For instance, if the seastar\n    // process is ready to run but the kernel hasn't scheduled us yet, that would be technically\n    // steal time but we have no ways to account it.\n    //\n    // Furthermore, not all steal is from other processes: time used by the syscall thread and any\n    // alien threads will show up as steal as well as any time spent in a system call that\n    // unexpectedly blocked (since CPU time won't tick up when that occurs).\n    //\n    // But what we have here should be good enough and at least has a well defined meaning.\n    //\n    // Because we calculate sleep time with timestamps around polling methods that may sleep, like\n    // io_getevents, we systematically over-count sleep time, since there is CPU usage within the\n    // period timed as sleep, before and after an actual sleep occurs (and no sleep may occur at all,\n    // e.g., if there are events immediately available). Over-counting sleep means we under-count the\n    // wall-clock awake time, and so if there is no \"true\" steal, we will generally have a small\n    // *negative* steal time, because we under-count awake wall clock time while thread CPU time does\n    // not have a corresponding error.\n    //\n    // Becuase we claim \"steal\" is a counter, we must ensure that it never deceases, because PromQL\n    // functions which use counters will produce non-sensical results if they do. Therefore we clamp\n    // the output such that it never decreases.\n    //\n    // Finally, we don't just clamp difference of awake and CPU time since proces start at 0, but\n    // take the last value we returned from this function and then calculate the incremental steal\n    // time since that measurement, clamped to 0. This means that as soon as steal time becomes\n    // positive, it will be reflected in the measurement, rather than needing to \"consume\" all the\n    // accumulated negative steal time before positive steal times start showing up.\n\n\n    auto true_steal = total_awake_time() - total_cpu_time();\n    auto mono_steal = _last_mono_steal + std::max(true_steal - _last_true_steal, 0ns);\n\n    _last_true_steal = true_steal;\n    _last_mono_steal = mono_steal;\n\n    return mono_steal;\n}\n\nstatic std::atomic<unsigned long> s_next_scheduling_group_specific_key{0};\n\nstatic\nunsigned long\nallocate_scheduling_group_specific_key() noexcept {\n    return  s_next_scheduling_group_specific_key.fetch_add(1, std::memory_order_relaxed);\n}\n\nstatic\ninternal::scheduling_group_specific_thread_local_data::specific_val\nallocate_scheduling_group_specific_data(scheduling_group sg, unsigned long key_id, const lw_shared_ptr<scheduling_group_key_config>& cfg) {\n    using val_ptr = internal::scheduling_group_specific_thread_local_data::val_ptr;\n    using specific_val = internal::scheduling_group_specific_thread_local_data::specific_val;\n\n    val_ptr valp(aligned_alloc(cfg->alignment, cfg->allocation_size), &specific_val::free);\n    if (!valp) {\n        throw std::runtime_error(\"memory allocation failed\");\n    }\n    return specific_val(std::move(valp), cfg);\n}\n\nfuture<>\nreactor::rename_scheduling_group_specific_data(scheduling_group sg) {\n    return with_shared(_scheduling_group_keys_mutex, [this, sg] {\n        return with_scheduling_group(sg, [this, sg] {\n            get_sg_data(sg).rename();\n        });\n    });\n}\n\nfuture<>\nreactor::init_scheduling_group_specific_data(scheduling_group sg) {\n    return with_scheduling_group(sg, [this, sg] () {\n        auto& sg_data = _scheduling_group_specific_data;\n        auto& this_sg = get_sg_data(sg);\n        for (const auto& [key_id, cfg] : sg_data.scheduling_group_key_configs) {\n            this_sg.specific_vals.resize(std::max<size_t>(this_sg.specific_vals.size(), key_id+1));\n            this_sg.specific_vals[key_id] = allocate_scheduling_group_specific_data(sg, key_id, cfg);\n        }\n    });\n}\n\nfuture<scheduling_group>\nreactor::init_scheduling_group(sstring name, sstring shortname, float shares, scheduling_supergroup parent) {\n    return with_shared(_scheduling_group_keys_mutex, [this, name = std::move(name), shortname = std::move(shortname), shares, parent] {\n        unsigned id = 0;\n        while (id < max_scheduling_groups() && _task_queues[id] != nullptr) {\n            id++;\n        }\n        if (id == max_scheduling_groups()) {\n            return make_exception_future<scheduling_group>(std::runtime_error(fmt::format(\"Scheduling group limit exceeded while creating {}\", name)));\n        }\n\n        auto* group = &_cpu_sched;\n        if (!parent.is_root()) {\n            if (_supergroups[parent.index()] == nullptr) {\n                return make_exception_future<scheduling_group>(std::runtime_error(\"Requested supergroup doesn't exist\"));\n            }\n            group = _supergroups[parent.index()].get();\n        }\n        if (group->_nr_children == max_scheduling_groups()) {\n            return make_exception_future<scheduling_group>(std::runtime_error(fmt::format(\"Supergroup children limit exceeded while creating {}\", name)));\n        }\n\n        auto sg = scheduling_group(id);\n        get_sg_data(sg).queue_is_initialized = true;\n        _task_queues[sg._id] = std::make_unique<task_queue>(group, sg._id, name, shortname, shares);\n\n        return init_scheduling_group_specific_data(sg).then([sg] {\n            return make_ready_future<scheduling_group>(sg);\n        });\n    });\n}\n\nfuture<>\nreactor::init_scheduling_group(seastar::scheduling_group sg, sstring name, sstring shortname, float shares, scheduling_supergroup parent) {\n    return with_shared(_scheduling_group_keys_mutex, [this, sg, name = std::move(name), shortname = std::move(shortname), shares, parent] {\n        get_sg_data(sg).queue_is_initialized = true;\n        auto* group = &_cpu_sched;\n        if (!parent.is_root()) {\n            group = _supergroups[parent.index()].get();\n        }\n        _task_queues[sg._id] = std::make_unique<task_queue>(group, sg._id, name, shortname, shares);\n        return init_scheduling_group_specific_data(sg);\n    });\n}\n\nfuture<>\nreactor::init_new_scheduling_group_key(scheduling_group_key key, scheduling_group_key_config cfg) {\n    return with_lock(_scheduling_group_keys_mutex, [this, key, cfg] {\n        auto key_id = internal::scheduling_group_key_id(key);\n        auto cfgp = make_lw_shared<scheduling_group_key_config>(std::move(cfg));\n        _scheduling_group_specific_data.scheduling_group_key_configs[key_id] = cfgp;\n        return parallel_for_each(_task_queues, [this, cfgp, key_id] (std::unique_ptr<task_queue>& tq) {\n            if (tq) {\n                scheduling_group sg = scheduling_group(tq->_id);\n                if (tq.get() == _at_destroy_tasks) {\n                    // fake the group by assuming it here\n                    auto curr = current_scheduling_group();\n                    auto cleanup = defer([curr] () noexcept { *internal::current_scheduling_group_ptr() = curr; });\n                    *internal::current_scheduling_group_ptr() = sg;\n                    auto& this_sg = get_sg_data(sg);\n                    this_sg.specific_vals.resize(std::max<size_t>(this_sg.specific_vals.size(), key_id+1));\n                    this_sg.specific_vals[key_id] = allocate_scheduling_group_specific_data(sg, key_id, cfgp);\n                } else {\n                    return with_scheduling_group(sg, [this, key_id, sg, cfgp = std::move(cfgp)] () {\n                        auto& this_sg = get_sg_data(sg);\n                        this_sg.specific_vals.resize(std::max<size_t>(this_sg.specific_vals.size(), key_id+1));\n                        this_sg.specific_vals[key_id] = allocate_scheduling_group_specific_data(sg, key_id, cfgp);\n                    });\n                }\n            }\n            return make_ready_future();\n        });\n    });\n}\n\nfuture<>\nreactor::destroy_scheduling_group(scheduling_group sg) noexcept {\n    if (sg._id >= max_scheduling_groups()) {\n        on_fatal_internal_error(seastar_logger, format(\"Invalid scheduling_group {}\", sg._id));\n    }\n    return with_scheduling_group(sg, [this, sg] () {\n        get_sg_data(sg).specific_vals.clear();\n    }).then( [this, sg] () {\n        get_sg_data(sg).queue_is_initialized = false;\n        _task_queues[sg._id].reset();\n        for (auto&& queue : _io_queues) {\n            queue.second->destroy_priority_class(internal::priority_class(sg));\n        }\n    });\n\n}\n\nstd::span<const unsigned> smp::shard_to_numa_node_mapping() const noexcept {\n    return _shard_to_numa_node_mapping;\n}\n\nconst smp& reactor::smp() const noexcept {\n    return *_smp;\n}\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nnamespace internal {\n\nscheduling_group_specific_thread_local_data** get_scheduling_group_specific_thread_local_data_ptr() noexcept {\n    static thread_local scheduling_group_specific_thread_local_data* data;\n    return &data;\n}\n\n}\n#endif\n\nvoid\ninternal::no_such_scheduling_group(scheduling_group sg) {\n    throw std::invalid_argument(format(\"The scheduling group does not exist ({})\", internal::scheduling_group_index(sg)));\n}\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nscheduling_group*\ninternal::current_scheduling_group_ptr() noexcept {\n    // Slow unless constructor is constexpr\n    static thread_local scheduling_group sg;\n    return &sg;\n}\n#endif\n\nconst sstring&\nscheduling_group::name() const noexcept {\n    return engine()._task_queues[_id]->_name;\n}\n\nconst sstring&\nscheduling_group::short_name() const noexcept {\n    auto& task_queue = engine()._task_queues[_id];\n    if (task_queue) {\n        return task_queue->_shortname;\n    }\n    // we might want to print logging messages before task_queues are ready.\n    // pad this string so its length is task_queue->shortname_size\n    static const sstring not_available(\"n/a \");\n    return not_available;\n}\n\n\nfloat scheduling_group::get_shares() const noexcept {\n    return engine()._task_queues[_id]->_shares;\n}\n\nfloat scheduling_supergroup::get_shares() const noexcept {\n    SEASTAR_ASSERT(!is_root());\n    return engine()._supergroups[index()]->_shares;\n}\n\nvoid\nscheduling_group::set_shares(float shares) noexcept {\n    engine()._task_queues[_id]->set_shares(shares);\n    engine().update_shares_for_queues(internal::priority_class(*this), shares);\n}\n\nfuture<> scheduling_group::update_io_bandwidth(uint64_t bandwidth) const {\n    return engine().update_bandwidth_for_queues(internal::priority_class(*this), bandwidth);\n}\n\nfuture<> scheduling_supergroup::update_io_bandwidth(uint64_t bandwidth) const {\n    SEASTAR_ASSERT(!is_root());\n    return engine().update_bandwidth_for_queues(index(), bandwidth);\n}\n\nvoid scheduling_supergroup::set_shares(float shares) noexcept {\n    if (!is_root()) {\n        engine()._supergroups[index()]->set_shares(shares);\n        engine().update_group_shares_for_queues(index(), shares);\n    }\n}\n\nfuture<scheduling_supergroup> create_scheduling_supergroup(float shares) noexcept {\n    auto index = co_await smp::submit_to(0, [shares] () -> unsigned {\n        auto& r = engine();\n        if (r._cpu_sched._nr_children == max_scheduling_groups()) {\n            throw std::runtime_error(\"Supergroup children limit exceeded while creating nested supergroup\");\n        }\n        auto ssg = std::make_unique<reactor::task_queue_group>(&r._cpu_sched, shares);\n        for (unsigned index = 0; index < r._supergroups.size(); index++) {\n            if (r._supergroups[index] == nullptr) {\n                r._supergroups[index] = std::move(ssg);\n                return index;\n            }\n        }\n\n        r._supergroups.push_back(std::move(ssg));\n        return r._supergroups.size() - 1;\n    });\n\n    std::exception_ptr ex;\n    try {\n        co_await smp::invoke_on_all([index, shares] {\n            if (this_shard_id() != 0) {\n                auto& r = engine();\n                if (r._supergroups.size() <= index) {\n                    r._supergroups.resize(index + 1);\n                }\n                r._supergroups[index] = std::make_unique<reactor::task_queue_group>(&r._cpu_sched, shares);\n            }\n        });\n    } catch (...) {\n        ex = std::current_exception();\n    }\n\n    if (ex != nullptr) {\n        co_await smp::invoke_on_all([index] () noexcept {\n            engine()._supergroups[index].reset();\n        });\n\n        std::rethrow_exception(std::move(ex));\n    }\n\n    co_return scheduling_supergroup(index);\n}\n\nfuture<> destroy_scheduling_supergroup(scheduling_supergroup sg) noexcept {\n    if (sg.is_root()) {\n        throw std::runtime_error(\"Root supergroup cannot be destroyed\");\n    }\n\n    unsigned index = sg.index();\n    co_await smp::submit_to(0, [index] {\n        auto& r = engine();\n        if (r._supergroups[index]->_nr_children != 0) {\n            throw std::runtime_error(\"Supergroup is still populated, destroy all subgroups first\");\n        }\n\n        r._supergroups[index].reset();\n    });\n\n    co_await smp::invoke_on_all([index] () noexcept {\n        if (this_shard_id() != 0) {\n            auto& r = engine();\n            SEASTAR_ASSERT(r._supergroups[index]->_nr_children == 0);\n            r._supergroups[index].reset();\n        }\n    });\n}\n\nfuture<scheduling_group>\ncreate_scheduling_group(sstring name, sstring shortname, float shares, scheduling_supergroup parent) noexcept {\n    auto sg = co_await smp::submit_to(0, [name, shortname, shares, parent] {\n        return engine().init_scheduling_group(name, shortname, shares, parent);\n    });\n    co_await smp::invoke_on_all([sg, name, shortname, shares, parent] {\n        if (this_shard_id() == 0) {\n            return make_ready_future<>();\n        }\n        return engine().init_scheduling_group(sg, name, shortname, shares, parent);\n    });\n    co_return sg;\n}\n\nfuture<scheduling_group>\ncreate_scheduling_group(sstring name, sstring shortname, float shares) noexcept {\n    return create_scheduling_group(name, shortname, shares, scheduling_supergroup());\n}\n\nfuture<scheduling_group>\ncreate_scheduling_group(sstring name, float shares) noexcept {\n    return create_scheduling_group(name, {}, shares);\n}\n\nfuture<scheduling_group_key>\nscheduling_group_key_create(scheduling_group_key_config cfg) noexcept {\n    scheduling_group_key key = allocate_scheduling_group_specific_key();\n    return smp::invoke_on_all([key, cfg] {\n        return engine().init_new_scheduling_group_key(key, cfg);\n    }).then([key] {\n        return make_ready_future<scheduling_group_key>(key);\n    });\n}\n\nfuture<>\ndestroy_scheduling_group(scheduling_group sg) noexcept {\n    if (sg == default_scheduling_group()) {\n        return make_exception_future<>(make_backtraced_exception_ptr<std::runtime_error>(\"Attempt to destroy the default scheduling group\"));\n    }\n    if (sg == current_scheduling_group()) {\n        return make_exception_future<>(make_backtraced_exception_ptr<std::runtime_error>(\"Attempt to destroy the current scheduling group\"));\n    }\n    return smp::invoke_on_all([sg] {\n        return engine().destroy_scheduling_group(sg);\n    });\n}\n\nfuture<>\nrename_scheduling_group(scheduling_group sg, sstring new_name) noexcept {\n    return rename_scheduling_group(sg, new_name, {});\n}\n\nfuture<>\nrename_scheduling_group(scheduling_group sg, sstring new_name, sstring new_shortname) noexcept {\n    if (sg == default_scheduling_group()) {\n        return make_exception_future<>(make_backtraced_exception_ptr<std::runtime_error>(\"Attempt to rename the default scheduling group\"));\n    }\n    return smp::invoke_on_all([sg, new_name, new_shortname] {\n        engine()._task_queues[internal::scheduling_group_index(sg)]->rename(new_name, new_shortname);\n        internal::execution_stage_manager::get().update_scheduling_group_name(sg);\n        engine().rename_queues(internal::priority_class(sg), new_name);\n        return engine().rename_scheduling_group_specific_data(sg);\n    });\n}\n\nnamespace internal {\n\nvoid add_to_flush_poller(output_stream<char>& os) noexcept {\n    engine()._flush_batching.push_back(os);\n}\n\ninline\nsched_clock::duration\ntimeval_to_duration(::timeval tv) {\n    return std::chrono::seconds(tv.tv_sec) + std::chrono::microseconds(tv.tv_usec);\n}\n\nclass reactor_stall_sampler : public reactor::pollfn {\n    sched_clock::time_point _run_start;\n    ::rusage _run_start_rusage;\n    uint64_t _kernel_stalls = 0;\n    sched_clock::duration _nonsleep_cpu_time = {};\n    sched_clock::duration _nonsleep_wall_time = {};\nprivate:\n    static ::rusage get_rusage() {\n        struct ::rusage ru;\n        ::getrusage(RUSAGE_THREAD, &ru);\n        return ru;\n    }\n    static sched_clock::duration cpu_time(const ::rusage& ru) {\n        return timeval_to_duration(ru.ru_stime) + timeval_to_duration(ru.ru_utime);\n    }\n    void mark_run_start() {\n        _run_start = reactor::now();\n        _run_start_rusage = get_rusage();\n    }\n    void mark_run_end() {\n        auto start_nvcsw = _run_start_rusage.ru_nvcsw;\n        auto start_cpu_time = cpu_time(_run_start_rusage);\n        auto start_time = _run_start;\n        _run_start = reactor::now();\n        _run_start_rusage = get_rusage();\n        _kernel_stalls += _run_start_rusage.ru_nvcsw - start_nvcsw;\n        _nonsleep_cpu_time += cpu_time(_run_start_rusage) - start_cpu_time;\n        _nonsleep_wall_time += _run_start - start_time;\n    }\npublic:\n    reactor_stall_sampler() { mark_run_start(); }\n    virtual bool poll() override { return false; }\n    virtual bool pure_poll() override { return false; }\n    virtual bool try_enter_interrupt_mode() override {\n        // try_enter_interrupt_mode marks the end of a reactor run that should be context-switch free\n        mark_run_end();\n        return true;\n    }\n    virtual void exit_interrupt_mode() override {\n        // start a reactor run that should be context switch free\n        mark_run_start();\n    }\n    stall_report report() const {\n        stall_report r;\n        // mark_run_end() with an immediate mark_run_start() is logically a no-op,\n        // but each one of them has an effect, so they can't be marked const\n        const_cast<reactor_stall_sampler*>(this)->mark_run_end();\n        r.kernel_stalls = _kernel_stalls;\n        r.run_wall_time = _nonsleep_wall_time;\n        r.stall_time = _nonsleep_wall_time - _nonsleep_cpu_time;\n        const_cast<reactor_stall_sampler*>(this)->mark_run_start();\n        return r;\n    }\n};\n\nfuture<stall_report>\nreport_reactor_stalls(noncopyable_function<future<> ()> uut) {\n    auto reporter = std::make_unique<reactor_stall_sampler>();\n    auto p_reporter = reporter.get();\n    auto poller = reactor::poller(std::move(reporter));\n    return uut().then([poller = std::move(poller), p_reporter] () mutable {\n        return p_reporter->report();\n    });\n}\n\nstd::ostream& operator<<(std::ostream& os, const stall_report& sr) {\n    auto to_ms = [] (sched_clock::duration d) -> float {\n        return std::chrono::duration<float>(d) / 1ms;\n    };\n    return os << format(\"{} stalls, {} ms stall time, {} ms run time\", sr.kernel_stalls, to_ms(sr.stall_time), to_ms(sr.run_wall_time));\n}\n\nsize_t scheduling_group_count() {\n    return max_scheduling_groups() - std::count(engine()._task_queues.begin(), engine()._task_queues.end(), nullptr);\n}\n\nvoid\nrun_in_background(future<> f) {\n    engine().run_in_background(std::move(f));\n}\n\nvoid log_timer_callback_exception(std::exception_ptr ex) noexcept {\n    seastar_logger.error(\"Timer callback failed: {}\", ex);\n}\n\nvoid set_current_task(task* t) {\n    local_engine->_current_task = t;\n}\n\n}\n\n#ifdef SEASTAR_TASK_BACKTRACE\n\nvoid task::make_backtrace() noexcept {\n    memory::disable_backtrace_temporarily dbt;\n    try {\n        _bt = make_lw_shared<simple_backtrace>(current_backtrace_tasklocal());\n    } catch (...) {\n        _bt = nullptr;\n    }\n}\n\n#endif\n\n}\n"
  },
  {
    "path": "src/core/reactor_backend.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#include <atomic>\n#include <chrono>\n#include <filesystem>\n#include <thread>\n#include <utility>\n#include <fcntl.h>\n#include <signal.h>\n#include <sys/epoll.h>\n#include <poll.h>\n#include <sys/syscall.h>\n#include <sys/resource.h>\n#include <boost/container/small_vector.hpp>\n#include <fmt/core.h>\n#include <seastar/util/assert.hh>\n\n#ifdef SEASTAR_HAVE_URING\n#include <liburing.h>\n#endif\n\n#include \"core/reactor_backend.hh\"\n#include \"core/thread_pool.hh\"\n#include \"core/syscall_result.hh\"\n#include <seastar/core/internal/buffer_allocator.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n#include <seastar/core/internal/uname.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/read_first_line.hh>\n\nnamespace seastar {\n\nusing namespace std::chrono_literals;\nusing namespace internal;\nusing namespace internal::linux_abi;\nnamespace fs = std::filesystem;\n\nclass pollable_fd_state_completion : public kernel_completion {\n    promise<> _pr;\npublic:\n    virtual void complete_with(ssize_t res) override {\n        _pr.set_value();\n    }\n    future<> get_future() {\n        return _pr.get_future();\n    }\n};\n\nvoid prepare_iocb(const io_request& req, io_completion* desc, iocb& iocb) {\n    switch (req.opcode()) {\n    case io_request::operation::fdatasync:\n        iocb = make_fdsync_iocb(req.as<io_request::operation::fdatasync>().fd);\n        break;\n    case io_request::operation::write: {\n        const auto& op = req.as<io_request::operation::write>();\n        iocb = make_write_iocb(op.fd, op.pos, op.addr, op.size);\n        set_nowait(iocb, op.nowait_works);\n        break;\n    }\n    case io_request::operation::writev: {\n        const auto& op = req.as<io_request::operation::writev>();\n        iocb = make_writev_iocb(op.fd, op.pos, op.iovec, op.iov_len);\n        set_nowait(iocb, op.nowait_works);\n        break;\n    }\n    case io_request::operation::read: {\n        const auto& op = req.as<io_request::operation::read>();\n        iocb = make_read_iocb(op.fd, op.pos, op.addr, op.size);\n        set_nowait(iocb, op.nowait_works);\n        break;\n    }\n    case io_request::operation::readv: {\n        const auto& op = req.as<io_request::operation::readv>();\n        iocb = make_readv_iocb(op.fd, op.pos, op.iovec, op.iov_len);\n        set_nowait(iocb, op.nowait_works);\n        break;\n    }\n    default:\n        seastar_logger.error(\"Invalid operation for iocb: {}\", req.opname());\n        std::abort();\n    }\n    set_user_data(iocb, desc);\n}\n\naio_storage_context::iocb_pool::iocb_pool() {\n    for (unsigned i = 0; i != max_aio; ++i) {\n        _free_iocbs.push(&_all_iocbs[i]);\n    }\n}\n\naio_storage_context::aio_storage_context(reactor& r)\n    : _r(r)\n    , _io_context(0) {\n    static_assert(max_aio >= reactor::max_queues * reactor::max_queues,\n                  \"Mismatch between maximum allowed io and what the IO queues can produce\");\n    internal::setup_aio_context(max_aio, &_io_context);\n    _r.do_at_exit([this] { return stop(); });\n}\n\naio_storage_context::~aio_storage_context() {\n    internal::io_destroy(_io_context);\n}\n\nfuture<> aio_storage_context::stop() noexcept {\n    return std::exchange(_pending_aio_retry_fut, make_ready_future<>()).finally([this] {\n        return do_until([this] { return !_iocb_pool.outstanding(); }, [this] {\n            reap_completions(false);\n            return make_ready_future<>();\n        });\n    });\n}\n\ninline\ninternal::linux_abi::iocb&\naio_storage_context::iocb_pool::get_one() {\n    auto io = _free_iocbs.top();\n    _free_iocbs.pop();\n    return *io;\n}\n\ninline\nvoid\naio_storage_context::iocb_pool::put_one(internal::linux_abi::iocb* io) {\n    _free_iocbs.push(io);\n}\n\ninline\nunsigned\naio_storage_context::iocb_pool::outstanding() const {\n    return max_aio - _free_iocbs.size();\n}\n\ninline\nbool\naio_storage_context::iocb_pool::has_capacity() const {\n    return !_free_iocbs.empty();\n}\n\n// Returns: number of iocbs consumed (0 or 1)\nsize_t\naio_storage_context::handle_aio_error(linux_abi::iocb* iocb, int ec) {\n    switch (ec) {\n        case EAGAIN:\n            return 0;\n        case EBADF: {\n            auto desc = get_user_data<kernel_completion>(*iocb);\n            _iocb_pool.put_one(iocb);\n            desc->complete_with(-EBADF);\n            // if EBADF, it means that the first request has a bad fd, so\n            // we will only remove it from _pending_io and try again.\n            return 1;\n        }\n        case EINVAL:\n            // happens when the filesystem does not implement aio read or write\n            [[fallthrough]];\n        case ENOTSUP: {\n            seastar_logger.error(\"io_submit failed: this happens when \"\n                                 \"accessing filesystem which does not supports \"\n                                 \"asynchronous direct I/O\");\n            auto desc = get_user_data<kernel_completion>(*iocb);\n            _iocb_pool.put_one(iocb);\n            desc->complete_with(-ENOTSUP);\n            return 1;\n        }\n        default:\n            ++_r._io_stats.aio_errors;\n            throw std::system_error(ec, std::system_category(), \"io_submit\");\n    }\n}\n\nbool\naio_storage_context::submit_work() {\n    bool did_work = false;\n\n    _submission_queue.clear();\n    size_t to_submit = _r._io_sink.drain([this] (const internal::io_request& req, io_completion* desc) -> bool {\n        if (!_iocb_pool.has_capacity()) {\n            return false;\n        }\n\n        auto& io = _iocb_pool.get_one();\n        prepare_iocb(req, desc, io);\n\n        if (_r._aio_eventfd) {\n            set_eventfd_notification(io, _r._aio_eventfd->get_fd());\n        }\n        _submission_queue.push_back(&io);\n        return true;\n    });\n\n    if (__builtin_expect(_r._cfg.kernel_page_cache, false)) {\n        // linux-aio is not asynchronous when the page cache is used,\n        // so we don't want to call io_submit() from the reactor thread.\n        //\n        // Pretend that all aio failed with EAGAIN and submit them\n        // via schedule_retry(), below.\n        did_work = !_submission_queue.empty();\n        for (auto& iocbp : _submission_queue) {\n            set_nowait(*iocbp, false);\n            _pending_aio_retry.push_back(iocbp);\n        }\n        to_submit = 0;\n    }\n\n    size_t nr_consumed = 0;\n    for (auto iocbs = _submission_queue.data(), end = iocbs + to_submit; iocbs < end; iocbs += nr_consumed) {\n        auto nr = end - iocbs;\n        auto r = io_submit(_io_context, nr, iocbs);\n        if (r == -1) {\n            nr_consumed = handle_aio_error(iocbs[0], errno);\n        } else {\n            nr_consumed = size_t(r);\n        }\n        did_work = true;\n    }\n\n    if (need_to_retry() && !retry_in_progress()) {\n        schedule_retry();\n    }\n\n    return did_work;\n}\n\nvoid aio_storage_context::schedule_retry() {\n    // loop until both _pending_aio_retry and _aio_retries are empty.\n    // While retrying _aio_retries, new retries may be queued onto _pending_aio_retry.\n    _pending_aio_retry_fut = do_until([this] {\n        if (_aio_retries.empty()) {\n            if (_pending_aio_retry.empty()) {\n                return true;\n            }\n            // _pending_aio_retry, holding a batch of new iocbs to retry,\n            // is swapped with the empty _aio_retries.\n            std::swap(_aio_retries, _pending_aio_retry);\n        }\n        return false;\n    }, [this] {\n        return _r._thread_pool->submit<syscall_result<int>>(\n                internal::thread_pool_submit_reason::aio_fallback, [this] () mutable {\n            auto r = io_submit(_io_context, _aio_retries.size(), _aio_retries.data());\n            return wrap_syscall<int>(r);\n        }).then_wrapped([this] (future<syscall_result<int>> f) {\n            // If submit failed, just log the error and exit the loop.\n            // The next call to submit_work will call schedule_retry again.\n            if (f.failed()) {\n                auto ex = f.get_exception();\n                seastar_logger.warn(\"aio_storage_context::schedule_retry failed: {}\", std::move(ex));\n                return;\n            }\n            auto result = f.get();\n            auto iocbs = _aio_retries.data();\n            size_t nr_consumed = 0;\n            if (result.result == -1) {\n                try {\n                    nr_consumed = handle_aio_error(iocbs[0], result.error);\n                } catch (...) {\n                    seastar_logger.error(\"aio retry failed: {}. Aborting.\", std::current_exception());\n                    abort();\n                }\n            } else {\n                nr_consumed = result.result;\n            }\n            _aio_retries.erase(_aio_retries.begin(), _aio_retries.begin() + nr_consumed);\n        });\n    });\n}\n\nbool aio_storage_context::reap_completions(bool allow_retry)\n{\n    struct timespec timeout = {0, 0};\n    auto n = io_getevents(_io_context, 1, max_aio, _ev_buffer, &timeout, _r._cfg.force_io_getevents_syscall);\n    if (n == -1 && errno == EINTR) {\n        n = 0;\n    }\n    SEASTAR_ASSERT(n >= 0);\n    for (size_t i = 0; i < size_t(n); ++i) {\n        auto iocb = get_iocb(_ev_buffer[i]);\n        if (_ev_buffer[i].res == -EAGAIN && allow_retry) {\n            set_nowait(*iocb, false);\n            _r._io_stats.aio_retries++;\n            _pending_aio_retry.push_back(iocb);\n            continue;\n        }\n        _iocb_pool.put_one(iocb);\n        auto desc = get_user_data<kernel_completion>(_ev_buffer[i]);\n        desc->complete_with(_ev_buffer[i].res);\n    }\n    return n;\n}\n\nbool aio_storage_context::can_sleep() const {\n    // Because aio depends on polling, it cannot generate events to wake us up, Therefore, sleep\n    // is only possible if there are no in-flight aios. If there are, we need to keep polling.\n    //\n    // Alternatively, if we enabled _aio_eventfd, we can always enter\n    unsigned executing = _iocb_pool.outstanding();\n    return executing == 0 || _r._aio_eventfd;\n}\n\naio_general_context::aio_general_context(size_t nr)\n        : iocbs(new iocb*[nr])\n        , last(iocbs.get())\n        , end(iocbs.get() + nr)\n{\n    setup_aio_context(nr, &io_context);\n}\n\naio_general_context::~aio_general_context() {\n    io_destroy(io_context);\n}\n\nvoid aio_general_context::queue(linux_abi::iocb* iocb) {\n    SEASTAR_ASSERT(last < end);\n    *last++ = iocb;\n}\n\nsize_t aio_general_context::flush() {\n    auto begin = iocbs.get();\n    using clock = std::chrono::steady_clock;\n    constexpr clock::time_point no_time_point = clock::time_point(clock::duration(0));\n    auto retry_until = no_time_point;\n    while (begin != last) {\n        auto r = io_submit(io_context, last - begin, begin);\n        if (__builtin_expect(r > 0, true)) {\n            begin += r;\n            continue;\n        }\n        // errno == EAGAIN is expected here. We don't explicitly assert that\n        // since the assert below prevents an endless loop for any reason.\n        if (retry_until == no_time_point) {\n            // allow retrying for 1 second\n            retry_until = clock::now() + 1s;\n        } else {\n            SEASTAR_ASSERT(clock::now() < retry_until);\n        }\n    }\n    auto nr = last - iocbs.get();\n    last = iocbs.get();\n    return nr;\n}\n\ncompletion_with_iocb::completion_with_iocb(int fd, int events, void* user_data)\n    : _iocb(make_poll_iocb(fd, events)) {\n    set_user_data(_iocb, user_data);\n}\n\nvoid completion_with_iocb::maybe_queue(aio_general_context& context) {\n    if (!_in_context) {\n        _in_context = true;\n        context.queue(&_iocb);\n    }\n}\n\nhrtimer_aio_completion::hrtimer_aio_completion(reactor& r, file_desc& fd)\n    : fd_kernel_completion(fd)\n    , completion_with_iocb(fd.get(), POLLIN, this)\n    , _r(r) {}\n\ntask_quota_aio_completion::task_quota_aio_completion(file_desc& fd)\n    : fd_kernel_completion(fd)\n    , completion_with_iocb(fd.get(), POLLIN, this) {}\n\nsmp_wakeup_aio_completion::smp_wakeup_aio_completion(file_desc& fd)\n        : fd_kernel_completion(fd)\n        , completion_with_iocb(fd.get(), POLLIN, this) {}\n\nvoid\nhrtimer_aio_completion::complete_with(ssize_t ret) {\n    uint64_t expirations = 0;\n    (void)_fd.read(&expirations, 8);\n    if (expirations) {\n        _r.service_highres_timer();\n    }\n    completion_with_iocb::completed();\n}\n\nvoid\ntask_quota_aio_completion::complete_with(ssize_t ret) {\n    uint64_t v;\n    (void)_fd.read(&v, 8);\n    completion_with_iocb::completed();\n}\n\nvoid\nsmp_wakeup_aio_completion::complete_with(ssize_t ret) {\n    uint64_t ignore = 0;\n    (void)_fd.read(&ignore, 8);\n    completion_with_iocb::completed();\n}\n\npreempt_io_context::preempt_io_context(reactor& r, file_desc& task_quota, file_desc& hrtimer)\n    : _r(r)\n    , _task_quota_aio_completion(task_quota)\n    , _hrtimer_aio_completion(r, hrtimer)\n{}\n\nvoid preempt_io_context::start_tick() {\n    // Preempt whenever an event (timer tick or signal) is available on the\n    // _preempting_io ring\n    set_need_preempt_var(reinterpret_cast<const preemption_monitor*>(_context.io_context + 8));\n    // preempt_io_context::request_preemption() will write to reactor::_preemption_monitor, which is now ignored\n}\n\nvoid preempt_io_context::stop_tick() {\n    set_need_preempt_var(&_r._preemption_monitor);\n}\n\nvoid preempt_io_context::request_preemption() {\n    ::itimerspec expired = {};\n    expired.it_value.tv_nsec = 1;\n    // will trigger immediately, triggering the preemption monitor\n    _hrtimer_aio_completion.fd().timerfd_settime(TFD_TIMER_ABSTIME, expired);\n\n    // This might have been called from poll_once. If that is the case, we cannot assume that timerfd is being\n    // monitored.\n    _hrtimer_aio_completion.maybe_queue(_context);\n    _context.flush();\n\n    // The kernel is not obliged to deliver the completion immediately, so wait for it\n    while (!need_preempt()) {\n        std::atomic_signal_fence(std::memory_order_seq_cst);\n    }\n}\n\nvoid preempt_io_context::reset_preemption_monitor() {\n    service_preempting_io();\n    _hrtimer_aio_completion.maybe_queue(_context);\n    _task_quota_aio_completion.maybe_queue(_context);\n    flush();\n}\n\nbool preempt_io_context::service_preempting_io() {\n    linux_abi::io_event a[2];\n    auto r = io_getevents(_context.io_context, 0, 2, a, 0);\n    SEASTAR_ASSERT(r != -1);\n    bool did_work = r > 0;\n    for (unsigned i = 0; i != unsigned(r); ++i) {\n        auto desc = get_user_data<kernel_completion>(a[i]);\n        desc->complete_with(a[i].res);\n    }\n    return did_work;\n}\n\nfile_desc reactor_backend_aio::make_timerfd() {\n    return file_desc::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC|TFD_NONBLOCK);\n}\n\nbool reactor_backend_aio::await_events(int timeout, const sigset_t* active_sigmask) {\n    ::timespec ts = {};\n    ::timespec* tsp = [&] () -> ::timespec* {\n        if (timeout == 0) {\n            return &ts;\n        } else if (timeout == -1) {\n            return nullptr;\n        } else {\n            ts = posix::to_timespec(timeout * 1ms);\n            return &ts;\n        }\n    }();\n    constexpr size_t batch_size = 128;\n    io_event batch[batch_size];\n    bool did_work = false;\n    int r;\n    do {\n        const bool may_sleep = !tsp || (tsp->tv_nsec + tsp->tv_sec > 0);\n        const auto before_getevents = may_sleep ? sched_clock::now() : sched_clock::time_point{};\n        r = io_pgetevents(_polling_io.io_context, 1, batch_size, batch, tsp, active_sigmask);\n        if (may_sleep) {\n            _r._total_sleep += sched_clock::now() - before_getevents;\n        }\n        if (r == -1 && errno == EINTR) {\n            return true;\n        }\n        SEASTAR_ASSERT(r != -1);\n        for (unsigned i = 0; i != unsigned(r); ++i) {\n            did_work = true;\n            auto& event = batch[i];\n            auto* desc = get_user_data<kernel_completion>(event);\n            desc->complete_with(event.res);\n        }\n        // For the next iteration, don't use a timeout, since we may have waited already\n        ts = {};\n        tsp = &ts;\n    } while (r == batch_size);\n    return did_work;\n}\n\nvoid reactor_backend_aio::signal_received(int signo, siginfo_t* siginfo, void* ignore) {\n    _r._signals.action(signo, siginfo, ignore);\n}\n\nreactor_backend_aio::reactor_backend_aio(reactor& r)\n    : reactor_backend(uses_blocking_io::no, supports_aio_fdatasync::yes)\n    , _r(r)\n    , _hrtimer_timerfd(make_timerfd())\n    , _storage_context(_r)\n    , _preempting_io(_r, _r._task_quota_timer, _hrtimer_timerfd)\n    , _polling_io(_r._cfg.max_networking_aio_io_control_blocks)\n    , _hrtimer_poll_completion(_r, _hrtimer_timerfd)\n    , _smp_wakeup_aio_completion(_r._notify_eventfd)\n{\n    // Protect against spurious wakeups - if we get notified that the timer has\n    // expired when it really hasn't, we don't want to block in read(tfd, ...).\n    auto tfd = _r._task_quota_timer.get();\n    ::fcntl(tfd, F_SETFL, ::fcntl(tfd, F_GETFL) | O_NONBLOCK);\n\n    sigset_t mask = make_sigset_mask(hrtimer_signal());\n    auto e = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n    SEASTAR_ASSERT(e == 0);\n}\n\nstd::string_view reactor_backend_aio::get_backend_name() const {\n    return \"linux-aio\";\n}\n\nbool reactor_backend_aio::reap_kernel_completions() {\n    bool did_work = await_events(0, nullptr);\n    did_work |= _storage_context.reap_completions();\n    return did_work;\n}\n\nbool reactor_backend_aio::kernel_submit_work() {\n    _hrtimer_poll_completion.maybe_queue(_polling_io);\n    bool did_work = _polling_io.flush();\n    did_work |= _storage_context.submit_work();\n    return did_work;\n}\n\nbool reactor_backend_aio::kernel_events_can_sleep() const {\n    return _storage_context.can_sleep();\n}\n\nvoid reactor_backend_aio::wait_and_process_events(const sigset_t* active_sigmask) {\n    int timeout = -1;\n    bool did_work = _preempting_io.service_preempting_io();\n    if (did_work) {\n        timeout = 0;\n    }\n\n    _hrtimer_poll_completion.maybe_queue(_polling_io);\n    _smp_wakeup_aio_completion.maybe_queue(_polling_io);\n    _polling_io.flush();\n    await_events(timeout, active_sigmask);\n    _preempting_io.service_preempting_io(); // clear task quota timer\n}\n\nclass aio_pollable_fd_state : public pollable_fd_state {\n    internal::linux_abi::iocb _iocb_pollin;\n    pollable_fd_state_completion _completion_pollin;\n\n    internal::linux_abi::iocb _iocb_pollout;\n    pollable_fd_state_completion _completion_pollout;\n\n    internal::linux_abi::iocb _iocb_pollrdhup;\n    pollable_fd_state_completion _completion_pollrdhup;\npublic:\n    pollable_fd_state_completion* get_desc(int events) {\n        if (events & POLLIN) {\n            return &_completion_pollin;\n        }\n        if (events & POLLOUT) {\n            return &_completion_pollout;\n        }\n        return &_completion_pollrdhup;\n    }\n    internal::linux_abi::iocb* get_iocb(int events) {\n        if (events & POLLIN) {\n            return &_iocb_pollin;\n        }\n        if (events & POLLOUT) {\n            return &_iocb_pollout;\n        }\n        return &_iocb_pollrdhup;\n    }\n    explicit aio_pollable_fd_state(file_desc fd, speculation speculate)\n        : pollable_fd_state(std::move(fd), std::move(speculate))\n    {}\n    future<> get_completion_future(int events) {\n        return get_desc(events)->get_future();\n    }\n};\n\nfuture<> reactor_backend_aio::poll(pollable_fd_state& fd, int events) {\n    try {\n        if (events & fd.events_known) {\n            fd.events_known &= ~events;\n            return make_ready_future<>();\n        }\n\n        fd.events_rw = events == (POLLIN|POLLOUT);\n\n        auto* pfd = static_cast<aio_pollable_fd_state*>(&fd);\n        auto* iocb = pfd->get_iocb(events);\n        auto* desc = pfd->get_desc(events);\n        *iocb = make_poll_iocb(fd.fd.get(), events);\n        *desc = pollable_fd_state_completion{};\n        set_user_data(*iocb, desc);\n        _polling_io.queue(iocb);\n        return pfd->get_completion_future(events);\n    } catch (...) {\n        return make_exception_future<>(std::current_exception());\n    }\n}\n\nfuture<> reactor_backend_aio::readable(pollable_fd_state& fd) {\n    return poll(fd, POLLIN);\n}\n\nfuture<> reactor_backend_aio::writeable(pollable_fd_state& fd) {\n    return poll(fd, POLLOUT);\n}\n\nfuture<> reactor_backend_aio::readable_or_writeable(pollable_fd_state& fd) {\n    return poll(fd, POLLIN|POLLOUT);\n}\n\nfuture<> reactor_backend_aio::poll_rdhup(pollable_fd_state& fd) {\n    return poll(fd, POLLRDHUP);\n}\n\nvoid reactor_backend_aio::forget(pollable_fd_state& fd) noexcept {\n    auto* pfd = static_cast<aio_pollable_fd_state*>(&fd);\n    delete pfd;\n    // ?\n}\n\nfuture<std::tuple<pollable_fd, socket_address>>\nreactor_backend_aio::accept(pollable_fd_state& listenfd) {\n    return _r.do_accept(listenfd);\n}\n\nfuture<> reactor_backend_aio::connect(pollable_fd_state& fd, socket_address& sa) {\n    return _r.do_connect(fd, sa);\n}\n\nfuture<size_t>\nreactor_backend_aio::read(pollable_fd_state& fd, void* buffer, size_t len) {\n    return _r.do_read(fd, buffer, len);\n}\n\nfuture<size_t>\nreactor_backend_aio::recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) {\n    return _r.do_recvmsg(fd, iov);\n}\n\nfuture<temporary_buffer<char>>\nreactor_backend_aio::read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return _r.do_read_some(fd, ba);\n}\n\n#if SEASTAR_API_LEVEL < 9\nfuture<size_t>\nreactor_backend_aio::send(pollable_fd_state& fd, const void* buffer, size_t len) {\n    return _r.do_send(fd, buffer, len);\n}\n#endif\n\nfuture<size_t>\nreactor_backend_aio::sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) {\n    return _r.do_sendmsg(fd, iovs, len);\n}\n\nfuture<temporary_buffer<char>>\nreactor_backend_aio::recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return _r.do_recv_some(fd, ba);\n}\n\nvoid reactor_backend_aio::start_tick() {\n    _preempting_io.start_tick();\n}\n\nvoid reactor_backend_aio::stop_tick() {\n    _preempting_io.stop_tick();\n}\n\nvoid reactor_backend_aio::arm_highres_timer(const ::itimerspec& its) {\n    _hrtimer_timerfd.timerfd_settime(TFD_TIMER_ABSTIME, its);\n}\n\nvoid reactor_backend_aio::reset_preemption_monitor() {\n    _preempting_io.reset_preemption_monitor();\n}\n\nvoid reactor_backend_aio::request_preemption() {\n    _preempting_io.request_preemption();\n}\n\nvoid reactor_backend_aio::start_handling_signal() {\n    // The aio backend only uses SIGHUP/SIGTERM/SIGINT. We don't need to handle them right away and our\n    // implementation of request_preemption is not signal safe, so do nothing.\n}\n\npollable_fd_state_ptr\nreactor_backend_aio::make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) {\n    return pollable_fd_state_ptr(new aio_pollable_fd_state(std::move(fd), std::move(speculate)));\n}\n\nreactor_backend_epoll::reactor_backend_epoll(reactor& r)\n        : reactor_backend(uses_blocking_io::no, supports_aio_fdatasync::yes)\n        , _r(r)\n        , _steady_clock_timer_reactor_thread(file_desc::timerfd_create(CLOCK_MONOTONIC, TFD_NONBLOCK|TFD_CLOEXEC))\n        , _steady_clock_timer_timer_thread(file_desc::timerfd_create(CLOCK_MONOTONIC, TFD_NONBLOCK|TFD_CLOEXEC))\n        , _epollfd(file_desc::epoll_create(EPOLL_CLOEXEC))\n        , _storage_context(_r) {\n    ::epoll_event event;\n    event.events = EPOLLIN;\n    event.data.ptr = nullptr;\n    auto ret = ::epoll_ctl(_epollfd.get(), EPOLL_CTL_ADD, _r._notify_eventfd.get(), &event);\n    throw_system_error_on(ret == -1);\n    event.events = EPOLLIN;\n    event.data.ptr = &_steady_clock_timer_reactor_thread;\n    ret = ::epoll_ctl(_epollfd.get(), EPOLL_CTL_ADD, _steady_clock_timer_reactor_thread.get(), &event);\n    throw_system_error_on(ret == -1);\n}\n\nvoid\nreactor_backend_epoll::task_quota_timer_thread_fn() {\n    auto thread_name = seastar::format(\"timer-{}\", _r._id);\n    pthread_setname_np(pthread_self(), thread_name.c_str());\n\n    sigset_t mask;\n    sigfillset(&mask);\n    for (auto sig : { SIGSEGV }) {\n        sigdelset(&mask, sig);\n    }\n    auto r = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n    if (r) {\n        seastar_logger.error(\"Thread {}: failed to block signals. Aborting.\", thread_name.c_str());\n        abort();\n    }\n\n    // We need to wait until task quota is set before we can calculate how many ticks are to\n    // a minute. Technically task_quota is used from many threads, but since it is read-only here\n    // and only used during initialization we will avoid complicating the code.\n    {\n        uint64_t events;\n        _r._task_quota_timer.read(&events, 8);\n        _r.request_preemption();\n    }\n\n    while (!_dying.load(std::memory_order_relaxed)) {\n        // Wait for either the task quota timer, or the high resolution timer, or both,\n        // to expire.\n        struct pollfd pfds[2] = {};\n        pfds[0].fd = _r._task_quota_timer.get();\n        pfds[0].events = POLL_IN;\n        pfds[1].fd = _steady_clock_timer_timer_thread.get();\n        pfds[1].events = POLL_IN;\n        int r = poll(pfds, 2, -1);\n        SEASTAR_ASSERT(r != -1);\n\n        uint64_t events;\n        if (pfds[0].revents & POLL_IN) {\n            _r._task_quota_timer.read(&events, 8);\n        }\n        if (pfds[1].revents & POLL_IN) {\n            _steady_clock_timer_timer_thread.read(&events, 8);\n            _highres_timer_pending.store(true, std::memory_order_relaxed);\n        }\n        _r.request_preemption();\n\n        // We're in a different thread, but guaranteed to be on the same core, so even\n        // a signal fence is overdoing it\n        std::atomic_signal_fence(std::memory_order_seq_cst);\n    }\n}\n\nreactor_backend_epoll::~reactor_backend_epoll() = default;\n\nstd::string_view reactor_backend_epoll::get_backend_name() const {\n    return \"epoll\";\n}\n\nvoid reactor_backend_epoll::start_tick() {\n    _task_quota_timer_thread = std::thread(&reactor_backend_epoll::task_quota_timer_thread_fn, this);\n\n    ::sched_param sp;\n    sp.sched_priority = 1;\n    auto sched_ok = pthread_setschedparam(_task_quota_timer_thread.native_handle(), SCHED_FIFO, &sp);\n    if (sched_ok != 0 && _r._id == 0) {\n        seastar_logger.warn(\"Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE\");\n    }\n}\n\nvoid reactor_backend_epoll::stop_tick() {\n    _dying.store(true, std::memory_order_relaxed);\n    _r._task_quota_timer.timerfd_settime(0, seastar::posix::to_relative_itimerspec(1ns, 1ms)); // Make the timer fire soon\n    _task_quota_timer_thread.join();\n}\n\nvoid reactor_backend_epoll::arm_highres_timer(const ::itimerspec& its) {\n    _steady_clock_timer_deadline = its;\n    _steady_clock_timer_timer_thread.timerfd_settime(TFD_TIMER_ABSTIME, its);\n}\n\nvoid\nreactor_backend_epoll::switch_steady_clock_timers(file_desc& from, file_desc& to) {\n    auto& deadline = _steady_clock_timer_deadline;\n    if (deadline.it_value.tv_sec == 0 && deadline.it_value.tv_nsec == 0) {\n        return;\n    }\n    // Enable-then-disable, so the hardware timer doesn't have to be reprogrammed. Probably pointless.\n    to.timerfd_settime(TFD_TIMER_ABSTIME, _steady_clock_timer_deadline);\n    from.timerfd_settime(TFD_TIMER_ABSTIME, {});\n}\n\nvoid reactor_backend_epoll::maybe_switch_steady_clock_timers(int timeout, file_desc& from, file_desc& to) {\n    if (timeout != 0) {\n        switch_steady_clock_timers(from, to);\n    }\n}\n\nbool\nreactor_backend_epoll::wait_and_process(int timeout, const sigset_t* active_sigmask) {\n    // If we plan to sleep, disable the timer thread steady clock timer (since it won't\n    // wake us up from sleep, and timer thread wakeup will just waste CPU time) and enable\n    // reactor thread steady clock timer.\n    maybe_switch_steady_clock_timers(timeout, _steady_clock_timer_timer_thread, _steady_clock_timer_reactor_thread);\n    auto undo_timer_switch = defer([&] () noexcept {\n      try {\n        maybe_switch_steady_clock_timers(timeout, _steady_clock_timer_reactor_thread, _steady_clock_timer_timer_thread);\n      } catch (...) {\n        seastar_logger.error(\"Switching steady_clock timers back failed: {}. Aborting...\", std::current_exception());\n        abort();\n      }\n    });\n    std::array<epoll_event, 128> eevt;\n    const auto before_pwait = sched_clock::now();\n    int nr = ::epoll_pwait(_epollfd.get(), eevt.data(), eevt.size(), timeout, active_sigmask);\n    _r._total_sleep += sched_clock::now() - before_pwait;\n    if (nr == -1 && errno == EINTR) {\n        return false; // gdb can cause this\n    }\n    SEASTAR_ASSERT(nr != -1);\n    for (int i = 0; i < nr; ++i) {\n        auto& evt = eevt[i];\n        auto pfd = reinterpret_cast<pollable_fd_state*>(evt.data.ptr);\n        if (!pfd) {\n            char dummy[8];\n            _r._notify_eventfd.read(dummy, 8);\n            continue;\n        }\n        if (evt.data.ptr == &_steady_clock_timer_reactor_thread) {\n            char dummy[8];\n            _steady_clock_timer_reactor_thread.read(dummy, 8);\n            _highres_timer_pending.store(true, std::memory_order_relaxed);\n            _steady_clock_timer_deadline = {};\n            continue;\n        }\n        bool has_error = evt.events & (EPOLLHUP | EPOLLERR);\n        if (has_error) {\n            // treat the events as required events when error occurs, let\n            // send/recv/accept/connect handle the specific error.\n            evt.events = pfd->events_requested;\n        }\n        auto events = evt.events & (EPOLLIN | EPOLLOUT | EPOLLRDHUP);\n        auto events_to_remove = events & ~pfd->events_requested;\n        complete_epoll_event(*pfd, events, EPOLLRDHUP);\n        if (pfd->events_rw) {\n            // accept() signals normal completions via EPOLLIN, but errors (due to shutdown())\n            // via EPOLLOUT|EPOLLHUP, so we have to wait for both EPOLLIN and EPOLLOUT with the\n            // same future\n            complete_epoll_event(*pfd, events, EPOLLIN|EPOLLOUT);\n        } else {\n            // Normal processing where EPOLLIN and EPOLLOUT are waited for via different\n            // futures.\n            complete_epoll_event(*pfd, events, EPOLLIN);\n            complete_epoll_event(*pfd, events, EPOLLOUT);\n        }\n        if (events_to_remove) {\n            pfd->events_epoll &= ~events_to_remove;\n            evt.events = pfd->events_epoll;\n            auto op = evt.events ? EPOLL_CTL_MOD : EPOLL_CTL_DEL;\n            ::epoll_ctl(_epollfd.get(), op, pfd->fd.get(), &evt);\n        } else if (has_error) {\n            // In the error case, all requested events are cleared (as we handle\n            // all requested events on error), so unconditionally delete the fd\n            // from epoll, which avoids edge conditions where we otherwise might\n            // get stuck spinning.\n            pfd->events_epoll = 0;\n            ::epoll_ctl(_epollfd.get(), EPOLL_CTL_DEL, pfd->fd.get(), nullptr);\n        }\n    }\n    return nr;\n}\n\nclass epoll_pollable_fd_state : public pollable_fd_state {\n    pollable_fd_state_completion _pollin;\n    pollable_fd_state_completion _pollout;\n    pollable_fd_state_completion _pollrdhup;\n\n    pollable_fd_state_completion* get_desc(int events) {\n        if (events & EPOLLIN) {\n            return &_pollin;\n        }\n        if (events & EPOLLOUT) {\n            return &_pollout;\n        }\n        return &_pollrdhup;\n    }\npublic:\n    explicit epoll_pollable_fd_state(file_desc fd, speculation speculate)\n        : pollable_fd_state(std::move(fd), std::move(speculate))\n    {}\n    future<> get_completion_future(int event) {\n        auto desc = get_desc(event);\n        *desc = pollable_fd_state_completion{};\n        return desc->get_future();\n    }\n\n    void complete_with(int event) {\n        get_desc(event)->complete_with(event);\n    }\n};\n\nbool reactor_backend_epoll::reap_kernel_completions() {\n    // epoll does not have a separate submission stage, and just\n    // calls epoll_ctl everytime it needs, so this method and\n    // kernel_submit_work are essentially the same. Ordering also\n    // doesn't matter much. wait_and_process is actually completing,\n    // but we prefer to call it in kernel_submit_work because the\n    // reactor register two pollers for completions and one for submission,\n    // since completion is cheaper for other backends like aio. This avoids\n    // calling epoll_wait twice.\n    //\n    // We will only reap the io completions\n    return _storage_context.reap_completions();\n}\n\nbool reactor_backend_epoll::kernel_submit_work() {\n    bool result = false;\n    _storage_context.submit_work();\n    if (_need_epoll_events) {\n        result |= wait_and_process(0, nullptr);\n    }\n\n    result |= complete_hrtimer();\n\n    return result;\n}\n\nbool reactor_backend_epoll::complete_hrtimer() {\n    // This can be set from either the task quota timer thread, or\n    // wait_and_process(), above.\n    if (_highres_timer_pending.load(std::memory_order_relaxed)) {\n        _highres_timer_pending.store(false, std::memory_order_relaxed);\n        _r.service_highres_timer();\n        return true;\n    }\n    return false;\n}\n\nbool reactor_backend_epoll::kernel_events_can_sleep() const {\n    return _storage_context.can_sleep();\n}\n\nvoid reactor_backend_epoll::wait_and_process_events(const sigset_t* active_sigmask) {\n    wait_and_process(-1 , active_sigmask);\n    complete_hrtimer();\n}\n\nvoid reactor_backend_epoll::complete_epoll_event(pollable_fd_state& pfd, int events, int event) {\n    if (pfd.events_requested & events & event) {\n        pfd.events_requested &= ~event;\n        pfd.events_known &= ~event;\n        auto* fd = static_cast<epoll_pollable_fd_state*>(&pfd);\n        return fd->complete_with(event);\n    }\n}\n\nvoid reactor_backend_epoll::signal_received(int signo, siginfo_t* siginfo, void* ignore) {\n    if (engine_is_ready()) {\n        _r._signals.action(signo, siginfo, ignore);\n    } else {\n        reactor::signals::failed_to_handle(signo);\n    }\n}\n\nfuture<> reactor_backend_epoll::get_epoll_future(pollable_fd_state& pfd, int event) {\n    if (pfd.events_known & event) {\n        pfd.events_known &= ~event;\n        return make_ready_future();\n    }\n    pfd.events_rw = event == (EPOLLIN | EPOLLOUT);\n    pfd.events_requested |= event;\n    if ((pfd.events_epoll & event) != event) {\n        auto ctl = pfd.events_epoll ? EPOLL_CTL_MOD : EPOLL_CTL_ADD;\n        pfd.events_epoll |= event;\n        ::epoll_event eevt;\n        eevt.events = pfd.events_epoll;\n        eevt.data.ptr = &pfd;\n        int r = ::epoll_ctl(_epollfd.get(), ctl, pfd.fd.get(), &eevt);\n        SEASTAR_ASSERT(r == 0);\n        _need_epoll_events = true;\n    }\n\n    auto* fd = static_cast<epoll_pollable_fd_state*>(&pfd);\n    return fd->get_completion_future(event);\n}\n\nfuture<> reactor_backend_epoll::readable(pollable_fd_state& fd) {\n    return get_epoll_future(fd, EPOLLIN);\n}\n\nfuture<> reactor_backend_epoll::writeable(pollable_fd_state& fd) {\n    return get_epoll_future(fd, EPOLLOUT);\n}\n\nfuture<> reactor_backend_epoll::readable_or_writeable(pollable_fd_state& fd) {\n    return get_epoll_future(fd, EPOLLIN | EPOLLOUT);\n}\n\nfuture<> reactor_backend_epoll::poll_rdhup(pollable_fd_state& fd) {\n    return get_epoll_future(fd, POLLRDHUP);\n}\n\nvoid reactor_backend_epoll::forget(pollable_fd_state& fd) noexcept {\n    if (fd.events_epoll) {\n        ::epoll_ctl(_epollfd.get(), EPOLL_CTL_DEL, fd.fd.get(), nullptr);\n    }\n    auto* efd = static_cast<epoll_pollable_fd_state*>(&fd);\n    delete efd;\n}\n\nfuture<std::tuple<pollable_fd, socket_address>>\nreactor_backend_epoll::accept(pollable_fd_state& listenfd) {\n    return _r.do_accept(listenfd);\n}\n\nfuture<> reactor_backend_epoll::connect(pollable_fd_state& fd, socket_address& sa) {\n    return _r.do_connect(fd, sa);\n}\n\nfuture<size_t>\nreactor_backend_epoll::read(pollable_fd_state& fd, void* buffer, size_t len) {\n    return _r.do_read(fd, buffer, len);\n}\n\nfuture<size_t>\nreactor_backend_epoll::recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) {\n    return _r.do_recvmsg(fd, iov);\n}\n\nfuture<temporary_buffer<char>>\nreactor_backend_epoll::read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return _r.do_read_some(fd, ba);\n}\n\n#if SEASTAR_API_LEVEL < 9\nfuture<size_t>\nreactor_backend_epoll::send(pollable_fd_state& fd, const void* buffer, size_t len) {\n    return _r.do_send(fd, buffer, len);\n}\n#endif\n\nfuture<size_t>\nreactor_backend_epoll::sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) {\n    return _r.do_sendmsg(fd, iovs, len);\n}\n\nfuture<temporary_buffer<char>>\nreactor_backend_epoll::recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) {\n    return _r.do_recv_some(fd, ba);\n}\n\nvoid\nreactor_backend_epoll::request_preemption() {\n    _r._preemption_monitor.head.store(1, std::memory_order_relaxed);\n}\n\nvoid reactor_backend_epoll::start_handling_signal() {\n    // The epoll backend uses signals for the high resolution timer. That is used for thread_scheduling_group, so we\n    // request preemption so when we receive a signal.\n    request_preemption();\n}\n\npollable_fd_state_ptr\nreactor_backend_epoll::make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) {\n    return pollable_fd_state_ptr(new epoll_pollable_fd_state(std::move(fd), std::move(speculate)));\n}\n\nvoid reactor_backend_epoll::reset_preemption_monitor() {\n    _r._preemption_monitor.head.store(0, std::memory_order_relaxed);\n}\n\n#ifdef SEASTAR_HAVE_URING\n\nstatic\nstd::optional<::io_uring>\ntry_create_uring(unsigned queue_len, bool throw_on_error) {\n    auto required_features =\n            IORING_FEAT_SUBMIT_STABLE\n            | IORING_FEAT_NODROP;\n    auto required_ops = {\n            IORING_OP_POLL_ADD, // linux 5.1\n            IORING_OP_READV,\n            IORING_OP_WRITEV,\n            IORING_OP_FSYNC,\n            IORING_OP_SENDMSG,  // linux 5.3\n            IORING_OP_RECVMSG,\n            IORING_OP_ACCEPT,\n            IORING_OP_CONNECT,\n            IORING_OP_READ,     // linux 5.6\n            IORING_OP_WRITE,\n            IORING_OP_SEND,\n            IORING_OP_RECV,\n            };\n    auto maybe_throw = [&] (auto exception) {\n        if (throw_on_error) {\n            throw exception;\n        }\n    };\n\n    auto params = ::io_uring_params{};\n    ::io_uring ring;\n    auto err = ::io_uring_queue_init_params(queue_len, &ring, &params);\n    if (err != 0) {\n        maybe_throw(std::system_error(std::error_code(-err, std::system_category()), \"trying to create io_uring\"));\n        return std::nullopt;\n    }\n    auto free_ring = defer([&] () noexcept { ::io_uring_queue_exit(&ring); });\n    ::io_uring_ring_dontfork(&ring);\n    if (~ring.features & required_features) {\n        maybe_throw(std::runtime_error(fmt::format(\"missing required io_ring features, required 0x{:x} available 0x{:x}\", required_features, ring.features)));\n        return std::nullopt;\n    }\n\n    auto probe = ::io_uring_get_probe_ring(&ring);\n    if (!probe) {\n        maybe_throw(std::runtime_error(\"unable to create io_uring probe\"));\n        return std::nullopt;\n    }\n    auto free_probe = defer([&] () noexcept { ::io_uring_free_probe(probe); });\n\n    for (auto op : required_ops) {\n        if (!io_uring_opcode_supported(probe, op)) {\n            maybe_throw(std::runtime_error(fmt::format(\"required io_uring opcode {} not supported\", static_cast<int>(op))));\n            return std::nullopt;\n        }\n    }\n    free_ring.cancel();\n\n    return ring;\n}\n\nstatic\nbool\nhave_md_devices() {\n    namespace fs = std::filesystem;\n    for (auto entry : fs::directory_iterator(\"/sys/block\")) {\n        if (entry.is_directory() && fs::exists(entry.path() / \"md\")) {\n            return true;\n        }\n    }\n    return false;\n}\n\nstatic size_t mlock_limit() {\n    struct ::rlimit lim;\n    int r = ::getrlimit(RLIMIT_MEMLOCK, &lim);\n    if (r == -1) {\n        return 0; // assume the worst; this is advisory anyway\n    }\n    return lim.rlim_cur;\n}\n\nstatic\nbool\ndetect_io_uring() {\n    if (!kernel_uname().whitelisted({\"5.17\"}) && have_md_devices()) {\n        // Older kernels fall back to workqueues for RAID devices\n        return false;\n    }\n    if (!kernel_uname().whitelisted({\"5.12\"}) && mlock_limit() < (8 << 20)) {\n        // Older kernels lock about 32k/vcpu for the ring itself. Require 8MB of\n        // locked memory to be safe (8MB is what newer kernels and newer systemd provide)\n        return false;\n    }\n    auto ring_opt = try_create_uring(1, false);\n    if (ring_opt) {\n        ::io_uring_queue_exit(&ring_opt.value());\n    }\n    return bool(ring_opt);\n}\n\nclass reactor_backend_uring final : public reactor_backend {\n    // s_queue_len is more or less arbitrary. Too low and we'll be\n    // issuing too small batches, too high and we require too much locked\n    // memory, but otherwise it doesn't matter.\n    static constexpr unsigned s_queue_len = 200;\n    reactor& _r;\n    ::io_uring _uring;\n    bool _did_work_while_getting_sqe = false;\n    bool _has_pending_submissions = false;\n    file_desc _hrtimer_timerfd;\n    preempt_io_context _preempt_io_context;\n\n    class uring_pollable_fd_state : public pollable_fd_state {\n        pollable_fd_state_completion _completion_pollin;\n        pollable_fd_state_completion _completion_pollout;\n        pollable_fd_state_completion _completion_pollrdhup;\n    public:\n        explicit uring_pollable_fd_state(file_desc desc, speculation speculate)\n                : pollable_fd_state(std::move(desc), std::move(speculate)) {\n        }\n        pollable_fd_state_completion* get_desc(int events) {\n            if (events & POLLIN) {\n                return &_completion_pollin;\n            } else if (events & POLLOUT) {\n                return &_completion_pollout;\n            } else {\n                return &_completion_pollrdhup;\n            }\n        }\n        future<> get_completion_future(int events) {\n            return get_desc(events)->get_future();\n        }\n    };\n\n    // eventfd and timerfd both need an 8-byte read after completion\n    class recurring_eventfd_or_timerfd_completion : public fd_kernel_completion {\n        bool _armed = false;\n    public:\n        explicit recurring_eventfd_or_timerfd_completion(file_desc& fd) : fd_kernel_completion(fd) {}\n        virtual void complete_with(ssize_t res) override {\n            char garbage[8];\n            auto ret = _fd.read(garbage, 8);\n            // Note: for hrtimer_completion we can have spurious wakeups,\n            // since we wait for this using both _preempt_io_context and the\n            // ring. So don't assert that we read anything.\n            SEASTAR_ASSERT(!ret || *ret == 8);\n            _armed = false;\n        }\n        void maybe_rearm(reactor_backend_uring& be) {\n            if (_armed) {\n                return;\n            }\n            auto sqe = be.get_sqe();\n            ::io_uring_prep_poll_add(sqe, fd().get(), POLLIN);\n            ::io_uring_sqe_set_data(sqe, static_cast<kernel_completion*>(this));\n            _armed = true;\n            be._has_pending_submissions = true;\n        }\n    };\n\n    // Completion for high resolution timerfd, used in wait_and_process_events()\n    // (while running tasks it's waited for in _preempt_io_context)\n    class hrtimer_completion : public recurring_eventfd_or_timerfd_completion {\n        reactor& _r;\n    public:\n        explicit hrtimer_completion(reactor& r, file_desc& timerfd)\n                : recurring_eventfd_or_timerfd_completion(timerfd), _r(r) {\n        }\n        virtual void complete_with(ssize_t res) override {\n            recurring_eventfd_or_timerfd_completion::complete_with(res);\n            _r.service_highres_timer();\n        }\n    };\n\n    using smp_wakeup_completion = recurring_eventfd_or_timerfd_completion;\n\n    hrtimer_completion _hrtimer_completion;\n    smp_wakeup_completion _smp_wakeup_completion;\nprivate:\n    static file_desc make_timerfd() {\n        return file_desc::timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC|TFD_NONBLOCK);\n    }\n\n    // Can fail if the completion queue is full\n    ::io_uring_sqe* try_get_sqe() {\n        return ::io_uring_get_sqe(&_uring);\n    }\n\n    bool do_flush_submission_ring() {\n        if (_has_pending_submissions) {\n            _has_pending_submissions = false;\n            _did_work_while_getting_sqe = false;\n            io_uring_submit(&_uring);\n            return true;\n        } else {\n            return std::exchange(_did_work_while_getting_sqe, false);\n        }\n    }\n\n    ::io_uring_sqe* get_sqe() {\n        ::io_uring_sqe* sqe;\n        while (__builtin_expect((sqe = try_get_sqe()) == nullptr, false)) {\n            do_flush_submission_ring();\n            do_process_kernel_completions_step();\n            _did_work_while_getting_sqe = true;\n        }\n        return sqe;\n    }\n\n    future<> poll(pollable_fd_state& fd, int events) {\n        auto sqe = get_sqe();\n        ::io_uring_prep_poll_add(sqe, fd.fd.get(), events);\n        auto ufd = static_cast<uring_pollable_fd_state*>(&fd);\n        ::io_uring_sqe_set_data(sqe, static_cast<kernel_completion*>(ufd->get_desc(events)));\n        _has_pending_submissions = true;\n        return ufd->get_completion_future(events);\n    }\n\n    void submit_io_request(const internal::io_request& req, io_completion* completion) {\n        auto sqe = get_sqe();\n        using o = internal::io_request::operation;\n        switch (req.opcode()) {\n            case o::read: {\n                const auto& op = req.as<io_request::operation::read>();\n                ::io_uring_prep_read(sqe, op.fd, op.addr, op.size, op.pos);\n                break;\n            }\n            case o::write: {\n                const auto& op = req.as<io_request::operation::write>();\n                ::io_uring_prep_write(sqe, op.fd, op.addr, op.size, op.pos);\n                break;\n            }\n            case o::readv: {\n                const auto& op = req.as<io_request::operation::readv>();\n                ::io_uring_prep_readv(sqe, op.fd, op.iovec, op.iov_len, op.pos);\n                break;\n            }\n            case o::writev: {\n                const auto& op = req.as<io_request::operation::writev>();\n                ::io_uring_prep_writev(sqe, op.fd, op.iovec, op.iov_len, op.pos);\n                break;\n            }\n            case o::fdatasync: {\n                const auto& op = req.as<io_request::operation::fdatasync>();\n                ::io_uring_prep_fsync(sqe, op.fd, IORING_FSYNC_DATASYNC);\n                break;\n            }\n            case o::recv: {\n                const auto& op = req.as<io_request::operation::recv>();\n                ::io_uring_prep_recv(sqe, op.fd, op.addr, op.size, op.flags);\n                break;\n            }\n            case o::recvmsg: {\n                const auto& op = req.as<io_request::operation::recvmsg>();\n                ::io_uring_prep_recvmsg(sqe, op.fd, op.msghdr, op.flags);\n                break;\n            }\n            case o::send: {\n                const auto& op = req.as<io_request::operation::send>();\n                ::io_uring_prep_send(sqe, op.fd, op.addr, op.size, op.flags);\n                break;\n            }\n            case o::sendmsg: {\n                const auto& op = req.as<io_request::operation::sendmsg>();\n                ::io_uring_prep_sendmsg(sqe, op.fd, op.msghdr, op.flags);\n                break;\n            }\n            case o::accept: {\n                const auto& op = req.as<io_request::operation::accept>();\n                ::io_uring_prep_accept(sqe, op.fd, op.sockaddr, op.socklen_ptr, op.flags);\n                break;\n            }\n            case o::connect: {\n                const auto& op = req.as<io_request::operation::connect>();\n                ::io_uring_prep_connect(sqe, op.fd, op.sockaddr, op.socklen);\n                break;\n            }\n            case o::poll_add:\n            case o::poll_remove:\n            case o::cancel:\n                // The reactor does not generate these types of I/O requests yet, so\n                // this path is unreachable. As more features of io_uring are exploited,\n                // we'll utilize more of these opcodes.\n                seastar_logger.error(\"Invalid operation for iocb: {}\", req.opname());\n                abort();\n        }\n        ::io_uring_sqe_set_data(sqe, completion);\n\n        _has_pending_submissions = true;\n    }\n\n    // Returns true if any work was done\n    bool queue_pending_file_io() {\n        return _r._io_sink.drain([&] (const internal::io_request& req, io_completion* completion) -> bool {\n            submit_io_request(req, completion);\n            return true;\n        });\n    }\n\n    // Process kernel completions already extracted from the ring.\n    // This is needed because we sometimes extract completions without\n    // waiting, and sometimes with waiting.\n    void do_process_ready_kernel_completions(::io_uring_cqe** buf, size_t nr) {\n        for (auto p = buf; p != buf + nr; ++p) {\n            auto cqe = *p;\n            auto completion = reinterpret_cast<kernel_completion*>(cqe->user_data);\n            completion->complete_with(cqe->res);\n        }\n    }\n\n    // Returns true if completions were processed\n    bool do_process_kernel_completions_step() {\n        struct ::io_uring_cqe* buf[s_queue_len];\n        auto n = ::io_uring_peek_batch_cqe(&_uring, buf, s_queue_len);\n        do_process_ready_kernel_completions(buf, n);\n        ::io_uring_cq_advance(&_uring, n);\n        return n != 0;\n    }\n\n    // Returns true if completions were processed\n    bool do_process_kernel_completions() {\n        auto did_work = false;\n        while (do_process_kernel_completions_step()) {\n            did_work = true;\n        }\n        return did_work | std::exchange(_did_work_while_getting_sqe, false);\n    }\n\n    template<typename Completion>\n    auto submit_request(std::unique_ptr<Completion> desc, io_request&& req) noexcept {\n        auto fut = desc->get_future();\n        _r._io_sink.submit(desc.release(), std::move(req));\n        return fut;\n    }\npublic:\n    explicit reactor_backend_uring(reactor& r)\n            : reactor_backend(uses_blocking_io::yes, supports_aio_fdatasync::yes)\n            , _r(r)\n            , _uring(try_create_uring(s_queue_len, true).value())\n            , _hrtimer_timerfd(make_timerfd())\n            , _preempt_io_context(_r, _r._task_quota_timer, _hrtimer_timerfd)\n            , _hrtimer_completion(_r, _hrtimer_timerfd)\n            , _smp_wakeup_completion(_r._notify_eventfd) {\n        // Protect against spurious wakeups - if we get notified that the timer has\n        // expired when it really hasn't, we don't want to block in read(tfd, ...).\n        auto tfd = _r._task_quota_timer.get();\n        ::fcntl(tfd, F_SETFL, ::fcntl(tfd, F_GETFL) | O_NONBLOCK);\n    }\n    ~reactor_backend_uring() {\n        ::io_uring_queue_exit(&_uring);\n    }\n    virtual std::string_view get_backend_name() const override {\n        return \"io_uring\";\n    }\n    virtual bool reap_kernel_completions() override {\n        return do_process_kernel_completions();\n    }\n    virtual bool kernel_submit_work() override {\n        bool did_work = false;\n        did_work |= _preempt_io_context.service_preempting_io();\n        did_work |= queue_pending_file_io();\n        did_work |= ::io_uring_submit(&_uring);\n        return did_work;\n    }\n    virtual bool kernel_events_can_sleep() const override {\n        // We never need to spin while I/O is in flight.\n        return true;\n    }\n    virtual void wait_and_process_events(const sigset_t* active_sigmask) override {\n        _smp_wakeup_completion.maybe_rearm(*this);\n        _hrtimer_completion.maybe_rearm(*this);\n        ::io_uring_submit(&_uring);\n        bool did_work = false;\n        did_work |= _preempt_io_context.service_preempting_io();\n        did_work |= std::exchange(_did_work_while_getting_sqe, false);\n        if (did_work) {\n            return;\n        }\n        struct ::io_uring_cqe* cqe = nullptr;\n        sigset_t sigs = *active_sigmask; // io_uring_wait_cqes() wants non-const\n        const auto before_wait_cqes = sched_clock::now();\n        auto r = ::io_uring_wait_cqes(&_uring, &cqe, 1, nullptr, &sigs);\n        _r._total_sleep += sched_clock::now() - before_wait_cqes;\n        if (__builtin_expect(r < 0, false)) {\n            switch (-r) {\n            case EINTR:\n                return;\n            default:\n                abort();\n            }\n        }\n        did_work |= do_process_kernel_completions();\n        _preempt_io_context.service_preempting_io();\n    }\n    virtual future<> readable(pollable_fd_state& fd) override {\n        return poll(fd, POLLIN);\n    }\n    virtual future<> writeable(pollable_fd_state& fd) override {\n        return poll(fd, POLLOUT);\n    }\n    virtual future<> readable_or_writeable(pollable_fd_state& fd) override {\n        return poll(fd, POLLIN | POLLOUT);\n    }\n    virtual future<> poll_rdhup(pollable_fd_state& fd) override {\n        return poll(fd, POLLRDHUP);\n    }\n    virtual void forget(pollable_fd_state& fd) noexcept override {\n        auto* pfd = static_cast<uring_pollable_fd_state*>(&fd);\n        delete pfd;\n    }\n    virtual future<std::tuple<pollable_fd, socket_address>> accept(pollable_fd_state& listenfd) override {\n        if (listenfd.take_speculation(POLLIN)) {\n            try {\n                listenfd.maybe_no_more_recv();\n                socket_address sa;\n                auto maybe_fd = listenfd.fd.try_accept(sa, SOCK_CLOEXEC);\n                if (maybe_fd) {\n                    listenfd.speculate_epoll(EPOLLIN);\n                    pollable_fd pfd(std::move(*maybe_fd), pollable_fd::speculation(EPOLLOUT));\n                    return make_ready_future<std::tuple<pollable_fd, socket_address>>(std::move(pfd), std::move(sa));\n                }\n            } catch (...) {\n                return current_exception_as_future<std::tuple<pollable_fd, socket_address>>();\n            }\n        }\n        class accept_completion final : public io_completion {\n            pollable_fd_state& _listenfd;\n            socket_address _sa;\n            promise<std::tuple<pollable_fd, socket_address>> _result;\n        public:\n            accept_completion(pollable_fd_state& listenfd)\n                : _listenfd(listenfd) {}\n            void complete(size_t fd) noexcept final {\n                _listenfd.speculate_epoll(EPOLLIN);\n                pollable_fd pfd(file_desc::from_fd(fd), pollable_fd::speculation(EPOLLOUT));\n                _result.set_value(std::move(pfd), std::move(_sa));\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                try {\n                    std::rethrow_exception(eptr);\n                } catch (const std::system_error& e) {\n                    if (e.code() == std::errc::invalid_argument) {\n                        try {\n                            // The chances are that we shutting down the connection.\n                            _listenfd.maybe_no_more_recv();\n                        } catch (...) {\n                            eptr = std::current_exception();\n                        }\n                    }\n                } catch (...) {}\n                _result.set_exception(eptr);\n                delete this;\n            }\n            future<std::tuple<pollable_fd, socket_address>> get_future() {\n                return _result.get_future();\n            }\n            ::sockaddr* posix_sockaddr() {\n                return &_sa.as_posix_sockaddr();\n            }\n            socklen_t* socklen_ptr() {\n                return &_sa.addr_length;\n            }\n        };\n        return readable_or_writeable(listenfd).then([this, &listenfd] {\n            auto desc = std::make_unique<accept_completion>(listenfd);\n            auto req = internal::io_request::make_accept(listenfd.fd.get(), desc->posix_sockaddr(), desc->socklen_ptr(), SOCK_NONBLOCK | SOCK_CLOEXEC);\n            return submit_request(std::move(desc), std::move(req));\n        });\n    }\n    virtual future<> connect(pollable_fd_state& fd, socket_address& sa) override {\n        class connect_completion final : public io_completion {\n            pollable_fd_state& _fd;\n            socket_address _sa;\n            promise<> _result;\n        public:\n            connect_completion(pollable_fd_state& fd, const socket_address& sa)\n                : _fd(fd), _sa(sa) {}\n            void complete(size_t fd) noexcept final {\n                _fd.speculate_epoll(POLLOUT);\n                _result.set_value();\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                _result.set_exception(eptr);\n                delete this;\n            }\n            future<> get_future() {\n                return _result.get_future();\n            }\n            ::sockaddr* posix_sockaddr() {\n                return &_sa.as_posix_sockaddr();\n            }\n            socklen_t socklen() const {\n                return _sa.addr_length;\n            }\n        };\n        auto desc = std::make_unique<connect_completion>(fd, sa);\n        auto req = internal::io_request::make_connect(fd.fd.get(), desc->posix_sockaddr(), desc->socklen());\n        return submit_request(std::move(desc), std::move(req));\n    }\n    virtual future<size_t> read(pollable_fd_state& fd, void* buffer, size_t len) override {\n        return _r.do_read(fd, buffer, len);\n    }\n    virtual future<size_t> recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) override {\n        if (fd.take_speculation(POLLIN)) {\n            ::msghdr mh = {};\n            mh.msg_iov = const_cast<iovec*>(iov.data());\n            mh.msg_iovlen = iov.size();\n            try {\n                auto r = fd.fd.recvmsg(&mh, MSG_DONTWAIT);\n                if (r) {\n                    if (size_t(*r) == internal::iovec_len(iov)) {\n                        fd.speculate_epoll(EPOLLIN);\n                    }\n                    return make_ready_future<size_t>(*r);\n                }\n            } catch (...) {\n                return current_exception_as_future<size_t>();\n            }\n        }\n        class read_completion final : public io_completion {\n            pollable_fd_state& _fd;\n            std::vector<iovec> _iov;\n            ::msghdr _mh = {};\n            promise<size_t> _result;\n        public:\n            read_completion(pollable_fd_state& fd, const std::vector<iovec>& iov)\n                : _fd(fd), _iov(iov) {\n                _mh.msg_iov = const_cast<iovec*>(_iov.data());\n                _mh.msg_iovlen = _iov.size();\n            }\n            void complete(size_t bytes) noexcept final {\n                if (bytes == internal::iovec_len(_iov)) {\n                    _fd.speculate_epoll(EPOLLIN);\n                }\n                _result.set_value(bytes);\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                _result.set_exception(eptr);\n                delete this;\n            }\n            ::msghdr* msghdr() {\n                return &_mh;\n            }\n            future<size_t> get_future() {\n                return _result.get_future();\n            }\n        };\n        auto desc = std::make_unique<read_completion>(fd, iov);\n        auto req = internal::io_request::make_recvmsg(fd.fd.get(), desc->msghdr(), 0);\n        return submit_request(std::move(desc), std::move(req));\n    }\n    virtual future<temporary_buffer<char>> read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override {\n        if (fd.take_speculation(POLLIN)) {\n            auto buffer = ba->allocate_buffer();\n            try {\n                auto r = fd.fd.read(buffer.get_write(), buffer.size());\n                if (r) {\n                    if (size_t(*r) == buffer.size()) {\n                        fd.speculate_epoll(EPOLLIN);\n                    }\n                    buffer.trim(*r);\n                    return make_ready_future<temporary_buffer<char>>(std::move(buffer));\n                }\n            } catch (...) {\n                return current_exception_as_future<temporary_buffer<char>>();\n            }\n        }\n        return readable(fd).then([this, &fd, ba] {\n            class read_completion final : public io_completion {\n                pollable_fd_state& _fd;\n                temporary_buffer<char> _buffer;\n                promise<temporary_buffer<char>> _result;\n            public:\n                read_completion(pollable_fd_state& fd, temporary_buffer<char> buffer)\n                    : _fd(fd), _buffer(std::move(buffer)) {}\n                void complete(size_t bytes) noexcept final {\n                    if (bytes == _buffer.size()) {\n                        _fd.speculate_epoll(EPOLLIN);\n                    }\n                    _buffer.trim(bytes);\n                    _result.set_value(std::move(_buffer));\n                    delete this;\n                }\n                void set_exception(std::exception_ptr eptr) noexcept final {\n                    _result.set_exception(eptr);\n                    delete this;\n                }\n                future<temporary_buffer<char>> get_future() {\n                    return _result.get_future();\n                }\n                char* get_write() {\n                    return _buffer.get_write();\n                }\n                size_t get_size() {\n                    return _buffer.size();\n                }\n            };\n            auto desc = std::make_unique<read_completion>(fd, ba->allocate_buffer());\n            auto req = internal::io_request::make_read(fd.fd.get(), -1, desc->get_write(), desc->get_size(), false);\n            return submit_request(std::move(desc), std::move(req));\n        });\n    }\n    virtual future<size_t> sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) final {\n        if (fd.take_speculation(EPOLLOUT)) {\n            ::msghdr mh = {};\n            mh.msg_iov = iovs.data();\n            mh.msg_iovlen = std::min<size_t>(iovs.size(), IOV_MAX);\n            try {\n                auto r = fd.fd.sendmsg(&mh, MSG_NOSIGNAL | MSG_DONTWAIT);\n                if (r) {\n                    if (size_t(*r) == len) {\n                        fd.speculate_epoll(EPOLLOUT);\n                    }\n                    return make_ready_future<size_t>(*r);\n                }\n            } catch (...) {\n                return current_exception_as_future<size_t>();\n            }\n        }\n        class write_completion final : public io_completion {\n            pollable_fd_state& _fd;\n            ::msghdr _mh = {};\n            const size_t _to_write;\n            promise<size_t> _result;\n        public:\n            write_completion(pollable_fd_state& fd, std::span<iovec> iovs, size_t len)\n                : _fd(fd), _to_write(len) {\n                _mh.msg_iov = iovs.data();\n                _mh.msg_iovlen = std::min<size_t>(iovs.size(), IOV_MAX);\n            }\n            void complete(size_t bytes) noexcept final {\n                if (bytes == _to_write) {\n                    _fd.speculate_epoll(EPOLLOUT);\n                }\n                _result.set_value(bytes);\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                _result.set_exception(eptr);\n                delete this;\n            }\n            ::msghdr* msghdr() {\n                return &_mh;\n            }\n            future<size_t> get_future() {\n                return _result.get_future();\n            }\n        };\n        auto desc = std::make_unique<write_completion>(fd, iovs, len);\n        auto req = internal::io_request::make_sendmsg(fd.fd.get(), desc->msghdr(), MSG_NOSIGNAL);\n        return submit_request(std::move(desc), std::move(req));\n    }\n\n#if SEASTAR_API_LEVEL < 9\n    virtual future<size_t> send(pollable_fd_state& fd, const void* buffer, size_t len) override {\n        if (fd.take_speculation(EPOLLOUT)) {\n            try {\n                auto r = fd.fd.send(buffer, len, MSG_NOSIGNAL | MSG_DONTWAIT);\n                if (r) {\n                    if (size_t(*r) == len) {\n                        fd.speculate_epoll(EPOLLOUT);\n                    }\n                    return make_ready_future<size_t>(*r);\n                }\n            } catch (...) {\n                return current_exception_as_future<size_t>();\n            }\n        }\n        class write_completion final : public io_completion {\n            pollable_fd_state& _fd;\n            const size_t _to_write;\n            promise<size_t> _result;\n        public:\n            write_completion(pollable_fd_state& fd, size_t to_write)\n                : _fd(fd), _to_write(to_write) {}\n            void complete(size_t bytes) noexcept final {\n                if (bytes == _to_write) {\n                    _fd.speculate_epoll(EPOLLOUT);\n                }\n                _result.set_value(bytes);\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                _result.set_exception(eptr);\n                delete this;\n            }\n            future<size_t> get_future() {\n                return _result.get_future();\n            }\n        };\n        auto desc = std::make_unique<write_completion>(fd, len);\n        auto req = internal::io_request::make_send(fd.fd.get(), buffer, len, MSG_NOSIGNAL);\n        return submit_request(std::move(desc), std::move(req));\n    }\n#endif\n\n    virtual future<temporary_buffer<char>> recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override {\n        if (fd.take_speculation(POLLIN)) {\n            auto buffer = ba->allocate_buffer();\n            try {\n                auto r = fd.fd.recv(buffer.get_write(), buffer.size(), MSG_DONTWAIT);\n                if (r) {\n                    if (size_t(*r) == buffer.size()) {\n                        fd.speculate_epoll(EPOLLIN);\n                    }\n                    buffer.trim(*r);\n                    return make_ready_future<temporary_buffer<char>>(std::move(buffer));\n                }\n            } catch (...) {\n                return current_exception_as_future<temporary_buffer<char>>();\n            }\n        }\n        class recv_completion final : public io_completion {\n            pollable_fd_state& _fd;\n            temporary_buffer<char> _buffer;\n            promise<temporary_buffer<char>> _result;\n        public:\n            recv_completion(pollable_fd_state& fd, temporary_buffer<char> buffer)\n                : _fd(fd), _buffer(std::move(buffer)) {}\n            void complete(size_t bytes) noexcept final {\n                if (bytes == _buffer.size()) {\n                    _fd.speculate_epoll(EPOLLIN);\n                }\n                _buffer.trim(bytes);\n                _result.set_value(std::move(_buffer));\n                delete this;\n            }\n            void set_exception(std::exception_ptr eptr) noexcept final {\n                _result.set_exception(eptr);\n                delete this;\n            }\n            future<temporary_buffer<char>> get_future() {\n                return _result.get_future();\n            }\n            char* get_write() {\n                return _buffer.get_write();\n            }\n            size_t get_size() {\n                return _buffer.size();\n            }\n        };\n        auto desc = std::make_unique<recv_completion>(fd, ba->allocate_buffer());\n        auto req = internal::io_request::make_recv(fd.fd.get(), desc->get_write(), desc->get_size(), 0);\n        return submit_request(std::move(desc), std::move(req));\n    }\n\n    virtual void signal_received(int signo, siginfo_t* siginfo, void* ignore) override {\n        _r._signals.action(signo, siginfo, ignore);\n    }\n    virtual void start_tick() override {\n        _preempt_io_context.start_tick();\n    }\n    virtual void stop_tick() override {\n        _preempt_io_context.stop_tick();\n    }\n    virtual void arm_highres_timer(const ::itimerspec& its) override {\n        _hrtimer_timerfd.timerfd_settime(TFD_TIMER_ABSTIME, its);\n    }\n    virtual void reset_preemption_monitor() override {\n        _preempt_io_context.reset_preemption_monitor();\n    }\n    virtual void request_preemption() override {\n        _preempt_io_context.request_preemption();\n    }\n    virtual void start_handling_signal() override {\n        // We don't have anything special wrt. signals\n    }\n    virtual pollable_fd_state_ptr make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) override {\n        return pollable_fd_state_ptr(new uring_pollable_fd_state(std::move(fd), std::move(speculate)));\n    }\n};\n\n#endif\n\nstatic bool detect_aio_poll() {\n    auto fd = file_desc::eventfd(0, 0);\n    aio_context_t ioc{};\n    setup_aio_context(1, &ioc);\n    auto cleanup = defer([&] () noexcept { io_destroy(ioc); });\n    linux_abi::iocb iocb = internal::make_poll_iocb(fd.get(), POLLIN|POLLOUT);\n    linux_abi::iocb* a[1] = { &iocb };\n    auto r = io_submit(ioc, 1, a);\n    if (r != 1) {\n        return false;\n    }\n    uint64_t one = 1;\n    fd.write(&one, 8);\n    io_event ev[1];\n    // We set force_syscall = true (the last parameter) to ensure\n    // the system call exists and is usable. If IOCB_CMD_POLL exists then\n    // io_pgetevents() will also exist, but some versions of docker\n    // have a syscall whitelist that does not include io_pgetevents(),\n    // which causes it to fail with -EPERM. See\n    // https://github.com/moby/moby/issues/38894.\n    r = io_pgetevents(ioc, 1, 1, ev, nullptr, nullptr, true);\n    return r == 1;\n}\n\nbool reactor_backend_selector::has_enough_aio_nr() {\n    auto aio_max_nr = read_first_line_as<unsigned>(\"/proc/sys/fs/aio-max-nr\");\n    auto aio_nr = read_first_line_as<unsigned>(\"/proc/sys/fs/aio-nr\");\n    /* reactor_backend_selector::available() will be execute in early stage,\n     * it's before io_setup() issued, and not per-cpu basis.\n     * So this method calculates:\n     *  Available AIO on the system - (request AIO per-cpu * ncpus)\n     */\n    if (aio_max_nr - aio_nr < reactor::max_aio * smp::count) {\n        return false;\n    }\n    return true;\n}\n\nstd::unique_ptr<reactor_backend> reactor_backend_selector::create(reactor& r) {\n    if (_name == \"io_uring\") {\n#ifdef SEASTAR_HAVE_URING\n        return std::make_unique<reactor_backend_uring>(r);\n#else\n        throw std::runtime_error(\"io_uring backend not compiled in\");\n#endif\n    }\n    if (_name == \"linux-aio\") {\n        return std::make_unique<reactor_backend_aio>(r);\n    } else if (_name == \"epoll\") {\n        return std::make_unique<reactor_backend_epoll>(r);\n    }\n    throw std::logic_error(\"bad reactor backend\");\n}\n\nreactor_backend_selector reactor_backend_selector::default_backend() {\n    return available()[0];\n}\n\nstd::vector<reactor_backend_selector> reactor_backend_selector::available() {\n    std::vector<reactor_backend_selector> ret;\n#ifdef SEASTAR_HAVE_URING\n    if (detect_io_uring()) {\n        ret.push_back(reactor_backend_selector(\"io_uring\"));\n    }\n#endif\n    if (has_enough_aio_nr() && detect_aio_poll()) {\n        ret.push_back(reactor_backend_selector(\"linux-aio\"));\n    }\n    ret.push_back(reactor_backend_selector(\"epoll\"));\n    return ret;\n}\n\n}\n"
  },
  {
    "path": "src/core/reactor_backend.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/future.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/internal/io_desc.hh>\n#include <seastar/core/internal/pollable_fd.hh>\n#include <seastar/core/internal/poll.hh>\n#include <seastar/core/internal/linux-aio.hh>\n#include <seastar/core/cacheline.hh>\n#include <seastar/util/bool_class.hh>\n\n#include <fmt/ostream.h>\n#include <sys/time.h>\n#include <thread>\n#include <stack>\n#include <boost/any.hpp>\n#include <boost/program_options.hpp>\n#include <boost/container/static_vector.hpp>\n\n\nnamespace seastar {\n\nclass reactor;\n\n// FIXME: merge it with storage context below. At this point the\n// main thing to do is unify the iocb list\nstruct aio_general_context {\n    explicit aio_general_context(size_t nr);\n    ~aio_general_context();\n    internal::linux_abi::aio_context_t io_context{};\n    std::unique_ptr<internal::linux_abi::iocb*[]> iocbs;\n    internal::linux_abi::iocb** last;\n    internal::linux_abi::iocb** const end;\n    void queue(internal::linux_abi::iocb* iocb);\n    // submit all queued iocbs and return their count.\n    size_t flush();\n};\n\nclass aio_storage_context {\n    static constexpr unsigned max_aio = 1024;\n\n    class iocb_pool {\n        alignas(cache_line_size) std::array<internal::linux_abi::iocb, max_aio> _all_iocbs;\n        std::stack<internal::linux_abi::iocb*, boost::container::static_vector<internal::linux_abi::iocb*, max_aio>> _free_iocbs;\n    public:\n        iocb_pool();\n        internal::linux_abi::iocb& get_one();\n        void put_one(internal::linux_abi::iocb* io);\n        unsigned outstanding() const;\n        bool has_capacity() const;\n    };\n\n    reactor& _r;\n    internal::linux_abi::aio_context_t _io_context;\n    boost::container::static_vector<internal::linux_abi::iocb*, max_aio> _submission_queue;\n    iocb_pool _iocb_pool;\n    size_t handle_aio_error(internal::linux_abi::iocb* iocb, int ec);\n    using pending_aio_retry_t = boost::container::static_vector<internal::linux_abi::iocb*, max_aio>;\n    pending_aio_retry_t _pending_aio_retry; // Pending retries iocbs\n    pending_aio_retry_t _aio_retries;       // Currently retried iocbs\n    future<> _pending_aio_retry_fut = make_ready_future<>();\n    internal::linux_abi::io_event _ev_buffer[max_aio];\n\n    bool need_to_retry() const noexcept {\n        return !_pending_aio_retry.empty() || !_aio_retries.empty();\n    }\n\n    bool retry_in_progress() const noexcept {\n        return !_pending_aio_retry_fut.available();\n    }\n\npublic:\n    explicit aio_storage_context(reactor& r);\n    ~aio_storage_context();\n\n    bool reap_completions(bool allow_retry = true);\n    void schedule_retry();\n    bool submit_work();\n    bool can_sleep() const;\n    future<> stop() noexcept;\n};\n\nclass completion_with_iocb {\n    bool _in_context = false;\n    internal::linux_abi::iocb _iocb;\nprotected:\n    completion_with_iocb(int fd, int events, void* user_data);\n    void completed() {\n        _in_context = false;\n    }\npublic:\n    void maybe_queue(aio_general_context& context);\n};\n\nclass fd_kernel_completion : public kernel_completion {\nprotected:\n    file_desc& _fd;\n    fd_kernel_completion(file_desc& fd) : _fd(fd) {}\npublic:\n    file_desc& fd() {\n        return _fd;\n    }\n};\n\nstruct hrtimer_aio_completion : public fd_kernel_completion,\n                                public completion_with_iocb {\nprivate:\n    reactor& _r;\npublic:\n    hrtimer_aio_completion(reactor& r, file_desc& fd);\n    virtual void complete_with(ssize_t value) override;\n};\n\nstruct task_quota_aio_completion : public fd_kernel_completion,\n                                   public completion_with_iocb {\n    task_quota_aio_completion(file_desc& fd);\n    virtual void complete_with(ssize_t value) override;\n};\n\nstruct smp_wakeup_aio_completion : public fd_kernel_completion,\n                                   public completion_with_iocb {\n    smp_wakeup_aio_completion(file_desc& fd);\n    virtual void complete_with(ssize_t value) override;\n};\n\n// Common aio-based Implementation of the task quota and hrtimer.\nclass preempt_io_context {\n    reactor& _r;\n    aio_general_context _context{2};\n\n    task_quota_aio_completion _task_quota_aio_completion;\n    hrtimer_aio_completion _hrtimer_aio_completion;\npublic:\n    preempt_io_context(reactor& r, file_desc& task_quota, file_desc& hrtimer);\n    bool service_preempting_io();\n\n    size_t flush() {\n        return _context.flush();\n    }\n\n    void reset_preemption_monitor();\n    void request_preemption();\n    void start_tick();\n    void stop_tick();\n};\n\n// The \"reactor_backend\" interface provides a method of waiting for various\n// basic events on one thread. We have one implementation based on epoll and\n// file-descriptors (reactor_backend_epoll), one implementation based on\n// linux aio, and one implementation based on io_uring.\nclass reactor_backend {\nprotected:\n    using uses_blocking_io = bool_class<struct uses_blocking_io_tag>;\n    using supports_aio_fdatasync = bool_class<struct supports_aio_fdatasync_tag>;\n\nprivate:\n    const uses_blocking_io _blocking_io;\n    const supports_aio_fdatasync _aio_fdatasync;\n\npublic:\n    virtual ~reactor_backend() {};\n    virtual std::string_view get_backend_name() const = 0;\n    // The methods below are used to communicate with the kernel.\n    // reap_kernel_completions() will complete any previous async\n    // work that is ready to consume.\n    // kernel_submit_work() submit new events that were produced.\n    // Both of those methods are asynchronous and will never block.\n    //\n    // wait_and_process_events on the other hand may block, and is called when\n    // we are about to go to sleep.\n    virtual bool reap_kernel_completions() = 0;\n    virtual bool kernel_submit_work() = 0;\n    virtual bool kernel_events_can_sleep() const = 0;\n    virtual void wait_and_process_events(const sigset_t* active_sigmask = nullptr) = 0;\n\n    // Methods that allow polling on file descriptors. This will only work on\n    // reactor_backend_epoll. Other reactor_backend will probably abort if\n    // they are called (which is fine if no file descriptors are waited on):\n    virtual future<> readable(pollable_fd_state& fd) = 0;\n    virtual future<> writeable(pollable_fd_state& fd) = 0;\n    virtual future<> readable_or_writeable(pollable_fd_state& fd) = 0;\n    virtual future<> poll_rdhup(pollable_fd_state& fd) = 0;\n    virtual void forget(pollable_fd_state& fd) noexcept = 0;\n\n    virtual future<std::tuple<pollable_fd, socket_address>>\n    accept(pollable_fd_state& listenfd) = 0;\n    virtual future<> connect(pollable_fd_state& fd, socket_address& sa) = 0;\n    virtual future<size_t> read(pollable_fd_state& fd, void* buffer, size_t len) = 0;\n    virtual future<size_t> recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) = 0;\n    virtual future<temporary_buffer<char>> read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) = 0;\n    virtual future<size_t> sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) = 0;\n#if SEASTAR_API_LEVEL < 9\n    virtual future<size_t> send(pollable_fd_state& fd, const void* buffer, size_t len) = 0;\n#endif\n    virtual future<temporary_buffer<char>> recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) = 0;\n\n    bool do_blocking_io() const noexcept {\n        return bool(_blocking_io);\n    }\n    bool have_aio_fdatasync() const noexcept {\n        return bool(_aio_fdatasync);\n    }\n    virtual void signal_received(int signo, siginfo_t* siginfo, void* ignore) = 0;\n    virtual void start_tick() = 0;\n    virtual void stop_tick() = 0;\n    virtual void arm_highres_timer(const ::itimerspec& ts) = 0;\n    virtual void reset_preemption_monitor() = 0;\n    virtual void request_preemption() = 0;\n    virtual void start_handling_signal() = 0;\n\n    virtual pollable_fd_state_ptr make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) = 0;\n\nprotected:\n    reactor_backend(uses_blocking_io blocking_io, supports_aio_fdatasync aio_fdatasync)\n        : _blocking_io(blocking_io)\n        , _aio_fdatasync(aio_fdatasync)\n    {}\n};\n\n// reactor backend using file-descriptor & epoll, suitable for running on\n// Linux. Can wait on multiple file descriptors, and converts other events\n// (such as timers, signals, inter-thread notifications) into file descriptors\n// using mechanisms like timerfd, signalfd and eventfd respectively.\nclass reactor_backend_epoll : public reactor_backend {\n    reactor& _r;\n    std::atomic<bool> _highres_timer_pending = {};\n    std::thread _task_quota_timer_thread;\n    ::itimerspec _steady_clock_timer_deadline = {};\n    // These two timers are used for high resolution timer<>s, one for\n    // the reactor thread (when sleeping) and one for the timer thread\n    // (when awake). We can't use one timer because of races between the\n    // timer thread and reactor thread.\n    //\n    // Only one of the two is active at any time.\n    file_desc _steady_clock_timer_reactor_thread;\n    file_desc _steady_clock_timer_timer_thread;\n    std::atomic<bool> _dying{false};\nprivate:\n    file_desc _epollfd;\n    void task_quota_timer_thread_fn();\n    future<> get_epoll_future(pollable_fd_state& fd, int event);\n    void complete_epoll_event(pollable_fd_state& fd, int events, int event);\n    aio_storage_context _storage_context;\n    void switch_steady_clock_timers(file_desc& from, file_desc& to);\n    void maybe_switch_steady_clock_timers(int timeout, file_desc& from, file_desc& to);\n    bool wait_and_process(int timeout, const sigset_t* active_sigmask);\n    bool complete_hrtimer();\n    bool _need_epoll_events = false;\npublic:\n    explicit reactor_backend_epoll(reactor& r);\n    virtual ~reactor_backend_epoll() override;\n\n    virtual std::string_view get_backend_name() const override;\n    virtual bool reap_kernel_completions() override;\n    virtual bool kernel_submit_work() override;\n    virtual bool kernel_events_can_sleep() const override;\n    virtual void wait_and_process_events(const sigset_t* active_sigmask) override;\n    virtual future<> readable(pollable_fd_state& fd) override;\n    virtual future<> writeable(pollable_fd_state& fd) override;\n    virtual future<> readable_or_writeable(pollable_fd_state& fd) override;\n    virtual future<> poll_rdhup(pollable_fd_state& fd) override;\n    virtual void forget(pollable_fd_state& fd) noexcept override;\n\n    virtual future<std::tuple<pollable_fd, socket_address>>\n    accept(pollable_fd_state& listenfd) override;\n    virtual future<> connect(pollable_fd_state& fd, socket_address& sa) override;\n    virtual future<size_t> read(pollable_fd_state& fd, void* buffer, size_t len) override;\n    virtual future<size_t> recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) override;\n    virtual future<temporary_buffer<char>> read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override;\n    virtual future<size_t> sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) override;\n#if SEASTAR_API_LEVEL < 9\n    virtual future<size_t> send(pollable_fd_state& fd, const void* buffer, size_t len) override;\n#endif\n    virtual future<temporary_buffer<char>> recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override;\n\n    virtual void signal_received(int signo, siginfo_t* siginfo, void* ignore) override;\n    virtual void start_tick() override;\n    virtual void stop_tick() override;\n    virtual void arm_highres_timer(const ::itimerspec& ts) override;\n    virtual void reset_preemption_monitor() override;\n    virtual void request_preemption() override;\n    virtual void start_handling_signal() override;\n\n    virtual pollable_fd_state_ptr\n    make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) override;\n};\n\nclass reactor_backend_aio : public reactor_backend {\n    reactor& _r;\n    file_desc _hrtimer_timerfd;\n    aio_storage_context _storage_context;\n    // We use two aio contexts, one for preempting events (the timer tick and\n    // signals), the other for non-preempting events (fd poll).\n    preempt_io_context _preempting_io; // Used for the timer tick and the high resolution timer\n    aio_general_context _polling_io; // FIXME: unify with disk aio_context\n    hrtimer_aio_completion _hrtimer_poll_completion;\n    smp_wakeup_aio_completion _smp_wakeup_aio_completion;\n    static file_desc make_timerfd();\n    bool await_events(int timeout, const sigset_t* active_sigmask);\n    future<> poll(pollable_fd_state& fd, int events);\npublic:\n    explicit reactor_backend_aio(reactor& r);\n\n    virtual std::string_view get_backend_name() const override;\n    virtual bool reap_kernel_completions() override;\n    virtual bool kernel_submit_work() override;\n    virtual bool kernel_events_can_sleep() const override;\n    virtual void wait_and_process_events(const sigset_t* active_sigmask) override;\n    virtual future<> readable(pollable_fd_state& fd) override;\n    virtual future<> writeable(pollable_fd_state& fd) override;\n    virtual future<> readable_or_writeable(pollable_fd_state& fd) override;\n    virtual future<> poll_rdhup(pollable_fd_state& fd) override;\n    virtual void forget(pollable_fd_state& fd) noexcept override;\n\n    virtual future<std::tuple<pollable_fd, socket_address>>\n    accept(pollable_fd_state& listenfd) override;\n    virtual future<> connect(pollable_fd_state& fd, socket_address& sa) override;\n    virtual future<size_t> read(pollable_fd_state& fd, void* buffer, size_t len) override;\n    virtual future<size_t> recvmsg(pollable_fd_state& fd, const std::vector<iovec>& iov) override;\n    virtual future<temporary_buffer<char>> read_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override;\n    virtual future<size_t> sendmsg(pollable_fd_state& fd, std::span<iovec> iovs, size_t len) override;\n#if SEASTAR_API_LEVEL < 9\n    virtual future<size_t> send(pollable_fd_state& fd, const void* buffer, size_t len) override;\n#endif\n    virtual future<temporary_buffer<char>> recv_some(pollable_fd_state& fd, internal::buffer_allocator* ba) override;\n\n    virtual void signal_received(int signo, siginfo_t* siginfo, void* ignore) override;\n    virtual void start_tick() override;\n    virtual void stop_tick() override;\n    virtual void arm_highres_timer(const ::itimerspec& its) override;\n    virtual void reset_preemption_monitor() override;\n    virtual void request_preemption() override;\n    virtual void start_handling_signal() override;\n\n    virtual pollable_fd_state_ptr\n    make_pollable_fd_state(file_desc fd, pollable_fd::speculation speculate) override;\n};\n\nclass reactor_backend_uring;\n\nclass reactor_backend_selector {\n    std::string _name;\nprivate:\n    static bool has_enough_aio_nr();\n    explicit reactor_backend_selector(std::string name) : _name(std::move(name)) {}\npublic:\n    const std::string& name() const { return _name; }\n    std::unique_ptr<reactor_backend> create(reactor& r);\n    static reactor_backend_selector default_backend();\n    static std::vector<reactor_backend_selector> available();\n    friend std::ostream& operator<<(std::ostream& os, const reactor_backend_selector& rbs) {\n        return os << rbs._name;\n    }\n};\n\n}\n\n\ntemplate <> struct fmt::formatter<seastar::reactor_backend_selector> : fmt::ostream_formatter {};\n\n"
  },
  {
    "path": "src/core/resource.cc",
    "content": "\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <boost/program_options.hpp>\n#include <boost/algorithm/string.hpp>\n#include <boost/range/adaptor/map.hpp>\n#include <boost/range/algorithm/copy.hpp>\n#include <ranges>\n#include <regex>\n#include <stdlib.h>\n#include <unistd.h>\n#include <limits>\n#include <filesystem>\n#include <unordered_map>\n#include <fmt/core.h>\n#include <seastar/util/assert.hh>\n\n#include <seastar/core/resource.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/read_first_line.hh>\n#include <seastar/util/log.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/print.hh>\n#include \"cgroup.hh\"\n\n#if SEASTAR_HAVE_HWLOC\n#include <hwloc.h>\n#include <hwloc/glibc-sched.h>\n#endif\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\nnamespace resource {\n\n// This function was made optional because of validate. It needs to\n// throw an error when a non parseable input is given.\nstd::optional<cpuset> parse_cpuset(std::string value) {\n    static std::regex r(\"(\\\\d+-)?(\\\\d+)(,(\\\\d+-)?(\\\\d+))*\");\n\n    std::smatch match;\n    if (std::regex_match(value, match, r)) {\n        std::vector<std::string> ranges;\n        boost::split(ranges, value, boost::is_any_of(\",\"));\n        resource::cpuset ret;\n        for (auto&& range: ranges) {\n            std::string beg = range;\n            std::string end = range;\n            auto dash = range.find('-');\n            if (dash != range.npos) {\n                beg = range.substr(0, dash);\n                end = range.substr(dash + 1);\n            }\n            auto b = boost::lexical_cast<unsigned>(beg);\n            auto e = boost::lexical_cast<unsigned>(end);\n\n            if (b > e) {\n                return std::nullopt;\n            }\n\n            for (auto i = b; i <= e; ++i) {\n                ret.insert(i);\n            }\n        }\n        return ret;\n    }\n    return std::nullopt;\n}\n\n}\n\nnamespace cgroup {\n\nnamespace fs = std::filesystem;\n\noptional<cpuset> cpu_set() {\n    auto cpuset = read_setting_V1V2_as<std::string>(\n                              \"cpuset/cpuset.cpus\",\n                              \"cpuset.cpus.effective\");\n    if (cpuset) {\n        return seastar::resource::parse_cpuset(*cpuset);\n    }\n\n    seastar_logger.warn(\"Unable to parse cgroup's cpuset. Ignoring.\");\n    return std::nullopt;\n}\n\nsize_t memory_limit() {\n    return read_setting_V1V2_as<size_t>(\n                             \"memory/memory.limit_in_bytes\",\n                             \"memory.max\")\n        .value_or(std::numeric_limits<size_t>::max());\n}\n\ntemplate <typename T>\noptional<T> read_setting_as(std::string path) {\n    try {\n        auto line = read_first_line(path);\n        return boost::lexical_cast<T>(line);\n    } catch (...) {\n        seastar_logger.warn(\"Couldn't read cgroup file {}.\", path);\n    }\n\n    return std::nullopt;\n}\n\n/*\n * what cgroup do we belong to?\n *\n * For cgroups V2, /proc/self/cgroup should read \"0::<cgroup-dir-path>\"\n * Note: true only for V2-only systems, but there is no reason to support\n * a hybrid configuration.\n */\nstatic optional<fs::path> cgroup2_path_my_pid() {\n    seastar::sstring cline;\n    try {\n        cline = read_first_line(fs::path{\"/proc/self/cgroup\"});\n    } catch (...) {\n        // '/proc/self/cgroup' must be there. If not - there is an issue\n        // with the system configuration.\n        throw std::runtime_error(\"no cgroup data for our process\");\n    }\n\n    // for a V2-only system, we expect exactly one line:\n    // 0::<abs-path-to-cgroup>\n    if (cline.at(0) != '0') {\n        // This is either a v1 system, or system configured with a hybrid of v1 & v2.\n        // We do not support such combinations of v1 and v2 at this point.\n        seastar_logger.debug(\"Not a cgroups-v2-only system\");\n        return std::nullopt;\n    }\n\n    // the path is guaranteed to start with '0::/'\n    return fs::path{\"/sys/fs/cgroup/\" + cline.substr(4)};\n}\n\n/*\n * traverse the cgroups V2 hierarchy bottom-up, starting from our process'\n * specific cgroup up to /sys/fs/cgroup, looking for the named file.\n */\nstatic optional<fs::path> locate_lowest_cgroup2(fs::path lowest_subdir, std::string filename) {\n    // locate the lowest subgroup containing the named file (i.e.\n    // handles the requested control by itself)\n    do {\n        //  does the cgroup settings file exist?\n        auto set_path = lowest_subdir / filename;\n        if (fs::exists(set_path) ) {\n            return set_path;\n        }\n\n        lowest_subdir = lowest_subdir.parent_path();\n    } while (lowest_subdir.compare(\"/sys/fs\"));\n\n    return std::nullopt;\n}\n\n/*\n * Read a settings value from either the cgroups V2 or the corresponding\n * cgroups V1 files.\n * For V2, look for the lowest cgroup in our hierarchy that manages the\n * requested settings.\n */\ntemplate <typename T>\noptional<T> read_setting_V1V2_as(std::string cg1_path, std::string cg2_fname) {\n    // on v2-systems, cg2_path will be initialized with the leaf cgroup that\n    // controls this process\n    static optional<fs::path> cg2_path{cgroup2_path_my_pid()};\n\n    if (cg2_path) {\n        // this is a v2 system\n        seastar::sstring line;\n        try {\n            line = read_first_line(locate_lowest_cgroup2(*cg2_path, cg2_fname).value());\n        } catch (...) {\n            seastar_logger.warn(\"Could not read cgroups v2 file ({}).\", cg2_fname);\n            return std::nullopt;\n        }\n        if (line.compare(\"max\")) {\n            try {\n                return boost::lexical_cast<T>(line);\n            } catch (...) {\n                seastar_logger.warn(\"Malformed cgroups file ({}) contents.\", cg2_fname);\n            }\n        }\n        return std::nullopt;\n    }\n\n    // try cgroups v1:\n    try {\n        auto line = read_first_line(fs::path{\"/sys/fs/cgroup\"} / cg1_path);\n        return boost::lexical_cast<T>(line);\n    } catch (...) {\n        seastar_logger.warn(\"Could not parse cgroups v1 file ({}).\", cg1_path);\n    }\n\n    return std::nullopt;\n}\n\n}\n\nnamespace resource {\n\nstatic unsigned long get_machine_memory_from_sysconf() {\n    return ::sysconf(_SC_PAGESIZE) * size_t(::sysconf(_SC_PHYS_PAGES));\n}\n\nstatic\nsize_t\nkernel_memory_reservation() {\n    try {\n        return read_first_line_as<size_t>(\"/proc/sys/vm/min_free_kbytes\") * 1024;\n    } catch (...) {\n        return 0;\n    }\n}\n\nsize_t calculate_memory(const configuration& c, size_t available_memory, float panic_factor = 1) {\n    auto kernel_reservation = kernel_memory_reservation();\n    if (kernel_reservation >= 200'000'000) {\n        // The standard setting is sqrt(mem)*128. This is 128MB at 1TB RAM. With 64kB pages and transparent hugepages,\n        // the kernel increases this significantly, wasting memory.\n        seastar_logger.warn(\"Kernel memory reservation (/proc/sys/vm/min_free_kbytes) unexpectedly high ({}), check your configuration\", kernel_reservation);\n    }\n    available_memory -= kernel_reservation;\n    size_t default_reserve_memory = std::max<size_t>(1536 * 1024 * 1024, 0.07 * available_memory) * panic_factor;\n    auto reserve = c.reserve_memory.value_or(default_reserve_memory);\n    auto reserve_additional = c.reserve_additional_memory_per_shard * c.cpus;\n    reserve += reserve_additional;\n    size_t min_memory = 500'000'000;\n    if (available_memory >= reserve + min_memory) {\n        available_memory -= reserve;\n    } else {\n        // Allow starting up even in low memory configurations (e.g. 2GB boot2docker VM)\n        available_memory = min_memory;\n    }\n    if (!c.total_memory.has_value()) {\n        return available_memory;\n    }\n    if (*c.total_memory < reserve_additional) {\n        throw std::runtime_error(format(\"insufficient total memory: reserve {} total {}\", reserve_additional, *c.total_memory));\n    }\n    size_t needed_memory = *c.total_memory - reserve_additional;\n    if (needed_memory > available_memory) {\n        throw std::runtime_error(format(\"insufficient physical memory: needed {} available {}\", needed_memory, available_memory));\n    }\n    return needed_memory;\n}\n\nio_queue_topology::io_queue_topology() {\n}\n\nio_queue_topology::~io_queue_topology() {\n}\n\nio_queue_topology::io_queue_topology(io_queue_topology&& o)\n    : queues(std::move(o.queues))\n    , shard_to_group(std::move(o.shard_to_group))\n    , shards_in_group(std::move(o.shards_in_group))\n    , groups(std::move(o.groups))\n    , lock() // unused until now, so just initialize\n{ }\n\n}\n\n}\n\n#ifdef SEASTAR_HAVE_HWLOC\n\nnamespace seastar {\n\nnamespace resource {\n\nstatic hwloc_uint64_t get_memory_from_hwloc_obj(hwloc_obj_t obj) {\n#if HWLOC_API_VERSION >= 0x00020000\n    auto total_memory = obj->total_memory;\n#else\n    auto total_memory = obj->memory.total_memory;\n#endif\n    return total_memory;\n}\n\nstatic void set_memory_to_hwloc_obj(hwloc_obj_t machine, hwloc_uint64_t memory) {\n#if HWLOC_API_VERSION >= 0x00020000\n    machine->total_memory = memory;\n#else\n    machine->memory.total_memory = memory;\n#endif\n}\n\nstatic size_t alloc_from_node(cpu& this_cpu, hwloc_obj_t node, std::unordered_map<hwloc_obj_t, size_t>& used_mem, size_t alloc) {\n    auto local_memory = get_memory_from_hwloc_obj(node);\n    auto taken = std::min(local_memory - used_mem[node], alloc);\n    if (taken) {\n        used_mem[node] += taken;\n        auto node_id = hwloc_bitmap_first(node->nodeset);\n        SEASTAR_ASSERT(node_id != -1);\n        this_cpu.mem.push_back({taken, unsigned(node_id)});\n        seastar_logger.debug(\"CPU{} allocated {} bytes from NODE{}\", this_cpu.cpu_id, taken, node_id);\n    }\n    return taken;\n}\n\n// Find the numa node that contains a specific PU.\nstatic hwloc_obj_t get_numa_node_for_pu(hwloc_topology_t topology, hwloc_obj_t pu) {\n    // Can't use ancestry because hwloc 2.0 NUMA nodes are not ancestors of PUs\n    hwloc_obj_t tmp = NULL;\n    auto depth = hwloc_get_type_or_above_depth(topology, HWLOC_OBJ_NUMANODE);\n    while ((tmp = hwloc_get_next_obj_by_depth(topology, depth, tmp)) != NULL) {\n        if (hwloc_bitmap_intersects(tmp->cpuset, pu->cpuset)) {\n            return tmp;\n        }\n    }\n    return nullptr;\n}\n\nstatic hwloc_obj_t hwloc_get_ancestor(hwloc_obj_type_t type, hwloc_topology_t topology, unsigned cpu_id) {\n    auto cur = hwloc_get_pu_obj_by_os_index(topology, cpu_id);\n\n    while (cur != nullptr) {\n        if (cur->type == type) {\n            break;\n        }\n        cur = cur->parent;\n    }\n\n    return cur;\n}\n\nstatic std::unordered_map<hwloc_obj_t, std::vector<unsigned>> break_cpus_into_groups(hwloc_topology_t topology,\n        const std::vector<unsigned>& cpus, hwloc_obj_type_t type) {\n    std::unordered_map<hwloc_obj_t, std::vector<unsigned>> groups;\n\n    for (auto&& cpu_id : cpus) {\n        hwloc_obj_t anc = hwloc_get_ancestor(type, topology, cpu_id);\n        groups[anc].push_back(cpu_id);\n    }\n\n    return groups;\n}\n\nstruct distribute_objects {\n    std::vector<hwloc_cpuset_t> cpu_sets;\n    hwloc_obj_t root;\n\n    distribute_objects(hwloc_topology_t topology, size_t nobjs) : cpu_sets(nobjs), root(hwloc_get_root_obj(topology)) {\n#if HWLOC_API_VERSION >= 0x00010900\n        hwloc_distrib(topology, &root, 1, cpu_sets.data(), cpu_sets.size(), INT_MAX, 0);\n#else\n        hwloc_distribute(topology, root, cpu_sets.data(), cpu_sets.size(), INT_MAX);\n#endif\n    }\n\n    ~distribute_objects() {\n        for (auto&& cs : cpu_sets) {\n            hwloc_bitmap_free(cs);\n        }\n    }\n    std::vector<hwloc_cpuset_t>& operator()() {\n        return cpu_sets;\n    }\n};\n\nstatic io_queue_topology\nallocate_io_queues(hwloc_topology_t topology, const std::vector<cpu>& cpus, const std::unordered_map<unsigned, hwloc_obj_t>& cpu_to_node, unsigned num_io_groups, unsigned& last_node_idx) {\n    auto node_of_shard = [&cpus, &cpu_to_node] (unsigned shard) {\n        auto node = cpu_to_node.at(cpus[shard].cpu_id);\n        return hwloc_bitmap_first(node->nodeset);\n    };\n\n    // There are two things we are trying to achieve by populating a numa_nodes map.\n    //\n    // The first is to find out how many nodes we have in the system. We can't use\n    // hwloc for that, because at this point we are not longer talking about the physical system,\n    // but the actual booted seastar server instead. So if we have restricted the run to a subset\n    // of the available processors, counting topology nodes won't spur the same result.\n    //\n    // Secondly, we need to find out which processors live in each node. For a reason similar to the\n    // above, hwloc won't do us any good here. Later on, we will use this information to assign\n    // shards to coordinators that are node-local to themselves.\n    std::unordered_map<unsigned, std::set<unsigned>> numa_nodes;\n    for (auto shard: std::views::iota(0, int(cpus.size()))) {\n        auto node_id = node_of_shard(shard);\n\n        if (numa_nodes.count(node_id) == 0) {\n            numa_nodes.emplace(node_id, std::set<unsigned>());\n        }\n        numa_nodes.at(node_id).insert(shard);\n    }\n\n    io_queue_topology ret;\n    ret.shard_to_group.resize(cpus.size());\n    ret.shards_in_group.resize(cpus.size(), 0); // worst case\n\n    if (num_io_groups == 0) {\n        num_io_groups = numa_nodes.size();\n        SEASTAR_ASSERT(num_io_groups != 0);\n        seastar_logger.debug(\"Auto-configure {} IO groups\", num_io_groups);\n    } else if (num_io_groups > cpus.size()) {\n        // User may be playing with --smp option, but num_io_groups was independently\n        // determined by iotune, so adjust for any conflicts.\n        fmt::print(\"Warning: number of IO queues ({:d}) greater than logical cores ({:d}). Adjusting downwards.\\n\", num_io_groups, cpus.size());\n        num_io_groups = cpus.size();\n    }\n\n    auto find_shard = [&cpus] (unsigned cpu_id) {\n        auto idx = 0u;\n        for (auto& c: cpus) {\n            if (c.cpu_id == cpu_id) {\n                return idx;\n            }\n            idx++;\n        }\n        SEASTAR_ASSERT(0);\n    };\n\n    auto cpu_sets = distribute_objects(topology, num_io_groups);\n    ret.queues.resize(cpus.size());\n    unsigned nr_groups = 0;\n\n    // First step: distribute the IO queues given the information returned in cpu_sets.\n    // If there is one IO queue per processor, only this loop will be executed.\n    std::unordered_map<unsigned, std::vector<unsigned>> node_coordinators;\n    for (auto&& cs : cpu_sets()) {\n        auto io_coordinator = find_shard(hwloc_bitmap_first(cs));\n        unsigned group_idx = nr_groups++;\n        ret.shard_to_group[io_coordinator] = group_idx;\n        ret.shards_in_group[group_idx]++;\n\n        auto node_id = node_of_shard(io_coordinator);\n        if (node_coordinators.count(node_id) == 0) {\n            node_coordinators.emplace(node_id, std::vector<unsigned>());\n        }\n        node_coordinators.at(node_id).push_back(io_coordinator);\n        numa_nodes[node_id].erase(io_coordinator);\n    }\n\n    ret.groups.resize(nr_groups);\n\n    auto available_nodes = boost::copy_range<std::vector<unsigned>>(node_coordinators | boost::adaptors::map_keys);\n\n    // If there are more processors than coordinators, we will have to assign them to existing\n    // coordinators. We prefer do that within the same NUMA node, but if not possible we assign\n    // the shard to a random node.\n    for (auto& node: numa_nodes) {\n        auto cid_idx = 0;\n        for (auto& remaining_shard: node.second) {\n            auto my_node = node.first;\n            // No I/O queue in this node, round-robin shards from this node into existing ones.\n            if (!node_coordinators.count(node.first)) {\n                my_node = available_nodes[last_node_idx++ % available_nodes.size()];\n            }\n            auto idx = cid_idx++ % node_coordinators.at(my_node).size();\n            auto io_coordinator = node_coordinators.at(my_node)[idx];\n            unsigned group_idx = ret.shard_to_group[io_coordinator];\n            ret.shard_to_group[remaining_shard] = group_idx;\n            ret.shards_in_group[group_idx]++;\n        }\n    }\n\n    return ret;\n}\n\nnamespace hwloc::internal {\n\ntopology_holder::topology_holder(topology_holder&& o) noexcept\n    : _topology(std::exchange(o._topology, nullptr))\n{ }\n\ntopology_holder::~topology_holder() {\n    if (_topology) {\n        hwloc_topology_destroy(_topology);\n    }\n}\n\ntopology_holder& topology_holder::operator=(topology_holder&& o) noexcept {\n    if (this != &o) {\n        std::swap(_topology, o._topology);\n    }\n    return *this;\n}\n\nvoid topology_holder::init_and_load() {\n    hwloc_topology_init(&_topology);\n    // hwloc_topology_destroy is required after hwloc_topology_init\n    // on success, _topology will not be null anymore\n\n    hwloc_topology_load(_topology);\n}\n\nhwloc_topology_t topology_holder::get() {\n    if (!_topology) {\n        init_and_load();\n    }\n    return _topology;\n}\n\n} // namespace hwloc::internal\n\nstatic\nstd::unordered_map<unsigned, cpuset>\nnuma_node_id_to_cpuset(hwloc_topology_t topo) {\n    auto ret = std::unordered_map<unsigned, cpuset>();\n    for (auto numa_node = hwloc_get_next_obj_by_type(topo, HWLOC_OBJ_NUMANODE, NULL);\n            numa_node;\n            numa_node = hwloc_get_next_obj_by_type(topo, HWLOC_OBJ_NUMANODE, numa_node)) {\n        auto parent = numa_node->parent;\n        auto cpuset = parent->cpuset;\n        cpu_set_t os_cpuset;\n        hwloc_cpuset_to_glibc_sched_affinity(topo, cpuset, &os_cpuset, sizeof(os_cpuset));\n        for (unsigned idx = 0; idx < CPU_SETSIZE; ++idx) {\n            if (CPU_ISSET(idx, &os_cpuset)) {\n                ret[numa_node->os_index].insert(idx);\n            }\n        }\n    }\n    return ret;\n}\n\nresources allocate(configuration& c) {\n    auto topology = c.topology.get();\n    auto bm = hwloc_bitmap_alloc();\n    auto free_bm = defer([&] () noexcept { hwloc_bitmap_free(bm); });\n    for (auto idx : c.cpu_set) {\n        hwloc_bitmap_set(bm, idx);\n    }\n    auto r = hwloc_topology_restrict(topology, bm,\n#if HWLOC_API_VERSION >= 0x00020000\n            0\n#else\n            HWLOC_RESTRICT_FLAG_ADAPT_DISTANCES\n#endif\n            | HWLOC_RESTRICT_FLAG_ADAPT_MISC\n            | HWLOC_RESTRICT_FLAG_ADAPT_IO);\n    if (r == -1) {\n        if (errno == ENOMEM) {\n            throw std::bad_alloc();\n        }\n        if (errno == EINVAL) {\n            throw std::runtime_error(\"bad cpuset\");\n        }\n        abort();\n    }\n    unsigned procs = c.cpus;\n    if (unsigned available_procs = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_PU);\n        !c.overcommit && procs > available_procs) {\n        throw std::runtime_error(format(\"insufficient processing units: needed {} available {}\", procs, available_procs));\n    }\n    if (procs == 0) {\n        throw std::runtime_error(\"number of processing units must be positive\");\n    }\n\n    size_t available_memory = 0;\n\n    // Get the list of NUMA nodes available\n    std::vector<hwloc_obj_t> nodes;\n\n    hwloc_obj_t tmp = NULL;\n    auto num_nodes = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NUMANODE);\n    SEASTAR_ASSERT(num_nodes > 0);\n    auto nodes_depth = hwloc_get_type_or_above_depth(topology, HWLOC_OBJ_NUMANODE);\n    while ((tmp = hwloc_get_next_obj_by_depth(topology, nodes_depth, tmp)) != NULL) {\n        available_memory += get_memory_from_hwloc_obj(tmp);\n        nodes.push_back(tmp);\n    }\n\n    if (!available_memory) {\n        auto machine_depth = hwloc_get_type_depth(topology, HWLOC_OBJ_MACHINE);\n        SEASTAR_ASSERT(hwloc_get_nbobjs_by_depth(topology, machine_depth) == 1);\n        auto machine = hwloc_get_obj_by_depth(topology, machine_depth, 0);\n        available_memory = get_memory_from_hwloc_obj(machine);\n        if (!available_memory) {\n            available_memory = get_machine_memory_from_sysconf();\n            set_memory_to_hwloc_obj(machine, available_memory);\n            seastar_logger.warn(\"hwloc failed to detect machine-wide memory size, using memory size fetched from sysconf\");\n        }\n\n        auto one_node_mem = available_memory / num_nodes;\n        auto one_node_mem_remainder = available_memory % num_nodes;\n        for (auto& node : nodes) {\n            set_memory_to_hwloc_obj(node, one_node_mem + one_node_mem_remainder);\n            one_node_mem_remainder = 0;\n        }\n    }\n\n    size_t mem = calculate_memory(c, std::min(available_memory,\n                                              cgroup::memory_limit()));\n    seastar::memory::internal::global_setup(procs);\n    auto mem_per_proc = seastar::memory::internal::per_shard_memory(mem, procs);\n\n    resources ret;\n    std::unordered_map<unsigned, hwloc_obj_t> cpu_to_node;\n    std::vector<unsigned> orphan_pus;\n    std::unordered_map<hwloc_obj_t, size_t> topo_used_mem;\n    std::vector<std::pair<cpu, size_t>> remains;\n\n    auto cpu_sets = distribute_objects(topology, procs);\n\n    for (auto&& cs : cpu_sets()) {\n        auto cpu_id = hwloc_bitmap_first(cs);\n        SEASTAR_ASSERT(cpu_id != -1);\n        auto pu = hwloc_get_pu_obj_by_os_index(topology, cpu_id);\n        auto node = get_numa_node_for_pu(topology, pu);\n        if (node == nullptr) {\n            orphan_pus.push_back(cpu_id);\n        } else {\n            cpu_to_node[cpu_id] = node;\n            seastar_logger.debug(\"Assign CPU{} to NUMA{}\", cpu_id, node->os_index);\n        }\n    }\n\n    if (!orphan_pus.empty()) {\n        if (!c.assign_orphan_cpus) {\n            seastar_logger.error(\"CPUs without local NUMA nodes are disabled by the \"\n                        \"--allow-cpus-in-remote-numa-nodes=false option.\\n\");\n            throw std::runtime_error(\"no NUMA node for CPU\");\n        }\n\n        seastar_logger.warn(\"Assigning some CPUs to remote NUMA nodes\");\n\n        // Group orphan CPUs by ... some sane enough feature\n        std::unordered_map<hwloc_obj_t, std::vector<unsigned>> grouped;\n        hwloc_obj_type_t group_by[] = {\n            HWLOC_OBJ_L3CACHE,\n            HWLOC_OBJ_L2CACHE,\n            HWLOC_OBJ_L1CACHE,\n            HWLOC_OBJ_PU,\n        };\n\n        for (auto&& gb : group_by) {\n            grouped = break_cpus_into_groups(topology, orphan_pus, gb);\n            if (grouped.size() >= nodes.size()) {\n                seastar_logger.debug(\"Grouped orphan CPUs by {}\", hwloc_obj_type_string(gb));\n                break;\n            }\n            // Try to scatter orphans into as much NUMA nodes as possible\n            // by grouping them with more specific selection\n        }\n\n        // Distribute PUs among the nodes by groups\n        unsigned nid = 0;\n        for (auto&& grp : grouped) {\n            for (auto&& cpu_id : grp.second) {\n                cpu_to_node[cpu_id] = nodes[nid];\n                seastar_logger.debug(\"Assign orphan CPU{} to NUMA{}\", cpu_id, nodes[nid]->os_index);\n            }\n            nid = (nid + 1) % nodes.size();\n        }\n    }\n\n    // Divide local memory to cpus\n    for (auto&& cs : cpu_sets()) {\n        auto cpu_id = hwloc_bitmap_first(cs);\n        SEASTAR_ASSERT(cpu_id != -1);\n        auto node = cpu_to_node.at(cpu_id);\n        cpu this_cpu;\n        this_cpu.cpu_id = cpu_id;\n        size_t remain = mem_per_proc - alloc_from_node(this_cpu, node, topo_used_mem, mem_per_proc);\n\n        remains.emplace_back(std::move(this_cpu), remain);\n    }\n\n    // Divide the rest of the memory\n    for (auto&& [this_cpu, remain] : remains) {\n        auto node = cpu_to_node.at(this_cpu.cpu_id);\n        auto obj = node;\n\n        while (remain) {\n            remain -= alloc_from_node(this_cpu, obj, topo_used_mem, remain);\n            do {\n                obj = hwloc_get_next_obj_by_depth(topology, nodes_depth, obj);\n            } while (!obj);\n            if (obj == node)\n                break;\n        }\n        SEASTAR_ASSERT(!remain);\n        ret.cpus.push_back(std::move(this_cpu));\n    }\n\n    unsigned last_node_idx = 0;\n    for (auto q : c.io_queues) {\n        ret.ioq_topology.emplace(q, allocate_io_queues(topology, ret.cpus, cpu_to_node, c.num_io_groups, last_node_idx));\n    }\n\n    ret.numa_node_id_to_cpuset = numa_node_id_to_cpuset(topology);\n\n    return ret;\n}\n\nunsigned nr_processing_units(configuration& c) {\n    return hwloc_get_nbobjs_by_type(c.topology.get(), HWLOC_OBJ_PU);\n}\n\n}\n\n}\n\n#else\n\nnamespace seastar {\n\nnamespace resource {\n\n// Without hwloc, we don't support tuning the number of IO queues. So each CPU gets their.\nstatic io_queue_topology\nallocate_io_queues(configuration c, std::vector<cpu> cpus) {\n    io_queue_topology ret;\n\n    unsigned nr_cpus = unsigned(cpus.size());\n    ret.queues.resize(nr_cpus);\n    ret.shard_to_group.resize(nr_cpus);\n    ret.shards_in_group.resize(1, 0);\n    ret.groups.resize(1);\n\n    for (unsigned shard = 0; shard < nr_cpus; ++shard) {\n        ret.shard_to_group[shard] = 0;\n        ret.shards_in_group[0]++;\n    }\n    return ret;\n}\n\n\nresources allocate(configuration& c) {\n    resources ret;\n\n    auto available_memory = get_machine_memory_from_sysconf();\n    auto mem = calculate_memory(c, available_memory);\n    auto procs = c.cpus;\n    ret.cpus.reserve(procs);\n    seastar::memory::internal::global_setup(procs);\n    auto mem_per_proc = seastar::memory::internal::per_shard_memory(mem, procs);\n    for (auto cpuid : c.cpu_set) {\n        ret.cpus.push_back(cpu{cpuid, {{mem_per_proc, 0}}});\n    }\n\n    ret.ioq_topology.emplace(0, allocate_io_queues(c, ret.cpus));\n    return ret;\n}\n\nunsigned nr_processing_units(configuration&) {\n    return ::sysconf(_SC_NPROCESSORS_ONLN);\n}\n\n}\n\n}\n\n#endif\n"
  },
  {
    "path": "src/core/scollectd-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/scollectd.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/net/api.hh>\n#include <chrono>\n\nnamespace seastar {\n\nnamespace scollectd {\n\nusing namespace std::chrono_literals;\nusing duration = std::chrono::milliseconds;\n\nstatic const ipv4_addr default_addr(\"239.192.74.66:25826\");\nstatic const std::chrono::milliseconds default_period(1s);\n\nclass impl {\n    net::datagram_channel _chan;\n    timer<> _timer;\n\n    sstring _host = \"localhost\";\n    ipv4_addr _addr = default_addr;\n    std::chrono::milliseconds _period = default_period;\n    uint64_t _num_packets = 0;\n    uint64_t _millis = 0;\n    uint64_t _bytes = 0;\n    double _avg = 0;\n\npublic:\n    typedef seastar::metrics::impl::value_map value_list_map;\n    typedef value_list_map::value_type value_list_pair;\n\n    void add_polled(const type_instance_id & id,\n            const shared_ptr<value_list> & values, bool enable = true);\n    void remove_polled(const type_instance_id & id);\n    // explicitly send a type_instance value list (outside polling)\n    future<> send_metric(const type_instance_id & id,\n            const value_list & values);\n    future<> send_notification(const type_instance_id & id,\n            const sstring & msg);\n    // initiates actual value polling -> send to target \"loop\"\n    void start(const sstring & host, const ipv4_addr & addr, const std::chrono::milliseconds period);\n    void stop();\n\n    value_list_map& get_value_list_map();\n    const sstring& host() const {\n        return _host;\n    }\n\nprivate:\n    void arm();\n    void run();\n\npublic:\n    shared_ptr<value_list> get_values(const type_instance_id & id) const;\n    std::vector<type_instance_id> get_instance_ids() const;\n    sstring get_collectd_description_str(const scollectd::type_instance_id&) const;\nprivate:\n    const value_list_map& values() const {\n        return seastar::metrics::impl::get_value_map();\n    }\n    seastar::metrics::metric_groups _metrics;\n};\n\nimpl & get_impl();\n\n};\n\n}\n"
  },
  {
    "path": "src/core/scollectd.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <sys/socket.h>\n#include <chrono>\n#include <cmath>\n#include <cstdint>\n#include <functional>\n#include <unordered_map>\n#include <utility>\n#include <string>\n#include <map>\n#include <iostream>\n#include <unordered_map>\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/scollectd_api.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/shared_ptr.hh>\n\n#include \"core/scollectd-impl.hh\"\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nusing seastar::metrics::impl::labels_type;\n\nvoid scollectd::type_instance_id::truncate(sstring& field, const char* field_desc) {\n    if (field.size() > max_collectd_field_text_len) {\n        auto suffix_len = std::ceil(std::log10(++_next_truncated_idx)) + 1;\n        sstring new_field(seastar::format(\n            \"{}~{:d}\", sstring(field.data(), max_collectd_field_text_len - suffix_len), _next_truncated_idx));\n\n        logger.warn(\"Truncating \\\"{}\\\" to {} chars: \\\"{}\\\" -> \\\"{}\\\"\", field_desc, max_collectd_field_text_len, field,\n            new_field);\n        field = std::move(new_field);\n    }\n}\n\nbool scollectd::type_instance_id::operator<(\n        const scollectd::type_instance_id& id2) const {\n    auto& id1 = *this;\n    return std::tie(id1.plugin(), id1.plugin_instance(), id1.type(),\n            id1.type_instance())\n            < std::tie(id2.plugin(), id2.plugin_instance(), id2.type(),\n                    id2.type_instance());\n}\nbool scollectd::type_instance_id::operator==(\n        const scollectd::type_instance_id & id2) const {\n    auto& id1 = *this;\n    return std::tie(id1.plugin(), id1.plugin_instance(), id1.type(),\n            id1.type_instance())\n            == std::tie(id2.plugin(), id2.plugin_instance(), id2.type(),\n                    id2.type_instance());\n}\n\nnamespace scollectd {\n\n::seastar::logger logger(\"scollectd\");\nthread_local unsigned type_instance_id::_next_truncated_idx = 0;\n\nregistration::~registration() {\n    unregister();\n}\n\nregistration::registration(const type_instance_id& id)\n: _id(id), _impl(seastar::metrics::impl::get_local_impl()) {\n}\n\nregistration::registration(type_instance_id&& id)\n: _id(std::move(id)), _impl(seastar::metrics::impl::get_local_impl()) {\n}\n\nseastar::metrics::impl::metric_id to_metrics_id(const type_instance_id & id) {\n    seastar::metrics::impl::labels_type labels {{seastar::metrics::shard_label.name(), seastar::metrics::impl::shard()}};\n    auto internalized_labels = make_lw_shared<seastar::metrics::impl::labels_type>(std::move(labels));\n    return seastar::metrics::impl::metric_id(id.plugin(), id.type_instance(), std::move(internalized_labels));\n}\n\n\nconst plugin_instance_id per_cpu_plugin_instance(\"#cpu\");\n\nstatic const size_t payload_size = 1024;\n\nenum class part_type : uint16_t {\n    Host = 0x0000, // The name of the host to associate with subsequent data values\n    Time = 0x0001, // Time  Numeric The timestamp to associate with subsequent data values, unix time format (seconds since epoch)\n    TimeHr = 0x0008, // Time (high resolution)  Numeric The timestamp to associate with subsequent data values. Time is defined in 2–30 seconds since epoch. New in Version 5.0.\n    Plugin = 0x0002, // Plugin String The plugin name to associate with subsequent data values, e.g. \"cpu\"\n    PluginInst = 0x0003, // Plugin instance String  The plugin instance name to associate with subsequent data values, e.g. \"1\"\n    Type = 0x0004, // Type String The type name to associate with subsequent data values, e.g. \"cpu\"\n    TypeInst = 0x0005, // Type instance     String  The type instance name to associate with subsequent data values, e.g. \"idle\"\n    Values = 0x0006, // Values  other   Data values, see above\n    Interval = 0x0007, // Interval Numeric Interval used to set the \"step\" when creating new RRDs unless rrdtool plugin forces StepSize. Also used to detect values that have timed out.\n    IntervalHr = 0x0009, // Interval (high resolution)  Numeric     The interval in which subsequent data values are collected. The interval is given in 2–30 seconds. New in Version 5.0.\n    Message = 0x0100, // Message (notifications) String\n    Severity = 0x0101, // Severity  Numeric\n    Signature = 0x0200, // Signature (HMAC-SHA-256)     other (todo)\n    Encryption = 0x0210, // Encryption (AES-256/OFB\n};\n\n// \"Time is defined in 2^–30 seconds since epoch. New in Version 5.0.\"\ntypedef std::chrono::duration<uint64_t, std::ratio<1, 0x40000000>> collectd_hres_duration;\n\n// yet another writer type, this one to construct collectd network\n// protocol data.\nstruct cpwriter {\n    typedef std::array<char, payload_size> buffer_type;\n    typedef buffer_type::iterator mark_type;\n    typedef buffer_type::const_iterator const_mark_type;\n\n    buffer_type _buf = {};\n    mark_type _pos;\n    bool _overflow = false;\n\n    std::unordered_map<uint16_t, sstring> _cache;\n\n    cpwriter()\n            : _pos(_buf.begin()) {\n    }\n    mark_type mark() const {\n        return _pos;\n    }\n    bool overflow() const {\n        return _overflow;\n    }\n    void reset(mark_type m) {\n        _pos = m;\n        _overflow = false;\n    }\n    size_t size() const {\n        return std::distance(_buf.begin(), const_mark_type(_pos));\n    }\n    bool empty() const {\n        return _pos == _buf.begin();\n    }\n    void clear() {\n        reset(_buf.begin());\n        _cache.clear();\n        _overflow = false;\n    }\n    const char * data() const {\n        return &_buf.at(0);\n    }\n    char * data() {\n        return &_buf.at(0);\n    }\n    cpwriter& check(size_t sz) {\n        size_t av = std::distance(_pos, _buf.end());\n        _overflow |= av < sz;\n        return *this;\n    }\n\n    sstring get_type_instance(const metrics::metric_name_type& name, const metrics::impl::labels_type& labels) {\n        if (labels.empty()) {\n            return name;\n        }\n        sstring res = name;\n        for (auto i : labels) {\n            if (i.first != seastar::metrics::shard_label.name()) {\n                res += \"-\" + i.second.value();\n            }\n        }\n        return res;\n    }\n    explicit operator bool() const {\n        return !_overflow;\n    }\n    bool operator!() const {\n        return !operator bool();\n    }\n    template<typename _Iter>\n    cpwriter & write(_Iter s, _Iter e) {\n        if (check(std::distance(s, e))) {\n            _pos = std::copy(s, e, _pos);\n        }\n        return *this;\n    }\n    template<std::integral T>\n    cpwriter& write(\n            const T & t) {\n        T tmp = net::hton(t);\n        auto * p = reinterpret_cast<const uint8_t *>(&tmp);\n        auto * e = p + sizeof(T);\n        write(p, e);\n        return *this;\n    }\n    template<std::integral T>\n    cpwriter& write_le(const T & t) {\n        T tmp = cpu_to_le(t);\n        auto * p = reinterpret_cast<const uint8_t *>(&tmp);\n        auto * e = p + sizeof(T);\n        write(p, e);\n        return *this;\n    }\n    void write_value(const seastar::metrics::impl::metric_value& v) {\n        switch (v.type()) {\n            case data_type::GAUGE: {\n                double tmpd = v.d();\n                uint64_t tmpi;\n                std::copy_n(reinterpret_cast<const char*>(&tmpd), 8, reinterpret_cast<char*>(&tmpi));\n                write_le(tmpi);\n                break;\n            }\n            case data_type::COUNTER:\n            case data_type::REAL_COUNTER:\n                write(v.ui()); // unsigned int 64, big endian\n                break;\n            default:\n                SEASTAR_ASSERT(0);\n        }\n    }\n    cpwriter & write(const sstring & s) {\n        write(s.begin(), s.end() + 1); // include \\0\n        return *this;\n    }\n    cpwriter & put(part_type type, const sstring & s) {\n        write(uint16_t(type));\n        write(uint16_t(4 + s.size() + 1)); // include \\0\n        write(s); // include \\0\n        return *this;\n    }\n    cpwriter & put_cached(part_type type, const sstring & s) {\n        auto & cached = _cache[uint16_t(type)];\n        if (cached != s) {\n            put(type, s);\n            cached = s;\n        }\n        return *this;\n    }\n    template<std::integral T>\n    cpwriter& put(\n            part_type type, T t) {\n        write(uint16_t(type));\n        write(uint16_t(4 + sizeof(t)));\n        write(t);\n        return *this;\n    }\n    cpwriter & put(part_type type, const value_list & v) {\n        auto s = v.size();\n        auto sz = 6 + s + s * sizeof(uint64_t);\n        if (check(sz)) {\n            write(uint16_t(type));\n            write(uint16_t(sz));\n            write(uint16_t(s));\n            v.types(reinterpret_cast<data_type *>(&(*_pos)));\n            _pos += s;\n            v.values(reinterpret_cast<net::packed<uint64_t> *>(&(*_pos)));\n            _pos += s * sizeof(uint64_t);\n        }\n        return *this;\n    }\n\n    cpwriter & put(part_type type, const seastar::metrics::impl::metric_value & v) {\n        auto sz = 7 +  sizeof(uint64_t);\n        if (check(sz)) {\n            write(uint16_t(type));\n            write(uint16_t(sz));\n            write(uint16_t(1));\n            write(static_cast<uint8_t>(v.type()));\n            write_value(v);\n        }\n        return *this;\n    }\n    cpwriter & put(const sstring & host, const metrics::group_name_type& group_name, const metrics::metric_name_type& name,\n            const metrics::impl::labels_type& labels, const type_id& type) {\n        const auto ts = std::chrono::system_clock::now().time_since_epoch();\n        const auto lrts =\n                std::chrono::duration_cast<std::chrono::seconds>(ts).count();\n\n        put_cached(part_type::Host, host);\n        put(part_type::Time, uint64_t(lrts));\n        // Seems hi-res timestamp does not work very well with\n        // at the very least my default collectd in fedora (or I did it wrong?)\n        // Use lo-res ts for now, it is probably quite sufficient.\n        put_cached(part_type::Plugin, group_name);\n        // Optional\n        sstring instance_id = labels.at(metrics::shard_label.name());\n        put_cached(part_type::PluginInst,\n                instance_id == per_cpu_plugin_instance ?\n                        to_sstring(this_shard_id()) : instance_id);\n        put_cached(part_type::Type, type);\n        // Optional\n        put_cached(part_type::TypeInst, get_type_instance(name, labels));\n        return *this;\n    }\n    cpwriter & put(const sstring & host,\n            const duration & period,\n            const type_instance_id & id, const value_list & v) {\n        const auto ps = std::chrono::duration_cast<collectd_hres_duration>(\n                        period).count();\n            auto mid = to_metrics_id(id);\n            put(host, mid.group_name(), mid.name(), mid.labels(), id.type());\n            put(part_type::Values, v);\n            if (ps != 0) {\n                put(part_type::IntervalHr, ps);\n            }\n            return *this;\n    }\n\n    cpwriter & put(const sstring & host,\n            const duration & period,\n            const type_id& type,\n            const metrics::group_name_type& group_name, const metrics::metric_name_type& name,\n            const metrics::impl::labels_type& labels, const seastar::metrics::impl::metric_value & v) {\n        const auto ps = std::chrono::duration_cast<collectd_hres_duration>(\n                period).count();\n        put(host, group_name, name, labels, type);\n        put(part_type::Values, v);\n        if (ps != 0) {\n            put(part_type::IntervalHr, ps);\n        }\n        return *this;\n    }\n};\n\nvoid impl::add_polled(const type_instance_id & id,\n        const shared_ptr<value_list> & values, bool enable) {\n    // do nothing\n    // add_polled is now implemented on the metrics layer\n\n}\n\nvoid impl::remove_polled(const type_instance_id & id) {\n    seastar::metrics::impl::unregister_metric(to_metrics_id(id));\n}\n\n// explicitly send a type_instance value list (outside polling)\nfuture<> impl::send_metric(const type_instance_id & id,\n        const value_list & values) {\n    if (values.empty()) {\n        return make_ready_future();\n    }\n    cpwriter out;\n    out.put(_host, duration(), id, values);\n    temporary_buffer<char> buf(out.data(), out.size());\n    return _chan.send(_addr, std::span(&buf, 1));\n}\n\nfuture<> impl::send_notification(const type_instance_id & id,\n        const sstring & msg) {\n    cpwriter out;\n    auto mid = to_metrics_id(id);\n    out.put(_host, mid.group_name(), mid.name(), mid.labels(), id.type());\n    out.put(part_type::Message, msg);\n    temporary_buffer<char> buf(out.data(), out.size());\n    return _chan.send(_addr, std::span(&buf, 1));\n}\n\n// initiates actual value polling -> send to target \"loop\"\nvoid impl::start(const sstring & host, const ipv4_addr & addr, const duration period) {\n    _period = period;\n    _addr = addr;\n    _host = host;\n    _chan = make_unbound_datagram_channel(AF_INET);\n    _timer.set_callback(std::bind(&impl::run, this));\n\n    // dogfood ourselves\n    namespace sm = seastar::metrics;\n\n    _metrics.add_group(\"scollectd\", {\n        // total_bytes      value:DERIVE:0:U\n        sm::make_counter(\"total_bytes_sent\", sm::description(\"total bytes sent\"), _bytes),\n        // total_requests      value:DERIVE:0:U\n        sm::make_counter(\"total_requests\", sm::description(\"total requests\"), _num_packets),\n        // latency          value:GAUGE:0:U\n        sm::make_gauge(\"latency\", sm::description(\"avrage latency\"), _avg),\n        // total_time_in_ms    value:DERIVE:0:U\n        sm::make_counter(\"total_time_in_ms\", sm::description(\"total time in milliseconds\"), _millis),\n        // total_values     value:DERIVE:0:U\n        sm::make_gauge(\"total_values\", sm::description(\"current number of values reported\"), [this] {return values().size();}),\n        // records          value:GAUGE:0:U\n        sm::make_gauge(\"records\", sm::description(\"number of records reported\"), [this] {return values().size();}),\n    });\n\n    // FIXME: future is discarded\n    (void)send_notification(\n            type_instance_id(\"scollectd\", per_cpu_plugin_instance,\n                    \"network\"), \"daemon started\");\n    arm();\n}\n\nvoid impl::stop() {\n    _timer.cancel();\n    _metrics.clear();\n}\n\n\nvoid impl::arm() {\n    if (_period != duration()) {\n        _timer.arm(_period);\n    }\n}\n\nvoid impl::run() {\n    typedef size_t metric_family_id;\n    typedef seastar::metrics::impl::value_vector::iterator value_iterator;\n    typedef seastar::metrics::impl::metric_metadata_fifo::iterator metadata_iterator;\n    typedef std::tuple<metric_family_id, metadata_iterator, value_iterator, type_id, cpwriter> context;\n\n    auto ctxt = make_lw_shared<context>();\n    foreign_ptr<shared_ptr<seastar::metrics::impl::values_copy>> vals =  seastar::metrics::impl::get_values();\n\n    // note we're doing this unsynced since we assume\n    // all registrations to this instance will be done on the\n    // same cpu, and without interuptions (no wait-states)\n\n    auto& values = vals->values;\n    auto metadata = vals->metadata;\n    std::get<metric_family_id>(*ctxt) = 0;\n    if (values.size() > 0) {\n        std::get<value_iterator>(*ctxt) = values[0].begin();\n        std::get<metadata_iterator>(*ctxt) = metadata->at(0).metrics.begin();\n        std::get<type_id>(*ctxt) = metadata->at(0).mf.inherit_type;\n    }\n\n    auto stop_when = [ctxt, metadata]() {\n        auto done = std::get<metric_family_id>(*ctxt) == metadata->size();\n        return done;\n    };\n    // append as many values as we can fit into a packet (1024 bytes)\n    auto send_packet = [this, ctxt, &values, metadata]() mutable {\n        auto start = steady_clock_type::now();\n        auto& mf = std::get<metric_family_id>(*ctxt);\n        auto & md_iterator = std::get<metadata_iterator>(*ctxt);\n        auto & i = std::get<value_iterator>(*ctxt);\n        auto & out = std::get<cpwriter>(*ctxt);\n\n        out.clear();\n\n        bool out_of_space = false;\n        while (!out_of_space && mf < values.size()) {\n            while (i != values[mf].end()) {\n                if (i->type() == seastar::metrics::impl::data_type::HISTOGRAM) {\n                    ++i;\n                    ++md_iterator;\n                    continue;\n                }\n                auto m = out.mark();\n                out.put(_host, _period, std::get<type_id>(*ctxt),\n                    md_iterator->group_name(), md_iterator->name(), md_iterator->labels(), *i);\n                if (!out) {\n                    out.reset(m);\n                    out_of_space = true;\n                    break;\n                }\n                ++i;\n                ++md_iterator;\n            }\n            if (out_of_space) {\n                break;\n            }\n            ++mf;\n            if (mf < values.size()) {\n                i = values[mf].begin();\n                md_iterator = metadata->at(mf).metrics.begin();\n                std::get<type_id>(*ctxt) = metadata->at(mf).mf.inherit_type;\n            }\n        }\n        if (out.empty()) {\n            return make_ready_future();\n        }\n        temporary_buffer<char> buf(out.data(), out.size());\n        return _chan.send(_addr, std::span(&buf, 1)).then([start, ctxt, this]() {\n                    auto & out = std::get<cpwriter>(*ctxt);\n                    auto now = steady_clock_type::now();\n                    // dogfood stats\n                    ++_num_packets;\n                    _millis += std::chrono::duration_cast<std::chrono::milliseconds>(now - start).count();\n                    _bytes += out.size();\n                    _avg = double(_millis) / _num_packets;\n                }).then_wrapped([] (auto&& f) {\n                    try {\n                        f.get();\n                    } catch (std::exception & ex) {\n                        std::cout << \"send failed: \" << ex.what() << std::endl;\n                    } catch (...) {\n                        std::cout << \"send failed: - unknown exception\" << std::endl;\n                    }\n                });\n    };\n    // No need to wait for future.\n    // The caller has to call impl::stop() to synchronize.\n    (void)do_until(stop_when, send_packet).finally([this, vals = std::move(vals)]() mutable {\n        arm();\n    });\n}\n\nstd::vector<type_instance_id> impl::get_instance_ids() const {\n    std::vector<type_instance_id> res;\n    for (auto&& v: values()) {\n        // Need to check for empty value_list, since unreg is two-stage.\n        // Not an issue for most uses, but unit testing etc that would like\n        // fully deterministic operation here would like us to only return\n        // actually active ids\n        for (auto i : v.second) {\n            if (i.second) {\n                res.emplace_back(i.second->get_id(), v.second.info().inherit_type);\n            }\n        }\n    }\n    return res;\n}\n\nvoid add_polled(const type_instance_id & id,\n        const shared_ptr<value_list> & values, bool enabled) {\n    get_impl().add_polled(id, values, enabled);\n}\n\nvoid remove_polled_metric(const type_instance_id & id) {\n    get_impl().remove_polled(id);\n}\n\nfuture<> send_notification(const type_instance_id & id,\n        const sstring & msg) {\n    return get_impl().send_notification(id, msg);\n}\n\nfuture<> send_metric(const type_instance_id & id,\n        const value_list & values) {\n    return get_impl().send_metric(id, values);\n}\n\nvoid configure(const options& opts) {\n    bool enable = opts.collectd.get_value();\n    if (!enable) {\n        return;\n    }\n    auto addr = ipv4_addr(opts.collectd_address.get_value());\n    auto period = std::chrono::milliseconds(opts.collectd_poll_period.get_value());\n\n    auto host = (opts.collectd_hostname.get_value() == \"\")\n            ? seastar::metrics::impl::get_local_impl()->get_config().hostname\n            : sstring(opts.collectd_hostname.get_value());\n\n    // Now create send loops on each cpu\n    for (unsigned c = 0; c < smp::count; c++) {\n        // FIXME: future is discarded\n        (void)smp::submit_to(c, [=] () {\n            get_impl().start(host, addr, period);\n        });\n    }\n}\n\noptions::options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"COLLECTD options\")\n    , collectd(*this, \"collectd\", false,\n            \"enable collectd daemon\")\n    , collectd_address(*this, \"collectd-address\",\n            \"239.192.74.66:25826\",\n            \"address to send/broadcast metrics to\")\n    , collectd_poll_period(*this, \"collectd-poll-period\",\n            1000,\n            \"poll period - frequency of sending counter metrics (default: 1000ms, 0 disables)\")\n    , collectd_hostname(*this, \"collectd-hostname\",\n            \"\",\n            \"Deprecated option, use metrics-hostname instead\")\n{\n}\n\nstatic seastar::metrics::impl::register_ref get_register(const scollectd::type_instance_id& i) {\n    seastar::metrics::impl::metric_id id = to_metrics_id(i);\n    return seastar::metrics::impl::get_value_map().at(id.full_name()).at(id.internalized_labels());\n}\n\nstd::vector<collectd_value> get_collectd_value(\n        const scollectd::type_instance_id& id) {\n    std::vector<collectd_value> vals;\n    const seastar::metrics::impl::registered_metric& val = *get_register(id);\n    vals.push_back(val());\n    return vals;\n}\n\nstd::vector<scollectd::type_instance_id> get_collectd_ids() {\n    return get_impl().get_instance_ids();\n}\n\nbool is_enabled(const scollectd::type_instance_id& id) {\n    return get_register(id)->is_enabled();\n}\n\nvoid enable(const scollectd::type_instance_id& id, bool enable) {\n    get_register(id)->set_enabled(enable);\n}\n\ntype_instance_id plugin_instance_metrics::add_impl(const typed_value& v) {\n    type_instance_id id(_plugin_id, _plugin_instance, v.type(), v.type_instance());\n    get_impl().add_polled(id, v.values());\n    return id;\n}\n\nvoid plugin_instance_metrics::add(const typed_value& v) {\n    _registrations.emplace_back(add_impl(v));\n}\n\nstd::vector<type_instance_id> plugin_instance_metrics::bound_ids() const {\n    std::vector<type_instance_id> res;\n    res.reserve(_registrations.size());\n    std::transform(_registrations.begin(), _registrations.end(), std::back_inserter(res), [](const registration& r) {\n       return r._id;\n    });\n    return res;\n}\n\ntype_id type_id_for(known_type t) {\n    switch (t) {\n    case known_type::absolute:\n        return \"absolute\";\n    case known_type::backends:\n        return \"backends\";\n    case known_type::bitrate:\n        return \"bitrate\";\n    case known_type::blocked_clients:\n        return \"blocked_clients\";\n    case known_type::bytes:\n        return \"bytes\";\n    case known_type::cache_eviction:\n        return \"cache_eviction\";\n    case known_type::cache_operation:\n        return \"cache_operation\";\n    case known_type::cache_ratio:\n        return \"cache_ratio\";\n    case known_type::cache_result:\n        return \"cache_result\";\n    case known_type::cache_size:\n        return \"cache_size\";\n    case known_type::capacity:\n        return \"capacity\";\n    case known_type::changes_since_last_save:\n        return \"changes_since_last_save\";\n    case known_type::charge:\n        return \"charge\";\n    case known_type::clock_last_meas:\n        return \"clock_last_meas\";\n    case known_type::clock_last_update:\n        return \"clock_last_update\";\n    case known_type::clock_mode:\n        return \"clock_mode\";\n    case known_type::clock_reachability:\n        return \"clock_reachability\";\n    case known_type::clock_skew_ppm:\n        return \"clock_skew_ppm\";\n    case known_type::clock_state:\n        return \"clock_state\";\n    case known_type::clock_stratum:\n        return \"clock_stratum\";\n    case known_type::compression:\n        return \"compression\";\n    case known_type::compression_ratio:\n        return \"compression_ratio\";\n    case known_type::connections:\n        return \"connections\";\n    case known_type::conntrack:\n        return \"conntrack\";\n    case known_type::contextswitch:\n        return \"contextswitch\";\n    case known_type::count:\n        return \"count\";\n    case known_type::counter:\n        return \"counter\";\n    case known_type::cpu:\n        return \"cpu\";\n    case known_type::cpufreq:\n        return \"cpufreq\";\n    case known_type::current:\n        return \"current\";\n    case known_type::current_connections:\n        return \"current_connections\";\n    case known_type::current_sessions:\n        return \"current_sessions\";\n    case known_type::delay:\n        return \"delay\";\n    case known_type::derive:\n        return \"derive\";\n    case known_type::df:\n        return \"df\";\n    case known_type::df_complex:\n        return \"df_complex\";\n    case known_type::df_inodes:\n        return \"df_inodes\";\n    case known_type::disk_io_time:\n        return \"disk_io_time\";\n    case known_type::disk_latency:\n        return \"disk_latency\";\n    case known_type::disk_merged:\n        return \"disk_merged\";\n    case known_type::disk_octets:\n        return \"disk_octets\";\n    case known_type::disk_ops:\n        return \"disk_ops\";\n    case known_type::disk_ops_complex:\n        return \"disk_ops_complex\";\n    case known_type::disk_time:\n        return \"disk_time\";\n    case known_type::dns_answer:\n        return \"dns_answer\";\n    case known_type::dns_notify:\n        return \"dns_notify\";\n    case known_type::dns_octets:\n        return \"dns_octets\";\n    case known_type::dns_opcode:\n        return \"dns_opcode\";\n    case known_type::dns_qtype:\n        return \"dns_qtype\";\n    case known_type::dns_qtype_cached:\n        return \"dns_qtype_cached\";\n    case known_type::dns_query:\n        return \"dns_query\";\n    case known_type::dns_question:\n        return \"dns_question\";\n    case known_type::dns_rcode:\n        return \"dns_rcode\";\n    case known_type::dns_reject:\n        return \"dns_reject\";\n    case known_type::dns_request:\n        return \"dns_request\";\n    case known_type::dns_resolver:\n        return \"dns_resolver\";\n    case known_type::dns_response:\n        return \"dns_response\";\n    case known_type::dns_transfer:\n        return \"dns_transfer\";\n    case known_type::dns_update:\n        return \"dns_update\";\n    case known_type::dns_zops:\n        return \"dns_zops\";\n    case known_type::drbd_resource:\n        return \"drbd_resource\";\n    case known_type::duration:\n        return \"duration\";\n    case known_type::email_check:\n        return \"email_check\";\n    case known_type::email_count:\n        return \"email_count\";\n    case known_type::email_size:\n        return \"email_size\";\n    case known_type::entropy:\n        return \"entropy\";\n    case known_type::evicted_keys:\n        return \"evicted_keys\";\n    case known_type::expired_keys:\n        return \"expired_keys\";\n    case known_type::fanspeed:\n        return \"fanspeed\";\n    case known_type::file_handles:\n        return \"file_handles\";\n    case known_type::file_size:\n        return \"file_size\";\n    case known_type::files:\n        return \"files\";\n    case known_type::flow:\n        return \"flow\";\n    case known_type::fork_rate:\n        return \"fork_rate\";\n    case known_type::frequency:\n        return \"frequency\";\n    case known_type::frequency_error:\n        return \"frequency_error\";\n    case known_type::frequency_offset:\n        return \"frequency_offset\";\n    case known_type::fscache_stat:\n        return \"fscache_stat\";\n    case known_type::gauge:\n        return \"gauge\";\n    case known_type::hash_collisions:\n        return \"hash_collisions\";\n    case known_type::http_request_methods:\n        return \"http_request_methods\";\n    case known_type::http_requests:\n        return \"http_requests\";\n    case known_type::http_response_codes:\n        return \"http_response_codes\";\n    case known_type::humidity:\n        return \"humidity\";\n    case known_type::if_collisions:\n        return \"if_collisions\";\n    case known_type::if_dropped:\n        return \"if_dropped\";\n    case known_type::if_errors:\n        return \"if_errors\";\n    case known_type::if_multicast:\n        return \"if_multicast\";\n    case known_type::if_octets:\n        return \"if_octets\";\n    case known_type::if_packets:\n        return \"if_packets\";\n    case known_type::if_rx_errors:\n        return \"if_rx_errors\";\n    case known_type::if_rx_octets:\n        return \"if_rx_octets\";\n    case known_type::if_tx_errors:\n        return \"if_tx_errors\";\n    case known_type::if_tx_octets:\n        return \"if_tx_octets\";\n    case known_type::invocations:\n        return \"invocations\";\n    case known_type::io_octets:\n        return \"io_octets\";\n    case known_type::io_packets:\n        return \"io_packets\";\n    case known_type::ipt_bytes:\n        return \"ipt_bytes\";\n    case known_type::ipt_packets:\n        return \"ipt_packets\";\n    case known_type::irq:\n        return \"irq\";\n    case known_type::latency:\n        return \"latency\";\n    case known_type::links:\n        return \"links\";\n    case known_type::load:\n        return \"load\";\n    case known_type::md_disks:\n        return \"md_disks\";\n    case known_type::memory:\n        return \"memory\";\n    case known_type::memory_lua:\n        return \"memory_lua\";\n    case known_type::memory_throttle_count:\n        return \"memory_throttle_count\";\n    case known_type::multimeter:\n        return \"multimeter\";\n    case known_type::mutex_operations:\n        return \"mutex_operations\";\n    case known_type::objects:\n        return \"objects\";\n    case known_type::operations:\n        return \"operations\";\n    case known_type::packets:\n        return \"packets\";\n    case known_type::pending_operations:\n        return \"pending_operations\";\n    case known_type::percent:\n        return \"percent\";\n    case known_type::percent_bytes:\n        return \"percent_bytes\";\n    case known_type::percent_inodes:\n        return \"percent_inodes\";\n    case known_type::ping:\n        return \"ping\";\n    case known_type::ping_droprate:\n        return \"ping_droprate\";\n    case known_type::ping_stddev:\n        return \"ping_stddev\";\n    case known_type::players:\n        return \"players\";\n    case known_type::power:\n        return \"power\";\n    case known_type::pressure:\n        return \"pressure\";\n    case known_type::protocol_counter:\n        return \"protocol_counter\";\n    case known_type::pubsub:\n        return \"pubsub\";\n    case known_type::queue_length:\n        return \"queue_length\";\n    case known_type::records:\n        return \"records\";\n    case known_type::requests:\n        return \"requests\";\n    case known_type::response_code:\n        return \"response_code\";\n    case known_type::response_time:\n        return \"response_time\";\n    case known_type::root_delay:\n        return \"root_delay\";\n    case known_type::root_dispersion:\n        return \"root_dispersion\";\n    case known_type::route_etx:\n        return \"route_etx\";\n    case known_type::route_metric:\n        return \"route_metric\";\n    case known_type::routes:\n        return \"routes\";\n    case known_type::segments:\n        return \"segments\";\n    case known_type::serial_octets:\n        return \"serial_octets\";\n    case known_type::signal_noise:\n        return \"signal_noise\";\n    case known_type::signal_power:\n        return \"signal_power\";\n    case known_type::signal_quality:\n        return \"signal_quality\";\n    case known_type::snr:\n        return \"snr\";\n    case known_type::spl:\n        return \"spl\";\n    case known_type::swap:\n        return \"swap\";\n    case known_type::swap_io:\n        return \"swap_io\";\n    case known_type::tcp_connections:\n        return \"tcp_connections\";\n    case known_type::temperature:\n        return \"temperature\";\n    case known_type::threads:\n        return \"threads\";\n    case known_type::time_dispersion:\n        return \"time_dispersion\";\n    case known_type::time_offset:\n        return \"time_offset\";\n    case known_type::time_offset_ntp:\n        return \"time_offset_ntp\";\n    case known_type::time_offset_rms:\n        return \"time_offset_rms\";\n    case known_type::time_ref:\n        return \"time_ref\";\n    case known_type::timeleft:\n        return \"timeleft\";\n    case known_type::total_bytes:\n        return \"total_bytes\";\n    case known_type::total_connections:\n        return \"total_connections\";\n    case known_type::total_objects:\n        return \"total_objects\";\n    case known_type::total_operations:\n        return \"total_operations\";\n    case known_type::total_requests:\n        return \"total_requests\";\n    case known_type::total_sessions:\n        return \"total_sessions\";\n    case known_type::total_threads:\n        return \"total_threads\";\n    case known_type::total_time_in_ms:\n        return \"total_time_in_ms\";\n    case known_type::total_values:\n        return \"total_values\";\n    case known_type::uptime:\n        return \"uptime\";\n    case known_type::users:\n        return \"users\";\n    case known_type::vcl:\n        return \"vcl\";\n    case known_type::vcpu:\n        return \"vcpu\";\n    case known_type::virt_cpu_total:\n        return \"virt_cpu_total\";\n    case known_type::virt_vcpu:\n        return \"virt_vcpu\";\n    case known_type::vmpage_action:\n        return \"vmpage_action\";\n    case known_type::vmpage_faults:\n        return \"vmpage_faults\";\n    case known_type::vmpage_io:\n        return \"vmpage_io\";\n    case known_type::vmpage_number:\n        return \"vmpage_number\";\n    case known_type::volatile_changes:\n        return \"volatile_changes\";\n    case known_type::voltage:\n        return \"voltage\";\n    case known_type::voltage_threshold:\n        return \"voltage_threshold\";\n    case known_type::vs_memory:\n        return \"vs_memory\";\n    case known_type::vs_processes:\n        return \"vs_processes\";\n    case known_type::vs_threads:\n        return \"vs_threads\";\n    default:\n        throw std::invalid_argument(\"Unknown type\");\n    }\n}\n\nmetrics::impl::value_map get_value_map() {\n    return metrics::impl::get_value_map();\n}\n\n}\n\nthread_local scollectd::impl scollectd_impl;\n\nscollectd::impl & scollectd::get_impl() {\n    return scollectd_impl;\n}\n\n}\n"
  },
  {
    "path": "src/core/semaphore.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n\n#include <fmt/format.h>\n\n#include <seastar/core/semaphore.hh>\n\nnamespace seastar {\n\n// Exception Factory for standard semaphore\n//\n// constructs standard semaphore exceptions\n// \\see semaphore_timed_out and broken_semaphore\n\nstatic_assert(std::is_nothrow_default_constructible_v<semaphore_default_exception_factory>);\nstatic_assert(std::is_nothrow_move_constructible_v<semaphore_default_exception_factory>);\n\nstatic_assert(std::is_nothrow_constructible_v<semaphore, size_t>);\nstatic_assert(std::is_nothrow_constructible_v<semaphore, size_t, semaphore_default_exception_factory&&>);\nstatic_assert(std::is_nothrow_move_constructible_v<semaphore>);\n\n\nconst char* broken_semaphore::what() const noexcept {\n    return \"Semaphore broken\";\n}\n\nconst char* semaphore_timed_out::what() const noexcept {\n    return \"Semaphore timedout\";\n}\n\nconst char* semaphore_aborted::what() const noexcept {\n    return \"Semaphore aborted\";\n}\n\nsemaphore_timed_out semaphore_default_exception_factory::timeout() noexcept {\n    static_assert(std::is_nothrow_default_constructible_v<semaphore_timed_out>);\n    return semaphore_timed_out();\n}\n\nbroken_semaphore semaphore_default_exception_factory::broken() noexcept {\n    static_assert(std::is_nothrow_default_constructible_v<broken_semaphore>);\n    return broken_semaphore();\n}\n\nsemaphore_aborted semaphore_default_exception_factory::aborted() noexcept {\n    static_assert(std::is_nothrow_default_constructible_v<semaphore_aborted>);\n    return semaphore_aborted();\n}\n\n// A factory of semaphore exceptions that contain additional context: the semaphore name\n// auto sem = named_semaphore(0, named_semaphore_exception_factory{\"file_opening_limit_semaphore\"});\n\nstatic_assert(std::is_nothrow_default_constructible_v<named_semaphore_exception_factory>);\nstatic_assert(std::is_nothrow_move_constructible_v<named_semaphore_exception_factory>);\n\nstatic_assert(std::is_nothrow_constructible_v<named_semaphore, size_t>);\nstatic_assert(std::is_nothrow_constructible_v<named_semaphore, size_t, named_semaphore_exception_factory&&>);\nstatic_assert(std::is_nothrow_move_constructible_v<named_semaphore>);\n\nnamed_semaphore_timed_out::named_semaphore_timed_out(std::string_view msg) noexcept : _msg() {\n    try {\n        _msg = seastar::format(\"Semaphore timed out: {}\", msg);\n    } catch (...) {\n        // ignore, empty _msg will generate a static message in what().\n    }\n}\n\nbroken_named_semaphore::broken_named_semaphore(std::string_view msg) noexcept : _msg() {\n    try {\n        _msg = seastar::format(\"Semaphore broken: {}\", msg);\n    } catch (...) {\n        // ignore, empty _msg will generate a static message in what().\n    }\n}\n\nnamed_semaphore_aborted::named_semaphore_aborted(std::string_view msg) noexcept : _msg() {\n    try {\n        _msg = seastar::format(\"Semaphore aborted: {}\", msg);\n    } catch (...) {\n        // ignore, empty _msg will generate a static message in what().\n    }\n}\n\nconst char* named_semaphore_timed_out::what() const noexcept {\n    // return a static message if generating the dynamic message failed.\n    return _msg.empty() ? \"Named semaphore timed out\" : _msg.c_str();\n}\n\nconst char* broken_named_semaphore::what() const noexcept {\n    // return a static message if generating the dynamic message failed.\n    return _msg.empty() ? \"Broken named semaphore\" : _msg.c_str();\n}\n\nconst char* named_semaphore_aborted::what() const noexcept {\n    // return a static message if generating the dynamic message failed.\n    return _msg.empty() ? \"Named semaphore aborted\" : _msg.c_str();\n}\n\nnamed_semaphore_timed_out named_semaphore_exception_factory::timeout() const noexcept {\n    return named_semaphore_timed_out(name);\n}\n\nbroken_named_semaphore named_semaphore_exception_factory::broken() const noexcept {\n    return broken_named_semaphore(name);\n}\n\nnamed_semaphore_aborted named_semaphore_exception_factory::aborted() const noexcept {\n    return named_semaphore_aborted(name);\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "src/core/sharded.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB\n */\n\n\n#include <ranges>\n\n#include <seastar/core/sharded.hh>\n#include <seastar/core/loop.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\n\nfuture<>\nsharded_parallel_for_each(unsigned nr_shards, on_each_shard_func on_each_shard) noexcept(std::is_nothrow_move_constructible_v<on_each_shard_func>) {\n    return parallel_for_each(std::views::iota(0u, nr_shards), std::move(on_each_shard));\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/core/signal.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n\n#include <stdexcept>\n\n#include <seastar/core/signal.hh>\n#include <seastar/core/reactor.hh>\n\nnamespace seastar {\n\nvoid handle_signal(int signo, noncopyable_function<void ()>&& handler, bool once) {\n    auto& r = engine();\n    if (once) {\n        r._signals.handle_signal_once(signo, std::move(handler));\n    } else {\n        r._signals.handle_signal(signo, std::move(handler));\n    }\n}\n\n}\n"
  },
  {
    "path": "src/core/smp.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#include <boost/range/algorithm/find_if.hpp>\n#include <atomic>\n#include <vector>\n#include <regex>\n#include <sys/mman.h>\n#include <unistd.h>\n#include <fcntl.h>\n\n#include <seastar/core/smp.hh>\n#include <seastar/core/alien.hh>\n#include <seastar/core/resource.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/on_internal_error.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/align.hh>\n#include \"prefault.hh\"\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nshard_id* internal::this_shard_id_ptr() noexcept {\n    static thread_local shard_id g_this_shard_id;\n    return &g_this_shard_id;\n}\n#endif\n\nvoid smp_message_queue::work_item::process() {\n    schedule(this);\n}\n\nstruct smp_service_group_impl {\n    std::vector<smp_service_group_semaphore> clients;   // one client per server shard\n#ifdef SEASTAR_DEBUG\n    unsigned version = 0;\n#endif\n};\n\nstatic thread_local smp_service_group_semaphore smp_service_group_management_sem{1, named_semaphore_exception_factory{\"smp_service_group_management_sem\"}};\nstatic thread_local std::vector<smp_service_group_impl> smp_service_groups;\n\nstatic named_semaphore_exception_factory make_service_group_semaphore_exception_factory(unsigned id, shard_id client_cpu, shard_id this_cpu, std::optional<sstring> smp_group_name) {\n    if (smp_group_name) {\n        return named_semaphore_exception_factory{format(\"smp_service_group:'{}' (#{}) {}->{} semaphore\", *smp_group_name, id, client_cpu, this_cpu)};\n    } else {\n        return named_semaphore_exception_factory{format(\"smp_service_group#{} {}->{} semaphore\", id, client_cpu, this_cpu)};\n    }\n\n}\n\nstatic_assert(std::is_nothrow_copy_constructible_v<smp_service_group>);\nstatic_assert(std::is_nothrow_move_constructible_v<smp_service_group>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<smp_submit_to_options>);\nstatic_assert(std::is_nothrow_copy_constructible_v<smp_submit_to_options>);\nstatic_assert(std::is_nothrow_move_constructible_v<smp_submit_to_options>);\n\nfuture<smp_service_group> create_smp_service_group(smp_service_group_config ssgc) noexcept {\n    ssgc.max_nonlocal_requests = std::max(ssgc.max_nonlocal_requests, smp::count - 1);\n    return smp::submit_to(0, [ssgc] {\n        return with_semaphore(smp_service_group_management_sem, 1, [ssgc] {\n            auto it = boost::range::find_if(smp_service_groups, [&] (smp_service_group_impl& ssgi) { return ssgi.clients.empty(); });\n            size_t id = it - smp_service_groups.begin();\n            return parallel_for_each(smp::all_cpus(), [ssgc, id] (unsigned cpu) {\n              return smp::submit_to(cpu, [ssgc, id, cpu] {\n                if (id >= smp_service_groups.size()) {\n                    smp_service_groups.resize(id + 1); // may throw\n                }\n                smp_service_groups[id].clients.reserve(smp::count); // may throw\n                auto per_client = smp::count > 1 ? ssgc.max_nonlocal_requests / (smp::count - 1) : 0u;\n                for (unsigned i = 0; i != smp::count; ++i) {\n                    smp_service_groups[id].clients.emplace_back(per_client, make_service_group_semaphore_exception_factory(id, i, cpu, ssgc.group_name));\n                }\n              });\n            }).handle_exception([id] (std::exception_ptr e) {\n                // rollback\n                return smp::invoke_on_all([id] {\n                    if (smp_service_groups.size() > id) {\n                        smp_service_groups[id].clients.clear();\n                    }\n                }).then([e = std::move(e)] () mutable {\n                    std::rethrow_exception(std::move(e));\n                });\n            }).then([id] {\n                auto ret = smp_service_group(id);\n#ifdef SEASTAR_DEBUG\n                ret._version = smp_service_groups[id].version;\n#endif\n                return ret;\n            });\n        });\n    });\n}\n\nfuture<> destroy_smp_service_group(smp_service_group ssg) noexcept {\n    return smp::submit_to(0, [ssg] {\n        return with_semaphore(smp_service_group_management_sem, 1, [ssg] {\n            auto id = internal::smp_service_group_id(ssg);\n            if (id >= smp_service_groups.size()) {\n                on_fatal_internal_error(seastar_logger, format(\"destroy_smp_service_group id={}: out of range\", id));\n            }\n#ifdef SEASTAR_DEBUG\n            if (ssg._version != smp_service_groups[id].version) {\n                on_fatal_internal_error(seastar_logger, format(\"destroy_smp_service_group id={}: stale version={}: current_version={}\", id, ssg._version, smp_service_groups[id].version));\n            }\n#endif\n            return smp::invoke_on_all([id] {\n                smp_service_groups[id].clients.clear();\n#ifdef SEASTAR_DEBUG\n                ++smp_service_groups[id].version;\n#endif\n            });\n        });\n    });\n}\n\nvoid init_default_smp_service_group(shard_id cpu) {\n    // default_smp_service_group == smp_service_group(0) -> we assume service groups are empty\n    // at this point. If they are not, it is quite possibly because we are running repeated\n    // reactors in the program. Probably a test (see #2148).\n    // This would be fine, we just create extra junk here, _but_ it is quite possible\n    // that we actually run with different cpu count (see smp_options::smp), in which case\n    // the `get_smp_service_groups_semaphore` below can cause us to return uninitialized memory.\n    smp_service_groups.clear();\n    smp_service_groups.emplace_back();\n    auto& ssg0 = smp_service_groups.back();\n    ssg0.clients.reserve(smp::count);\n    for (unsigned i = 0; i != smp::count; ++i) {\n        ssg0.clients.emplace_back(smp_service_group_semaphore::max_counter(), make_service_group_semaphore_exception_factory(0, i, cpu, {\"default\"}));\n    }\n}\n\nsmp_service_group_semaphore& get_smp_service_groups_semaphore(unsigned ssg_id, shard_id t) noexcept {\n    return smp_service_groups[ssg_id].clients[t];\n}\n\nsmp::smp(alien::instance& alien)\n        : _alien(alien) {\n}\n\n\nsmp::~smp() = default;\n\nvoid\nsmp::setup_prefaulter(const seastar::resource::resources& res, seastar::memory::internal::numa_layout layout) {\n    // Stack guards mprotect() random pages, so the prefaulter will hard-fault.\n#ifndef SEASTAR_THREAD_STACK_GUARDS\n    _prefaulter = std::make_unique<internal::memory_prefaulter>(_alien, res, std::move(layout));\n#endif\n}\n\nstatic\nstd::optional<size_t>\nget_huge_page_size() {\n    auto meminfo_fd = file_desc::open(\"/proc/meminfo\", O_RDONLY | O_CLOEXEC);\n    std::string meminfo;\n    char buf[4096];\n    while (auto size_opt = meminfo_fd.read(buf, sizeof(buf))) {\n        if (!*size_opt) {\n            break;\n        }\n        meminfo.append(buf, *size_opt);\n    }\n    static std::regex re(R\"(Hugepagesize:\\s*(\\d+) kB)\");\n    auto m = std::smatch{};\n    if (std::regex_search(meminfo, m, re)) {\n        return std::stoi(m[1]) * size_t(1024);\n    }\n    return std::nullopt;\n}\n\ninternal::memory_prefaulter::memory_prefaulter(alien::instance& alien, const resource::resources& res, memory::internal::numa_layout layout) {\n    for (auto& range : layout.ranges) {\n        _layout_by_node_id[range.numa_node_id].push_back(std::move(range));\n    }\n    auto page_size = getpagesize();\n    auto huge_page_size_opt = get_huge_page_size();\n    for (auto& numa_node_id_and_ranges : _layout_by_node_id) {\n        auto& numa_node_id = numa_node_id_and_ranges.first;\n        auto& ranges = numa_node_id_and_ranges.second;\n        posix_thread::attr a;\n        auto i = res.numa_node_id_to_cpuset.find(numa_node_id);\n        if (i != res.numa_node_id_to_cpuset.end()) {\n            cpu_set_t cpuset;\n            CPU_ZERO(&cpuset);\n            for (auto cpu : i->second) {\n                CPU_SET(cpu, &cpuset);\n            }\n            a.set(cpuset);\n        }\n        _worker_threads.emplace_back(a, [this, &alien, &ranges, page_size, huge_page_size_opt] {\n            ++_active_threads;\n            work(ranges, page_size, huge_page_size_opt);\n            if (!--_active_threads) {\n                run_on(alien, 0, [this] () noexcept { join_threads(); });\n            }\n        });\n    }\n}\n\nvoid\ninternal::memory_prefaulter::join_threads() noexcept {\n    for (auto& t : _worker_threads) {\n        t.join();\n    }\n    _worker_threads.clear();\n    _layout_by_node_id.clear();\n}\n\ninternal::memory_prefaulter::~memory_prefaulter() {\n    _stop_request.store(true, std::memory_order_relaxed);\n    join_threads();\n}\n\nvoid\ninternal::memory_prefaulter::work(std::vector<memory::internal::memory_range>& ranges, size_t page_size,\n        std::optional<size_t> huge_page_size_opt) {\n    sched_param param = { .sched_priority = 0 };\n    // SCHED_IDLE doesn't work via thread attributes\n    pthread_setschedparam(pthread_self(), SCHED_IDLE, &param);\n    size_t current_range = 0;\n    const size_t batch_size = huge_page_size_opt.value_or(512*4096);\n    auto populate_memory_madvise = [works = true] (char* start, char* end) mutable {\n#ifdef MADV_POPULATE_WRITE\n        if (works) {\n            auto result = madvise(start, end - start, MADV_POPULATE_WRITE);\n            if (result == -1 && errno == EINVAL) {\n                // Unsupported by kernel\n                works = false;\n                return false;\n            }\n            // Ignore other errors. This is just an optimization anyway.\n#ifdef MADV_COLLAPSE\n            madvise(start, end - start, MADV_COLLAPSE);\n            // Also ignore problems with MADV_COLLAPSE, it's just an optimization.\n#endif\n            return true;\n        };\n#endif\n        (void)works; // suppress warning if block above is elided\n        return false;\n    };\n    auto fault_in_memory = [] (char* p) {\n        // Touch the page for write, but be sure not to modify anything\n        // The compilers tend to optimize things, so prefer assembly\n#if defined(__x86_64__)\n        asm volatile (\"lock orb $0, %0\" : \"=&m\"(*p));\n#elif defined(__aarch64__)\n        int byte; // ldxrb likes 32-bit registers\n        int need_loop;\n        asm volatile (\"1: ldxrb %w0, %2; stxrb %w1, %w0, %2; cbnz %w1, 1b\"\n                : \"=&r\"(byte), \"=&r\"(need_loop), \"+Q\"(*p));\n#else\n        // atomic_ref would be better, but alas C++20 only\n        auto p1 = reinterpret_cast<volatile std::atomic<char>*>(p);\n        p1->fetch_or(0, std::memory_order_relaxed);\n#endif\n    };\n    auto populate_memory = [&] (char* start, char* end) {\n        if (populate_memory_madvise(start, end)) {\n            return;\n        }\n        while (start < end) {\n            fault_in_memory(start);\n            start += page_size;\n        }\n    };\n    while (!_stop_request.load(std::memory_order_relaxed) && !ranges.empty()) {\n        auto& range = ranges[current_range];\n\n        auto batch_end = std::min(align_up(range.start + 1, batch_size), range.end);\n        populate_memory(range.start, batch_end);\n        range.start = batch_end;\n\n        // An end-to-start scan for applications that manage two heaps that\n        // grow towards each other.\n        auto batch_start = std::max(align_down(range.end - 1, batch_size), range.start);\n        populate_memory(batch_start, range.end);\n        range.end = batch_start;\n\n        if (range.start >= range.end) {\n            ranges.erase(ranges.begin() + current_range);\n            current_range = 0;\n        }\n        current_range += 1;\n        if (current_range >= ranges.size()) {\n            current_range = 0;\n        }\n    }\n}\n\n}\n"
  },
  {
    "path": "src/core/sstring.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB\n */\n\n#include <seastar/core/sstring.hh>\n\nusing namespace seastar;\n\n[[noreturn]] void internal::throw_bad_alloc() {\n    throw std::bad_alloc();\n}\n\n[[noreturn]] void internal::throw_sstring_overflow() {\n    throw std::overflow_error(\"sstring overflow\");\n}\n\n[[noreturn]] void internal::throw_sstring_out_of_range() {\n    throw std::out_of_range(\"sstring out of range\");\n}\n"
  },
  {
    "path": "src/core/syscall_result.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\nnamespace seastar {\n\nnamespace fs = std::filesystem;\n\n#pragma once\ntemplate <typename T>\nstruct syscall_result {\n    T result;\n    int error;\n    syscall_result(T result, int error) : result{std::move(result)}, error{error} {\n    }\n    void throw_if_error() const {\n        if (long(result) == -1) {\n            throw std::system_error(ec());\n        }\n    }\n\n    void throw_fs_exception(const sstring& reason, const fs::path& path) const {\n        throw fs::filesystem_error(reason, path, ec());\n    }\n\n    void throw_fs_exception(const sstring& reason, const fs::path& path1, const fs::path& path2) const {\n        throw fs::filesystem_error(reason, path1, path2, ec());\n    }\n\n    void throw_fs_exception_if_error(const sstring& reason, const sstring& path) const {\n        if (long(result) == -1) {\n            throw_fs_exception(reason, fs::path(path));\n        }\n    }\n\n    void throw_fs_exception_if_error(const sstring& reason, const sstring& path1, const sstring& path2) const {\n        if (long(result) == -1) {\n            throw_fs_exception(reason, fs::path(path1), fs::path(path2));\n        }\n    }\n\n    std::error_code ec() const {\n        return std::error_code(error, std::system_category());\n    }\n};\n\n// Wrapper for a system call result containing the return value,\n// an output parameter that was returned from the syscall, and errno.\ntemplate <typename Extra>\nstruct syscall_result_extra : public syscall_result<int> {\n    Extra extra;\n    syscall_result_extra(int result, int error, Extra e) : syscall_result<int>{result, error}, extra{std::move(e)} {\n    }\n};\n\ntemplate <typename T>\nsyscall_result<T>\nwrap_syscall(T result) {\n    return syscall_result<T>{std::move(result), errno};\n}\n\ntemplate <typename Extra>\nsyscall_result_extra<Extra>\nwrap_syscall(int result, const Extra& extra) {\n    return syscall_result_extra<Extra>{result, errno, extra};\n}\n\n}\n"
  },
  {
    "path": "src/core/syscall_work_queue.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n#pragma once\n\n#include <seastar/core/internal/pollable_fd.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <boost/lockfree/spsc_queue.hpp>\n\nnamespace seastar {\n\nclass syscall_work_queue {\n    static constexpr size_t queue_length = 128;\n    struct work_item;\n    using lf_queue = boost::lockfree::spsc_queue<work_item*,\n                            boost::lockfree::capacity<queue_length>>;\n    lf_queue _pending;\n    lf_queue _completed;\n    writeable_eventfd _start_eventfd;\n    semaphore _queue_has_room = { queue_length };\n    struct work_item {\n        virtual ~work_item() {}\n        virtual void process() = 0;\n        virtual void complete() = 0;\n        virtual void set_exception(std::exception_ptr) = 0;\n    };\n    template <typename T>\n    struct work_item_returning :  work_item {\n        noncopyable_function<T ()> _func;\n        promise<T> _promise;\n        std::optional<T> _result;\n        work_item_returning(noncopyable_function<T ()> func) : _func(std::move(func)) {}\n        virtual void process() override { _result = this->_func(); }\n        virtual void complete() override { _promise.set_value(std::move(*_result)); }\n        virtual void set_exception(std::exception_ptr eptr) override { _promise.set_exception(eptr); };\n        future<T> get_future() { return _promise.get_future(); }\n    };\npublic:\n    syscall_work_queue();\n    template <typename T>\n    future<T> submit(noncopyable_function<T ()> func) noexcept {\n      try {\n        auto wi = std::make_unique<work_item_returning<T>>(std::move(func));\n        auto fut = wi->get_future();\n        submit_item(std::move(wi));\n        return fut;\n      } catch (...) {\n        return current_exception_as_future<T>();\n      }\n    }\nprivate:\n    void work();\n    // Scans the _completed queue, that contains the requests already handled by the syscall thread,\n    // effectively opening up space for more requests to be submitted. One consequence of this is\n    // that from the reactor's point of view, a request is not considered handled until it is\n    // removed from the _completed queue.\n    //\n    // Returns the number of requests handled.\n    unsigned complete();\n    void submit_item(std::unique_ptr<syscall_work_queue::work_item> wi);\n\n    friend class thread_pool;\n};\n\n}\n"
  },
  {
    "path": "src/core/systemwide_memory_barrier.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Scylla DB\n */\n\n\n#include <sys/mman.h>\n#include <unistd.h>\n#include <atomic>\n#include <mutex>\n\n#if SEASTAR_HAS_MEMBARRIER\n#include <linux/membarrier.h>\n#include <sys/syscall.h>\n#include <unistd.h>\n#endif\n\n#include <seastar/core/internal/systemwide_memory_barrier.hh>\n#include <seastar/core/cacheline.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\nnamespace internal {\n\n\n#ifdef SEASTAR_HAS_MEMBARRIER\n\nstatic bool has_native_membarrier = [] {\n    auto r = syscall(SYS_membarrier, MEMBARRIER_CMD_QUERY, 0);\n    if (r == -1) {\n        return false;\n    }\n    int needed = MEMBARRIER_CMD_PRIVATE_EXPEDITED | MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED;\n    if ((r & needed) != needed) {\n        return false;\n    }\n    syscall(SYS_membarrier, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED, 0);\n    return true;\n}();\n\nstatic bool try_native_membarrier() {\n    if (has_native_membarrier) {\n        syscall(SYS_membarrier, MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0);\n        return true;\n    }\n    return false;\n}\n\n#else\n\nstatic bool try_native_membarrier() {\n    return false;\n}\n\n#endif\n\n// cause all threads to invoke a full memory barrier\nvoid\nsystemwide_memory_barrier() {\n    if (try_native_membarrier()) {\n        return;\n    }\n\n    // FIXME: use sys_membarrier() when available\n    static thread_local char* mem = [] {\n       void* mem = mmap(nullptr, getpagesize(),\n               PROT_READ | PROT_WRITE,\n               MAP_PRIVATE | MAP_ANONYMOUS,\n               -1, 0) ;\n       SEASTAR_ASSERT(mem != MAP_FAILED);\n\n       // If the user specified --lock-memory, then madvise() below will fail\n       // with EINVAL, so we unlock here:\n       auto r = munlock(mem, getpagesize());\n       // munlock may fail on old kernels if we don't have permission. That's not\n       // a problem, since if we don't have permission to unlock, we didn't have\n       // permissions to lock.\n       SEASTAR_ASSERT(r == 0 || errno == EPERM);\n\n       return reinterpret_cast<char*>(mem);\n    }();\n    // Force page into memory to make madvise() have real work to do\n    *mem = 3;\n    // Evict page to force kernel to send IPI to all threads, with\n    // a side effect of executing a memory barrier on those threads\n    // FIXME: does this work on ARM?\n    int r2 = madvise(mem, getpagesize(), MADV_DONTNEED);\n    SEASTAR_ASSERT(r2 == 0);\n}\n\nstruct alignas(cache_line_size) aligned_flag {\n    std::atomic<bool> flag;\n    bool try_lock() noexcept {\n        return !flag.exchange(true, std::memory_order_relaxed);\n    }\n    void unlock() noexcept {\n        flag.store(false, std::memory_order_relaxed);\n    }\n};\nstatic aligned_flag membarrier_lock;\n\nbool try_systemwide_memory_barrier() {\n    // In 944d5fe50f3f, Linux started serializing membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED)\n    // calls. This means that if all reactors enter sleep mode at the same time, they will sleep\n    // on a kernel mutex while doing so. While they wait on the mutex, they cannot be woken.\n    //\n    // To fix this, only we serialize membarrier calls ourselves, but instead of sleeping, we just\n    // return to the reactor poll loop. If an event is ready, we will wake up immediately.\n  if (auto lck = std::unique_lock(membarrier_lock, std::try_to_lock)) {\n\n    if (try_native_membarrier()) {\n        return true;\n    }\n\n#ifdef __aarch64__\n\n    // Some (not all) ARM processors can broadcast TLB invalidations using the\n    // TLBI instruction. On those, the mprotect trick won't work.\n    static std::once_flag warn_once;\n    std::call_once(warn_once, [] {\n        seastar_logger.warn(\"membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED) is not available, reactor will not sleep when idle. Upgrade to Linux 4.14 or later\");\n    });\n\n    return false;\n\n#endif\n\n    systemwide_memory_barrier();\n    return true;\n  }\n  return false; // couldn't get the lock\n}\n\n}\n}\n\n"
  },
  {
    "path": "src/core/thread.cc",
    "content": "// If _FORTIFY_SOURCE is defined then longjmp ends up using longjmp_chk\n// which asserts that you're jumping to the same stack. However, here we\n// are intentionally switching stacks when longjmp'ing, so undefine this\n// option to always use normal longjmp.\n#undef _FORTIFY_SOURCE\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <ucontext.h>\n#ifndef SEASTAR_ASAN_ENABLED\n#include <setjmp.h>\n#endif\n#include <stdint.h>\n#include <valgrind/valgrind.h>\n#include <exception>\n#include <utility>\n#include <boost/intrusive/list.hpp>\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/internal/current_task.hh>\n#include <seastar/util/assert.hh>\n\n/// \\cond internal\n\nnamespace seastar {\n\nthread_local jmp_buf_link g_unthreaded_context;\nthread_local jmp_buf_link* g_current_context;\n\n#ifdef SEASTAR_ASAN_ENABLED\n\nnamespace {\n\n#ifdef SEASTAR_HAVE_ASAN_FIBER_SUPPORT\n// ASan provides two functions as a means of informing it that user context\n// switch has happened. First __sanitizer_start_switch_fiber() needs to be\n// called with a place to store the fake stack pointer and the new stack\n// information as arguments. Then, ucontext switch may be performed after which\n// __sanitizer_finish_switch_fiber() needs to be called with a pointer to the\n// current context fake stack and a place to store stack information of the\n// previous ucontext.\n\nextern \"C\" {\nvoid __sanitizer_start_switch_fiber(void** fake_stack_save, const void* stack_bottom, size_t stack_size);\nvoid __sanitizer_finish_switch_fiber(void* fake_stack_save, const void** stack_bottom_old, size_t* stack_size_old);\n}\n#else\nstatic inline void __sanitizer_start_switch_fiber(...) { }\nstatic inline void __sanitizer_finish_switch_fiber(...) { }\n#endif\n\nthread_local jmp_buf_link* g_previous_context;\n\n}\n\nvoid jmp_buf_link::initial_switch_in(ucontext_t* initial_context, const void* stack_bottom, size_t stack_size)\n{\n    auto prev = std::exchange(g_current_context, this);\n    link = prev;\n    g_previous_context = prev;\n    __sanitizer_start_switch_fiber(&prev->fake_stack, stack_bottom, stack_size);\n    swapcontext(&prev->context, initial_context);\n    __sanitizer_finish_switch_fiber(g_current_context->fake_stack, &g_previous_context->stack_bottom,\n                                    &g_previous_context->stack_size);\n}\n\nvoid jmp_buf_link::switch_in()\n{\n    auto prev = std::exchange(g_current_context, this);\n    link = prev;\n    g_previous_context = prev;\n    __sanitizer_start_switch_fiber(&prev->fake_stack, stack_bottom, stack_size);\n    swapcontext(&prev->context, &context);\n    __sanitizer_finish_switch_fiber(g_current_context->fake_stack, &g_previous_context->stack_bottom,\n                                    &g_previous_context->stack_size);\n}\n\nvoid jmp_buf_link::switch_out()\n{\n    g_current_context = link;\n    g_previous_context = this;\n    __sanitizer_start_switch_fiber(&fake_stack, g_current_context->stack_bottom,\n                                   g_current_context->stack_size);\n    swapcontext(&context, &g_current_context->context);\n    __sanitizer_finish_switch_fiber(g_current_context->fake_stack, &g_previous_context->stack_bottom,\n                                    &g_previous_context->stack_size);\n}\n\nvoid jmp_buf_link::initial_switch_in_completed()\n{\n    // This is a new thread and it doesn't have the fake stack yet. ASan will\n    // create it lazily, for now just pass nullptr.\n    __sanitizer_finish_switch_fiber(nullptr, &g_previous_context->stack_bottom, &g_previous_context->stack_size);\n}\n\nvoid jmp_buf_link::final_switch_out()\n{\n    g_current_context = link;\n    g_previous_context = this;\n    // Since the thread is about to die we pass nullptr as fake_stack_save argument\n    // so that ASan knows it can destroy the fake stack if it exists.\n    __sanitizer_start_switch_fiber(nullptr, g_current_context->stack_bottom, g_current_context->stack_size);\n    setcontext(&g_current_context->context);\n}\n\n#else\n\ninline void jmp_buf_link::initial_switch_in(ucontext_t* initial_context, const void*, size_t)\n{\n    auto prev = std::exchange(g_current_context, this);\n    link = prev;\n    if (setjmp(prev->jmpbuf) == 0) {\n        setcontext(initial_context);\n    }\n}\n\ninline void jmp_buf_link::switch_in()\n{\n    auto prev = std::exchange(g_current_context, this);\n    link = prev;\n    if (setjmp(prev->jmpbuf) == 0) {\n        longjmp(jmpbuf, 1);\n    }\n}\n\ninline void jmp_buf_link::switch_out()\n{\n    g_current_context = link;\n    if (setjmp(jmpbuf) == 0) {\n        longjmp(g_current_context->jmpbuf, 1);\n    }\n}\n\ninline void jmp_buf_link::initial_switch_in_completed()\n{\n}\n\ninline void jmp_buf_link::final_switch_out()\n{\n    g_current_context = link;\n    longjmp(g_current_context->jmpbuf, 1);\n}\n\n#endif\n\n// Both asan and optimizations can increase the stack used by a\n// function. When both are used, we need more than 128 KiB.\n#if defined(SEASTAR_ASAN_ENABLED)\nstatic constexpr size_t base_stack_size = 256 * 1024;\n#else\nstatic constexpr size_t base_stack_size = 128 * 1024;\n#endif\n\nstatic size_t get_stack_size(thread_attributes attr) {\n#if defined(__OPTIMIZE__) && defined(SEASTAR_ASAN_ENABLED)\n    return std::max(base_stack_size, attr.stack_size);\n#else\n    return attr.stack_size ? attr.stack_size : base_stack_size;\n#endif\n}\n\nthread_context::thread_context(thread_attributes attr, noncopyable_function<void ()> func)\n        : task(attr.sched_group.value_or(current_scheduling_group()))\n        , _stack(make_stack(get_stack_size(attr)))\n        , _func(std::move(func)) {\n    setup(get_stack_size(attr));\n    _all_threads.push_front(*this);\n}\n\nthread_context::~thread_context() {\n#ifdef SEASTAR_THREAD_STACK_GUARDS\n    auto mp_result = mprotect(_stack.get(), getpagesize(), PROT_READ | PROT_WRITE);\n    SEASTAR_ASSERT(mp_result == 0);\n#endif\n    _all_threads.erase(_all_threads.iterator_to(*this));\n}\n\nthread_context::stack_deleter::stack_deleter(int valgrind_id) : valgrind_id(valgrind_id) {}\n\nthread_context::stack_holder\nthread_context::make_stack(size_t stack_size) {\n#ifdef SEASTAR_THREAD_STACK_GUARDS\n    size_t page_size = getpagesize();\n    size_t alignment = page_size;\n#else\n    size_t alignment = 16; // ABI requirement on x86_64\n#endif\n    void* mem = ::aligned_alloc(alignment, stack_size);\n    if (mem == nullptr) {\n        throw std::bad_alloc();\n    }\n    int valgrind_id = VALGRIND_STACK_REGISTER(mem, reinterpret_cast<char*>(mem) + stack_size);\n    auto stack = stack_holder(new (mem) char[stack_size], stack_deleter(valgrind_id));\n#ifdef SEASTAR_ASAN_ENABLED\n    // Avoid ASAN false positive due to garbage on stack\n    std::memset(stack.get(), 0, stack_size);\n#endif\n\n#ifdef SEASTAR_THREAD_STACK_GUARDS\n    auto mp_status = mprotect(stack.get(), page_size, PROT_READ);\n    throw_system_error_on(mp_status != 0, \"mprotect\");\n#endif\n\n    return stack;\n}\n\nvoid thread_context::stack_deleter::operator()(char* ptr) const noexcept {\n    VALGRIND_STACK_DEREGISTER(valgrind_id);\n    free(ptr);\n}\n\nvoid\nthread_context::setup(size_t stack_size) {\n    // use setcontext() for the initial jump, as it allows us\n    // to set up a stack, but continue with longjmp() as it's\n    // much faster.\n    ucontext_t initial_context;\n    auto q = uint64_t(reinterpret_cast<uintptr_t>(this));\n    auto main = reinterpret_cast<void (*)()>(&thread_context::s_main);\n    auto r = getcontext(&initial_context);\n    throw_system_error_on(r == -1);\n    initial_context.uc_stack.ss_sp = _stack.get();\n    initial_context.uc_stack.ss_size = stack_size;\n    initial_context.uc_link = nullptr;\n    makecontext(&initial_context, main, 2, int(q), int(q >> 32));\n    _context.thread = this;\n    _context.initial_switch_in(&initial_context, _stack.get(), stack_size);\n}\n\nvoid\nthread_context::switch_in() {\n    internal::set_current_task(nullptr); // thread_wake_task is on the stack and will be invalid when we resume\n    _context.switch_in();\n}\n\nvoid\nthread_context::switch_out() {\n    _context.switch_out();\n}\n\nbool\nthread_context::should_yield() const {\n    return need_preempt();\n}\n\nthread_local thread_context::all_thread_list thread_context::_all_threads;\n\nvoid\nthread_context::run_and_dispose() noexcept {\n    switch_in();\n}\n\nvoid\nthread_context::yield() {\n    schedule(this);\n    switch_out();\n}\n\nvoid\nthread_context::reschedule() {\n    schedule(this);\n}\n\nvoid\nthread_context::s_main(int lo, int hi) {\n    uintptr_t q = uint64_t(uint32_t(lo)) | uint64_t(hi) << 32;\n    reinterpret_cast<thread_context*>(q)->main();\n}\n\nvoid\nthread_context::main() {\n#ifdef __x86_64__\n    // There is no caller of main() in this context. We need to annotate this frame like this so that\n    // unwinders don't try to trace back past this frame.\n    // See https://github.com/scylladb/scylla/issues/1909.\n    asm(\".cfi_undefined rip\");\n#elif defined(__PPC__)\n    asm(\".cfi_undefined lr\");\n#elif defined(__aarch64__)\n    asm(\".cfi_undefined x30\");\n#elif defined(__s390x__)\n    asm(\".cfi_undefined %r14\");\n#else\n    #warning \"Backtracing from seastar threads may be broken\"\n#endif\n    _context.initial_switch_in_completed();\n    if (group() != current_scheduling_group()) {\n        yield();\n    }\n    try {\n        _func();\n        _done.set_value();\n    } catch (...) {\n        _done.set_exception(std::current_exception());\n    }\n\n    _context.final_switch_out();\n}\n\nnamespace thread_impl {\n\nvoid yield() {\n    g_current_context->thread->yield();\n}\n\nvoid switch_in(thread_context* to) {\n    to->switch_in();\n}\n\nvoid switch_out(thread_context* from) {\n    from->switch_out();\n}\n\nvoid init() {\n    g_unthreaded_context.link = nullptr;\n    g_unthreaded_context.thread = nullptr;\n    g_current_context = &g_unthreaded_context;\n}\n\n}\n\nvoid thread::yield() {\n    thread_impl::get()->yield();\n}\n\n}\n\n/// \\endcond\n"
  },
  {
    "path": "src/core/thread_pool.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n\n\n#include <atomic>\n#include <cstdint>\n#include <array>\n#include <pthread.h>\n#include <signal.h>\n\n#include \"core/thread_pool.hh\"\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nthread_pool::thread_pool(sstring name, file_desc& notify) : _notify_eventfd(notify), _worker_thread([this, name] { work(name); }) {\n}\n\nvoid thread_pool::work(sstring name) {\n    pthread_setname_np(pthread_self(), name.c_str());\n    sigset_t mask;\n    sigfillset(&mask);\n    auto r = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n    throw_pthread_error(r);\n    std::array<syscall_work_queue::work_item*, syscall_work_queue::queue_length> tmp_buf;\n    while (true) {\n        uint64_t count;\n        auto r = ::read(inter_thread_wq._start_eventfd.get_read_fd(), &count, sizeof(count));\n        SEASTAR_ASSERT(r == sizeof(count));\n        if (_stopped.load(std::memory_order_relaxed)) {\n            break;\n        }\n        auto end = tmp_buf.data();\n        inter_thread_wq._pending.consume_all([&] (syscall_work_queue::work_item* wi) {\n            *end++ = wi;\n        });\n        for (auto p = tmp_buf.data(); p != end; ++p) {\n            auto wi = *p;\n            wi->process();\n            inter_thread_wq._completed.push(wi);\n\n            // Prevent the following load of _main_thread_idle to be hoisted before the writes to _completed above.\n            std::atomic_thread_fence(std::memory_order_seq_cst);\n            if (_main_thread_idle.load(std::memory_order_relaxed)) {\n                uint64_t one = 1;\n                auto res = ::write(_notify_eventfd.get(), &one, 8);\n                SEASTAR_ASSERT(res == 8 && \"write(2) failed on _reactor._notify_eventfd\");\n            }\n        }\n    }\n}\n\nthread_pool::~thread_pool() {\n    _stopped.store(true, std::memory_order_relaxed);\n    inter_thread_wq._start_eventfd.signal(1);\n    _worker_thread.join();\n}\n\n}\n"
  },
  {
    "path": "src/core/thread_pool.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include \"syscall_work_queue.hh\"\n\nnamespace seastar {\n\nclass file_desc;\n\nnamespace internal {\n// Reasons for why a function had to be submitted to the thread_pool\nenum class thread_pool_submit_reason : size_t {\n    // Used for aio operations what would block in `io_submit`.\n    aio_fallback,\n    // Used for file operations that don't have non-blocking alternatives.\n    file_operation,\n    // Used for process operations that don't have non-blocking alternatives.\n    process_operation,\n};\n\nclass submit_metrics {\n    uint64_t _counters[static_cast<size_t>(thread_pool_submit_reason::process_operation) + 1]{};\n\npublic:\n    void record_reason(thread_pool_submit_reason reason) {\n        ++_counters[static_cast<size_t>(reason)];\n    }\n\n    uint64_t count_for(thread_pool_submit_reason reason) const {\n        return _counters[static_cast<size_t>(reason)];\n    }\n};\n} // namespace internal\n\nclass thread_pool {\n    file_desc& _notify_eventfd;\n    internal::submit_metrics metrics;\n    syscall_work_queue inter_thread_wq;\n    posix_thread _worker_thread;\n    std::atomic<bool> _stopped = { false };\n    std::atomic<bool> _main_thread_idle = { false };\npublic:\n    explicit thread_pool(sstring thread_name, file_desc& notify);\n    ~thread_pool();\n    template <typename T, typename Func>\n    future<T> submit(internal::thread_pool_submit_reason reason, Func func) noexcept {\n        metrics.record_reason(reason);\n        return inter_thread_wq.submit<T>(std::move(func));\n    }\n    uint64_t count(internal::thread_pool_submit_reason r) const { return metrics.count_for(r); }\n\n    unsigned complete() { return inter_thread_wq.complete(); }\n    // Before we enter interrupt mode, we must make sure that the syscall thread will properly\n    // generate signals to wake us up. This means we need to make sure that all modifications to\n    // the pending and completed fields in the inter_thread_wq are visible to all threads.\n    //\n    // Simple release-acquire won't do because we also need to serialize all writes that happens\n    // before the syscall thread loads this value, so we'll need full seq_cst.\n    void enter_interrupt_mode() { _main_thread_idle.store(true, std::memory_order_seq_cst); }\n    // When we exit interrupt mode, however, we can safely used relaxed order. If any reordering\n    // takes place, we'll get an extra signal and complete will be called one extra time, which is\n    // harmless.\n    void exit_interrupt_mode() { _main_thread_idle.store(false, std::memory_order_relaxed); }\n\nprivate:\n    void work(sstring thread_name);\n};\n\n\n}\n"
  },
  {
    "path": "src/core/uname.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n\n#include <memory>\n#include <optional>\n#include <regex>\n#include <boost/algorithm/cxx11/any_of.hpp>\n#include <sys/utsname.h>\n#include <iostream>\n\n#include <seastar/core/internal/uname.hh>\n\nnamespace seastar {\n\nnamespace internal {\n\nint uname_t::component_count() const {\n    if (distro_patch) {\n        return 5;\n    }\n    if (subsublevel) {\n        return 4;\n    }\n    if (sublevel) {\n        return 3;\n    }\n    return 2;\n}\n\nbool uname_t::has_distro_extra(std::string extra) const {\n    return distro_extra.find(extra) != std::string::npos;\n}\n\n// Can't use optional compares, C++17 only\nstatic int cmp(const std::optional<int>& u1, const std::optional<int>& u2) {\n    return int(u1.value_or(0) - u2.value_or(0));\n}\n\nbool uname_t::same_as_or_descendant_of(const uname_t& x) const {\n    if (version < x.version) {\n        return false; // 4.2 vs. 5.1\n    }\n    if (version == x.version && patchlevel < x.patchlevel) {\n        return false; // 4.0 vs 4.1\n    }\n    if (!has_distro_extra(x.distro_extra)) {\n        return false;\n    }\n    switch (x.component_count()) {\n    case 5:\n        return version == x.version\n                && patchlevel == x.patchlevel\n                && cmp(sublevel, x.sublevel) == 0\n                && cmp(subsublevel, x.subsublevel) == 0\n                && cmp(distro_patch, x.distro_patch) >= 0;\n    case 4:\n        return version == x.version\n                && patchlevel == x.patchlevel\n                && cmp(sublevel, x.sublevel) == 0\n                && cmp(subsublevel, x.subsublevel) >= 0;\n    case 3:\n        return version == x.version\n                && patchlevel == x.patchlevel\n                && cmp(sublevel, x.sublevel) >= 0;\n    case 2:\n        return true;\n    default:\n        return false;\n    }\n}\n\nuname_t parse_uname(const char* u) {\n    static std::regex re(R\"XX((\\d+)\\.(\\d+)(?:\\.(\\d+)(?:\\.(\\d+))?)?(?:-(\\d*)(.+))?)XX\");\n    std::cmatch m;\n    if (std::regex_match(u, m, re)) {\n        auto num = [] (std::csub_match sm) -> std::optional<int> {\n            if (sm.length() > 0) {\n                return std::atoi(sm.str().c_str());\n            } else {\n                return std::nullopt;\n            }\n        };\n        return uname_t{*num(m[1]), *num(m[2]), num(m[3]), num(m[4]), num(m[5]), m[6].str()};\n    } else {\n        return uname_t{0, 0, std::nullopt, std::nullopt, std::nullopt, \"\"};\n    }\n}\n\n\nbool uname_t::whitelisted(std::initializer_list<const char*> wl) const {\n    return boost::algorithm::any_of(wl, [this] (const char* v) {\n        return same_as_or_descendant_of(parse_uname(v));\n    });\n}\n\nstd::ostream& operator<<(std::ostream& os, const uname_t& u) {\n    os << u.version << \".\" << u.patchlevel;\n    if (u.sublevel) {\n        os << \".\" << *u.sublevel;\n    }\n    if (u.subsublevel) {\n        os << \".\" << *u.subsublevel;\n    }\n    if (u.distro_patch || !u.distro_extra.empty()) {\n        os << \"-\";\n    }\n    if (u.distro_patch) {\n        os << *u.distro_patch;\n    }\n    os << u.distro_extra;\n    return os;\n}\n\n\nuname_t kernel_uname() {\n    struct ::utsname buf;\n    ::uname(&buf);\n    return parse_uname(buf.release);\n}\n\n}\n}\n"
  },
  {
    "path": "src/core/vla.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/util/assert.hh>\n#include <memory>\n\nnamespace seastar {\n\n// Some C APIs have a structure with a variable length array at the end.\n// This is a helper function to help allocate it.\n//\n// for a structure\n//\n//   struct xx { int a; float b[0]; };\n//\n// use\n//\n//   make_struct_with_vla(&xx::b, number_of_bs);\n//\n// to allocate it.\n//\ntemplate <class S, typename E>\ninline\nstd::unique_ptr<S, free_deleter>\nmake_struct_with_vla(E S::*last, size_t nr) {\n    auto fake = reinterpret_cast<S*>(0);\n    size_t offset = reinterpret_cast<uintptr_t>(&(fake->*last));\n    size_t element_size = sizeof((fake->*last)[0]);\n    SEASTAR_ASSERT(offset == sizeof(S));\n    auto p = std::unique_ptr<char, free_deleter>(\n            reinterpret_cast<char*>(::malloc(offset + element_size * nr)));\n    auto s = std::unique_ptr<S, free_deleter>(new (p.get()) S());\n    p.release();\n    return s;\n}\n\n}\n"
  },
  {
    "path": "src/http/api_docs.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#include <seastar/http/api_docs.hh>\n#include <seastar/http/handlers.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/http/transformers.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/http/transformers.hh>\n#include <seastar/core/loop.hh>\n\nusing namespace std;\n\nnamespace seastar {\n\nnamespace httpd {\n\nconst sstring api_registry_builder_base::DEFAULT_PATH = \"/api-doc\";\nconst sstring api_registry_builder_base::DEFAULT_DIR = \".\";\n\ndoc_entry get_file_reader(sstring file_name) {\n    return [file_name] (output_stream<char>& os) {\n        return open_file_dma(file_name, open_flags::ro).then([&os] (file f) mutable {\n            return do_with(input_stream<char>(make_file_input_stream(std::move(f))), [&os](input_stream<char>& is) {\n                return copy(is, os).then([&is] {\n                    return is.close();\n                });\n            });\n        });\n    };\n}\n\nfuture<> api_docs_20::write(output_stream<char>&& os, std::unique_ptr<http::request> req) {\n    return do_with(output_stream<char>(_transform.transform(std::move(req), \"\", std::move(os))), [this] (output_stream<char>& os) {\n        return do_for_each(_apis, [&os](doc_entry& api) {\n            return api(os);\n        }).then([&os] {\n            return os.write(\"},\\\"definitions\\\": {\");\n        }).then([this, &os] {\n            return do_for_each(_definitions, [&os](doc_entry& api) {\n                return api(os);\n            });\n        }).then([&os] {\n            return os.write(\"}}\");\n        }).then([&os] {\n            return os.flush();\n        }).finally([&os] {\n            return os.close();\n        });\n    });\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/chunk_parsers.rl",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/ragel.hh>\n#include <seastar/core/sstring.hh>\n#include <unordered_map>\n\nnamespace seastar {\n\n%% machine chunk_head;\n\n%%{\n\naccess _fsm_;\n\naction mark {\n    g.mark_start(p);\n}\n\naction store_name {\n    _name = str();\n}\n\naction store_value {\n    _value = str();\n}\n\naction store_size {\n    _size = str();\n}\n\naction assign_value {\n    _extensions[_name] = std::move(_value);\n}\n\naction done {\n    done = true;\n    fbreak;\n}\n\ncrlf  = '\\r\\n';\nsp    = ' ';\nht    = '\\t';\nsp_ht = sp | ht;\ntchar = alpha | digit | '-' | '!' | '#' | '$' | '%' | '&' | '\\'' | '*'\n        | '+' | '.' | '^' | '_' | '`' | '|' | '~';\n\n# names correspond to the ones in RFC 7230 (with hyphens instead of underscores)\nobs_text = 0x80..0xFF;\nqdtext = any - (cntrl | '\"' | '\\\\');\nquoted_pair = ('\\\\' >{ g.mark_end(p); }) ((any - cntrl) >mark) ;\ntoken = tchar+;\nquoted_string = '\"' ((qdtext | quoted_pair)* >mark %store_value) '\"';\nchunk_ext_name = token >mark %store_name;\nchunk_ext_val = (token >mark %store_value) | quoted_string ;\nchunk_size = xdigit+ >mark %store_size;\nchunk_ext = (';' chunk_ext_name ('=' chunk_ext_val)? %assign_value)*;\n\nmain := chunk_size chunk_ext crlf @done;\n\n}%%\n\nclass http_chunk_size_and_ext_parser : public ragel_parser_base<http_chunk_size_and_ext_parser> {\n    %% write data nofinal noprefix;\npublic:\n    enum class state {\n        error,\n        eof,\n        done,\n    };\n    std::unordered_map<sstring, sstring> _extensions;\n    sstring _name;\n    sstring _value;\n    sstring _size;\n    state _state;\npublic:\n    void init() {\n        init_base();\n        _extensions = std::unordered_map<sstring, sstring>();\n        _value = \"\";\n        _size = \"\";\n        _state = state::eof;\n        %% write init;\n    }\n    char* parse(char* p, char* pe, char* eof) {\n        sstring_builder::guard g(_builder, p, pe);\n        [[maybe_unused]] auto str = [this, &g, &p] { g.mark_end(p); return get_str(); };\n        bool done = false;\n        if (p != pe) {\n            _state = state::error;\n        }\n#ifdef __clang__\n#pragma clang diagnostic push\n#pragma clang diagnostic ignored \"-Wmisleading-indentation\"\n#endif\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wimplicit-fallthrough\"\n        %% write exec;\n#pragma GCC diagnostic pop\n#ifdef __clang__\n#pragma clang diagnostic pop\n#endif\n        if (!done) {\n            if (p == eof) {\n                _state = state::eof;\n            } else if (p != pe) {\n                _state = state::error;\n            } else {\n                p = nullptr;\n            }\n        } else {\n            _state = state::done;\n        }\n        return p;\n    }\n    auto get_parsed_extensions() {\n        return std::move(_extensions);\n    }\n    auto get_size() {\n        return std::move(_size);\n    }\n    bool eof() const {\n        return _state == state::eof;\n    }\n    bool failed() const {\n        return _state == state::error;\n    }\n};\n\n%% machine chunk_tail;\n// the headers in the chunk trailer are parsed the same way as in the request_parser.rl\n%%{\n\naccess _fsm_;\n\naction mark {\n    g.mark_start(p);\n}\n\naction store_field_name {\n    _field_name = str();\n}\n\naction trim_trailing_whitespace_and_store_value {\n    _value = str();\n    trim_trailing_spaces_and_tabs(_value);\n    g.mark_start(nullptr);\n}\n\naction assign_field {\n    if (_headers.count(_field_name)) {\n        _headers[_field_name] += sstring(\",\") + std::move(_value);\n    } else {\n        _headers[_field_name] = std::move(_value);\n    }\n}\n\naction extend_field  {\n    _headers[_field_name] += sstring(\" \") + std::move(_value);\n}\n\naction done {\n    done = true;\n    fbreak;\n}\n\ncrlf = '\\r\\n';\ntchar = alpha | digit | '-' | '!' | '#' | '$' | '%' | '&' | '\\'' | '*'\n        | '+' | '.' | '^' | '_' | '`' | '|' | '~';\nsp = ' ';\nht = '\\t';\nsp_ht = sp | ht;\n\nobs_text = 0x80..0xFF; # defined in RFC 7230, Section 3.2.6.\nfield_vchar = (graph | obs_text);\n# RFC 9110, Section 5.5 allows single ' '/'\\t' separators between\n# field_vchar words. We are less strict and allow any number of spaces\n# between words.\n# Trailing spaces are trimmed in postprocessing.\nfield_content = (field_vchar | sp_ht)*;\n\nfield = tchar+ >mark %store_field_name;\nvalue = field_content >mark %trim_trailing_whitespace_and_store_value;\nheader_1st = field ':' sp_ht* <: value crlf %assign_field;\nheader_cont = (sp_ht+ <: value crlf) %extend_field;\nheader = header_1st header_cont*;\nmain := header* crlf @done;\n\n}%%\n\nclass http_chunk_trailer_parser : public ragel_parser_base<http_chunk_trailer_parser> {\n    %% write data nofinal noprefix;\npublic:\n    enum class state {\n        error,\n        eof,\n        done,\n    };\n    std::unordered_map<sstring, sstring> _headers;\n    sstring _field_name;\n    sstring _value;\n    state _state;\npublic:\n    void init() {\n        init_base();\n        _headers = std::unordered_map<sstring, sstring>();\n        _field_name = \"\";\n        _value = \"\";\n        _state = state::eof;\n        %% write init;\n    }\n    char* parse(char* p, char* pe, char* eof) {\n        sstring_builder::guard g(_builder, p, pe);\n        [[maybe_unused]] auto str = [this, &g, &p] { g.mark_end(p); return get_str(); };\n        bool done = false;\n        if (p != pe) {\n            _state = state::error;\n        }\n#ifdef __clang__\n#pragma clang diagnostic push\n#pragma clang diagnostic ignored \"-Wmisleading-indentation\"\n#endif\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wimplicit-fallthrough\"\n        %% write exec;\n#pragma GCC diagnostic pop\n#ifdef __clang__\n#pragma clang diagnostic pop\n#endif\n        if (!done) {\n            if (p == eof) {\n                _state = state::eof;\n            } else if (p != pe) {\n                _state = state::error;\n            } else {\n                p = nullptr;\n            }\n        } else {\n            _state = state::done;\n        }\n        return p;\n    }\n    auto get_parsed_headers() {\n        return std::move(_headers);\n    }\n    bool eof() const {\n        return _state == state::eof;\n    }\n    bool failed() const {\n        return _state == state::error;\n    }\n};\n\n}\n"
  },
  {
    "path": "src/http/client.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Scylladb, Ltd.\n */\n\n\n#include <cassert>\n#include <concepts>\n#include <gnutls/gnutls.h>\n#include <memory>\n#include <optional>\n#include <stdexcept>\n#include <utility>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/http/client.hh>\n#include <seastar/http/request.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/http/response_parser.hh>\n#include <seastar/http/internal/content_source.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/short_streams.hh>\n#include <seastar/util/string_utils.hh>\n\nnamespace seastar {\nlogger http_log(\"http\");\nnamespace http {\nnamespace internal {\n\nclient_ref::client_ref(http::experimental::client* c) noexcept : _c(c) {\n    _c->_nr_connections++;\n}\n\nclient_ref::~client_ref() {\n    if (_c != nullptr) {\n        _c->_nr_connections--;\n        _c->_wait_con.broadcast();\n    }\n}\n\n}\n\nnamespace experimental {\n\nconnection::connection(connected_socket&& fd, internal::client_ref cr)\n        : _fd(std::move(fd))\n        , _read_buf(_fd.input())\n        , _write_buf(_fd.output())\n        , _closed(_fd.wait_input_shutdown().finally([me = shared_from_this()] {}))\n        , _ref(std::move(cr))\n{\n}\n\nfuture<> connection::write_body(const request& req) {\n    if (req.body_writer) {\n        if (req.content_length != 0) {\n            return req.body_writer(internal::make_http_content_length_output_stream(_write_buf, req.content_length, req._bytes_written)).then([&req] {\n                if (req.content_length == req._bytes_written) {\n                    return make_ready_future<>();\n                } else {\n                    return make_exception_future<>(std::runtime_error(format(\"partial request body write, need {} sent {}\", req.content_length, req._bytes_written)));\n                }\n            });\n        }\n        return req.body_writer(internal::make_http_chunked_output_stream(_write_buf)).then([this] {\n            return _write_buf.write(\"0\\r\\n\\r\\n\");\n        });\n    } else if (auto& c = internal::deprecated_content(req); !c.empty()) {\n        return _write_buf.write(c);\n    } else {\n        return make_ready_future<>();\n    }\n}\n\nfuture<connection::reply_ptr> connection::maybe_wait_for_continue(const request& req) {\n    if (req.get_header(\"Expect\") == \"\") {\n        return make_ready_future<reply_ptr>(nullptr);\n    }\n\n    return _write_buf.flush().then([this] {\n        return recv_reply().then([] (reply_ptr rep) {\n            if (rep->_status == reply::status_type::continue_) {\n                return make_ready_future<reply_ptr>(nullptr);\n            } else {\n                return make_ready_future<reply_ptr>(std::move(rep));\n            }\n        });\n    });\n}\n\nstatic void validate_request(const request& req) {\n    if (req._version.empty()) {\n        throw std::runtime_error(\"HTTP version not set\");\n    }\n    if (req.content_length != 0 && !req.body_writer && internal::deprecated_content(req).empty()) {\n        throw std::runtime_error(\"Request body writer not set and content is empty\");\n    }\n}\n\nfuture<> connection::send_request_head(const request& req) {\n    return _write_buf.write(req.request_line()).then([this, &req] {\n        return req.write_request_headers(_write_buf).then([this] {\n            return _write_buf.write(\"\\r\\n\", 2);\n        });\n    });\n}\n\nfuture<connection::reply_ptr> connection::recv_reply() {\n    http_response_parser parser;\n    return do_with(std::move(parser), [this] (auto& parser) {\n        parser.init();\n        return _read_buf.consume(parser).then([this, &parser] {\n            if (parser.eof()) {\n                http_log.trace(\"Parsing response EOFed\");\n                throw std::system_error(ECONNABORTED, std::system_category());\n            }\n            if (parser.failed()) {\n                http_log.trace(\"Parsing response failed\");\n                throw httpd::response_parsing_exception(format(\"Invalid http server response. Reason: {}\", parser.error_message()));\n            }\n\n            auto resp = parser.get_parsed_response();\n            sstring length_header = resp->get_header(\"Content-Length\");\n            resp->content_length = strtol(length_header.c_str(), nullptr, 10);\n            if ((resp->_version != \"1.1\") || seastar::internal::case_insensitive_cmp()(resp->get_header(\"Connection\"), \"close\")) {\n                _persistent = false;\n            }\n            return make_ready_future<reply_ptr>(std::move(resp));\n        });\n    });\n}\n\nfuture<connection::reply_ptr> connection::do_make_request(const request& req) {\n    return send_request_head(req).then([this, &req] {\n        return maybe_wait_for_continue(req).then([this, &req] (reply_ptr cont) {\n            if (cont) {\n                return make_ready_future<reply_ptr>(std::move(cont));\n            }\n\n            return write_body(req).then([this] {\n                return _write_buf.flush().then([this] {\n                    return recv_reply();\n                });\n            });\n        });\n    });\n}\n\nfuture<reply> connection::make_request(request req) {\n    try {\n        validate_request(req);\n    } catch (...) {\n        return current_exception_as_future<reply>();\n    }\n    return do_with(std::move(req), [this] (auto& req) {\n        return do_make_request(req).then([] (reply_ptr rep) {\n            return make_ready_future<reply>(std::move(*rep));\n        });\n    });\n}\n\ninput_stream<char> connection::in(reply& rep) {\n    if (seastar::internal::case_insensitive_cmp()(rep.get_header(\"Transfer-Encoding\"), \"chunked\")) {\n        return input_stream<char>(data_source(std::make_unique<httpd::internal::chunked_source_impl>(_read_buf, rep.chunk_extensions, rep.trailing_headers, rep.left_content_length)));\n    }\n\n    return input_stream<char>(data_source(std::make_unique<httpd::internal::content_length_source_impl>(_read_buf, rep.content_length, rep.left_content_length)));\n}\n\nvoid connection::shutdown() noexcept {\n    _persistent = false;\n    _fd.shutdown_input();\n}\n\nfuture<> connection::close() {\n    // #2661. At least output stream close can fail with exception, because\n    // the stream will do a flush. If connection never managed to send data, we\n    // will throw again here. Need to suppress these exceptions.\n    return when_all(_read_buf.close().handle_exception([](auto&&){}), _write_buf.close().handle_exception([](auto&&){})).discard_result().then([this] {\n        auto la = _fd.local_address();\n        return std::move(_closed).then([la = std::move(la)] {\n            http_log.trace(\"destroyed connection {}\", la);\n        });\n    });\n}\n\nclient::client(socket_address addr)\n        : client(std::make_unique<basic_connection_factory>(std::move(addr)))\n{\n}\n\nclient::client(socket_address addr, shared_ptr<tls::certificate_credentials> creds, sstring host)\n        : client(std::make_unique<tls_connection_factory>(std::move(addr), std::move(creds), std::move(host)))\n{\n}\n\nstatic std::unique_ptr<retry_strategy> get_strategy(client::retry_requests retry) {\n    if (retry == client::retry_requests::no) {\n        return std::make_unique<no_retry_strategy>();\n    }\n    return std::make_unique<default_retry_strategy>();\n}\n\nclient::client(std::unique_ptr<connection_factory> f, unsigned max_connections, retry_requests retry, size_t max_bytes_to_drain) : client(\n    std::move(f),\n    max_connections,\n    max_bytes_to_drain,\n    get_strategy(retry)) {\n}\n\nclient::client(std::unique_ptr<connection_factory> f, unsigned max_connections, size_t max_bytes_to_drain, std::unique_ptr<retry_strategy>&& retry_strategy)\n        : _new_connections(std::move(f))\n        , _max_connections(max_connections)\n        , _max_bytes_to_drain(max_bytes_to_drain)\n        , _retry_strategy(std::move(retry_strategy))\n{\n    assert(_retry_strategy);\n}\n\nfuture<client::connection_ptr> client::get_connection(abort_source* as) {\n    if (!_pool.empty()) {\n        connection_ptr con = _pool.front().shared_from_this();\n        _pool.pop_front();\n        http_log.trace(\"pop http connection {} from pool\", con->_fd.local_address());\n        return make_ready_future<connection_ptr>(con);\n    }\n\n    if (_nr_connections >= _max_connections) {\n        auto sub = as ? as->subscribe([this] () noexcept { _wait_con.broadcast(); }) : std::nullopt;\n        return _wait_con.wait().then([this, as, sub = std::move(sub)] {\n            if (as != nullptr && as->abort_requested()) {\n                return make_exception_future<client::connection_ptr>(as->abort_requested_exception_ptr());\n            }\n            return get_connection(as);\n        });\n    }\n\n    return make_connection(as);\n}\n\nfuture<client::connection_ptr> client::make_connection(abort_source* as) {\n    _total_new_connections++;\n    return _new_connections->make(as).then([cr = internal::client_ref(this)] (connected_socket cs) mutable {\n        http_log.trace(\"created new http connection {}\", cs.local_address());\n        auto con = seastar::make_shared<connection>(std::move(cs), std::move(cr));\n        return make_ready_future<connection_ptr>(std::move(con));\n    });\n}\n\nfuture<> client::put_connection(connection_ptr con) {\n    if (con->_persistent && (_nr_connections <= _max_connections)) {\n        http_log.trace(\"push http connection {} to pool\", con->_fd.local_address());\n        _pool.push_back(*con);\n        _wait_con.signal();\n        return make_ready_future<>();\n    }\n\n    http_log.trace(\"dropping connection {}\", con->_fd.local_address());\n    return con->close().finally([con] {});\n}\n\nfuture<> client::shrink_connections() {\n    if (_nr_connections <= _max_connections) {\n        return make_ready_future<>();\n    }\n\n    if (!_pool.empty()) {\n        connection_ptr con = _pool.front().shared_from_this();\n        _pool.pop_front();\n        return con->close().finally([this, con] {\n            return shrink_connections();\n        });\n    }\n\n    return _wait_con.wait().then([this] {\n        return shrink_connections();\n    });\n}\n\nfuture<> client::set_maximum_connections(unsigned nr) {\n    if (nr > _max_connections) {\n        _max_connections = nr;\n        _wait_con.broadcast();\n        return make_ready_future<>();\n    }\n\n    _max_connections = nr;\n    return shrink_connections();\n}\n\ntemplate <std::invocable<connection&> Fn>\nauto client::with_connection(Fn&& fn, abort_source* as) {\n    return get_connection(as).then([this, fn = std::move(fn)] (connection_ptr con) mutable {\n        return fn(*con).finally([this, con = std::move(con)] () mutable {\n            return put_connection(std::move(con));\n        });\n    });\n}\n\ntemplate <typename Fn>\nrequires std::invocable<Fn, connection&>\nauto client::with_new_connection(Fn&& fn, abort_source* as) {\n    return make_connection(as).then([this, fn = std::move(fn)] (connection_ptr con) mutable {\n        return fn(*con).finally([this, con = std::move(con)] () mutable {\n            return put_connection(std::move(con));\n        });\n    });\n}\n\nfuture<> client::make_request(request&& req, reply_handler&& handle, std::optional<reply::status_type>&& expected, abort_source* as) {\n    return do_with(std::move(req), std::move(handle), [this, expected, as](const request& req, reply_handler& handle) mutable {\n        return make_request(req, handle, expected, as);\n    });\n}\n\nfuture<> client::make_request(request&& req, reply_handler&& handle, const retry_strategy& strategy, std::optional<reply::status_type>&& expected, abort_source* as) {\n    return do_with(std::move(req), std::move(handle), [this, &strategy, expected, as](const request& req, reply_handler& handle) mutable {\n        return make_request(req, handle, strategy, expected, as);\n    });\n}\n\nfuture<> client::make_request(const request& req, reply_handler& handle, std::optional<reply::status_type> expected, abort_source* as) {\n    return make_request(req, handle, *_retry_strategy, expected, as);\n}\n\nfuture<> client::maybe_retry_request(std::exception_ptr ex,\n                                     unsigned retry_count,\n                                     const request& req,\n                                     reply_handler& handle,\n                                     const retry_strategy& strategy,\n                                     std::optional<reply::status_type> expected,\n                                     abort_source* as) {\n    return strategy.should_retry(ex, retry_count).then([this, ex = std::move(ex), retry_count, &req, &handle, &strategy, as, expected](bool retry) mutable {\n        if (!retry) {\n            return make_exception_future<>(std::move(ex));\n        }\n        return with_new_connection([this, &req, &handle, as, expected](connection& con) {\n                                       return do_make_request(con, req, handle, as, expected);\n                                   },\n                                   as).handle_exception([this, retry_count, &req, &handle, &strategy, as, expected](std::exception_ptr ex) {\n            return maybe_retry_request(std::move(ex), retry_count + 1, req, handle, strategy, expected, as);\n        });\n    });\n}\n\nfuture<> client::make_request(const request& req, reply_handler& handle, const retry_strategy& strategy, std::optional<reply::status_type> expected, abort_source* as) {\n    if (as && as->abort_requested()) {\n        return make_exception_future(as->abort_requested_exception_ptr());\n    }\n    try {\n        validate_request(req);\n    } catch (...) {\n        return current_exception_as_future();\n    }\n    return with_connection([this, &req, &handle, as, expected] (connection& con) {\n        return do_make_request(con, req, handle, as, expected);\n    }, as).handle_exception([this, &req, &handle, &strategy, as, expected] (std::exception_ptr ex) {\n        if (as && as->abort_requested()) {\n            return make_exception_future<>(as->abort_requested_exception_ptr());\n        }\n        return maybe_retry_request(std::move(ex), 0, req, handle, strategy, expected, as);\n    });\n}\n\nclass skip_body_source : public data_source_impl {\npublic:\n    skip_body_source(reply& rep) {\n        // nothing to consume here\n        rep.left_content_length = 0;\n        http_log.trace(\"Skipping HEAD reply body\");\n    }\n\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        return make_ready_future<temporary_buffer<char>>();\n    }\n\n    virtual future<temporary_buffer<char>> get() override {\n        return make_ready_future<temporary_buffer<char>>();\n    }\n};\n\nfuture<> client::do_make_request(connection& con, const request& req, reply_handler& handle, abort_source* as, std::optional<reply::status_type> expected) {\n    auto sub = as ? as->subscribe([&con] () noexcept { con.shutdown(); }) : std::nullopt;\n    return con.do_make_request(req).then([this, &con, &req, &handle, expected] (connection::reply_ptr reply) mutable {\n        auto& rep = *reply;\n        if (expected.has_value() && rep._status != expected.value()) {\n            if (!http_log.is_enabled(log_level::debug)) {\n                return make_exception_future<>(httpd::unexpected_status_error(rep._status));\n            }\n\n            return do_with(con.in(rep), [reply = std::move(reply)] (auto& in) mutable {\n                return util::read_entire_stream_contiguous(in).then([reply = std::move(reply)] (auto message) {\n                    http_log.debug(\"request finished with {}: {}\", reply->_status, message);\n                    return make_exception_future<>(httpd::unexpected_status_error(reply->_status));\n                });\n            });\n        }\n\n        auto in = req._method != \"HEAD\" ? con.in(rep) : input_stream<char>(data_source(std::make_unique<skip_body_source>(rep)));\n        return handle(rep, std::move(in)).then([this, reply = std::move(reply), &con] {\n            if (reply->left_content_length > 0) {\n                auto bytes_left = reply->left_content_length;\n                /*\n                 * Reply with known body size (content_length_source_impl) tracks this counter\n                 * carefully and sets it with the exact number of bytes left to be read. If this\n                 * value is low enough, it's cheaper to read the body up to the end and instantly\n                 * discard it rather than to close the connection and establish a new one for\n                 * next request.\n                 *\n                 * Reply with \"dynamic\" body (chunked_source_impl) will set this counter to zero\n                 * after reading all body contents, otherwise it will be set to maximum uint64_t\n                 * value, so the below check will be false in this case.\n                 *\n                 * The \"HEAD\" reply doesn't have body and will report zero left bytes.\n                 */\n                if (bytes_left <= _max_bytes_to_drain) {\n                    http_log.trace(\"content was not fully consumed, {} bytes were left behind, skipping and returning the connection to the pool\", bytes_left);\n                    return con._read_buf.skip(bytes_left);\n                }\n                http_log.trace(\"content was not fully consumed, content length is {} but {} left, will close the connection\",\n                               reply->content_length,\n                               reply->left_content_length);\n                con._persistent = false;\n            }\n            return make_ready_future<>();\n        });\n    }).handle_exception([&con] (auto ex) mutable {\n        con._persistent = false;\n        return make_exception_future<>(std::move(ex));\n    }).finally([sub = std::move(sub)] {});\n}\n\nfuture<> client::close() {\n    if (_pool.empty()) {\n        return _new_connections->close();\n    }\n\n    connection_ptr con = _pool.front().shared_from_this();\n    _pool.pop_front();\n    http_log.trace(\"closing connection {}\", con->_fd.local_address());\n    return con->close().then([this, con] {\n        return close();\n    });\n}\n\n} // experimental namespace\n} // http namespace\n} // seastar namespace\n"
  },
  {
    "path": "src/http/common.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <cstdlib>\n#include <memory>\n#include <utility>\n#include <numeric>\n#include <span>\n\n#include <seastar/http/common.hh>\n#include <seastar/core/iostream-impl.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\noperation_type str2type(const sstring& type) {\n    if (type == \"DELETE\") {\n        return DELETE;\n    }\n    if (type == \"POST\") {\n        return POST;\n    }\n    if (type == \"PUT\") {\n        return PUT;\n    }\n    if (type == \"HEAD\") {\n        return HEAD;\n    }\n    if (type == \"OPTIONS\") {\n        return OPTIONS;\n    }\n    if (type == \"TRACE\") {\n        return TRACE;\n    }\n    if (type == \"CONNECT\") {\n        return CONNECT;\n    }\n    if (type == \"PATCH\") {\n        return PATCH;\n    }\n    return GET;\n}\n\nsstring type2str(operation_type type) {\n    if (type == DELETE) {\n        return \"DELETE\";\n    }\n    if (type == POST) {\n        return \"POST\";\n    }\n    if (type == PUT) {\n        return \"PUT\";\n    }\n    if (type == HEAD) {\n        return \"HEAD\";\n    }\n    if (type == OPTIONS) {\n        return \"OPTIONS\";\n    }\n    if (type == TRACE) {\n        return \"TRACE\";\n    }\n    if (type == CONNECT) {\n        return \"CONNECT\";\n    }\n    if (type == PATCH) {\n        return \"PATCH\";\n    }\n    return \"GET\";\n}\n\n}\n\nnamespace http {\nnamespace internal {\n\nstatic constexpr size_t default_body_sink_buffer_size = 32000;\n\n// Data sinks below are running \"on top\" of socket output stream and provide\n// reliable and handy way of generating request bodies according to selected\n// encoding type and content-length.\n//\n// Respectively, both .close() methods should not close the underlying stream,\n// because the socket in question may continue being in use for keep-alive\n// connections, and closing it would just break the keep-alive-ness\n\nclass http_chunked_data_sink_impl : public data_sink_impl {\n    output_stream<char>& _out;\n\n    future<> write_size(size_t s) {\n        auto req = format(\"{:x}\\r\\n\", s);\n        return _out.write(req);\n    }\npublic:\n    http_chunked_data_sink_impl(output_stream<char>& out) : _out(out) {\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> data) override {\n        return data_sink_impl::fallback_put(data, [this] (temporary_buffer<char>&& buf) {\n            return do_put(std::move(buf));\n        });\n    }\n#else\n    virtual future<> put(net::packet data) override {\n        return data_sink_impl::fallback_put(std::move(data));\n    }\n    using data_sink_impl::put;\n    virtual future<> put(temporary_buffer<char> buf) override {\n        return do_put(std::move(buf));\n    }\n#endif\nprivate:\n    future<> do_put(temporary_buffer<char> buf) {\n        if (buf.size() == 0) {\n            // size 0 buffer should be ignored, some server\n            // may consider it an end of message\n            return make_ready_future<>();\n        }\n        auto size = buf.size();\n        return write_size(size).then([this, buf = std::move(buf)] () mutable {\n            return _out.write(buf.get(), buf.size());\n        }).then([this] () mutable {\n            return _out.write(\"\\r\\n\", 2);\n        });\n    }\n    virtual future<> close() override {\n        return  make_ready_future<>();\n    }\n};\n\nclass http_chunked_data_sink : public data_sink {\npublic:\n    http_chunked_data_sink(output_stream<char>& out)\n        : data_sink(std::make_unique<http_chunked_data_sink_impl>(\n                out)) {}\n};\n\noutput_stream<char> make_http_chunked_output_stream(output_stream<char>& out) {\n    output_stream_options opts;\n    opts.trim_to_size = true;\n    return output_stream<char>(http_chunked_data_sink(out), default_body_sink_buffer_size, opts);\n}\n\nclass http_content_length_data_sink_impl : public data_sink_impl {\n    output_stream<char>& _out;\n    const size_t _limit;\n    size_t& _bytes_written;\n\npublic:\n    http_content_length_data_sink_impl(output_stream<char>& out, size_t total_len, size_t& bytes_written)\n            : _out(out)\n            , _limit(total_len)\n            , _bytes_written(bytes_written)\n    {\n        // at the very beginning, 0 bytes were written\n        _bytes_written = 0;\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> data) override {\n        size_t size = std::accumulate(data.begin(), data.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n        if (size == 0) {\n            return make_ready_future<>();\n        }\n        if (_bytes_written + size > _limit) {\n            return make_exception_future<>(std::runtime_error(format(\"body content length overflow: want {} limit {}\", _bytes_written + size, _limit)));\n        }\n        return _out.write(data).then([this, size] {\n            _bytes_written += size;\n        });\n    }\n#else\n    virtual future<> put(net::packet data) override {\n        auto size = data.len();\n        if (size == 0) {\n            return make_ready_future<>();\n        }\n        if (_bytes_written + size > _limit) {\n            return make_exception_future<>(std::runtime_error(format(\"body content length overflow: want {} limit {}\", _bytes_written + size, _limit)));\n        }\n        return _out.write(std::move(data)).then([this, size] {\n            _bytes_written += size;\n        });\n    }\n    using data_sink_impl::put;\n    virtual future<> put(temporary_buffer<char> buf) override {\n        auto size = buf.size();\n        if (size == 0) {\n            return make_ready_future<>();\n        }\n\n        if (_bytes_written + size > _limit) {\n            return make_exception_future<>(std::runtime_error(format(\"body content length overflow: want {} limit {}\", _bytes_written + buf.size(), _limit)));\n        }\n\n        return _out.write(buf.get(), size).then([this, size] {\n            _bytes_written += size;\n        });\n    }\n#endif\n    virtual future<> close() override {\n        return make_ready_future<>();\n    }\n};\n\nclass http_content_length_data_sink : public data_sink {\npublic:\n    http_content_length_data_sink(output_stream<char>& out, size_t total_len, size_t& bytes_written)\n        : data_sink(std::make_unique<http_content_length_data_sink_impl>(out, total_len, bytes_written))\n    {\n    }\n};\n\noutput_stream<char> make_http_content_length_output_stream(output_stream<char>& out, size_t total_len, size_t& bytes_written) {\n    output_stream_options opts;\n    opts.trim_to_size = true;\n    return output_stream<char>(http_content_length_data_sink(out, total_len, bytes_written), default_body_sink_buffer_size, opts);\n}\n\n}\n}\n\n}\n\n"
  },
  {
    "path": "src/http/file_handler.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <algorithm>\n#include <iostream>\n#include <memory>\n\n#include <seastar/http/file_handler.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/http/exception.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\ndirectory_handler::directory_handler(const sstring& doc_root,\n        file_transformer* transformer)\n        : file_interaction_handler(transformer), doc_root(doc_root) {\n}\n\nfuture<std::unique_ptr<http::reply>> directory_handler::handle(const sstring& path,\n        std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n    sstring full_path = doc_root + req->param.get_decoded_param(\"path\");\n    auto h = this;\n    return engine().file_type(full_path).then(\n            [h, full_path, req = std::move(req), rep = std::move(rep)](auto val) mutable {\n                if (val) {\n                    if (val.value() == directory_entry_type::directory) {\n                        if (h->redirect_if_needed(*req.get(), *rep.get())) {\n                            return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                        }\n                        full_path += \"/index.html\";\n                    }\n                    return h->read(full_path, std::move(req), std::move(rep));\n                }\n                rep->set_status(http::reply::status_type::not_found).done();\n                return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n\n            });\n}\n\nfile_interaction_handler::~file_interaction_handler() {\n    delete transformer;\n}\n\nsstring file_interaction_handler::get_extension(const sstring& file) {\n    size_t last_slash_pos = file.find_last_of('/');\n    size_t last_dot_pos = file.find_last_of('.');\n    sstring extension;\n    if (last_dot_pos != sstring::npos && last_dot_pos > last_slash_pos) {\n        extension = file.substr(last_dot_pos + 1);\n    }\n    // normalize file extension for mime type\n    std::transform(extension.begin(), extension.end(), extension.begin(), ::tolower);\n    return extension;\n}\n\noutput_stream<char> file_interaction_handler::get_stream(std::unique_ptr<http::request> req,\n        const sstring& extension, output_stream<char>&& s) {\n    if (transformer) {\n        return transformer->transform(std::move(req), extension, std::move(s));\n    }\n    return std::move(s);\n}\n\nfuture<std::unique_ptr<http::reply>> file_interaction_handler::read(\n        sstring file_name, std::unique_ptr<http::request> req,\n        std::unique_ptr<http::reply> rep) {\n    sstring extension = get_extension(file_name);\n    rep->write_body(extension, [req = std::move(req), extension, file_name, this] (output_stream<char>&& s) mutable {\n        return do_with(get_stream(std::move(req), extension, std::move(s)),\n                [file_name] (output_stream<char>& os) {\n            return open_file_dma(file_name, open_flags::ro).then([&os] (file f) {\n                return do_with(make_file_input_stream(std::move(f)), [&os](input_stream<char>& is) {\n                    return copy(is, os).finally([&os] {\n                        return os.close();\n                    }).finally([&is] {\n                        return is.close();\n                    });\n                });\n            });\n        });\n    });\n    return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n}\n\nbool file_interaction_handler::redirect_if_needed(const http::request& req,\n        http::reply& rep) const {\n    if (req._url.length() == 0 || req._url.back() != '/') {\n        rep.set_status(http::reply::status_type::moved_permanently);\n        rep._headers[\"Location\"] = req.get_url() + \"/\";\n        rep.done();\n        return true;\n    }\n    return false;\n}\n\nfuture<std::unique_ptr<http::reply>> file_handler::handle(const sstring& path,\n        std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n    if (force_path && redirect_if_needed(*req.get(), *rep.get())) {\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n    return read(file, std::move(req), std::move(rep));\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/httpd.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <memory>\n#include <algorithm>\n#include <bitset>\n#include <cctype>\n#include <chrono>\n#include <cstdint>\n#include <functional>\n#include <iostream>\n#include <limits>\n#include <queue>\n#include <unordered_map>\n#include <vector>\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/print.hh>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/internal/content_source.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/util/short_streams.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/string_utils.hh>\n\n\nusing namespace std::chrono_literals;\n\nnamespace seastar {\n\nlogger hlogger(\"httpd\");\n\nnamespace httpd {\nhttp_stats::http_stats(http_server& server, const sstring& name)\n {\n    namespace sm = seastar::metrics;\n    std::vector<sm::label_instance> labels;\n\n    labels.push_back(sm::label_instance(\"service\", name));\n    _metric_groups.add_group(\"httpd\", {\n            sm::make_counter(\"connections_total\", [&server] { return server.total_connections(); }, sm::description(\"The total number of connections opened\"), labels),\n            sm::make_gauge(\"connections_current\", [&server] { return server.current_connections(); }, sm::description(\"The current number of open  connections\"), labels),\n            sm::make_counter(\"read_errors\", [&server] { return server.read_errors(); }, sm::description(\"The total number of errors while reading http requests\"), labels),\n            sm::make_counter(\"reply_errors\", [&server] { return server.reply_errors(); }, sm::description(\"The total number of errors while replying to http\"), labels),\n            sm::make_counter(\"requests_served\", [&server] { return server.requests_served(); }, sm::description(\"The total number of http requests served\"), labels)\n    });\n}\n\nsstring http_server_control::generate_server_name() {\n    static thread_local uint16_t idgen;\n    return seastar::format(\"http-{}\", idgen++);\n}\n\nfuture<> connection::do_response_loop() {\n    return _replies.pop_eventually().then(\n        [this] (std::unique_ptr<http::reply> resp) {\n            if (!resp) {\n                // eof\n                return make_ready_future<>();\n            }\n            _resp = std::move(resp);\n            return start_response().then([this] {\n                        return do_response_loop();\n                    });\n        });\n}\n\nfuture<> connection::start_response() {\n    return _resp->write_reply(out()).then_wrapped([this] (auto f) {\n        if (f.failed()) {\n            // In case of an error during the write close the connection\n            _server._respond_errors++;\n            _done = true;\n            _replies.abort(std::make_exception_ptr(std::logic_error(\"Unknown exception during body creation\")));\n            _replies.push(std::unique_ptr<http::reply>());\n            f.ignore_ready_future();\n        }\n        return make_ready_future<>();\n    }).then_wrapped([this ] (auto f) {\n        if (f.failed()) {\n            // We could not write the closing sequence\n            // Something is probably wrong with the connection,\n            // we should close it, so the client will disconnect\n            _done = true;\n            _replies.abort(std::make_exception_ptr(std::logic_error(\"Unknown exception during body creation\")));\n            _replies.push(std::unique_ptr<http::reply>());\n            f.ignore_ready_future();\n            return make_ready_future<>();\n        } else {\n            return _write_buf.flush();\n        }\n    }).then_wrapped([this] (auto f) {\n        if (f.failed()) {\n            // flush failed. just close the connection\n            _done = true;\n            _replies.abort(std::make_exception_ptr(std::logic_error(\"Unknown exception during body creation\")));\n            _replies.push(std::unique_ptr<http::reply>());\n            f.ignore_ready_future();\n        }\n        _resp.reset();\n        return make_ready_future<>();\n    });\n}\n\nconnection::~connection() {\n    --_server._current_connections;\n    _server._connections.erase(_server._connections.iterator_to(*this));\n}\n\nvoid connection::on_new_connection() {\n    ++_server._total_connections;\n    ++_server._current_connections;\n    _fd.set_nodelay(true);\n    _server._connections.push_back(*this);\n}\n\nfuture<> connection::read() {\n    return do_until([this] {return _done;}, [this] {\n        return read_one();\n    }).then_wrapped([this] (future<> f) {\n        // swallow error\n        if (f.failed()) {\n            _server._read_errors++;\n        }\n        f.ignore_ready_future();\n        return _replies.push_eventually( {});\n    });\n}\n\nstatic input_stream<char> make_content_stream(http::request* req, input_stream<char>& buf) {\n    // Create an input stream based on the requests body encoding or lack thereof\n    if (seastar::internal::case_insensitive_cmp()(req->get_header(\"Transfer-Encoding\"), \"chunked\")) {\n        return input_stream<char>(data_source(std::make_unique<internal::chunked_source_impl>(buf, req->chunk_extensions, req->trailing_headers)));\n    } else {\n        return input_stream<char>(data_source(std::make_unique<internal::content_length_source_impl>(buf, req->content_length)));\n    }\n}\n\nstatic future<std::unique_ptr<http::request>>\nset_request_content(std::unique_ptr<http::request> req, input_stream<char>* content_stream, bool streaming) {\n    req->content_stream = content_stream;\n\n    if (streaming) {\n        return make_ready_future<std::unique_ptr<http::request>>(std::move(req));\n    } else {\n        // Read the entire content into the request content string\n        return util::read_entire_stream_contiguous(*content_stream).then([req = std::move(req)] (sstring content) mutable {\n            http::internal::deprecated_content(*req) = std::move(content);\n            return make_ready_future<std::unique_ptr<http::request>>(std::move(req));\n        });\n    }\n}\n\nstatic void set_header_connection(http::reply& resp, bool keep_alive) {\n    if (keep_alive) {\n        if (resp._version == \"1.0\") {\n            resp.add_header(\"Connection\", \"Keep-Alive\");\n        }\n    } else {\n        if (resp._version == \"1.1\") {\n            resp.add_header(\"Connection\", \"close\");\n        }\n    }\n}\n\nvoid connection::generate_error_reply_and_close(std::unique_ptr<http::request> req, http::reply::status_type status, const sstring& msg) {\n    auto resp = std::make_unique<http::reply>();\n    // TODO: Handle HTTP/2.0 when it releases\n    resp->set_version(req->_version);\n    resp->set_status(status, msg);\n    set_header_connection(*resp, false);\n    resp->done();\n    _done = true;\n    _replies.push(std::move(resp));\n}\n\nfuture<> connection::read_one() {\n    _parser.init();\n    return _read_buf.consume(_parser).then([this] () mutable {\n        if (_parser.eof()) {\n            _done = true;\n            return make_ready_future<>();\n        }\n        ++_server._requests_served;\n        std::unique_ptr<http::request> req = _parser.get_parsed_request();\n\n        req->_server_address = this->_server_addr;\n        req->_client_address = this->_client_addr;\n\n        if (_tls) {\n            req->protocol_name = \"https\";\n        }\n        if (_parser.failed()) {\n            if (req->_version.empty()) {\n                // we might have failed to parse even the version\n                req->_version = \"1.1\";\n            }\n            generate_error_reply_and_close(std::move(req), http::reply::status_type::bad_request, \"Can't parse the request\");\n            return make_ready_future<>();\n        }\n\n        size_t content_length_limit = _server.get_content_length_limit();\n        sstring length_header = req->get_header(\"Content-Length\");\n        req->content_length = strtol(length_header.c_str(), nullptr, 10);\n\n        if (req->content_length > content_length_limit) {\n            auto msg = format(\"Content length limit ({}) exceeded: {}\", content_length_limit, req->content_length);\n            generate_error_reply_and_close(std::move(req), http::reply::status_type::payload_too_large, std::move(msg));\n            return make_ready_future<>();\n        }\n\n        sstring encoding = req->get_header(\"Transfer-Encoding\");\n        if (encoding.size() && !seastar::internal::case_insensitive_cmp()(encoding, \"chunked\")){\n            //TODO: add \"identity\", \"gzip\"(\"x-gzip\"), \"compress\"(\"x-compress\"), and \"deflate\" encodings and their combinations\n            generate_error_reply_and_close(std::move(req), http::reply::status_type::not_implemented, format(\"Encodings other than \\\"chunked\\\" are not implemented (received encoding: \\\"{}\\\")\", encoding));\n            return make_ready_future<>();\n        }\n\n        auto maybe_reply_continue = [this, req = std::move(req)] () mutable {\n            if (req->_version == \"1.1\" && seastar::internal::case_insensitive_cmp()(req->get_header(\"Expect\"), \"100-continue\")){\n                return _replies.not_full().then([req = std::move(req), this] () mutable {\n                    auto continue_reply = std::make_unique<http::reply>();\n                    set_headers(*continue_reply);\n                    continue_reply->set_version(req->_version);\n                    continue_reply->set_status(http::reply::status_type::continue_).done();\n                    this->_replies.push(std::move(continue_reply));\n                    return make_ready_future<std::unique_ptr<http::request>>(std::move(req));\n                });\n            } else {\n                return make_ready_future<std::unique_ptr<http::request>>(std::move(req));\n            }\n        };\n\n        return maybe_reply_continue().then([this] (std::unique_ptr<http::request> req) {\n            return do_with(make_content_stream(req.get(), _read_buf), sstring(req->_version), std::move(req), [this] (input_stream<char>& content_stream, sstring& version, std::unique_ptr<http::request>& req) {\n                return set_request_content(std::move(req), &content_stream, _server.get_content_streaming()).then([this, &content_stream] (std::unique_ptr<http::request> req) {\n                    return _replies.not_full().then([this, req = std::move(req)] () mutable {\n                        return generate_reply(std::move(req));\n                    }).then([this, &content_stream](bool done) {\n                        _done = done;\n                        // If the handler did not read the entire request\n                        // content, this connection cannot be reused so we\n                        // need to close it (via \"_done = true\"). But we can't\n                        // just check content_stream.eof(): It may only become\n                        // true after read(). Issue #907.\n                        return content_stream.read().then([this] (temporary_buffer<char> buf) {\n                            if (!buf.empty()) {\n                                _done = true;\n                            }\n                        });\n                    });\n                }).handle_exception_type([this, &version] (const base_exception& e) mutable {\n                    // If the request had a \"Transfer-Encoding: chunked\" header and content streaming wasn't enabled, we might have failed\n                    // before passing the request to handler - when we were parsing chunks\n                    auto err_req = std::make_unique<http::request>();\n                    err_req->_version = version;\n                    generate_error_reply_and_close(std::move(err_req), e.status(), e.str());\n                });\n            });\n        });\n    });\n}\n\nfuture<> connection::process() {\n    // Launch read and write \"threads\" simultaneously:\n    return when_all(read(), respond()).then(\n            [] (std::tuple<future<>, future<>> joined) {\n        try {\n            std::get<0>(joined).get();\n        } catch (...) {\n            hlogger.debug(\"Read exception encountered: {}\", std::current_exception());\n        }\n        try {\n            std::get<1>(joined).get();\n        } catch (...) {\n            hlogger.debug(\"Response exception encountered: {}\", std::current_exception());\n        }\n        return make_ready_future<>();\n    }).finally([this]{\n        return _read_buf.close().handle_exception([](std::exception_ptr e) {\n            hlogger.debug(\"Close exception encountered: {}\", e);\n        });\n    });\n}\nvoid connection::shutdown() {\n    _fd.shutdown_input();\n    _fd.shutdown_output();\n}\n\noutput_stream<char>& connection::out() {\n    return _write_buf;\n}\n\nfuture<> connection::respond() {\n    return do_response_loop().then_wrapped([this] (future<> f) {\n        // swallow error\n        if (f.failed()) {\n            _server._respond_errors++;\n        }\n        f.ignore_ready_future();\n        return _write_buf.close();\n    });\n}\n\nvoid connection::set_headers(http::reply& resp) {\n    resp._headers[\"Server\"] = \"Seastar httpd\";\n    resp._headers[\"Date\"] = _server._date;\n}\n\nfuture<bool> connection::generate_reply(std::unique_ptr<http::request> req) {\n    auto resp = std::make_unique<http::reply>();\n    resp->set_version(req->_version);\n    set_headers(*resp);\n    bool keep_alive = req->should_keep_alive();\n    set_header_connection(*resp, keep_alive);\n\n    sstring url = req->parse_query_param();\n    sstring version = req->_version;\n    if (req->_method == \"HEAD\") {\n        resp->skip_body();\n    }\n    return _server._routes.handle(url, std::move(req), std::move(resp)).\n    // Caller guarantees enough room\n    then([this, keep_alive , version = std::move(version)](std::unique_ptr<http::reply> rep) {\n        rep->set_version(version).done();\n        this->_replies.push(std::move(rep));\n        return make_ready_future<bool>(!keep_alive);\n    });\n}\n\nvoid http_server::set_tls_credentials(server_credentials_ptr credentials) {\n    _credentials = credentials;\n}\n\nsize_t http_server::get_content_length_limit() const {\n    return _content_length_limit;\n}\n\nvoid http_server::set_content_length_limit(size_t limit) {\n    _content_length_limit = limit;\n}\n\nbool http_server::get_content_streaming() const {\n    return _content_streaming;\n}\n\nvoid http_server::set_content_streaming(bool b) {\n    _content_streaming = b;\n}\n\nfuture<> http_server::listen(socket_address addr, listen_options lo,\n            server_credentials_ptr listener_credentials) {\n    if (listener_credentials) {\n        _listeners.push_back(seastar::tls::listen(listener_credentials, addr, lo));\n    } else {\n        _listeners.push_back(seastar::listen(addr, lo));\n    }\n    return do_accepts(_listeners.size() - 1, listener_credentials != nullptr);\n}\n\nfuture<> http_server::listen(socket_address addr, listen_options lo) {\n    return listen(addr, lo, _credentials);\n}\n\nfuture<> http_server::listen(socket_address addr,\n            server_credentials_ptr listener_credentials) {\n    listen_options lo;\n    lo.reuse_address = true;\n    return listen(addr, lo, listener_credentials);\n}\n\nfuture<> http_server::listen(socket_address addr) {\n    listen_options lo;\n    lo.reuse_address = true;\n    return listen(addr, lo);\n}\nfuture<> http_server::stop() {\n    future<> tasks_done = _task_gate.close();\n    for (auto&& l : _listeners) {\n        l.abort_accept();\n    }\n    for (auto&& c : _connections) {\n        c.shutdown();\n    }\n    return tasks_done;\n}\n\n// FIXME: This could return void\nfuture<> http_server::do_accepts(int which, bool tls) {\n    (void)try_with_gate(_task_gate, [this, which, tls] {\n        return keep_doing([this, which, tls] {\n            return try_with_gate(_task_gate, [this, which, tls] {\n                return do_accept_one(which, tls);\n            });\n        }).handle_exception_type([](const gate_closed_exception& e) {});\n    }).handle_exception_type([](const gate_closed_exception& e) {});\n    return make_ready_future<>();\n}\n\nfuture<> http_server::do_accepts(int which){\n    return do_accepts(which, _credentials != nullptr);\n}\n\nfuture<> http_server::do_accept_one(int which, bool tls) {\n    return _listeners[which].accept().then([this, tls] (accept_result ar) mutable {\n        if (_keepalive_params) {\n            ar.connection.set_keepalive(true);\n            ar.connection.set_keepalive_parameters(_keepalive_params.value());\n        }\n        auto local_address = ar.connection.local_address();\n        auto conn = std::make_unique<connection>(*this, std::move(ar.connection),\n                std::move(ar.remote_address), std::move(local_address), tls);\n        (void)try_with_gate(_task_gate, [conn = std::move(conn)]() mutable {\n            return conn->process().handle_exception([conn = std::move(conn)] (std::exception_ptr ex) {\n                hlogger.error(\"request error: {}\", ex);\n            });\n        }).handle_exception_type([] (const gate_closed_exception& e) {});\n    }).handle_exception_type([] (const std::system_error &e) {\n        // We expect a ECONNABORTED when http_server::stop is called,\n        // no point in warning about that.\n        if (e.code().value() != ECONNABORTED) {\n            hlogger.error(\"accept failed: {}\", e);\n        }\n    }).handle_exception([] (std::exception_ptr ex) {\n        hlogger.error(\"accept failed: {}\", ex);\n    });\n}\n\nuint64_t http_server::total_connections() const {\n    return _total_connections;\n}\nuint64_t http_server::current_connections() const {\n    return _current_connections;\n}\nuint64_t http_server::requests_served() const {\n    return _requests_served;\n}\nuint64_t http_server::read_errors() const {\n    return _read_errors;\n}\nuint64_t http_server::reply_errors() const {\n    return _respond_errors;\n}\n\n// Write the current date in the specific \"preferred format\" defined in\n// RFC 7231, Section 7.1.1.1, a.k.a. IMF (Internet Message Format) fixdate.\n// For example: Sun, 06 Nov 1994 08:49:37 GMT\nsstring http_server::http_date() {\n    auto t = ::time(nullptr);\n    struct tm tm;\n    gmtime_r(&t, &tm);\n    // Using strftime() would have been easier, but unfortunately relies on\n    // the current locale, and we need the month and day names in English.\n    static const char* days[] = {\n        \"Sun\", \"Mon\", \"Tue\", \"Wed\", \"Thu\", \"Fri\", \"Sat\"\n    };\n    static const char* months[] = {\n        \"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\",\n        \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"\n    };\n    return seastar::format(\"{}, {:02d} {} {} {:02d}:{:02d}:{:02d} GMT\",\n        days[tm.tm_wday], tm.tm_mday, months[tm.tm_mon], 1900 + tm.tm_year,\n        tm.tm_hour, tm.tm_min, tm.tm_sec);\n}\n\n\nfuture<> http_server_control::start(const sstring& name) {\n    return _server_dist->start(name);\n}\n\nfuture<> http_server_control::stop() noexcept {\n    return _server_dist->stop();\n}\n\nfuture<> http_server_control::set_routes(std::function<void(routes& r)> fun) {\n    return _server_dist->invoke_on_all([fun](http_server& server) {\n        fun(server._routes);\n    });\n}\n\nfuture<> http_server_control::listen(socket_address addr) {\n    return _server_dist->invoke_on_all<future<> (http_server::*)(socket_address)>(&http_server::listen, addr);\n}\n\nfuture<> http_server_control::listen(socket_address addr, http_server::server_credentials_ptr credentials) {\n    return _server_dist->invoke_on_all<future<> (http_server::*)(socket_address, http_server::server_credentials_ptr)>(&http_server::listen, addr, credentials);\n}\n\nfuture<> http_server_control::listen(socket_address addr, listen_options lo) {\n    return _server_dist->invoke_on_all<future<> (http_server::*)(socket_address, listen_options)>(&http_server::listen, addr, lo);\n}\n\nfuture<> http_server_control::listen(socket_address addr, listen_options lo, http_server::server_credentials_ptr credentials) {\n    return _server_dist->invoke_on_all<future<> (http_server::*)(socket_address, listen_options, http_server::server_credentials_ptr)>(&http_server::listen, addr, lo, credentials);\n}\n\nsharded<http_server>& http_server_control::server() {\n    return *_server_dist;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/json_path.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#include <seastar/http/json_path.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nusing namespace std;\n\nvoid path_description::set(routes& _routes, handler_base* handler) const {\n    for (auto& i : mandatory_queryparams) {\n        handler->mandatory(i);\n    }\n\n    if (params.size() == 0)\n        _routes.put(operations.method, path, handler);\n    else {\n        match_rule* rule = new match_rule(handler);\n        rule->add_str(path);\n        for (auto&& i : params) {\n            if (i.type == url_component_type::FIXED_STRING) {\n                rule->add_str(i.name);\n            } else {\n                rule->add_param(i.name, i.type == url_component_type::PARAM_UNTIL_END_OF_PATH);\n            }\n        }\n        _cookie = _routes.add_cookie(rule, operations.method);\n    }\n}\n\nvoid path_description::set(routes& _routes,\n        const json_request_function& f) const {\n    set(_routes, new function_handler(f));\n}\n\nvoid path_description::set(routes& _routes, const future_json_function& f) const {\n    set(_routes, new function_handler(f));\n}\n\nvoid path_description::unset(routes& _routes) const {\n    if (params.size() == 0) {\n        _routes.drop(operations.method, path);\n    } else {\n        auto rule = _routes.del_cookie(_cookie, operations.method);\n        delete rule;\n    }\n}\n\npath_description::path_description(const sstring& path, operation_type method,\n        const sstring& nickname,\n        const std::vector<std::pair<sstring, bool>>& path_parameters,\n        const std::vector<sstring>& mandatory_params)\n        : path(path), operations(method, nickname) {\n\n    for (auto man : mandatory_params) {\n        pushmandatory_param(man);\n    }\n    for (auto& [param, all_path] : path_parameters) {\n        pushparam(param, all_path);\n    }\n}\n\npath_description::path_description(const sstring& path, operation_type method,\n        const sstring& nickname,\n        const std::initializer_list<path_part>& path_parameters,\n        const std::vector<sstring>& mandatory_params)\n        : path(path), operations(method, nickname) {\n\n    for (auto man : mandatory_params) {\n        pushmandatory_param(man);\n    }\n    for (auto param : path_parameters) {\n        params.push_back(param);\n    }\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/matcher.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <iostream>\n\n#include <seastar/http/matcher.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nusing namespace std;\n\n/**\n * Search for the end of the url parameter.\n * @param url the url to search\n * @param ind the position in the url\n * @param entire_path when set to true, take all the reminaing url\n * when set to false, search for the next slash\n * @return the position in the url of the end of the parameter\n */\nstatic size_t find_end_param(const sstring& url, size_t ind, bool entire_path) {\n    size_t pos = (entire_path) ? url.length() : url.find('/', ind + 1);\n    if (pos == sstring::npos) {\n        return url.length();\n    }\n    return pos;\n}\n\nsize_t param_matcher::match(const sstring& url, size_t ind, parameters& param) {\n    size_t last = find_end_param(url, ind, _entire_path);\n    if (last == ind) {\n        /*\n         * empty parameter allows only for the case of entire_path\n         */\n        if (_entire_path) {\n            param.set(_name, \"\");\n            return ind;\n        }\n        return sstring::npos;\n    }\n    param.set(_name, url.substr(ind, last - ind));\n    return last;\n}\n\nsize_t str_matcher::match(const sstring& url, size_t ind, parameters& param) {\n    if (url.length() >= _len + ind && (url.find(_cmp, ind) == ind)\n            && (url.length() == _len + ind || url.at(_len + ind) == '/')) {\n        return _len + ind;\n    }\n    return sstring::npos;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/mime_types.cc",
    "content": "//\n// mime_types.cpp\n// ~~~~~~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n//\n\n\n#include <string_view>\n\n#include <seastar/http/mime_types.hh>\n\n\nnamespace seastar {\n\nnamespace http {\nnamespace mime_types {\n\nstruct mapping {\n    const char* extension;\n    const char* mime_type;\n} mappings[] = {\n        { \"json\", \"application/json\"},\n\t{ \"xml\", \"application/xml\"},\n        { \"gif\", \"image/gif\" },\n        { \"htm\", \"text/html\" },\n        { \"css\", \"text/css\" },\n        { \"js\", \"text/javascript\" },\n        { \"html\", \"text/html\" },\n        { \"jpg\", \"image/jpeg\" },\n        { \"svg\", \"image/svg+xml\" },\n        { \"png\", \"image/png\" },\n        { \"txt\", \"text/plain\" },\n        { \"ico\", \"image/x-icon\" },\n        { \"bin\", \"application/octet-stream\" },\n        { \"proto\", \"application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited\"},\n};\n\nconst char* extension_to_type(std::string_view extension)\n{\n    for (mapping m : mappings) {\n        if (extension == m.extension) {\n            return m.mime_type;\n        }\n    }\n    return \"text/plain\";\n}\n\n} // namespace mime_types\n\n} // http\n\n}\n"
  },
  {
    "path": "src/http/reply.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n//\n// response.cpp\n// ~~~~~~~~~\n//\n// Copyright (c) 2003-2013 Christopher M. Kohlhoff (chris at kohlhoff dot com)\n//\n// Distributed under the Boost Software License, Version 1.0. (See accompanying\n// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)\n//\n\n#include <seastar/http/reply.hh>\n#include <seastar/core/print.hh>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/common.hh>\n#include <seastar/http/response_parser.hh>\n#include <seastar/core/loop.hh>\n\nnamespace seastar {\n\nnamespace http {\n\nnamespace status_strings {\n\nstatic const std::unordered_map<reply::status_type, std::string_view> status_strings = {\n    {reply::status_type::continue_, \"100 Continue\"},\n    {reply::status_type::switching_protocols, \"101 Switching Protocols\"},\n    {reply::status_type::ok, \"200 OK\"},\n    {reply::status_type::created, \"201 Created\"},\n    {reply::status_type::accepted, \"202 Accepted\"},\n    {reply::status_type::nonauthoritative_information, \"203 Non-Authoritative Information\"},\n    {reply::status_type::no_content, \"204 No Content\"},\n    {reply::status_type::reset_content, \"205 Reset Content\"},\n    {reply::status_type::partial_content, \"206 Partial Content\"},\n    {reply::status_type::multiple_choices, \"300 Multiple Choices\"},\n    {reply::status_type::moved_permanently, \"301 Moved Permanently\"},\n    {reply::status_type::moved_temporarily, \"302 Moved Temporarily\"},\n    {reply::status_type::see_other, \"303 See Other\"},\n    {reply::status_type::not_modified, \"304 Not Modified\"},\n    {reply::status_type::use_proxy, \"305 Use Proxy\"},\n    {reply::status_type::temporary_redirect, \"307 Temporary Redirect\"},\n    {reply::status_type::permanent_redirect, \"308 Permanent Redirect\"},\n    {reply::status_type::bad_request, \"400 Bad Request\"},\n    {reply::status_type::unauthorized, \"401 Unauthorized\"},\n    {reply::status_type::payment_required, \"402 Payment Required\"},\n    {reply::status_type::forbidden, \"403 Forbidden\"},\n    {reply::status_type::not_found, \"404 Not Found\"},\n    {reply::status_type::method_not_allowed, \"405 Method Not Allowed\"},\n    {reply::status_type::not_acceptable, \"406 Not Acceptable\"},\n    {reply::status_type::request_timeout, \"408 Request Timeout\"},\n    {reply::status_type::conflict, \"409 Conflict\"},\n    {reply::status_type::gone, \"410 Gone\"},\n    {reply::status_type::length_required, \"411 Length Required\"},\n    {reply::status_type::payload_too_large, \"413 Payload Too Large\"},\n    {reply::status_type::uri_too_long, \"414 URI Too Long\"},\n    {reply::status_type::unsupported_media_type, \"415 Unsupported Media Type\"},\n    {reply::status_type::expectation_failed, \"417 Expectation Failed\"},\n    {reply::status_type::page_expired, \"419 Page Expired\"},\n    {reply::status_type::unprocessable_entity, \"422 Unprocessable Entity\"},\n    {reply::status_type::upgrade_required, \"426 Upgrade Required\"},\n    {reply::status_type::too_many_requests, \"429 Too Many Requests\"},\n    {reply::status_type::login_timeout, \"440 Login Timeout\"},\n    {reply::status_type::internal_server_error, \"500 Internal Server Error\"},\n    {reply::status_type::not_implemented, \"501 Not Implemented\"},\n    {reply::status_type::bad_gateway, \"502 Bad Gateway\"},\n    {reply::status_type::service_unavailable, \"503 Service Unavailable\"},\n    {reply::status_type::gateway_timeout, \"504 Gateway Timeout\"},\n    {reply::status_type::http_version_not_supported, \"505 HTTP Version Not Supported\"},\n    {reply::status_type::insufficient_storage, \"507 Insufficient Storage\"},\n    {reply::status_type::bandwidth_limit_exceeded, \"509 Bandwidth Limit Exceeded\"},\n    {reply::status_type::network_read_timeout, \"598 Network Read Timeout\"},\n    {reply::status_type::network_connect_timeout, \"599 Network Connect Timeout\"}};\n\ntemplate<typename Func>\nstatic auto with_string_view(reply::status_type status, Func&& func) -> std::invoke_result_t<Func, std::string_view> {\n  if (auto found = status_strings.find(status); found != status_strings.end()) [[likely]] {\n     return func(found->second);\n  }\n  auto dummy_buf = std::to_string(int(status));\n  return func(dummy_buf);\n}\n\n} // namespace status_strings\n\nstd::ostream& operator<<(std::ostream& os, reply::status_type st) {\n    return status_strings::with_string_view(st, [&](std::string_view txt) -> std::ostream& {\n        return os << txt;\n    });\n}\n\nsstring reply::response_line() const {\n    return status_strings::with_string_view(_status, [this](std::string_view txt) {\n        return seastar::format(\"HTTP/{} {}\\r\\n\", _version, txt);\n    });\n}\n\nvoid reply::write_body(const sstring& content_type, body_writer_type&& body_writer) {\n    set_content_type(content_type);\n    _body_writer  = std::move(body_writer);\n}\n\nvoid reply::write_body(const sstring& content_type, sstring content) {\n    _content = std::move(content);\n    done(content_type);\n}\n\nfuture<> reply::write_reply(output_stream<char>& out) {\n    return out.write(response_line()).then([this, &out] {\n        if (_body_writer) {\n            add_header(\"Transfer-Encoding\", \"chunked\");\n        } else {\n            add_header(\"Content-Length\", to_sstring(_content.size()));\n        }\n\n        return write_reply_headers(out).then([&out] {\n            return out.write(\"\\r\\n\", 2);\n        }).then([this, &out] {\n            if (_skip_body) {\n                return make_ready_future<>();\n            }\n            if (_body_writer) {\n                return _body_writer(http::internal::make_http_chunked_output_stream(out)).then([&out] {\n                    return out.write(\"0\\r\\n\\r\\n\", 5);\n                });\n            } else {\n                return out.write(_content.data(), _content.size());\n            }\n        });\n    });\n}\n\nfuture<> reply::write_reply_headers(output_stream<char>& out) {\n    return do_for_each(_headers, [&out](auto& h) {\n        return out.write(h.first + \": \" + h.second + \"\\r\\n\");\n    }).then([this, &out] {\n        return do_for_each(_cookies, [&out] (auto& c) {\n            return out.write(\"Set-Cookie: \" + c.first + \"=\" + c.second + \"\\r\\n\");\n        });\n    });\n}\n\n}\n} // namespace server\n"
  },
  {
    "path": "src/http/request.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Scylladb, Ltd.\n */\n\n\n#include <string_view>\n#include <unordered_map>\n#include <utility>\n\n#include <seastar/http/request.hh>\n#include <seastar/http/url.hh>\n#include <seastar/http/common.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\nnamespace http {\n\nnamespace internal {\n// NOTE: Remove this once `query_parameters` is removed\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n    const std::unordered_map<sstring, sstring>& deprecated_query_parameters(const request& r) noexcept {\n        return r.query_parameters;\n    }\n    std::unordered_map<sstring, sstring>& deprecated_query_parameters(request& r) noexcept {\n        return r.query_parameters;\n    }\n#pragma GCC diagnostic pop\n}\n\nsstring request::format_url() const {\n    sstring query = \"\";\n    if (!_query_params.empty()) {\n        sstring delim = \"&\";\n        for (const auto& [key, values] : _query_params) {\n            auto key_component = delim + internal::url_encode(key) + \"=\";\n            if (values.empty()) {\n                query += key_component;\n            } else {\n                for (const auto& val : values) {\n                    query += key_component + internal::url_encode(val);\n                }\n            }\n        }\n        if (!query.empty()) {\n            query[0] = '?';\n        }\n    } else {\n        sstring delim = \"?\";\n        for (const auto& p : internal::deprecated_query_parameters(*this)) {\n            query += delim + internal::url_encode(p.first);\n            if (!p.second.empty()) {\n                query += \"=\" + internal::url_encode(p.second);\n            }\n            delim = \"&\";\n        }\n    }\n\n    return _url + query;\n}\n\nsstring request::request_line() const {\n    SEASTAR_ASSERT(!_version.empty());\n    return _method + \" \" + format_url() + \" HTTP/\" + _version + \"\\r\\n\";\n}\n\n// FIXME -- generalize with reply::write_request_headers\nfuture<> request::write_request_headers(output_stream<char>& out) const {\n    return do_for_each(_headers, [&out] (auto& h) {\n        return out.write(h.first + \": \" + h.second + \"\\r\\n\");\n    });\n}\n\nvoid request::add_query_param(std::string_view param) {\n    size_t split = param.find('=');\n    auto& deprecated_query_parameters = http::internal::deprecated_query_parameters(*this);\n    if (split >= param.length() - 1) {\n        sstring key;\n        if (http::internal::url_decode(param.substr(0,split) , key)) {\n            _query_params[key].push_back(\"\");\n            deprecated_query_parameters[key] = \"\";\n        }\n    } else {\n        sstring key;\n        sstring value;\n        if (http::internal::url_decode(param.substr(0,split), key)\n                && http::internal::url_decode(param.substr(split + 1), value)) {\n            deprecated_query_parameters[key] = value;\n            _query_params[key].push_back(std::move(value));\n        }\n    }\n\n}\n\nsstring request::parse_query_param() {\n    http::internal::deprecated_query_parameters(*this).clear();\n    _query_params.clear();\n    size_t pos = _url.find('?');\n    if (pos == sstring::npos) {\n        return _url;\n    }\n    size_t curr = pos + 1;\n    size_t end_param;\n    std::string_view url = _url;\n    while ((end_param = _url.find('&', curr)) != sstring::npos) {\n        add_query_param(url.substr(curr, end_param - curr) );\n        curr = end_param + 1;\n    }\n    add_query_param(url.substr(curr));\n    return _url.substr(0, pos);\n}\n\nvoid request::write_body(const sstring& content_type, sstring content) {\n    set_content_type(content_type);\n    content_length = content.size();\n    _headers[\"Content-Length\"] = to_sstring(content_length);\n    internal::deprecated_content(*this) = std::move(content);\n}\n\nvoid request::write_body(const sstring& content_type, body_writer_type&& body_writer) {\n    set_content_type(content_type);\n    _headers[\"Transfer-Encoding\"] = \"chunked\";\n    this->body_writer = std::move(body_writer);\n}\n\nvoid request::write_body(const sstring& content_type, size_t len, body_writer_type&& body_writer) {\n    set_content_type(content_type);\n    content_length = len;\n    _headers[\"Content-Length\"] = to_sstring(content_length);\n    if (len > 0) {\n        // At the time of this writing, connection::write_body()\n        // assumes that `body_writer` is unset if `content_length` is 0.\n        this->body_writer = std::move(body_writer);\n    }\n}\n\nvoid request::set_expects_continue() {\n    _headers[\"Expect\"] = \"100-continue\";\n}\n\nrequest request::make(sstring method, sstring host, sstring path) {\n    request rq;\n    rq._version = \"1.1\";\n    rq._method = std::move(method);\n    rq._url = std::move(path);\n    rq._headers[\"Host\"] = std::move(host);\n    return rq;\n}\n\nrequest request::make(httpd::operation_type type, sstring host, sstring path) {\n    return make(httpd::type2str(type), std::move(host), std::move(path));\n}\n\n} // http namespace\n} // seastar namespace\n"
  },
  {
    "path": "src/http/request_parser.rl",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/core/ragel.hh>\n#include <memory>\n#include <unordered_map>\n#include <seastar/http/request.hh>\n\nnamespace seastar {\n\n%% machine request;\n\n%%{\n\naccess _fsm_;\n\naction mark {\n    g.mark_start(p);\n}\n\naction store_method {\n    _req->_method = str();\n}\n\naction store_uri {\n    _req->_url = str();\n}\n\naction store_version {\n    _req->_version = str();\n}\n\naction store_field_name {\n    _field_name = str();\n}\n\naction store_value {\n    _value = str();\n}\n\naction trim_trailing_whitespace_and_store_value {\n    _value = str();\n    trim_trailing_spaces_and_tabs(_value);\n    g.mark_start(nullptr);\n}\n\naction assign_field {\n    auto [iter, inserted] = _req->_headers.try_emplace(_field_name, std::move(_value));\n    if (!inserted) {\n        // RFC 7230, section 3.2.2.  Field Parsing:\n        // A recipient MAY combine multiple header fields with the same field name into one\n        // \"field-name: field-value\" pair, without changing the semantics of the message,\n        // by appending each subsequent field value to the combined field value in order, separated by a comma.\n        iter->second += sstring(\",\") + std::move(_value);\n    }\n}\n\naction extend_field  {\n    // RFC 7230, section 3.2.4.  Field Order:\n    // A server that receives an obs-fold in a request message that is not\n    // within a message/http container MUST either reject the message [...]\n    // or replace each received obs-fold with one or more SP octets [...]\n    _req->_headers[_field_name] += sstring(\" \") + std::move(_value);\n}\n\naction done {\n    done = true;\n    fbreak;\n}\n\ncrlf = '\\r\\n';\ntchar = alpha | digit | '-' | '!' | '#' | '$' | '%' | '&' | '\\'' | '*'\n        | '+' | '.' | '^' | '_' | '`' | '|' | '~';\n\nsp = ' ';\nht = '\\t';\n\nsp_ht = sp | ht;\n\nop_char = upper;\n\noperation = op_char+ >mark %store_method;\nuri = (any - sp)+ >mark %store_uri;\nhttp_version = 'HTTP/' (digit '.' digit) >mark %store_version;\n\nobs_text = 0x80..0xFF; # defined in RFC 7230, Section 3.2.6.\nfield_vchar = (graph | obs_text);\n# RFC 9110, Section 5.5 allows single ' '/'\\t' separators between\n# field_vchar words. We are less strict and allow any number of spaces\n# between words.\n# Trailing spaces are trimmed in postprocessing.\nfield_content = (field_vchar | sp_ht)*;\n\nfield = tchar+ >mark %store_field_name;\nvalue = field_content >mark %trim_trailing_whitespace_and_store_value;\nstart_line = ((operation sp uri sp http_version) -- crlf) crlf;\nheader_1st = (field ':' sp_ht* <: value crlf) %assign_field;\nheader_cont = (sp_ht+ <: value crlf) %extend_field;\nheader = header_1st header_cont*;\nmain := start_line header* (crlf @done);\n\n}%%\n\nclass http_request_parser : public ragel_parser_base<http_request_parser> {\n    %% write data nofinal noprefix;\npublic:\n    enum class state {\n        error,\n        eof,\n        done,\n    };\n    std::unique_ptr<http::request> _req;\n    sstring _field_name;\n    sstring _value;\n    state _state;\npublic:\n    void init() {\n        init_base();\n        _req.reset(new http::request());\n        _state = state::eof;\n        %% write init;\n    }\n    char* parse(char* p, char* pe, char* eof) {\n        sstring_builder::guard g(_builder, p, pe);\n        [[maybe_unused]] auto str = [this, &g, &p] { g.mark_end(p); return get_str(); };\n        bool done = false;\n        if (p != pe) {\n            _state = state::error;\n        }\n#ifdef __clang__\n#pragma clang diagnostic push\n#pragma clang diagnostic ignored \"-Wmisleading-indentation\"\n#endif\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wimplicit-fallthrough\"\n        %% write exec;\n#pragma GCC diagnostic pop\n#ifdef __clang__\n#pragma clang diagnostic pop\n#endif\n        if (!done) {\n            if (p == eof) {\n                _state = state::eof;\n            } else if (p != pe) {\n                _state = state::error;\n            } else {\n                p = nullptr;\n            }\n        } else {\n            _state = state::done;\n        }\n        return p;\n    }\n    auto get_parsed_request() {\n        return std::move(_req);\n    }\n    bool eof() const {\n        return _state == state::eof;\n    }\n    bool failed() const {\n        return _state == state::error;\n    }\n};\n\n}\n"
  },
  {
    "path": "src/http/response_parser.rl",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <memory>\n#include <unordered_map>\n\n#include <seastar/core/ragel.hh>\n#include <seastar/http/reply.hh>\n\nnamespace seastar {\n\n\n%% machine reply;\n\n%%{\n\naccess _fsm_;\n\naction mark {\n    g.mark_start(p);\n}\n\naction store_version {\n    _rsp->_version = str();\n}\n\naction store_field_name {\n    _field_name = str();\n}\n\naction store_value {\n    _value = str();\n}\n\naction trim_trailing_whitespace_and_store_value {\n    _value = str();\n    trim_trailing_spaces_and_tabs(_value);\n    g.mark_start(nullptr);\n}\n\naction assign_field {\n    auto [iter, inserted] = _rsp->_headers.try_emplace(_field_name, std::move(_value));\n    if (!inserted) {\n        // RFC 7230, section 3.2.2.  Field Parsing:\n        // A recipient MAY combine multiple header fields with the same field name into one\n        // \"field-name: field-value\" pair, without changing the semantics of the message,\n        // by appending each subsequent field value to the combined field value in order, separated by a comma.\n        iter->second += sstring(\",\") + std::move(_value);\n    }\n}\n\naction extend_field  {\n    // RFC 7230, section 3.2.4.  Field Order:\n    // A server that receives an obs-fold in a request message that is not\n    // within a message/http container MUST either reject the message [...]\n    // or replace each received obs-fold with one or more SP octets [...]\n    _rsp->_headers[_field_name] += sstring(\" \") + std::move(_value);\n}\n\naction store_status {\n    _rsp->_status = static_cast<http::reply::status_type>(std::atoi(str().c_str()));\n}\n\naction done {\n    done = true;\n    fbreak;\n}\n\ncr = '\\r';\nlf = '\\n';\ncrlf = '\\r\\n';\ntchar = alpha | digit | '-' | '!' | '#' | '$' | '%' | '&' | '\\'' | '*'\n        | '+' | '.' | '^' | '_' | '`' | '|' | '~';\n\nsp = ' ';\nht = '\\t';\n\nsp_ht = sp | ht;\n\nhttp_version = 'HTTP/' (digit '.' digit) >mark %store_version;\n\nobs_text = 0x80..0xFF; # defined in RFC 7230, Section 3.2.6.\nfield_vchar = (graph | obs_text);\n# RFC 9110, Section 5.5 allows single ' '/'\\t' separators between\n# field_vchar words. We are less strict and allow any number of spaces\n# between words.\n# Trailing spaces are trimmed in postprocessing.\nfield_content = (field_vchar | sp_ht)*;\n\nfield = tchar+ >mark %store_field_name;\nvalue = field_content >mark %trim_trailing_whitespace_and_store_value;\nstatus_code = (digit digit digit) >mark %store_status;\nstart_line = http_version space status_code space (any - cr - lf)* crlf;\nheader_1st = (field ':' sp_ht* <: value crlf) %assign_field;\nheader_cont = (sp_ht+ <: value crlf) %extend_field;\nheader = header_1st header_cont*;\nmain := start_line header* (crlf @done);\n\n}%%\n\nclass http_response_parser : public ragel_parser_base<http_response_parser> {\n    %% write data nofinal noprefix;\npublic:\n    enum class state {\n        error,\n        eof,\n        done,\n    };\n    std::unique_ptr<http::reply> _rsp;\n    sstring _field_name;\n    sstring _value;\n    state _state;\n    sstring _error_message;\n\npublic:\n    void init() {\n        init_base();\n        _rsp.reset(new http::reply());\n        _state = state::eof;\n        _error_message = {};\n        %% write init;\n    }\n    char* parse(char* p, char* pe, char* eof) {\n        // Save the start pointer to compute offsets for error context.\n        char* start_ptr = p;\n        sstring_builder::guard g(_builder, p, pe);\n        auto str = [this, &g, &p] { g.mark_end(p); return get_str(); };\n        bool done = false;\n        if (p != pe) {\n            _state = state::error;\n        }\n#ifdef __clang__\n#pragma clang diagnostic push\n#pragma clang diagnostic ignored \"-Wmisleading-indentation\"\n#endif\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wimplicit-fallthrough\"\n        %% write exec;\n#pragma GCC diagnostic pop\n#ifdef __clang__\n#pragma clang diagnostic pop\n#endif\n        if (done) {\n            _state = state::done;\n        } else if (p == eof) {\n            _state = state::eof;\n            _error_message = \"Incomplete HTTP response header: reached end-of-file before parsing completed.\";\n        } else if (p != pe) {\n            _state = state::error;\n            // Get the error offset and extract a snippet from the current pointer.\n            size_t offset = static_cast<size_t>(p - start_ptr);\n            size_t available = static_cast<size_t>(pe - p);\n            size_t context_len = std::min(available, 32ul);\n            sstring encountered(p, context_len);\n            _error_message = sstring(\"Parsing error at offset \") + std::to_string(offset) + \": encountered \\\"\" + encountered +\n                                         \"\\\". Expected valid HTTP response header format (e.g., a complete start line with HTTP version, three-digit status code, header \"\n                                         \"fields, and a terminating CRLF).\";\n        } else {\n            p = nullptr;\n        }\n        return p;\n    }\n    auto get_parsed_response() {\n        return std::move(_rsp);\n    }\n    bool eof() const {\n        return _state == state::eof;\n    }\n    bool failed() const {\n        return _state == state::error;\n    }\n    const sstring& error_message() const {\n        return _error_message;\n    }\n};\n}\n"
  },
  {
    "path": "src/http/retry_strategy.cc",
    "content": "/*\n * Copyright (C) 2025-present ScyllaDB\n */\n\n/*\n * SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0\n */\n\n\n#include <chrono>\n#include <coroutine>\n#include <gnutls/gnutls.h>\n\n#include <seastar/http/exception.hh>\n#include <seastar/http/retry_strategy.hh>\n#include <seastar/util/short_streams.hh>\n\n\nusing namespace std::chrono_literals;\n\nnamespace seastar {\nextern logger http_log;\n\nnamespace http::experimental {\nlogger rs_logger(\"default_http_retry_strategy\");\n\nstatic bool is_retryable_exception(std::exception_ptr ex) {\n    while (ex) {\n        try {\n            std::rethrow_exception(ex);\n        } catch (const std::system_error& sys_err) {\n            auto code = sys_err.code().value();\n            if (code == EPIPE || code == ECONNABORTED || code == ECONNRESET || code == GNUTLS_E_PREMATURE_TERMINATION) {\n                return true;\n            }\n            try {\n                std::rethrow_if_nested(sys_err);\n            } catch (...) {\n                ex = std::current_exception();\n                continue;\n            }\n            return false;\n        } catch (const httpd::response_parsing_exception&) {\n            return true;\n        } catch (const std::exception& e) {\n            try {\n                std::rethrow_if_nested(e);\n            } catch (...) {\n                ex = std::current_exception();\n                continue;\n            }\n            return false;\n        } catch (...) {\n            return false;\n        }\n    }\n    return false;\n}\n\ndefault_retry_strategy::default_retry_strategy(unsigned max_retries)\n    : _max_retries(max_retries) {\n}\n\ndefault_retry_strategy::default_retry_strategy()\n    : default_retry_strategy(1) {\n}\n\nfuture<bool> default_retry_strategy::should_retry(std::exception_ptr error, unsigned attempted_retries) const {\n    if (attempted_retries >= _max_retries) {\n        rs_logger.debug(\"Retries exhausted. Retry# {}\", attempted_retries);\n        co_return false;\n    }\n\n    co_return is_retryable_exception(error);\n}\n\nfuture<bool> no_retry_strategy::should_retry(std::exception_ptr error, unsigned) const {\n    co_return false;\n}\n\n}\n}\n"
  },
  {
    "path": "src/http/routes.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#include <seastar/http/routes.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/http/request.hh>\n#include <seastar/http/exception.hh>\n#include <seastar/http/json_path.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nusing namespace std;\n\nroutes::routes() : _general_handler([this](std::exception_ptr eptr) mutable {\n    return exception_reply(eptr);\n}) {}\n\nroutes::~routes() {\n    for (int i = 0; i < NUM_OPERATION; i++) {\n        for (auto kv : _map[i]) {\n            delete kv.second;\n        }\n    }\n    for (int i = 0; i < NUM_OPERATION; i++) {\n        for (auto r : _rules[i]) {\n            delete r.second;\n        }\n    }\n\n}\n\nnamespace internal {\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\nstd::string to_json(const auto& e) {\n    return seastar::httpd::json_exception(e).to_json();\n}\n#pragma GCC diagnostic pop\n} // internal namespace\n\nstd::unique_ptr<http::reply> routes::exception_reply(std::exception_ptr eptr) {\n    auto rep = std::make_unique<http::reply>();\n    try {\n        // go over the register exception handler\n        // if one of them handle the exception, return.\n        for (auto e: _exceptions) {\n            try {\n                return e.second(eptr);\n            } catch (...) {\n                // this is needed if there are more then one register exception handler\n                // so if the exception handler throw a new exception, they would\n                // get the new exception and not the original one.\n                eptr = std::current_exception();\n            }\n        }\n        std::rethrow_exception(eptr);\n    } catch (const redirect_exception& _e) {\n       rep.reset(new http::reply());\n       rep->add_header(\"Location\", _e.url).set_status(_e.status());\n    } catch (const base_exception& e) {\n        rep->set_status(e.status(), internal::to_json(e));\n    } catch (...) {\n        rep->set_status(http::reply::status_type::internal_server_error,\n                internal::to_json(std::current_exception()));\n    }\n\n    rep->done(\"json\");\n    return rep;\n}\n\nfuture<std::unique_ptr<http::reply> > routes::handle(const sstring& path, std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n    handler_base* handler = get_handler(str2type(req->_method),\n            normalize_url(path), req->param);\n    if (handler != nullptr) {\n        try {\n            handler->verify_mandatory_params(*req);\n            auto r =  handler->handle(path, std::move(req), std::move(rep));\n            return r.handle_exception(_general_handler);\n        } catch (...) {\n            rep = exception_reply(std::current_exception());\n        }\n    } else {\n        rep.reset(new http::reply());\n        rep->set_status(http::reply::status_type::not_found, internal::to_json(not_found_exception(\"Not found\"))).done(\n                \"json\");\n    }\n    return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n}\n\nsstring routes::normalize_url(const sstring& url) {\n    if (url.length() < 2 || url.at(url.length() - 1) != '/') {\n        return url;\n    }\n    return url.substr(0, url.length() - 1);\n}\n\nhandler_base* routes::get_handler(operation_type type, const sstring& url,\n        parameters& params) {\n    handler_base* handler = get_exact_match(type, url);\n    if (handler != nullptr) {\n        return handler;\n    }\n\n    for (auto&& rule : _rules[type]) {\n        handler = rule.second->get(url, params);\n        if (handler != nullptr) {\n            return handler;\n        }\n        params.clear();\n    }\n    return _default_handler;\n}\n\nroutes& routes::add(operation_type type, const url& url,\n        handler_base* handler) {\n    match_rule* rule = new match_rule(handler);\n    rule->add_str(url._path);\n    if (url._param != \"\") {\n        rule->add_param(url._param, true);\n    }\n    return add(rule, type);\n}\n\nroutes& routes::add_default_handler(handler_base* handler) {\n    _default_handler = handler;\n    return *this;\n}\n\ntemplate <typename Map, typename Key>\nstatic auto delete_rule_from(operation_type type, Key& key, Map& map) {\n    auto& bucket = map[type];\n    auto ret = bucket.find(key);\n    using ret_type = decltype(ret->second);\n    if (ret != bucket.end()) {\n        ret_type v = ret->second;\n        bucket.erase(ret);\n        return v;\n    }\n    return static_cast<ret_type>(nullptr);\n}\n\nhandler_base* routes::drop(operation_type type, const sstring& url) {\n    return delete_rule_from(type, url, _map);\n}\n\nroutes& routes::put(operation_type type, const sstring& url, handler_base* handler) {\n    auto it = _map[type].emplace(url, handler);\n    if (it.second == false) {\n        throw std::runtime_error(format(\"Handler for {} already exists.\", url));\n    }\n    return *this;\n}\n\nmatch_rule* routes::del_cookie(rule_cookie cookie, operation_type type) {\n    return delete_rule_from(type, cookie, _rules);\n}\n\nvoid routes::add_alias(const path_description& old_path, const path_description& new_path) {\n    httpd::parameters p;\n    stringstream path;\n    path << old_path.path;\n    for (const auto& p : old_path.params) {\n        // the path_description path does not contains the path parameters\n        // so just add a fake parameter to the path for each of the parameters,\n        // and add the string for each fixed string part.\n        if (p.type == path_description::url_component_type::FIXED_STRING) {\n            path << p.name;\n        } else {\n            path << \"/k\";\n        }\n\n    }\n    auto a = get_handler(old_path.operations.method, path.str(), p);\n    if (!a) {\n        throw std::runtime_error(\"routes::add_alias path_description not found: \" + old_path.path);\n    }\n    // if a handler is found then it must be a function_handler\n    new_path.set(*this, new function_handler(*static_cast<function_handler*>(a)));\n}\n\nrule_registration::rule_registration(routes& rs, match_rule& rule, operation_type op)\n        : _routes(rs) , _op(op)\n        , _cookie(_routes.add_cookie(&rule, _op)) {}\n\nrule_registration::~rule_registration() {\n    _routes.del_cookie(_cookie, _op);\n}\n\nhandler_registration::handler_registration(routes& rs, handler_base& h, const sstring& url, operation_type op)\n        : _routes(rs), _url(url), _op(op) {\n    _routes.put(_op, _url, &h);\n}\n\nhandler_registration::~handler_registration() {\n    _routes.drop(_op, _url);\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/transformers.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <boost/algorithm/string/replace.hpp>\n#include <list>\n#include <memory>\n\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/http/transformers.hh>\n\nnamespace seastar {\n\nnamespace httpd {\n\nusing namespace std;\n\nstruct potential_match_entry {\n    const char* begin;\n    const char* end;\n    size_t pos;\n};\n\n/*!\n * \\brief holds the buffer replace object current state\n * The way the matching algorithm works, is that when there's a match\n * it will be the first entry\n */\nclass buffer_replace_state {\n    std::list<potential_match_entry> _potential_match;\n\npublic:\n    using iterator = std::list<potential_match_entry>::iterator;\n\n    void add_potential_match(const char* s, const char* e, size_t pos) {\n        _potential_match.emplace_back(potential_match_entry{s, e, pos});\n    }\n\n    iterator begin() {\n        return _potential_match.begin();\n    }\n\n    iterator end() {\n        return _potential_match.end();\n    }\n\n    bool empty() const {\n        return _potential_match.empty();\n    }\n\n    bool last() const {\n        return _potential_match.size() == 1;\n    }\n\n    auto erase(const iterator& i) {\n        return _potential_match.erase(i);\n    }\n\n    /*!\n     * \\brief gets the key/value position in the buffer_replace of the match\n     */\n    size_t get_pos() const {\n        return (*_potential_match.begin()).pos;\n    }\n\n    /*!\n     * \\brief gets the length of the remaining string\n     */\n    size_t get_remaining_length() const {\n        return _potential_match.begin()->end - _potential_match.begin()->begin;\n    }\n\n    void clear() {\n        _potential_match.clear();\n    }\n};\n\n/*!\n *\\brief a helper class to replace strings in a buffer\n * The keys to replace are surrounded by braces\n */\nclass buffer_replace {\n    std::vector<std::tuple<sstring, sstring>> _values;\n    buffer_replace_state _current;\n    const sstring& get_value(size_t pos) const;\n    const sstring& get_key(size_t pos) const;\npublic:\n    /*!\n     * \\brief Add a key and value to be replaced\n     */\n    buffer_replace& add(sstring key, sstring value) {\n        _values.emplace_back(std::make_tuple(\"{{\" + key + \"}}\", value));\n        return *this;\n    }\n\n    /*!\n     * \\brief if there are no more buffers to consume get\n     * the remaining chars stored in the buffer_replace\n     */\n    temporary_buffer<char> get_remaining();\n\n    /*!\n     * \\brief check if the given buffer still match any of the current potential matches\n     *\n     */\n    temporary_buffer<char> match(temporary_buffer<char>& buf);\n    /*!\n     * \\brief replace the buffer content\n     *\n     * The returned result is after translation. The method consumes what it read\n     * from the buf, so the caller  should check that buf is not empty.\n     *\n     * For example: if buf is: \"abcd{{key}}\"\n     * res = replace(buf);\n     *\n     * res will be \"abcd\"\n     * and buf will be \"{{key}}\"\n     */\n    temporary_buffer<char> replace(temporary_buffer<char>& buf);\n\n    /*!\n     * \\brief check if we are currently in the middle of consuming\n     */\n    bool is_consuming() const {\n        return !_current.empty();\n    }\n};\n\n\nclass content_replace_data_sink_impl : public data_sink_impl {\n    output_stream<char> _out;\n    buffer_replace _br;\npublic:\n    content_replace_data_sink_impl(output_stream<char>&& out, std::vector<std::tuple<sstring,sstring>>&& key_value) : _out(std::move(out)) {\n        for (auto& i : key_value) {\n            _br.add(std::get<0>(i), std::get<1>(i));\n        }\n    }\n\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        return data_sink_impl::fallback_put(bufs, [this] (temporary_buffer<char>&& buf) {\n            return do_put(std::move(buf));\n        });\n    }\n#else\n    virtual future<> put(net::packet data)  override {\n        return make_ready_future<>();\n    }\n\n    using data_sink_impl::put;\n\n    virtual future<> put(temporary_buffer<char> buf) override {\n        return do_put(std::move(buf));\n    }\n#endif\n\nprivate:\n    future<> do_put(temporary_buffer<char> buf) {\n        if (buf.empty()) {\n            return make_ready_future<>();\n        }\n        return do_with(temporary_buffer<char>(std::move(buf)), [this] (temporary_buffer<char>& buf) {\n            return repeat([&buf, this] {\n                auto bf = _br.replace(buf);\n                return _out.write(bf.get(), bf.size()).then([&buf] {\n                    return (buf.empty()) ? stop_iteration::yes : stop_iteration::no;\n                });\n            });\n        });\n    }\n\n    virtual future<> flush() override {\n        return _out.flush();\n    }\n\n    virtual future<> close() override {\n        // if we are in the middle of a consuming a key\n        // there will be no match, write the remaining.\n        if (_br.is_consuming()) {\n            return do_with(temporary_buffer<char>(_br.get_remaining()), [this](temporary_buffer<char>& buf) {\n                return _out.write(buf.get(), buf.size()).then([this] {\n                    return _out.flush();\n                });\n            });\n        }\n        return _out.flush();\n    }\n};\n\nclass content_replace_data_sink : public data_sink {\npublic:\n    content_replace_data_sink(output_stream<char>&& out, std::vector<std::tuple<sstring,sstring>> key_value)\n        : data_sink(std::make_unique<content_replace_data_sink_impl>(\n                std::move(out), std::move(key_value))) {}\n};\n\noutput_stream<char> content_replace::transform(std::unique_ptr<http::request> req,\n            const sstring& extension, output_stream<char>&& s) {\n    sstring host = req->get_header(\"Host\");\n    if (host == \"\" || (this->extension != \"\" && extension != this->extension)) {\n        return std::move(s);\n    }\n    sstring protocol = req->get_protocol_name();\n    output_stream_options opts;\n    opts.trim_to_size = true;\n    return output_stream<char>(content_replace_data_sink(std::move(s), {std::make_tuple(\"Protocol\", protocol), std::make_tuple(\"Host\", host)}), 32000, opts);\n\n}\n\n/*!\n * \\brief find the open brace that surround a parameter\n * it is either two consecutive braces or a single brace, if it's the last char in the buffer\n */\nssize_t find_braces(const char* s, const char* end) {\n    for (size_t i = 0; s != end; s++, i++) {\n        if (*s == '{' && ((s + 1) == end || *(s + 1) == '{')) {\n            return i;\n        }\n    }\n    return -1;\n}\n\nconst sstring& buffer_replace::get_value(size_t pos) const {\n    return std::get<1>(_values[pos]);\n}\n\nconst sstring& buffer_replace::get_key(size_t pos) const {\n    return std::get<0>(_values[pos]);\n}\n\ntemporary_buffer<char> buffer_replace::match(temporary_buffer<char>& buf) {\n    if (_current.empty()) {\n        return temporary_buffer<char>();\n    }\n    auto buf_len = buf.size();\n    auto first = _current.begin();\n    while (first != _current.end()) {\n        auto& pos = first->begin;\n        auto end = first->end;\n        size_t len_compare = std::min(buf_len, static_cast<size_t>(end - pos));\n        if (strncmp(pos, buf.begin(), len_compare)) {\n            // No match remove the entry unless it's the last one\n            // In that case, there is no match\n            // we should return what we consumed so far;\n            if (_current.last()) {\n                auto res = get_remaining();\n                _current.erase(first);\n                return res;\n            }\n            first = _current.erase(first);\n        } else {\n            // we found a match\n            if (pos + len_compare == end) {\n                // this is a full match, there could be only one so this is the first\n                // consume the buffer and return\n                const sstring& value = get_value(_current.get_pos());\n                temporary_buffer<char> res(value.data(), value.size());\n                buf.trim_front(len_compare);\n                _current.clear();\n                return res;\n            }\n            // only partial match\n            pos += len_compare;\n            ++first;\n        }\n    }\n    // if we are here we run out of buffer\n    buf.trim_front(buf_len);\n    return temporary_buffer<char>();\n}\n\ntemporary_buffer<char> buffer_replace::get_remaining() {\n    if (!is_consuming()) {\n        return temporary_buffer<char>();\n    }\n    size_t pos = _current.get_pos();\n    const sstring& key = get_key(pos);\n    auto size = key.size() - _current.get_remaining_length();\n    return temporary_buffer<char>(key.data(), size);\n}\n\ntemporary_buffer<char> buffer_replace::replace(temporary_buffer<char>& buf) {\n    if (buf.empty()) {\n        return std::move(buf);\n    }\n    if (is_consuming()) {\n        return match(buf);\n    }\n    auto start = find_braces(buf.begin(), buf.end());\n    if (start >= 0) {\n        // we found an opening brace that is followed by a second brace or buffer end.\n        // 1. All values are a possible match\n        // 2. We can output the beginning of the buffer\n        // 3. Need to continue matching the remaining buffer\n        size_t pos = 0;\n        for (auto&& i : _values) {\n            sstring& key = std::get<0>(i);\n            _current.add_potential_match(key.data() + 1, key.data() + key.size(), pos++);\n        }\n        temporary_buffer<char> res = buf.share(0, start);\n        buf.trim_front(start + 1);\n        return res;\n    }\n\n    return std::move(buf);\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/http/url.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Scylladb, Ltd.\n */\n\n\n#include <string_view>\n\n#include <seastar/http/url.hh>\n\nnamespace seastar {\nnamespace http {\nnamespace internal {\n\nnamespace {\n\nshort hex_to_byte(char c) {\n    if (c >='a' && c <= 'z') {\n        return c - 'a' + 10;\n    } else if (c >='A' && c <= 'Z') {\n        return c - 'A' + 10;\n    }\n    return c - '0';\n}\n\n/**\n * Convert a hex encoded 2 bytes substring to char\n */\nchar hexstr_to_char(std::string_view in, size_t from) {\n\n    return static_cast<char>(hex_to_byte(in[from]) * 16 + hex_to_byte(in[from + 1]));\n}\n\nbool should_encode(char c) {\n    return !(\n        (c >= 'a' && c <= 'z') ||\n        (c >= 'A' && c <= 'Z') ||\n        (c >= '0' && c <= '9') ||\n        (c == '-' || c == '_' || c == '.' || c == '~')\n    );\n}\n\ninline char char_to_hex(unsigned char val) {\n    return \"0123456789ABCDEF\"[val];\n}\n\nbool decode(bool replace_plus, std::string_view in, sstring& out) {\n    size_t pos = 0;\n    sstring buff(in.length(), 0);\n    for (size_t i = 0; i < in.length(); ++i) {\n        if (in[i] == '%') {\n            if (i + 3 <= in.size()) {\n                buff[pos++] = hexstr_to_char(in, i + 1);\n                i += 2;\n            } else {\n                return false;\n            }\n        } else if (replace_plus && in[i] == '+') {\n            buff[pos++] = ' ';\n        } else {\n            buff[pos++] = in[i];\n        }\n    }\n    buff.resize(pos);\n    out = std::move(buff);\n    return true;\n}\n\n}\n\nbool url_decode(std::string_view in, sstring& out) {\n    return decode(true, in, out);\n}\n\nbool path_decode(std::string_view in, sstring& out) {\n    return decode(false, in, out);\n}\n\n\nsstring url_encode(std::string_view in) {\n    size_t encodable_chars = 0;\n    for (size_t i = 0; i < in.length(); i++) {\n        if (should_encode(in[i])) {\n            encodable_chars++;\n        }\n    }\n\n    if (encodable_chars == 0) {\n        return sstring(in);\n    }\n\n    sstring ret(in.length() + encodable_chars * 2, 0);\n    size_t o = 0;\n    for (size_t i = 0; i < in.length(); i++) {\n        if (should_encode(in[i])) {\n            ret[o++] = '%';\n            ret[o++] = char_to_hex(((unsigned char)in[i]) >> 4);\n            ret[o++] = char_to_hex(in[i] & 0xF);\n        } else {\n            ret[o++] = in[i];\n        }\n    }\n    return ret;\n}\n\n} // internal namespace\n} // http namespace\n} // seastar namespace\n"
  },
  {
    "path": "src/json/formatter.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <cmath>\n#include <algorithm>\n#include <iomanip>\n#include <ios>\n#include <sstream>\n#include <string_view>\n\n#include <seastar/json/formatter.hh>\n#include <seastar/json/json_elements.hh>\n\nnamespace seastar {\n\nusing namespace std;\n\nnamespace json {\n\nsstring formatter::begin(state s) {\n    switch (s) {\n    case state::array: return \"[\";\n    case state::map: return \"{\";\n    default: return {};\n    }\n}\n\nsstring formatter::end(state s) {\n    switch (s) {\n    case state::array: return \"]\";\n    case state::map: return \"}\";\n    default: return {};\n    }\n}\n\nstatic inline bool is_control_char(char c) {\n    return c >= 0 && c <= 0x1F;\n}\n\nstatic bool needs_escaping(const string_view& str) {\n    return std::any_of(str.begin(), str.end(), [] (char c) {\n        return is_control_char(c) || c == '\"' || c == '\\\\';\n    });\n}\n\nstatic sstring string_view_to_json(const string_view& str) {\n    if (!needs_escaping(str)) {\n        return seastar::format(\"\\\"{}\\\"\", str);\n    }\n\n    ostringstream oss;\n    oss << std::hex << std::uppercase << std::setfill('0');\n    oss.put('\"');\n    for (char c : str) {\n        switch (c) {\n        case '\"':\n            oss.put('\\\\').put('\"');\n            break;\n        case '\\\\':\n            oss.put('\\\\').put('\\\\');\n            break;\n        case '\\b':\n            oss.put('\\\\').put('b');\n            break;\n        case '\\f':\n            oss.put('\\\\').put('f');\n            break;\n        case '\\n':\n            oss.put('\\\\').put('n');\n            break;\n        case '\\r':\n            oss.put('\\\\').put('r');\n            break;\n        case '\\t':\n            oss.put('\\\\').put('t');\n            break;\n        default:\n            if (is_control_char(c)) {\n                oss.put('\\\\').put('u') << std::setw(4) << static_cast<int>(c);\n            } else {\n                oss.put(c);\n            }\n            break;\n        }\n    }\n    oss.put('\"');\n    return oss.str();\n}\n\nsstring formatter::to_json(std::string_view str) {\n    return string_view_to_json(str);\n}\n\nsstring formatter::to_json(const char* str) {\n    return string_view_to_json(str);\n}\n\nsstring formatter::to_json(const char* str, size_t len) {\n    return string_view_to_json(string_view{str, len});\n}\n\nsstring formatter::to_json(int n) {\n    return to_string(n);\n}\n\nsstring formatter::to_json(unsigned n) {\n    return to_string(n);\n}\n\nsstring formatter::to_json(long n) {\n    return to_string(n);\n}\n\nsstring formatter::to_json(float f) {\n    if (std::isinf(f)) {\n        throw out_of_range(\"Infinite float value is not supported\");\n    } else if (std::isnan(f)) {\n        throw invalid_argument(\"Invalid float value\");\n    }\n    return to_sstring(f);\n}\n\nsstring formatter::to_json(double d) {\n    if (std::isinf(d)) {\n        throw out_of_range(\"Infinite double value is not supported\");\n    } else if (std::isnan(d)) {\n        throw invalid_argument(\"Invalid double value\");\n    }\n    return to_sstring(d);\n}\n\nsstring formatter::to_json(bool b) {\n    return (b) ? \"true\" : \"false\";\n}\n\nsstring formatter::to_json(const date_time& d) {\n    // use RFC3339/RFC8601 \"internet format\"\n    // which is stipulated as mandatory for swagger\n    // dates\n    // Note that this assumes dates are in UTC timezone\n    static constexpr const char* TIME_FORMAT = \"%FT%TZ\";\n\n    char buff[50];\n    sstring res = \"\\\"\";\n    strftime(buff, 50, TIME_FORMAT, &d);\n    res += buff;\n    return res + \"\\\"\";\n}\n\nsstring formatter::to_json(const jsonable& obj) {\n    return obj.to_json();\n}\n\nsstring formatter::to_json(unsigned long l) {\n    return to_string(l);\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/json/json_elements.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <string.h>\n#include <string>\n#include <vector>\n#include <sstream>\n#include <fmt/core.h>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/print.hh>\n#include <seastar/json/json_elements.hh>\n\nnamespace seastar {\n\nusing namespace std;\n\nnamespace json {\n\n\nclass json_stream_builder;\n/**\n * The json builder is a helper class\n * To help create a json object\n *\n */\nclass json_builder {\npublic:\n    json_builder()\n            : first(true) {\n        result << OPEN;\n    }\n\n    /**\n     * add a name value to an object\n     * @param name the name of the element\n     * @param str the value already formated\n     */\n    void add(const string& name, const string& str) {\n        if (first) {\n            first = false;\n        } else {\n            result << \", \";\n        }\n        result << '\"' << name << \"\\\": \" << str;\n    }\n\n    /**\n     * add a json element to the an object\n     * @param element\n     */\n    void add(json_base_element* element) {\n        if (element == nullptr || element->_set == false) {\n            return;\n        }\n        try {\n            add(element->_name, element->to_string());\n        } catch (...) {\n            std::throw_with_nested(std::runtime_error(fmt::format(\"Json generation failed for field: {}\",element->_name)));\n        }\n    }\n\n    /**\n     * Get the string representation of the object\n     * @return a string of accumulative object\n     */\n    string as_json() {\n        result << CLOSE;\n        return result.str();\n    }\n\nprivate:\n    static const string OPEN;\n    static const string CLOSE;\n    friend class json_stream_builder;\n    stringstream result;\n    bool first;\n\n};\n\n/**\n * The json builder is a helper class\n * To help create a json object\n *\n */\nclass json_stream_builder {\npublic:\n    json_stream_builder(output_stream<char>& s)\n            : first(true), open(false), _s(s) {\n    }\n\n    /**\n     * add a name value to an object\n     * @param name the name of the element\n     * @param str the value already formated\n     */\n    future<> add(const string& name, const json_base_element& element) {\n        if (!open) {\n            open = true;\n            return _s.write(json_builder::OPEN).then([this, &name, &element] {\n                return add(name, element);\n            });\n        }\n        return _s.write(((first)? '\"' + name : \",\\\"\" + name) + \"\\\":\").then([this, &element] {\n            first = false;\n            return element.write(_s);\n        });\n    }\n\n    /**\n     * add a json element to the an object\n     * @param element\n     */\n    future<> add(json_base_element* element) {\n        if (element == nullptr || element->_set == false) {\n            return make_ready_future<>();\n        }\n        return add(element->_name, *element);\n    }\n\n    /**\n     * Get the string representation of the object\n     * @return a string of accumulative object\n     */\n    future<> done() {\n        return _s.write(json_builder::CLOSE);\n    }\n\nprivate:\n\n    bool first;\n    bool open;\n    output_stream<char>& _s;\n};\n\nconst string json_builder::OPEN(\"{\");\nconst string json_builder::CLOSE(\"}\");\n\nvoid json_base::add(json_base_element* element, string name, bool mandatory) {\n    element->_mandatory = mandatory;\n    element->_name = name;\n    _elements.push_back(element);\n}\n\nstring json_base::to_json() const {\n    json_builder res;\n    for (auto i : _elements) {\n        res.add(i);\n    }\n    return res.as_json();\n}\n\nfuture<> json_base::write(output_stream<char>& s) const {\n    return do_with(json_stream_builder(s), [this] (json_stream_builder& builder) {\n        return do_for_each(_elements, [&builder] (auto m) {\n            return builder.add(m);\n        }).then([&builder] {\n            return builder.done();\n        });\n    });\n}\n\nbool json_base::is_verify() const {\n    for (auto i : _elements) {\n        if (!i->is_verify()) {\n            return false;\n        }\n    }\n    return true;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/arp.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/arp.hh>\n\nnamespace seastar {\n\nnamespace net {\n\narp_for_protocol::arp_for_protocol(arp& a, uint16_t proto_num)\n    : _arp(a), _proto_num(proto_num) {\n    _arp.add(proto_num, this);\n}\n\narp_for_protocol::~arp_for_protocol() {\n    _arp.del(_proto_num);\n}\n\narp::arp(interface* netif) : _netif(netif), _proto(netif, eth_protocol_num::arp, [this] { return get_packet(); })\n{\n    // FIXME: ignored future\n    (void)_proto.receive(\n        [this](packet p, ethernet_address ea) {\n            return process_packet(std::move(p), ea);\n        },\n        [this](forward_hash& out_hash_data, packet& p, size_t off) {\n            return forward(out_hash_data, p, off);\n        });\n}\n\nstd::optional<l3_protocol::l3packet> arp::get_packet() {\n    std::optional<l3_protocol::l3packet> p;\n    if (!_packetq.empty()) {\n        p = std::move(_packetq.front());\n        _packetq.pop_front();\n    }\n    return p;\n}\n\nbool arp::forward(forward_hash& out_hash_data, packet& p, size_t off) {\n    auto ah = p.get_header<arp_hdr>(off);\n    auto i = _arp_for_protocol.find(ntoh(ah->ptype));\n    if (i != _arp_for_protocol.end()) {\n        return i->second->forward(out_hash_data, p, off);\n    }\n    return false;\n}\n\nvoid arp::add(uint16_t proto_num, arp_for_protocol* afp) {\n    _arp_for_protocol[proto_num] = afp;\n}\n\nvoid arp::del(uint16_t proto_num) {\n    _arp_for_protocol.erase(proto_num);\n}\n\nfuture<>\narp::process_packet(packet p, ethernet_address from) {\n    auto h = p.get_header(0, arp_hdr::size());\n    if (!h) {\n        return make_ready_future<>();\n    }\n    auto ah = arp_hdr::read(h);\n    auto i = _arp_for_protocol.find(ah.ptype);\n    if (i != _arp_for_protocol.end()) {\n        return i->second->received(std::move(p));\n    }\n    return make_ready_future<>();\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/config.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 Marek Waszkiewicz ( marek.waszkiewicz77@gmail.com )\n */\n\n\n#include <boost/algorithm/cxx11/all_of.hpp>\n#include <boost/algorithm/cxx11/none_of.hpp>\n#include <boost/next_prior.hpp>\n#include <yaml-cpp/yaml.h>\n#include <algorithm>\n#include <istream>\n#include <optional>\n#include <unordered_map>\n#include <string>\n\n#include <seastar/net/config.hh>\n#include <seastar/core/print.hh>\n\nusing namespace boost::algorithm;\n\nnamespace seastar {\nnamespace net {\n\n    // list of supported config keys\n    std::string config_keys[]{ \"pci-address\", \"port-index\", \"ip\", \"gateway\", \"netmask\", \"dhcp\", \"lro\", \"tso\", \"ufo\", \"hw-fc\", \"event-index\", \"csum-offload\",\"ring-size\" };\n\n    std::unordered_map<std::string, device_config>\n    parse_config(std::istream& input) {\n        std::unordered_map<std::string, device_config> device_configs;\n\n        YAML::Node doc = YAML::Load(input);\n        for (auto&& item : doc) {\n            device_configs[item.first.as<std::string>()] = item.second.as<device_config>();\n        }\n\n        bool port_index_used = false;\n        bool pci_address_used = false;\n\n        for (auto&& item : device_configs) {\n\n            if (item.second.hw_cfg.port_index) {\n                port_index_used = true;\n            }\n\n            if (!item.second.hw_cfg.pci_address.empty()) {\n                pci_address_used = true;\n            }\n\n            if (port_index_used && pci_address_used) {\n                throw config_exception(\"port_index and pci_address cannot be used together\");\n            }\n        }\n\n        // check if all of ip,gw,nm are specified when dhcp is off\n        if (all_of(device_configs, [](std::pair<std::string, device_config> p) {\n                return !(!p.second.ip_cfg.dhcp\n                    && (!p.second.ip_cfg.ip.empty() && !p.second.ip_cfg.gateway.empty()\n                           && !p.second.ip_cfg.netmask.empty()));\n            })) {\n            throw config_exception(\n                \"when dhcp is off then all of ip, gateway, netmask has to be specified\");\n        }\n\n        // check if dhcp is not used when ip/gw/nm are specified\n        if (all_of(device_configs, [](std::pair<std::string, device_config> p) {\n                return p.second.ip_cfg.dhcp\n                    && !(p.second.ip_cfg.ip.empty() || p.second.ip_cfg.gateway.empty()\n                           || p.second.ip_cfg.netmask.empty());\n            })) {\n            throw config_exception(\"dhcp and ip cannot be used together\");\n        }\n        return device_configs;\n    }\n}\n}\n\n/// YAML parsing functions\nnamespace YAML {\ntemplate <>\nstruct convert<seastar::net::device_config> {\n    static bool\n    decode(const Node& node, seastar::net::device_config& dev_cfg) {\n        // test for unsupported key\n\n        for (auto&& item : node) {\n            if (none_of(seastar::net::config_keys, [&item](std::string s) {\n                    return s == item.first.as<std::string>();\n                })) {\n                throw seastar::net::config_exception(\n                    seastar::format(\"unsuppoted key {}\", item.first.as<std::string>()));\n            }\n        }\n\n        if (node[\"pci-address\"]) {\n            dev_cfg.hw_cfg.pci_address = node[\"pci-address\"].as<std::string>();\n        }\n\n        if (node[\"port-index\"]) {\n            dev_cfg.hw_cfg.port_index = node[\"port-index\"].as<unsigned>();\n        }\n\n        if (node[\"lro\"]) {\n            dev_cfg.hw_cfg.lro = node[\"lro\"].as<bool>();\n        }\n\n        if (node[\"tso\"]) {\n            dev_cfg.hw_cfg.tso = node[\"tso\"].as<bool>();\n        }\n\n        if (node[\"ufo\"]) {\n            dev_cfg.hw_cfg.ufo = node[\"ufo\"].as<bool>();\n        }\n\n        if (node[\"hw-fc\"]) {\n            dev_cfg.hw_cfg.hw_fc = node[\"hw-fc\"].as<bool>();\n        }\n\n        if (node[\"event-index\"]) {\n            dev_cfg.hw_cfg.event_index = node[\"event-index\"].as<bool>();\n        }\n\n        if (node[\"ring-size\"]) {\n            dev_cfg.hw_cfg.ring_size = node[\"ring-size\"].as<unsigned>();\n        }\n\n        if (node[\"ip\"]) {\n            dev_cfg.ip_cfg.ip = node[\"ip\"].as<std::string>();\n        }\n\n        if (node[\"netmask\"]) {\n            dev_cfg.ip_cfg.netmask = node[\"netmask\"].as<std::string>();\n        }\n\n        if (node[\"gateway\"]) {\n            dev_cfg.ip_cfg.gateway = node[\"gateway\"].as<std::string>();\n        }\n\n        if (node[\"dhcp\"]) {\n            dev_cfg.ip_cfg.dhcp = node[\"dhcp\"].as<bool>();\n        }\n\n        return true;\n    }\n};\n}\n"
  },
  {
    "path": "src/net/dhcp.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n\n#include <chrono>\n#include <unordered_map>\n#include <array>\n#include <random>\n#include <iostream>\n#include <arpa/inet.h>\n\n#include <seastar/net/dhcp.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/udp.hh>\n#include <seastar/net/stack.hh>\n\nnamespace seastar {\n\nusing namespace std::literals::chrono_literals;\n\nclass net::dhcp::impl : public ip_packet_filter {\npublic:\n\n    decltype(std::cout) & log() {\n        return std::cout << \"DHCP \";\n    }\n\n    enum class state {\n        NONE,\n        DISCOVER,\n        REQUEST,\n        DONE,\n        FAIL,\n    };\n\n    enum class m_type : uint8_t {\n        BOOTREQUEST = 1,\n        BOOTREPLY = 2\n    };\n\n    enum class htype : uint8_t {\n        ETHERNET = 1\n    };\n\n    enum class opt_type : uint8_t {\n        PAD = 0,\n        SUBNET_MASK = 1,\n        ROUTER = 3,\n        DOMAIN_NAME_SERVERS = 6,\n        INTERFACE_MTU = 26,\n        BROADCAST_ADDRESS = 28,\n        REQUESTED_ADDRESS = 50,\n        LEASE_TIME = 51,\n        MESSAGE_TYPE = 53,\n        DHCP_SERVER = 54,\n        PARAMETER_REQUEST_LIST = 55,\n        RENEWAL_TIME = 58,\n        REBINDING_TIME = 59,\n        CLASSLESS_ROUTE = 121,\n        END = 255\n    };\n\n    enum class msg_type : uint8_t {\n        DISCOVER = 1,\n        OFFER = 2,\n        REQUEST = 3,\n        DECLINE = 4,\n        ACK = 5,\n        NAK = 6,\n        RELEASE = 7,\n        INFORM = 8,\n        LEASEQUERY = 10,\n        LEASEUNASSIGNED = 11,\n        LEASEUNKNOWN = 12,\n        LEASEACTIVE = 13,\n        INVALID = 255\n    };\n\n    struct dhcp_header {\n        m_type op = m_type::BOOTREQUEST; // Message op code / message type.\n        htype type = htype::ETHERNET;             // Hardware address type\n        uint8_t hlen = 6;           // Hardware address length\n        uint8_t hops = 0;           // Client sets to zero, used by relay agents\n        packed<uint32_t> xid = 0;           // Client sets Transaction ID, a random number\n        packed<uint16_t> secs = 0;          // Client sets seconds elapsed since op start\n        packed<uint16_t> flags = 0;         // Flags\n        ipv4_address ciaddr;  // Client IP address\n        ipv4_address yiaddr;  // 'your' (client) IP address.\n        ipv4_address siaddr;  // IP address of next server to use in bootstrap\n        ipv4_address giaddr;  // Relay agent IP address\n        uint8_t chaddr[16] = { 0, };     // Client hardware address.\n        char sname[64] = { 0, };         // unused\n        char file[128] = { 0, };         // unused\n\n        template <typename Adjuster>\n        auto adjust_endianness(Adjuster a) {\n            return a(xid, secs, flags, ciaddr, yiaddr, siaddr, giaddr);\n        }\n    } __attribute__((packed));\n\n    typedef std::array<opt_type, 5> req_opt_type;\n\n    static const req_opt_type requested_options;\n\n    struct option_mark {\n        option_mark(opt_type t = opt_type::END) : type(t) {};\n        opt_type type;\n    } __attribute__((packed));\n\n    struct option : public option_mark {\n        option(opt_type t, uint8_t l = 1) : option_mark(t), len(l) {};\n        uint8_t len;\n    } __attribute__((packed));\n\n    struct type_option : public option {\n        type_option(msg_type t) : option(opt_type::MESSAGE_TYPE), type(t) {}\n        msg_type type;\n    } __attribute__((packed));\n\n    struct mtu_option : public option {\n        mtu_option(uint16_t v) : option(opt_type::INTERFACE_MTU, 2), mtu((::htons)(v)) {}\n        packed<uint16_t> mtu;\n    } __attribute__((packed));\n\n    struct ip_option : public option {\n        ip_option(opt_type t = opt_type::BROADCAST_ADDRESS, const ipv4_address & ip = ipv4_address()) : option(t, sizeof(uint32_t)), ip(htonl(ip.ip)) {}\n        packed<uint32_t> ip;\n    } __attribute__((packed));\n\n    struct time_option : public option {\n        time_option(opt_type t, uint32_t v) : option(t, sizeof(uint32_t)), time(htonl(v)) {}\n        packed<uint32_t> time;\n    } __attribute__((packed));\n\n\n    struct requested_option: public option {\n        requested_option()\n                : option(opt_type::PARAMETER_REQUEST_LIST,\n                        uint8_t(requested_options.size())), req(\n                        requested_options) {\n        }\n        req_opt_type req;\n    }__attribute__((packed));\n\n    static const uint16_t client_port = 68;\n    static const uint16_t server_port = 67;\n\n    typedef std::array<uint8_t, 4> magic_tag;\n\n    static const magic_tag options_magic;\n\n    struct dhcp_payload {\n        dhcp_header bootp;\n        magic_tag magic = options_magic;\n\n        template <typename Adjuster>\n        auto adjust_endianness(Adjuster a) {\n            return a(bootp);\n        }\n    } __attribute__((packed));\n\n    struct dhcp_packet_base {\n        dhcp_payload dhp;\n\n        template <typename Adjuster>\n        auto adjust_endianness(Adjuster a) {\n            return a(dhp);\n        }\n    } __attribute__((packed));\n\n    struct ip_info : public lease {\n        msg_type type = msg_type();\n\n        void set(opt_type type, const ipv4_address & ip) {\n            switch (type) {\n            case opt_type::SUBNET_MASK: netmask = ip; break;\n            case opt_type::ROUTER: gateway = ip; break;\n            case opt_type::BROADCAST_ADDRESS: broadcast = ip; break;\n            case opt_type::DHCP_SERVER: dhcp_server = ip; break;\n            case opt_type::DOMAIN_NAME_SERVERS:\n                name_servers.emplace_back(ip);\n                break;\n            default:\n                break;\n            }\n        }\n\n        void set(opt_type type, std::chrono::seconds s) {\n            switch (type) {\n            case opt_type::LEASE_TIME: lease_time = s; break;\n            case opt_type::RENEWAL_TIME: renew_time = s; break;\n            case opt_type::REBINDING_TIME: rebind_time = s; break;\n            default:\n                break;\n            }\n        }\n\n        void parse_options(packet & p, size_t off) {\n            for (;;) {\n                auto * m = p.get_header<option_mark>(off);\n                if (m == nullptr || m->type == opt_type::END) {\n                    break;\n                }\n                auto * o = p.get_header<option>(off);\n                if (o == nullptr) {\n                    // TODO: report broken packet?\n                    break;\n                }\n\n                switch (o->type) {\n                case opt_type::SUBNET_MASK:\n                case opt_type::ROUTER:\n                case opt_type::BROADCAST_ADDRESS:\n                case opt_type::DHCP_SERVER:\n                case opt_type::DOMAIN_NAME_SERVERS:\n                {\n                    auto ipo = p.get_header<ip_option>(off);\n                    if (ipo != nullptr) {\n                        set(o->type, ipv4_address(ntohl(ipo->ip)));\n                    }\n                }\n                break;\n                case opt_type::MESSAGE_TYPE:\n                {\n                    auto to = p.get_header<type_option>(off);\n                    if (to != nullptr) {\n                        type = to->type;\n                    }\n                }\n                break;\n                case opt_type::INTERFACE_MTU:\n                {\n                    auto mo = p.get_header<mtu_option>(off);\n                    if (mo != nullptr) {\n                        mtu = (::ntohs)(uint16_t(mo->mtu));\n                    }\n                }\n                break;\n                case opt_type::LEASE_TIME:\n                case opt_type::RENEWAL_TIME:\n                case opt_type::REBINDING_TIME:\n                {\n                    auto to = p.get_header<time_option>(off);\n                    if (to != nullptr) {\n                        set(o->type, std::chrono::seconds(ntohl(to->time)));\n                    }\n                }\n                break;\n                default:\n                    break;\n                }\n\n                off += sizeof(*o) + o->len;\n            }\n        }\n    };\n\n    impl(ipv4 & stack)\n       : _stack(stack)\n    {\n        _sock = _stack.get_udp().make_channel({0, client_port});\n    }\n\n    future<> process_packet(packet p, dhcp_payload* dhp, size_t opt_off) {\n        _retry_timer.cancel();\n\n        auto h = ntoh(*dhp);\n\n        ip_info info;\n\n        info.ip = h.bootp.yiaddr;\n        info.parse_options(p, opt_off);\n\n        switch (_state) {\n        case state::DISCOVER:\n            if (info.type != msg_type::OFFER) {\n                // TODO: log?\n                break;\n            }\n            log() << \"Got offer for \" << info.ip << std::endl;\n            // TODO, check for minimum valid/required fields sent back?\n            return send_request(info);\n        case state::REQUEST:\n            if (info.type == msg_type::NAK) {\n                log() << \"Got nak on request\" << std::endl;\n                _state = state::NONE;\n                return send_discover();\n            }\n            if (info.type != msg_type::ACK) {\n                break;\n            }\n            log() << \"Got ack on request\" << std::endl;\n            log() << \" ip: \" << info.ip << std::endl;\n            log() << \" nm: \" << info.netmask << std::endl;\n            log() << \" gw: \" << info.gateway << std::endl;\n            _state = state::DONE;\n            _result.set_value(info);\n            break;\n        default:\n            break;\n        }\n        return make_ready_future<>();\n    }\n\n    future<> handle(packet& p, ip_hdr* iph, ethernet_address from, bool & handled) override {\n        if (_state == state::NONE || p.len() < sizeof(dhcp_packet_base)) {\n            return make_ready_future<>();\n        }\n\n        auto ipl = iph->ihl * 4;\n        auto udp = p.get_header<udp_hdr>(ipl);\n        auto dhp = p.get_header<dhcp_payload>(ipl + sizeof(*udp));\n\n        const auto opt_off = ipl + sizeof(*udp) + sizeof(dhcp_payload);\n\n        if (udp == nullptr || dhp == nullptr\n                || iph->ip_proto != uint8_t(ip_protocol_num::udp)\n                || (::ntohs)(udp->dst_port) != client_port\n                || iph->len < (opt_off + sizeof(option_mark))\n                || dhp->magic != options_magic) {\n            return make_ready_future<>();\n        }\n        handled = true;\n        auto src_cpu = this_shard_id();\n        if (src_cpu == 0) {\n            return process_packet(std::move(p), dhp, opt_off);\n        }\n        // FIXME: future is discarded\n        (void)smp::submit_to(0, [this, p = std::move(p), src_cpu, dhp, opt_off]() mutable {\n            return process_packet(p.free_on_cpu(src_cpu), dhp, opt_off);\n        });\n        return make_ready_future<>();\n    }\n\n    future<std::optional<lease>> run(const lease & l,\n            const steady_clock_type::duration & timeout) {\n\n        _state = state::NONE;\n        _timer.set_callback([this]() {\n            _state = state::FAIL;\n            log() << \"timeout\" << std::endl;\n            _retry_timer.cancel();\n            _result.set_value(std::nullopt);\n        });\n\n        log() << \"sending discover\" << std::endl;\n        (void)send_discover(l.ip); // FIXME: ignoring return\n        if (timeout.count()) {\n            _timer.arm(timeout);\n        }\n        _retry_timer.set_callback([this, l] {\n            // FIXME: ignoring return\n            (void)send_discover(l.ip);\n        });\n        _retry_timer.arm_periodic(1s);\n        return _result.get_future();\n    }\n\n    template<typename T>\n    future<> send(T && pkt) {\n        pkt.dhp.bootp.xid = _xid;\n        auto ipf = _stack.netif();\n        auto mac = ipf->hw_address().mac;\n        std::copy(mac.begin(), mac.end(), std::begin(pkt.dhp.bootp.chaddr));\n\n        pkt = hton(pkt);\n\n        // FIXME: future is discarded\n        temporary_buffer<char> buf(reinterpret_cast<char *>(&pkt), sizeof(pkt));\n        (void)_sock.send({0xffffffff, server_port}, std::span(&buf, 1));\n\n        return make_ready_future<>();\n    }\n\n    future<> send_discover(const ipv4_address & ip = ipv4_address()) {\n        struct discover : public dhcp_packet_base {\n            type_option type = type_option(msg_type::DISCOVER);\n            ip_option requested_ip;\n            requested_option req;\n            option_mark end;\n        } __attribute__((packed));\n\n        discover d;\n\n        d.requested_ip = ip_option(opt_type::REQUESTED_ADDRESS, ip);\n\n        std::random_device rd;\n        std::default_random_engine e1(rd());\n        std::uniform_int_distribution<uint32_t> xid_dist{};\n\n        _xid = xid_dist(e1);\n        _state = state::DISCOVER;\n        return send(d);\n    }\n\n    future<> send_request(const lease & info) {\n        struct request : public dhcp_packet_base {\n            type_option type = type_option(msg_type::REQUEST);\n            ip_option dhcp_server;\n            ip_option requested_ip;\n            requested_option req;\n            option_mark end;\n        } __attribute__((packed));\n\n        request d;\n\n        d.dhcp_server = ip_option(opt_type::DHCP_SERVER, info.dhcp_server);\n        d.requested_ip = ip_option(opt_type::REQUESTED_ADDRESS, info.ip);\n\n        log() << \"sending request for \" << info.ip << std::endl;\n        _state = state::REQUEST;\n        return send(d);\n    }\n\nprivate:\n    promise<std::optional<lease>> _result;\n    state _state = state::NONE;\n    timer<> _timer;\n    timer<> _retry_timer;\n    ipv4 & _stack;\n    udp_channel _sock;\n    uint32_t _xid = 0;\n};\n\nconst net::dhcp::impl::req_opt_type net::dhcp::impl::requested_options = { {\n        opt_type::SUBNET_MASK, opt_type::ROUTER, opt_type::DOMAIN_NAME_SERVERS,\n        opt_type::INTERFACE_MTU, opt_type::BROADCAST_ADDRESS } };\n\nconst net::dhcp::impl::magic_tag net::dhcp::impl::options_magic = { { 0x63, 0x82, 0x53,\n        0x63 } };\n\nconst uint16_t net::dhcp::impl::client_port;\nconst uint16_t net::dhcp::impl::server_port;\n\nconst steady_clock_type::duration net::dhcp::default_timeout = std::chrono::duration_cast<steady_clock_type::duration>(std::chrono::seconds(30));\n\nnet::dhcp::dhcp(ipv4 & ip)\n: _impl(std::make_unique<impl>(ip))\n{}\n\nnet::dhcp::dhcp(dhcp && v) noexcept = default;\n\nnet::dhcp::~dhcp()\n{}\n\nnet::dhcp::result_type net::dhcp::discover(const steady_clock_type::duration & timeout) {\n    return _impl->run(lease(), timeout);\n}\n\nnet::dhcp::result_type net::dhcp::renew(const lease & l, const steady_clock_type::duration & timeout) {\n    return _impl->run(l, timeout);\n}\n\nnet::ip_packet_filter* net::dhcp::get_ipv4_filter() {\n    return _impl.get();\n}\n\n}\n"
  },
  {
    "path": "src/net/dns.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 Cloudius Systems\n */\n\n/*\n * DNS resolver implementation using c-ares\n *\n * Version compatibility matrix:\n * - c-ares >= 1.34.0 (ARES_VERSION >= 0x012200):\n *   * Uses ares_set_socket_functions_ex (non-deprecated)\n *   * Uses ares_process_fds directly without FD polling (no ares_getsock/ares_fds)\n *   * Uses ares_query_dnsrec for SRV records (if >= 1.28.0)\n *   * Zero deprecation warnings\n *\n * - c-ares >= 1.19.0 (ARES_VERSION >= 0x011300):\n *   * Uses ARES_OPT_SOCK_STATE_CB for event-driven socket monitoring (when available)\n *   * Works WITH custom socket functions (proven through testing)\n *   * Eliminates polling-based socket monitoring entirely\n *\n * - c-ares 1.13.0 - 1.33.x (ARES_VERSION >= 0x010D00 && < 0x012200):\n *   * Uses ares_set_socket_functions (deprecated in 1.34.0+)\n *   * Uses ares_getsock for FD polling\n *   * Uses ares_process for event processing\n *   * Uses ares_query_dnsrec for SRV records (if >= 1.28.0)\n */\n\n#include <arpa/nameser.h>\n#include <chrono>\n#include <memory>\n\n#include <ares.h>\n#include <boost/lexical_cast.hpp>\n\n#include <ostream>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/lazy.hh>\n#include <seastar/net/inet_address.hh>\n\n#include <seastar/net/ip.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/print.hh>\n\nnamespace seastar::net {\n\n// NOTE: Should be prior to <seastar/util/log.hh> include because\n// logger::stringer_for<T> needs to see the corresponding `operator <<`\n// declaration at the call site\n//\n// This doesn't need to be in the public API, so leave it there instead of placing into `inet_address.hh`\nstd::ostream& operator<<(std::ostream& os, const opt_family& f) {\n    if (f) {\n        return os << *f;\n    } else {\n        return os << \"ANY\";\n    }\n}\n\n}\n\n#if (ARES_VERSION < 0x011600)\n// ares_channel_t is only present since c-ares 1.22.0 (November 2023)\ntypedef struct ares_channeldata ares_channel_t;\n#endif\n\ntemplate <> struct fmt::formatter<seastar::net::opt_family> : fmt::ostream_formatter {};\n\n#include <seastar/util/log.hh>\n\nnamespace seastar::net {\n\nstatic logger dns_log(\"dns_resolver\");\n\nclass ares_error_category : public std::error_category {\npublic:\n    constexpr ares_error_category() noexcept : std::error_category{} {}\n    const char * name() const noexcept {\n        return \"C-Ares\";\n    }\n    std::string message(int error) const {\n        switch (error) {\n            /* Server error codes (ARES_ENODATA indicates no relevant answer) */\n            case ARES_ENODATA: return \"No data\";\n            case ARES_EFORMERR: return \"Form error\";\n            case ARES_ESERVFAIL: return \"Server failure\";\n            case ARES_ENOTFOUND: return \"Not found\";\n            case ARES_ENOTIMP: return \"Not implemented\";\n            case ARES_EREFUSED: return \"Refused\";\n\n            /* Locally generated error codes */\n            case ARES_EBADQUERY: return \"Bad query\";\n            case ARES_EBADNAME: return \"Bad name\";\n            case ARES_EBADFAMILY: return \"Bad family\";\n            case ARES_EBADRESP: return \"Bad response\";\n            case ARES_ECONNREFUSED :return \"Connection refused\";\n            case ARES_ETIMEOUT: return \"Timeout\";\n            case ARES_EOF: return \"EOF\";\n            case ARES_EFILE: return \"File error\";\n            case ARES_ENOMEM: return \"No memory\";\n            case ARES_EDESTRUCTION: return \"Destruction\";\n            case ARES_EBADSTR: return \"Bad string\";\n\n            /* ares_getnameinfo error codes */\n            case ARES_EBADFLAGS: return \"Invalid flags\";\n\n            /* ares_getaddrinfo error codes */\n            case ARES_ENONAME: return \"No name\";\n            case ARES_EBADHINTS: return \"Bad hints\";\n\n            /* Uninitialized library error code */\n            case ARES_ENOTINITIALIZED: return \"Not initialized\";\n\n            /* ares_library_init error codes */\n            case ARES_ELOADIPHLPAPI: return \"Load PHLPAPI\";\n            case ARES_EADDRGETNETWORKPARAMS: return \"Get network parameters\";\n\n            /* More error codes */\n            case ARES_ECANCELLED: return \"Cancelled\";\n            default:\n            return \"Unknown error\";\n        }\n    }\n};\n\nstatic void check_ares_error(int error) {\n    if (error != ARES_SUCCESS) {\n        throw std::system_error(error, dns::error_category());\n    }\n}\n\nstruct ares_initializer {\n    ares_initializer() {\n        check_ares_error(ares_library_init(ARES_LIB_INIT_NONE));\n    }\n    ~ares_initializer() {\n        ares_library_cleanup();\n    }\n};\n\nclass dns_resolver::impl\n    : public enable_shared_from_this<impl>\n{\npublic:\n    impl(network_stack& stack, const options& opts);\n    ~impl();\n    future<inet_address> resolve_name(sstring name, opt_family family);\n    future<hostent> get_host_by_name(sstring name, opt_family family);\n    future<hostent> get_host_by_addr(inet_address addr);\n    future<srv_records> get_srv_records(srv_proto proto,\n                                        const sstring& service,\n                                        const sstring& domain);\n    future<sstring> resolve_addr(inet_address addr);\n    future<> close();\nprivate:\n    enum class type {\n        none, tcp, udp\n    };\n    struct dns_call {\n        dns_call(impl & i)\n            : _i(i)\n            , _c(++i._calls)\n        {}\n        ~dns_call() {\n            // If a query does not immediately complete\n            // it might never do so, unless data actually\n            // comes back to us and a waiting recv promise\n            // is fulfilled.\n            // We need to add a timer to do polling at ~timeout\n            // ms later, so the ares logic can detect this and\n            // tell us we're over.\n            if (_c == 1 && _i._calls != 0) {\n                _i._timer.arm_periodic(_i._timeout);\n            }\n        }\n        impl& _i;\n        uint64_t _c;\n    };\n\n    void end_call();\n    void poll_sockets();\n    static srv_records make_srv_records(ares_srv_reply* start);\n    static hostent make_hostent(const ares_addrinfo* ai);\n    static hostent make_hostent(const ::hostent& host);\n\n    // We need to partially ref-count our socket entries\n    // when we have pending reads/writes, so we don't erase the\n    // entry to early.\n    void use(ares_socket_t fd);\n    void release(ares_socket_t fd);\n\n    ares_socket_t do_socket(int af, int type, int protocol);\n    int do_close(ares_socket_t fd);\n    socket_address sock_addr(const sockaddr * addr, socklen_t len);\n    int do_connect(ares_socket_t fd, const sockaddr * addr, socklen_t len);\n    ssize_t do_recvfrom(ares_socket_t fd, void * dst, size_t len, int flags, struct sockaddr * from, socklen_t * from_len);\n#if ARES_VERSION >= 0x012200\n    ssize_t do_send(ares_socket_t fd, const void * buf, size_t bytes);\n#else\n    ssize_t do_sendv(ares_socket_t fd, const iovec * vec, int len);\n#endif\n\n    // Note: cannot use to much here, because fd_sets only handle\n    // ~1024 fd:s. Set to something below that in case you need to\n    // debug (maybe)\n    static constexpr ares_socket_t socket_offset = 1;\n\n    ares_socket_t next_fd();\n    struct tcp_entry {\n        tcp_entry(connected_socket s)\n                        : socket(std::move(s)) {\n        }\n        ;\n        connected_socket socket;\n        std::optional<input_stream<char>> in;\n        std::optional<data_sink> out;\n        temporary_buffer<char> indata;\n    };\n    struct udp_entry {\n        udp_entry(datagram_channel c)\n                        : channel(std::move(c)) {\n        }\n        datagram_channel channel;\n        std::optional<datagram> in;;\n        socket_address dst;\n        future<> f = make_ready_future<>();\n    };\n    struct sock_entry {\n        union {\n            tcp_entry tcp;\n            udp_entry udp;\n        };\n        type typ;\n        int avail = 0;\n        int pending = 0;\n        bool closed = false;\n\n        sock_entry(sock_entry&& e)\n            : typ(e.typ)\n            , avail(e.avail)\n        {\n            e.typ = type::none;\n            switch (typ) {\n            case type::tcp:\n                new (&tcp) tcp_entry(std::move(e.tcp));\n                break;\n            case type::udp:\n                new (&udp) udp_entry(std::move(e.udp));\n                break;\n            default:\n                break;\n            }\n        }\n        sock_entry(connected_socket s)\n            : tcp(tcp_entry{std::move(s)})\n            , typ(type::tcp)\n        {}\n        sock_entry(datagram_channel c)\n            : udp(udp_entry{std::move(c)})\n            , typ(type::udp)\n        {}\n        ~sock_entry() {\n            switch (typ) {\n            case type::tcp: tcp.~tcp_entry(); break;\n            case type::udp: udp.~udp_entry(); break;\n            default: break;\n            }\n        }\n    };\n\n#if ARES_VERSION >= 0x012200\n    using send_packet_t = temporary_buffer<char>;\n#else\n    using send_packet_t = std::vector<temporary_buffer<char>>;\n#endif\n    ssize_t do_send_tcp(sock_entry& e, send_packet_t p, size_t len, ares_socket_t fd);\n    ssize_t do_send_udp(sock_entry& e, send_packet_t p, size_t len, ares_socket_t fd);\n\n    sock_entry& get_socket_entry(ares_socket_t fd);\n\n    using socket_map = std::unordered_map<ares_socket_t, sock_entry>;\n\n    friend struct dns_call;\n\n    socket_map _sockets;\n    network_stack & _stack;\n\n    ares_channel_t* _channel;\n    uint64_t _calls = 0;\n    std::chrono::milliseconds _timeout;\n    timer<> _timer;\n    gate _gate;\n    bool _closed = false;\n\n#if ARES_VERSION >= 0x011300  // c-ares 1.19.0+ supports ARES_OPT_SOCK_STATE_CB\n    void handle_socket_state_change(ares_socket_t fd, int readable, int writable);\n\n    // Track which sockets need monitoring\n    struct socket_monitor {\n        bool wants_read = false;\n        bool wants_write = false;\n    };\n    std::unordered_map<ares_socket_t, socket_monitor> _socket_monitors;\n#endif\n};\n\ndns_resolver::impl::impl(network_stack& stack, const options& opts)\n    : _stack(stack)\n    , _timeout(opts.timeout ? *opts.timeout : std::chrono::milliseconds(5000) /* from ares private */)\n    , _timer(std::bind(&impl::poll_sockets, this))\n{\n    static const ares_initializer a_init;\n\n    // this can \"block\" ever so slightly, because it will\n    // look in resolv.conf etc for query setup. We could\n    // do this ourselves, and instead set ares options\n    // here, but it seems more error prone (me parsing\n    // resolv.conf -> hah!)\n    ares_options a_opts = {};\n\n    // For now, use the default \"fb\" query order\n    // (set explicitly lest we forget).\n    // We only do querying dns server really async.\n    // Reading hosts files is doen by c-ares internally\n    // and with normal fread calls. Thus they theorectically\n    // block. This can potentially be an issue for some application\n    // and if so, we need to revisit this. For now, assume\n    // it won't block us in any measurable way.\n    char buf[3] = \"fb\";\n    a_opts.lookups = buf; // only net\n    // Always set the timeout\n    a_opts.timeout = _timeout.count();\n    int flags = ARES_OPT_LOOKUPS|ARES_OPT_TIMEOUTMS;\n\n    static auto get_impl = [](void * p) { return reinterpret_cast<impl *>(p); };\n\n#if ARES_VERSION >= 0x011300  // c-ares 1.19.0+ supports ARES_OPT_SOCK_STATE_CB\n    // Use socket state callback for event-driven monitoring\n    // This works WITH custom socket functions (proven through testing)\n    a_opts.sock_state_cb = [](void* p, ares_socket_t s, int readable, int writable) {\n      return get_impl(p)->handle_socket_state_change(s, readable, writable);\n    };\n    a_opts.sock_state_cb_data = this;\n    flags |= ARES_OPT_SOCK_STATE_CB;\n#endif\n\n    if (opts.use_tcp_query && *opts.use_tcp_query) {\n        a_opts.flags = ARES_FLAG_USEVC | ARES_FLAG_PRIMARY;\n        flags |= ARES_OPT_FLAGS;\n    }\n    std::vector<in_addr> addr_tmp;\n    if (opts.servers) {\n        std::transform(opts.servers->begin(), opts.servers->end(), std::back_inserter(addr_tmp), [](const inet_address& a) {\n            if (a.in_family() != inet_address::family::INET) {\n                throw std::invalid_argument(\"Servers must be ipv4 addresses\");\n            }\n            in_addr in = a;\n            return in;\n        });\n        a_opts.servers = addr_tmp.data();\n        a_opts.nservers = int(addr_tmp.size());\n        flags |= ARES_OPT_SERVERS;\n    }\n    std::vector<const char *> dom_tmp;\n    if (opts.domains) {\n        std::transform(opts.domains->begin(), opts.domains->end(), std::back_inserter(dom_tmp), [](const sstring& s) {\n            return s.data();\n        });\n        a_opts.domains = const_cast<char **>(dom_tmp.data());\n        a_opts.ndomains = int(dom_tmp.size());\n        flags |= ARES_OPT_DOMAINS;\n    }\n    if (opts.tcp_port) {\n        a_opts.tcp_port = *opts.tcp_port;\n        flags |= ARES_OPT_TCP_PORT;\n    }\n    if (opts.udp_port) {\n        a_opts.udp_port = *opts.udp_port;\n        flags |= ARES_OPT_UDP_PORT;\n    }\n\n    check_ares_error(ares_init_options(&_channel, &a_opts, flags));\n\n    // Set up custom socket functions to integrate with Seastar's networking stack\n    // Note: These work together with ARES_OPT_SOCK_STATE_CB (when available)\n#if ARES_VERSION >= 0x012200  // ares_set_socket_functions_ex available since 1.34.0\n    // Use the new extended socket functions API to avoid deprecation warning\n    static const ares_socket_functions_ex callbacks_ex = {\n        .version = 1,  // Required ABI version\n        .flags = ARES_SOCKFUNC_FLAG_NONBLOCKING,  // Our sockets are always non-blocking\n        .asocket = [](int af, int type, int protocol, void * p) {\n            return get_impl(p)->do_socket(af, type, protocol);\n        },\n        .aclose = [](ares_socket_t s, void * p) {\n            return get_impl(p)->do_close(s);\n        },\n        .asetsockopt = [](ares_socket_t s, ares_socket_opt_t opt, const void * val, ares_socklen_t val_size, void * p) {\n            // No-op: c-ares explicitly handles ENOSYS as \"intentionally not supported\" per API docs.\n            // Socket management (buffer sizes, etc.) is handled by Seastar's networking stack.\n            // These options are optional and not configured in Seastar's DNS resolver anyway.\n            dns_log.trace(\"c-ares socket option request: fd={} opt={} size={}\", s, static_cast<int>(opt), val_size);\n            errno = ENOSYS;\n            return -1;\n        },\n        .aconnect = [](ares_socket_t s, const struct sockaddr * addr, ares_socklen_t len, unsigned int flags, void * p) {\n            // flags parameter is currently unused by our implementation\n            return get_impl(p)->do_connect(s, addr, len);\n        },\n        .arecvfrom = [](ares_socket_t s, void * dst, size_t len, int flags, struct sockaddr * addr, ares_socklen_t * alen, void * p) {\n            return static_cast<ares_ssize_t>(get_impl(p)->do_recvfrom(s, dst, len, flags, addr, alen));\n        },\n        .asendto = [](ares_socket_t s, const void * buffer, size_t len, int flags, const struct sockaddr * addr, ares_socklen_t addrlen, void * p) {\n            return static_cast<ares_ssize_t>(get_impl(p)->do_send(s, buffer, len));\n        },\n        .agetsockname = nullptr,  // Not needed\n        .abind = nullptr,  // Not needed\n        .aif_nametoindex = nullptr,  // Not needed\n        .aif_indextoname = nullptr,  // Not needed\n    };\n\n    ares_status_t status = ares_set_socket_functions_ex(_channel, &callbacks_ex, this);\n    if (status != ARES_SUCCESS) {\n        throw std::system_error(status, dns::error_category());\n    }\n#else\n    // Use the older API for compatibility with c-ares < 1.34.0\n    static const ares_socket_functions callbacks = {\n        [](int af, int type, int protocol, void * p) { return get_impl(p)->do_socket(af, type, protocol); },\n        [](ares_socket_t s, void * p) { return get_impl(p)->do_close(s); },\n        [](ares_socket_t s, const struct sockaddr * addr, socklen_t len, void * p) { return get_impl(p)->do_connect(s, addr, len); },\n        [](ares_socket_t s, void * dst, size_t len, int flags, struct sockaddr * addr, socklen_t * alen, void * p) {\n            return get_impl(p)->do_recvfrom(s, dst, len, flags, addr, alen);\n        },\n        [](ares_socket_t s, const struct iovec * vec, int len, void * p) {\n            return get_impl(p)->do_sendv(s, vec, len);\n        },\n    };\n\n    ares_set_socket_functions(_channel, &callbacks, this);\n#endif\n\n    // just in case you need printf-debug.\n    // dns_log.set_level(log_level::trace);\n}\n\ndns_resolver::impl::~impl() {\n    _timer.cancel();\n    if (_channel) {\n        ares_destroy(_channel);\n    }\n}\n\n#if ARES_VERSION >= 0x011300  // c-ares 1.19.0+ supports ARES_OPT_SOCK_STATE_CB\nvoid dns_resolver::impl::handle_socket_state_change(ares_socket_t fd, int readable, int writable) {\n    dns_log.trace(\"Socket state change: fd={} readable={} writable={}\", fd, readable, writable);\n\n    auto it = _sockets.find(fd);\n\n    if (!readable && !writable) {\n        // Socket is being closed by c-ares\n        dns_log.trace(\"c-ares closing socket {}\", fd);\n\n        // Update our monitoring state\n        _socket_monitors.erase(fd);\n\n        // Note: The actual socket cleanup is handled by do_close()\n        // which c-ares will call through our custom socket functions\n    } else {\n        // c-ares wants us to monitor this socket\n        auto& monitor = _socket_monitors[fd];\n        monitor.wants_read = readable;\n        monitor.wants_write = writable;\n\n        // If we have the socket entry, update its monitoring state\n        if (it != _sockets.end()) {\n            // For event-driven operation with sock_state_cb, we rely on\n            // our existing socket infrastructure to handle events.\n            // When sockets become readable/writable, poll_sockets will\n            // process them using ares_process_fd.\n\n            dns_log.trace(\"Socket {} monitoring updated: read={} write={}\",\n                         fd, readable, writable);\n        } else {\n            // Socket not yet in our map - it will be added by do_socket()\n            dns_log.trace(\"Socket {} will be monitored: read={} write={}\",\n                         fd, readable, writable);\n        }\n    }\n}\n#endif\n\nfuture<inet_address>\ndns_resolver::impl::resolve_name(sstring name, opt_family family) {\n    return get_host_by_name(std::move(name), family).then([](hostent h) {\n        return make_ready_future<inet_address>(h.addr_entries.front().addr);\n    });\n}\n\nfuture<hostent>\ndns_resolver::impl::get_host_by_name(sstring name, opt_family family)  {\n    class promise_wrap : public promise<hostent> {\n    public:\n        promise_wrap(sstring s)\n            : name(std::move(s))\n        {}\n        sstring name;\n    };\n\n    dns_log.debug(\"Query name {} ({})\", name, family);\n\n    if (!family) {\n        auto res = inet_address::parse_numerical(name);\n        if (res) {\n            return make_ready_future<hostent>(hostent({std::move(name)}, {{*res}}, {{*res}}));\n        }\n    }\n\n    auto p = new promise_wrap(std::move(name));\n    auto f = p->get_future();\n\n    dns_call call(*this);\n\n    auto af = family ? int(*family) : AF_UNSPEC;\n\n// The following pragma is needed to work around a false-positive warning\n// in Gcc 11 (see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96003).\n#pragma GCC diagnostic ignored \"-Wnonnull\"\n\n    ares_addrinfo_hints hints = {\n        .ai_flags = ARES_AI_CANONNAME,\n        .ai_family = af,\n        .ai_socktype = 0,\n        .ai_protocol = 0,\n    };\n    ares_getaddrinfo(_channel, p->name.c_str(), nullptr, &hints, [](void* arg, int status, int timeouts, ares_addrinfo* addrinfo) {\n        // we do potentially allocating operations below, so wrap the pointer in a\n        // unique here.\n        std::unique_ptr<promise_wrap> p(reinterpret_cast<promise_wrap *>(arg));\n\n        switch (status) {\n        default:\n            dns_log.debug(\"Query failed: {}\", status);\n            p->set_exception(std::system_error(status, dns::error_category(), p->name));\n            break;\n        case ARES_SUCCESS:\n            p->set_value(make_hostent(addrinfo));\n            break;\n        }\n        ares_freeaddrinfo(addrinfo);\n\n    }, reinterpret_cast<void *>(p));\n\n    poll_sockets();\n\n    return f.finally([this] {\n        end_call();\n    });\n}\n\nfuture<hostent>\ndns_resolver::impl::get_host_by_addr(inet_address addr) {\n    class promise_wrap : public promise<hostent> {\n    public:\n        promise_wrap(inet_address a)\n            : addr(std::move(a))\n        {}\n        inet_address addr;\n    };\n\n    dns_log.debug(\"Query addr {}\", addr);\n\n    auto p = new promise_wrap(std::move(addr));\n    auto f = p->get_future();\n\n    dns_call call(*this);\n\n    ares_gethostbyaddr(_channel, p->addr.data(), p->addr.size(), int(p->addr.in_family()), [](void* arg, int status, int timeouts, ::hostent* host) {\n        // we do potentially allocating operations below, so wrap the pointer in a\n        // unique here.\n        std::unique_ptr<promise_wrap> p(reinterpret_cast<promise_wrap *>(arg));\n\n        switch (status) {\n        default:\n            dns_log.debug(\"Query failed: {}\", status);\n            p->set_exception(std::system_error(status, dns::error_category(), boost::lexical_cast<std::string>(p->addr)));\n            break;\n        case ARES_SUCCESS:\n            p->set_value(make_hostent(*host));\n            break;\n        }\n\n    }, reinterpret_cast<void *>(p));\n\n\n    poll_sockets();\n\n    return f.finally([this] {\n        end_call();\n    });\n}\n\nfuture<dns_resolver::srv_records>\ndns_resolver::impl::get_srv_records(srv_proto proto,\n                                    const sstring& service,\n                                    const sstring& domain) {\n    auto p = std::make_unique<promise<srv_records>>();\n    auto f = p->get_future();\n\n    const auto query = format(\"_{}._{}.{}\",\n                                service,\n                                proto == srv_proto::tcp ? \"tcp\" : \"udp\",\n                                domain);\n\n    dns_log.debug(\"Query srv {}\", query);\n\n    dns_call call(*this);\n\n#if ARES_VERSION >= 0x011c00\n    // Use modern DNS record API introduced in c-ares 1.28.0\n    ares_query_dnsrec(_channel, query.c_str(), ARES_CLASS_IN, ARES_REC_TYPE_SRV,\n                        [](void* arg, ares_status_t status, size_t timeouts,\n                            const ares_dns_record *dnsrec) {\n        auto p = std::unique_ptr<promise<srv_records>>(\n            reinterpret_cast<promise<srv_records> *>(arg));\n        if (status != ARES_SUCCESS) {\n            dns_log.debug(\"Query failed: {}\", fmt::underlying(status));\n            p->set_exception(std::system_error(status, dns::error_category()));\n            return;\n        }\n        const size_t rr_count = ares_dns_record_rr_cnt(dnsrec, ARES_SECTION_ANSWER);\n        srv_records replies;\n        for (size_t i = 0; i < rr_count; i++) {\n            const ares_dns_rr_t* rr = ares_dns_record_rr_get(\n                const_cast<ares_dns_record*>(dnsrec),\n                ARES_SECTION_ANSWER, i);\n            if (!rr) {\n                // not likely, but still..\n                status = ARES_EBADRESP;\n                break;\n            }\n            if (ares_dns_rr_get_class(rr) != ARES_CLASS_IN ||\n                ares_dns_rr_get_type(rr) != ARES_REC_TYPE_SRV) {\n                continue;\n            }\n            replies.push_back({\n                ares_dns_rr_get_u16(rr, ARES_RR_SRV_PRIORITY),\n                ares_dns_rr_get_u16(rr, ARES_RR_SRV_WEIGHT),\n                ares_dns_rr_get_u16(rr, ARES_RR_SRV_PORT),\n                sstring{ares_dns_rr_get_str(rr, ARES_RR_SRV_TARGET)}\n            });\n        }\n        if (status != ARES_SUCCESS) {\n            dns_log.debug(\"Parse failed: {}\", fmt::underlying(status));\n            p->set_exception(std::system_error(status, dns::error_category()));\n            return;\n        }\n            p->set_value(std::move(replies));\n    }, reinterpret_cast<void *>(p.release()), nullptr);\n#else\n    // Legacy API for older c-ares versions - uses deprecated ares_parse_srv_reply\n    ares_query(_channel, query.c_str(), ns_c_in, ns_t_srv,\n                [](void* arg, int status, int timeouts,\n                    unsigned char* buf, int len) {\n        auto p = std::unique_ptr<promise<srv_records>>(\n            reinterpret_cast<promise<srv_records> *>(arg));\n        if (status != ARES_SUCCESS) {\n            dns_log.debug(\"Query failed: {}\", status);\n            p->set_exception(std::system_error(status, dns::error_category()));\n            return;\n        }\n        ares_srv_reply* start = nullptr;\n        status = ares_parse_srv_reply(buf, len, &start);\n        if (status != ARES_SUCCESS) {\n            dns_log.debug(\"Parse failed: {}\", status);\n            p->set_exception(std::system_error(status, dns::error_category()));\n            return;\n        }\n        try {\n            p->set_value(make_srv_records(start));\n        } catch (...) {\n            p->set_exception(std::current_exception());\n        }\n        ares_free_data(start);\n    }, reinterpret_cast<void *>(p.release()));\n#endif\n\n\n    poll_sockets();\n\n    return f.finally([this] {\n        end_call();\n    });\n}\n\nfuture<sstring>\ndns_resolver::impl::resolve_addr(inet_address addr) {\n    return get_host_by_addr(addr).then([](hostent h) {\n        return make_ready_future<sstring>(h.names.front());\n    });\n}\n\nfuture<>\ndns_resolver::impl::close() {\n    _closed = true;\n    ares_cancel(_channel);\n    dns_log.trace(\"Shutting down {} sockets\", _sockets.size());\n    for (auto & p : _sockets) {\n        do_close(p.first);\n    }\n    dns_log.trace(\"Closing gate\");\n    return _gate.close();\n}\n\nvoid\ndns_resolver::impl::end_call() {\n    if (--_calls == 0) {\n        _timer.cancel();\n    }\n}\n\n#define USE_CARES_EVENTFD (ARES_VERSION >= 0x012200)  // ares_process_fds available since 1.34.0\n#define USE_ARES_GETSOCK (ARES_VERSION >= 0x010D00 && ARES_VERSION < 0x012200)  // ares_getsock only for intermediate versions\n// Note: For c-ares >= 1.34.0, we use ares_process_fds without needing any FD polling\n// For intermediate versions (1.13.0 - 1.33.x), we use ares_getsock\n\nvoid\ndns_resolver::impl::poll_sockets() {\n    dns_log.trace(\"Poll sockets\");\n\n#if ARES_VERSION >= 0x011300 && USE_CARES_EVENTFD\n    // When using ARES_OPT_SOCK_STATE_CB with modern c-ares >= 1.34.0\n    // we know exactly which sockets c-ares cares about through the callback\n    if (!_socket_monitors.empty()) {\n        // Most DNS queries use few sockets, so optimize for the common case by\n        // stack-allocating a small buffer and only allocating on heap if needed\n        constexpr int MAX_EVENTS = 16;\n        ares_fd_events_t stack_events[MAX_EVENTS];\n        std::unique_ptr<ares_fd_events_t[]> heap_events;\n        ares_fd_events_t* events = stack_events;\n\n        // Count monitored sockets that have data available\n        int available_count = 0;\n        for (auto& [fd, monitor] : _socket_monitors) {\n            auto it = _sockets.find(fd);\n            if (it != _sockets.end() && it->second.avail != 0 && !it->second.closed) {\n                available_count++;\n            }\n        }\n\n        if (available_count > MAX_EVENTS) {\n            heap_events = std::make_unique<ares_fd_events_t[]>(available_count);\n            events = heap_events.get();\n        }\n\n        int event_count = 0;\n        for (auto& [fd, monitor] : _socket_monitors) {\n            auto it = _sockets.find(fd);\n            if (it == _sockets.end() || it->second.closed) {\n                continue;\n            }\n            auto& e = it->second;\n\n            events[event_count].fd = fd;\n            events[event_count].events = 0;\n\n            // Only process events that c-ares is interested in\n            if (monitor.wants_read && (e.avail & POLLIN)) {\n                events[event_count].events |= ARES_FD_EVENT_READ;\n            }\n            if (monitor.wants_write && (e.avail & POLLOUT)) {\n                events[event_count].events |= ARES_FD_EVENT_WRITE;\n            }\n\n            if (events[event_count].events) {\n                event_count++;\n                if (event_count >= available_count) break;\n            }\n        }\n\n        if (event_count > 0) {\n            ares_process_fds(_channel, events, event_count, ARES_PROCESS_FLAG_NONE);\n        } else {\n            // No sockets ready, just process timeouts\n            ares_process_fd(_channel, ARES_SOCKET_BAD, ARES_SOCKET_BAD);\n        }\n    } else {\n        // No sockets monitored, just handle timeouts\n        ares_process_fd(_channel, ARES_SOCKET_BAD, ARES_SOCKET_BAD);\n    }\n#elif USE_CARES_EVENTFD\n    // For modern c-ares >= 1.34.0 without sock_state_cb, we directly process\n    // sockets using ares_process_fds. Iterate through tracked sockets and\n    // process any that have available data.\n\n    // Most DNS queries use few sockets, so optimize for the common case by\n    // stack-allocating a small buffer and only allocating on heap if needed\n    constexpr int MAX_EVENTS = 16;\n    ares_fd_events_t stack_events[MAX_EVENTS];\n    std::unique_ptr<ares_fd_events_t[]> heap_events;\n    ares_fd_events_t* events = stack_events;\n\n    int available_count = 0;\n    for (auto& [fd, e] : _sockets) {\n        if (e.avail != 0 && !e.closed) {\n            available_count++;\n        }\n    }\n\n    if (available_count > MAX_EVENTS) {\n        heap_events = std::make_unique<ares_fd_events_t[]>(available_count);\n        events = heap_events.get();\n    }\n\n    int event_count = 0;\n    for (auto& [fd, e] : _sockets) {\n        if (e.avail != 0 && !e.closed) {\n            events[event_count].fd = fd;\n            events[event_count].events = 0;\n            if (e.avail & POLLIN) {\n                events[event_count].events |= ARES_FD_EVENT_READ;\n            }\n            if (e.avail & POLLOUT) {\n                events[event_count].events |= ARES_FD_EVENT_WRITE;\n            }\n            if (events[event_count].events) {\n                event_count++;\n                if (event_count >= available_count) break;\n            }\n        }\n    }\n\n    if (event_count > 0) {\n        ares_process_fds(_channel, events, event_count, ARES_PROCESS_FLAG_NONE);\n    } else {\n        // No sockets ready, just process timeouts\n        ares_process_fd(_channel, ARES_SOCKET_BAD, ARES_SOCKET_BAD);\n    }\n#else\n    // For older c-ares versions, use the traditional FD polling approach\n    bool processed = false;\n    for (;;) {\n        // Retrieve the set of file descriptors that the library wants us to monitor.\n#if USE_ARES_GETSOCK\n        // Use ares_getsock for c-ares >= 1.13.0\n        ares_socket_t socks[ARES_GETSOCK_MAXNUM];\n        int bitmask = ares_getsock(_channel, socks, ARES_GETSOCK_MAXNUM);\n\n        if (bitmask == 0) {\n            break;\n        }\n\n        // Convert bitmask to fd_sets for compatibility with existing code\n        fd_set readers, writers;\n        FD_ZERO(&readers);\n        FD_ZERO(&writers);\n\n        int nr_fds = 0;\n        for (int i = 0; i < ARES_GETSOCK_MAXNUM; i++) {\n            if (ARES_GETSOCK_READABLE(bitmask, i) || ARES_GETSOCK_WRITABLE(bitmask, i)) {\n                if (ARES_GETSOCK_READABLE(bitmask, i)) {\n                    FD_SET(socks[i], &readers);\n                }\n                if (ARES_GETSOCK_WRITABLE(bitmask, i)) {\n                    FD_SET(socks[i], &writers);\n                }\n                nr_fds++;\n            }\n        }\n\n        dns_log.trace(\"ares_getsock: {} sockets\", nr_fds);\n#endif\n\n#if USE_CARES_EVENTFD\n        // avoid allocations on every poll. the ares_process_fds will not reenter this,\n        // as any read/write/close callbacks will either complete or create a pending\n        // future which cannot execute until the call returns.\n        // I.e. we cannot get here while this call is running, so safe to make this\n        // thread static\n        static thread_local ares_fd_events_t events[FD_SETSIZE];\n#endif\n        int processed_fds = 0;\n\n        for (auto& [fd, e] : _sockets) {\n            bool read_monitor = FD_ISSET(fd, &readers);\n            bool write_monitor = FD_ISSET(fd, &writers);\n            bool read_avail = e.avail & POLLIN;\n            bool write_avail = e.avail & POLLOUT;\n\n            dns_log.trace(\"fd {} {}{}/{}{}\", fd,\n                          read_monitor ? \"r\" : \"\",\n                          write_monitor ? \"w\" : \"\",\n                          read_avail ? \"r\" : \"\",\n                          write_avail ? \"w\" : \"\");\n\n            // #2641 - don't do callbacks per fd, instead use\n            // ares_process or ares_process_fds if available.\n            // Use ares_process_fds if possible, since this is the\n            // recommended API and avoids allocations.\n            // Note: For c-ares >= 1.19.0 with ARES_OPT_SOCK_STATE_CB,\n            // the modern code path above already uses socket state callbacks.\n\n            // clear fd state\n#if USE_CARES_EVENTFD\n            events[processed_fds] = {0,};\n            // Update read/write state based on our poll info\n            if (read_monitor && read_avail) {\n                events[processed_fds].fd = fd;\n                events[processed_fds].events |= ARES_FD_EVENT_READ;\n            }\n            if (write_monitor && write_avail) {\n                events[processed_fds].fd = fd;\n                events[processed_fds].events |= ARES_FD_EVENT_WRITE;\n            }\n            if (events[processed_fds].events) {\n                ++processed_fds;\n            }\n#else\n            FD_CLR(fd, &readers);\n            FD_CLR(fd, &writers);\n            // Update read/write state based on our poll info\n            if (read_monitor && read_avail) {\n                FD_SET(fd, &readers);\n            }\n            if (write_monitor && write_avail) {\n                FD_SET(fd, &writers);\n            }\n            if (FD_ISSET(fd, &readers) || FD_ISSET(fd, &writers)) {\n                ++processed_fds;\n            }\n#endif\n        }\n        // no sockets of interest had polling values. done.\n        if (processed_fds == 0) {\n            break;\n        }\n        // call fd processing. this will clean up and close sockets as well.\n#if USE_CARES_EVENTFD\n        ares_process_fds(_channel, events, processed_fds, ARES_PROCESS_FLAG_NONE);\n#else\n        ares_process(_channel, &readers, &writers);\n#endif\n        processed = true;\n    }\n    // even if we did not process anything, do a single callback to maybe close\n    // broken sockets.\n    if (!processed) {\n        ares_process_fd(_channel, ARES_SOCKET_BAD, ARES_SOCKET_BAD);\n    }\n#endif  // USE_CARES_EVENTFD\n}\n\ndns_resolver::srv_records\ndns_resolver::impl::make_srv_records(ares_srv_reply* start) {\n    // Only used with deprecated ares_parse_srv_reply API for c-ares < 1.28.0\n    srv_records records;\n    for (auto reply = start; reply; reply = reply->next) {\n        srv_record record = {reply->priority,\n                                reply->weight,\n                                reply->port,\n                                sstring{reply->host}};\n        records.push_back(std::move(record));\n    }\n    return records;\n}\n\nhostent\ndns_resolver::impl::make_hostent(const ares_addrinfo* ai) {\n    hostent e;\n    if (!ai) {\n        return e;\n    }\n    if (ai->cnames) {\n        e.names.emplace_back(ai->cnames->name);\n    } else {\n        e.names.emplace_back(ai->name);\n    }\n    for (auto cname = ai->cnames; cname != nullptr; cname = cname->next) {\n        if (cname->alias == nullptr) {\n            continue;\n        }\n        e.names.emplace_back(cname->alias);\n    }\n    for (auto node = ai->nodes; node != nullptr; node = node->ai_next) {\n        // The TTL can be zero (dont cache) or greater up to 2^31 - 1 (in seconds)\n        // https://datatracker.ietf.org/doc/html/rfc2181#section-8\n        // https://datatracker.ietf.org/doc/html/rfc1035#section-3.2.1\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n        switch (node->ai_family) {\n            case AF_INET:\n                e.addr_list.emplace_back(reinterpret_cast<const sockaddr_in*>(node->ai_addr)->sin_addr);\n                e.addr_entries.emplace_back(hostent::address_entry{\n                    reinterpret_cast<const sockaddr_in*>(node->ai_addr)->sin_addr,\n                    std::chrono::seconds(std::max<unsigned int>(0, node->ai_ttl))\n                });\n                break;\n            case AF_INET6:\n                e.addr_list.emplace_back(reinterpret_cast<const sockaddr_in6*>(node->ai_addr)->sin6_addr);\n                e.addr_entries.emplace_back(hostent::address_entry{\n                    reinterpret_cast<const sockaddr_in6*>(node->ai_addr)->sin6_addr,\n                    std::chrono::seconds(std::max<unsigned int>(0, node->ai_ttl))\n                });\n                break;\n        }\n#pragma GCC diagnostic pop\n    }\n\n    dns_log.debug(\"Query success: {}/{}, TTL: {}s\", e.names.front(), e.addr_entries.front().addr, e.addr_entries.front().ttl.count());\n\n    return e;\n}\n\nhostent\ndns_resolver::impl::make_hostent(const ::hostent& host) {\n    hostent e;\n    e.names.emplace_back(host.h_name);\n    auto np = host.h_aliases;\n    while (*np != 0) {\n        e.names.emplace_back(*np++);\n    }\n    auto p = host.h_addr_list;\n    while (*p != nullptr) {\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n        switch (host.h_addrtype) {\n        case AF_INET:\n            SEASTAR_ASSERT(size_t(host.h_length) >= sizeof(in_addr));\n            e.addr_list.emplace_back(*reinterpret_cast<const in_addr*>(*p));\n            e.addr_entries.emplace_back(hostent::address_entry{\n                *reinterpret_cast<const in_addr*>(*p), std::chrono::seconds(std::numeric_limits<signed int>::max())\n            });\n            break;\n        case AF_INET6:\n            SEASTAR_ASSERT(size_t(host.h_length) >= sizeof(in6_addr));\n            e.addr_list.emplace_back(*reinterpret_cast<const in6_addr*>(*p));\n            e.addr_entries.emplace_back(hostent::address_entry{\n                *reinterpret_cast<const in6_addr*>(*p), std::chrono::seconds(std::numeric_limits<signed int>::max())\n            });\n            break;\n        default:\n            break;\n        }\n#pragma GCC diagnostic pop\n        ++p;\n    }\n\n    dns_log.debug(\"Query success: {}/{}, TTL: {}s\", e.names.front(), e.addr_entries.front().addr, e.addr_entries.front().ttl.count());\n\n    return e;\n}\n\nvoid\ndns_resolver::impl::use(ares_socket_t fd) {\n    _gate.enter();\n    auto& e = _sockets.at(fd);\n    ++e.pending;\n}\n\nvoid\ndns_resolver::impl::release(ares_socket_t fd) {\n    auto& e = _sockets.at(fd);\n    dns_log.trace(\"Release socket {} -> {}\", fd, e.pending -  1);\n    if (--e.pending < 0) {\n        _sockets.erase(fd);\n        dns_log.trace(\"Released socket {}\", fd);\n    }\n    _gate.leave();\n}\n\nares_socket_t\ndns_resolver::impl::do_socket(int af, int type, int protocol) {\n    if (_closed) {\n        return -1;\n    }\n    int fd = next_fd();\n    switch (type) {\n    case SOCK_STREAM:\n        _sockets.emplace(fd, connected_socket());\n        dns_log.trace(\"Created tcp socket {}\", fd);\n        break;\n    case SOCK_DGRAM:\n        _sockets.emplace(fd, _stack.make_unbound_datagram_channel(AF_INET));\n        dns_log.trace(\"Created udp socket {}\", fd);\n        break;\n    default: return -1;\n    }\n    return fd;\n}\n\nint\ndns_resolver::impl::do_close(ares_socket_t fd) {\n    dns_log.trace(\"Close socket {}\", fd);\n    auto& e = _sockets.at(fd);\n\n    // Mark as closed.\n    if (std::exchange(e.closed, true)) {\n        return 0;\n    }\n\n    _gate.enter(); // \"leave\" is done in release(fd)\n\n    switch (e.typ) {\n    case type::tcp:\n    {\n        dns_log.trace(\"Close tcp socket {}, {} pending\", fd, e.pending);\n        future<> f = make_ready_future();\n        if (e.tcp.in) {\n            e.tcp.socket.shutdown_input();\n            dns_log.trace(\"Closed tcp socket {} input\", fd);\n        }\n        if (e.tcp.out) {\n            f = f.then([&e] {\n                return e.tcp.out->close();\n            }).then([fd] {\n                dns_log.trace(\"Closed tcp socket {} output\", fd);\n            });\n        }\n        f = f.finally([me = shared_from_this(), fd] {\n            me->release(fd);\n        });\n        break;\n    }\n    case type::udp:\n        e.udp.channel.shutdown_input();\n        e.udp.channel.shutdown_output();\n        release(fd);\n        break;\n    default:\n        // should not happen\n        _gate.leave();\n        break;\n    }\n    return 0;\n}\n\nsocket_address\ndns_resolver::impl::sock_addr(const sockaddr * addr, socklen_t len) {\n    if (addr->sa_family != AF_INET) {\n        throw std::invalid_argument(\"No ipv6 yet\");\n    }\n    auto in = reinterpret_cast<const sockaddr_in *>(addr);\n    return *in;\n}\n\nint\ndns_resolver::impl::do_connect(ares_socket_t fd, const sockaddr * addr, socklen_t len) {\n    if (_closed) {\n        return -1;\n    }\n    try {\n        auto& e = get_socket_entry(fd);\n        auto sa = sock_addr(addr, len);\n\n        dns_log.trace(\"Connect {}({})->{}\", fd, int(e.typ), sa);\n\n        SEASTAR_ASSERT(e.avail == 0);\n\n        e.avail = POLLOUT|POLLIN; // until we know otherwise\n\n        switch (e.typ) {\n        case type::tcp: {\n            auto f = _stack.connect(sa);\n            if (!f.available()) {\n                dns_log.trace(\"Connection pending: {}\", fd);\n                e.avail = 0;\n                use(fd);\n                // FIXME: future is discarded\n                (void)f.then_wrapped([me = shared_from_this(), &e, fd](future<connected_socket> f) {\n                    try {\n                        e.tcp.socket = f.get();\n                        dns_log.trace(\"Connection complete: {}\", fd);\n                    } catch (...) {\n                        dns_log.debug(\"Connect {} failed: {}\", fd, std::current_exception());\n                    }\n                    e.avail = POLLOUT|POLLIN;\n                    me->poll_sockets();\n                    me->release(fd);\n                });\n                errno = EWOULDBLOCK;\n                return -1;\n            }\n            e.tcp.socket = f.get();\n            break;\n        }\n        case type::udp:\n            // we do not have udp connect, so just keep\n            // track of the destination\n            e.udp.dst = sa;\n            break;\n        default:\n            return -1;\n        }\n        return 0;\n    } catch (...) {\n        return -1;\n    }\n}\n\nssize_t\ndns_resolver::impl::do_recvfrom(ares_socket_t fd, void * dst, size_t len, int flags, struct sockaddr * from, socklen_t * from_len) {\n    if (_closed) {\n        return -1;\n    }\n    try {\n        auto& e = get_socket_entry(fd);\n        dns_log.trace(\"Read {}({})\", fd, int(e.typ));\n        // check if we're already reading.\n        if (!(e.avail & POLLIN)) {\n            dns_log.trace(\"Read already pending {}\", fd);\n            errno = EWOULDBLOCK;\n            return -1;\n        }\n        for (;;) {\n            switch (e.typ) {\n            case type::tcp: {\n                auto & tcp = e.tcp;\n                if (!tcp.indata.empty()) {\n                    dns_log.trace(\"Read {}. {} bytes available\", fd, tcp.indata.size());\n                    len = std::min(len, tcp.indata.size());\n                    std::copy(tcp.indata.begin(), tcp.indata.begin() + len, reinterpret_cast<char *>(dst));\n                    tcp.indata.trim_front(len);\n                    return len;\n                }\n                if (!tcp.socket) {\n                    errno = ENOTCONN;\n                    return -1;\n                }\n                if (!tcp.in) {\n                    tcp.in = tcp.socket.input();\n                }\n                auto f = tcp.in->read_up_to(len);\n                if (!f.available()) {\n                    dns_log.trace(\"Read {}: data unavailable\", fd);\n                    e.avail &= ~POLLIN;\n                    use(fd);\n                    // FIXME: future is discarded\n                    (void)f.then_wrapped([me = shared_from_this(), &e, fd](future<temporary_buffer<char>> f) {\n                        try {\n                            auto buf = f.get();\n                            dns_log.trace(\"Read {} -> {} bytes\", fd, buf.size());\n                            e.tcp.indata = std::move(buf);\n                        } catch (...) {\n                            dns_log.debug(\"Read {} failed: {}\", fd, std::current_exception());\n                        }\n                        e.avail |= POLLIN; // always reset state\n                        me->poll_sockets();\n                        me->release(fd);\n                    });\n                    errno = EWOULDBLOCK;\n                    return -1;\n                }\n\n                try {\n                    tcp.indata = f.get();\n                    continue; // loop will take care of data\n                } catch (std::system_error& e) {\n                    errno = e.code().value();\n                    return -1;\n                } catch (...) {\n                }\n                return -1;\n\n            }\n            case type::udp: {\n                auto & udp = e.udp;\n                if (udp.in) {\n                    auto bufs = udp.in->get_buffers();\n                    size_t available = std::accumulate(bufs.begin(), bufs.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n\n                    dns_log.trace(\"Read {}. {} bytes available from {}\", fd, available, udp.in->get_src());\n\n                    if (from != nullptr) {\n                        *from = socket_address(udp.in->get_src()).as_posix_sockaddr();\n                        if (from_len != nullptr) {\n                            // TODO: ipvv6\n                            *from_len = sizeof(sockaddr_in);\n                        }\n                    }\n\n                    size_t copied = 0;\n                    auto * out = reinterpret_cast<char *>(dst);\n                    for (auto& b : bufs) {\n                        size_t n = std::min(len - copied, b.size());\n                        out = std::copy_n(b.get(), n, out);\n                        copied += n;\n                        b.trim_front(n);\n                        if (copied == available) {\n                            udp.in = {};\n                            break;\n                        }\n                        if (copied == len) {\n                            break;\n                        }\n                    }\n                    return copied;\n                }\n                auto f = udp.channel.receive();\n                if (!f.available()) {\n                    e.avail &= ~POLLIN;\n                    use(fd);\n                    dns_log.trace(\"Read {}: data unavailable\", fd);\n                    // FIXME: future is discarded\n                    (void)f.then_wrapped([me = shared_from_this(), &e, fd](future<datagram> f) {\n                        try {\n                            auto d = f.get();\n                            dns_log.trace(\"Read {} -> {} bytes\", fd, value_of([&d] {\n                                    auto bufs = d.get_buffers();\n                                    return std::accumulate(bufs.begin(), bufs.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n                                }));\n                            e.udp.in = std::move(d);\n                            e.avail |= POLLIN;\n                        } catch (...) {\n                            dns_log.debug(\"Read {} failed: {}\", fd, std::current_exception());\n                        }\n                        me->poll_sockets();\n                        me->release(fd);\n                    });\n                    errno = EWOULDBLOCK;\n                    return -1;\n                }\n\n                try {\n                    udp.in = f.get();\n                    continue; // loop will take care of data\n                } catch (std::system_error& e) {\n                    errno = e.code().value();\n                    return -1;\n                } catch (...) {\n                }\n                return -1;\n            }\n            default:\n                return -1;\n            }\n        }\n    } catch (...) {\n    }\n    return -1;\n}\n\nssize_t dns_resolver::impl::do_send_tcp(sock_entry& e, send_packet_t p, size_t bytes, ares_socket_t fd) {\n    if (!e.tcp.out) {\n        e.tcp.out = e.tcp.socket.output(0).detach();\n    }\n    auto f = e.tcp.out->put(std::move(p));\n\n    if (!f.available()) {\n        dns_log.trace(\"Send {} unavailable.\", fd);\n        e.avail &= ~POLLOUT;\n        // FIXME: future is discarded\n        (void)f.then_wrapped([me = shared_from_this(), &e, bytes, fd](future<> f) {\n            try {\n                f.get();\n                dns_log.trace(\"Send {}. {} bytes sent.\", fd, bytes);\n            } catch (...) {\n                dns_log.debug(\"Send {} failed: {}\", fd, std::current_exception());\n            }\n            e.avail |= POLLOUT;\n            me->poll_sockets();\n            me->release(fd);\n        });\n\n        // For tcp we also pretend we're done, to make sure we don't have to deal with\n        // matching sent data\n        return bytes;\n    }\n\n    release(fd);\n\n    if (f.failed()) {\n        try {\n            f.get();\n        } catch (std::system_error& e) {\n            errno = e.code().value();\n        } catch (...) {\n        }\n        return -1;\n    }\n\n    return bytes;\n}\n\nssize_t dns_resolver::impl::do_send_udp(sock_entry& e, send_packet_t p, size_t bytes, ares_socket_t fd) {\n    // always chain UDP sends\n    e.udp.f = e.udp.f.finally([&e, p = std::move(p)]() mutable {\n#if ARES_VERSION >= 0x012200\n        std::span<temporary_buffer<char>> sp(&p, 1);\n#else\n        std::span<temporary_buffer<char>> sp(p);\n#endif\n        return e.udp.channel.send(e.udp.dst, sp);\n    }).finally([fd, me = shared_from_this()] {\n        me->release(fd);\n    });\n\n    if (e.udp.f.available()) {\n        // if we have a fast-fail, give error.\n        if (e.udp.f.failed()) {\n            try {\n                e.udp.f.get();\n            } catch (std::system_error& e) {\n                errno = e.code().value();\n            } catch (...) {\n            }\n            e.udp.f = make_ready_future<>();\n            return -1;\n        }\n    } else {\n        // ensure that no exception from channel.send is left uncaught\n        e.udp.f = e.udp.f.handle_exception_type([](std::system_error const& e){\n            dns_log.warn(\"UDP send exception: {}\", e.what());\n        });\n    }\n    // c-ares does _not_ use non-blocking retry for udp sockets. We just pretend\n    // all is fine even though we have no idea. Barring stack/adapter failure it\n    // is close to the same guarantee a \"normal\" message send would have anyway.\n    return bytes;\n}\n\nssize_t\n#if ARES_VERSION >= 0x012200\ndns_resolver::impl::do_send(ares_socket_t fd, const void * buf, size_t bytes) {\n#else\ndns_resolver::impl::do_sendv(ares_socket_t fd, const iovec * vec, int len) {\n#endif\n    if (_closed) {\n        return -1;\n    }\n    try {\n        auto& e = _sockets.at(fd);\n        dns_log.trace(\"Send {}({})\", fd, int(e.typ));\n\n        // Assume we will be able to send data eventually very soon\n        // and just assume that unless we get immediate\n        // failures, we'll be ok. If we're not, the\n        // timeout logic will have to handle the problem.\n        //\n        // This saves us on two accounts:\n        // 1.) c-ares does not handle EWOULDBLOCK for\n        //     udp sockets. Must pretend to finish\n        //     immediately there anyway\n        // 2.) Doing so for tcp writes saves us having to\n        //     match iovec->packet fragments. Downside is we\n        //     have to copy the data, but we pretty much\n        //     have to anyway, since we could otherwise\n        //     get a query time out while we're sending\n        //     with zero-copy and suddenly have freed\n        //     memory in packets. Bad.\n\n\n            // check if we're already writing.\n            if (!(e.avail & POLLOUT)) {\n                dns_log.trace(\"Send already pending {}\", fd);\n                errno = EWOULDBLOCK;\n                return -1;\n            }\n\n            if (e.typ == type::tcp && !e.tcp.socket) {\n                errno = ENOTCONN;\n                return -1;\n            }\n\n#if ARES_VERSION >= 0x012200\n            temporary_buffer<char> p(reinterpret_cast<const char *>(buf), bytes);\n#else\n            std::vector<temporary_buffer<char>> p;\n            p.reserve(len);\n            size_t bytes = 0;\n            for (int i = 0; i < len; ++i) {\n                bytes += vec[i].iov_len;\n                p.emplace_back(reinterpret_cast<const char *>(vec[i].iov_base), vec[i].iov_len);\n            }\n#endif\n\n            use(fd);\n\n            switch (e.typ) {\n            case type::tcp:\n                return do_send_tcp(e, std::move(p), bytes, fd);\n            case type::udp:\n                return do_send_udp(e, std::move(p), bytes, fd);\n            default:\n                return -1;\n            }\n    } catch (...) {\n    }\n    return -1;\n}\n\nares_socket_t\ndns_resolver::impl::next_fd() {\n    ares_socket_t fd = ares_socket_t(_sockets.size() + socket_offset);\n    while (_sockets.count(fd)) {\n        ++fd;\n    }\n    return fd;\n}\n\ndns_resolver::impl::sock_entry&\ndns_resolver::impl::get_socket_entry(ares_socket_t fd) {\n    auto& e = _sockets.at(fd);\n    if (e.closed) {\n        throw std::runtime_error(\"Socket closed\");\n    }\n    return e;\n}\n\ndns_resolver::dns_resolver()\n    : dns_resolver(options())\n{}\n\ndns_resolver::dns_resolver(const options& opts)\n    : dns_resolver(engine().net(), opts)\n{}\n\ndns_resolver::dns_resolver(network_stack& stack, const options& opts)\n    : _impl(make_shared<impl>(stack, opts))\n{}\n\ndns_resolver::~dns_resolver()\n{}\n\ndns_resolver::dns_resolver(dns_resolver&&) noexcept = default;\ndns_resolver& dns_resolver::operator=(dns_resolver&&) noexcept = default;\n\nfuture<hostent> dns_resolver::get_host_by_name(const sstring& name, opt_family family) {\n    return _impl->get_host_by_name(name, family);\n}\n\nfuture<hostent> dns_resolver::get_host_by_addr(const inet_address& addr) {\n    return _impl->get_host_by_addr(addr);\n}\n\nfuture<inet_address> dns_resolver::resolve_name(const sstring& name, opt_family family) {\n    return _impl->resolve_name(name, family);\n}\n\nfuture<sstring> dns_resolver::resolve_addr(const inet_address& addr) {\n    return _impl->resolve_addr(addr);\n}\n\nfuture<dns_resolver::srv_records> dns_resolver::get_srv_records(dns_resolver::srv_proto proto,\n                                                                          const sstring& service,\n                                                                          const sstring& domain) {\n    return _impl->get_srv_records(proto, service, domain);\n}\n\nfuture<> dns_resolver::close() {\n    return _impl->close();\n}\n\nstatic dns_resolver& resolver() {\n    static thread_local dns_resolver resolver;\n    return resolver;\n}\n\n\nfuture<hostent> dns::get_host_by_name(const sstring& name, opt_family family) {\n    return resolver().get_host_by_name(name, family);\n}\n\nfuture<hostent> dns::get_host_by_addr(const inet_address& addr) {\n    return resolver().get_host_by_addr(addr);\n}\n\nfuture<inet_address> dns::resolve_name(const sstring& name, opt_family family) {\n    return resolver().resolve_name(name, family);\n}\n\nfuture<sstring> dns::resolve_addr(const inet_address& addr) {\n    return resolver().resolve_addr(addr);\n}\n\nfuture<dns_resolver::srv_records> dns::get_srv_records(dns_resolver::srv_proto proto,\n                                                                 const sstring& service,\n                                                                 const sstring& domain) {\n    return resolver().get_srv_records(proto, service, domain);\n}\n\nfuture<sstring> inet_address::hostname() const {\n    return dns::resolve_addr(*this);\n}\n\nfuture<std::vector<sstring>> inet_address::aliases() const {\n    return dns::get_host_by_addr(*this).then([](hostent e) {\n        return make_ready_future<std::vector<sstring>>(std::move(e.names));\n    });\n}\n\nfuture<inet_address> inet_address::find(\n                const sstring& name) {\n    return dns::resolve_name(name);\n}\n\nfuture<inet_address> inet_address::find(\n                const sstring& name, family f) {\n    return dns::resolve_name(name, f);\n}\n\nfuture<std::vector<inet_address>> inet_address::find_all_impl(\n                const sstring& name, opt_family f) {\n    return dns::get_host_by_name(name, f).then([](hostent e) {\n        auto rng = e.addr_entries | std::views::transform([](auto& entry) { return entry.addr; });\n        return make_ready_future<std::vector<inet_address>>(rng.begin(), rng.end());\n    });\n}\n\nfuture<std::vector<inet_address>> inet_address::find_all(\n                const sstring& name) {\n    return inet_address::find_all_impl(name, std::nullopt);\n}\n\nfuture<std::vector<inet_address>> inet_address::find_all(\n                const sstring& name, family f) {\n    return inet_address::find_all_impl(name, std::make_optional(f));\n}\n\nconst std::error_category& dns::error_category() {\n    static const ares_error_category ares_errorc;\n\n    return ares_errorc;\n}\n\n}\n"
  },
  {
    "path": "src/net/dpdk.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n#ifdef SEASTAR_HAVE_DPDK\n\n\n#include <cinttypes>\n#include <atomic>\n#include <iostream>\n#include <vector>\n#include <queue>\n#include <getopt.h>\n#include <malloc.h>\n\n#include <cinttypes>\n#include <rte_config.h>\n#include <rte_common.h>\n#include <rte_eal.h>\n#include <rte_pci.h>\n#include <rte_ethdev.h>\n#include <rte_cycles.h>\n#include <rte_memzone.h>\n#include <rte_vfio.h>\n\n#include <boost/preprocessor.hpp>\n\n#include <seastar/core/posix.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/net/virtio-interface.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/internal/poll.hh>\n#include <seastar/core/units.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/function_input_iterator.hh>\n#include <seastar/util/transform_iterator.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/const.hh>\n#include <seastar/core/dpdk_rte.hh>\n#include <seastar/net/dpdk.hh>\n#include <seastar/net/toeplitz.hh>\n#include <seastar/net/native-stack.hh>\n#include \"core/vla.hh\"\n\n#if RTE_VERSION <= RTE_VERSION_NUM(2,0,0,16)\n\nstatic\ninline\nchar*\nrte_mbuf_to_baddr(rte_mbuf* mbuf) {\n    return reinterpret_cast<char*>(RTE_MBUF_TO_BADDR(mbuf));\n}\n\nvoid* as_cookie(struct rte_pktmbuf_pool_private& p) {\n    return reinterpret_cast<void*>(uint64_t(p.mbuf_data_room_size));\n};\n\n#else\n\nvoid* as_cookie(struct rte_pktmbuf_pool_private& p) {\n    return &p;\n};\n\n#endif\n\n#ifndef MARKER\ntypedef void    *MARKER[0];   /**< generic marker for a point in a structure */\n#endif\n\n// Calculate maximum amount of memory required to store given number of objects\nstatic size_t\nget_mempool_xmem_size(uint32_t elt_num, size_t total_elt_sz, uint32_t pg_shift)\n{\n    size_t obj_per_page, pg_num, pg_sz;\n\n    if (total_elt_sz == 0) {\n        return 0;\n    }\n\n    if (pg_shift == 0) {\n        return total_elt_sz * elt_num;\n    }\n\n    pg_sz = (size_t)1 << pg_shift;\n    obj_per_page = pg_sz / total_elt_sz;\n    if (obj_per_page == 0) {\n        return RTE_ALIGN_CEIL(total_elt_sz, pg_sz) * elt_num;\n    }\n\n    pg_num = (elt_num + obj_per_page - 1) / obj_per_page;\n    return pg_num << pg_shift;\n}\n\nusing namespace seastar::net;\n\nnamespace seastar {\n\nnamespace dpdk {\n\n/******************* Net device related constatns *****************************/\nstatic constexpr uint16_t default_ring_size      = 512;\n\n//\n// We need 2 times the ring size of buffers because of the way PMDs\n// refill the ring.\n//\nstatic constexpr uint16_t mbufs_per_queue_rx     = 2 * default_ring_size;\nstatic constexpr uint16_t rx_gc_thresh           = 64;\n\n//\n// No need to keep more descriptors in the air than can be sent in a single\n// rte_eth_tx_burst() call.\n//\nstatic constexpr uint16_t mbufs_per_queue_tx     = 2 * default_ring_size;\n\nstatic constexpr uint16_t mbuf_cache_size        = 512;\nstatic constexpr uint16_t mbuf_overhead          =\n                                 sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM;\n//\n// We'll allocate 2K data buffers for an inline case because this would require\n// a single page per mbuf. If we used 4K data buffers here it would require 2\n// pages for a single buffer (due to \"mbuf_overhead\") and this is a much more\n// demanding memory constraint.\n//\nstatic constexpr size_t   inline_mbuf_data_size  = 2048;\n\n//\n// Size of the data buffer in the non-inline case.\n//\n// We may want to change (increase) this value in future, while the\n// inline_mbuf_data_size value will unlikely change due to reasons described\n// above.\n//\nstatic constexpr size_t   mbuf_data_size         = 2048;\n\n// (INLINE_MBUF_DATA_SIZE(2K)*32 = 64K = Max TSO/LRO size) + 1 mbuf for headers\nstatic constexpr uint8_t  max_frags              = 32 + 1;\n\n//\n// Intel's 40G NIC HW limit for a number of fragments in an xmit segment.\n//\n// See Chapter 8.4.1 \"Transmit Packet in System Memory\" of the xl710 devices\n// spec. for more details.\n//\nstatic constexpr uint8_t  i40e_max_xmit_segment_frags = 8;\n\n//\n// VMWare's virtual NIC limit for a number of fragments in an xmit segment.\n//\n// see drivers/net/vmxnet3/base/vmxnet3_defs.h VMXNET3_MAX_TXD_PER_PKT\n//\nstatic constexpr uint8_t vmxnet3_max_xmit_segment_frags = 16;\n\nstatic constexpr uint16_t inline_mbuf_size       =\n                                inline_mbuf_data_size + mbuf_overhead;\n\nuint32_t qp_mempool_obj_size(bool hugetlbfs_membackend)\n{\n    uint32_t mp_size = 0;\n    struct rte_mempool_objsz mp_obj_sz = {};\n\n    //\n    // We will align each size to huge page size because DPDK allocates\n    // physically contiguous memory region for each pool object.\n    //\n\n    // Rx\n    if (hugetlbfs_membackend) {\n        mp_size +=\n            align_up(rte_mempool_calc_obj_size(mbuf_overhead, 0, &mp_obj_sz)+\n                                        sizeof(struct rte_pktmbuf_pool_private),\n                                               memory::huge_page_size);\n    } else {\n        mp_size +=\n            align_up(rte_mempool_calc_obj_size(inline_mbuf_size, 0, &mp_obj_sz)+\n                                        sizeof(struct rte_pktmbuf_pool_private),\n                                               memory::huge_page_size);\n    }\n    //Tx\n    std::memset(&mp_obj_sz, 0, sizeof(mp_obj_sz));\n    mp_size += align_up(rte_mempool_calc_obj_size(inline_mbuf_size, 0,\n                                                  &mp_obj_sz)+\n                                        sizeof(struct rte_pktmbuf_pool_private),\n                                                  memory::huge_page_size);\n    return mp_size;\n}\n\nstatic constexpr const char* pktmbuf_pool_name   = \"dpdk_pktmbuf_pool\";\n\n/*\n * When doing reads from the NIC queues, use this batch size\n */\nstatic constexpr uint8_t packet_read_size        = 32;\n/******************************************************************************/\n\nstruct port_stats {\n    port_stats() : rx{}, tx{} {}\n\n    struct {\n        struct {\n            uint64_t mcast;        // number of received multicast packets\n            uint64_t pause_xon;    // number of received PAUSE XON frames\n            uint64_t pause_xoff;   // number of received PAUSE XOFF frames\n        } good;\n\n        struct {\n            uint64_t dropped;      // missed packets (e.g. full FIFO)\n            uint64_t crc;          // packets with CRC error\n            uint64_t len;          // packets with a bad length\n            uint64_t total;        // total number of erroneous received packets\n        } bad;\n    } rx;\n\n    struct {\n        struct {\n            uint64_t pause_xon;   // number of sent PAUSE XON frames\n            uint64_t pause_xoff;  // number of sent PAUSE XOFF frames\n        } good;\n\n        struct {\n            uint64_t total;   // total number of failed transmitted packets\n        } bad;\n    } tx;\n};\n\n#define XSTATS_ID_LIST \\\n        (rx_multicast_packets) \\\n        (rx_xon_packets) \\\n        (rx_xoff_packets) \\\n        (rx_crc_errors) \\\n        (rx_length_errors) \\\n        (rx_undersize_errors) \\\n        (rx_oversize_errors) \\\n        (tx_xon_packets) \\\n        (tx_xoff_packets)\n\nclass dpdk_xstats {\npublic:\n    dpdk_xstats(uint16_t port_id)\n        : _port_id(port_id)\n    {\n    }\n\n    ~dpdk_xstats()\n    {\n        if (_xstats)\n            delete[] _xstats;\n        if (_xstat_names)\n            delete[] _xstat_names;\n    }\n\n    enum xstat_id {\n        BOOST_PP_SEQ_ENUM(XSTATS_ID_LIST)\n    };\n\n\n    void start() {\n        _len = rte_eth_xstats_get_names(_port_id, NULL, 0);\n        _xstats = new rte_eth_xstat[_len];\n        _xstat_names = new rte_eth_xstat_name[_len];\n        update_xstats();\n        update_xstat_names();\n        update_offsets();\n    }\n\n    void update_xstats() {\n        auto len = rte_eth_xstats_get(_port_id, _xstats, _len);\n        SEASTAR_ASSERT(len == _len);\n    }\n\n    uint64_t get_value(const xstat_id id) {\n        auto off = _offsets[static_cast<int>(id)];\n        if (off == -1) {\n            return 0;\n        }\n        return _xstats[off].value;\n    }\n\nprivate:\n    uint16_t _port_id;\n    int _len;\n    struct rte_eth_xstat *_xstats = nullptr;\n    struct rte_eth_xstat_name *_xstat_names = nullptr;\n    int _offsets[BOOST_PP_SEQ_SIZE(XSTATS_ID_LIST)];\n\n    static const sstring id_to_str(const xstat_id id) {\n#define ENUM_TO_STR(r, data, elem) \\\n        if (id == elem) \\\n            return BOOST_PP_STRINGIZE(elem);\n\n        BOOST_PP_SEQ_FOR_EACH(ENUM_TO_STR, ~, XSTATS_ID_LIST)\n        return \"\";\n    }\n\n    int get_offset_by_name(const xstat_id id, const int len) {\n        for (int i = 0; i < len; i++) {\n            if (id_to_str(id) == _xstat_names[i].name)\n                return i;\n        }\n        return -1;\n    }\n\n    void update_xstat_names() {\n        auto len = rte_eth_xstats_get_names(_port_id, _xstat_names, _len);\n        SEASTAR_ASSERT(len == _len);\n    }\n\n    void update_offsets() {\n#define FIND_OFFSET(r, data, elem) \\\n        _offsets[static_cast<int>(elem)] = \\\n            get_offset_by_name(elem, _len);\n\n        BOOST_PP_SEQ_FOR_EACH(FIND_OFFSET, ~, XSTATS_ID_LIST)\n    }\n};\n\nclass dpdk_device : public device {\n    uint16_t _port_idx;\n    uint16_t _num_queues;\n    net::hw_features _hw_features;\n    uint16_t _queues_ready = 0;\n    unsigned _home_cpu;\n    bool _use_lro;\n    bool _enable_fc;\n    std::vector<uint8_t> _redir_table;\n    rss_key_type _rss_key;\n    port_stats _stats;\n    timer<> _stats_collector;\n    const std::string _stats_plugin_name;\n    const std::string _stats_plugin_inst;\n    seastar::metrics::metric_groups _metrics;\n    bool _is_i40e_device = false;\n    bool _is_vmxnet3_device = false;\n    dpdk_xstats _xstats;\n\npublic:\n    rte_eth_dev_info _dev_info = {};\n    promise<> _link_ready_promise;\n\nprivate:\n    /**\n     * Port initialization consists of 3 main stages:\n     * 1) General port initialization which ends with a call to\n     *    rte_eth_dev_configure() where we request the needed number of Rx and\n     *    Tx queues.\n     * 2) Individual queues initialization. This is done in the constructor of\n     *    dpdk_qp class. In particular the memory pools for queues are allocated\n     *    in this stage.\n     * 3) The final stage of the initialization which starts with the call of\n     *    rte_eth_dev_start() after which the port becomes fully functional. We\n     *    will also wait for a link to get up in this stage.\n     */\n\n\n    /**\n     * First stage of the port initialization.\n     *\n     * @return 0 in case of success and an appropriate error code in case of an\n     *         error.\n     */\n    int init_port_start();\n\n    /**\n     * The final stage of a port initialization.\n     * @note Must be called *after* all queues from stage (2) have been\n     *       initialized.\n     */\n    void init_port_fini();\n\n    /**\n     * Check the link status of out port in up to 9s, and print them finally.\n     */\n    void check_port_link_status();\n\n    /**\n     * Configures the HW Flow Control\n     */\n    void set_hw_flow_control();\n\npublic:\n    dpdk_device(uint16_t port_idx, uint16_t num_queues, bool use_lro,\n                bool enable_fc)\n        : _port_idx(port_idx)\n        , _num_queues(num_queues)\n        , _home_cpu(this_shard_id())\n        , _use_lro(use_lro)\n        , _enable_fc(enable_fc)\n        , _stats_plugin_name(\"network\")\n        , _stats_plugin_inst(std::string(\"port\") + std::to_string(_port_idx))\n        , _xstats(port_idx)\n    {\n\n        /* now initialise the port we will use */\n        int ret = init_port_start();\n        if (ret != 0) {\n            rte_exit(EXIT_FAILURE, \"Cannot initialise port %u\\n\", _port_idx);\n        }\n\n        // Register port statistics pollers\n        namespace sm = seastar::metrics;\n        _metrics.add_group(_stats_plugin_name, {\n            // Rx Good\n            sm::make_counter(\"rx_multicast\", _stats.rx.good.mcast,\n                            sm::description(\"Counts a number of received multicast packets.\"), {sm::shard_label(_stats_plugin_inst)}),\n            // Rx Errors\n            sm::make_counter(\"rx_crc_errors\", _stats.rx.bad.crc,\n                            sm::description(\"Counts a number of received packets with a bad CRC value. \"\n                                            \"A non-zero value of this metric usually indicates a HW problem, e.g. a bad cable.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"rx_dropped\", _stats.rx.bad.dropped,\n                            sm::description(\"Counts a number of dropped received packets. \"\n                                            \"A non-zero value of this counter indicated the overflow of ingress HW buffers. \"\n                                            \"This usually happens because of a rate of a sender on the other side of the link is higher than we can process as a receiver.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"rx_bad_length_errors\", _stats.rx.bad.len,\n                            sm::description(\"Counts a number of received packets with a bad length value. \"\n                                            \"A non-zero value of this metric usually indicates a HW issue: e.g. bad cable.\"), {sm::shard_label(_stats_plugin_inst)}),\n            // Coupled counters:\n            // Good\n            sm::make_counter(\"rx_pause_xon\", _stats.rx.good.pause_xon,\n                            sm::description(\"Counts a number of received PAUSE XON frames (PAUSE frame with a quanta of zero). \"\n                                            \"When PAUSE XON frame is received our port may resume sending L2 frames. \"\n                                            \"PAUSE XON frames are sent to resume sending that was previously paused with a PAUSE XOFF frame. If ingress \"\n                                            \"buffer falls below the low watermark threshold before the timeout configured in the original PAUSE XOFF frame the receiver may decide to send PAUSE XON frame. \"\n                                            \"A non-zero value of this metric may mean that our sender is bursty and that the spikes overwhelm the receiver on the other side of the link.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"tx_pause_xon\", _stats.tx.good.pause_xon,\n                            sm::description(\"Counts a number of sent PAUSE XON frames (L2 flow control frames). \"\n                                            \"A non-zero value of this metric indicates that our ingress path doesn't keep up with the rate of a sender on the other side of the link. \"\n                                            \"Note that if a sender port respects PAUSE frames this will prevent it from sending from ALL its egress queues because L2 flow control is defined \"\n                                            \"on a per-link resolution.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"rx_pause_xoff\", _stats.rx.good.pause_xoff,\n                            sm::description(\"Counts a number of received PAUSE XOFF frames. \"\n                                            \"A non-zero value of this metric indicates that our egress overwhelms the receiver on the other side of the link and it has to send PAUSE frames to make us stop sending. \"\n                                            \"Note that if our port respects PAUSE frames a reception of a PAUSE XOFF frame will cause ALL egress queues of this port to stop sending.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"tx_pause_xoff\", _stats.tx.good.pause_xoff,\n                            sm::description(\"Counts a number of sent PAUSE XOFF frames. \"\n                                            \"A non-zero value of this metric indicates that our ingress path (SW and HW) doesn't keep up with the rate of a sender on the other side of the link and as a result \"\n                                            \"our ingress HW buffers overflow.\"), {sm::shard_label(_stats_plugin_inst)}),\n            // Errors\n            sm::make_counter(\"rx_errors\", _stats.rx.bad.total,\n                            sm::description(\"Counts the total number of ingress errors: CRC errors, bad length errors, etc.\"), {sm::shard_label(_stats_plugin_inst)}),\n\n            sm::make_counter(\"tx_errors\", _stats.tx.bad.total,\n                            sm::description(\"Counts a total number of egress errors. A non-zero value usually indicated a problem with a HW or a SW driver.\"), {sm::shard_label(_stats_plugin_inst)}),\n        });\n    }\n\n    ~dpdk_device() {\n        _stats_collector.cancel();\n    }\n\n    ethernet_address hw_address() override {\n        struct rte_ether_addr mac;\n        rte_eth_macaddr_get(_port_idx, &mac);\n\n        return mac.addr_bytes;\n    }\n    net::hw_features hw_features() override {\n        return _hw_features;\n    }\n\n    net::hw_features& hw_features_ref() { return _hw_features; }\n\n    const rte_eth_rxconf* def_rx_conf() const {\n        return &_dev_info.default_rxconf;\n    }\n\n    const rte_eth_txconf* def_tx_conf() const {\n        return &_dev_info.default_txconf;\n    }\n\n    /**\n     *  Set the RSS table in the device and store it in the internal vector.\n     */\n    void set_rss_table();\n\n    virtual uint16_t hw_queues_count() override { return _num_queues; }\n    virtual future<> link_ready() override { return _link_ready_promise.get_future(); }\n    virtual std::unique_ptr<qp> init_local_queue(const program_options::option_group& opts, uint16_t qid) override;\n    virtual unsigned hash2qid(uint32_t hash) override {\n        SEASTAR_ASSERT(_redir_table.size());\n        return _redir_table[hash & (_redir_table.size() - 1)];\n    }\n    uint16_t port_idx() { return _port_idx; }\n    bool is_i40e_device() const {\n        return _is_i40e_device;\n    }\n    bool is_vmxnet3_device() const {\n        return _is_vmxnet3_device;\n    }\n\n    virtual rss_key_type rss_key() const override { return _rss_key; }\n};\n\ntemplate <bool HugetlbfsMemBackend>\nclass dpdk_qp : public net::qp {\n    class tx_buf_factory;\n\n    class tx_buf {\n    friend class dpdk_qp;\n    public:\n        static tx_buf* me(rte_mbuf* mbuf) {\n            return reinterpret_cast<tx_buf*>(mbuf);\n        }\n\n    private:\n        /**\n         * Checks if the original packet of a given cluster should be linearized\n         * due to HW limitations.\n         *\n         * @param head head of a cluster to check\n         *\n         * @return TRUE if a packet should be linearized.\n         */\n        static bool i40e_should_linearize(rte_mbuf *head) {\n            bool is_tso = head->ol_flags & RTE_MBUF_F_TX_TCP_SEG;\n\n            // For a non-TSO case: number of fragments should not exceed 8\n            if (!is_tso){\n                return head->nb_segs > i40e_max_xmit_segment_frags;\n            }\n\n            //\n            // For a TSO case each MSS window should not include more than 8\n            // fragments including headers.\n            //\n\n            // Calculate the number of frags containing headers.\n            //\n            // Note: we support neither VLAN nor tunneling thus headers size\n            // accounting is super simple.\n            //\n            size_t headers_size = head->l2_len + head->l3_len + head->l4_len;\n            unsigned hdr_frags = 0;\n            size_t cur_payload_len = 0;\n            rte_mbuf *cur_seg = head;\n\n            while (cur_seg && cur_payload_len < headers_size) {\n                cur_payload_len += cur_seg->data_len;\n                cur_seg = cur_seg->next;\n                hdr_frags++;\n            }\n\n            //\n            // Header fragments will be used for each TSO segment, thus the\n            // maximum number of data segments will be 8 minus the number of\n            // header fragments.\n            //\n            // It's unclear from the spec how the first TSO segment is treated\n            // if the last fragment with headers contains some data bytes:\n            // whether this fragment will be accounted as a single fragment or\n            // as two separate fragments. We prefer to play it safe and assume\n            // that this fragment will be accounted as two separate fragments.\n            //\n            size_t max_win_size = i40e_max_xmit_segment_frags - hdr_frags;\n\n            if (head->nb_segs <= max_win_size) {\n                return false;\n            }\n\n            // Get the data (without headers) part of the first data fragment\n            size_t prev_frag_data = cur_payload_len - headers_size;\n            auto mss = head->tso_segsz;\n\n            while (cur_seg) {\n                unsigned frags_in_seg = 0;\n                size_t cur_seg_size = 0;\n\n                if (prev_frag_data) {\n                    cur_seg_size = prev_frag_data;\n                    frags_in_seg++;\n                    prev_frag_data = 0;\n                }\n\n                while (cur_seg_size < mss && cur_seg) {\n                    cur_seg_size += cur_seg->data_len;\n                    cur_seg = cur_seg->next;\n                    frags_in_seg++;\n\n                    if (frags_in_seg > max_win_size) {\n                        return true;\n                    }\n                }\n\n                if (cur_seg_size > mss) {\n                    prev_frag_data = cur_seg_size - mss;\n                }\n            }\n\n            return false;\n        }\n\n        /**\n         * Sets the offload info in the head buffer of an rte_mbufs cluster.\n         *\n         * @param p an original packet the cluster is built for\n         * @param qp QP handle\n         * @param head a head of an rte_mbufs cluster\n         */\n        static void set_cluster_offload_info(const packet& p, const dpdk_qp& qp, rte_mbuf* head) {\n            // Handle TCP checksum offload\n            auto oi = p.get_offload_info();\n            if (oi.needs_ip_csum) {\n                head->ol_flags |= RTE_MBUF_F_TX_IP_CKSUM;\n                // TODO: Take a VLAN header into an account here\n                head->l2_len = sizeof(struct rte_ether_hdr);\n                head->l3_len = oi.ip_hdr_len;\n            }\n            if (qp.port().hw_features().tx_csum_l4_offload) {\n                if (oi.protocol == ip_protocol_num::tcp) {\n                    head->ol_flags |= RTE_MBUF_F_TX_TCP_CKSUM;\n                    // TODO: Take a VLAN header into an account here\n                    head->l2_len = sizeof(struct rte_ether_hdr);\n                    head->l3_len = oi.ip_hdr_len;\n\n                    if (oi.tso_seg_size) {\n                        SEASTAR_ASSERT(oi.needs_ip_csum);\n                        head->ol_flags |= RTE_MBUF_F_TX_TCP_SEG;\n                        head->l4_len = oi.tcp_hdr_len;\n                        head->tso_segsz = oi.tso_seg_size;\n                    }\n                } else if (oi.protocol == ip_protocol_num::udp) {\n                    head->ol_flags |= RTE_MBUF_F_TX_UDP_CKSUM;\n                    // TODO: Take a VLAN header into an account here\n                    head->l2_len = sizeof(struct rte_ether_hdr);\n                    head->l3_len = oi.ip_hdr_len;\n                }\n            }\n        }\n\n        /**\n         * Creates a tx_buf cluster representing a given packet in a \"zero-copy\"\n         * way.\n         *\n         * @param p packet to translate\n         * @param qp dpdk_qp handle\n         *\n         * @return the HEAD tx_buf of the cluster or nullptr in case of a\n         *         failure\n         */\n        static tx_buf* from_packet_zc(packet&& p, dpdk_qp& qp) {\n\n            // Too fragmented - linearize\n            if (p.nr_frags() > max_frags) {\n                p.linearize();\n                ++qp._stats.tx.linearized;\n            }\n\nbuild_mbuf_cluster:\n            rte_mbuf *head = nullptr, *last_seg = nullptr;\n            unsigned nsegs = 0;\n\n            // Create a HEAD of the fragmented packet\n            if (!translate_one_frag(qp, p.frag(0), head, last_seg, nsegs)) {\n                return nullptr;\n            }\n\n            unsigned total_nsegs = nsegs;\n\n            for (unsigned i = 1; i < p.nr_frags(); i++) {\n                rte_mbuf *h = nullptr, *new_last_seg = nullptr;\n                if (!translate_one_frag(qp, p.frag(i), h, new_last_seg, nsegs)) {\n                    me(head)->recycle();\n                    return nullptr;\n                }\n\n                total_nsegs += nsegs;\n\n                // Attach a new buffers' chain to the packet chain\n                last_seg->next = h;\n                last_seg = new_last_seg;\n            }\n\n            // Update the HEAD buffer with the packet info\n            head->pkt_len = p.len();\n            head->nb_segs = total_nsegs;\n\n            set_cluster_offload_info(p, qp, head);\n\n            //\n            // If a packet hasn't been linearized already and the resulting\n            // cluster requires the linearisation due to HW limitation:\n            //\n            //    - Recycle the cluster.\n            //    - Linearize the packet.\n            //    - Build the cluster once again\n            //\n            if (head->nb_segs > max_frags ||\n                (p.nr_frags() > 1 && qp.port().is_i40e_device() && i40e_should_linearize(head)) ||\n                (p.nr_frags() > vmxnet3_max_xmit_segment_frags && qp.port().is_vmxnet3_device())) {\n                me(head)->recycle();\n                p.linearize();\n                ++qp._stats.tx.linearized;\n\n                goto build_mbuf_cluster;\n            }\n\n            me(last_seg)->set_packet(std::move(p));\n\n            return me(head);\n        }\n\n        /**\n         * Copy the contents of the \"packet\" into the given cluster of\n         * rte_mbuf's.\n         *\n         * @note Size of the cluster has to be big enough to accommodate all the\n         *       contents of the given packet.\n         *\n         * @param p packet to copy\n         * @param head head of the rte_mbuf's cluster\n         */\n        static void copy_packet_to_cluster(const packet& p, rte_mbuf* head) {\n            rte_mbuf* cur_seg = head;\n            size_t cur_seg_offset = 0;\n            unsigned cur_frag_idx = 0;\n            size_t cur_frag_offset = 0;\n\n            while (true) {\n                size_t to_copy = std::min(p.frag(cur_frag_idx).size - cur_frag_offset,\n                                          inline_mbuf_data_size - cur_seg_offset);\n\n                memcpy(rte_pktmbuf_mtod_offset(cur_seg, void*, cur_seg_offset),\n                       p.frag(cur_frag_idx).base + cur_frag_offset, to_copy);\n\n                cur_frag_offset += to_copy;\n                cur_seg_offset += to_copy;\n\n                if (cur_frag_offset >= p.frag(cur_frag_idx).size) {\n                    ++cur_frag_idx;\n                    if (cur_frag_idx >= p.nr_frags()) {\n                        //\n                        // We are done - set the data size of the last segment\n                        // of the cluster.\n                        //\n                        cur_seg->data_len = cur_seg_offset;\n                        break;\n                    }\n\n                    cur_frag_offset = 0;\n                }\n\n                if (cur_seg_offset >= inline_mbuf_data_size) {\n                    cur_seg->data_len = inline_mbuf_data_size;\n                    cur_seg = cur_seg->next;\n                    cur_seg_offset = 0;\n\n                    // FIXME: assert in a fast-path - remove!!!\n                    SEASTAR_ASSERT(cur_seg);\n                }\n            }\n        }\n\n        /**\n         * Creates a tx_buf cluster representing a given packet in a \"copy\" way.\n         *\n         * @param p packet to translate\n         * @param qp dpdk_qp handle\n         *\n         * @return the HEAD tx_buf of the cluster or nullptr in case of a\n         *         failure\n         */\n        static tx_buf* from_packet_copy(packet&& p, dpdk_qp& qp) {\n            // sanity\n            if (!p.len()) {\n                return nullptr;\n            }\n\n            /*\n             * Here we are going to use the fact that the inline data size is a\n             * power of two.\n             *\n             * We will first try to allocate the cluster and only if we are\n             * successful - we will go and copy the data.\n             */\n            auto aligned_len = align_up((size_t)p.len(), inline_mbuf_data_size);\n            unsigned nsegs = aligned_len / inline_mbuf_data_size;\n            rte_mbuf *head = nullptr, *last_seg = nullptr;\n\n            tx_buf* buf = qp.get_tx_buf();\n            if (!buf) {\n                return nullptr;\n            }\n\n            head = buf->rte_mbuf_p();\n            last_seg = head;\n            for (unsigned i = 1; i < nsegs; i++) {\n                buf = qp.get_tx_buf();\n                if (!buf) {\n                    me(head)->recycle();\n                    return nullptr;\n                }\n\n                last_seg->next = buf->rte_mbuf_p();\n                last_seg = last_seg->next;\n            }\n\n            //\n            // If we've got here means that we have succeeded already!\n            // We only need to copy the data and set the head buffer with the\n            // relevant info.\n            //\n            head->pkt_len = p.len();\n            head->nb_segs = nsegs;\n\n            copy_packet_to_cluster(p, head);\n            set_cluster_offload_info(p, qp, head);\n\n            return me(head);\n        }\n\n        /**\n         * Zero-copy handling of a single net::fragment.\n         *\n         * @param do_one_buf Functor responsible for a single rte_mbuf\n         *                   handling\n         * @param qp dpdk_qp handle (in)\n         * @param frag Fragment to copy (in)\n         * @param head Head of the cluster (out)\n         * @param last_seg Last segment of the cluster (out)\n         * @param nsegs Number of segments in the cluster (out)\n         *\n         * @return TRUE in case of success\n         */\n        template <class DoOneBufFunc>\n        static bool do_one_frag(DoOneBufFunc do_one_buf, dpdk_qp& qp,\n                                fragment& frag, rte_mbuf*& head,\n                                rte_mbuf*& last_seg, unsigned& nsegs) {\n            size_t len, left_to_set = frag.size;\n            char* base = frag.base;\n\n            rte_mbuf* m;\n\n            // TODO: SEASTAR_ASSERT() in a fast path! Remove me ASAP!\n            SEASTAR_ASSERT(frag.size);\n\n            // Create a HEAD of mbufs' cluster and set the first bytes into it\n            len = do_one_buf(qp, head, base, left_to_set);\n            if (!len) {\n                return false;\n            }\n\n            left_to_set -= len;\n            base += len;\n            nsegs = 1;\n\n            //\n            // Set the rest of the data into the new mbufs and chain them to\n            // the cluster.\n            //\n            rte_mbuf* prev_seg = head;\n            while (left_to_set) {\n                len = do_one_buf(qp, m, base, left_to_set);\n                if (!len) {\n                    me(head)->recycle();\n                    return false;\n                }\n\n                left_to_set -= len;\n                base += len;\n                nsegs++;\n\n                prev_seg->next = m;\n                prev_seg = m;\n            }\n\n            // Return the last mbuf in the cluster\n            last_seg = prev_seg;\n\n            return true;\n        }\n\n        /**\n         * Zero-copy handling of a single net::fragment.\n         *\n         * @param qp dpdk_qp handle (in)\n         * @param frag Fragment to copy (in)\n         * @param head Head of the cluster (out)\n         * @param last_seg Last segment of the cluster (out)\n         * @param nsegs Number of segments in the cluster (out)\n         *\n         * @return TRUE in case of success\n         */\n        static bool translate_one_frag(dpdk_qp& qp, fragment& frag,\n                                       rte_mbuf*& head, rte_mbuf*& last_seg,\n                                       unsigned& nsegs) {\n            return do_one_frag(set_one_data_buf, qp, frag, head,\n                               last_seg, nsegs);\n        }\n\n        /**\n         * Copies one net::fragment into the cluster of rte_mbuf's.\n         *\n         * @param qp dpdk_qp handle (in)\n         * @param frag Fragment to copy (in)\n         * @param head Head of the cluster (out)\n         * @param last_seg Last segment of the cluster (out)\n         * @param nsegs Number of segments in the cluster (out)\n         *\n         * We return the \"last_seg\" to avoid traversing the cluster in order to get\n         * it.\n         *\n         * @return TRUE in case of success\n         */\n        static bool copy_one_frag(dpdk_qp& qp, fragment& frag,\n                                  rte_mbuf*& head, rte_mbuf*& last_seg,\n                                  unsigned& nsegs) {\n            return do_one_frag(copy_one_data_buf, qp, frag, head,\n                               last_seg, nsegs);\n        }\n\n        /**\n         * Allocates a single rte_mbuf and sets it to point to a given data\n         * buffer.\n         *\n         * @param qp dpdk_qp handle (in)\n         * @param m New allocated rte_mbuf (out)\n         * @param va virtual address of a data buffer (in)\n         * @param buf_len length of the data to copy (in)\n         *\n         * @return The actual number of bytes that has been set in the mbuf\n         */\n        static size_t set_one_data_buf(\n            dpdk_qp& qp, rte_mbuf*& m, char* va, size_t buf_len) {\n            static constexpr size_t max_frag_len = 15_KiB;\n\n            //\n            // Currently we break a buffer on a 15K boundary because 82599\n            // devices have a 15.5K limitation on a maximum single fragment\n            // size.\n            //\n            rte_iova_t iova = rte_mem_virt2iova(va);\n\n            if (iova == RTE_BAD_IOVA) {\n                return copy_one_data_buf(qp, m, va, buf_len);\n            }\n\n            tx_buf* buf = qp.get_tx_buf();\n            if (!buf) {\n                return 0;\n            }\n\n            size_t len = std::min(buf_len, max_frag_len);\n\n            buf->set_zc_info(va, iova, len);\n            m = buf->rte_mbuf_p();\n\n            return len;\n        }\n\n        /**\n         *  Allocates a single rte_mbuf and copies a given data into it.\n         *\n         * @param qp dpdk_qp handle (in)\n         * @param m New allocated rte_mbuf (out)\n         * @param data Data to copy from (in)\n         * @param buf_len length of the data to copy (in)\n         *\n         * @return The actual number of bytes that has been copied\n         */\n        static size_t copy_one_data_buf(\n            dpdk_qp& qp, rte_mbuf*& m, char* data, size_t buf_len)\n        {\n            tx_buf* buf = qp.get_tx_buf();\n            if (!buf) {\n                return 0;\n            }\n\n            size_t len = std::min(buf_len, inline_mbuf_data_size);\n\n            m = buf->rte_mbuf_p();\n\n            // mbuf_put()\n            m->data_len = len;\n            m->pkt_len  = len;\n\n            qp._stats.tx.good.update_copy_stats(1, len);\n\n            memcpy(rte_pktmbuf_mtod(m, void*), data, len);\n\n            return len;\n        }\n\n    public:\n        tx_buf(tx_buf_factory& fc) : _fc(fc) {\n\n            _buf_iova     = _mbuf.buf_iova;\n            _data_off     = _mbuf.data_off;\n        }\n\n        rte_mbuf* rte_mbuf_p() { return &_mbuf; }\n\n        void set_zc_info(void* va, rte_iova_t iova, size_t len) {\n            // mbuf_put()\n            _mbuf.data_len           = len;\n            _mbuf.pkt_len            = len;\n\n            // Set the mbuf to point to our data\n            _mbuf.buf_addr           = va;\n            _mbuf.buf_iova           = iova;\n            _mbuf.data_off           = 0;\n            _is_zc                   = true;\n        }\n\n        void reset_zc() {\n            //\n            // If this mbuf was the last in a cluster and contains an\n            // original packet object then call the destructor of the\n            // original packet object.\n            //\n            if (_p) {\n                //\n                // Reset the std::optional. This in particular is going\n                // to call the \"packet\"'s destructor and reset the\n                // \"optional\" state to \"nonengaged\".\n                //\n                _p = std::nullopt;\n\n            } else if (!_is_zc) {\n                return;\n            }\n\n            // Restore the rte_mbuf fields we trashed in set_zc_info()\n            _mbuf.buf_iova     = _buf_iova;\n            _mbuf.buf_addr     = rte_mbuf_to_baddr(&_mbuf);\n            _mbuf.data_off     = _data_off;\n\n            _is_zc             = false;\n        }\n\n        void recycle() {\n            struct rte_mbuf *m = &_mbuf, *m_next;\n\n            while (m != nullptr) {\n                m_next = m->next;\n                rte_pktmbuf_reset(m);\n                _fc.put(me(m));\n                m = m_next;\n            }\n        }\n\n        void set_packet(packet&& p) {\n            _p = std::move(p);\n        }\n\n    private:\n        struct rte_mbuf _mbuf;\n        MARKER private_start;\n        std::optional<packet> _p;\n        rte_iova_t _buf_iova;\n        uint16_t _data_off;\n        // TRUE if underlying mbuf has been used in the zero-copy flow\n        bool _is_zc = false;\n        // buffers' factory the buffer came from\n        tx_buf_factory& _fc;\n        MARKER private_end;\n    };\n\n    class tx_buf_factory {\n        //\n        // Number of buffers to free in each GC iteration:\n        // We want the buffers to be allocated from the mempool as many as\n        // possible.\n        //\n        // On the other hand if there is no Tx for some time we want the\n        // completions to be eventually handled. Thus we choose the smallest\n        // possible packets count number here.\n        //\n        static constexpr int gc_count = 1;\n    public:\n        tx_buf_factory(uint16_t qid) {\n            using namespace memory;\n\n            sstring name = sstring(pktmbuf_pool_name) + to_sstring(qid) + \"_tx\";\n            printf(\"Creating Tx mbuf pool '%s' [%u mbufs] ...\\n\",\n                   name.c_str(), mbufs_per_queue_tx);\n\n            if (HugetlbfsMemBackend) {\n                size_t xmem_size;\n\n                _xmem.reset(dpdk_qp::alloc_mempool_xmem(mbufs_per_queue_tx,\n                                                        inline_mbuf_size,\n                                                        xmem_size));\n                if (!_xmem.get()) {\n                    printf(\"Can't allocate a memory for Tx buffers\\n\");\n                    exit(1);\n                }\n\n                //\n                // We are going to push the buffers from the mempool into\n                // the circular_buffer and then poll them from there anyway, so\n                // we prefer to make a mempool non-atomic in this case.\n                //\n                _pool =\n                    rte_mempool_create_empty(name.c_str(),\n                                             mbufs_per_queue_tx,\n                                             inline_mbuf_size,\n                                             mbuf_cache_size,\n                                             sizeof(struct rte_pktmbuf_pool_private),\n                                             rte_socket_id(), 0);\n                if (_pool) {\n                    rte_pktmbuf_pool_init(_pool, nullptr);\n\n                    if (rte_mempool_populate_virt(_pool, (char*)(_xmem.get()),\n                                                  xmem_size, page_size,\n                                                  nullptr, nullptr) <= 0) {\n                        printf(\"Failed to populate mempool for Tx\\n\");\n                        exit(1);\n                    }\n\n                    rte_mempool_obj_iter(_pool, rte_pktmbuf_init, nullptr);\n                }\n\n            } else {\n                _pool =\n                    rte_mempool_create(name.c_str(),\n                                       mbufs_per_queue_tx, inline_mbuf_size,\n                                       mbuf_cache_size,\n                                       sizeof(struct rte_pktmbuf_pool_private),\n                                       rte_pktmbuf_pool_init, nullptr,\n                                       rte_pktmbuf_init, nullptr,\n                                       rte_socket_id(), 0);\n            }\n\n            if (!_pool) {\n                printf(\"Failed to create mempool for Tx\\n\");\n                exit(1);\n            }\n\n            //\n            // Fill the factory with the buffers from the mempool allocated\n            // above.\n            //\n            init_factory();\n        }\n\n        /**\n         * @note Should not be called if there are no free tx_buf's\n         *\n         * @return a free tx_buf object\n         */\n        tx_buf* get() {\n            // Take completed from the HW first\n            tx_buf *pkt = get_one_completed();\n            if (pkt) {\n                if (HugetlbfsMemBackend) {\n                    pkt->reset_zc();\n                }\n\n                return pkt;\n            }\n\n            //\n            // If there are no completed at the moment - take from the\n            // factory's cache.\n            //\n            if (_ring.empty()) {\n                return nullptr;\n            }\n\n            pkt = _ring.back();\n            _ring.pop_back();\n\n            return pkt;\n        }\n\n        void put(tx_buf* buf) {\n            if (HugetlbfsMemBackend) {\n                buf->reset_zc();\n            }\n            _ring.push_back(buf);\n        }\n\n        bool gc() {\n            for (int cnt = 0; cnt < gc_count; ++cnt) {\n                auto tx_buf_p = get_one_completed();\n                if (!tx_buf_p) {\n                    return false;\n                }\n\n                put(tx_buf_p);\n            }\n\n            return true;\n        }\n    private:\n        /**\n         * Fill the mbufs circular buffer: after this the _pool will become\n         * empty. We will use it to catch the completed buffers:\n         *\n         * - Underlying PMD drivers will \"free\" the mbufs once they are\n         *   completed.\n         * - We will poll the _pktmbuf_pool_tx till it's empty and release\n         *   all the buffers from the freed mbufs.\n         */\n        void init_factory() {\n            while (rte_mbuf* mbuf = rte_pktmbuf_alloc(_pool)) {\n                _ring.push_back(new(tx_buf::me(mbuf)) tx_buf{*this});\n            }\n        }\n\n        /**\n         * PMD puts the completed buffers back into the mempool they have\n         * originally come from.\n         *\n         * @note rte_pktmbuf_alloc() resets the mbuf so there is no need to call\n         *       rte_pktmbuf_reset() here again.\n         *\n         * @return a single tx_buf that has been completed by HW.\n         */\n        tx_buf* get_one_completed() {\n            return tx_buf::me(rte_pktmbuf_alloc(_pool));\n        }\n\n    private:\n        std::vector<tx_buf*> _ring;\n        rte_mempool* _pool = nullptr;\n        std::unique_ptr<void, free_deleter> _xmem;\n    };\n\npublic:\n    explicit dpdk_qp(dpdk_device* dev, uint16_t qid,\n                     const std::string stats_plugin_name);\n\n    virtual void rx_start() override;\n    virtual future<> send(packet p) override {\n        abort();\n    }\n    virtual ~dpdk_qp() {}\n\n    virtual uint32_t send(circular_buffer<packet>& pb) override {\n        if (HugetlbfsMemBackend) {\n            // Zero-copy send\n            return _send(pb, [&] (packet&& p) {\n                return tx_buf::from_packet_zc(std::move(p), *this);\n            });\n        } else {\n            // \"Copy\"-send\n            return _send(pb, [&](packet&& p) {\n                return tx_buf::from_packet_copy(std::move(p), *this);\n            });\n        }\n    }\n\n    dpdk_device& port() const { return *_dev; }\n    tx_buf* get_tx_buf() { return _tx_buf_factory.get(); }\nprivate:\n\n    template <class Func>\n    uint32_t _send(circular_buffer<packet>& pb, Func packet_to_tx_buf_p) {\n        if (_tx_burst.size() == 0) {\n            for (auto&& p : pb) {\n                // TODO: SEASTAR_ASSERT() in a fast path! Remove me ASAP!\n                SEASTAR_ASSERT(p.len());\n\n                tx_buf* buf = packet_to_tx_buf_p(std::move(p));\n                if (!buf) {\n                    break;\n                }\n\n                _tx_burst.push_back(buf->rte_mbuf_p());\n            }\n        }\n\n        uint16_t sent = rte_eth_tx_burst(_dev->port_idx(), _qid,\n                                         _tx_burst.data() + _tx_burst_idx,\n                                         _tx_burst.size() - _tx_burst_idx);\n\n        uint64_t nr_frags = 0, bytes = 0;\n\n        for (int i = 0; i < sent; i++) {\n            rte_mbuf* m = _tx_burst[_tx_burst_idx + i];\n            bytes    += m->pkt_len;\n            nr_frags += m->nb_segs;\n            pb.pop_front();\n        }\n\n        _stats.tx.good.update_frags_stats(nr_frags, bytes);\n\n        _tx_burst_idx += sent;\n\n        if (_tx_burst_idx == _tx_burst.size()) {\n            _tx_burst_idx = 0;\n            _tx_burst.clear();\n        }\n\n        return sent;\n    }\n\n    /**\n     * Allocate a new data buffer and set the mbuf to point to it.\n     *\n     * Do some DPDK hacks to work on PMD: it assumes that the buf_addr\n     * points to the private data of RTE_PKTMBUF_HEADROOM before the actual\n     * data buffer.\n     *\n     * @param m mbuf to update\n     */\n    static bool refill_rx_mbuf(rte_mbuf* m, size_t size = mbuf_data_size) {\n        char* data;\n\n        if (posix_memalign((void**)&data, size, size)) {\n            return false;\n        }\n\n        rte_iova_t iova = rte_mem_virt2iova(data);\n\n        //\n        // Set the mbuf to point to our data.\n        //\n        // Do some DPDK hacks to work on PMD: it assumes that the buf_addr\n        // points to the private data of RTE_PKTMBUF_HEADROOM before the\n        // actual data buffer.\n        //\n        m->buf_addr      = data - RTE_PKTMBUF_HEADROOM;\n        m->buf_iova      = iova - RTE_PKTMBUF_HEADROOM;\n        return true;\n    }\n\n    static bool init_noninline_rx_mbuf(rte_mbuf* m,\n                                       size_t size = mbuf_data_size) {\n        if (!refill_rx_mbuf(m, size)) {\n            return false;\n        }\n        // The below fields stay constant during the execution.\n        m->buf_len       = size + RTE_PKTMBUF_HEADROOM;\n        m->data_off      = RTE_PKTMBUF_HEADROOM;\n        return true;\n    }\n\n    bool init_rx_mbuf_pool();\n    bool map_dma();\n    bool rx_gc();\n    bool refill_one_cluster(rte_mbuf* head);\n\n    /**\n     * Allocates a memory chunk to accommodate the given number of buffers of\n     * the given size and fills a vector with underlying physical pages.\n     *\n     * The chunk is going to be used as an external memory buffer of the DPDK\n     * memory pool.\n     *\n     * The chunk size if calculated using get_mempool_xmem_size() function.\n     *\n     * @param num_bufs  Number of buffers (in)\n     * @param buf_sz    Size of each buffer (in)\n     * @param xmem_size Size of allocated memory chunk (out)\n     *\n     * @return a virtual address of the allocated memory chunk or nullptr in\n     *         case of a failure.\n     */\n    static void* alloc_mempool_xmem(uint16_t num_bufs, uint16_t buf_sz,\n                                    size_t& xmem_size);\n\n    /**\n     * Polls for a burst of incoming packets. This function will not block and\n     * will immediately return after processing all available packets.\n     *\n     */\n    bool poll_rx_once();\n\n    /**\n     * Translates an rte_mbuf's into net::packet and feeds them to _rx_stream.\n     *\n     * @param bufs An array of received rte_mbuf's\n     * @param count Number of buffers in the bufs[]\n     */\n    void process_packets(struct rte_mbuf **bufs, uint16_t count);\n\n    /**\n     * Translate rte_mbuf into the \"packet\".\n     * @param m mbuf to translate\n     *\n     * @return a \"optional\" object representing the newly received data if in an\n     *         \"engaged\" state or an error if in a \"disengaged\" state.\n     */\n    std::optional<packet> from_mbuf(rte_mbuf* m);\n\n    /**\n     * Transform an LRO rte_mbuf cluster into the \"packet\" object.\n     * @param m HEAD of the mbufs' cluster to transform\n     *\n     * @return a \"optional\" object representing the newly received LRO packet if\n     *         in an \"engaged\" state or an error if in a \"disengaged\" state.\n     */\n    std::optional<packet> from_mbuf_lro(rte_mbuf* m);\n\nprivate:\n    dpdk_device* _dev;\n    uint16_t _qid;\n    rte_mempool *_pktmbuf_pool_rx;\n    std::vector<rte_mbuf*> _rx_free_pkts;\n    std::vector<rte_mbuf*> _rx_free_bufs;\n    std::vector<fragment> _frags;\n    std::vector<char*> _bufs;\n    size_t _num_rx_free_segs = 0;\n    internal::poller _rx_gc_poller;\n    std::unique_ptr<void, free_deleter> _rx_xmem;\n    tx_buf_factory _tx_buf_factory;\n    std::optional<reactor::poller> _rx_poller;\n    internal::poller _tx_gc_poller;\n    std::vector<rte_mbuf*> _tx_burst;\n    uint16_t _tx_burst_idx = 0;\n    static constexpr phys_addr_t page_mask = ~(memory::page_size - 1);\n};\n\nint dpdk_device::init_port_start()\n{\n    SEASTAR_ASSERT(_port_idx < rte_eth_dev_count_avail());\n\n    rte_eth_dev_info_get(_port_idx, &_dev_info);\n\n    //\n    // This is a workaround for a missing handling of a HW limitation in the\n    // DPDK i40e driver. This and all related to _is_i40e_device code should be\n    // removed once this handling is added.\n    //\n    if (sstring(\"rte_i40evf_pmd\") == _dev_info.driver_name ||\n        sstring(\"rte_i40e_pmd\") == _dev_info.driver_name) {\n        printf(\"Device is an Intel's 40G NIC. Enabling 8 fragments hack!\\n\");\n        _is_i40e_device = true;\n    }\n\n    if (std::string(\"rte_vmxnet3_pmd\") == _dev_info.driver_name) {\n      printf(\"Device is a VMWare Virtual NIC. Enabling 16 fragments hack!\\n\");\n      _is_vmxnet3_device = true;\n    }\n\n    //\n    // Another workaround: this time for a lack of number of RSS bits.\n    // ixgbe PF NICs support up to 16 RSS queues.\n    // ixgbe VF NICs support up to 4 RSS queues.\n    // i40e PF NICs support up to 64 RSS queues.\n    // i40e VF NICs support up to 16 RSS queues.\n    //\n    if (sstring(\"rte_ixgbe_pmd\") == _dev_info.driver_name) {\n        _dev_info.max_rx_queues = std::min(_dev_info.max_rx_queues, (uint16_t)16);\n    } else if (sstring(\"rte_ixgbevf_pmd\") == _dev_info.driver_name) {\n        _dev_info.max_rx_queues = std::min(_dev_info.max_rx_queues, (uint16_t)4);\n    } else if (sstring(\"rte_i40e_pmd\") == _dev_info.driver_name) {\n        _dev_info.max_rx_queues = std::min(_dev_info.max_rx_queues, (uint16_t)64);\n    } else if (sstring(\"rte_i40evf_pmd\") == _dev_info.driver_name) {\n        _dev_info.max_rx_queues = std::min(_dev_info.max_rx_queues, (uint16_t)16);\n    }\n\n    // Hardware offload capabilities\n    // https://github.com/DPDK/dpdk/blob/v19.05/lib/librte_ethdev/rte_ethdev.h#L993-L1074\n\n    // We want to support all available offload features\n    // TODO: below features are implemented in 17.05, should support new ones\n    const uint64_t tx_offloads_wanted =\n        RTE_ETH_TX_OFFLOAD_VLAN_INSERT      |\n        RTE_ETH_TX_OFFLOAD_IPV4_CKSUM       |\n        RTE_ETH_TX_OFFLOAD_UDP_CKSUM        |\n        RTE_ETH_TX_OFFLOAD_TCP_CKSUM        |\n        RTE_ETH_TX_OFFLOAD_SCTP_CKSUM       |\n        RTE_ETH_TX_OFFLOAD_TCP_TSO          |\n        RTE_ETH_TX_OFFLOAD_UDP_TSO          |\n        RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM |\n        RTE_ETH_TX_OFFLOAD_QINQ_INSERT      |\n        RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO    |\n        RTE_ETH_TX_OFFLOAD_GRE_TNL_TSO      |\n        RTE_ETH_TX_OFFLOAD_IPIP_TNL_TSO     |\n        RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO   |\n        RTE_ETH_TX_OFFLOAD_MACSEC_INSERT;\n\n    _dev_info.default_txconf.offloads =\n        _dev_info.tx_offload_capa & tx_offloads_wanted;\n\n    /* for port configuration all features are off by default */\n    rte_eth_conf port_conf = { 0 };\n\n    /* setting tx offloads for port */\n    port_conf.txmode.offloads = _dev_info.default_txconf.offloads;\n\n    printf(\"Port %d: max_rx_queues %d max_tx_queues %d\\n\",\n           _port_idx, _dev_info.max_rx_queues, _dev_info.max_tx_queues);\n\n    _num_queues = std::min({_num_queues, _dev_info.max_rx_queues, _dev_info.max_tx_queues});\n\n    printf(\"Port %d: using %d %s\\n\", _port_idx, _num_queues,\n           (_num_queues > 1) ? \"queues\" : \"queue\");\n\n    // Set RSS mode: enable RSS if seastar is configured with more than 1 CPU.\n    // Even if port has a single queue we still want the RSS feature to be\n    // available in order to make HW calculate RSS hash for us.\n    if (smp::count > 1) {\n        if (_dev_info.hash_key_size == 40) {\n            _rss_key = default_rsskey_40bytes;\n        } else if (_dev_info.hash_key_size == 52) {\n            _rss_key = default_rsskey_52bytes;\n        } else if (_dev_info.hash_key_size != 0) {\n            // WTF?!!\n            rte_exit(EXIT_FAILURE,\n                \"Port %d: We support only 40 or 52 bytes RSS hash keys, %d bytes key requested\",\n                _port_idx, _dev_info.hash_key_size);\n        } else {\n            _rss_key = default_rsskey_40bytes;\n            _dev_info.hash_key_size = 40;\n        }\n\n        port_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_RSS;\n        /* enable all supported rss offloads */\n        port_conf.rx_adv_conf.rss_conf.rss_hf = _dev_info.flow_type_rss_offloads;\n        if (_dev_info.hash_key_size) {\n            port_conf.rx_adv_conf.rss_conf.rss_key = const_cast<uint8_t *>(_rss_key.data());\n            port_conf.rx_adv_conf.rss_conf.rss_key_len = _dev_info.hash_key_size;\n        }\n    } else {\n        port_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;\n    }\n\n    if (_num_queues > 1) {\n        if (_dev_info.reta_size) {\n            // RETA size should be a power of 2\n            SEASTAR_ASSERT((_dev_info.reta_size & (_dev_info.reta_size - 1)) == 0);\n\n            // Set the RSS table to the correct size\n            _redir_table.resize(_dev_info.reta_size);\n            _rss_table_bits = std::lround(std::log2(_dev_info.reta_size));\n            printf(\"Port %d: RSS table size is %d\\n\",\n                   _port_idx, _dev_info.reta_size);\n        } else {\n            _rss_table_bits = std::lround(std::log2(_dev_info.max_rx_queues));\n        }\n    } else {\n        _redir_table.push_back(0);\n    }\n\n    // Set Rx VLAN stripping\n    if (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_VLAN_STRIP) {\n        port_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_VLAN_STRIP;\n    }\n\n#ifdef RTE_ETHDEV_HAS_LRO_SUPPORT\n    // Enable LRO\n    if (_use_lro && (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_TCP_LRO)) {\n        printf(\"LRO is on\\n\");\n        port_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_TCP_LRO;\n        _hw_features.rx_lro = true;\n    } else\n#endif\n        printf(\"LRO is off\\n\");\n\n    // Check that all CSUM features are either all set all together or not set\n    // all together. If this assumption breaks we need to rework the below logic\n    // by splitting the csum offload feature bit into separate bits for IPv4,\n    // TCP and UDP.\n    SEASTAR_ASSERT(((_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_IPV4_CKSUM) &&\n            (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_UDP_CKSUM) &&\n            (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_TCP_CKSUM)) ||\n           (!(_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_IPV4_CKSUM) &&\n            !(_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_UDP_CKSUM) &&\n            !(_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_TCP_CKSUM)));\n\n    // Set Rx checksum checking\n    if (  (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_IPV4_CKSUM) &&\n          (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_UDP_CKSUM) &&\n          (_dev_info.rx_offload_capa & RTE_ETH_RX_OFFLOAD_TCP_CKSUM)) {\n        printf(\"RX checksum offload supported\\n\");\n        port_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_CHECKSUM;\n        _hw_features.rx_csum_offload = 1;\n    }\n\n    if ((_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_IPV4_CKSUM)) {\n        printf(\"TX ip checksum offload supported\\n\");\n        _hw_features.tx_csum_ip_offload = 1;\n    }\n\n    // TSO is supported starting from DPDK v1.8\n    if (_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_TCP_TSO) {\n        printf(\"TSO is supported\\n\");\n        _hw_features.tx_tso = 1;\n    }\n\n    // There is no UFO support in the PMDs yet.\n#if 0\n    if (_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_UDP_TSO) {\n        printf(\"UFO is supported\\n\");\n        _hw_features.tx_ufo = 1;\n    }\n#endif\n\n    // Check that Tx TCP and UDP CSUM features are either all set all together\n    // or not set all together. If this assumption breaks we need to rework the\n    // below logic by splitting the csum offload feature bit into separate bits\n    // for TCP and UDP.\n    SEASTAR_ASSERT(((_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_UDP_CKSUM) &&\n            (_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_TCP_CKSUM)) ||\n           (!(_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_UDP_CKSUM) &&\n            !(_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_TCP_CKSUM)));\n\n    if (  (_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_UDP_CKSUM) &&\n          (_dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_TCP_CKSUM)) {\n        printf(\"TX TCP&UDP checksum offload supported\\n\");\n        _hw_features.tx_csum_l4_offload = 1;\n    }\n\n    int retval;\n\n    printf(\"Port %u init ... \", _port_idx);\n    fflush(stdout);\n\n    /*\n     * Standard DPDK port initialisation - config port, then set up\n     * rx and tx rings.\n      */\n    if ((retval = rte_eth_dev_configure(_port_idx, _num_queues, _num_queues,\n                                        &port_conf)) != 0) {\n        return retval;\n    }\n\n    //rte_eth_promiscuous_enable(port_num);\n    printf(\"done: \\n\");\n\n    return 0;\n}\n\nvoid dpdk_device::set_hw_flow_control()\n{\n    // Read the port's current/default flow control settings\n    struct rte_eth_fc_conf fc_conf;\n    auto ret = rte_eth_dev_flow_ctrl_get(_port_idx, &fc_conf);\n\n    if (ret == -ENOTSUP) {\n        goto not_supported;\n    }\n\n    if (ret < 0) {\n        rte_exit(EXIT_FAILURE, \"Port %u: failed to get hardware flow control settings: (error %d)\\n\", _port_idx, ret);\n    }\n\n    if (_enable_fc) {\n        fc_conf.mode = RTE_ETH_FC_FULL;\n    } else {\n        fc_conf.mode = RTE_ETH_FC_NONE;\n    }\n\n    ret = rte_eth_dev_flow_ctrl_set(_port_idx, &fc_conf);\n    if (ret == -ENOTSUP) {\n        goto not_supported;\n    }\n\n    if (ret < 0) {\n        rte_exit(EXIT_FAILURE, \"Port %u: failed to set hardware flow control (error %d)\\n\", _port_idx, ret);\n    }\n\n    printf(\"Port %u: %s HW FC\\n\", _port_idx,\n                                  (_enable_fc ? \"Enabling\" : \"Disabling\"));\n    return;\n\nnot_supported:\n    printf(\"Port %u: Changing HW FC settings is not supported\\n\", _port_idx);\n}\n\nvoid dpdk_device::init_port_fini()\n{\n    // Changing FC requires HW reset, so set it before the port is initialized.\n    set_hw_flow_control();\n\n    if (rte_eth_dev_start(_port_idx) < 0) {\n        rte_exit(EXIT_FAILURE, \"Cannot start port %d\\n\", _port_idx);\n    }\n\n    /* need to defer initialize xstats since NIC specific xstat entries\n       show up only after port initization */\n    _xstats.start();\n\n    _stats_collector.set_callback([&] {\n        rte_eth_stats rte_stats = {};\n        int rc = rte_eth_stats_get(_port_idx, &rte_stats);\n\n        if (rc) {\n            printf(\"Failed to get port statistics: %s\\n\", strerror(rc));\n        }\n\n        _stats.rx.good.mcast      =\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_multicast_packets);\n        _stats.rx.good.pause_xon  =\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_xon_packets);\n        _stats.rx.good.pause_xoff =\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_xoff_packets);\n\n        _stats.rx.bad.crc        =\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_crc_errors);\n        _stats.rx.bad.len         =\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_length_errors) +\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_undersize_errors) +\n            _xstats.get_value(dpdk_xstats::xstat_id::rx_oversize_errors);\n        _stats.rx.bad.total       = rte_stats.ierrors;\n\n        _stats.tx.good.pause_xon  =\n            _xstats.get_value(dpdk_xstats::xstat_id::tx_xon_packets);\n        _stats.tx.good.pause_xoff =\n            _xstats.get_value(dpdk_xstats::xstat_id::tx_xoff_packets);\n\n        _stats.tx.bad.total       = rte_stats.oerrors;\n    });\n\n    // TODO: replace deprecated filter api with generic flow api\n    if (_num_queues > 1) {\n        set_rss_table();\n    }\n\n    // Wait for a link\n    check_port_link_status();\n\n    printf(\"Created DPDK device\\n\");\n}\n\ntemplate <bool HugetlbfsMemBackend>\nvoid* dpdk_qp<HugetlbfsMemBackend>::alloc_mempool_xmem(\n    uint16_t num_bufs, uint16_t buf_sz, size_t& xmem_size)\n{\n    using namespace memory;\n    char* xmem;\n    struct rte_mempool_objsz mp_obj_sz = {};\n\n    rte_mempool_calc_obj_size(buf_sz, 0, &mp_obj_sz);\n\n    xmem_size =\n        get_mempool_xmem_size(num_bufs,\n                              mp_obj_sz.elt_size + mp_obj_sz.header_size +\n                                                   mp_obj_sz.trailer_size,\n                              page_bits);\n\n    // Aligning to 2M causes the further failure in small allocations.\n    // TODO: Check why - and fix.\n    if (posix_memalign((void**)&xmem, page_size, xmem_size)) {\n        printf(\"Can't allocate %ld bytes aligned to %ld\\n\",\n               xmem_size, page_size);\n        return nullptr;\n    }\n\n    return xmem;\n}\n\ntemplate <bool HugetlbfsMemBackend>\nbool dpdk_qp<HugetlbfsMemBackend>::init_rx_mbuf_pool()\n{\n    using namespace memory;\n    sstring name = sstring(pktmbuf_pool_name) + to_sstring(_qid) + \"_rx\";\n\n    printf(\"Creating Rx mbuf pool '%s' [%u mbufs] ...\\n\",\n           name.c_str(), mbufs_per_queue_rx);\n\n    //\n    // If we have a hugetlbfs memory backend we may perform a virt2phys\n    // translation and memory is \"pinned\". Therefore we may provide an external\n    // memory for DPDK pools and this way significantly reduce the memory needed\n    // for the DPDK in this case.\n    //\n    if (HugetlbfsMemBackend) {\n        size_t xmem_size;\n\n        _rx_xmem.reset(alloc_mempool_xmem(mbufs_per_queue_rx, mbuf_overhead,\n                                          xmem_size));\n        if (!_rx_xmem.get()) {\n            printf(\"Can't allocate a memory for Rx buffers\\n\");\n            return false;\n        }\n\n        //\n        // Don't pass single-producer/single-consumer flags to mbuf create as it\n        // seems faster to use a cache instead.\n        //\n        struct rte_pktmbuf_pool_private roomsz = {};\n        roomsz.mbuf_data_room_size = mbuf_data_size + RTE_PKTMBUF_HEADROOM;\n        _pktmbuf_pool_rx =\n            rte_mempool_create_empty(name.c_str(),\n                                     mbufs_per_queue_rx, mbuf_overhead,\n                                     mbuf_cache_size,\n                                     sizeof(struct rte_pktmbuf_pool_private),\n                                     rte_socket_id(), 0);\n        if (!_pktmbuf_pool_rx) {\n            printf(\"Failed to create mempool for Rx\\n\");\n            exit(1);\n        }\n\n        rte_pktmbuf_pool_init(_pktmbuf_pool_rx, as_cookie(roomsz));\n\n        if (rte_mempool_populate_virt(_pktmbuf_pool_rx,\n                                      (char*)(_rx_xmem.get()), xmem_size,\n                                      page_size,\n                                      nullptr, nullptr) < 0) {\n            printf(\"Failed to populate mempool for Rx\\n\");\n            exit(1);\n        }\n\n        rte_mempool_obj_iter(_pktmbuf_pool_rx, rte_pktmbuf_init, nullptr);\n\n        // reserve the memory for Rx buffers containers\n        _rx_free_pkts.reserve(mbufs_per_queue_rx);\n        _rx_free_bufs.reserve(mbufs_per_queue_rx);\n\n        //\n        // 1) Pull all entries from the pool.\n        // 2) Bind data buffers to each of them.\n        // 3) Return them back to the pool.\n        //\n        for (int i = 0; i < mbufs_per_queue_rx; i++) {\n            rte_mbuf* m = rte_pktmbuf_alloc(_pktmbuf_pool_rx);\n            SEASTAR_ASSERT(m);\n            _rx_free_bufs.push_back(m);\n        }\n\n        for (auto&& m : _rx_free_bufs) {\n            if (!init_noninline_rx_mbuf(m)) {\n                printf(\"Failed to allocate data buffers for Rx ring. \"\n                       \"Consider increasing the amount of memory.\\n\");\n                exit(1);\n            }\n        }\n\n        rte_mempool_put_bulk(_pktmbuf_pool_rx, (void**)_rx_free_bufs.data(),\n                             _rx_free_bufs.size());\n\n        _rx_free_bufs.clear();\n    } else {\n        struct rte_pktmbuf_pool_private roomsz = {};\n        roomsz.mbuf_data_room_size = inline_mbuf_data_size + RTE_PKTMBUF_HEADROOM;\n        _pktmbuf_pool_rx =\n            rte_mempool_create(name.c_str(),\n                               mbufs_per_queue_rx, inline_mbuf_size,\n                               mbuf_cache_size,\n                               sizeof(struct rte_pktmbuf_pool_private),\n                               rte_pktmbuf_pool_init, as_cookie(roomsz),\n                               rte_pktmbuf_init, nullptr,\n                               rte_socket_id(), 0);\n    }\n\n    return _pktmbuf_pool_rx != nullptr;\n}\n\n// Map DMA address explicitly.\n// XXX: does NOT work with Mellanox NICs as they use IB libs instead of VFIO.\ntemplate <bool HugetlbfsMemBackend>\nbool dpdk_qp<HugetlbfsMemBackend>::map_dma()\n{\n    auto m = memory::get_memory_layout();\n    rte_iova_t iova = rte_mem_virt2iova((const void*)m.start);\n\n    return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,\n                                      m.start, iova, m.end - m.start) == 0;\n}\n\nvoid dpdk_device::check_port_link_status()\n{\n    using namespace std::literals::chrono_literals;\n    int count = 0;\n    constexpr auto check_interval = 100ms;\n\n    std::cout << \"\\nChecking link status \" << std::endl;\n    auto t = new timer<>;\n    t->set_callback([this, count, t] () mutable {\n        const int max_check_time = 90;  /* 9s (90 * 100ms) in total */\n        struct rte_eth_link link;\n        memset(&link, 0, sizeof(link));\n        rte_eth_link_get_nowait(_port_idx, &link);\n\n        if (link.link_status) {\n            std::cout <<\n                \"done\\nPort \" << static_cast<unsigned>(_port_idx) <<\n                \" Link Up - speed \" << link.link_speed <<\n                \" Mbps - \" << ((link.link_duplex == RTE_ETH_LINK_FULL_DUPLEX) ?\n                          (\"full-duplex\") : (\"half-duplex\\n\")) <<\n                std::endl;\n            _link_ready_promise.set_value();\n\n            // We may start collecting statistics only after the Link is UP.\n            _stats_collector.arm_periodic(2s);\n        } else if (count++ < max_check_time) {\n             std::cout << \".\" << std::flush;\n             return;\n        } else {\n            std::cout << \"done\\nPort \" << _port_idx << \" Link Down\" << std::endl;\n        }\n        t->cancel();\n        delete t;\n    });\n    t->arm_periodic(check_interval);\n}\n\n// This function uses offsetof with non POD types.\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Winvalid-offsetof\"\n\ntemplate <bool HugetlbfsMemBackend>\ndpdk_qp<HugetlbfsMemBackend>::dpdk_qp(dpdk_device* dev, uint16_t qid,\n                                      const std::string stats_plugin_name)\n     : qp(true, stats_plugin_name, qid), _dev(dev), _qid(qid),\n       _rx_gc_poller(reactor::poller::simple([&] { return rx_gc(); })),\n       _tx_buf_factory(qid),\n       _tx_gc_poller(reactor::poller::simple([&] { return _tx_buf_factory.gc(); }))\n{\n    if (!init_rx_mbuf_pool()) {\n        rte_exit(EXIT_FAILURE, \"Cannot initialize mbuf pools\\n\");\n    }\n\n    if (HugetlbfsMemBackend && !map_dma()) {\n        rte_exit(EXIT_FAILURE, \"Cannot map DMA\\n\");\n    }\n\n    static_assert(offsetof(class tx_buf, private_end) -\n                  offsetof(class tx_buf, private_start) <= RTE_PKTMBUF_HEADROOM,\n                  \"RTE_PKTMBUF_HEADROOM is less than dpdk_qp::tx_buf size! \"\n                  \"Increase the headroom size in the DPDK configuration\");\n    static_assert(offsetof(class tx_buf, _mbuf) == 0,\n                  \"There is a pad at the beginning of the tx_buf before _mbuf \"\n                  \"field!\");\n    static_assert((inline_mbuf_data_size & (inline_mbuf_data_size - 1)) == 0,\n                  \"inline_mbuf_data_size has to be a power of two!\");\n\n    if (rte_eth_rx_queue_setup(_dev->port_idx(), _qid, default_ring_size,\n            rte_eth_dev_socket_id(_dev->port_idx()),\n            _dev->def_rx_conf(), _pktmbuf_pool_rx) < 0) {\n        rte_exit(EXIT_FAILURE, \"Cannot initialize rx queue\\n\");\n    }\n\n    if (rte_eth_tx_queue_setup(_dev->port_idx(), _qid, default_ring_size,\n            rte_eth_dev_socket_id(_dev->port_idx()), _dev->def_tx_conf()) < 0) {\n        rte_exit(EXIT_FAILURE, \"Cannot initialize tx queue\\n\");\n    }\n\n    // Register error statistics: Rx total and checksum errors\n    namespace sm = seastar::metrics;\n    _metrics.add_group(_stats_plugin_name, {\n        sm::make_counter(_queue_name + \"_rx_csum_errors\", _stats.rx.bad.csum,\n                        sm::description(\"Counts a number of packets received by this queue that have a bad CSUM value. \"\n                                        \"A non-zero value of this metric usually indicates a HW issue, e.g. a bad cable.\")),\n\n        sm::make_counter(_queue_name + \"_rx_errors\", _stats.rx.bad.total,\n                        sm::description(\"Counts a total number of errors in the ingress path for this queue: CSUM errors, etc.\")),\n\n        sm::make_counter(_queue_name + \"_rx_no_memory_errors\", _stats.rx.bad.no_mem,\n                        sm::description(\"Counts a number of ingress packets received by this HW queue but dropped by the SW due to low memory. \"\n                                        \"A non-zero value indicates that seastar doesn't have enough memory to handle the packet reception or the memory is too fragmented.\")),\n    });\n}\n\n#pragma GCC diagnostic pop\n\ntemplate <bool HugetlbfsMemBackend>\nvoid dpdk_qp<HugetlbfsMemBackend>::rx_start() {\n    _rx_poller = reactor::poller::simple([&] { return poll_rx_once(); });\n}\n\ntemplate<>\ninline std::optional<packet>\ndpdk_qp<false>::from_mbuf_lro(rte_mbuf* m)\n{\n    //\n    // Try to allocate a buffer for the whole packet's data.\n    // If we fail - construct the packet from mbufs.\n    // If we succeed - copy the data into this buffer, create a packet based on\n    // this buffer and return the mbuf to its pool.\n    //\n    auto pkt_len = rte_pktmbuf_pkt_len(m);\n    char* buf = (char*)malloc(pkt_len);\n    if (buf) {\n        // Copy the contents of the packet into the buffer we've just allocated\n        size_t offset = 0;\n        for (rte_mbuf* m1 = m; m1 != nullptr; m1 = m1->next) {\n            char* data = rte_pktmbuf_mtod(m1, char*);\n            auto len = rte_pktmbuf_data_len(m1);\n\n            rte_memcpy(buf + offset, data, len);\n            offset += len;\n        }\n\n        rte_pktmbuf_free(m);\n\n        return packet(fragment{buf, pkt_len}, make_free_deleter(buf));\n    }\n\n    // Drop if allocation failed\n    rte_pktmbuf_free(m);\n\n    return std::nullopt;\n}\n\ntemplate<>\ninline std::optional<packet>\ndpdk_qp<false>::from_mbuf(rte_mbuf* m)\n{\n    if (!_dev->hw_features_ref().rx_lro || rte_pktmbuf_is_contiguous(m)) {\n        //\n        // Try to allocate a buffer for packet's data. If we fail - give the\n        // application an mbuf itself. If we succeed - copy the data into this\n        // buffer, create a packet based on this buffer and return the mbuf to\n        // its pool.\n        //\n        auto len = rte_pktmbuf_data_len(m);\n        char* buf = (char*)malloc(len);\n\n        if (!buf) {\n            // Drop if allocation failed\n            rte_pktmbuf_free(m);\n\n            return std::nullopt;\n        } else {\n            rte_memcpy(buf, rte_pktmbuf_mtod(m, char*), len);\n            rte_pktmbuf_free(m);\n\n            return packet(fragment{buf, len}, make_free_deleter(buf));\n        }\n    } else {\n        return from_mbuf_lro(m);\n    }\n}\n\ntemplate<>\ninline std::optional<packet>\ndpdk_qp<true>::from_mbuf_lro(rte_mbuf* m)\n{\n    _frags.clear();\n    _bufs.clear();\n\n    for (; m != nullptr; m = m->next) {\n        char* data = rte_pktmbuf_mtod(m, char*);\n\n        _frags.emplace_back(fragment{data, rte_pktmbuf_data_len(m)});\n        _bufs.push_back(data);\n    }\n\n    return packet(_frags.begin(), _frags.end(),\n                  make_deleter(deleter(),\n                          [bufs_vec = std::move(_bufs)] {\n                              for (auto&& b : bufs_vec) {\n                                  free(b);\n                              }\n                          }));\n}\n\ntemplate<>\ninline std::optional<packet> dpdk_qp<true>::from_mbuf(rte_mbuf* m)\n{\n    _rx_free_pkts.push_back(m);\n    _num_rx_free_segs += m->nb_segs;\n\n    if (!_dev->hw_features_ref().rx_lro || rte_pktmbuf_is_contiguous(m)) {\n        char* data = rte_pktmbuf_mtod(m, char*);\n\n        return packet(fragment{data, rte_pktmbuf_data_len(m)},\n                      make_free_deleter(data));\n    } else {\n        return from_mbuf_lro(m);\n    }\n}\n\ntemplate <bool HugetlbfsMemBackend>\ninline bool dpdk_qp<HugetlbfsMemBackend>::refill_one_cluster(rte_mbuf* head)\n{\n    for (; head != nullptr; head = head->next) {\n        if (!refill_rx_mbuf(head)) {\n            //\n            // If we failed to allocate a new buffer - push the rest of the\n            // cluster back to the free_packets list for a later retry.\n            //\n            _rx_free_pkts.push_back(head);\n            return false;\n        }\n        _rx_free_bufs.push_back(head);\n    }\n\n    return true;\n}\n\ntemplate <bool HugetlbfsMemBackend>\nbool dpdk_qp<HugetlbfsMemBackend>::rx_gc()\n{\n    if (_num_rx_free_segs >= rx_gc_thresh) {\n        while (!_rx_free_pkts.empty()) {\n            //\n            // Use back() + pop_back() semantics to avoid an extra\n            // _rx_free_pkts.clear() at the end of the function - clear() has a\n            // linear complexity.\n            //\n            auto m = _rx_free_pkts.back();\n            _rx_free_pkts.pop_back();\n\n            if (!refill_one_cluster(m)) {\n                break;\n            }\n        }\n\n        if (_rx_free_bufs.size()) {\n            rte_mempool_put_bulk(_pktmbuf_pool_rx,\n                                 (void **)_rx_free_bufs.data(),\n                                 _rx_free_bufs.size());\n\n            // TODO: SEASTAR_ASSERT() in a fast path! Remove me ASAP!\n            SEASTAR_ASSERT(_num_rx_free_segs >= _rx_free_bufs.size());\n\n            _num_rx_free_segs -= _rx_free_bufs.size();\n            _rx_free_bufs.clear();\n\n            // TODO: SEASTAR_ASSERT() in a fast path! Remove me ASAP!\n            SEASTAR_ASSERT((_rx_free_pkts.empty() && !_num_rx_free_segs) ||\n                   (!_rx_free_pkts.empty() && _num_rx_free_segs));\n        }\n    }\n\n    return _num_rx_free_segs >= rx_gc_thresh;\n}\n\n\ntemplate <bool HugetlbfsMemBackend>\nvoid dpdk_qp<HugetlbfsMemBackend>::process_packets(\n    struct rte_mbuf **bufs, uint16_t count)\n{\n    uint64_t nr_frags = 0, bytes = 0;\n\n    for (uint16_t i = 0; i < count; i++) {\n        struct rte_mbuf *m = bufs[i];\n        offload_info oi;\n\n        std::optional<packet> p = from_mbuf(m);\n\n        // Drop the packet if translation above has failed\n        if (!p) {\n            _stats.rx.bad.inc_no_mem();\n            continue;\n        }\n\n        nr_frags += m->nb_segs;\n        bytes    += m->pkt_len;\n\n        // Set stripped VLAN value if available\n        if ((m->ol_flags & RTE_MBUF_F_RX_VLAN_STRIPPED) &&\n            (m->ol_flags & RTE_MBUF_F_RX_VLAN)) {\n\n            oi.vlan_tci = m->vlan_tci;\n        }\n\n        if (_dev->hw_features().rx_csum_offload) {\n            if (m->ol_flags & (RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD)) {\n                // Packet with bad checksum, just drop it.\n                _stats.rx.bad.inc_csum_err();\n                continue;\n            }\n            // Note that when _hw_features.rx_csum_offload is on, the receive\n            // code for ip, tcp and udp will assume they don't need to check\n            // the checksum again, because we did this here.\n        }\n\n        (*p).set_offload_info(oi);\n        if (m->ol_flags & RTE_MBUF_F_RX_RSS_HASH) {\n            (*p).set_rss_hash(m->hash.rss);\n        }\n\n        _dev->l2receive(std::move(*p));\n    }\n\n    _stats.rx.good.update_pkts_bunch(count);\n    _stats.rx.good.update_frags_stats(nr_frags, bytes);\n\n    if (!HugetlbfsMemBackend) {\n        _stats.rx.good.copy_frags = _stats.rx.good.nr_frags;\n        _stats.rx.good.copy_bytes = _stats.rx.good.bytes;\n    }\n}\n\ntemplate <bool HugetlbfsMemBackend>\nbool dpdk_qp<HugetlbfsMemBackend>::poll_rx_once()\n{\n    struct rte_mbuf *buf[packet_read_size];\n\n    /* read a port */\n    uint16_t rx_count = rte_eth_rx_burst(_dev->port_idx(), _qid,\n                                         buf, packet_read_size);\n\n    /* Now process the NIC packets read */\n    if (likely(rx_count > 0)) {\n        process_packets(buf, rx_count);\n    }\n\n    return rx_count;\n}\n\nvoid dpdk_device::set_rss_table()\n{\n    if (_dev_info.reta_size == 0)\n        return;\n\n    int reta_conf_size =\n        std::max(1, _dev_info.reta_size / RTE_ETH_RETA_GROUP_SIZE);\n    std::vector<rte_eth_rss_reta_entry64> reta_conf(reta_conf_size);\n\n    // Configure the HW indirection table\n    unsigned i = 0;\n    for (auto& x : reta_conf) {\n        x.mask = ~0ULL;\n        for (auto& r: x.reta) {\n            r = i++ % _num_queues;\n        }\n    }\n\n    if (rte_eth_dev_rss_reta_update(_port_idx, reta_conf.data(), _dev_info.reta_size)) {\n        rte_exit(EXIT_FAILURE, \"Port %d: Failed to update an RSS indirection table\", _port_idx);\n    }\n\n    // Fill our local indirection table. Make it in a separate loop to keep things simple.\n    i = 0;\n    for (auto& r : _redir_table) {\n        r = i++ % _num_queues;\n    }\n}\n\nstd::unique_ptr<qp> dpdk_device::init_local_queue(const program_options::option_group& opts, uint16_t qid) {\n    auto net_opts = dynamic_cast<const net::native_stack_options*>(&opts);\n    SEASTAR_ASSERT(net_opts);\n\n    std::unique_ptr<qp> qp;\n    if (net_opts->_hugepages) {\n        qp = std::make_unique<dpdk_qp<true>>(this, qid,\n                                 _stats_plugin_name + \"-\" + _stats_plugin_inst);\n    } else {\n        qp = std::make_unique<dpdk_qp<false>>(this, qid,\n                                 _stats_plugin_name + \"-\" + _stats_plugin_inst);\n    }\n\n    // FIXME: future is discarded\n    (void)smp::submit_to(_home_cpu, [this] () mutable {\n        if (++_queues_ready == _num_queues) {\n            init_port_fini();\n        }\n    });\n    return qp;\n}\n} // namespace dpdk\n\n/******************************** Interface functions *************************/\n\nstd::unique_ptr<net::device> create_dpdk_net_device(\n                                    uint16_t port_idx,\n                                    uint16_t num_queues,\n                                    bool use_lro,\n                                    bool enable_fc)\n{\n    static bool called = false;\n\n    SEASTAR_ASSERT(!called);\n    SEASTAR_ASSERT(dpdk::eal::initialized);\n\n    called = true;\n\n    // Check that we have at least one DPDK-able port\n    if (rte_eth_dev_count_avail() == 0) {\n        rte_exit(EXIT_FAILURE, \"No Ethernet ports - bye\\n\");\n    } else {\n        printf(\"ports number: %d\\n\", rte_eth_dev_count_avail());\n    }\n\n    return std::make_unique<dpdk::dpdk_device>(port_idx, num_queues, use_lro,\n                                               enable_fc);\n}\n\nstd::unique_ptr<net::device> create_dpdk_net_device(\n                                    const hw_config& hw_cfg)\n{\n    return create_dpdk_net_device(*hw_cfg.port_index, smp::count, hw_cfg.lro, hw_cfg.hw_fc);\n}\n\n}\n\n#else\n\n\n#include <seastar/net/dpdk.hh>\n\n#endif // SEASTAR_HAVE_DPDK\n\nnamespace seastar::net {\n\ndpdk_options::dpdk_options(program_options::option_group* parent_group)\n#ifdef SEASTAR_HAVE_DPDK\n    : program_options::option_group(parent_group, \"DPDK net options\")\n    , dpdk_port_index(*this, \"dpdk-port-index\",\n                0,\n                \"DPDK Port Index\")\n    , hw_fc(*this, \"hw-fc\",\n                \"on\",\n                \"Enable HW Flow Control (on / off)\")\n#else\n    : program_options::option_group(parent_group, \"DPDK net options\", program_options::unused{})\n    , dpdk_port_index(*this, \"dpdk-port-index\", program_options::unused{})\n    , hw_fc(*this, \"hw-fc\", program_options::unused{})\n#endif\n#if 0\n    opts.add_options()\n        (\"csum-offload\",\n                boost::program_options::value<std::string>()->default_value(\"on\"),\n                \"Enable checksum offload feature (on / off)\")\n        (\"tso\",\n                boost::program_options::value<std::string>()->default_value(\"on\"),\n                \"Enable TCP segment offload feature (on / off)\")\n        (\"ufo\",\n                boost::program_options::value<std::string>()->default_value(\"on\"),\n                \"Enable UDP fragmentation offload feature (on / off)\")\n        ;\n#endif\n{\n}\n\n}\n"
  },
  {
    "path": "src/net/ethernet.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/print.hh>\n#include <seastar/net/ethernet.hh>\n#include <fmt/ostream.h>\n#include <boost/algorithm/string.hpp>\n#include <string>\n\nnamespace seastar {\n\nnamespace net {\n\nstd::ostream& operator<<(std::ostream& os, ethernet_address ea) {\n    auto& m = ea.mac;\n    using u = uint32_t;\n    fmt::print(os, \"{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}\",\n            u(m[0]), u(m[1]), u(m[2]), u(m[3]), u(m[4]), u(m[5]));\n    return os;\n}\n\nethernet_address parse_ethernet_address(std::string addr)\n{\n    std::vector<std::string> v;\n    boost::split(v, addr , boost::algorithm::is_any_of(\":\"));\n\n    if (v.size() != 6) {\n        throw std::runtime_error(\"invalid mac address\\n\");\n    }\n\n    ethernet_address a;\n    unsigned i = 0;\n    for (auto &x: v) {\n        a.mac[i++] = std::stoi(x, nullptr,16);\n    }\n    return a;\n}\n}\n\n}\n\n"
  },
  {
    "path": "src/net/inet_address.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n\n\n#include <algorithm>\n#include <ostream>\n#include <arpa/inet.h>\n#include <boost/functional/hash.hpp>\n#include <fmt/ostream.h>\n\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/print.hh>\n\nstatic_assert(std::is_nothrow_default_constructible_v<seastar::net::ipv4_address>);\nstatic_assert(std::is_nothrow_copy_constructible_v<seastar::net::ipv4_address>);\nstatic_assert(std::is_nothrow_move_constructible_v<seastar::net::ipv4_address>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<seastar::net::ipv6_address>);\nstatic_assert(std::is_nothrow_copy_constructible_v<seastar::net::ipv6_address>);\nstatic_assert(std::is_nothrow_move_constructible_v<seastar::net::ipv6_address>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<seastar::net::inet_address>);\nstatic_assert(std::is_nothrow_copy_constructible_v<seastar::net::inet_address>);\nstatic_assert(std::is_nothrow_move_constructible_v<seastar::net::inet_address>);\n\nseastar::net::inet_address::inet_address() noexcept\n                : inet_address(::in6_addr{})\n{}\n\nseastar::net::inet_address::inet_address(family f) noexcept\n                : _in_family(f)\n{\n    memset(&_in6, 0, sizeof(_in6));\n}\n\nseastar::net::inet_address::inet_address(::in_addr i) noexcept\n                : _in_family(family::INET), _in(i) {\n}\n\nseastar::net::inet_address::inet_address(::in6_addr i, uint32_t scope) noexcept\n                : _in_family(family::INET6), _in6(i), _scope(scope) {\n}\n\nstd::optional<seastar::net::inet_address>\nseastar::net::inet_address::parse_numerical(const sstring& addr) {\n    inet_address in;\n    if (::inet_pton(AF_INET, addr.c_str(), &in._in)) {\n        in._in_family = family::INET;\n        return in;\n    }\n    auto i = addr.find_last_of('%');\n    if (i != sstring::npos) {\n        auto ext = addr.substr(i + 1);\n        auto src = addr.substr(0, i);\n        auto res = parse_numerical(src);\n\n        if (res) {\n            uint32_t index = std::numeric_limits<uint32_t>::max();\n            try {\n                index = std::stoul(ext);\n            } catch (...) {\n            }\n            for (auto& nwif : engine().net().network_interfaces()) {\n                if (nwif.index() == index || nwif.name() == ext || nwif.display_name() == ext) {\n                    res->_scope = nwif.index();\n                    break;\n                }\n            }\n            return *res;\n        }\n    }\n    if (::inet_pton(AF_INET6, addr.c_str(), &in._in6)) {\n        in._in_family = family::INET6;\n        return in;\n    }\n    return {};\n}\n\nseastar::net::inet_address::inet_address(const sstring& addr)\n                : inet_address([&addr] {\n    auto res = parse_numerical(addr);\n    if (res) {\n        return std::move(*res);\n    }\n    throw std::invalid_argument(addr);\n}())\n{}\n\nseastar::net::inet_address::inet_address(const ipv4_address& in) noexcept\n    : inet_address(::in_addr{hton(in.ip)})\n{}\n\nseastar::net::inet_address::inet_address(const ipv6_address& in, uint32_t scope) noexcept\n    : inet_address([&] {\n        ::in6_addr tmp;\n        std::copy(in.bytes().begin(), in.bytes().end(), tmp.s6_addr);\n        return tmp;\n    }(), scope)\n{}\n\nseastar::net::ipv4_address seastar::net::inet_address::as_ipv4_address() const {\n    in_addr in = *this;\n    return ipv4_address(ntoh(in.s_addr));\n}\n\nseastar::net::ipv6_address seastar::net::inet_address::as_ipv6_address() const noexcept {\n    in6_addr in6 = *this;\n    return ipv6_address{in6};\n}\n\nbool seastar::net::inet_address::operator==(const inet_address& o) const noexcept {\n    if (o._in_family != _in_family) {\n        return false;\n    }\n    switch (_in_family) {\n    case family::INET:\n        return _in.s_addr == o._in.s_addr;\n    case family::INET6:\n        return std::equal(std::begin(_in6.s6_addr), std::end(_in6.s6_addr), std::begin(o._in6.s6_addr));\n    default:\n        return false;\n    }\n}\n\nseastar::net::inet_address::operator ::in_addr() const {\n    if (_in_family != family::INET) {\n        if (IN6_IS_ADDR_V4MAPPED(&_in6)) {\n            ::in_addr in;\n            in.s_addr = _in6.s6_addr32[3];\n            return in;\n        }\n        throw std::invalid_argument(\"Not an IPv4 address\");\n    }\n    return _in;\n}\n\nseastar::net::inet_address::operator ::in6_addr() const noexcept {\n    if (_in_family == family::INET) {\n        in6_addr in6 = IN6ADDR_ANY_INIT;\n        in6.s6_addr32[2] = htonl(0xffff);\n        in6.s6_addr32[3] = _in.s_addr;\n        return in6;\n    }\n    return _in6;\n}\n\nseastar::net::inet_address::operator seastar::net::ipv6_address() const noexcept {\n    return as_ipv6_address();\n}\n\nsize_t seastar::net::inet_address::size() const noexcept {\n    switch (_in_family) {\n    case family::INET:\n        return sizeof(::in_addr);\n    case family::INET6:\n        return sizeof(::in6_addr);\n    default:\n        return 0;\n    }\n}\n\nconst void * seastar::net::inet_address::data() const noexcept {\n    return &_in;\n}\n\nbool seastar::net::inet_address::is_loopback() const noexcept {\n    switch (_in_family) {\n    case family::INET:\n        return (net::ntoh(_in.s_addr) & 0xff000000) == 0x7f000000;\n    case family::INET6:\n        return std::equal(std::begin(_in6.s6_addr), std::end(_in6.s6_addr), std::begin(::in6addr_loopback.s6_addr));\n    default:\n        return false;\n    }\n}\n\nbool seastar::net::inet_address::is_addr_any() const noexcept {\n    switch (_in_family) {\n    case family::INET:\n        return _in.s_addr == INADDR_ANY;\n    case family::INET6:\n        return std::equal(std::begin(_in6.s6_addr), std::end(_in6.s6_addr), std::begin(::in6addr_any.s6_addr));\n    default:\n        return false;\n    }\n}\n\nseastar::net::ipv6_address::ipv6_address(const ::in6_addr& in) noexcept {\n    std::copy(std::begin(in.s6_addr), std::end(in.s6_addr), ip.begin());\n}\n\nseastar::net::ipv6_address::ipv6_address(const ipv6_bytes& in) noexcept\n    : ip(in)\n{}\n\nseastar::net::ipv6_address::ipv6_address(const ipv6_addr& addr) noexcept\n    : ipv6_address(addr.ip)\n{}\n\nseastar::net::ipv6_address::ipv6_address() noexcept\n    : ipv6_address(::in6addr_any)\n{}\n\nseastar::net::ipv6_address::ipv6_address(const std::string& addr) {\n    if (!::inet_pton(AF_INET6, addr.c_str(), ip.data())) {\n        throw std::runtime_error(fmt::format(\"Wrong format for IPv6 address {}. Please ensure it's in colon-hex format\",\n                                        addr));\n    }\n}\n\nseastar::net::ipv6_address seastar::net::ipv6_address::read(const char* s) noexcept {\n    auto* b = reinterpret_cast<const uint8_t *>(s);\n    ipv6_address in;\n    std::copy(b, b + ipv6_address::size(), in.ip.begin());\n    return in;\n}\n\nseastar::net::ipv6_address seastar::net::ipv6_address::consume(const char*& p) noexcept {\n    auto res = read(p);\n    p += size();\n    return res;\n}\n\nvoid seastar::net::ipv6_address::write(char* p) const noexcept {\n    std::copy(ip.begin(), ip.end(), p);\n}\n\nvoid seastar::net::ipv6_address::produce(char*& p) const noexcept {\n    write(p);\n    p += size();\n}\n\nbool seastar::net::ipv6_address::is_unspecified() const noexcept {\n    return std::all_of(ip.begin(), ip.end(), [](uint8_t b) { return b == 0; });\n}\n\nstd::ostream& seastar::net::operator<<(std::ostream& os, const ipv4_address& a) {\n    auto ip = a.ip;\n    fmt::print(os, \"{:d}.{:d}.{:d}.{:d}\",\n            (ip >> 24) & 0xff,\n            (ip >> 16) & 0xff,\n            (ip >> 8) & 0xff,\n            (ip >> 0) & 0xff);\n    return os;\n}\n\nstd::ostream& seastar::net::operator<<(std::ostream& os, const ipv6_address& a) {\n    char buffer[64];\n    return os << ::inet_ntop(AF_INET6, a.ip.data(), buffer, sizeof(buffer));\n}\n\nseastar::ipv6_addr::ipv6_addr(const ipv6_bytes& b, uint16_t p) noexcept\n    : ip(b), port(p)\n{}\n\nseastar::ipv6_addr::ipv6_addr(uint16_t p) noexcept\n    : ipv6_addr(net::inet_address(), p)\n{}\n\nseastar::ipv6_addr::ipv6_addr(const ::in6_addr& in6, uint16_t p) noexcept\n    : ipv6_addr(net::ipv6_address(in6).bytes(), p)\n{}\n\nseastar::ipv6_addr::ipv6_addr(const std::string& s)\n    : ipv6_addr([&] {\n        auto lc = s.find_last_of(']');\n        auto cp = s.find_first_of(':', lc);\n        auto port = cp != std::string::npos ? std::stoul(s.substr(cp + 1)) : 0;\n        auto ss = lc != std::string::npos ? s.substr(1, lc - 1) : s;\n        return ipv6_addr(net::ipv6_address(ss).bytes(), uint16_t(port));\n    }())\n{}\n\nseastar::ipv6_addr::ipv6_addr(const std::string& s, uint16_t p)\n    : ipv6_addr(net::ipv6_address(s).bytes(), p)\n{}\n\nseastar::ipv6_addr::ipv6_addr(const net::inet_address& i, uint16_t p) noexcept\n    : ipv6_addr(i.as_ipv6_address().bytes(), p)\n{}\n\nseastar::ipv6_addr::ipv6_addr(const ::sockaddr_in6& s) noexcept\n    : ipv6_addr(s.sin6_addr, net::ntoh(s.sin6_port))\n{}\n\nseastar::ipv6_addr::ipv6_addr(const socket_address& s) noexcept\n    : ipv6_addr(s.as_posix_sockaddr_in6())\n{}\n\nbool seastar::ipv6_addr::is_ip_unspecified() const noexcept {\n    return std::all_of(ip.begin(), ip.end(), [](uint8_t b) { return b == 0; });\n}\n\n\nseastar::net::inet_address seastar::socket_address::addr() const noexcept {\n    switch (family()) {\n    case AF_INET:\n        return net::inet_address(as_posix_sockaddr_in().sin_addr);\n    case AF_INET6:\n        return net::inet_address(as_posix_sockaddr_in6().sin6_addr, as_posix_sockaddr_in6().sin6_scope_id);\n    default:\n        return net::inet_address();\n    }\n}\n\n::in_port_t seastar::socket_address::port() const noexcept {\n    return net::ntoh(u.in.sin_port);\n}\n\nbool seastar::socket_address::is_wildcard() const noexcept {\n    switch (family()) {\n    case AF_INET: {\n            ipv4_addr addr(*this);\n            return addr.is_ip_unspecified() && addr.is_port_unspecified();\n        }\n    default:\n    case AF_INET6: {\n            ipv6_addr addr(*this);\n            return addr.is_ip_unspecified() && addr.is_port_unspecified();\n        }\n    case AF_UNIX:\n        return length() <= sizeof(::sa_family_t);\n    }\n}\n\nstd::ostream& seastar::net::operator<<(std::ostream& os, const inet_address& addr) {\n    char buffer[64];\n    os << inet_ntop(int(addr.in_family()), addr.data(), buffer, sizeof(buffer));\n    if (addr.scope() != inet_address::invalid_scope) {\n        os << \"%\" << addr.scope();\n    }\n    return os;\n}\n\nstd::ostream& seastar::net::operator<<(std::ostream& os, const inet_address::family& f) {\n    switch (f) {\n    case inet_address::family::INET:\n        os << \"INET\";\n        break;\n    case inet_address::family::INET6:\n        os << \"INET6\";\n        break;\n    default:\n        break;\n    }\n    return os;\n}\n\nstd::ostream& seastar::operator<<(std::ostream& os, const ipv4_addr& a) {\n    return os << seastar::socket_address(a);\n}\n\nstd::ostream& seastar::operator<<(std::ostream& os, const ipv6_addr& a) {\n    return os << seastar::socket_address(a);\n}\n\nsize_t std::hash<seastar::net::inet_address>::operator()(const seastar::net::inet_address& a) const {\n    switch (a.in_family()) {\n    case seastar::net::inet_address::family::INET:\n        return std::hash<seastar::net::ipv4_address>()(a.as_ipv4_address());\n    case seastar::net::inet_address::family::INET6:\n        return std::hash<seastar::net::ipv6_address>()(a.as_ipv6_address());\n    default:\n        return 0;\n    }\n}\n\nsize_t std::hash<seastar::net::ipv6_address>::operator()(const seastar::net::ipv6_address& a) const {\n    return boost::hash_range(a.ip.begin(), a.ip.end());\n}\n\nsize_t std::hash<seastar::ipv4_addr>::operator()(const seastar::ipv4_addr& x) const {\n    size_t h = x.ip;\n    boost::hash_combine(h, x.port);\n    return h;\n}\n"
  },
  {
    "path": "src/net/ip.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n#include <seastar/net/ip.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/net/toeplitz.hh>\n#include <seastar/core/metrics.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nipv4_address::ipv4_address(const std::string& addr) {\n    boost::system::error_code ec;\n    auto ipv4 = boost::asio::ip::make_address_v4(addr, ec);\n    if (ec) {\n        throw std::runtime_error(\n            fmt::format(\"Wrong format for IPv4 address {}. Please ensure it's in dotted-decimal format\", addr));\n    }\n    ip = static_cast<uint32_t>(std::move(ipv4).to_uint());\n}\n\nipv4::ipv4(interface* netif)\n    : _netif(netif)\n    , _global_arp(netif)\n    , _arp(_global_arp)\n    , _host_address(0)\n    , _gw_address(0)\n    , _netmask(0)\n    , _l3(netif, eth_protocol_num::ipv4, [this] { return get_packet(); })\n    , _tcp(*this)\n    , _icmp(*this)\n    , _udp(*this)\n    , _l4({ { uint8_t(ip_protocol_num::tcp), &_tcp }, { uint8_t(ip_protocol_num::icmp), &_icmp }, { uint8_t(ip_protocol_num::udp), &_udp }})\n{\n    namespace sm = seastar::metrics;\n    // FIXME: ignored future\n    (void)_l3.receive(\n        [this](packet p, ethernet_address ea) {\n            return handle_received_packet(std::move(p), ea);\n        },\n        [this](forward_hash& out_hash_data, packet& p, size_t off) {\n            return forward(out_hash_data, p, off);\n        });\n\n    _metrics.add_group(\"ipv4\", {\n        //\n        // Linearized events: DERIVE:0:u\n        //\n        sm::make_counter(\"linearizations\", [] { return ipv4_packet_merger::linearizations(); },\n                        sm::description(\"Counts a number of times a buffer linearization was invoked during buffers merge process. \"\n                                        \"Divide it by a total IPv4 receive packet rate to get an average number of lineraizations per packet.\"))\n    });\n    _frag_timer.set_callback([this] { frag_timeout(); });\n}\n\nbool ipv4::forward(forward_hash& out_hash_data, packet& p, size_t off)\n{\n    auto iph = p.get_header<ip_hdr>(off);\n\n    out_hash_data.push_back(iph->src_ip.ip);\n    out_hash_data.push_back(iph->dst_ip.ip);\n\n    auto h = ntoh(*iph);\n    auto l4 = _l4[h.ip_proto];\n    if (l4) {\n        if (h.mf() == false && h.offset() == 0) {\n            // This IP datagram is atomic, forward according to tcp or udp connection hash\n            l4->forward(out_hash_data, p, off + sizeof(ip_hdr));\n        }\n        // else forward according to ip fields only\n    }\n    return true;\n}\n\nbool ipv4::in_my_netmask(ipv4_address a) const {\n    return !((a.ip ^ _host_address.ip) & _netmask.ip);\n}\n\nbool ipv4::needs_frag(packet& p, ip_protocol_num prot_num, net::hw_features hw_features) {\n    if (p.len() + ipv4_hdr_len_min <= hw_features.mtu) {\n        return false;\n    }\n\n    if ((prot_num == ip_protocol_num::tcp && hw_features.tx_tso) ||\n        (prot_num == ip_protocol_num::udp && hw_features.tx_ufo)) {\n        return false;\n    }\n\n    return true;\n}\n\nfuture<>\nipv4::handle_received_packet(packet p, ethernet_address from) {\n    auto iph = p.get_header<ip_hdr>(0);\n    if (!iph) {\n        return make_ready_future<>();\n    }\n\n    // Skip checking csum of reassembled IP datagram\n    if (!hw_features().rx_csum_offload && !p.offload_info_ref().reassembled) {\n        checksummer csum;\n        csum.sum(reinterpret_cast<char*>(iph), sizeof(*iph));\n        if (csum.get() != 0) {\n            return make_ready_future<>();\n        }\n    }\n\n    auto h = ntoh(*iph);\n    unsigned ip_len = h.len;\n    unsigned ip_hdr_len = h.ihl * 4;\n    unsigned pkt_len = p.len();\n    auto offset = h.offset();\n    if (pkt_len > ip_len) {\n        // Trim extra data in the packet beyond IP total length\n        p.trim_back(pkt_len - ip_len);\n    } else if (pkt_len < ip_len) {\n        // Drop if it contains less than IP total length\n        return make_ready_future<>();\n    }\n    // Drop if the reassembled datagram will be larger than maximum IP size\n    if (offset + p.len() > net::ip_packet_len_max) {\n        return make_ready_future<>();\n    }\n\n    // FIXME: process options\n    if (in_my_netmask(h.src_ip) && h.src_ip != _host_address) {\n        _arp.learn(from, h.src_ip);\n    }\n\n    if (_packet_filter) {\n        bool handled = false;\n        auto r = _packet_filter->handle(p, &h, from, handled);\n        if (handled) {\n            return r;\n        }\n    }\n\n    if (h.dst_ip != _host_address) {\n        // FIXME: forward\n        return make_ready_future<>();\n    }\n\n    // Does this IP datagram need reassembly\n    auto mf = h.mf();\n    if (mf == true || offset != 0) {\n        frag_limit_mem();\n        auto frag_id = ipv4_frag_id{h.src_ip, h.dst_ip, h.id, h.ip_proto};\n        auto& frag = _frags[frag_id];\n        if (mf == false) {\n            frag.last_frag_received = true;\n        }\n        // This is a newly created frag_id\n        if (frag.mem_size == 0) {\n            _frags_age.push_back(frag_id);\n            frag.rx_time = clock_type::now();\n        }\n        auto added_size = frag.merge(h, offset, std::move(p));\n        _frag_mem += added_size;\n        if (frag.is_complete()) {\n            // All the fragments are received\n            auto dropped_size = frag.mem_size;\n            auto& ip_data = frag.data.map.begin()->second;\n            // Choose a cpu to forward this packet\n            auto cpu_id = this_shard_id();\n            auto l4 = _l4[h.ip_proto];\n            if (l4) {\n                if (smp::count == 1) {\n                    l4->received(std::move(ip_data), h.src_ip, h.dst_ip);\n                } else {\n                    size_t l4_offset = 0;\n                    forward_hash hash_data;\n                    hash_data.push_back(hton(h.src_ip.ip));\n                    hash_data.push_back(hton(h.dst_ip.ip));\n                    auto forwarded = l4->forward(hash_data, ip_data, l4_offset);\n                    if (forwarded) {\n                        cpu_id = _netif->hash2cpu(toeplitz_hash(_netif->rss_key(), hash_data));\n                        // No need to forward if the dst cpu is the current cpu\n                        if (cpu_id == this_shard_id()) {\n                            l4->received(std::move(ip_data), h.src_ip, h.dst_ip);\n                        } else {\n                            auto to = _netif->hw_address();\n                            auto pkt = frag.get_assembled_packet(from, to);\n                            _netif->forward(cpu_id, std::move(pkt));\n                        }\n                    }\n                }\n            }\n\n            // Delete this frag from _frags and _frags_age\n            frag_drop(frag_id, dropped_size);\n            _frags_age.remove(frag_id);\n        } else {\n            // Some of the fragments are missing\n            if (!_frag_timer.armed()) {\n                frag_arm();\n            }\n        }\n        return make_ready_future<>();\n    }\n\n    auto l4 = _l4[h.ip_proto];\n    if (l4) {\n        // Trim IP header and pass to upper layer\n        p.trim_front(ip_hdr_len);\n        l4->received(std::move(p), h.src_ip, h.dst_ip);\n    }\n    return make_ready_future<>();\n}\n\nfuture<ethernet_address> ipv4::get_l2_dst_address(ipv4_address to) {\n    // Figure out where to send the packet to. If it is a directly connected\n    // host, send to it directly, otherwise send to the default gateway.\n    ipv4_address dst;\n    if (in_my_netmask(to)) {\n        dst = to;\n    } else {\n        dst = _gw_address;\n    }\n\n    return _arp.lookup(dst);\n}\n\nvoid ipv4::send(ipv4_address to, ip_protocol_num proto_num, packet p, ethernet_address e_dst) {\n    auto needs_frag = this->needs_frag(p, proto_num, hw_features());\n\n    auto send_pkt = [this, to, proto_num, needs_frag, e_dst] (packet& pkt, uint16_t remaining, uint16_t offset) mutable  {\n        auto iph = pkt.prepend_header<ip_hdr>();\n        iph->ihl = sizeof(*iph) / 4;\n        iph->ver = 4;\n        iph->dscp = 0;\n        iph->ecn = 0;\n        iph->len = pkt.len();\n        // FIXME: a proper id\n        iph->id = 0;\n        if (needs_frag) {\n            uint16_t mf = remaining > 0;\n            // The fragment offset is measured in units of 8 octets (64 bits)\n            auto off = offset / 8;\n            iph->frag = (mf << uint8_t(ip_hdr::frag_bits::mf)) | off;\n        } else {\n            iph->frag = 0;\n        }\n        iph->ttl = 64;\n        iph->ip_proto = (uint8_t)proto_num;\n        iph->csum = 0;\n        iph->src_ip = _host_address;\n        iph->dst_ip = to;\n        *iph = hton(*iph);\n\n        if (hw_features().tx_csum_ip_offload) {\n            iph->csum = 0;\n            pkt.offload_info_ref().needs_ip_csum = true;\n        } else {\n            checksummer csum;\n            csum.sum(reinterpret_cast<char*>(iph), sizeof(*iph));\n            iph->csum = csum.get();\n        }\n\n        _packetq.push_back(l3_protocol::l3packet{eth_protocol_num::ipv4, e_dst, std::move(pkt)});\n    };\n\n    if (needs_frag) {\n        uint16_t offset = 0;\n        uint16_t remaining = p.len();\n        auto mtu = hw_features().mtu;\n\n        while (remaining) {\n            auto can_send = std::min(uint16_t(mtu - net::ipv4_hdr_len_min), remaining);\n            remaining -= can_send;\n            auto pkt = p.share(offset, can_send);\n            send_pkt(pkt, remaining, offset);\n            offset += can_send;\n        }\n    } else {\n        // The whole packet can be send in one shot\n        send_pkt(p, 0, 0);\n    }\n}\n\nstd::optional<l3_protocol::l3packet> ipv4::get_packet() {\n    // _packetq will be mostly empty here unless it hold remnants of previously\n    // fragmented packet\n    if (_packetq.empty()) {\n        for (size_t i = 0; i < _pkt_providers.size(); i++) {\n            auto l4p = _pkt_providers[_pkt_provider_idx++]();\n            if (_pkt_provider_idx == _pkt_providers.size()) {\n                _pkt_provider_idx = 0;\n            }\n            if (l4p) {\n                auto l4pv = std::move(l4p.value());\n                send(l4pv.to, l4pv.proto_num, std::move(l4pv.p), l4pv.e_dst);\n                break;\n            }\n        }\n    }\n\n    std::optional<l3_protocol::l3packet> p;\n    if (!_packetq.empty()) {\n        p = std::move(_packetq.front());\n        _packetq.pop_front();\n    }\n    return p;\n}\n\nvoid ipv4::set_host_address(ipv4_address ip) {\n    _host_address = ip;\n    _arp.set_self_addr(ip);\n}\n\nipv4_address ipv4::host_address() const {\n    return _host_address;\n}\n\nvoid ipv4::set_gw_address(ipv4_address ip) {\n    _gw_address = ip;\n}\n\nipv4_address ipv4::gw_address() const {\n    return _gw_address;\n}\n\nvoid ipv4::set_netmask_address(ipv4_address ip) {\n    _netmask = ip;\n}\n\nipv4_address ipv4::netmask_address() const {\n    return _netmask;\n}\n\nvoid ipv4::set_packet_filter(ip_packet_filter * f) {\n    _packet_filter = f;\n}\n\nip_packet_filter * ipv4::packet_filter() const {\n    return _packet_filter;\n}\n\nvoid ipv4::frag_limit_mem() {\n    if (_frag_mem <= _frag_high_thresh) {\n        return;\n    }\n    auto drop = _frag_mem - _frag_low_thresh;\n    while (drop) {\n        if (_frags_age.empty()) {\n            return;\n        }\n        // Drop the oldest frag (first element) from _frags_age\n        auto frag_id = _frags_age.front();\n        _frags_age.pop_front();\n\n        // Drop from _frags as well\n        auto& frag = _frags[frag_id];\n        auto dropped_size = frag.mem_size;\n        frag_drop(frag_id, dropped_size);\n\n        drop -= std::min(drop, dropped_size);\n    }\n}\n\nvoid ipv4::frag_timeout() {\n    if (_frags.empty()) {\n        return;\n    }\n    auto now = clock_type::now();\n    for (auto it = _frags_age.begin(); it != _frags_age.end();) {\n        auto frag_id = *it;\n        auto& frag = _frags[frag_id];\n        if (now > frag.rx_time + _frag_timeout) {\n            auto dropped_size = frag.mem_size;\n            // Drop from _frags\n            frag_drop(frag_id, dropped_size);\n            // Drop from _frags_age\n            it = _frags_age.erase(it);\n        } else {\n            // The further items can only be younger\n            break;\n        }\n    }\n    if (_frags.size() != 0) {\n        frag_arm(now);\n    } else {\n        _frag_mem = 0;\n    }\n}\n\nvoid ipv4::frag_drop(ipv4_frag_id frag_id, uint32_t dropped_size) {\n    _frags.erase(frag_id);\n    _frag_mem -= dropped_size;\n}\n\nint32_t ipv4::frag::merge(ip_hdr &h, uint16_t offset, packet p) {\n    uint32_t old = mem_size;\n    unsigned ip_hdr_len = h.ihl * 4;\n    // Store IP header\n    if (offset == 0) {\n        header = p.share(0, ip_hdr_len);\n    }\n    // Sotre IP payload\n    p.trim_front(ip_hdr_len);\n    data.merge(offset, std::move(p));\n    // Update mem size\n    mem_size = header.memory();\n    for (const auto& x : data.map) {\n        mem_size += x.second.memory();\n    }\n    auto added_size = mem_size - old;\n    return added_size;\n}\n\nbool ipv4::frag::is_complete() {\n    // If all the fragments are received, ipv4::frag::merge() should merge all\n    // the fragments into a single packet\n    auto offset = data.map.begin()->first;\n    auto nr_packet = data.map.size();\n    return last_frag_received && nr_packet == 1 && offset == 0;\n}\n\npacket ipv4::frag::get_assembled_packet(ethernet_address from, ethernet_address to) {\n    auto& ip_header = header;\n    auto& ip_data = data.map.begin()->second;\n    // Append a ethernet header, needed for forwarding\n    auto eh = ip_header.prepend_header<eth_hdr>();\n    eh->src_mac = from;\n    eh->dst_mac = to;\n    eh->eth_proto = uint16_t(eth_protocol_num::ipv4);\n    *eh = hton(*eh);\n    // Prepare a packet contains both ethernet header, ip header and ip data\n    ip_header.append(std::move(ip_data));\n    auto pkt = std::move(ip_header);\n    auto iph = pkt.get_header<ip_hdr>(sizeof(eth_hdr));\n    // len is the sum of each fragment\n    iph->len = hton(uint16_t(pkt.len() - sizeof(eth_hdr)));\n    // No fragmentation for the assembled datagram\n    iph->frag = 0;\n    // Since each fragment's csum is checked, no need to csum\n    // again for the assembled datagram\n    offload_info oi;\n    oi.reassembled = true;\n    pkt.set_offload_info(oi);\n    return pkt;\n}\n\nvoid icmp::received(packet p, ipaddr from, ipaddr to) {\n    auto hdr = p.get_header<icmp_hdr>(0);\n    if (!hdr || hdr->type != icmp_hdr::msg_type::echo_request) {\n        return;\n    }\n    hdr->type = icmp_hdr::msg_type::echo_reply;\n    hdr->code = 0;\n    hdr->csum = 0;\n    checksummer csum;\n    csum.sum(reinterpret_cast<char*>(hdr), p.len());\n    hdr->csum = csum.get();\n\n    if (_queue_space.try_wait(p.len())) { // drop packets that do not fit the queue\n        // FIXME: future is discarded\n        (void)_inet.get_l2_dst_address(from).then([this, from, p = std::move(p)] (ethernet_address e_dst) mutable {\n            _packetq.emplace_back(ipv4_traits::l4packet{from, std::move(p), e_dst, ip_protocol_num::icmp});\n        });\n    }\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/ip_checksum.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <arpa/inet.h>\n\n#include <seastar/net/ip_checksum.hh>\n#include <seastar/net/net.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nvoid checksummer::sum(const char* data, size_t len) {\n    auto orig_len = len;\n    if (odd) {\n        csum += uint8_t(*data++);\n        --len;\n    }\n    auto p64 = reinterpret_cast<const packed<uint64_t>*>(data);\n    while (len >= 8) {\n        csum += ntohq(*p64++);\n        len -= 8;\n    }\n    auto p16 = reinterpret_cast<const packed<uint16_t>*>(p64);\n    while (len >= 2) {\n        csum += ntohs(*p16++);\n        len -= 2;\n    }\n    auto p8 = reinterpret_cast<const uint8_t*>(p16);\n    if (len) {\n        csum += *p8++ << 8;\n        len -= 1;\n    }\n    odd ^= orig_len & 1;\n}\n\nuint16_t checksummer::get() const {\n    __int128 csum1 = (csum & 0xffff'ffff'ffff'ffff) + (csum >> 64);\n    uint64_t csum = (csum1 & 0xffff'ffff'ffff'ffff) + (csum1 >> 64);\n    csum = (csum & 0xffff) + ((csum >> 16) & 0xffff) + ((csum >> 32) & 0xffff) + (csum >> 48);\n    csum = (csum & 0xffff) + (csum >> 16);\n    csum = (csum & 0xffff) + (csum >> 16);\n    return htons(~csum);\n}\n\nvoid checksummer::sum(const packet& p) {\n    for (auto&& f : p.fragments()) {\n        sum(f.base, f.size);\n    }\n}\n\nuint16_t ip_checksum(const void* data, size_t len) {\n    checksummer cksum;\n    cksum.sum(reinterpret_cast<const char*>(data), len);\n    return cksum.get();\n}\n\n\n}\n\n}\n"
  },
  {
    "path": "src/net/native-stack-impl.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#pragma once\n\n#include <seastar/net/stack.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/log.hh>\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\nnamespace internal {\n\nnamespace native_stack_net_stats {\n\ninline thread_local std::array<uint64_t, max_scheduling_groups()> bytes_sent = {};\ninline thread_local std::array<uint64_t, max_scheduling_groups()> bytes_received = {};\n\n};\n\n}\n\nnamespace net {\n\nusing namespace seastar;\n\ntemplate <typename Protocol>\nclass native_server_socket_impl;\n\ntemplate <typename Protocol>\nclass native_connected_socket_impl;\n\nclass native_network_stack;\n\n// native_server_socket_impl\ntemplate <typename Protocol>\nclass native_server_socket_impl : public server_socket_impl {\n    typename Protocol::listener _listener;\npublic:\n    native_server_socket_impl(Protocol& proto, uint16_t port, listen_options opt);\n    virtual future<accept_result> accept() override;\n    virtual void abort_accept() override;\n    virtual socket_address local_address() const override;\n};\n\ntemplate <typename Protocol>\nnative_server_socket_impl<Protocol>::native_server_socket_impl(Protocol& proto, uint16_t port, listen_options opt)\n    : _listener(proto.listen(port)) {\n}\n\ntemplate <typename Protocol>\nfuture<accept_result>\nnative_server_socket_impl<Protocol>::accept() {\n    return _listener.accept().then([] (typename Protocol::connection conn) {\n        // Save \"conn\" contents before call below function\n        // \"conn\" is moved in 1st argument, and used in 2nd argument\n        // It causes trouble on Arm which passes arguments from left to right\n        auto ip = conn.foreign_ip().ip;\n        auto port = conn.foreign_port();\n        return make_ready_future<accept_result>(accept_result{\n                connected_socket(std::make_unique<native_connected_socket_impl<Protocol>>(make_lw_shared(std::move(conn)))),\n                make_ipv4_address(ip, port)});\n    });\n}\n\ntemplate <typename Protocol>\nvoid\nnative_server_socket_impl<Protocol>::abort_accept() {\n    _listener.abort_accept();\n}\n\ntemplate <typename Protocol>\nsocket_address native_server_socket_impl<Protocol>::local_address() const {\n    return socket_address(_listener.get_tcp().inet().inet().host_address(), _listener.port());\n}\n\n// native_connected_socket_impl\ntemplate <typename Protocol>\nclass native_connected_socket_impl : public connected_socket_impl {\n    lw_shared_ptr<typename Protocol::connection> _conn;\n    class native_data_source_impl;\n    class native_data_sink_impl;\npublic:\n    explicit native_connected_socket_impl(lw_shared_ptr<typename Protocol::connection> conn)\n        : _conn(std::move(conn)) {}\n    using connected_socket_impl::source;\n    virtual data_source source() override;\n    virtual data_sink sink() override;\n    virtual void shutdown_input() override;\n    virtual void shutdown_output() override;\n    virtual void set_nodelay(bool nodelay) override;\n    virtual bool get_nodelay() const override;\n    void set_keepalive(bool keepalive) override;\n    bool get_keepalive() const override;\n    void set_keepalive_parameters(const keepalive_params&) override;\n    keepalive_params get_keepalive_parameters() const override;\n    int get_sockopt(int level, int optname, void* data, size_t len) const override;\n    void set_sockopt(int level, int optname, const void* data, size_t len) override;\n    socket_address local_address() const noexcept override;\n    socket_address remote_address() const noexcept override;\n    virtual future<> wait_input_shutdown() override;\n};\n\ntemplate <typename Protocol>\nclass native_socket_impl final : public socket_impl {\n    Protocol& _proto;\n    lw_shared_ptr<typename Protocol::connection> _conn;\npublic:\n    explicit native_socket_impl(Protocol& proto)\n        : _proto(proto), _conn(nullptr) { }\n\n    virtual future<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) override {\n        //TODO: implement SCTP\n        SEASTAR_ASSERT(proto == transport::TCP);\n\n        // FIXME: local is ignored since native stack does not support multiple IPs yet\n        SEASTAR_ASSERT(sa.as_posix_sockaddr().sa_family == AF_INET);\n\n        _conn = make_lw_shared<typename Protocol::connection>(_proto.connect(sa));\n        return _conn->connected().then([conn = _conn]() mutable {\n            auto csi = std::make_unique<native_connected_socket_impl<Protocol>>(std::move(conn));\n            return make_ready_future<connected_socket>(connected_socket(std::move(csi)));\n        });\n    }\n\n    virtual void set_reuseaddr(bool reuseaddr) override {\n        // FIXME: implement\n        seastar_logger.error(\"Reuseaddr is not supported by native stack\");\n    }\n\n    virtual bool get_reuseaddr() const override {\n        // FIXME: implement\n        return false;\n    }\n\n    virtual void shutdown() override {\n        if (_conn) {\n            _conn->shutdown_connect();\n        }\n    }\n};\n\ntemplate <typename Protocol>\nclass native_connected_socket_impl<Protocol>::native_data_source_impl final\n    : public data_source_impl {\n    typedef typename Protocol::connection connection_type;\n    lw_shared_ptr<connection_type> _conn;\n    size_t _cur_frag = 0;\n    bool _eof = false;\n    packet _buf;\npublic:\n    explicit native_data_source_impl(lw_shared_ptr<connection_type> conn)\n        : _conn(std::move(conn)) {}\n    virtual future<temporary_buffer<char>> get() override {\n        if (_eof) {\n            return make_ready_future<temporary_buffer<char>>(temporary_buffer<char>(0));\n        }\n        if (_cur_frag != _buf.nr_frags()) {\n            auto& f = _buf.fragments()[_cur_frag++];\n            return make_ready_future<temporary_buffer<char>>(\n                    temporary_buffer<char>(f.base, f.size,\n                            make_deleter(deleter(), [p = _buf.share()] () mutable {})));\n        }\n        return _conn->wait_for_data().then([this] {\n            _buf = _conn->read();\n            auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n            internal::native_stack_net_stats::bytes_received[sg_id] += _buf.len();\n            _cur_frag = 0;\n            _eof = !_buf.len();\n            return get();\n        });\n    }\n    future<> close() override {\n        _conn->close_write();\n        return make_ready_future<>();\n    }\n};\n\ntemplate <typename Protocol>\nclass native_connected_socket_impl<Protocol>::native_data_sink_impl final\n    : public data_sink_impl {\n    typedef typename Protocol::connection connection_type;\n    lw_shared_ptr<connection_type> _conn;\npublic:\n    explicit native_data_sink_impl(lw_shared_ptr<connection_type> conn)\n        : _conn(std::move(conn)) {}\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n        auto sizes = bufs | std::views::transform(&temporary_buffer<char>::size);\n        auto len = std::accumulate(sizes.begin(), sizes.end(), size_t(0));\n        internal::native_stack_net_stats::bytes_sent[sg_id] += len;\n        return _conn->send(bufs);\n    }\n#else\n    using data_sink_impl::put;\n    virtual future<> put(packet p) override {\n        auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n        internal::native_stack_net_stats::bytes_sent[sg_id] += p.len();\n        auto v = std::move(p).release();\n        return _conn->send(v);\n    }\n#endif\n    virtual future<> close() override {\n        _conn->close_write();\n        return make_ready_future<>();\n    }\n    virtual bool can_batch_flushes() const noexcept override { return true; }\n    virtual void on_batch_flush_error() noexcept override {\n        _conn->close_read();\n    }\n};\n\ntemplate <typename Protocol>\ndata_source native_connected_socket_impl<Protocol>::source() {\n    return data_source(std::make_unique<native_data_source_impl>(_conn));\n}\n\ntemplate <typename Protocol>\ndata_sink native_connected_socket_impl<Protocol>::sink() {\n    return data_sink(std::make_unique<native_data_sink_impl>(_conn));\n}\n\ntemplate <typename Protocol>\nvoid\nnative_connected_socket_impl<Protocol>::shutdown_input() {\n    _conn->close_read();\n}\n\ntemplate <typename Protocol>\nvoid\nnative_connected_socket_impl<Protocol>::shutdown_output() {\n    _conn->close_write();\n}\n\ntemplate <typename Protocol>\nvoid\nnative_connected_socket_impl<Protocol>::set_nodelay(bool nodelay) {\n    // FIXME: implement\n}\n\ntemplate <typename Protocol>\nbool\nnative_connected_socket_impl<Protocol>::get_nodelay() const {\n    // FIXME: implement\n    return true;\n}\n\ntemplate <typename Protocol>\nvoid native_connected_socket_impl<Protocol>::set_keepalive(bool keepalive) {\n    // FIXME: implement\n    seastar_logger.error(\"Keepalive is not supported by native stack\");\n}\ntemplate <typename Protocol>\nbool native_connected_socket_impl<Protocol>::get_keepalive() const {\n    // FIXME: implement\n    return false;\n}\n\ntemplate <typename Protocol>\nvoid native_connected_socket_impl<Protocol>::set_keepalive_parameters(const keepalive_params&) {\n    // FIXME: implement\n    seastar_logger.error(\"Keepalive parameters are not supported by native stack\");\n}\n\ntemplate <typename Protocol>\nkeepalive_params native_connected_socket_impl<Protocol>::get_keepalive_parameters() const {\n    // FIXME: implement\n    return tcp_keepalive_params {std::chrono::seconds(0), std::chrono::seconds(0), 0};\n}\n\ntemplate<typename Protocol>\nvoid native_connected_socket_impl<Protocol>::set_sockopt(int level, int optname, const void* data, size_t len) {\n    throw std::runtime_error(\"Setting custom socket options is not supported for native stack\");\n}\n\ntemplate<typename Protocol>\nint native_connected_socket_impl<Protocol>::get_sockopt(int level, int optname, void* data, size_t len) const {\n    throw std::runtime_error(\"Getting custom socket options is not supported for native stack\");\n}\n\ntemplate<typename Protocol>\nsocket_address native_connected_socket_impl<Protocol>::local_address() const noexcept {\n    return {_conn->local_ip(), _conn->local_port()};\n}\n\ntemplate<typename Protocol>\nsocket_address native_connected_socket_impl<Protocol>::remote_address() const noexcept {\n    return {_conn->foreign_ip(), _conn->foreign_port()};\n}\n\ntemplate <typename Protocol>\nfuture<> native_connected_socket_impl<Protocol>::wait_input_shutdown() {\n    return _conn->wait_input_shutdown();\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/native-stack.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <chrono>\n#include <fstream>\n#include <functional>\n#include <map>\n#include <memory>\n#include <optional>\n#include <queue>\n\n#include <seastar/util/assert.hh>\n\n#include <sys/types.h>\n#include <sys/stat.h>\n#include <arpa/inet.h>\n#include <unistd.h>\n\n#include <seastar/net/native-stack.hh>\n#include \"net/native-stack-impl.hh\"\n#include <seastar/net/net.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/tcp-stack.hh>\n#include <seastar/net/tcp.hh>\n#include <seastar/net/udp.hh>\n#include <seastar/net/virtio.hh>\n#include <seastar/net/dpdk.hh>\n#include <seastar/net/proxy.hh>\n#include <seastar/net/dhcp.hh>\n#include <seastar/net/config.hh>\n#include <seastar/core/reactor.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nusing namespace seastar;\n\nvoid create_native_net_device(const native_stack_options& opts) {\n\n    bool deprecated_config_used = true;\n\n    std::stringstream net_config;\n\n    if ( opts.net_config) {\n        deprecated_config_used = false;\n        net_config << opts.net_config.get_value();\n    }\n    if ( opts.net_config_file) {\n        deprecated_config_used = false;\n        std::fstream fs(opts.net_config_file.get_value());\n        net_config << fs.rdbuf();\n    }\n\n    std::unique_ptr<device> dev;\n\n    if ( deprecated_config_used) {\n#ifdef SEASTAR_HAVE_DPDK\n        if ( opts.dpdk_pmd) {\n             dev = create_dpdk_net_device(opts.dpdk_opts.dpdk_port_index.get_value(), smp::count,\n                !(opts.lro && opts.lro.get_value() == \"off\"),\n                !(opts.dpdk_opts.hw_fc && opts.dpdk_opts.hw_fc.get_value() == \"off\"));\n       } else\n#endif\n        dev = create_virtio_net_device(opts.virtio_opts, opts.lro);\n    }\n    else {\n        auto device_configs = parse_config(net_config);\n\n        if ( device_configs.size() > 1) {\n            std::runtime_error(\"only one network interface is supported\");\n        }\n\n        for ( auto&& device_config : device_configs) {\n            auto& hw_config = device_config.second.hw_cfg;\n#ifdef SEASTAR_HAVE_DPDK\n            if ( hw_config.port_index || !hw_config.pci_address.empty() ) {\n\t            dev = create_dpdk_net_device(hw_config);\n\t        } else\n#endif\n            {\n                (void)hw_config;\n                std::runtime_error(\"only DPDK supports new configuration format\");\n            }\n        }\n    }\n\n    auto sem = std::make_shared<semaphore>(0);\n    std::shared_ptr<device> sdev(dev.release());\n    // set_local_queue on all shard in the background,\n    // signal when done.\n    // FIXME: handle exceptions\n    for (unsigned i = 0; i < smp::count; i++) {\n        (void)smp::submit_to(i, [&opts, sdev] {\n            uint16_t qid = this_shard_id();\n            if (qid < sdev->hw_queues_count()) {\n                auto qp = sdev->init_local_queue(opts, qid);\n                std::map<unsigned, float> cpu_weights;\n                for (unsigned i = sdev->hw_queues_count() + qid % sdev->hw_queues_count(); i < smp::count; i+= sdev->hw_queues_count()) {\n                    cpu_weights[i] = 1;\n                }\n                cpu_weights[qid] = opts.hw_queue_weight.get_value();\n                qp->configure_proxies(cpu_weights);\n                sdev->set_local_queue(std::move(qp));\n            } else {\n                auto master = qid % sdev->hw_queues_count();\n                sdev->set_local_queue(create_proxy_net_device(master, sdev.get()));\n            }\n        }).then([sem] {\n            sem->signal();\n        });\n    }\n    // wait for all shards to set their local queue,\n    // then when link is ready, communicate the native_stack to the caller\n    // via `create_native_stack` (that sets the ready_promise value)\n    (void)sem->wait(smp::count).then([&opts, sdev] {\n        // FIXME: future is discarded\n        (void)sdev->link_ready().then([&opts, sdev] {\n            for (unsigned i = 0; i < smp::count; i++) {\n                // FIXME: future is discarded\n                (void)smp::submit_to(i, [&opts, sdev] {\n                    create_native_stack(opts, sdev);\n                });\n            }\n        });\n    });\n}\n\n// native_network_stack\nclass native_network_stack : public network_stack {\npublic:\n    static thread_local promise<std::unique_ptr<network_stack>> ready_promise;\nprivate:\n    interface _netif;\n    ipv4 _inet;\n    bool _dhcp = false;\n    promise<> _config;\n    timer<> _timer;\n\n    future<> run_dhcp(bool is_renew = false, const dhcp::lease & res = dhcp::lease());\n    void on_dhcp(std::optional<dhcp::lease> lease, bool is_renew);\n    void set_ipv4_packet_filter(ip_packet_filter* filter) {\n        _inet.set_packet_filter(filter);\n    }\n    using tcp4 = tcp<ipv4_traits>;\npublic:\n    explicit native_network_stack(const native_stack_options& opts, std::shared_ptr<device> dev);\n    virtual server_socket listen(socket_address sa, listen_options opt) override;\n    virtual ::seastar::socket socket() override;\n    virtual udp_channel make_udp_channel(const socket_address& addr) override;\n    virtual net::datagram_channel make_unbound_datagram_channel(sa_family_t) override;\n    virtual net::datagram_channel make_bound_datagram_channel(const socket_address& local) override;\n    virtual future<> initialize() override;\n    static future<std::unique_ptr<network_stack>> create(const program_options::option_group& opts) {\n        auto ns_opts = dynamic_cast<const native_stack_options*>(&opts);\n        SEASTAR_ASSERT(ns_opts);\n        if (this_shard_id() == 0) {\n            create_native_net_device(*ns_opts);\n        }\n        return ready_promise.get_future();\n    }\n    virtual bool has_per_core_namespace() override { return true; };\n    void arp_learn(ethernet_address l2, ipv4_address l3) {\n        _inet.learn(l2, l3);\n    }\n    friend class native_server_socket_impl<tcp4>;\n\n    class native_network_interface;\n    friend class native_network_interface;\n\n    std::vector<network_interface> network_interfaces() override;\n\n    virtual statistics stats(unsigned scheduling_group_id) override {\n        return statistics{\n            internal::native_stack_net_stats::bytes_sent[scheduling_group_id],\n            internal::native_stack_net_stats::bytes_received[scheduling_group_id],\n        };\n    }\n\n    virtual void clear_stats(unsigned scheduling_group_id) override {\n        internal::native_stack_net_stats::bytes_sent[scheduling_group_id] = 0;\n        internal::native_stack_net_stats::bytes_received[scheduling_group_id] = 0;\n    }\n};\n\nthread_local promise<std::unique_ptr<network_stack>> native_network_stack::ready_promise;\n\nudp_channel\nnative_network_stack::make_udp_channel(const socket_address& addr) {\n    return _inet.get_udp().make_channel(addr);\n}\n\nnet::datagram_channel native_network_stack::make_unbound_datagram_channel(sa_family_t family) {\n    if (family != AF_INET) {\n        throw std::runtime_error(\"Unsupported address family\");\n    }\n\n    return _inet.get_udp().make_channel({});\n}\n\nnet::datagram_channel native_network_stack::make_bound_datagram_channel(const socket_address& local) {\n    return _inet.get_udp().make_channel(local);\n}\n\nnative_network_stack::native_network_stack(const native_stack_options& opts, std::shared_ptr<device> dev)\n    : _netif(std::move(dev))\n    , _inet(&_netif) {\n    _inet.get_udp().set_queue_size(opts.udpv4_queue_size.get_value());\n    _dhcp = opts.host_ipv4_addr.defaulted()\n            && opts.gw_ipv4_addr.defaulted()\n            && opts.netmask_ipv4_addr.defaulted() && opts.dhcp.get_value();\n    if (!_dhcp) {\n        _inet.set_host_address(ipv4_address(opts.host_ipv4_addr.get_value()));\n        _inet.set_gw_address(ipv4_address(opts.gw_ipv4_addr.get_value()));\n        _inet.set_netmask_address(ipv4_address(opts.netmask_ipv4_addr.get_value()));\n    }\n}\n\nserver_socket\nnative_network_stack::listen(socket_address sa, listen_options opts) {\n    SEASTAR_ASSERT(sa.family() == AF_INET || sa.is_unspecified());\n    return tcpv4_listen(_inet.get_tcp(), ntohs(sa.as_posix_sockaddr_in().sin_port), opts);\n}\n\nseastar::socket native_network_stack::socket() {\n    return tcpv4_socket(_inet.get_tcp());\n}\n\nusing namespace std::chrono_literals;\n\nfuture<> native_network_stack::run_dhcp(bool is_renew, const dhcp::lease& res) {\n    dhcp d(_inet);\n    // Hijack the ip-stack.\n    auto f = d.get_ipv4_filter();\n    return smp::invoke_on_all([f] {\n        auto & ns = static_cast<native_network_stack&>(engine().net());\n        ns.set_ipv4_packet_filter(f);\n    }).then([this, d = std::move(d), is_renew, res = res]() mutable {\n        net::dhcp::result_type fut = is_renew ? d.renew(res) : d.discover();\n        return fut.then([this, is_renew](std::optional<dhcp::lease> lease) {\n            return smp::invoke_on_all([] {\n                auto & ns = static_cast<native_network_stack&>(engine().net());\n                ns.set_ipv4_packet_filter(nullptr);\n            }).then(std::bind(&net::native_network_stack::on_dhcp, this, lease, is_renew));\n        }).finally([d = std::move(d)] {});\n    });\n}\n\nvoid native_network_stack::on_dhcp(std::optional<dhcp::lease> lease, bool is_renew) {\n    if (lease) {\n        auto& res = *lease;\n        _inet.set_host_address(res.ip);\n        _inet.set_gw_address(res.gateway);\n        _inet.set_netmask_address(res.netmask);\n    }\n    // Signal waiters.\n    if (!is_renew) {\n        _config.set_value();\n    }\n\n    if (this_shard_id() == 0) {\n        // And the other cpus, which, in the case of initial discovery,\n        // will be waiting for us.\n        for (unsigned i = 1; i < smp::count; i++) {\n            (void)smp::submit_to(i, [lease, is_renew]() {\n                auto & ns = static_cast<native_network_stack&>(engine().net());\n                ns.on_dhcp(lease, is_renew);\n            });\n        }\n        if (lease) {\n            // And set up to renew the lease later on.\n            auto& res = *lease;\n            _timer.set_callback(\n                    [this, res]() {\n                        _config = promise<>();\n                        // callback ignores future result\n                        (void)run_dhcp(true, res);\n                    });\n            _timer.arm(\n                    std::chrono::duration_cast<steady_clock_type::duration>(\n                            res.lease_time));\n        }\n    }\n}\n\nfuture<> native_network_stack::initialize() {\n    return network_stack::initialize().then([this]() {\n        if (!_dhcp) {\n            return make_ready_future();\n        }\n\n        // Only run actual discover on main cpu.\n        // All other cpus must simply for main thread to complete and signal them.\n        if (this_shard_id() == 0) {\n            // FIXME: future is discarded\n            (void)run_dhcp();\n        }\n        return _config.get_future();\n    });\n}\n\nvoid arp_learn(ethernet_address l2, ipv4_address l3)\n{\n    // Run arp_learn on all shard in the background\n    (void)smp::invoke_on_all([l2, l3] {\n        auto & ns = static_cast<native_network_stack&>(engine().net());\n        ns.arp_learn(l2, l3);\n    });\n}\n\nvoid create_native_stack(const native_stack_options& opts, std::shared_ptr<device> dev) {\n    native_network_stack::ready_promise.set_value(std::unique_ptr<network_stack>(std::make_unique<native_network_stack>(opts, std::move(dev))));\n}\n\nnative_stack_options::native_stack_options()\n    : program_options::option_group(nullptr, \"Native networking stack options\")\n    // these two are ghost options\n    , net_config(*this, \"net-config\", program_options::unused{})\n    , net_config_file(*this, \"net-config-file\", program_options::unused{})\n    , tap_device(*this, \"tap-device\",\n                \"tap0\",\n                \"tap device to connect to\")\n    , host_ipv4_addr(*this, \"host-ipv4-addr\",\n                \"192.168.122.2\",\n                \"static IPv4 address to use\")\n    , gw_ipv4_addr(*this, \"gw-ipv4-addr\",\n                \"192.168.122.1\",\n                \"static IPv4 gateway to use\")\n    , netmask_ipv4_addr(*this, \"netmask-ipv4-addr\",\n                \"255.255.255.0\",\n                \"static IPv4 netmask to use\")\n    , udpv4_queue_size(*this, \"udpv4-queue-size\",\n                ipv4_udp::default_queue_size,\n                \"Default size of the UDPv4 per-channel packet queue\")\n    , dhcp(*this, \"dhcp\",\n                true,\n                        \"Use DHCP discovery\")\n    , hw_queue_weight(*this, \"hw-queue-weight\",\n                1.0f,\n                \"Weighing of a hardware network queue relative to a software queue (0=no work, 1=equal share)\")\n#ifdef SEASTAR_HAVE_DPDK\n    , dpdk_pmd(*this, \"dpdk-pmd\", \"Use DPDK PMD drivers\")\n#else\n    , dpdk_pmd(*this, \"dpdk-pmd\", program_options::unused{})\n#endif\n    , lro(*this, \"lro\",\n                \"on\",\n                \"Enable LRO\")\n    , virtio_opts(this)\n    , dpdk_opts(this)\n{\n}\n\nnetwork_stack_entry register_native_stack() {\n    return network_stack_entry{\"native\", std::make_unique<native_stack_options>(), native_network_stack::create, false};\n}\n\nclass native_network_stack::native_network_interface : public net::network_interface_impl {\n    const native_network_stack& _stack;\n    std::vector<net::inet_address> _addresses;\n    std::vector<uint8_t> _hardware_address;\npublic:\n    native_network_interface(const native_network_stack& stack)\n        : _stack(stack)\n        , _addresses(1, _stack._inet.host_address())\n    {\n        const auto mac = _stack._inet.netif()->hw_address().mac;\n        _hardware_address = std::vector<uint8_t>{mac.cbegin(), mac.cend()};\n    }\n    native_network_interface(const native_network_interface&) = default;\n\n    uint32_t index() const override {\n        return 0;\n    }\n    uint32_t mtu() const override {\n        return _stack._inet.netif()->hw_features().mtu;\n    }\n    const sstring& name() const override {\n        static const sstring name = \"if0\";\n        return name;\n    }\n    const sstring& display_name() const override {\n        return name();\n    }\n    const std::vector<net::inet_address>& addresses() const override {\n        return _addresses;\n    }\n    const std::vector<uint8_t> hardware_address() const override {\n        return _hardware_address;\n    }\n    bool is_loopback() const override {\n        return false;\n    }\n    bool is_virtual() const override {\n        return false;\n    }\n    bool is_up() const override {\n        return true;\n    }\n    bool supports_ipv6() const override {\n        return false;\n    }\n};\n\nstd::vector<network_interface> native_network_stack::network_interfaces() {\n    if (!_inet.netif()) {\n        return {};\n    }\n\n    static const native_network_interface nwif(*this);\n\n    std::vector<network_interface> res;\n    res.emplace_back(make_shared<native_network_interface>(nwif));\n    return res;\n}\n\n} // namespace net\n\n}\n"
  },
  {
    "path": "src/net/net.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n *\n */\n\n\n#include <boost/asio/ip/address_v4.hpp>\n#include <boost/algorithm/string.hpp>\n#include <map>\n#include <utility>\n\n#include <seastar/net/net.hh>\n#include <seastar/net/toeplitz.hh>\n#include <seastar/core/internal/poll.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/print.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nstatic_assert(std::is_nothrow_default_constructible_v<ipv4_addr>);\nstatic_assert(std::is_nothrow_copy_constructible_v<ipv4_addr>);\nstatic_assert(std::is_nothrow_move_constructible_v<ipv4_addr>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<ipv6_addr>);\nstatic_assert(std::is_nothrow_copy_constructible_v<ipv6_addr>);\nstatic_assert(std::is_nothrow_move_constructible_v<ipv6_addr>);\n\nusing std::move;\n\nipv4_addr::ipv4_addr(const std::string &addr) {\n    std::vector<std::string> items;\n    boost::split(items, addr, boost::is_any_of(\":\"));\n    if (items.size() == 1) {\n        ip = boost::asio::ip::make_address_v4(addr).to_uint();\n        port = 0;\n    } else if (items.size() == 2) {\n        ip = boost::asio::ip::make_address_v4(items[0]).to_uint();\n        port = std::stoul(items[1]);\n    } else {\n        throw std::invalid_argument(\"invalid format: \" + addr);\n    }\n}\n\nipv4_addr::ipv4_addr(const std::string &addr, uint16_t port_) : ip(boost::asio::ip::make_address_v4(addr).to_uint()), port(port_) {}\n\nipv4_addr::ipv4_addr(const net::inet_address& a, uint16_t port)\n    : ipv4_addr(::in_addr(a), port)\n{}\n\nipv4_addr::ipv4_addr(const socket_address &sa) noexcept\n    : ipv4_addr(sa.addr(), sa.port())\n{}\n\nipv4_addr::ipv4_addr(const ::in_addr& in, uint16_t p) noexcept\n    : ip(net::ntoh(in.s_addr)), port(p)\n{}\n\nnamespace net {\n\ninline\nbool qp::poll_tx() {\n    if (_tx_packetq.size() < 16) {\n        // refill send queue from upper layers\n        uint32_t work;\n        do {\n            work = 0;\n            for (auto&& pr : _pkt_providers) {\n                auto p = pr();\n                if (p) {\n                    work++;\n                    _tx_packetq.push_back(std::move(p.value()));\n                    if (_tx_packetq.size() == 128) {\n                        break;\n                    }\n                }\n            }\n        } while (work && _tx_packetq.size() < 128);\n    }\n    if (!_tx_packetq.empty()) {\n        _stats.tx.good.update_pkts_bunch(send(_tx_packetq));\n        return true;\n    }\n\n    return false;\n}\n\nqp::qp(bool register_copy_stats,\n       const std::string stats_plugin_name, uint8_t qid)\n        : _tx_poller(std::make_unique<internal::poller>(reactor::poller::simple([this] { return poll_tx(); })))\n        , _stats_plugin_name(stats_plugin_name)\n        , _queue_name(std::string(\"queue\") + std::to_string(qid))\n{\n    namespace sm = metrics;\n\n    _metrics.add_group(_stats_plugin_name, {\n        //\n        // Packets rate: DERIVE:0:u\n        //\n        sm::make_counter(_queue_name + \"_rx_packets\", _stats.rx.good.packets,\n                        sm::description(\"This metric is a receive packet rate for this queue.\")),\n\n        sm::make_counter(_queue_name + \"_tx_packets\", _stats.tx.good.packets,\n                        sm::description(\"This metric is a transmit packet rate for this queue.\")),\n        //\n        // Bytes rate: DERIVE:0:U\n        //\n        sm::make_counter(_queue_name + \"_rx_bytes\", _stats.rx.good.bytes,\n                        sm::description(\"This metric is a receive throughput for this queue.\")),\n\n        sm::make_counter(_queue_name + \"_tx_bytes\", _stats.tx.good.bytes,\n                        sm::description(\"This metric is a transmit throughput for this queue.\")),\n        //\n        // Queue length: GAUGE:0:U\n        //\n        // Tx\n        sm::make_gauge(_queue_name + \"_tx_packet_queue\", [this] { return _tx_packetq.size(); },\n                        sm::description(\"Holds a number of packets pending to be sent. \"\n                                        \"This metric will have high values if the network backend doesn't keep up with the upper layers or if upper layers send big bursts of packets.\")),\n\n        //\n        // Linearization counter: DERIVE:0:U\n        //\n        sm::make_counter(_queue_name + \"_xmit_linearized\", _stats.tx.linearized,\n                        sm::description(\"Counts a number of linearized Tx packets. High value indicates that we send too fragmented packets.\")),\n\n        //\n        // Number of packets in last bunch: GAUGE:0:U\n        //\n        // Tx\n        sm::make_gauge(_queue_name + \"_tx_packet_queue_last_bunch\", _stats.tx.good.last_bunch,\n                        sm::description(seastar::format(\"Holds a number of packets sent in the bunch. \"\n                                        \"A high value in conjunction with a high value of a {} indicates an efficient Tx packets bulking.\", _queue_name + \"_tx_packet_queue\"))),\n        // Rx\n        sm::make_gauge(_queue_name + \"_rx_packet_queue_last_bunch\", _stats.rx.good.last_bunch,\n                        sm::description(\"Holds a number of packets received in the last Rx bunch. High value indicates an efficient Rx packets bulking.\")),\n\n        //\n        // Fragments rate: DERIVE:0:U\n        //\n        // Tx\n        sm::make_counter(_queue_name + \"_tx_frags\", _stats.tx.good.nr_frags,\n                        sm::description(seastar::format(\"Counts a number of sent fragments. Divide this value by a {} to get an average number of fragments in a Tx packet.\", _queue_name + \"_tx_packets\"))),\n        // Rx\n        sm::make_counter(_queue_name + \"_rx_frags\", _stats.rx.good.nr_frags,\n                        sm::description(seastar::format(\"Counts a number of received fragments. Divide this value by a {} to get an average number of fragments in an Rx packet.\", _queue_name + \"_rx_packets\"))),\n    });\n\n    if (register_copy_stats) {\n        _metrics.add_group(_stats_plugin_name, {\n            //\n            // Non-zero-copy data bytes rate: DERIVE:0:u\n            //\n            // Tx\n            sm::make_counter(_queue_name + \"_tx_copy_bytes\", _stats.tx.good.copy_bytes,\n                        sm::description(seastar::format(\"Counts a number of sent bytes that were handled in a non-zero-copy way. Divide this value by a {} to get a portion of data sent using a non-zero-copy flow.\", _queue_name + \"_tx_bytes\"))),\n            // Rx\n            sm::make_counter(_queue_name + \"_rx_copy_bytes\", _stats.rx.good.copy_bytes,\n                        sm::description(seastar::format(\"Counts a number of received bytes that were handled in a non-zero-copy way. Divide this value by an {} to get a portion of received data handled using a non-zero-copy flow.\", _queue_name + \"_rx_bytes\"))),\n\n            //\n            // Non-zero-copy data fragments rate: DERIVE:0:u\n            //\n            // Tx\n            sm::make_counter(_queue_name + \"_tx_copy_frags\", _stats.tx.good.copy_frags,\n                        sm::description(seastar::format(\"Counts a number of sent fragments that were handled in a non-zero-copy way. Divide this value by a {} to get a portion of fragments sent using a non-zero-copy flow.\", _queue_name + \"_tx_frags\"))),\n            // Rx\n            sm::make_counter(_queue_name + \"_rx_copy_frags\", _stats.rx.good.copy_frags,\n                        sm::description(seastar::format(\"Counts a number of received fragments that were handled in a non-zero-copy way. Divide this value by a {} to get a portion of received fragments handled using a non-zero-copy flow.\", _queue_name + \"_rx_frags\"))),\n\n        });\n    }\n}\n\nqp::~qp() {\n}\n\nvoid qp::configure_proxies(const std::map<unsigned, float>& cpu_weights) {\n    SEASTAR_ASSERT(!cpu_weights.empty());\n    if ((cpu_weights.size() == 1 && cpu_weights.begin()->first == this_shard_id())) {\n        // special case queue sending to self only, to avoid requiring a hash value\n        return;\n    }\n    register_packet_provider([this] {\n        std::optional<packet> p;\n        if (!_proxy_packetq.empty()) {\n            p = std::move(_proxy_packetq.front());\n            _proxy_packetq.pop_front();\n        }\n        return p;\n    });\n    build_sw_reta(cpu_weights);\n}\n\nvoid qp::build_sw_reta(const std::map<unsigned, float>& cpu_weights) {\n    float total_weight = 0;\n    for (auto&& x : cpu_weights) {\n        total_weight += x.second;\n    }\n    float accum = 0;\n    unsigned idx = 0;\n    std::array<uint8_t, 128> reta;\n    for (auto&& entry : cpu_weights) {\n        auto cpu = entry.first;\n        auto weight = entry.second;\n        accum += weight;\n        while (idx < (accum / total_weight * reta.size() - 0.5)) {\n            reta[idx++] = cpu;\n        }\n    }\n    _sw_reta = reta;\n}\n\nfuture<>\ndevice::receive(std::function<future<> (packet)> next_packet) {\n    auto sub = _queues[this_shard_id()]->_rx_stream.listen(std::move(next_packet));\n    _queues[this_shard_id()]->rx_start();\n    return sub.done();\n}\n\nvoid device::set_local_queue(std::unique_ptr<qp> dev) {\n    SEASTAR_ASSERT(!_queues[this_shard_id()]);\n    _queues[this_shard_id()] = dev.get();\n    internal::at_destroy([dev = std::move(dev)] {});\n}\n\n\nl3_protocol::l3_protocol(interface* netif, eth_protocol_num proto_num, packet_provider_type func)\n    : _netif(netif), _proto_num(proto_num)  {\n        _netif->register_packet_provider(std::move(func));\n}\n\nfuture<> l3_protocol::receive(\n        std::function<future<> (packet p, ethernet_address from)> rx_fn,\n        std::function<bool (forward_hash&, packet&, size_t)> forward) {\n    return _netif->register_l3(_proto_num, std::move(rx_fn), std::move(forward));\n};\n\ninterface::interface(std::shared_ptr<device> dev)\n    : _dev(dev)\n    , _hw_address(_dev->hw_address())\n    , _hw_features(_dev->hw_features()) {\n    // FIXME: ignored future\n    (void)_dev->receive([this] (packet p) {\n        return dispatch_packet(std::move(p));\n    });\n    dev->local_queue().register_packet_provider([this, idx = 0u] () mutable {\n            std::optional<packet> p;\n            for (size_t i = 0; i < _pkt_providers.size(); i++) {\n                auto l3p = _pkt_providers[idx++]();\n                if (idx == _pkt_providers.size())\n                    idx = 0;\n                if (l3p) {\n                    auto l3pv = std::move(l3p.value());\n                    auto eh = l3pv.p.prepend_header<eth_hdr>();\n                    eh->dst_mac = l3pv.to;\n                    eh->src_mac = _hw_address;\n                    eh->eth_proto = uint16_t(l3pv.proto_num);\n                    *eh = hton(*eh);\n                    p = std::move(l3pv.p);\n                    return p;\n                }\n            }\n            return p;\n        });\n}\n\nfuture<>\ninterface::register_l3(eth_protocol_num proto_num,\n        std::function<future<> (packet p, ethernet_address from)> next,\n        std::function<bool (forward_hash&, packet& p, size_t)> forward) {\n    auto i = _proto_map.emplace(std::piecewise_construct, std::make_tuple(uint16_t(proto_num)), std::forward_as_tuple(std::move(forward)));\n    SEASTAR_ASSERT(i.second);\n    l3_rx_stream& l3_rx = i.first->second;\n    return l3_rx.packet_stream.listen(std::move(next)).done();\n}\n\nunsigned interface::hash2cpu(uint32_t hash) {\n    return _dev->hash2cpu(hash);\n}\n\nuint16_t interface::hw_queues_count() {\n    return _dev->hw_queues_count();\n}\n\nrss_key_type interface::rss_key() const {\n    return _dev->rss_key();\n}\n\nvoid interface::forward(unsigned cpuid, packet p) {\n    static __thread unsigned queue_depth;\n\n    if (queue_depth < 1000) {\n        queue_depth++;\n        auto src_cpu = this_shard_id();\n        // FIXME: future is discarded\n        (void)smp::submit_to(cpuid, [this, p = std::move(p), src_cpu]() mutable {\n            _dev->l2receive(p.free_on_cpu(src_cpu));\n        }).then([] {\n            queue_depth--;\n        });\n    }\n}\n\nfuture<> interface::dispatch_packet(packet p) {\n    auto eh = p.get_header<eth_hdr>();\n    if (eh) {\n        auto i = _proto_map.find(ntoh(eh->eth_proto));\n        if (i != _proto_map.end()) {\n            l3_rx_stream& l3 = i->second;\n            auto fw = _dev->forward_dst(this_shard_id(), [&p, &l3, this] () {\n                auto hwrss = p.rss_hash();\n                if (hwrss) {\n                    return hwrss.value();\n                } else {\n                    forward_hash data;\n                    if (l3.forward(data, p, sizeof(eth_hdr))) {\n                        return toeplitz_hash(rss_key(), data);\n                    }\n                    return 0u;\n                }\n            });\n            if (fw != this_shard_id()) {\n                forward(fw, std::move(p));\n            } else {\n                auto h = ntoh(*eh);\n                auto from = h.src_mac;\n                p.trim_front(sizeof(*eh));\n                // avoid chaining, since queue lenth is unlimited\n                // drop instead.\n                if (l3.ready.available()) {\n                    l3.ready = l3.packet_stream.produce(std::move(p), from);\n                }\n            }\n        }\n    }\n    return make_ready_future<>();\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/packet.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <iostream>\n#include <algorithm>\n#include <cctype>\n#include <cstdint>\n#include <functional>\n#include <memory>\n\n#include <seastar/core/print.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/net/packet.hh>\n\nnamespace seastar {\n\nstatic_assert(std::is_nothrow_constructible_v<deleter>);\nstatic_assert(std::is_nothrow_move_constructible_v<deleter>);\n\nnamespace net {\n\nstatic_assert(std::is_nothrow_move_constructible_v<packet>);\n\nvoid packet::linearize(size_t at_frag, size_t desired_size) {\n    _impl->unuse_internal_data();\n    size_t nr_frags = 0;\n    size_t accum_size = 0;\n    while (accum_size < desired_size) {\n        accum_size += _impl->_frags[at_frag + nr_frags].size;\n        ++nr_frags;\n    }\n    std::unique_ptr<char[]> new_frag{new char[accum_size]};\n    auto p = new_frag.get();\n    for (size_t i = 0; i < nr_frags; ++i) {\n        auto& f = _impl->_frags[at_frag + i];\n        p = std::copy(f.base, f.base + f.size, p);\n    }\n    // collapse nr_frags into one fragment\n    std::copy(_impl->_frags + at_frag + nr_frags, _impl->_frags + _impl->_nr_frags,\n            _impl->_frags + at_frag + 1);\n    _impl->_nr_frags -= nr_frags - 1;\n    _impl->_frags[at_frag] = fragment{new_frag.get(), accum_size};\n    if (at_frag == 0 && desired_size == len()) {\n        // We can drop the old buffer safely\n        auto x = std::move(_impl->_deleter);\n        _impl->_deleter = make_deleter([buf = std::move(new_frag)] {});\n    } else {\n        _impl->_deleter = make_deleter(std::move(_impl->_deleter), [buf = std::move(new_frag)] {});\n    }\n}\n\n\npacket packet::free_on_cpu(unsigned cpu, std::function<void()> cb)\n{\n    // make new deleter that runs old deleter on an origin cpu\n    _impl->_deleter = make_deleter(deleter(), [d = std::move(_impl->_deleter), cpu, cb = std::move(cb)] () mutable {\n        // FIXME: future is discarded\n        (void)smp::submit_to(cpu, [d = std::move(d), cb = std::move(cb)] () mutable {\n            // deleter needs to be moved from lambda capture to be destroyed here\n            // otherwise deleter destructor will be called on a cpu that called smp::submit_to()\n            // when work_item is destroyed.\n            deleter xxx(std::move(d));\n            cb();\n        });\n    });\n\n    return packet(impl::copy(_impl.get()));\n}\n\nstd::ostream& operator<<(std::ostream& os, const packet& p) {\n    os << \"packet{\";\n    bool first = true;\n    for (auto&& frag : p.fragments()) {\n        if (!first) {\n            os << \", \";\n        }\n        first = false;\n        if (std::all_of(frag.base, frag.base + frag.size, [] (int c) { return c >= 9 && c <= 0x7f; })) {\n            os << '\"';\n            for (auto p = frag.base; p != frag.base + frag.size; ++p) {\n                auto c = *p;\n                if (isprint(c)) {\n                    os << c;\n                } else if (c == '\\r') {\n                    os << \"\\\\r\";\n                } else if (c == '\\n') {\n                    os << \"\\\\n\";\n                } else if (c == '\\t') {\n                    os << \"\\\\t\";\n                } else {\n                    uint8_t b = c;\n                    os << \"\\\\x\" << (b / 16) << (b % 16);\n                }\n            }\n            os << '\"';\n        } else {\n            os << \"{\";\n            bool nfirst = true;\n            for (auto p = frag.base; p != frag.base + frag.size; ++p) {\n                if (!nfirst) {\n                    os << \" \";\n                }\n                nfirst = false;\n                uint8_t b = *p;\n                os << format(\"{:02x}\", unsigned(b));\n            }\n            os << \"}\";\n        }\n    }\n    os << \"}\";\n    return os;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/posix-stack.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <chrono>\n#include <cstring>\n#include <functional>\n#include <random>\n#include <variant>\n#include <coroutine>\n\n#include <unistd.h>\n#include <linux/if.h>\n#include <linux/netlink.h>\n#include <linux/rtnetlink.h>\n#include <arpa/inet.h>\n#include <net/route.h>\n#include <netinet/tcp.h>\n#include <netinet/sctp.h>\n#include <sys/socket.h>\n#include <seastar/util/assert.hh>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/net/posix-stack.hh>\n#include <seastar/net/net.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/std-compat.hh>\n\nnamespace std {\n\ntemplate <>\nstruct hash<seastar::net::posix_ap_server_socket_impl::protocol_and_socket_address> {\n    size_t operator()(const seastar::net::posix_ap_server_socket_impl::protocol_and_socket_address& t_sa) const {\n        auto h1 = std::hash<int>()(std::get<0>(t_sa));\n        auto h2 = std::hash<seastar::net::socket_address>()(std::get<1>(t_sa));\n        return h1 ^ h2;\n    }\n};\n\n}\n\n\nnamespace {\n\n// reinterpret_cast<foo*>() on a pointer that the compiler knows points to an\n// object with a different type is disliked by the compiler as it violates\n// strict aliasing rules. This safe version does the same thing but keeps the\n// compiler happy.\ntemplate <typename T>\nT\ncopy_reinterpret_cast(const void* ptr) {\n    T tmp;\n    std::memcpy(&tmp, ptr, sizeof(T));\n    return tmp;\n}\n\nthread_local std::array<uint64_t, seastar::max_scheduling_groups()> bytes_sent = {};\nthread_local std::array<uint64_t, seastar::max_scheduling_groups()> bytes_received = {};\n\n}\n\nnamespace seastar {\n\nnamespace internal {\n\nstd::pair<size_t, deleter> wrapped_iovecs::populate(std::span<temporary_buffer<char>> bufs) {\n    size_t total = 0;\n    deleter del;\n\n    if (!v.empty()) {\n        on_internal_error(seastar_logger, \"wrapped_iovecs are re-populated live\");\n    }\n\n    v.reserve(bufs.size());\n    for (auto&& buf : bufs) {\n        auto b = std::move(buf);\n        total += b.size();\n        v.push_back({ .iov_base = b.get_write(), .iov_len = b.size() });\n        deleter d = b.release();\n        d.append(std::move(del));\n        del = std::move(d);\n    }\n\n    return std::make_pair(total, make_deleter(std::move(del), [this] { v.clear(); }));\n}\n\n}\n\nnamespace net {\n\nusing namespace seastar;\n\nclass posix_connected_socket_operations {\npublic:\n    virtual ~posix_connected_socket_operations() = default;\n    virtual void set_nodelay(file_desc& fd, bool nodelay) const = 0;\n    virtual bool get_nodelay(file_desc& fd) const = 0;\n    virtual void set_keepalive(file_desc& _fd, bool keepalive) const = 0;\n    virtual bool get_keepalive(file_desc& _fd) const = 0;\n    virtual void set_keepalive_parameters(file_desc& _fd, const keepalive_params& params) const = 0;\n    virtual keepalive_params get_keepalive_parameters(file_desc& _fd) const = 0;\n    virtual void set_sockopt(file_desc& _fd, int level, int optname, const void* data, size_t len) const {\n        _fd.setsockopt(level, optname, data, socklen_t(len));\n    }\n    virtual int get_sockopt(file_desc& _fd, int level, int optname, void* data, size_t len) const {\n        return _fd.getsockopt(level, optname, reinterpret_cast<char*>(data), socklen_t(len));\n    }\n    virtual socket_address local_address(file_desc& _fd) const {\n        return _fd.get_address();\n    }\n    virtual socket_address remote_address(file_desc& _fd) const {\n        try {\n            return _fd.get_remote_address();\n        } catch (const std::system_error& e) {\n            if (e.code().value() != ENOTCONN) {\n                throw;\n            }\n            return socket_address();\n        }\n    }\n};\n\nthread_local posix_ap_server_socket_impl::port_map_t posix_ap_server_socket_impl::ports{};\nthread_local posix_ap_server_socket_impl::sockets_map_t posix_ap_server_socket_impl::sockets{};\nthread_local posix_ap_server_socket_impl::conn_map_t posix_ap_server_socket_impl::conn_q{};\n\nclass posix_tcp_connected_socket_operations : public posix_connected_socket_operations {\npublic:\n    virtual void set_nodelay(file_desc& _fd, bool nodelay) const override {\n        _fd.setsockopt(IPPROTO_TCP, TCP_NODELAY, int(nodelay));\n    }\n    virtual bool get_nodelay(file_desc& _fd) const override {\n        return _fd.getsockopt<int>(IPPROTO_TCP, TCP_NODELAY);\n    }\n    virtual void set_keepalive(file_desc& _fd, bool keepalive) const override {\n        _fd.setsockopt(SOL_SOCKET, SO_KEEPALIVE, int(keepalive));\n    }\n    virtual bool get_keepalive(file_desc& _fd) const override {\n        return _fd.getsockopt<int>(SOL_SOCKET, SO_KEEPALIVE);\n    }\n    virtual void set_keepalive_parameters(file_desc& _fd, const keepalive_params& params) const override {\n        const tcp_keepalive_params& pms = std::get<tcp_keepalive_params>(params);\n        _fd.setsockopt(IPPROTO_TCP, TCP_KEEPCNT, pms.count);\n        _fd.setsockopt(IPPROTO_TCP, TCP_KEEPIDLE, int(pms.idle.count()));\n        _fd.setsockopt(IPPROTO_TCP, TCP_KEEPINTVL, int(pms.interval.count()));\n    }\n    virtual keepalive_params get_keepalive_parameters(file_desc& _fd) const override {\n        return tcp_keepalive_params {\n            std::chrono::seconds(_fd.getsockopt<int>(IPPROTO_TCP, TCP_KEEPIDLE)),\n            std::chrono::seconds(_fd.getsockopt<int>(IPPROTO_TCP, TCP_KEEPINTVL)),\n            _fd.getsockopt<unsigned>(IPPROTO_TCP, TCP_KEEPCNT)\n        };\n    }\n};\n\nclass posix_sctp_connected_socket_operations : public posix_connected_socket_operations {\npublic:\n    virtual void set_nodelay(file_desc& _fd, bool nodelay) const override {\n        _fd.setsockopt(SOL_SCTP, SCTP_NODELAY, int(nodelay));\n    }\n    virtual bool get_nodelay(file_desc& _fd) const override {\n        return _fd.getsockopt<int>(SOL_SCTP, SCTP_NODELAY);\n    }\n    virtual void set_keepalive(file_desc& _fd, bool keepalive) const override {\n        auto heartbeat = _fd.getsockopt<sctp_paddrparams>(SOL_SCTP, SCTP_PEER_ADDR_PARAMS);\n        if (keepalive) {\n            heartbeat.spp_flags |= SPP_HB_ENABLE;\n        } else {\n            heartbeat.spp_flags &= ~SPP_HB_ENABLE;\n        }\n        _fd.setsockopt(SOL_SCTP, SCTP_PEER_ADDR_PARAMS, heartbeat);\n    }\n    virtual bool get_keepalive(file_desc& _fd) const override {\n        return _fd.getsockopt<sctp_paddrparams>(SOL_SCTP, SCTP_PEER_ADDR_PARAMS).spp_flags & SPP_HB_ENABLE;\n    }\n    virtual void set_keepalive_parameters(file_desc& _fd, const keepalive_params& kpms) const override {\n        const sctp_keepalive_params& pms = std::get<sctp_keepalive_params>(kpms);\n        auto params = _fd.getsockopt<sctp_paddrparams>(SOL_SCTP, SCTP_PEER_ADDR_PARAMS);\n        params.spp_hbinterval = pms.interval.count() * 1000; // in milliseconds\n        params.spp_pathmaxrxt = pms.count;\n        _fd.setsockopt(SOL_SCTP, SCTP_PEER_ADDR_PARAMS, params);\n    }\n    virtual keepalive_params get_keepalive_parameters(file_desc& _fd) const override {\n        auto params = _fd.getsockopt<sctp_paddrparams>(SOL_SCTP, SCTP_PEER_ADDR_PARAMS);\n        return sctp_keepalive_params {\n            std::chrono::seconds(params.spp_hbinterval/1000), // in seconds\n            params.spp_pathmaxrxt\n        };\n    }\n};\n\nclass posix_unix_stream_connected_socket_operations : public posix_connected_socket_operations {\npublic:\n    virtual void set_nodelay(file_desc& fd, bool nodelay) const override {\n        SEASTAR_ASSERT(nodelay); // make sure nobody actually tries to use this non-existing functionality\n    }\n    virtual bool get_nodelay(file_desc& fd) const override {\n        return true;\n    }\n    virtual void set_keepalive(file_desc& fd, bool keepalive) const override {}\n    virtual bool get_keepalive(file_desc& fd) const override {\n        return false;\n    }\n    virtual void set_keepalive_parameters(file_desc& fd, const keepalive_params& p) const override {}\n    virtual keepalive_params get_keepalive_parameters(file_desc& fd) const override {\n        return keepalive_params{};\n    }\n};\n\nstatic const posix_connected_socket_operations*\nget_posix_connected_socket_ops(sa_family_t family, int protocol) {\n    static posix_tcp_connected_socket_operations tcp_ops;\n    static posix_sctp_connected_socket_operations sctp_ops;\n    static posix_unix_stream_connected_socket_operations unix_ops;\n    switch (family) {\n    case AF_INET:\n    case AF_INET6:\n        switch (protocol) {\n        case IPPROTO_TCP: return &tcp_ops;\n        case IPPROTO_SCTP: return &sctp_ops;\n        default: abort();\n        }\n    case AF_UNIX:\n        return &unix_ops;\n    default:\n        abort();\n    }\n}\n\nstatic void shutdown_socket_fd(pollable_fd& fd, int how) noexcept {\n    try {\n        // file_desc::shutdown ignores ENOTCONN. Other reasons for exception\n        // EINVAL (wrong \"how\") -- impossible\n        // ENOTSOCK (not a socket) -- incredible\n        // EBADF (invalid file descriptor) -- irretrievable\n        fd.shutdown(how);\n    } catch (...) {\n        on_internal_error(seastar_logger, seastar::format(\"socket shutdown({}, {}) failed: {}\", fd.get_file_desc().fdinfo(), how, std::current_exception()));\n    }\n}\n\nclass posix_connected_socket_impl : public connected_socket_impl {\n    pollable_fd _fd;\n    const posix_connected_socket_operations* _ops;\n    conntrack::handle _handle;\n    std::pmr::polymorphic_allocator<char>* _allocator;\npublic:\n    explicit posix_connected_socket_impl(sa_family_t family, int protocol, pollable_fd fd, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) :\n        _fd(std::move(fd)), _ops(get_posix_connected_socket_ops(family, protocol)), _allocator(allocator) {}\n    explicit posix_connected_socket_impl(sa_family_t family, int protocol, pollable_fd fd, conntrack::handle&& handle,\n        std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator) : _fd(std::move(fd))\n                , _ops(get_posix_connected_socket_ops(family, protocol)), _handle(std::move(handle)), _allocator(allocator) {}\npublic:\n    virtual data_source source() override {\n        return source(connected_socket_input_stream_config());\n    }\n    virtual data_source source(connected_socket_input_stream_config csisc) override {\n        return data_source(std::make_unique<posix_data_source_impl>(_fd, csisc, _allocator));\n    }\n    virtual data_sink sink() override {\n        return data_sink(std::make_unique< posix_data_sink_impl>(_fd));\n    }\n    virtual void shutdown_input() override {\n        shutdown_socket_fd(_fd, SHUT_RD);\n    }\n    virtual void shutdown_output() override {\n        shutdown_socket_fd(_fd, SHUT_WR);\n    }\n    virtual void set_nodelay(bool nodelay) override {\n        return _ops->set_nodelay(_fd.get_file_desc(), nodelay);\n    }\n    virtual bool get_nodelay() const override {\n        return _ops->get_nodelay(_fd.get_file_desc());\n    }\n    void set_keepalive(bool keepalive) override {\n        return _ops->set_keepalive(_fd.get_file_desc(), keepalive);\n    }\n    bool get_keepalive() const override {\n        return _ops->get_keepalive(_fd.get_file_desc());\n    }\n    void set_keepalive_parameters(const keepalive_params& p) override {\n        return _ops->set_keepalive_parameters(_fd.get_file_desc(), p);\n    }\n    keepalive_params get_keepalive_parameters() const override {\n        return _ops->get_keepalive_parameters(_fd.get_file_desc());\n    }\n    void set_sockopt(int level, int optname, const void* data, size_t len) override {\n        return _ops->set_sockopt(_fd.get_file_desc(), level, optname, data, len);\n    }\n    int get_sockopt(int level, int optname, void* data, size_t len) const override {\n        return _ops->get_sockopt(_fd.get_file_desc(), level, optname, data, len);\n    }\n    socket_address local_address() const noexcept override {\n        return _ops->local_address(_fd.get_file_desc());\n    }\n    socket_address remote_address() const noexcept override {\n        return _ops->remote_address(_fd.get_file_desc());\n    }\n    future<> wait_input_shutdown() override {\n        return _fd.poll_rdhup();\n    }\n\n    friend class posix_server_socket_impl;\n    friend class posix_ap_server_socket_impl;\n    friend class posix_reuseport_server_socket_impl;\n    friend class posix_network_stack;\n    friend class posix_ap_network_stack;\n    friend class posix_socket_impl;\n};\n\n// Like posix_connected_socket_impl, but overrides local_address() and remote_address()\n// to return the addresses from the PROXY protocol v2 header.\nclass posix_proxied_connected_socket_impl final : public posix_connected_socket_impl {\n    socket_address _proxy_local_addr;\n    socket_address _proxy_remote_addr;\npublic:\n    posix_proxied_connected_socket_impl(\n            sa_family_t family,\n            int protocol,\n            pollable_fd fd,\n            conntrack::handle&& handle,\n            socket_address proxy_local_addr,\n            socket_address proxy_remote_addr,\n            std::pmr::polymorphic_allocator<char>* allocator = memory::malloc_allocator)\n        : posix_connected_socket_impl(family, protocol, std::move(fd), std::move(handle), allocator)\n        , _proxy_local_addr(proxy_local_addr)\n        , _proxy_remote_addr(proxy_remote_addr) {\n    }\n    virtual socket_address local_address() const noexcept override {\n        return _proxy_local_addr;\n    }\n    virtual socket_address remote_address() const noexcept override {\n        return _proxy_remote_addr;\n    }\n};\n\nstatic void resolve_outgoing_address(socket_address& a) {\n    if (a.family() != AF_INET6\n        || a.as_posix_sockaddr_in6().sin6_scope_id != inet_address::invalid_scope\n        || !IN6_IS_ADDR_LINKLOCAL(&a.as_posix_sockaddr_in6().sin6_addr)\n    ) {\n        return;\n    }\n\n    FILE *f;\n\n    if (!(f = fopen(\"/proc/net/ipv6_route\", \"r\"))) {\n        throw std::system_error(errno, std::system_category(), \"resolve_address\");\n    }\n\n    auto holder = std::unique_ptr<FILE, int(*)(FILE *)>(f, &::fclose);\n\n    /**\n      Here all configured IPv6 routes are shown in a special format. The example displays for loopback interface only. The meaning is shown below (see net/ipv6/route.c for more).\n\n    # cat /proc/net/ipv6_route\n    00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00000001 00200200 lo\n    +------------------------------+ ++ +------------------------------+ ++ +------------------------------+ +------+ +------+ +------+ +------+ ++\n    |                                |  |                                |  |                                |        |        |        |        |\n    1                                2  3                                4  5                                6        7        8        9        10\n\n    1: IPv6 destination network displayed in 32 hexadecimal chars without colons as separator\n\n    2: IPv6 destination prefix length in hexadecimal\n\n    3: IPv6 source network displayed in 32 hexadecimal chars without colons as separator\n\n    4: IPv6 source prefix length in hexadecimal\n\n    5: IPv6 next hop displayed in 32 hexadecimal chars without colons as separator\n\n    6: Metric in hexadecimal\n\n    7: Reference counter\n\n    8: Use counter\n\n    9: Flags\n\n    10: Device name\n\n    */\n\n    uint32_t prefix_len, src_prefix_len;\n    unsigned long flags;\n    char device[16];\n    char dest_str[40];\n\n    for (;;) {\n        auto n = fscanf(f, \"%4s%4s%4s%4s%4s%4s%4s%4s %02x \"\n                            \"%*4s%*4s%*4s%*4s%*4s%*4s%*4s%*4s %02x \"\n                            \"%*4s%*4s%*4s%*4s%*4s%*4s%*4s%*4s \"\n                            \"%*08x %*08x %*08x %08lx %8s\",\n                            &dest_str[0], &dest_str[5], &dest_str[10], &dest_str[15],\n                            &dest_str[20], &dest_str[25], &dest_str[30], &dest_str[35],\n                            &prefix_len,\n                            &src_prefix_len,\n                            &flags, device);\n        if (n != 12) {\n            break;\n        }\n\n        if ((prefix_len > 128)  || (src_prefix_len != 0)\n            || (flags & (RTF_POLICY | RTF_FLOW))\n            || ((flags & RTF_REJECT) && prefix_len == 0) /* reject all */) {\n            continue;\n        }\n\n        dest_str[4] = dest_str[9] = dest_str[14] = dest_str[19] = dest_str[24] = dest_str[29] = dest_str[34] = ':';\n        dest_str[39] = '\\0';\n\n        struct in6_addr addr;\n        if (inet_pton(AF_INET6, dest_str, &addr) < 0) {\n            /* not an Ipv6 address */\n            continue;\n        }\n\n        auto bytes = prefix_len / 8;\n        auto bits = prefix_len % 8;\n\n        auto& src = a.as_posix_sockaddr_in6().sin6_addr;\n\n        if (bytes > 0 && memcmp(&src, &addr, bytes)) {\n            continue;\n        }\n        if (bits > 0) {\n            auto c1 = src.s6_addr[bytes];\n            auto c2 = addr.s6_addr[bytes];\n            auto mask = 0xffu << (8 - bits);\n            if ((c1 & mask) != (c2 & mask)) {\n                continue;\n            }\n        }\n\n        // found the route.\n        for (auto& nif : engine().net().network_interfaces()) {\n            if (nif.name() == device || nif.display_name() == device) {\n                a.as_posix_sockaddr_in6().sin6_scope_id = nif.index();\n                return;\n            }\n        }\n    }\n}\n\nclass posix_socket_impl final : public socket_impl {\n    pollable_fd _fd;\n    std::pmr::polymorphic_allocator<char>* _allocator;\n    bool _reuseaddr = false;\n    const int _sock_flags;\n\n    future<> find_port_and_connect(socket_address sa, socket_address local, transport proto = transport::TCP) {\n        static thread_local std::default_random_engine random_engine{std::random_device{}()};\n        static thread_local std::uniform_int_distribution<uint16_t> u(49152/smp::count + 1, 65535/smp::count - 1);\n        // If no explicit local address, set to dest address family wildcard.\n        if (local.is_unspecified()) {\n            local = net::inet_address(sa.addr().in_family());\n        }\n        resolve_outgoing_address(sa);\n        return repeat([this, sa, local, proto, attempts = 0, requested_port = ntoh(local.as_posix_sockaddr_in().sin_port)] () mutable {\n            _fd = file_desc::socket(sa.u.sa.sa_family, _sock_flags, int(proto));\n            _fd.get_file_desc().setsockopt(SOL_SOCKET, SO_REUSEADDR, int(_reuseaddr));\n            uint16_t port = attempts++ < 5 && requested_port == 0 && proto == transport::TCP ? u(random_engine) * smp::count + this_shard_id() : requested_port;\n            local.as_posix_sockaddr_in().sin_port = hton(port);\n            return internal::posix_connect(_fd, sa, local).then_wrapped([port, requested_port] (future<> f) {\n                try {\n                    f.get();\n                    return stop_iteration::yes;\n                } catch (std::system_error& err) {\n                    if (port != requested_port && (err.code().value() == EADDRINUSE || err.code().value() == EADDRNOTAVAIL)) {\n                        return stop_iteration::no;\n                    }\n                    throw;\n                }\n            });\n        });\n    }\n\n    /// an aux function to handle unix-domain-specific requests\n    future<connected_socket> connect_unix_domain(socket_address sa, socket_address local) {\n        // note that if the 'local' address was not set by the client, it was created as an undefined address\n        if (local.is_unspecified()) {\n            local = socket_address{unix_domain_addr{std::string{}}};\n        }\n\n        _fd = file_desc::socket(sa.u.sa.sa_family, _sock_flags, 0);\n        return internal::posix_connect(_fd, sa, local).then(\n            [fd = _fd, allocator = _allocator](){\n                // a problem with 'private' interaction with 'unique_ptr'\n                std::unique_ptr<connected_socket_impl> csi(new posix_connected_socket_impl{AF_UNIX, 0, std::move(fd), allocator});\n                return make_ready_future<connected_socket>(connected_socket(std::move(csi)));\n            }\n        );\n    }\n\npublic:\n    posix_socket_impl(bool need_nonblock, std::pmr::polymorphic_allocator<char>* allocator=memory::malloc_allocator)\n            : _allocator(allocator)\n            , _sock_flags(SOCK_STREAM | SOCK_CLOEXEC | (need_nonblock ? SOCK_NONBLOCK : 0))\n    {}\n\n    virtual future<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) override {\n        if (sa.is_af_unix()) {\n            return connect_unix_domain(sa, local);\n        }\n        return find_port_and_connect(sa, local, proto).then([this, sa, proto, allocator = _allocator] () mutable {\n            std::unique_ptr<connected_socket_impl> csi(new posix_connected_socket_impl(sa.family(), static_cast<int>(proto), _fd, allocator));\n            return make_ready_future<connected_socket>(connected_socket(std::move(csi)));\n        });\n    }\n\n    void set_reuseaddr(bool reuseaddr) override {\n        _reuseaddr = reuseaddr;\n        if (_fd) {\n            _fd.get_file_desc().setsockopt(SOL_SOCKET, SO_REUSEADDR, int(reuseaddr));\n        }\n    }\n\n    bool get_reuseaddr() const override {\n        if(_fd) {\n            return _fd.get_file_desc().getsockopt<int>(SOL_SOCKET, SO_REUSEADDR);\n        } else {\n            return _reuseaddr;\n        }\n    }\n\n    virtual void shutdown() override {\n        if (_fd) {\n            try {\n                _fd.shutdown(SHUT_RDWR);\n            } catch (std::system_error& e) {\n                if (e.code().value() != ENOTCONN) {\n                    throw;\n                }\n            }\n        }\n    }\n};\n\nstatic\nunsigned\nget_port_or_counter(const socket_address& sa) {\n    static thread_local constinit unsigned counter = 0;\n    switch (sa.family()) {\n    case AF_INET:\n        return ntoh(sa.as_posix_sockaddr_in().sin_port);\n    case AF_INET6:\n        return ntoh(sa.as_posix_sockaddr_in6().sin6_port);\n    default:\n        // AF_UNIX? Shouldn't happen but not worth raising an error for it\n        return counter++;\n    }\n}\n\nstatic\nproxy_data\nlocal_proxy_data(const pollable_fd& pfd) {\n    auto& fd = pfd.get_file_desc();\n\n    auto local_sa = fd.get_address();\n    auto remote_sa = fd.get_remote_address();\n\n    proxy_data header = {\n        .remote_address = remote_sa,\n        .local_address = local_sa,\n    };\n\n    return header;\n}\n\n// Parses proxy protocol v2 header; returns std::nullopt if no valid header is found.\nstatic\nfuture<std::optional<proxy_data>>\nread_proxy_data(pollable_fd& fd) {\n    constexpr size_t pp2_header_len = 16;\n    char header_buf[pp2_header_len];\n    auto n_read = co_await fd.read_some(header_buf, pp2_header_len);\n    if (n_read < pp2_header_len) {\n        co_return std::nullopt;\n    }\n    static const char pp2_signature[12] = {\n        0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a\n    };\n    if (std::memcmp(header_buf, pp2_signature, sizeof(pp2_signature)) != 0) {\n        co_return std::nullopt;\n    }\n\n    auto len = read_be<uint16_t>(header_buf + 14);\n\n    char stack_buffer[36]; // Suitable for IPv6 without extra TLVs\n    std::unique_ptr<char[]> heap_buffer;\n    auto* buffer = stack_buffer;\n\n    if (len > sizeof(stack_buffer)) {\n        heap_buffer = std::make_unique<char[]>(len);\n        buffer = heap_buffer.get();\n    }\n\n    auto xlen = co_await fd.read_some(buffer, len);\n    if (xlen < len) {\n        co_return std::nullopt;\n    }\n\n    uint8_t fam_proto = header_buf[13];\n    switch (header_buf[12]) { // version and command\n    case 0x20: // v2, LOCAL\n        if (fam_proto != 0x00) { // UNSPEC\n            co_return std::nullopt;\n        }\n        co_return local_proxy_data(fd);\n    case 0x21: // v2, PROXY\n        // Mainline continues after the switch\n        break;\n    default:   // Not defined, must reject\n        co_return std::nullopt;\n    }\n\n    auto fam = fam_proto >> 4;\n    auto proto = fam_proto & 0x0F;\n\n    // We could, in principle, support DGRAM here, but there's no\n    // real need for it. Real-world proxies support proxying UDP as UDP.\n    if (proto != 0x1) { // STREAM\n        co_return std::nullopt;\n    }\n\n    proxy_data addr_data;\n\n    switch (fam) {\n    case 0x1: { // INET\n        if (len < 12) {\n            co_return std::nullopt;\n        }\n        addr_data.remote_address = ipv4_addr(inet_address(copy_reinterpret_cast<in_addr>(buffer)), read_be<uint16_t>(buffer + 8));\n        addr_data.local_address = ipv4_addr(inet_address(copy_reinterpret_cast<in_addr>(buffer + 4)), read_be<uint16_t>(buffer + 10));\n        break;\n    }\n    case 0x2: { // INET6\n        if (len < 36) {\n            co_return std::nullopt;\n        }\n        addr_data.remote_address = ipv6_addr(inet_address(copy_reinterpret_cast<in6_addr>(buffer)), read_be<uint16_t>(buffer + 32));\n        addr_data.local_address = ipv6_addr(inet_address(copy_reinterpret_cast<in6_addr>(buffer + 16)), read_be<uint16_t>(buffer + 34));\n        break;\n    }\n    default:\n        co_return std::nullopt;\n    }\n    co_return addr_data;\n}\n\nstatic\nstd::unique_ptr<connected_socket_impl>\nmake_maybe_proxied_connected_socket_impl(\n        sa_family_t family,\n        int protocol,\n        pollable_fd fd,\n        conntrack::handle&& handle,\n        std::optional<proxy_data> addr_data_opt,\n        std::pmr::polymorphic_allocator<char>* allocator = memory::malloc_allocator) {\n    if (addr_data_opt) {\n        return std::make_unique<posix_proxied_connected_socket_impl>(\n            family,\n            protocol,\n            std::move(fd),\n            std::move(handle),\n            std::move(addr_data_opt->local_address),\n            std::move(addr_data_opt->remote_address),\n            allocator);\n    } else {\n        return std::make_unique<posix_connected_socket_impl>(\n            family,\n            protocol,\n            std::move(fd),\n            std::move(handle),\n            allocator);\n    }\n}\n\nfuture<accept_result>\nposix_server_socket_impl::accept() {\n    while (true) { // exited via co_return\n        auto [fd, sa] = co_await _lfd.accept();\n\n        std::optional<proxy_data> addr_data_opt;\n        if (_proxy_protocol) {\n            addr_data_opt = co_await read_proxy_data(fd);\n            if (!addr_data_opt) {\n                continue; // drop the connection\n            }\n            sa = addr_data_opt->remote_address;\n        }\n\n        auto cth = conntrack::handle();\n        switch(_lba) {\n        case server_socket::load_balancing_algorithm::connection_distribution:\n            cth = _conntrack.get_handle();\n            break;\n        case server_socket::load_balancing_algorithm::port:\n            cth = _conntrack.get_handle(get_port_or_counter(sa) % smp::count);\n            break;\n        case server_socket::load_balancing_algorithm::fixed:\n            cth = _conntrack.get_handle(_fixed_cpu);\n            break;\n        default: abort();\n        }\n\n        auto cpu = cth.cpu();\n        if (cpu == this_shard_id()) {\n            auto csi = make_maybe_proxied_connected_socket_impl(\n                sa.family(),\n                _protocol,\n                std::move(fd),\n                std::move(cth),\n                std::move(addr_data_opt),\n                _allocator);\n            co_return accept_result{connected_socket(std::move(csi)), sa};\n        } else {\n            // FIXME: future is discarded\n            (void)smp::submit_to(cpu, [protocol = _protocol, ssa = _sa, fd = std::move(fd.get_file_desc()), sa, cth = std::move(cth), addr_data_opt = std::move(addr_data_opt), allocator = _allocator] () mutable {\n                posix_ap_server_socket_impl::move_connected_socket(protocol, ssa, pollable_fd(std::move(fd)), sa, std::move(cth), std::move(addr_data_opt), allocator);\n            });\n        }\n    };\n}\n\nvoid\nposix_server_socket_impl::abort_accept() {\n    _lfd.shutdown(SHUT_RD, pollable_fd::shutdown_kernel_only::no);\n}\n\nsocket_address posix_server_socket_impl::local_address() const {\n    return _lfd.get_file_desc().get_address();\n}\n\nposix_ap_server_socket_impl::posix_ap_server_socket_impl(int protocol, socket_address sa, std::pmr::polymorphic_allocator<char>* allocator)\n        : _protocol(protocol), _sa(sa), _allocator(allocator)\n{\n    auto it = ports.emplace(std::make_tuple(_protocol, _sa));\n    if (!it.second) {\n        throw std::system_error(EADDRINUSE, std::system_category());\n    }\n}\n\nposix_ap_server_socket_impl::~posix_ap_server_socket_impl() {\n    ports.erase(std::make_tuple(_protocol, _sa));\n}\n\nfuture<accept_result> posix_ap_server_socket_impl::accept() {\n    auto t_sa = std::make_tuple(_protocol, _sa);\n    auto conni = conn_q.find(t_sa);\n    if (conni != conn_q.end()) {\n        connection c = std::move(conni->second);\n        conn_q.erase(conni);\n        try {\n            std::unique_ptr<connected_socket_impl> csi(\n                    new posix_connected_socket_impl(_sa.family(), _protocol, std::move(c.fd), std::move(c.connection_tracking_handle), _allocator));\n            return make_ready_future<accept_result>(accept_result{connected_socket(std::move(csi)), std::move(c.addr)});\n        } catch (...) {\n            return make_exception_future<accept_result>(std::current_exception());\n        }\n    } else {\n        try {\n            auto i = sockets.emplace(std::piecewise_construct, std::make_tuple(t_sa), std::make_tuple());\n            SEASTAR_ASSERT(i.second);\n            return i.first->second.get_future();\n        } catch (...) {\n            return make_exception_future<accept_result>(std::current_exception());\n        }\n    }\n}\n\nvoid\nposix_ap_server_socket_impl::abort_accept() {\n    auto t_sa = std::make_tuple(_protocol, _sa);\n    conn_q.erase(t_sa);\n    auto i = sockets.find(t_sa);\n    if (i != sockets.end()) {\n        i->second.set_exception(std::system_error(ECONNABORTED, std::system_category()));\n        sockets.erase(i);\n    }\n}\n\nfuture<accept_result>\nposix_reuseport_server_socket_impl::accept() {\n    return _lfd.accept().then_unpack([allocator = _allocator, protocol = _protocol] (pollable_fd fd, socket_address sa) {\n        std::unique_ptr<connected_socket_impl> csi(\n                new posix_connected_socket_impl(sa.family(), protocol, std::move(fd), allocator));\n        return make_ready_future<accept_result>(\n            accept_result{connected_socket(std::move(csi)), sa});\n    });\n}\n\nvoid\nposix_reuseport_server_socket_impl::abort_accept() {\n    _lfd.shutdown(SHUT_RD, pollable_fd::shutdown_kernel_only::no);\n}\n\nsocket_address posix_reuseport_server_socket_impl::local_address() const {\n    return _lfd.get_file_desc().get_address();\n}\n\nvoid\nposix_ap_server_socket_impl::move_connected_socket(int protocol, socket_address sa, pollable_fd fd, socket_address addr, conntrack::handle cth, std::optional<proxy_data> addr_data_opt, std::pmr::polymorphic_allocator<char>* allocator) {\n    auto t_sa = std::make_tuple(protocol, sa);\n    auto i = sockets.find(t_sa);\n    if (i != sockets.end()) {\n        try {\n            auto csi = make_maybe_proxied_connected_socket_impl(\n                sa.family(),\n                protocol,\n                std::move(fd),\n                std::move(cth),\n                std::move(addr_data_opt),\n                allocator);\n            i->second.set_value(accept_result{connected_socket(std::move(csi)), std::move(addr)});\n        } catch (...) {\n            i->second.set_exception(std::current_exception());\n        }\n        sockets.erase(i);\n    } else {\n        conn_q.emplace(std::piecewise_construct, std::make_tuple(t_sa), std::make_tuple(std::move(fd), std::move(addr), std::move(cth), std::move(addr_data_opt)));\n    }\n}\n\nfuture<temporary_buffer<char>>\nposix_data_source_impl::get() {\n    return _fd.recv_some(static_cast<internal::buffer_allocator*>(this)).then([this] (temporary_buffer<char> b) {\n        if (b.size() >= _config.buffer_size) {\n            _config.buffer_size *= 2;\n            _config.buffer_size = std::min(_config.buffer_size, _config.max_buffer_size);\n        } else if (b.size() <= _config.buffer_size / 4) {\n            _config.buffer_size /= 2;\n            _config.buffer_size = std::max(_config.buffer_size, _config.min_buffer_size);\n        }\n        auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n        bytes_received[sg_id] += b.size();\n        return b;\n    });\n}\n\ntemporary_buffer<char>\nposix_data_source_impl::allocate_buffer() {\n    return make_temporary_buffer<char>(_buffer_allocator, _config.buffer_size);\n}\n\nfuture<> posix_data_source_impl::close() {\n    _fd.shutdown(SHUT_RD);\n    return make_ready_future<>();\n}\n\n#if SEASTAR_API_LEVEL >= 9\nfuture<> posix_data_sink_impl::put(std::span<temporary_buffer<char>> bufs) {\n    auto [ total, del ] = _vecs.populate(bufs);\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    bytes_sent[sg_id] += total;\n    return _fd.write_all(_vecs.v).finally([del = std::move(del)] {} );\n}\n#else\nfuture<>\nposix_data_sink_impl::put(temporary_buffer<char> buf) {\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    bytes_sent[sg_id] += buf.size();\n    return _fd.write_all(buf.get(), buf.size()).then([d = buf.release()] {});\n}\n\nfuture<>\nposix_data_sink_impl::put(packet p) {\n    SEASTAR_ASSERT(!_p);\n    _p = std::move(p);\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    bytes_sent[sg_id] += _p.len();\n    return _fd.write_all(_p).finally([this] { _p.reset(); });\n}\n#endif\n\nfuture<>\nposix_data_sink_impl::close() {\n    _fd.shutdown(SHUT_WR);\n    return make_ready_future<>();\n}\n\nvoid posix_data_sink_impl::on_batch_flush_error() noexcept {\n    shutdown_socket_fd(_fd, SHUT_RD);\n}\n\nposix_network_stack::posix_network_stack(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator)\n        : _reuseport(engine().posix_reuseport_available())\n        , _sock_need_nonblock(engine().posix_sock_need_nonblock())\n        , _allocator(allocator)\n{\n}\n\nserver_socket\nposix_network_stack::listen(socket_address sa, listen_options opt) {\n    using server_socket = seastar::server_socket;\n    // allow unspecified bind address -> default to ipv4 wildcard\n    if (sa.is_unspecified()) {\n        sa = inet_address(inet_address::family::INET);\n    }\n    if (sa.is_af_unix()) {\n        return server_socket(std::make_unique<posix_server_socket_impl>(0, sa, internal::posix_listen(sa, opt), opt.lba, opt.fixed_cpu, opt.proxy_protocol, _allocator));\n    }\n    auto protocol = static_cast<int>(opt.proto);\n    return _reuseport ?\n        server_socket(std::make_unique<posix_reuseport_server_socket_impl>(protocol, sa, internal::posix_listen(sa, opt), _allocator))\n        :\n        server_socket(std::make_unique<posix_server_socket_impl>(protocol, sa, internal::posix_listen(sa, opt), opt.lba, opt.fixed_cpu, opt.proxy_protocol, _allocator));\n}\n\n::seastar::socket posix_network_stack::socket() {\n    return ::seastar::socket(std::make_unique<posix_socket_impl>(_sock_need_nonblock, _allocator));\n}\n\nposix_ap_network_stack::posix_ap_network_stack(const program_options::option_group& opts, std::pmr::polymorphic_allocator<char>* allocator)\n        : posix_network_stack(opts, allocator)\n{\n}\n\nserver_socket\nposix_ap_network_stack::listen(socket_address sa, listen_options opt) {\n    using server_socket = seastar::server_socket;\n    // allow unspecified bind address -> default to ipv4 wildcard\n    if (sa.is_unspecified()) {\n        sa = inet_address(inet_address::family::INET);\n    }\n    if (sa.is_af_unix()) {\n        return server_socket(std::make_unique<posix_ap_server_socket_impl>(0, sa, _allocator));\n    }\n    auto protocol = static_cast<int>(opt.proto);\n    return posix_network_stack::_reuseport ?\n        server_socket(std::make_unique<posix_reuseport_server_socket_impl>(protocol, sa, internal::posix_listen(sa, opt), _allocator))\n        :\n        server_socket(std::make_unique<posix_ap_server_socket_impl>(protocol, sa, _allocator));\n}\n\nstruct cmsg_with_pktinfo {\n    struct cmsghdrcmh;\n    union {\n        struct in_pktinfo pktinfo;\n        struct in6_pktinfo pkt6info;\n    };\n};\n\nclass posix_datagram_channel : public datagram_channel_impl {\nprivate:\n    static constexpr int MAX_DATAGRAM_SIZE = 65507;\n    struct recv_ctx {\n        struct msghdr _hdr;\n        struct iovec _iov;\n        socket_address _src_addr;\n        char* _buffer;\n        cmsg_with_pktinfo _cmsg;\n\n        recv_ctx(bool use_pktinfo) {\n            memset(&_hdr, 0, sizeof(_hdr));\n            _hdr.msg_iov = &_iov;\n            _hdr.msg_iovlen = 1;\n            _hdr.msg_name = &_src_addr.u.sa;\n            _hdr.msg_namelen = sizeof(_src_addr.u.sas);\n\n            if (use_pktinfo) {\n                memset(&_cmsg, 0, sizeof(_cmsg));\n                _hdr.msg_control = &_cmsg;\n                _hdr.msg_controllen = sizeof(_cmsg);\n            } else {\n                _hdr.msg_control = nullptr;\n                _hdr.msg_controllen = 0;\n            }\n        }\n\n        recv_ctx(const recv_ctx&) = delete;\n        recv_ctx(recv_ctx&&) = delete;\n\n        void prepare() {\n            _buffer = new char[MAX_DATAGRAM_SIZE];\n            _iov.iov_base = _buffer;\n            _iov.iov_len = MAX_DATAGRAM_SIZE;\n        }\n    };\n    struct send_ctx {\n        struct msghdr _hdr;\n        internal::wrapped_iovecs _vecs;\n        socket_address _dst;\n\n        send_ctx() {\n            memset(&_hdr, 0, sizeof(_hdr));\n            _hdr.msg_name = &_dst.u.sa;\n            _hdr.msg_namelen = _dst.addr_length;\n        }\n\n        send_ctx(const send_ctx&) = delete;\n        send_ctx(send_ctx&&) = delete;\n\n        auto prepare(const socket_address& dst, std::span<temporary_buffer<char>> bufs) {\n            _dst = dst;\n            _hdr.msg_namelen = _dst.addr_length;\n            auto ret = _vecs.populate(bufs);\n            _hdr.msg_iov = _vecs.v.data();\n            _hdr.msg_iovlen = _vecs.v.size();\n            resolve_outgoing_address(_dst);\n            return ret;\n        }\n    };\n\n    static bool is_inet(sa_family_t family) {\n        return family == AF_INET || family == AF_INET6;\n    }\n\n    static file_desc create_socket(sa_family_t family) {\n        file_desc fd = file_desc::socket(family, SOCK_DGRAM | SOCK_NONBLOCK | SOCK_CLOEXEC, 0);\n\n        if (is_inet(family)) {\n            fd.setsockopt(SOL_IP, IP_PKTINFO, true);\n            if (engine().posix_reuseport_available()) {\n                fd.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1);\n            }\n        }\n\n        return fd;\n    }\n\n    pollable_fd _fd;\n    socket_address _address;\n    recv_ctx _recv;\n    send_ctx _send;\n    bool _closed;\npublic:\n    /// Creates a channel that is not bound to any socket address. The channel\n    /// can be used to communicate with adressess that belong to the \\param\n    /// family.\n    posix_datagram_channel(sa_family_t family)\n        : _recv(is_inet(family)), _closed(false) {\n        auto fd = create_socket(family);\n\n        _address = fd.get_address();\n        _fd = std::move(fd);\n    }\n\n    /// Creates a channel that is bound to the specified local address. It can be used to\n    /// communicate with addresses that belong to the family of \\param local.\n    posix_datagram_channel(socket_address local)\n        : _recv(is_inet(local.family())), _closed(false) {\n        auto fd = create_socket(local.family());\n        fd.bind(local.u.sa, local.addr_length);\n\n        _address = fd.get_address();\n        _fd = std::move(fd);\n    }\n\n    virtual ~posix_datagram_channel() { if (!_closed) close(); };\n    virtual future<datagram> receive() override;\n    virtual future<> send(const socket_address& dst, const char *msg) override;\n    virtual future<> send(const socket_address& dst, std::span<temporary_buffer<char>> bufs) override;\n    virtual void shutdown_input() override {\n        _fd.shutdown(SHUT_RD, pollable_fd::shutdown_kernel_only::no);\n    }\n    virtual void shutdown_output() override {\n        _fd.shutdown(SHUT_WR, pollable_fd::shutdown_kernel_only::no);\n    }\n    virtual void close() override {\n        _closed = true;\n        _fd = {};\n    }\n    virtual bool is_closed() const override { return _closed; }\n    socket_address local_address() const override {\n        SEASTAR_ASSERT(_address.u.sas.ss_family != AF_INET6 || (_address.addr_length > 20));\n        return _address;\n    }\n};\n\nfuture<> posix_datagram_channel::send(const socket_address& dst, const char *message) {\n    auto len = strlen(message);\n    auto a = dst;\n    resolve_outgoing_address(a);\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    bytes_sent[sg_id] += len;\n    return _fd.sendto(a, message, len)\n            .then([len] (size_t size) { SEASTAR_ASSERT(size == len); });\n}\n\nfuture<> posix_datagram_channel::send(const socket_address& dst, std::span<temporary_buffer<char>> bufs) {\n    auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n    auto [ len, del ] = _send.prepare(dst, bufs);\n    bytes_sent[sg_id] += len;\n    return _fd.sendmsg(&_send._hdr)\n            .then([len, del = std::move(del) ] (size_t size) { SEASTAR_ASSERT(size == len); });\n}\n\nudp_channel\nposix_network_stack::make_udp_channel(const socket_address& addr) {\n    if (!addr.is_unspecified()) {\n        return make_bound_datagram_channel(addr);\n    } else {\n        // Preserve the default behavior of make_udp_channel({}) which is to\n        // create an unbound channel that can be used to send UDP datagrams.\n        return make_unbound_datagram_channel(AF_INET);\n    }\n}\n\ndatagram_channel\nposix_network_stack::make_unbound_datagram_channel(sa_family_t family) {\n    return datagram_channel(std::make_unique<posix_datagram_channel>(family));\n}\n\ndatagram_channel\nposix_network_stack::make_bound_datagram_channel(const socket_address& local) {\n    return datagram_channel(std::make_unique<posix_datagram_channel>(local));\n}\n\nbool\nposix_network_stack::supports_ipv6() const {\n    static bool has_ipv6 = [] {\n        try {\n            posix_datagram_channel c(ipv6_addr{\"::1\"});\n            c.close();\n            return true;\n        } catch (...) {}\n        return false;\n    }();\n\n    return has_ipv6;\n}\n\nclass posix_datagram : public datagram_impl {\nprivate:\n    socket_address _src;\n    socket_address _dst;\n    temporary_buffer<char> _buf;\npublic:\n    posix_datagram(const socket_address& src, const socket_address& dst, temporary_buffer<char> b) : _src(src), _dst(dst), _buf(std::move(b)) {}\n    virtual socket_address get_src() override { return _src; }\n    virtual socket_address get_dst() override { return _dst; }\n    virtual uint16_t get_dst_port() override {\n        if (_dst.family() != AF_INET && _dst.family() != AF_INET6) {\n            throw std::runtime_error(format(\"get_dst_port() called on non-IP address: {}\", _dst));\n        }\n        return _dst.port();\n    }\n    virtual std::span<temporary_buffer<char>> get_buffers() override { return std::span(&_buf, 1); }\n};\n\nfuture<datagram>\nposix_datagram_channel::receive() {\n    _recv.prepare();\n    return _fd.recvmsg(&_recv._hdr).then([this] (size_t size) {\n        std::optional<socket_address> dst;\n        for (auto* cmsg = CMSG_FIRSTHDR(&_recv._hdr); cmsg != nullptr; cmsg = CMSG_NXTHDR(&_recv._hdr, cmsg)) {\n            if (cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_PKTINFO) {\n                dst = ipv4_addr(copy_reinterpret_cast<in_pktinfo>(CMSG_DATA(cmsg)).ipi_addr, _address.port());\n                break;\n            } else if (cmsg->cmsg_level == IPPROTO_IPV6 && cmsg->cmsg_type == IPV6_PKTINFO) {\n                dst = ipv6_addr(copy_reinterpret_cast<in6_pktinfo>(CMSG_DATA(cmsg)).ipi6_addr, _address.port());\n                break;\n            }\n        }\n        auto sg_id = internal::scheduling_group_index(current_scheduling_group());\n        bytes_received[sg_id] += size;\n        return make_ready_future<datagram>(datagram(std::make_unique<posix_datagram>(\n            _recv._src_addr, dst ? *dst : _address, temporary_buffer<char>(_recv._buffer, size, make_deleter([buf = _recv._buffer] { delete[] buf; })))));\n    }).handle_exception([p = _recv._buffer](auto ep) {\n        delete[] p;\n        return make_exception_future<datagram>(std::move(ep));\n    });\n}\n\nnetwork_stack_entry register_posix_stack() {\n    return network_stack_entry{\n        \"posix\", std::make_unique<program_options::option_group>(nullptr, \"Posix\"),\n        [](const program_options::option_group& ops) {\n            return smp::main_thread() ? posix_network_stack::create(ops)\n                                      : posix_ap_network_stack::create(ops);\n        },\n        true};\n}\n\n// nw interface stuff\n\nstd::vector<network_interface> posix_network_stack::network_interfaces() {\n    class posix_network_interface_impl final : public network_interface_impl {\n    public:\n        uint32_t _index = 0, _mtu = 0;\n        sstring _name, _display_name;\n        std::vector<net::inet_address> _addresses;\n        std::vector<uint8_t> _hardware_address;\n        bool _loopback = false, _virtual = false, _up = false;\n\n        uint32_t index() const override {\n            return _index;\n        }\n        uint32_t mtu() const override {\n            return _mtu;\n        }\n        const sstring& name() const override {\n            return _name;\n        }\n        const sstring& display_name() const override {\n            return _display_name.empty() ? name() : _display_name;\n        }\n        const std::vector<net::inet_address>& addresses() const override {\n            return _addresses;\n        }\n        const std::vector<uint8_t> hardware_address() const override {\n            return _hardware_address;\n        }\n        bool is_loopback() const override {\n            return _loopback;\n        }\n        bool is_virtual() const override {\n            return _virtual;\n        }\n        bool is_up() const override {\n            // TODO: should be checked on query?\n            return _up;\n        }\n        bool supports_ipv6() const override {\n            // TODO: this is not 100% correct.\n            return std::any_of(_addresses.begin(), _addresses.end(), std::mem_fn(&inet_address::is_ipv6));\n        }\n    };\n\n    // For now, keep an immutable set of interfaces created on start, shared across\n    // shards\n    static const std::vector<posix_network_interface_impl> global_interfaces = [] {\n        auto fd = ::socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);\n        throw_system_error_on(fd < 0, \"could not open netlink socket\");\n\n        std::unique_ptr<int, void(*)(int*)> fd_guard(&fd, [](int* p) { ::close(*p); });\n\n        auto pid = ::getpid();\n\n        sockaddr_nl local = {\n          .nl_family = AF_NETLINK,\n          .nl_pid = static_cast<unsigned int>(pid),\n          .nl_groups = RTMGRP_IPV6_IFADDR|RTMGRP_IPV4_IFADDR,\n        };\n\n        throw_system_error_on(bind(fd, (struct sockaddr *) &local, sizeof(local)) < 0, \"could not bind netlink socket\");\n\n        /* RTNL socket is ready for use, prepare and send requests */\n\n        std::vector<posix_network_interface_impl> res;\n\n        for (auto msg : { RTM_GETLINK, RTM_GETADDR}) {\n            struct nl_req {\n                nlmsghdr hdr;\n                union {\n                    rtgenmsg gen;\n                    ifaddrmsg addr;\n                };\n            } req = {};\n\n            req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));\n            req.hdr.nlmsg_type = msg;\n            req.hdr.nlmsg_flags = NLM_F_REQUEST | NLM_F_ROOT;\n            req.hdr.nlmsg_seq = 1;\n            req.hdr.nlmsg_pid = pid;\n\n            if (msg == RTM_GETLINK) {\n                req.gen.rtgen_family = AF_PACKET; /*  no preferred AF, we will get *all* interfaces */\n            } else {\n                req.addr.ifa_family = AF_UNSPEC;\n            }\n\n            sockaddr_nl kernel = {\n              .nl_family = AF_NETLINK, /* fill-in kernel address (destination of our message) */\n            };\n            iovec io = {\n              .iov_base = &req,\n              .iov_len = req.hdr.nlmsg_len,\n            };\n            msghdr rtnl_msg = {\n              .msg_name = &kernel,\n              .msg_namelen = sizeof(kernel),\n              .msg_iov = &io,\n              .msg_iovlen = 1,\n            };\n\n            throw_system_error_on(::sendmsg(fd, (struct msghdr *) &rtnl_msg, 0) < 0, \"could not send netlink request\");\n            /* parse reply */\n\n            constexpr size_t reply_buffer_size = 8192;\n            char reply[reply_buffer_size];\n\n            bool done = false;\n\n            while (!done) {\n                iovec io_reply = {\n                  .iov_base = reply,\n                  .iov_len = reply_buffer_size,\n                };\n                msghdr rtnl_reply = {\n                  .msg_name = &kernel,\n                  .msg_namelen = sizeof(kernel),\n                  .msg_iov = &io_reply,\n                  .msg_iovlen = 1,\n                };\n\n                auto len = ::recvmsg(fd, &rtnl_reply, 0); /* read as much data as fits in the receive buffer */\n                if (len <= 0) {\n                    return res;\n                }\n\n                for (auto* msg_ptr = (struct nlmsghdr *) reply; NLMSG_OK(msg_ptr, len); msg_ptr = NLMSG_NEXT(msg_ptr, len)) {\n                    switch(msg_ptr->nlmsg_type) {\n                    case NLMSG_DONE: // that is all\n                        done = true;\n                        break;\n                    case RTM_NEWLINK:\n                    {\n                        auto* iface = reinterpret_cast<const ifinfomsg*>(NLMSG_DATA(msg_ptr));\n                        auto ilen = msg_ptr->nlmsg_len - NLMSG_LENGTH(sizeof(ifinfomsg));\n\n                        // todo: filter any non-network interfaces (family)\n\n                        posix_network_interface_impl nwif;\n\n                        nwif._index = iface->ifi_index;\n                        nwif._loopback = (iface->ifi_flags & IFF_LOOPBACK) != 0;\n                        nwif._up = (iface->ifi_flags & IFF_UP) != 0;\n    #if defined(IFF_802_1Q_VLAN) && defined(IFF_EBRIDGE) && defined(IFF_SLAVE_INACTIVE)\n                        nwif._virtual = (iface->ifi_flags & (IFF_802_1Q_VLAN|IFF_EBRIDGE|IFF_SLAVE_INACTIVE)) != 0;\n    #endif\n                        for (auto* attribute = IFLA_RTA(iface); RTA_OK(attribute, ilen); attribute = RTA_NEXT(attribute, ilen)) {\n                            switch(attribute->rta_type) {\n                            case IFLA_IFNAME:\n                                nwif._name = reinterpret_cast<const char *>(RTA_DATA(attribute));\n                                break;\n                            case IFLA_MTU:\n                                nwif._mtu = *reinterpret_cast<const uint32_t *>(RTA_DATA(attribute));\n                                break;\n                            case IFLA_ADDRESS:\n                                nwif._hardware_address.assign(reinterpret_cast<const uint8_t *>(RTA_DATA(attribute)), reinterpret_cast<const uint8_t *>(RTA_DATA(attribute)) + RTA_PAYLOAD(attribute));\n                                break;\n                            default:\n                                break;\n                            }\n                        }\n\n                        res.emplace_back(std::move(nwif));\n\n                        break;\n                    }\n                    case RTM_NEWADDR:\n                    {\n                        auto* addr = reinterpret_cast<const ifaddrmsg*>(NLMSG_DATA(msg_ptr));\n                        auto ilen = msg_ptr->nlmsg_len - NLMSG_LENGTH(sizeof(ifaddrmsg));\n\n                        for (auto& nwif : res) {\n                            if (nwif._index == addr->ifa_index) {\n                                for (auto* attribute = IFA_RTA(addr); RTA_OK(attribute, ilen); attribute = RTA_NEXT(attribute, ilen)) {\n                                    std::optional<inet_address> ia;\n\n                                    switch(attribute->rta_type) {\n                                    case IFA_LOCAL:\n                                    case IFA_ADDRESS: // ipv6 addresses are reported only as \"ADDRESS\"\n\n                                        if (RTA_PAYLOAD(attribute) == sizeof(::in_addr)) {\n                                            ia.emplace(*reinterpret_cast<const ::in_addr *>(RTA_DATA(attribute)));\n                                        } else if (RTA_PAYLOAD(attribute) == sizeof(::in6_addr)) {\n                                            ia.emplace(*reinterpret_cast<const ::in6_addr *>(RTA_DATA(attribute)), nwif.index());\n                                        }\n\n                                        if (ia && std::find(nwif._addresses.begin(), nwif._addresses.end(), *ia) == nwif._addresses.end()) {\n                                            nwif._addresses.emplace_back(*ia);\n                                        }\n\n                                        break;\n                                    default:\n                                        break;\n                                    }\n                                }\n\n                                break;\n                            }\n                        }\n\n                        break;\n                    }\n                    default:\n                        break;\n                    }\n                }\n            }\n        }\n\n        return res;\n    }();\n\n    // And a similarly immutable set of shared_ptr to network_interface_impl per shard, ready\n    // to be handed out to callers with minimal overhead\n    static const thread_local std::vector<shared_ptr<posix_network_interface_impl>> thread_local_interfaces = [] {\n        std::vector<shared_ptr<posix_network_interface_impl>> res;\n        res.reserve(global_interfaces.size());\n        std::transform(global_interfaces.begin(), global_interfaces.end(), std::back_inserter(res), [](const posix_network_interface_impl& impl) {\n            return make_shared<posix_network_interface_impl>(impl);\n        });\n        return res;\n    }();\n\n    return std::vector<network_interface>(thread_local_interfaces.begin(), thread_local_interfaces.end());\n}\n\nstatistics posix_network_stack::stats(unsigned scheduling_group_id) {\n    return statistics{\n        bytes_sent[scheduling_group_id],\n        bytes_received[scheduling_group_id],\n    };\n}\n\nvoid posix_network_stack::clear_stats(unsigned scheduling_group_id) {\n    bytes_sent[scheduling_group_id] = 0;\n    bytes_received[scheduling_group_id] = 0;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/proxy.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <utility>\n#include <vector>\n#include <cstdlib>\n#include <cstdint>\n#include <memory>\n\n#include <seastar/net/proxy.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nclass proxy_net_device : public qp {\nprivate:\n    static constexpr size_t _send_queue_length = 128;\n    size_t _send_depth = 0;\n    unsigned _cpu;\n    device* _dev;\n    std::vector<packet> _moving;\npublic:\n    explicit proxy_net_device(unsigned cpu, device* dev);\n    virtual future<> send(packet p) override {\n        abort();\n    }\n    virtual uint32_t send(circular_buffer<packet>& p) override;\n};\n\nproxy_net_device::proxy_net_device(unsigned cpu, device* dev) :\n        _cpu(cpu),\n        _dev(dev)\n{\n    _moving.reserve(_send_queue_length);\n}\n\nuint32_t proxy_net_device::send(circular_buffer<packet>& p)\n{\n    if (!_moving.empty() || _send_depth  == _send_queue_length) {\n        return 0;\n    }\n\n    for (; !p.empty() && _send_depth < _send_queue_length; _send_depth++) {\n        _moving.push_back(std::move(p.front()));\n        p.pop_front();\n    }\n\n    if (!_moving.empty()) {\n        qp* dev = &_dev->queue_for_cpu(_cpu);\n        auto cpu = this_shard_id();\n        // FIXME: future is discarded\n        (void)smp::submit_to(_cpu, [this, dev, cpu]() mutable {\n            for(size_t i = 0; i < _moving.size(); i++) {\n                dev->proxy_send(_moving[i].free_on_cpu(cpu, [this] { _send_depth--; }));\n            }\n        }).then([this] {\n            _moving.clear();\n        });\n    }\n\n    return _moving.size();\n}\n\nstd::unique_ptr<qp> create_proxy_net_device(unsigned master_cpu, device* dev) {\n    return std::make_unique<proxy_net_device>(master_cpu, dev);\n}\n}\n\n}\n"
  },
  {
    "path": "src/net/socket_address.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n/*! \\file\n    \\brief Some non-INET-specific socket address code\n\n    Extracted from inet_address.cc.\n */\n\n#include <arpa/inet.h>\n#include <sys/un.h>\n#include <ostream>\n#include <boost/functional/hash.hpp>\n\n#include <seastar/net/socket_defs.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/core/print.hh>\n\nusing namespace std::string_literals;\n\nsize_t std::hash<seastar::socket_address>::operator()(const seastar::socket_address& a) const {\n    auto h = std::hash<seastar::net::inet_address>()(a.addr());\n    boost::hash_combine(h, a.as_posix_sockaddr_in().sin_port);\n    return h;\n}\n\nnamespace seastar {\n\nstatic_assert(std::is_nothrow_default_constructible_v<socket_address>);\nstatic_assert(std::is_nothrow_copy_constructible_v<socket_address>);\nstatic_assert(std::is_nothrow_move_constructible_v<socket_address>);\n\nsocket_address::socket_address() noexcept\n    // set max addr_length, as we (probably) want to use the constructed object\n    // in accept() or get_address()\n    : addr_length(sizeof(::sockaddr_storage))\n{\n    static_assert(AF_UNSPEC == 0, \"just checking\");\n    memset(&u, 0, sizeof(u));\n}\n\nsocket_address::socket_address(uint16_t p) noexcept\n    : socket_address(ipv4_addr(p))\n{}\n\nsocket_address::socket_address(ipv4_addr addr) noexcept\n{\n    addr_length = sizeof(::sockaddr_in);\n    u.in.sin_family = AF_INET;\n    u.in.sin_port = htons(addr.port);\n    u.in.sin_addr.s_addr = htonl(addr.ip);\n}\n\nsocket_address::socket_address(const ipv6_addr& addr, uint32_t scope) noexcept\n{\n    addr_length = sizeof(::sockaddr_in6);\n    u.in6.sin6_family = AF_INET6;\n    u.in6.sin6_port = htons(addr.port);\n    u.in6.sin6_flowinfo = 0;\n    u.in6.sin6_scope_id = scope;\n    std::copy(addr.ip.begin(), addr.ip.end(), u.in6.sin6_addr.s6_addr);\n}\n\nsocket_address::socket_address(const ipv6_addr& addr) noexcept\n    : socket_address(addr, net::inet_address::invalid_scope)\n{}\n\nsocket_address::socket_address(uint32_t ipv4, uint16_t p) noexcept\n    : socket_address(make_ipv4_address(ipv4, p))\n{}\n\nsocket_address::socket_address(const net::inet_address& a, uint16_t p) noexcept\n    : socket_address(a.is_ipv6() ? socket_address(ipv6_addr(a, p), a.scope()) : socket_address(ipv4_addr(a, p)))\n{}\n\nsocket_address::socket_address(const unix_domain_addr& s) noexcept {\n    u.un.sun_family = AF_UNIX;\n    memset(u.un.sun_path, '\\0', sizeof(u.un.sun_path));\n    auto path_length = std::min((int)sizeof(u.un.sun_path), s.path_length());\n    memcpy(u.un.sun_path, s.path_bytes(), path_length);\n    addr_length = path_length + offsetof(struct ::sockaddr_un, sun_path);\n}\n\nbool socket_address::is_unspecified() const noexcept {\n    return u.sa.sa_family == AF_UNSPEC;\n}\n\nstatic int adjusted_path_length(const socket_address& a) noexcept {\n    int l = std::max(0, (int)a.addr_length-(int)(offsetof(sockaddr_un, sun_path)));\n    // \"un-count\" a trailing null in filesystem-namespace paths\n    if (a.u.un.sun_path[0]!='\\0' && (l > 1) && a.u.un.sun_path[l-1]=='\\0') {\n        --l;\n    }\n    return l;\n}\n\nbool socket_address::operator==(const socket_address& a) const noexcept {\n    if (u.sa.sa_family != a.u.sa.sa_family) {\n        return false;\n    }\n    if (u.sa.sa_family == AF_UNIX) {\n        // tolerate counting/not counting a terminating null in filesystem-namespace paths\n        int adjusted_len = adjusted_path_length(*this);\n        int a_adjusted_len = adjusted_path_length(a);\n        if (adjusted_len != a_adjusted_len) {\n            return false;\n        }\n        return (memcmp(u.un.sun_path, a.u.un.sun_path, adjusted_len) == 0);\n    }\n\n    // an INET address\n    if (u.in.sin_port != a.u.in.sin_port) {\n        return false;\n    }\n    switch (u.sa.sa_family) {\n    case AF_INET:\n        return u.in.sin_addr.s_addr == a.u.in.sin_addr.s_addr;\n    case AF_UNSPEC:\n    case AF_INET6:\n        // handled below\n        break;\n    default:\n        return false;\n    }\n\n    auto& in1 = as_posix_sockaddr_in6();\n    auto& in2 = a.as_posix_sockaddr_in6();\n\n    return IN6_ARE_ADDR_EQUAL(&in1.sin6_addr, &in2.sin6_addr);\n}\n\nstd::string unix_domain_addr_text(const socket_address& sa) {\n    if (sa.length() <= offsetof(sockaddr_un, sun_path)) {\n        return \"{unnamed}\"s;\n    }\n    if (sa.u.un.sun_path[0]) {\n        // regular (filesystem-namespace) path\n        return std::string{sa.u.un.sun_path};\n    }\n\n    const size_t  path_length{sa.length() - offsetof(sockaddr_un, sun_path)};\n    std::string ud_path(path_length, 0);\n    char* targ = &ud_path[0];\n    *targ++ = '@';\n    const char* src = sa.u.un.sun_path + 1;\n    int k = (int)path_length;\n\n    for (; --k > 0; src++) {\n        *targ++ = std::isprint(*src) ? *src : '_';\n    }\n    return ud_path;\n}\n\nstd::ostream& operator<<(std::ostream& os, const socket_address& a) {\n    if (a.is_af_unix()) {\n        return os << unix_domain_addr_text(a);\n    }\n\n    auto addr = a.addr();\n    // CMH. maybe skip brackets for ipv4-mapped\n    auto bracket = addr.in_family() == seastar::net::inet_address::family::INET6;\n\n    if (bracket) {\n        os << '[';\n    }\n    os << addr;\n    if (bracket) {\n        os << ']';\n    }\n\n    return os << ':' << ntohs(a.u.in.sin_port);\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "src/net/stack.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <memory>\n#include <type_traits>\n#include <utility>\n#include <vector>\n\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/net/inet_address.hh>\n\nnamespace seastar {\n\nstatic_assert(std::is_nothrow_default_constructible_v<connected_socket>);\nstatic_assert(std::is_nothrow_move_constructible_v<connected_socket>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<socket>);\nstatic_assert(std::is_nothrow_move_constructible_v<socket>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<server_socket>);\nstatic_assert(std::is_nothrow_move_constructible_v<server_socket>);\n\nnet::datagram_channel::datagram_channel() noexcept\n{}\n\nnet::datagram_channel::datagram_channel(std::unique_ptr<datagram_channel_impl> impl) noexcept : _impl(std::move(impl))\n{}\n\nnet::datagram_channel::~datagram_channel()\n{}\n\nnet::datagram_channel::datagram_channel(datagram_channel&&) noexcept = default;\nnet::datagram_channel& net::datagram_channel::operator=(datagram_channel&&) noexcept = default;\n\nsocket_address net::datagram_channel::local_address() const {\n    if (_impl) {\n        return _impl->local_address();\n    } else {\n        return {};\n    }\n}\n\nfuture<net::datagram> net::datagram_channel::receive() {\n    return _impl->receive();\n}\n\nfuture<> net::datagram_channel::send(const socket_address& dst, const char* msg) {\n    return _impl->send(dst, msg);\n}\n\nfuture<> net::datagram_channel::send(const socket_address& dst, packet p) {\n    auto bufs = std::move(p).release();\n    return _impl->send(dst, bufs);\n}\n\nfuture<> net::datagram_channel::send(const socket_address& dst, std::span<temporary_buffer<char>> bufs) {\n    return _impl->send(dst, bufs);\n}\n\nbool net::datagram_channel::is_closed() const {\n    return _impl->is_closed();\n}\n\nvoid net::datagram_channel::shutdown_input() {\n    _impl->shutdown_input();\n}\n\nvoid net::datagram_channel::shutdown_output() {\n    _impl->shutdown_output();\n}\n\n\nvoid net::datagram_channel::close() {\n    return _impl->close();\n}\n\nconnected_socket::connected_socket() noexcept\n{}\n\nconnected_socket::connected_socket(\n        std::unique_ptr<net::connected_socket_impl> csi) noexcept\n        : _csi(std::move(csi)) {\n}\n\nconnected_socket::connected_socket(connected_socket&& cs) noexcept = default;\nconnected_socket& connected_socket::operator=(connected_socket&& cs) noexcept = default;\n\nconnected_socket::~connected_socket()\n{}\n\ninput_stream<char> connected_socket::input(connected_socket_input_stream_config csisc) {\n    return input_stream<char>(_csi->source(csisc));\n}\n\noutput_stream<char> connected_socket::output(size_t buffer_size) {\n    output_stream_options opts;\n    opts.batch_flushes = true;\n    // TODO: allow user to determine buffer size etc\n    return output_stream<char>(_csi->sink(), buffer_size, opts);\n}\n\nvoid connected_socket::set_nodelay(bool nodelay) {\n    _csi->set_nodelay(nodelay);\n}\n\nbool connected_socket::get_nodelay() const {\n    return _csi->get_nodelay();\n}\nvoid connected_socket::set_keepalive(bool keepalive) {\n    _csi->set_keepalive(keepalive);\n}\nbool connected_socket::get_keepalive() const {\n    return _csi->get_keepalive();\n}\nvoid connected_socket::set_keepalive_parameters(const net::keepalive_params& p) {\n    _csi->set_keepalive_parameters(p);\n}\nnet::keepalive_params connected_socket::get_keepalive_parameters() const {\n    return _csi->get_keepalive_parameters();\n}\nvoid connected_socket::set_sockopt(int level, int optname, const void* data, size_t len) {\n    _csi->set_sockopt(level, optname, data, len);\n}\nint connected_socket::get_sockopt(int level, int optname, void* data, size_t len) const {\n    return _csi->get_sockopt(level, optname, data, len);\n}\n\nsocket_address connected_socket::local_address() const noexcept {\n    return _csi->local_address();\n}\n\nsocket_address connected_socket::remote_address() const noexcept {\n    return _csi->remote_address();\n}\n\nvoid connected_socket::shutdown_output() {\n    _csi->shutdown_output();\n}\n\nvoid connected_socket::shutdown_input() {\n    _csi->shutdown_input();\n}\n\nfuture<> connected_socket::wait_input_shutdown() {\n    return _csi->wait_input_shutdown();\n}\n\ndata_source\nnet::connected_socket_impl::source(connected_socket_input_stream_config csisc) {\n    // Default implementation falls back to non-parameterized data_source\n    return source();\n}\n\nsocket::~socket()\n{}\n\nsocket::socket(\n        std::unique_ptr<net::socket_impl> si) noexcept\n        : _si(std::move(si)) {\n}\n\nsocket::socket(socket&&) noexcept = default;\nsocket& socket::operator=(socket&&) noexcept = default;\n\nfuture<connected_socket> socket::connect(socket_address sa, socket_address local, transport proto) {\n    return _si->connect(sa, local, proto);\n}\n\nvoid socket::set_reuseaddr(bool reuseaddr) {\n    _si->set_reuseaddr(reuseaddr);\n}\n\nbool socket::get_reuseaddr() const {\n    return _si->get_reuseaddr();\n}\n\nvoid socket::shutdown() {\n    _si->shutdown();\n}\n\nserver_socket::server_socket() noexcept {\n}\n\nserver_socket::server_socket(std::unique_ptr<net::server_socket_impl> ssi) noexcept\n        : _ssi(std::move(ssi)) {\n}\nserver_socket::server_socket(server_socket&& ss) noexcept = default;\nserver_socket& server_socket::operator=(server_socket&& cs) noexcept = default;\n\nserver_socket::~server_socket() {\n}\n\nfuture<accept_result> server_socket::accept() {\n    if (_aborted) {\n        return make_exception_future<accept_result>(std::system_error(ECONNABORTED, std::system_category()));\n    }\n    return _ssi->accept();\n}\n\nvoid server_socket::abort_accept() {\n    _ssi->abort_accept();\n    _aborted = true;\n}\n\nsocket_address server_socket::local_address() const noexcept {\n    return _ssi->local_address();\n}\n\nnetwork_interface::network_interface(shared_ptr<net::network_interface_impl> impl) noexcept\n    : _impl(std::move(impl))\n{}\n\nnetwork_interface::network_interface(network_interface&&) noexcept = default;\nnetwork_interface& network_interface::operator=(network_interface&&) noexcept = default;\n\nuint32_t network_interface::index() const {\n    return _impl->index();\n}\n\nuint32_t network_interface::mtu() const {\n    return _impl->mtu();\n}\n\nconst sstring& network_interface::name() const {\n    return _impl->name();\n}\n\nconst sstring& network_interface::display_name() const {\n    return _impl->display_name();\n}\n\nconst std::vector<net::inet_address>& network_interface::addresses() const {\n    return _impl->addresses();\n}\n\nconst std::vector<uint8_t> network_interface::hardware_address() const {\n    return _impl->hardware_address();\n}\n\nbool network_interface::is_loopback() const {\n    return _impl->is_loopback();\n}\n\nbool network_interface::is_virtual() const {\n    return _impl->is_virtual();\n}\n\nbool network_interface::is_up() const {\n    return _impl->is_up();\n}\n\nbool network_interface::supports_ipv6() const {\n    return _impl->supports_ipv6();\n}\n\n\nfuture<connected_socket>\nnetwork_stack::connect(socket_address sa, socket_address local, transport proto) {\n    return do_with(socket(), [sa, local, proto](::seastar::socket& s) {\n        return s.connect(sa, local, proto);\n    });\n}\n\nstd::vector<network_interface> network_stack::network_interfaces() {\n    return {};\n}\n\nvoid register_net_metrics_for_scheduling_group(\n    metrics::metric_groups &metrics, unsigned sg_id, const metrics::label_instance& name) {\n    namespace sm = seastar::metrics;\n    metrics.add_group(\"network\", {\n        sm::make_counter(\"bytes_sent\", [sg_id] { return engine().net().stats(sg_id).bytes_sent; },\n                sm::description(\"Counts the number of bytes written to network sockets.\"), {name}),\n        sm::make_counter(\"bytes_received\", [sg_id] { return engine().net().stats(sg_id).bytes_received; },\n                sm::description(\"Counts the number of bytes received from network sockets.\"), {name}),\n    });\n\n    // need to clear stats in case we recreated a SG with the same id\n    // but avoid during reactor startup\n    if (engine_is_ready()) {\n        engine().net().clear_stats(sg_id);\n    }\n}\n\n}\n"
  },
  {
    "path": "src/net/tcp.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/tcp.hh>\n#include <seastar/net/tcp-stack.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/future.hh>\n#include \"net/native-stack-impl.hh\"\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nnamespace net {\n\nvoid tcp_option::parse(uint8_t* beg1, uint8_t* end1) {\n    const char* beg = reinterpret_cast<const char*>(beg1);\n    const char* end = reinterpret_cast<const char*>(end1);\n    while (beg < end) {\n        auto kind = option_kind(*beg);\n        if (kind != option_kind::nop && kind != option_kind::eol) {\n            // Make sure there is enough room for this option\n            auto len = uint8_t(beg[1]);\n            if (beg + len > end) {\n                return;\n            }\n        }\n        switch (kind) {\n        case option_kind::mss:\n            _mss_received = true;\n            _remote_mss = mss::read(beg).mss;\n            beg += option_len::mss;\n            break;\n        case option_kind::win_scale:\n            _win_scale_received = true;\n            _remote_win_scale = win_scale::read(beg).shift;\n            // We can turn on win_scale option, 7 is Linux's default win scale size\n            _local_win_scale = 7;\n            beg += option_len::win_scale;\n            break;\n        case option_kind::sack:\n            _sack_received = true;\n            beg += option_len::sack;\n            break;\n        case option_kind::nop:\n            beg += option_len::nop;\n            break;\n        case option_kind::eol:\n            return;\n        default:\n            // Ignore options we do not understand\n            uint8_t len = *(beg + 1);\n            beg += len;\n            // Prevent infinite loop\n            if (len == 0) {\n                return;\n            }\n            break;\n        }\n    }\n}\n\nuint8_t tcp_option::fill(void* h, const tcp_hdr* th, uint8_t options_size) {\n    auto hdr = reinterpret_cast<char*>(h);\n    auto off = hdr + tcp_hdr::len;\n    uint8_t size = 0;\n    bool syn_on = th->f_syn;\n    bool ack_on = th->f_ack;\n\n    if (syn_on) {\n        if (_mss_received || !ack_on) {\n            auto mss = tcp_option::mss();\n            mss.mss = _local_mss;\n            mss.write(off);\n            off += mss.len;\n            size += mss.len;\n        }\n        if (_win_scale_received || !ack_on) {\n            auto win_scale = tcp_option::win_scale();\n            win_scale.shift = _local_win_scale;\n            win_scale.write(off);\n            off += win_scale.len;\n            size += win_scale.len;\n        }\n    }\n    if (size > 0) {\n        // Insert NOP option\n        auto size_max = align_up(uint8_t(size + 1), tcp_option::align);\n        while (size < size_max - uint8_t(option_len::eol)) {\n            auto nop = tcp_option::nop();\n            nop.write(off);\n            off += option_len::nop;\n            size += option_len::nop;\n        }\n        auto eol = tcp_option::eol();\n        eol.write(off);\n        size += option_len::eol;\n    }\n    SEASTAR_ASSERT(size == options_size);\n\n    return size;\n}\n\nuint8_t tcp_option::get_size(bool syn_on, bool ack_on) {\n    uint8_t size = 0;\n    if (syn_on) {\n        if (_mss_received || !ack_on) {\n            size += option_len::mss;\n        }\n        if (_win_scale_received || !ack_on) {\n            size += option_len::win_scale;\n        }\n    }\n    if (size > 0) {\n        size += option_len::eol;\n        // Insert NOP option to align on 32-bit\n        size = align_up(size, tcp_option::align);\n    }\n    return size;\n}\n\nipv4_tcp::ipv4_tcp(ipv4& inet)\n\t: _inet_l4(inet), _tcp(std::make_unique<tcp<ipv4_traits>>(_inet_l4)) {\n}\n\nipv4_tcp::~ipv4_tcp() {\n}\n\nvoid ipv4_tcp::received(packet p, ipv4_address from, ipv4_address to) {\n    _tcp->received(std::move(p), from, to);\n}\n\nbool ipv4_tcp::forward(forward_hash& out_hash_data, packet& p, size_t off) {\n\n    return _tcp->forward(out_hash_data, p, off);\n}\n\nserver_socket\ntcpv4_listen(tcp<ipv4_traits>& tcpv4, uint16_t port, listen_options opts) {\n\treturn server_socket(std::make_unique<native_server_socket_impl<tcp<ipv4_traits>>>(\n\t\t\ttcpv4, port, opts));\n}\n\n::seastar::socket\ntcpv4_socket(tcp<ipv4_traits>& tcpv4) {\n    return ::seastar::socket(std::make_unique<native_socket_impl<tcp<ipv4_traits>>>(\n            tcpv4));\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/net/tls.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n\n#include <any>\n#include <filesystem>\n#include <stdexcept>\n#include <system_error>\n#include <memory>\n#include <chrono>\n#include <span>\n#include <unordered_set>\n#include <numeric>\n\n#include <seastar/util/assert.hh>\n\n#include <netinet/in.h>\n#include <sys/stat.h>\n#include <gnutls/gnutls.h>\n#include <gnutls/x509.h>\n\n#include <boost/any.hpp>\n#include <boost/range/iterator_range.hpp>\n#include <boost/range/adaptor/map.hpp>\n\n#include <fmt/core.h>\n#include <fmt/ostream.h>\n\n#include <seastar/core/loop.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/with_timeout.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/variant_utils.hh>\n#include <seastar/core/fsnotify.hh>\n\nnamespace seastar {\n\nusing namespace std::string_view_literals;\n\nclass net::get_impl {\npublic:\n    static std::unique_ptr<connected_socket_impl> get(connected_socket s) {\n        return std::move(s._csi);\n    }\n\n    static connected_socket_impl* maybe_get_ptr(connected_socket& s) {\n        if (s._csi) {\n            return s._csi.get();\n        }\n        return nullptr;\n    }\n};\n\nclass blob_wrapper: public gnutls_datum_t {\npublic:\n    blob_wrapper(const tls::blob& in)\n            : gnutls_datum_t {\n                    reinterpret_cast<uint8_t *>(const_cast<char *>(in.data())),\n                    unsigned(in.size()) } {\n    }\n};\n\nclass gnutlsinit {\npublic:\n    gnutlsinit() {\n        gnutls_global_init();\n    }\n    ~gnutlsinit() {\n        gnutls_global_deinit();\n    }\n};\n\n// Helper to ensure gnutls legacy init\n// is handled properly with regards to\n// object life spans. Could be better,\n// this version will not destroy the\n// gnutls stack until process exit.\nclass gnutlsobj {\npublic:\n    gnutlsobj() {\n        static gnutlsinit init;\n    }\n};\n\n// Helper\nstruct file_info {\n    sstring filename;\n    std::chrono::system_clock::time_point modified;\n};\n\nstruct file_result {\n    temporary_buffer<char> buf;\n    file_info file;\n    operator temporary_buffer<char>&&() && {\n        return std::move(buf);\n    }\n};\n\nstatic future<file_result> read_fully(const sstring& name, const sstring& what) {\n    return open_file_dma(name, open_flags::ro).then([name = name](file f) mutable {\n        return do_with(std::move(f), [name = std::move(name)](file& f) mutable {\n            return f.stat().then([&f, name = std::move(name)](struct stat s) mutable {\n                return f.dma_read_bulk<char>(0, s.st_size).then([s, name = std::move(name)](temporary_buffer<char> buf) mutable {\n                    return file_result{ std::move(buf), file_info{\n                        std::move(name), std::chrono::system_clock::from_time_t(s.st_mtim.tv_sec) +\n                            std::chrono::duration_cast<std::chrono::system_clock::duration>(std::chrono::nanoseconds(s.st_mtim.tv_nsec))\n                    } };\n                });\n            }).finally([&f]() {\n                return f.close();\n            });\n        });\n    }).handle_exception([name = name, what = what](std::exception_ptr ep) -> future<file_result> {\n       try {\n           std::rethrow_exception(std::move(ep));\n       } catch (...) {\n           std::throw_with_nested(std::runtime_error(sstring(\"Could not read \") + what + \" \" + name));\n       }\n    });\n}\n\n// Note: we are not using gnutls++ interfaces, mainly because we\n// want to keep _our_ interface reasonably non-gnutls (well...)\n// and once we get to this level, their abstractions don't help\n// that much anyway. And they are sooo c++98...\nclass gnutls_error_category : public std::error_category {\npublic:\n    constexpr gnutls_error_category() noexcept : std::error_category{} {}\n    const char * name() const noexcept override {\n        return \"GnuTLS\";\n    }\n    std::string message(int error) const override {\n        return gnutls_strerror(error);\n    }\n};\n\nconst std::error_category& tls::error_category() {\n    static const gnutls_error_category ec;\n    return ec;\n}\n\n// Checks a gnutls return value.\n// < 0 -> error.\nstatic void gtls_chk(int res) {\n    if (res < 0) {\n        throw std::system_error(res, tls::error_category());\n    }\n}\n\nnamespace {\n\n// helper for gnutls-functions for receiving a string\n// arguments\n//  func - the gnutls function that is returning a string (e.g. gnutls_x509_crt_get_issuer_dn)\n//  args - the arguments to func that come before the char array's ptr and size args\n// returns\n//  pair<int, string> - [gnutls error code, extracted string],\n//                      in case of no errors, the error code is zero\nstatic auto get_gtls_string = [](auto func, auto... args) noexcept {\n    size_t size = 0;\n    int ret = func(args..., nullptr, &size);\n\n    // by construction, we expect the SHORT_MEMORY_BUFFER error code here\n    if (ret != GNUTLS_E_SHORT_MEMORY_BUFFER) {\n        return std::make_pair(ret, sstring{});\n    }\n    SEASTAR_ASSERT(size != 0);\n    auto res = uninitialized_string(size - 1);\n    ret = func(args..., res.data(), &size);\n    return std::make_pair(ret, res);\n};\n\n}\n\nclass tls::dh_params::impl : gnutlsobj {\n    static gnutls_sec_param_t to_gnutls_level(level l) {\n        switch (l) {\n            case level::LEGACY: return GNUTLS_SEC_PARAM_LEGACY;\n#if GNUTLS_VERSION_NUMBER >= 0x030300\n            case level::MEDIUM: return GNUTLS_SEC_PARAM_MEDIUM;\n#else\n            case level::MEDIUM: return GNUTLS_SEC_PARAM_NORMAL;\n#endif\n            case level::HIGH: return GNUTLS_SEC_PARAM_HIGH;\n            case level::ULTRA: return GNUTLS_SEC_PARAM_ULTRA;\n            default:\n                throw std::runtime_error(format(\"Unknown value of dh_params::level: {:d}\", static_cast<std::underlying_type_t<level>>(l)));\n        }\n    }\n    using dh_ptr = std::unique_ptr<std::remove_pointer_t<gnutls_dh_params_t>, void(*)(gnutls_dh_params_t)>;\n\n    static dh_ptr new_dh_params() {\n        gnutls_dh_params_t params;\n        gtls_chk(gnutls_dh_params_init(&params));\n        return dh_ptr(params, &gnutls_dh_params_deinit);\n    }\npublic:\n    impl(dh_ptr p)\n        : _params(std::move(p))\n    {}\n    impl(level lvl)\n#if GNUTLS_VERSION_NUMBER >= 0x030506\n        : _params(nullptr, &gnutls_dh_params_deinit)\n        , _sec_param(to_gnutls_level(lvl))\n#else\n        : impl([&] {\n            auto bits = gnutls_sec_param_to_pk_bits(GNUTLS_PK_DH, to_gnutls_level(lvl));\n            auto ptr = new_dh_params();\n            gtls_chk(gnutls_dh_params_generate2(ptr.get(), bits));\n            return ptr;\n        }())\n#endif\n    {}\n    impl(const blob& pkcs3, x509_crt_format fmt)\n        : impl([&] {\n            auto ptr = new_dh_params();\n            blob_wrapper w(pkcs3);\n            gtls_chk(gnutls_dh_params_import_pkcs3(ptr.get(), &w, gnutls_x509_crt_fmt_t(fmt)));\n            return ptr;\n        }())\n    {}\n    impl(const impl& v)\n        : impl([&v] {\n            auto ptr = new_dh_params();\n            gtls_chk(gnutls_dh_params_cpy(ptr.get(), v));\n            return ptr;\n        }())\n    {}\n    ~impl() = default;\n\n    operator gnutls_dh_params_t() const {\n        return _params.get();\n    }\n#if GNUTLS_VERSION_NUMBER >= 0x030506\n    std::optional<gnutls_sec_param_t> sec_param() const {\n        return _sec_param;\n    }\n#endif\nprivate:\n    dh_ptr _params;\n#if GNUTLS_VERSION_NUMBER >= 0x030506\n    std::optional<gnutls_sec_param_t> _sec_param;\n#endif\n};\n\ntls::dh_params::dh_params(level lvl) : _impl(std::make_unique<impl>(lvl))\n{}\n\ntls::dh_params::dh_params(const blob& b, x509_crt_format fmt)\n        : _impl(std::make_unique<impl>(b, fmt)) {\n}\n\ntls::dh_params::~dh_params() {\n}\n\ntls::dh_params::dh_params(dh_params&&) noexcept = default;\ntls::dh_params& tls::dh_params::operator=(dh_params&&) noexcept = default;\n\nfuture<tls::dh_params> tls::dh_params::from_file(\n        const sstring& filename, x509_crt_format fmt) {\n    return read_fully(filename, \"dh parameters\").then([fmt](temporary_buffer<char> buf) {\n        return make_ready_future<dh_params>(dh_params(blob(buf.get()), fmt));\n    });\n}\n\nclass tls::x509_cert::impl : gnutlsobj {\npublic:\n    impl()\n            : _cert([] {\n                gnutls_x509_crt_t cert;\n                gtls_chk(gnutls_x509_crt_init(&cert));\n                return cert;\n            }()) {\n    }\n    impl(const blob& b, x509_crt_format fmt)\n        : impl()\n    {\n        blob_wrapper w(b);\n        gtls_chk(gnutls_x509_crt_import(*this, &w, gnutls_x509_crt_fmt_t(fmt)));\n    }\n    ~impl() {\n        if (_cert != nullptr) {\n            gnutls_x509_crt_deinit(_cert);\n        }\n    }\n    operator gnutls_x509_crt_t() const {\n        return _cert;\n    }\n\nprivate:\n    gnutls_x509_crt_t _cert;\n};\n\ntls::x509_cert::x509_cert(shared_ptr<impl> impl)\n        : _impl(std::move(impl)) {\n}\n\ntls::x509_cert::x509_cert(const blob& b, x509_crt_format fmt)\n        : x509_cert(::seastar::make_shared<impl>(b, fmt)) {\n}\n\nfuture<tls::x509_cert> tls::x509_cert::from_file(\n        const sstring& filename, x509_crt_format fmt) {\n    return read_fully(filename, \"x509 certificate\").then([fmt](temporary_buffer<char> buf) {\n        return make_ready_future<x509_cert>(x509_cert(blob(buf.get()), fmt));\n    });\n}\n\n// wrapper for gnutls_datum, with raii free\nstruct gnutls_datum : public gnutls_datum_t {\n    gnutls_datum(size_t s) {\n        data = reinterpret_cast<unsigned char*>(gnutls_malloc(s));\n        if (data == nullptr) {\n           throw std::bad_alloc();\n        }\n        size = s;\n    }\n    gnutls_datum() {\n        data = nullptr;\n        size = 0;\n    }\n    gnutls_datum(const gnutls_datum&) = delete;\n    gnutls_datum& operator=(gnutls_datum&& other) {\n        if (this == &other) {\n            return *this;\n        }\n        if (data != nullptr) {\n            ::gnutls_memset(data, 0, size);\n            ::gnutls_free(data);\n        }\n        data = std::exchange(other.data, nullptr);\n        size = std::exchange(other.size, 0);\n        return *this;\n    }\n    ~gnutls_datum() {\n        if (data != nullptr) {\n            ::gnutls_memset(data, 0, size);\n            ::gnutls_free(data);\n        }\n    }\n};\n\nclass tls::certificate_credentials::impl: public gnutlsobj {\npublic:\n    impl()\n            : _creds([] {\n                gnutls_certificate_credentials_t xcred;\n                gnutls_certificate_allocate_credentials(&xcred);\n                if (xcred == nullptr) {\n                    throw std::bad_alloc();\n                }\n                return xcred;\n            }()), _priority(nullptr, &gnutls_priority_deinit)\n    {}\n    ~impl() {\n        if (_creds != nullptr) {\n            gnutls_certificate_free_credentials (_creds);\n        }\n    }\n\n    operator gnutls_certificate_credentials_t() const {\n        return _creds;\n    }\n\n    void set_x509_trust(const blob& b, x509_crt_format fmt) {\n        blob_wrapper w(b);\n        gtls_chk(\n                gnutls_certificate_set_x509_trust_mem(_creds, &w,\n                        gnutls_x509_crt_fmt_t(fmt)));\n    }\n    void set_x509_crl(const blob& b, x509_crt_format fmt) {\n        blob_wrapper w(b);\n        gtls_chk(\n                gnutls_certificate_set_x509_crl_mem(_creds, &w,\n                        gnutls_x509_crt_fmt_t(fmt)));\n    }\n    void set_x509_key(const blob& cert, const blob& key, x509_crt_format fmt) {\n        blob_wrapper w1(cert);\n        blob_wrapper w2(key);\n        gtls_chk(\n                gnutls_certificate_set_x509_key_mem(_creds, &w1, &w2,\n                        gnutls_x509_crt_fmt_t(fmt)));\n    }\n    void set_simple_pkcs12(const blob& b, x509_crt_format fmt,\n            const sstring& password) {\n        blob_wrapper w(b);\n        gtls_chk(\n                gnutls_certificate_set_x509_simple_pkcs12_mem(_creds, &w,\n                        gnutls_x509_crt_fmt_t(fmt), password.c_str()));\n    }\n    void dh_params(const tls::dh_params& dh) {\n#if GNUTLS_VERSION_NUMBER >= 0x030506\n        auto sec_param = dh._impl->sec_param();\n        if (sec_param) {\n            gnutls_certificate_set_known_dh_params(*this, *sec_param);\n            return;\n        }\n#endif\n        auto cpy = std::make_unique<tls::dh_params::impl>(*dh._impl);\n        gnutls_certificate_set_dh_params(*this, *cpy);\n        _dh_params = std::move(cpy);\n    }\n    future<> set_system_trust() {\n        return async([this] {\n            gtls_chk(gnutls_certificate_set_x509_system_trust(_creds));\n            _load_system_trust = false; // should only do once, for whatever reason\n        });\n    }\n    void set_client_auth(client_auth ca) {\n        _client_auth = ca;\n    }\n    client_auth get_client_auth() const {\n        return _client_auth;\n    }\n    void set_session_resume_mode(session_resume_mode m, std::span<const uint8_t> key = {}) {\n        _session_resume_mode = m;\n        // (re-)generate session key\n        if (m != session_resume_mode::NONE) {\n            _session_resume_key = {};\n            if (key.empty()) {\n                gtls_chk(gnutls_session_ticket_key_generate(&_session_resume_key));\n            } else {\n                _session_resume_key = gnutls_datum(key.size());\n                std::copy(key.begin(), key.end(), _session_resume_key.data);\n            }\n        }\n    }\n    session_resume_mode get_session_resume_mode() const {\n        return _session_resume_mode;\n    }\n    const gnutls_datum_t* get_session_resume_key() const {\n        return &_session_resume_key;\n    }\n    void set_priority_string(const sstring& prio) {\n        const char * err = prio.c_str();\n        try {\n            gnutls_priority_t p;\n            gtls_chk(gnutls_priority_init(&p, prio.c_str(), &err));\n            _priority.reset(p);\n        } catch (...) {\n            std::throw_with_nested(std::invalid_argument(std::string(\"Could not set priority: \") + err));\n        }\n    }\n    gnutls_priority_t get_priority() const {\n        return _priority.get();\n    }\n\n    void set_dn_verification_callback(dn_callback cb) {\n        _dn_callback = std::move(cb);\n    }\n\n    void set_enable_certificate_verification(bool enable) {\n        _enable_certificate_verification = enable;\n    }\n\n    void set_alpn_protocols(const std::vector<sstring>& protocols) {\n        _alpn_protocols = protocols;\n    }\n\nprivate:\n    friend class credentials_builder;\n    friend class session;\n\n    bool need_load_system_trust() const {\n        return _load_system_trust;\n    }\n    future<> maybe_load_system_trust() {\n        return with_semaphore(_system_trust_sem, 1, [this] {\n            if (!_load_system_trust) {\n                return make_ready_future();\n            }\n            return set_system_trust();\n        });\n    }\n\n    gnutls_certificate_credentials_t _creds;\n    std::unique_ptr<tls::dh_params::impl> _dh_params;\n    std::unique_ptr<std::remove_pointer_t<gnutls_priority_t>, void(*)(gnutls_priority_t)> _priority;\n    client_auth _client_auth = client_auth::NONE;\n    session_resume_mode _session_resume_mode = session_resume_mode::NONE;\n    bool _load_system_trust = false;\n    semaphore _system_trust_sem {1};\n    dn_callback _dn_callback;\n    bool _enable_certificate_verification = true;\n    gnutls_datum _session_resume_key;\n    std::vector<sstring> _alpn_protocols;\n};\n\ntls::certificate_credentials::certificate_credentials()\n        : _impl(make_shared<impl>()) {\n}\n\ntls::certificate_credentials::~certificate_credentials() {\n}\n\ntls::certificate_credentials::certificate_credentials(\n        certificate_credentials&&) noexcept = default;\ntls::certificate_credentials& tls::certificate_credentials::operator=(\n        certificate_credentials&&) noexcept = default;\n\nvoid tls::certificate_credentials::set_x509_trust(const blob& b,\n        x509_crt_format fmt) {\n    _impl->set_x509_trust(b, fmt);\n}\n\nvoid tls::certificate_credentials::set_x509_crl(const blob& b,\n        x509_crt_format fmt) {\n    _impl->set_x509_crl(b, fmt);\n\n}\nvoid tls::certificate_credentials::set_x509_key(const blob& cert,\n        const blob& key, x509_crt_format fmt) {\n    _impl->set_x509_key(cert, key, fmt);\n}\n\nvoid tls::certificate_credentials::set_simple_pkcs12(const blob& b,\n        x509_crt_format fmt, const sstring& password) {\n    _impl->set_simple_pkcs12(b, fmt, password);\n}\n\nfuture<> tls::abstract_credentials::set_x509_trust_file(\n        const sstring& cafile, x509_crt_format fmt) {\n    return read_fully(cafile, \"trust file\").then([this, fmt](temporary_buffer<char> buf) {\n        set_x509_trust(blob(buf.get(), buf.size()), fmt);\n    });\n}\n\nfuture<> tls::abstract_credentials::set_x509_crl_file(\n        const sstring& crlfile, x509_crt_format fmt) {\n    return read_fully(crlfile, \"crl file\").then([this, fmt](temporary_buffer<char> buf) {\n        set_x509_crl(blob(buf.get(), buf.size()), fmt);\n    });\n}\n\nfuture<> tls::abstract_credentials::set_x509_key_file(\n        const sstring& cf, const sstring& kf, x509_crt_format fmt) {\n    return read_fully(cf, \"certificate file\").then([this, fmt, kf = kf](temporary_buffer<char> buf) {\n        return read_fully(kf, \"key file\").then([this, fmt, buf = std::move(buf)](temporary_buffer<char> buf2) {\n                    set_x509_key(blob(buf.get(), buf.size()), blob(buf2.get(), buf2.size()), fmt);\n                });\n    });\n}\n\nfuture<> tls::abstract_credentials::set_simple_pkcs12_file(\n        const sstring& pkcs12file, x509_crt_format fmt,\n        const sstring& password) {\n    return read_fully(pkcs12file, \"pkcs12 file\").then([this, fmt, password = password](temporary_buffer<char> buf) {\n        set_simple_pkcs12(blob(buf.get(), buf.size()), fmt, password);\n    });\n}\n\nfuture<> tls::certificate_credentials::set_system_trust() {\n    return _impl->set_system_trust();\n}\n\nvoid tls::certificate_credentials::set_priority_string(const sstring& prio) {\n    _impl->set_priority_string(prio);\n}\n\nvoid tls::certificate_credentials::set_dn_verification_callback(dn_callback cb) {\n    _impl->set_dn_verification_callback(std::move(cb));\n}\n\nvoid tls::certificate_credentials::set_enable_certificate_verification(bool enable) {\n    _impl->set_enable_certificate_verification(enable);\n}\n\ntls::server_credentials::server_credentials()\n#if GNUTLS_VERSION_NUMBER < 0x030600\n    : server_credentials(dh_params{})\n#endif\n{}\n\ntls::server_credentials::server_credentials(shared_ptr<dh_params> dh)\n    : server_credentials(*dh)\n{}\n\ntls::server_credentials::server_credentials(const dh_params& dh) {\n    _impl->dh_params(dh);\n}\n\ntls::server_credentials::server_credentials(server_credentials&&) noexcept = default;\ntls::server_credentials& tls::server_credentials::operator=(\n        server_credentials&&) noexcept = default;\n\nvoid tls::server_credentials::set_client_auth(client_auth ca) {\n    _impl->set_client_auth(ca);\n}\n\nvoid tls::server_credentials::set_session_resume_mode(session_resume_mode m) {\n    _impl->set_session_resume_mode(m);\n}\n\nvoid tls::server_credentials::set_alpn_protocols(const std::vector<sstring>& protocols) {\n    _impl->set_alpn_protocols(protocols);\n}\n\nconstexpr auto dh_level_key = \"dh_level\"sv;\nconstexpr auto x509_trust_key = \"x509_trust\"sv;\nconstexpr auto x509_crl_key = \"x509_crl\"sv;\nconstexpr auto x509_key_key = \"x509_key\"sv;\nconstexpr auto pkcs12_key = \"pkcs12\"sv;\nconstexpr auto system_trust = \"system_trust\"sv;\n\nusing buffer_type = std::basic_string<tls::blob::value_type, tls::blob::traits_type, std::allocator<tls::blob::value_type>>;\n\nstruct x509_simple {\n    buffer_type data;\n    tls::x509_crt_format format;\n    file_info file;\n};\n\nstruct x509_key {\n    buffer_type cert;\n    buffer_type key;\n    tls::x509_crt_format format;\n    file_info cert_file;\n    file_info key_file;\n};\n\nstruct pkcs12_simple {\n    buffer_type data;\n    tls::x509_crt_format format;\n    sstring password;\n    file_info file;\n};\n\nvoid tls::credentials_builder::set_dh_level(dh_params::level level) {\n    _blobs.emplace(dh_level_key, level);\n}\n\nvoid tls::credentials_builder::set_x509_trust(const blob& b, x509_crt_format fmt) {\n    _blobs.emplace(x509_trust_key, x509_simple{ std::string(b), fmt });\n}\n\nvoid tls::credentials_builder::set_x509_crl(const blob& b, x509_crt_format fmt) {\n    _blobs.emplace(x509_crl_key, x509_simple{ std::string(b), fmt });\n}\n\nvoid tls::credentials_builder::set_x509_key(const blob& cert, const blob& key, x509_crt_format fmt) {\n    _blobs.emplace(x509_key_key, x509_key { std::string(cert), std::string(key), fmt });\n}\n\nvoid tls::credentials_builder::set_simple_pkcs12(const blob& b, x509_crt_format fmt, const sstring& password) {\n    _blobs.emplace(pkcs12_key, pkcs12_simple{std::string(b), fmt, password });\n}\n\nstatic buffer_type to_buffer(const temporary_buffer<char>& buf) {\n    return buffer_type(buf.get(), buf.get() + buf.size());\n}\n\nfuture<> tls::credentials_builder::set_x509_trust_file(const sstring& cafile, x509_crt_format fmt) {\n    return read_fully(cafile, \"trust file\").then([this, fmt](file_result f) {\n        _blobs.emplace(x509_trust_key, x509_simple{ to_buffer(f.buf), fmt, std::move(f.file) });\n    });\n}\n\nfuture<> tls::credentials_builder::set_x509_crl_file(const sstring& crlfile, x509_crt_format fmt) {\n    return read_fully(crlfile, \"crl file\").then([this, fmt](file_result f) {\n        _blobs.emplace(x509_crl_key, x509_simple{ to_buffer(f.buf), fmt, std::move(f.file) });\n    });\n}\n\nfuture<> tls::credentials_builder::set_x509_key_file(const sstring& cf, const sstring& kf, x509_crt_format fmt) {\n    return read_fully(cf, \"certificate file\").then([this, fmt, kf = kf](file_result cf) {\n        return read_fully(kf, \"key file\").then([this, fmt, cf = std::move(cf)](file_result kf) {\n            _blobs.emplace(x509_key_key, x509_key{ to_buffer(cf.buf), to_buffer(kf.buf), fmt, std::move(cf.file), std::move(kf.file) });\n        });\n    });\n}\n\nfuture<> tls::credentials_builder::set_simple_pkcs12_file(const sstring& pkcs12file, x509_crt_format fmt, const sstring& password) {\n    return read_fully(pkcs12file, \"pkcs12 file\").then([this, fmt, password = password](file_result f) {\n        _blobs.emplace(pkcs12_key, pkcs12_simple{ to_buffer(f.buf), fmt, password, std::move(f.file) });\n    });\n}\n\nfuture<> tls::credentials_builder::set_system_trust() {\n    // TODO / Caveat:\n    // We cannot actually issue a loading of system trust here,\n    // because we have no actual tls context.\n    // And we probably _don't want to get into the guessing game\n    // of where the system trust cert chains are, since this is\n    // super distro dependent, and usually compiled into the library.\n    // Pretent it is raining, and just set a flag.\n    // Leave the function returning future, so if we change our\n    // minds and want to do explicit loading, we can...\n    _blobs.emplace(system_trust, true);\n    return make_ready_future();\n}\n\nvoid tls::credentials_builder::set_client_auth(client_auth auth) {\n    _client_auth = auth;\n}\n\nvoid tls::credentials_builder::set_priority_string(const sstring& prio) {\n    _priority = prio;\n}\n\nvoid tls::credentials_builder::set_session_resume_mode(session_resume_mode m) {\n    _session_resume_mode = m;\n    if (m != session_resume_mode::NONE) {\n        gnutls_datum key;\n        gtls_chk(gnutls_session_ticket_key_generate(&key));\n        _session_resume_key.assign(key.data, key.data + key.size);\n    }\n}\n\nvoid tls::credentials_builder::set_alpn_protocols(const std::vector<sstring>& protocols) {\n    _alpn_protocols = protocols;\n}\n\ntemplate<typename Blobs, typename Visitor>\nstatic void visit_blobs(Blobs& blobs, Visitor&& visitor) {\n    auto visit = [&](const std::string_view& key, auto* vt) {\n        auto tr = blobs.equal_range(key);\n        for (auto& p : boost::make_iterator_range(tr.first, tr.second)) {\n            auto* v = std::any_cast<std::decay_t<decltype(*vt)>>(&p.second);\n            visitor(key, *v);\n        }\n    };\n    visit(x509_trust_key, static_cast<x509_simple*>(nullptr));\n    visit(x509_crl_key, static_cast<x509_simple*>(nullptr));\n    visit(x509_key_key, static_cast<x509_key*>(nullptr));\n    visit(pkcs12_key, static_cast<pkcs12_simple*>(nullptr));\n}\n\nvoid tls::credentials_builder::apply_to(certificate_credentials& creds) const {\n    // Could potentially be templated down, but why bother...\n    visit_blobs(_blobs, make_visitor(\n        [&](const std::string_view& key, const x509_simple& info) {\n            if (key == x509_trust_key) {\n                creds.set_x509_trust(info.data, info.format);\n            } else if (key == x509_crl_key) {\n                creds.set_x509_crl(info.data, info.format);\n            }\n        },\n        [&](const std::string_view&, const x509_key& info) {\n            creds.set_x509_key(info.cert, info.key, info.format);\n        },\n        [&](const std::string_view&, const pkcs12_simple& info) {\n            creds.set_simple_pkcs12(info.data, info.format, info.password);\n        }\n    ));\n\n    // TODO / Caveat:\n    // We cannot do this immediately, because we are not a continuation, and\n    // potentially blocking calls are a no-no.\n    // Doing this detached would be indeterministic, so set a flag in\n    // credentials, and do actual loading in first handshake (see session)\n    if (_blobs.count(system_trust)) {\n        creds._impl->_load_system_trust = true;\n    }\n\n    if (!_priority.empty()) {\n        creds.set_priority_string(_priority);\n    }\n\n    creds._impl->set_client_auth(_client_auth);\n    // Note: this causes server session key rotation on cert reload\n    creds._impl->set_session_resume_mode(_session_resume_mode, std::span{_session_resume_key.begin(), _session_resume_key.end()});\n\n    if (!_alpn_protocols.empty()) {\n        creds._impl->set_alpn_protocols(_alpn_protocols);\n    }\n}\n\nshared_ptr<tls::certificate_credentials> tls::credentials_builder::build_certificate_credentials() const {\n    auto creds = make_shared<certificate_credentials>();\n    apply_to(*creds);\n    return creds;\n}\n\nshared_ptr<tls::server_credentials> tls::credentials_builder::build_server_credentials() const {\n    auto i = _blobs.find(dh_level_key);\n    if (i == _blobs.end()) {\n#if GNUTLS_VERSION_NUMBER < 0x030600\n        throw std::invalid_argument(\"No DH level set\");\n#else\n        auto creds = make_shared<server_credentials>();\n        apply_to(*creds);\n        return creds;\n#endif\n    }\n    auto creds = make_shared<server_credentials>(dh_params(std::any_cast<dh_params::level>(i->second)));\n    apply_to(*creds);\n    return creds;\n}\n\nusing namespace std::chrono_literals;\n\nclass tls::reloadable_credentials_base {\npublic:\n    using delay_type = std::chrono::milliseconds;\n    static inline constexpr delay_type default_tolerance = 500ms;\n\n    class reloading_builder\n        : public credentials_builder\n        , public enable_shared_from_this<reloading_builder>\n    {\n    public:\n        using time_point = std::chrono::system_clock::time_point;\n\n        reloading_builder(credentials_builder b, reload_callback_ex cb, reloadable_credentials_base* creds, delay_type delay)\n            : credentials_builder(std::move(b))\n            , _cb(std::move(cb))\n            , _creds(creds)\n            , _delay(delay)\n        {}\n        future<> init() {\n            std::vector<future<>> futures;\n            visit_blobs(_blobs, make_visitor(\n                [&](const std::string_view&, const x509_simple& info) {\n                    _all_files.emplace(info.file.filename);\n                },\n                [&](const std::string_view&, const x509_key& info) {\n                    _all_files.emplace(info.cert_file.filename);\n                    _all_files.emplace(info.key_file.filename);\n                },\n                [&](const std::string_view&, const pkcs12_simple& info) {\n                    _all_files.emplace(info.file.filename);\n                }\n            ));\n            return parallel_for_each(_all_files, [this](auto& f) {\n                if (!f.empty()) {\n                    return add_watch(f).discard_result();\n                }\n                return make_ready_future<>();\n            }).finally([me = shared_from_this()] {});\n        }\n        void start() {\n            // run the loop in a thread. makes code almost readable.\n            (void)async(std::bind(&reloading_builder::run, this)).finally([me = shared_from_this()] {});\n        }\n        void run() {\n            while (_creds) {\n                try {\n                    auto events = _fsn.wait().get();\n                    if (events.empty() && _creds == nullptr) {\n                        return;\n                    }\n                    rebuild(events);\n                    _timer.cancel();\n                } catch (...) {\n                    if (!_timer.armed()) {\n                        _timer.set_callback([this, ep = std::current_exception()]() mutable {\n                            do_callback(std::move(ep));\n                        });\n                        _timer.arm(_delay);\n                    }\n                }\n            }\n        }\n        void detach() {\n            _creds = nullptr;\n            _cb = {};\n            _fsn.shutdown();\n            _timer.cancel();\n        }\n    private:\n        using fsnotifier = experimental::fsnotifier;\n\n        // called from seastar::thread\n        void rebuild(const std::vector<fsnotifier::event>& events) {\n            for (auto& e : events) {\n                // don't use at. We could be getting two events for\n                // same watch (mod + delete), but we only need to care\n                // about one...\n                auto i = _watches.find(e.id);\n                if (i != _watches.end()) {\n                    auto& filename = i->second.second;\n                    // only add actual file watches to\n                    // query set. If this was a directory\n                    // watch, the file should already be\n                    // in there.\n                    if (_all_files.count(filename)) {\n                        _files[filename] = e.mask;\n                    }\n                    _watches.erase(i);\n                }\n            }\n            auto num_changed = 0;\n\n            auto maybe_reload = [&](const sstring& filename, buffer_type& dst) {\n                if (filename.empty() || !_files.count(filename)) {\n                    return;\n                }\n                // #756\n                // first, add a watch to nearest parent dir we\n                // can find. If user deleted folders, we could end\n                // up looking at modifications to root.\n                // The idea is that should adding a watch to actual file\n                // fail (deleted file/folder), we wait for changes to closest\n                // parent. When this happens, we will retry all files\n                // that have not been successfully replaced (and maybe more),\n                // repeating the process. At some point, we hopefully\n                // get new, current data.\n                add_dir_watch(filename);\n                // #756 add watch _first_. File could change while we are\n                // reading this.\n                try {\n                    add_watch(filename).get();\n                } catch (...) {\n                    // let's just assume if this happens, it's because the file or folder was deleted.\n                    // just ignore for now, and hope the dir watch will tell us when it is back...\n                    return;\n                }\n                temporary_buffer<char> buf = read_fully(filename, \"reloading\").get();\n                dst = to_buffer(buf);\n                ++num_changed;\n            };\n            visit_blobs(_blobs, make_visitor(\n                [&](const std::string_view&, x509_simple& info) {\n                    maybe_reload(info.file.filename, info.data);\n                },\n                [&](const std::string_view&, x509_key& info) {\n                    maybe_reload(info.cert_file.filename, info.cert);\n                    maybe_reload(info.key_file.filename, info.key);\n                },\n                [&](const std::string_view&, pkcs12_simple& info) {\n                    maybe_reload(info.file.filename, info.data);\n                }\n            ));\n            // only try this if anything was in fact successfully loaded.\n            // if files were missing, or pairs incomplete, we can just skip.\n            if (num_changed == 0) {\n                return;\n            }\n            try {\n                // force rebuilding session resume mode key if\n                // enabled. should not reuse sessions across certificate\n                // change (should not work anyway)\n                set_session_resume_mode(_session_resume_mode);\n                if (_creds) {\n                    _creds->rebuild(*this);\n                }\n            } catch (...) {\n                if (std::any_of(_files.begin(), _files.end(), [](auto& p) { return p.second == fsnotifier::flags::ignored; })) {\n                    // if any file in the reload set was deleted - i.e. we have not seen a \"closed\" yet - assume\n                    // this is a spurious reload and we'd better wait for next event - hopefully a \"closed\" -\n                    // and try again\n                    return;\n                }\n                throw;\n            }\n            // if we got here, all files loaded, all watches were created,\n            // and gnutls was ok with the content. success.\n            do_callback();\n            on_success();\n        }\n        void on_success() {\n            _files.clear();\n            // remove all directory watches, since we've successfully\n            // reloaded -> the file watches themselves should suffice now\n            auto i = _watches.begin();\n            auto e = _watches.end();\n            while (i != e) {\n                if (!_all_files.count(i->second.second)) {\n                    i = _watches.erase(i);\n                    continue;\n                }\n                ++i;\n            }\n        }\n        void do_callback(std::exception_ptr ep = {}) {\n            if (_cb && !_files.empty()) {\n                _cb(*this, boost::copy_range<std::unordered_set<sstring>>(_files | boost::adaptors::map_keys), std::move(ep)).get();\n            }\n        }\n        // called from seastar::thread\n        fsnotifier::watch_token add_dir_watch(const sstring& filename) {\n            auto dir = std::filesystem::path(filename).parent_path();\n            for (;;) {\n                try {\n                    return add_watch(dir.native(), fsnotifier::flags::create_child | fsnotifier::flags::move).get();\n                } catch (...) {\n                    auto parent = dir.parent_path();\n                    if (parent.empty() || dir == parent) {\n                        throw;\n                    }\n                    dir = std::move(parent);\n                    continue;\n                }\n            }\n        }\n        future<fsnotifier::watch_token> add_watch(const sstring& filename, fsnotifier::flags flags = fsnotifier::flags::close_write|fsnotifier::flags::delete_self) {\n            return _fsn.create_watch(filename, flags).then([this, filename = filename](fsnotifier::watch w) {\n                auto t = w.token();\n                // we might create multiple watches for same token in case of dirs, avoid deleting previously\n                // created one\n                if (_watches.count(t)) {\n                    w.release();\n                } else {\n                    _watches.emplace(t, std::make_pair(std::move(w), filename));\n                }\n                return t;\n            });\n        }\n\n        reload_callback_ex _cb;\n        reloadable_credentials_base* _creds;\n        fsnotifier _fsn;\n        std::unordered_map<fsnotifier::watch_token, std::pair<fsnotifier::watch, sstring>> _watches;\n        std::unordered_map<sstring, fsnotifier::flags> _files;\n        std::unordered_set<sstring> _all_files;\n        timer<> _timer;\n        delay_type _delay;\n    };\n    reloadable_credentials_base(credentials_builder builder, reload_callback_ex cb, delay_type delay = default_tolerance)\n        : _builder(seastar::make_shared<reloading_builder>(std::move(builder), std::move(cb), this, delay))\n    {\n        _builder->start();\n    }\n    future<> init() {\n        return _builder->init();\n    }\n    virtual ~reloadable_credentials_base() {\n        _builder->detach();\n    }\n    virtual void rebuild(const credentials_builder&) = 0;\nprivate:\n    shared_ptr<reloading_builder> _builder;\n};\n\ntemplate<typename Base>\nclass tls::reloadable_credentials : public Base, public tls::reloadable_credentials_base {\npublic:\n    reloadable_credentials(credentials_builder builder, reload_callback_ex cb, Base b, delay_type delay = default_tolerance)\n        : Base(std::move(b))\n        , tls::reloadable_credentials_base(std::move(builder), std::move(cb), delay)\n    {}\n    void rebuild(const credentials_builder&) override;\n};\n\ntemplate<>\nvoid tls::reloadable_credentials<tls::certificate_credentials>::rebuild(const credentials_builder& builder) {\n    builder.rebuild(*this);\n}\n\ntemplate<>\nvoid tls::reloadable_credentials<tls::server_credentials>::rebuild(const credentials_builder& builder) {\n    builder.rebuild(*this);\n}\n\nvoid tls::credentials_builder::rebuild(certificate_credentials& creds) const {\n    auto tmp = build_certificate_credentials();\n    creds._impl = std::move(tmp->_impl);\n}\n\nvoid tls::credentials_builder::rebuild(server_credentials& creds) const {\n    auto tmp = build_server_credentials();\n    creds._impl = std::move(tmp->_impl);\n}\n\nfuture<shared_ptr<tls::certificate_credentials>> tls::credentials_builder::build_reloadable_certificate_credentials(reload_callback_ex cb, std::optional<std::chrono::milliseconds> tolerance) const {\n    auto creds = seastar::make_shared<reloadable_credentials<tls::certificate_credentials>>(*this, std::move(cb), std::move(*build_certificate_credentials()), tolerance.value_or(reloadable_credentials_base::default_tolerance));\n    return creds->init().then([creds] {\n        return make_ready_future<shared_ptr<tls::certificate_credentials>>(creds);\n    });\n}\n\nfuture<shared_ptr<tls::server_credentials>> tls::credentials_builder::build_reloadable_server_credentials(reload_callback_ex cb, std::optional<std::chrono::milliseconds> tolerance) const {\n    auto creds = seastar::make_shared<reloadable_credentials<tls::server_credentials>>(*this, std::move(cb), std::move(*build_server_credentials()), tolerance.value_or(reloadable_credentials_base::default_tolerance));\n    return creds->init().then([creds] {\n        return make_ready_future<shared_ptr<tls::server_credentials>>(creds);\n    });\n}\n\nfuture<shared_ptr<tls::certificate_credentials>> tls::credentials_builder::build_reloadable_certificate_credentials(reload_callback cb, std::optional<std::chrono::milliseconds> tolerance) const {\n    return build_reloadable_certificate_credentials([cb = std::move(cb)](const credentials_builder&, const std::unordered_set<sstring>& files, std::exception_ptr p) {\n        cb(files, p);\n        return make_ready_future<>();\n    }, tolerance);\n}\n\nfuture<shared_ptr<tls::server_credentials>> tls::credentials_builder::build_reloadable_server_credentials(reload_callback cb, std::optional<std::chrono::milliseconds> tolerance) const {\n    return build_reloadable_server_credentials([cb = std::move(cb)](const credentials_builder&, const std::unordered_set<sstring>& files, std::exception_ptr p) {\n        cb(files, p);\n        return make_ready_future<>();\n    }, tolerance);\n}\n\nnamespace tls {\n\n/**\n * Session wraps gnutls session, and is the\n * actual conduit for an TLS/SSL data flow.\n *\n * We use a connected_socket and its sink/source\n * for IO. Note that we need to keep ownership\n * of these, since we handle handshake etc.\n *\n */\nclass session : public enable_lw_shared_from_this<session> {\npublic:\n    enum class type\n        : uint32_t {\n            CLIENT = GNUTLS_CLIENT, SERVER = GNUTLS_SERVER,\n    };\n\n    session(type t, shared_ptr<tls::certificate_credentials> creds,\n            std::unique_ptr<net::connected_socket_impl> sock, tls_options options = {})\n            : _type(t), _sock(std::move(sock)), _creds(creds->_impl),\n                    _in(_sock->source()), _out(_sock->sink()),\n                    _in_sem(1), _out_sem(1), _options(std::move(options)), _output_pending(\n                    make_ready_future<>()), _session([t] {\n                gnutls_session_t session;\n                gtls_chk(gnutls_init(&session, GNUTLS_NONBLOCK|uint32_t(t)));\n                return session;\n            }(), &gnutls_deinit) {\n        gtls_chk(gnutls_set_default_priority(*this));\n        gtls_chk(\n                gnutls_credentials_set(*this, GNUTLS_CRD_CERTIFICATE,\n                        *_creds));\n        if (_type == type::SERVER) {\n            switch (_creds->get_client_auth()) {\n                case client_auth::NONE:\n                default:\n                    gnutls_certificate_server_set_request(*this, GNUTLS_CERT_IGNORE);\n                    break;\n                case client_auth::REQUEST:\n                    gnutls_certificate_server_set_request(*this, GNUTLS_CERT_REQUEST);\n                    break;\n                case client_auth::REQUIRE:\n                    gnutls_certificate_server_set_request(*this, GNUTLS_CERT_REQUIRE);\n                    break;\n            }\n            // Maybe set up server session ticket support\n            switch (_creds->get_session_resume_mode()) {\n                case session_resume_mode::NONE:\n                default:\n                    break;\n                case session_resume_mode::TLS13_SESSION_TICKET:\n                    gnutls_session_ticket_enable_server(*this, _creds->get_session_resume_key());\n                    break;\n            }\n        }\n\n        auto prio = _creds->get_priority();\n        if (prio) {\n            gtls_chk(gnutls_priority_set(*this, prio));\n        }\n\n        gnutls_transport_set_ptr(*this, this);\n        gnutls_transport_set_vec_push_function(*this, &vec_push_wrapper);\n        gnutls_transport_set_pull_function(*this, &pull_wrapper);\n\n        // This would be nice, because we preferably want verification to\n        // abort hand shake so peer immediately knows we bailed...\n#if GNUTLS_VERSION_NUMBER >= 0x030406\n        if (_type == type::CLIENT) {\n            gnutls_session_set_verify_function(*this, &verify_wrapper);\n        }\n#endif\n        // if we are a client, check if we have a session ticket to unpack.\n        if (_type == type::CLIENT && !_options.session_resume_data.empty()) {\n            gtls_chk(gnutls_session_set_data(*this, _options.session_resume_data.data(), _options.session_resume_data.size()));\n        }\n        _options.session_resume_data.clear(); // no need to keep around\n\n        // ALPN setup\n        auto& alpn_protocols = _type == type::CLIENT ? _options.alpn_protocols : _creds->_alpn_protocols;\n        if (!alpn_protocols.empty()) {\n            std::vector<gnutls_datum_t> alpn_datums;\n            alpn_datums.reserve(alpn_protocols.size());\n            for (const auto& p_str : alpn_protocols) {\n                alpn_datums.push_back({const_cast<unsigned char*>(reinterpret_cast<const unsigned char*>(p_str.data())),\n                                       static_cast<unsigned int>(p_str.size())});\n            }\n            gtls_chk(gnutls_alpn_set_protocols(*this, alpn_datums.data(), alpn_datums.size(), 0));\n        }\n    }\n    session(type t, shared_ptr<certificate_credentials> creds,\n            connected_socket sock,\n            tls_options options = {})\n            : session(t, std::move(creds), net::get_impl::get(std::move(sock)),\n                      std::move(options)) {\n    }\n\n    ~session() {\n        SEASTAR_ASSERT(_output_pending.available());\n    }\n\n    typedef temporary_buffer<char> buf_type;\n\n    sstring cert_status_to_string(gnutls_certificate_type_t type, unsigned int status) {\n        gnutls_datum_t out;\n        gtls_chk(\n                gnutls_certificate_verification_status_print(status, type, &out,\n                        0));\n        sstring s(reinterpret_cast<const char *>(out.data), out.size);\n        gnutls_free(out.data);\n        return s;\n    }\n\n    future<> send_alert(gnutls_alert_level_t level, gnutls_alert_description_t desc) {\n        return repeat([this, level, desc]() {\n            auto res = gnutls_alert_send(*this, level, desc);\n            switch(res) {\n            case GNUTLS_E_SUCCESS:\n                return wait_for_output().then([] {\n                    return make_ready_future<stop_iteration>(stop_iteration::yes);\n                });\n            case GNUTLS_E_AGAIN:\n            case GNUTLS_E_INTERRUPTED:\n                return wait_for_output().then([] {\n                    return make_ready_future<stop_iteration>(stop_iteration::no);\n                });\n            default:\n                return handle_output_error(res).then([] {\n                    return make_ready_future<stop_iteration>(stop_iteration::yes);\n                });\n            }\n        });\n    }\n\n    future<> do_handshake_invoke(int (*func)(gnutls_session_t)) {\n        if (_type == type::CLIENT && !_options.server_name.empty()) {\n            gnutls_server_name_set(*this, GNUTLS_NAME_DNS, _options.server_name.data(), _options.server_name.size());\n        }\n        try {\n            auto res = func(*this);\n            if (res < 0) {\n                switch (res) {\n                case GNUTLS_E_AGAIN:\n                    // #453 always wait for output first.\n                    // If none is pending, it should be a no-op\n                {\n                    int dir = gnutls_record_get_direction(*this);\n                    return wait_for_output().then([this, dir, func] {\n                        // we actually E_AGAIN:ed in a write. Don't\n                        // wait for input.\n                        if (dir == 1) {\n                            return do_handshake_invoke(func);\n                        }\n                        return wait_for_input().then([this, func] {\n                            return do_handshake_invoke(func);\n                        });\n                    });\n                }\n                case GNUTLS_E_NO_CERTIFICATE_FOUND:\n                    return make_exception_future<>(verification_error(\"No certificate was found\"));\n#if GNUTLS_VERSION_NUMBER >= 0x030406\n                case GNUTLS_E_CERTIFICATE_ERROR:\n                    verify(); // should throw. otherwise, fallthrough\n                    [[fallthrough]];\n#endif\n                default:\n                    // Send the handshake error returned by gnutls_handshake()\n                    // to the client, as an alert. For example, if the protocol\n                    // version requested by the client is not supported, the\n                    // \"protocol_version\" alert is sent.\n                    auto alert = gnutls_alert_description_t(gnutls_error_to_alert(res, NULL));\n                    return handle_output_error(res).then_wrapped([this, alert = std::move(alert)] (future<> output_future) {\n                        return send_alert(GNUTLS_AL_FATAL, alert).then_wrapped([output_future = std::move(output_future)] (future<> f) mutable {\n                            // Return to the caller the original handshake error.\n                            // If send_alert() *also* failed, ignore that.\n                            f.ignore_ready_future();\n                            return std::move(output_future);\n                        });\n                    });\n                }\n            }\n            if (_type == type::CLIENT || _creds->get_client_auth() != client_auth::NONE) {\n                verify();\n            }\n            _connected = true;\n            // make sure we reset output_pending\n            return wait_for_output();\n        } catch (...) {\n            return make_exception_future<>(std::current_exception());\n        }\n    }\n    future<> do_handshake() {\n        if (_connected) {\n            return make_ready_future<>();\n        }\n        return do_handshake_invoke(&gnutls_handshake);\n    }\n    future<> do_handshake_sync(future<> (session::*func)()) {\n        // acquire both semaphores to sync both read & write\n        return with_semaphore(_in_sem, 1, [this, func] {\n            return with_semaphore(_out_sem, 1, [this, func] {\n                return std::invoke(func, this).handle_exception([this](auto ep) {\n                    if (!_error) {\n                        _error = ep;\n                    }\n                    return make_exception_future<>(_error);\n                });\n            });\n        });\n    }\n    future<> do_force_rehandshake() {\n        return do_handshake_invoke(gnutls_rehandshake);\n    }\n    future<> force_rehandshake() {\n        if (!_connected) {\n            return handshake();\n        }\n        // Note: only applicable to server.\n        if (_type == type::CLIENT) {\n            throw std::system_error(GNUTLS_E_INVALID_REQUEST, error_category(), \"re-handshake only applicable for server socket\");\n        }\n        return do_handshake_sync(&session::do_force_rehandshake);\n    }\n\n    future<> handshake() {\n        // maybe load system certificates before handshake, in case we\n        // have not done so yet...\n        if (_creds->need_load_system_trust()) {\n            return _creds->maybe_load_system_trust().then([this] {\n               return handshake();\n            });\n        }\n        return do_handshake_sync(&session::do_handshake);\n    }\n\n    size_t in_avail() const {\n        return _input.size();\n    }\n    bool eof() const {\n        return _eof;\n    }\n    future<> wait_for_input() {\n        if (!_input.empty()) {\n            return make_ready_future<>();\n        }\n        return _in.get().then([this](buf_type buf) {\n            _eof |= buf.empty();\n           _input = std::move(buf);\n        }).handle_exception([this](auto ep) {\n           _error = ep;\n           return make_exception_future(ep);\n        });\n    }\n    future<> wait_for_output() {\n        return std::exchange(_output_pending, make_ready_future()).handle_exception([this](auto ep) {\n           _error = ep;\n           return make_exception_future(ep);\n        });\n    }\n\n    static session * from_transport_ptr(gnutls_transport_ptr_t ptr) {\n        return static_cast<session *>(ptr);\n    }\n#if GNUTLS_VERSION_NUMBER >= 0x030406\n    static int verify_wrapper(gnutls_session_t gs) {\n        try {\n            from_transport_ptr(gnutls_transport_get_ptr(gs))->verify();\n            return 0;\n        } catch (...) {\n            return GNUTLS_E_CERTIFICATE_ERROR;\n        }\n    }\n#endif\n    static ssize_t vec_push_wrapper(gnutls_transport_ptr_t ptr, const giovec_t * iov, int iovcnt) {\n        return from_transport_ptr(ptr)->vec_push(iov, iovcnt);\n    }\n    static ssize_t pull_wrapper(gnutls_transport_ptr_t ptr, void* dst, size_t len) {\n        return from_transport_ptr(ptr)->pull(dst, len);\n    }\n\n    void verify() {\n        if (!_creds->_enable_certificate_verification) {\n            return;\n        }\n\n        unsigned int status;\n        auto res = gnutls_certificate_verify_peers3(*this, _type != type::CLIENT || _options.server_name.empty()\n                        ? nullptr : _options.server_name.c_str(), &status);\n        if (res == GNUTLS_E_NO_CERTIFICATE_FOUND && _type != type::CLIENT && _creds->get_client_auth() != client_auth::REQUIRE) {\n            return;\n        }\n        if (res < 0) {\n            throw std::system_error(res, error_category());\n        }\n        if (status & GNUTLS_CERT_INVALID) {\n            auto stat_str = cert_status_to_string(gnutls_certificate_type_get(*this), status);\n            auto dn = extract_dn_information();\n\n            // If possible, include issuer/subject info on the cert that failed verification.\n            if (dn) {\n                std::stringstream ss;\n                ss << stat_str;\n                if (stat_str.back() != ' ') {\n                    ss << ' ';\n                }\n                ss << \"(Issuer=[\" << dn->issuer << \"], Subject=[\" << dn->subject << \"])\";\n                stat_str = ss.str();\n            }\n            throw verification_error(stat_str);\n        }\n        if (_creds->_dn_callback) {\n            // if the user registered a DN (Distinguished Name) callback\n            // then extract subject and issuer from the (leaf) peer certificate and invoke the callback\n\n            auto dn = extract_dn_information();\n            SEASTAR_ASSERT(dn.has_value()); // otherwise we couldn't have gotten here\n\n            // a switch here might look overelaborate, however,\n            // the compiler will warn us if someone alters the definition of type\n            session_type t;\n            switch (_type) {\n            case type::CLIENT:\n                t = session_type::CLIENT;\n                break;\n            case type::SERVER:\n                t = session_type::SERVER;\n                break;\n            }\n\n            _creds->_dn_callback(t, std::move(dn->subject), std::move(dn->issuer));\n        }\n    }\n\n    future<temporary_buffer<char>> get() {\n        if (_error) {\n            return make_exception_future<temporary_buffer<char>>(_error);\n        }\n        if (_shutdown || eof()) {\n            return make_ready_future<temporary_buffer<char>>();\n        }\n        if (!_connected) {\n            return handshake().then(std::bind(&session::get, this));\n        }\n        return with_semaphore(_in_sem, 1, std::bind(&session::do_get, this)).then([this](temporary_buffer<char> buf) {\n            if (buf.empty() && !eof()) {\n                // this must mean we got a re-handshake request.\n                // see do_get.\n                // We there clear connected flag and return empty in case\n                // other side requests re-handshake. Now, someone else could have already dealt with it by the\n                // time we are here (continuation reordering). In fact, someone could have dealt with\n                // it and set the eof flag also, but in that case we're still eof...\n                return handshake().then(std::bind(&session::get, this));\n            }\n            return make_ready_future<temporary_buffer<char>>(std::move(buf));\n        });\n    }\n\n    future<temporary_buffer<char>> do_get() {\n        // gnutls might have stuff in its buffers.\n        auto avail = gnutls_record_check_pending(*this);\n        if (avail == 0) {\n            // or we might...\n            avail = in_avail();\n        }\n        if (avail != 0) {\n            // typically, unencrypted data can get smaller (padding),\n            // but not larger.\n            temporary_buffer<char> buf(avail);\n            auto n = gnutls_record_recv(*this, buf.get_write(), buf.size());\n            if (n < 0) {\n                switch (n) {\n                case GNUTLS_E_AGAIN:\n                    // Assume we got this because we read to little underlying\n                    // data to finish a tls packet\n                    // Our input buffer should be empty now, so just go again\n                    return do_get();\n                case GNUTLS_E_REHANDSHAKE:\n                    // server requests new HS. must release semaphore, so set new state\n                    // and return nada.\n                    _connected = false;\n                    return make_ready_future<temporary_buffer<char>>();\n                default:\n                    _error = std::make_exception_ptr(std::system_error(n, error_category()));\n                    return make_exception_future<temporary_buffer<char>>(_error);\n                }\n            }\n            buf.trim(n);\n            if (n == 0) {\n                _eof = true;\n            }\n            return make_ready_future<temporary_buffer<char>>(std::move(buf));\n        }\n        if (eof()) {\n            return make_ready_future<temporary_buffer<char>>();\n        }\n        // No input? wait for out buffers to fill...\n        return wait_for_input().then([this] {\n            return do_get();\n        });\n    }\n\nprivate:\n    future<> do_put(std::vector<temporary_buffer<char>> bufs) {\n        auto i = bufs.begin();\n        auto e = bufs.end();\n        return with_semaphore(_out_sem, 1, [this, i, e] {\n            SEASTAR_ASSERT(_output_pending.available());\n            return do_for_each(i, e, [this](temporary_buffer<char>& b) {\n                return do_put_one(b.get(), b.size());\n            });\n        }).finally([b = std::move(bufs)] {});\n    }\n\n    future<> do_put(temporary_buffer<char> buf) {\n        auto ptr = buf.get();\n        auto size = buf.size();\n        return with_semaphore(_out_sem, 1, [this, ptr, size] {\n            SEASTAR_ASSERT(_output_pending.available());\n            return do_put_one(ptr, size);\n        }).finally([b = std::move(buf)] {});\n    }\n\n    future<> do_put_one(const char* ptr, size_t size) {\n        // #2859\n        // Normally, gnutls_record_send will break up data into gnutls_record_get_max_size()\n        // sized chunks for us (only processing the first 16k or so of provided buffer).\n        // However, if the session is in a re-handshake state, gnutls will\n        // make an intermediate alloc to prepend the requested session key\n        // to the sent data. This can cause large alloc warnings.\n        // To avoid this, we explicitly break the message into\n        // block sized parts (same as normal case in gnutls)\n        auto max_record_len = gnutls_record_get_max_size(*this);\n\n        size_t off = 0; // here to appease eclipse cdt\n        return repeat([this, ptr, size, off, max_record_len]() mutable {\n            if (off == size) {\n                return make_ready_future<stop_iteration>(stop_iteration::yes);\n            }\n            auto n = std::min(max_record_len, size - off);\n            auto res = gnutls_record_send(*this, ptr + off, n);\n            if (res > 0) { // don't really need to check, but...\n                off += res;\n            }\n            // what will we wait for? error or results...\n            // NOTE: we _can_ get an EAGAIN here since the\n            // addition of force_rehandshake ability (and possibly before)\n            // due to the tls buffering. Just wait + retrying should work\n            // in all cases.\n            auto f = res < 0  && res != GNUTLS_E_AGAIN\n                ? handle_output_error(res)\n                : wait_for_output()\n                ;\n            return f.then([] {\n                return make_ready_future<stop_iteration>(stop_iteration::no);\n            });\n        });\n    }\n\npublic:\n    future<> put(std::span<temporary_buffer<char>> bufs) {\n        if (_error) {\n            return make_exception_future<>(_error);\n        }\n        if (_shutdown) {\n            return make_exception_future<>(std::system_error(EPIPE, std::system_category()));\n        }\n        if (!_connected) {\n            std::vector<temporary_buffer<char>> p;\n            p.reserve(bufs.size());\n            p.insert(p.end(), std::make_move_iterator(bufs.begin()), std::make_move_iterator(bufs.end()));\n            return handshake().then([this, p = std::move(p)]() mutable {\n               return put(std::span(p));\n            });\n        }\n\n        if (bufs.size() == 1) {\n            return do_put(std::move(bufs.front()));\n        }\n\n        // We want to make sure that we call gnutls_record_send with as large\n        // packets as possible. This is because each call to gnutls_record_send\n        // translates to a sendmsg syscall. Further it results in larger TLS\n        // records which makes encryption/decryption faster. Hence to avoid\n        // cases where we would do an extra syscall for something like a 100\n        // bytes header we linearize the packet if it's below the max TLS record\n        // size.\n\n        size_t size = std::accumulate(bufs.begin(), bufs.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n        if (size <= gnutls_record_get_max_size(*this)) {\n            temporary_buffer<char> linear(size);\n            char* pos = linear.get_write();\n            for (auto& buf : bufs) {\n                std::copy_n(buf.get(), buf.size(), pos);\n                pos += buf.size();\n            }\n            return do_put(std::move(linear));\n        }\n\n        std::vector<temporary_buffer<char>> p;\n        p.reserve(bufs.size());\n        p.insert(p.end(), std::make_move_iterator(bufs.begin()), std::make_move_iterator(bufs.end()));\n        return do_put(std::move(p));\n    }\n\n    ssize_t pull(void* dst, size_t len) {\n        if (eof()) {\n            return 0;\n        }\n        // If we have data in buffers, we can complete.\n        // Otherwise, we must be conservative.\n        if (_input.empty()) {\n            gnutls_transport_set_errno(*this, EAGAIN);\n            return -1;\n        }\n        auto n = std::min(len, _input.size());\n        memcpy(dst, _input.get(), n);\n        _input.trim_front(n);\n        return n;\n    }\n    ssize_t vec_push(const giovec_t * iov, int iovcnt) {\n        if (!_output_pending.available()) {\n            gnutls_transport_set_errno(*this, EAGAIN);\n            return -1;\n        }\n        try {\n            ssize_t n; // Set on the good path and unused on the bad path\n\n            if (!_output_pending.failed()) {\n                std::vector<temporary_buffer<char>> msg;\n                msg.reserve(iovcnt);\n                n = 0;\n                for (int i = 0; i < iovcnt; ++i) {\n                    msg.emplace_back(reinterpret_cast<const char*>(iov[i].iov_base), iov[i].iov_len);\n                    n += iov[i].iov_len;\n                }\n                _output_pending = _out.put(std::move(msg));\n            }\n            if (_output_pending.failed()) {\n                // exception is copied back into _output_pending\n                // by the catch handlers below\n                std::rethrow_exception(_output_pending.get_exception());\n            }\n            return n;\n        } catch (const std::system_error& e) {\n            gnutls_transport_set_errno(*this, e.code().value());\n            _output_pending = make_exception_future<>(std::current_exception());\n        } catch (...) {\n            gnutls_transport_set_errno(*this, EIO);\n            _output_pending = make_exception_future<>(std::current_exception());\n        }\n        return -1;\n    }\n\n    operator gnutls_session_t() const {\n        return _session.get();\n    }\n\n    future<>\n    handle_error(int res) {\n        _error = std::make_exception_ptr(std::system_error(res, error_category()));\n        return make_exception_future(_error);\n    }\n    future<>\n    handle_output_error(int res) {\n        _error = std::make_exception_ptr(std::system_error(res, error_category()));\n        // #453\n        // defensively wait for output before generating the error.\n        // if we have both error code and an exception in output\n        // future, throw both.\n        return wait_for_output().then_wrapped([this, res](auto f) {\n            try {\n                f.get();\n                // output was ok/done, just generate error code exception\n                return make_exception_future(_error);\n            } catch (...) {\n                std::throw_with_nested(std::system_error(res, error_category()));\n            }\n        });\n    }\n    future<> do_shutdown() {\n        if (_error || !_connected) {\n            return make_ready_future();\n        }\n        auto res = gnutls_bye(*this, GNUTLS_SHUT_WR);\n        if (res < 0) {\n            switch (res) {\n            case GNUTLS_E_AGAIN:\n                // We only send \"bye\" alert, letting a \"normal\" (either pending, or subsequent)\n                // read deal with reading the expected EOF alert.\n                SEASTAR_ASSERT(gnutls_record_get_direction(*this) == 1);\n                return wait_for_output().then([this] {\n                    return do_shutdown();\n                });\n            default:\n                return handle_output_error(res);\n            }\n        }\n        return wait_for_output();\n    }\n    future<> wait_for_eof() {\n        // read records until we get an eof alert\n        // since this call could time out, we must not ac\n        return with_semaphore(_in_sem, 1, [this] {\n            if (_error || !_connected) {\n                return make_ready_future();\n            }\n            if (!_options.wait_for_data_on_shutdown) {\n                if (eof()) {\n                    return make_ready_future<>();\n                }\n                return do_get().discard_result();\n            }\n\n            return repeat([this] {\n                if (eof()) {\n                    return make_ready_future<stop_iteration>(stop_iteration::yes);\n                }\n                return do_get().then([](auto buf) {\n                   return make_ready_future<stop_iteration>(stop_iteration::no);\n                });\n            });\n        });\n    }\n    future<> shutdown() {\n        // first, make sure any pending write is done.\n        // bye handshake is a flush operation, but this\n        // allows us to not pay extra attention to output state\n        //\n        // we only send a simple \"bye\" alert packet. Then we\n        // read from input until we see EOF. Any other reader\n        // before us will get it instead of us, and mark _eof = true\n        // in which case we will be no-op.\n        return with_semaphore(_out_sem, 1,\n                        std::bind(&session::do_shutdown, this)).then(\n                        std::bind(&session::wait_for_eof, this)).finally([me = shared_from_this()] {});\n        // note moved finally clause above. It is theorethically possible\n        // that we could complete do_shutdown just before the close calls\n        // below, get pre-empted, have \"close()\" finish, get freed, and\n        // then call wait_for_eof on stale pointer.\n    }\n    void close() noexcept {\n        // only do once.\n        if (!std::exchange(_shutdown, true)) {\n            auto me = shared_from_this();\n            auto f = _options.bye_timeout.count() > 0 && (_options.wait_for_eof_on_shutdown._value == true)\n                // try to bye-handshake us nicely, but after a timeout we forcefully close.\n                ? with_timeout(timer<>::clock::now() + _options.bye_timeout, shutdown())\n                : make_ready_future<>();\n            engine().run_in_background(std::move(f).finally([this] {\n                _eof = true;\n                try {\n                    (void)_in.close().handle_exception([](std::exception_ptr) {}); // should wake any waiters\n                } catch (...) {\n                }\n                try {\n                    (void)_out.close().handle_exception([](std::exception_ptr) {});\n                } catch (...) {\n                }\n                // make sure to wait for handshake attempt to leave semaphores. Must be in same order as\n                // handshake aqcuire, because in worst case, we get here while a reader is attempting\n                // re-handshake.\n                return with_semaphore(_in_sem, 1, [this] {\n                    return with_semaphore(_out_sem, 1, [] {});\n                });\n            }).then_wrapped([me = std::move(me)](future<> f) { // must keep object alive until here.\n                f.ignore_ready_future();\n            }));\n        }\n    }\n    // helper for sink\n    future<> flush() noexcept {\n        return with_semaphore(_out_sem, 1, [this] {\n            return _out.flush();\n        });\n    }\n\n    seastar::net::connected_socket_impl& socket() const {\n        return *_sock;\n    }\n\n    // helper routine.\n    template<typename Func, typename... Args>\n    auto state_checked_access(Func&& f, Args&& ...args) {\n        using future_type = typename futurize<std::invoke_result_t<Func, Args...>>::type;\n        using result_t = typename future_type::value_type;\n        if (_error) {\n            return make_exception_future<result_t>(_error);\n        }\n        if (_shutdown) {\n            return make_exception_future<result_t>(std::system_error(ENOTCONN, std::system_category()));\n        }\n        if (!_connected) {\n            return handshake().then([this, f = std::move(f), ...args = std::forward<Args>(args)]() mutable {\n                // always recurse, in case malicious api caller does a shutdown while the above handshake is\n                // happening. I.e. misuses the api.\n                return session::state_checked_access(std::move(f), std::forward<Args>(args)...);\n            });\n        }\n        return futurize_invoke(f, std::forward<Args>(args)...);\n    }\n\n    future<bool> is_resumed() {\n        return state_checked_access([this] {\n            return gnutls_session_is_resumed(*this) != 0;\n        });\n    }\n    future<session_data> get_session_resume_data() {\n        return state_checked_access([this] {\n            /**\n             * Session ticket data is not available just because handshake\n             * was done. First off, of course other part must support it,\n             * but we also (mostly?) need to actually transfer data before\n             * the ticket is received.\n             *\n             * Check session flags so we can return no data in the case\n             * none is avail. Gnutls returns a 4-byte \"empty marker\"\n             * on none avail.\n            */\n            auto flags = gnutls_session_get_flags(*this);\n            if ((flags & GNUTLS_SFLAGS_SESSION_TICKET) == 0) {\n                return session_data{};\n            }\n            gnutls_datum tmp;\n            gtls_chk(gnutls_session_get_data2(*this, &tmp));\n            return session_data(tmp.data, tmp.data + tmp.size);\n        });\n    }\n    future<std::optional<session_dn>> get_distinguished_name() {\n        return state_checked_access([this] {\n            return extract_dn_information();\n        });\n    }\n    future<std::vector<subject_alt_name>> get_alt_name_information(std::unordered_set<subject_alt_name_type> types) {\n        return state_checked_access([this](std::unordered_set<subject_alt_name_type> types) {\n            std::vector<subject_alt_name> res;\n\n            auto peer = get_peer_certificate();\n            if (!peer) {\n                return res;\n            }\n\n        \tfor (auto i = 0u; ; i++) {\n                size_t size = 0;\n\n                auto err = gnutls_x509_crt_get_subject_alt_name(peer.get(), i, nullptr, &size, nullptr);\n\n                if (err == GNUTLS_E_REQUESTED_DATA_NOT_AVAILABLE) {\n                    break;\n                }\n                if (err != GNUTLS_E_SHORT_MEMORY_BUFFER) {\n                    gtls_chk(err); // will throw\n                }\n                sstring buf;\n                buf.resize(size);\n\n                err = gnutls_x509_crt_get_subject_alt_name(peer.get(), i, buf.data(), &size, nullptr);\n                if (err < 0) {\n                    gtls_chk(err); // will throw\n                }\n\n                static_assert(int(subject_alt_name_type::dnsname) == GNUTLS_SAN_DNSNAME);\n                static_assert(int(subject_alt_name_type::rfc822name) == GNUTLS_SAN_RFC822NAME);\n                static_assert(int(subject_alt_name_type::uri) == GNUTLS_SAN_URI);\n                static_assert(int(subject_alt_name_type::ipaddress) == GNUTLS_SAN_IPADDRESS);\n                static_assert(int(subject_alt_name_type::othername) == GNUTLS_SAN_OTHERNAME);\n                static_assert(int(subject_alt_name_type::dn) == GNUTLS_SAN_DN);\n\n                subject_alt_name v;\n\n                v.type = subject_alt_name_type(err);\n\n                if (!types.empty() && !types.count(v.type)) {\n                    continue;\n                }\n\n                switch (v.type) {\n                    case subject_alt_name_type::ipaddress:\n                    {\n                        union {\n                            char c;\n                            ::in_addr in;\n                            ::in6_addr in6;\n                        } tmp;\n\n                        memcpy(&tmp.c, buf.data(), size);\n                        if (size == sizeof(::in_addr)) {\n                            v.value = net::inet_address(tmp.in);\n                        } else if (size == sizeof(::in6_addr)) {\n                            v.value = net::inet_address(tmp.in6);\n                        } else {\n                            throw std::runtime_error(fmt::format(\"Unexpected size {} for ipaddress alt name value\", size));\n                        }\n                        break;\n                    }\n                    default:\n                        // data we get back is null-terminated.\n                        while (buf.back() == 0) {\n                            buf.resize(buf.size() - 1);\n                        }\n                        v.value = std::move(buf);\n                        break;\n                }\n\n                res.emplace_back(std::move(v));\n        \t}\n            return res;\n        }, std::move(types));\n    }\n\n    future<std::vector<certificate_data>> get_peer_certificate_chain() {\n        return state_checked_access([this] {\n            unsigned int list_size = 0;\n            const gnutls_datum_t* client_cert_list = gnutls_certificate_get_peers(*this, &list_size);\n            auto res = std::vector<certificate_data>{};\n            res.reserve(list_size);\n            if (client_cert_list) {\n                for (auto const& client_cert : std::span{client_cert_list, list_size}) {\n                    res.emplace_back(client_cert.size);\n                    std::copy_n(client_cert.data, client_cert.size, res.back().data());\n                }\n            }\n            return res;\n        });\n    }\n\n    future<std::optional<sstring>> get_selected_alpn_protocol() {\n        return state_checked_access([this]() -> std::optional<sstring> {\n            gnutls_datum_t selected_proto_datum = { nullptr, 0 };\n            int rv = gnutls_alpn_get_selected_protocol(*this, &selected_proto_datum);\n            if (rv != 0) {\n                return std::nullopt;\n            }\n            return {{reinterpret_cast<const char*>(selected_proto_datum.data), selected_proto_datum.size}};\n        });\n    }\n\n    struct session_ref;\n\n    future<sstring> get_cipher_suite() {\n        return state_checked_access([this] {\n            return sstring(gnutls_ciphersuite_get(*this));\n        });\n    }\n\n    future<sstring> get_protocol_version() {\n        return state_checked_access([this]() {\n            return sstring(gnutls_protocol_get_name(gnutls_protocol_get_version(*this)));\n        });\n    }\n\nprivate:\n\n    using x509_ctr_ptr = std::unique_ptr<gnutls_x509_crt_int, void (*)(gnutls_x509_crt_t)>;\n\n    x509_ctr_ptr get_peer_certificate() const {\n        unsigned int list_size = 0;\n        const gnutls_datum_t* client_cert_list = gnutls_certificate_get_peers(*this, &list_size);\n        if (client_cert_list && list_size > 0) {\n            gnutls_x509_crt_t peer_leaf_cert = nullptr;\n            gtls_chk(gnutls_x509_crt_init(&peer_leaf_cert));\n\n            x509_ctr_ptr res(peer_leaf_cert, &gnutls_x509_crt_deinit);\n            gtls_chk(gnutls_x509_crt_import(peer_leaf_cert, &(client_cert_list[0]), GNUTLS_X509_FMT_DER));\n            return res;\n        }\n        return x509_ctr_ptr(nullptr, &gnutls_x509_crt_deinit);\n    }\n\n    std::optional<session_dn> extract_dn_information() const {\n        auto peer_leaf_cert = get_peer_certificate();\n        if (!peer_leaf_cert) {\n            return std::nullopt;\n        }\n        auto [ec, subject] = get_gtls_string(gnutls_x509_crt_get_dn, peer_leaf_cert.get());\n        auto [ec2, issuer] = get_gtls_string(gnutls_x509_crt_get_issuer_dn, peer_leaf_cert.get());\n        if (ec || ec2) {\n            throw std::runtime_error(\"error while extracting certificate DN strings\");\n        }\n        return session_dn{.subject=subject, .issuer=issuer};\n    }\n\n    type _type;\n\n    std::unique_ptr<net::connected_socket_impl> _sock;\n    shared_ptr<tls::certificate_credentials::impl> _creds;\n    data_source _in;\n    data_sink _out;\n\n    semaphore _in_sem, _out_sem;\n\n    tls_options _options;\n\n    bool _eof = false;\n    bool _shutdown = false;\n    bool _connected = false;\n    std::exception_ptr _error;\n\n    future<> _output_pending;\n    buf_type _input;\n\n    // modify this to a unique_ptr to handle exceptions in our constructor.\n    std::unique_ptr<std::remove_pointer_t<gnutls_session_t>, void(*)(gnutls_session_t)> _session;\n};\n\nstruct session::session_ref {\n    session_ref() = default;\n    session_ref(lw_shared_ptr<session> session)\n                    : _session(std::move(session)) {\n    }\n    session_ref(session_ref&&) = default;\n    session_ref(const session_ref&) = default;\n    ~session_ref() {\n        // This is not super pretty. But we take some care to only own sessions\n        // through session_ref, and we need to initiate shutdown on \"last owner\",\n        // since we cannot revive the session in destructor.\n        if (_session && _session.use_count() == 1) {\n            _session->close();\n        }\n    }\n\n    session_ref& operator=(session_ref&&) = default;\n    session_ref& operator=(const session_ref&) = default;\n\n    lw_shared_ptr<session> _session;\n};\n\nclass tls_connected_socket_impl : public net::connected_socket_impl, public session::session_ref {\npublic:\n    tls_connected_socket_impl(session_ref&& sess)\n        : session_ref(std::move(sess))\n    {}\n\n    class source_impl;\n    class sink_impl;\n\n    using net::connected_socket_impl::source;\n    data_source source() override;\n    data_sink sink() override;\n\n    void shutdown_input() override {\n        _session->close();\n    }\n    void shutdown_output() override {\n        _session->close();\n    }\n    void set_nodelay(bool nodelay) override {\n        _session->socket().set_nodelay(nodelay);\n    }\n    bool get_nodelay() const override {\n        return _session->socket().get_nodelay();\n    }\n    void set_keepalive(bool keepalive) override {\n        _session->socket().set_keepalive(keepalive);\n    }\n    bool get_keepalive() const override {\n        return _session->socket().get_keepalive();\n    }\n    void set_keepalive_parameters(const net::keepalive_params& p) override {\n        _session->socket().set_keepalive_parameters(p);\n    }\n    net::keepalive_params get_keepalive_parameters() const override {\n        return _session->socket().get_keepalive_parameters();\n    }\n    void set_sockopt(int level, int optname, const void* data, size_t len) override {\n        _session->socket().set_sockopt(level, optname, data, len);\n    }\n    int get_sockopt(int level, int optname, void* data, size_t len) const override {\n        return _session->socket().get_sockopt(level, optname, data, len);\n    }\n    socket_address local_address() const noexcept override {\n        return _session->socket().local_address();\n    }\n    socket_address remote_address() const noexcept override {\n        return _session->socket().remote_address();\n    }\n    future<std::optional<session_dn>> get_distinguished_name() {\n        return _session->get_distinguished_name();\n    }\n    future<std::vector<subject_alt_name>> get_alt_name_information(std::unordered_set<subject_alt_name_type> types) {\n        return _session->get_alt_name_information(std::move(types));\n    }\n    future<std::vector<certificate_data>> get_peer_certificate_chain() {\n        return _session->get_peer_certificate_chain();\n    }\n    future<> wait_input_shutdown() override {\n        return _session->socket().wait_input_shutdown();\n    }\n    future<bool> check_session_is_resumed() {\n        return _session->is_resumed();\n    }\n    future<session_data> get_session_resume_data() {\n        return _session->get_session_resume_data();\n    }\n    future<std::optional<sstring>> get_selected_alpn_protocol() {\n        return _session->get_selected_alpn_protocol();\n    }\n    future<sstring> get_cipher_suite() const {\n        return _session->get_cipher_suite();\n    }\n    future<sstring> get_protocol_version() const {\n        return _session->get_protocol_version();\n    }\n    future<> force_rehandshake() {\n        return _session->force_rehandshake();\n    }\n};\n\n\nclass tls_connected_socket_impl::source_impl: public data_source_impl, public session::session_ref {\npublic:\n    using session_ref::session_ref;\nprivate:\n    future<temporary_buffer<char>> get() override {\n        return _session->get();\n    }\n    future<> close() override {\n        _session->close();\n        return make_ready_future<>();\n    }\n};\n\n// Note: source/sink, and by extension, the in/out streams\n// produced, cannot exist outside the direct life span of\n// the connected_socket itself. This is consistent with\n// other sockets in seastar, though I am than less fond of it...\nclass tls_connected_socket_impl::sink_impl: public data_sink_impl, public session::session_ref {\npublic:\n    using session_ref::session_ref;\nprivate:\n    future<> flush() override {\n        return _session->flush();\n    }\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        return _session->put(bufs);\n    }\n#else\n    using data_sink_impl::put;\n    future<> put(net::packet p) override {\n        auto vec = p.release();\n        return _session->put(std::span(vec));\n    }\n#endif\n    future<> close() override {\n        _session->close();\n        return make_ready_future<>();\n    }\n    bool can_batch_flushes() const noexcept override { return true; }\n    void on_batch_flush_error() noexcept override {\n        _session->close();\n    }\n};\n\nclass server_session : public net::server_socket_impl {\npublic:\n    server_session(shared_ptr<server_credentials> creds, server_socket sock)\n            : _creds(std::move(creds)), _sock(std::move(sock)) {\n    }\n    future<accept_result> accept() override {\n        // We're not actually doing anything very SSL until we get\n        // an actual connection. Then we create a \"server\" session\n        // and wrap it up after handshaking.\n        return _sock.accept().then([this](accept_result ar) {\n            return wrap_server(_creds, std::move(ar.connection)).then([addr = std::move(ar.remote_address)](connected_socket s) {\n                return make_ready_future<accept_result>(accept_result{std::move(s), addr});\n            });\n        });\n    }\n    void abort_accept() override  {\n        _sock.abort_accept();\n    }\n    socket_address local_address() const override {\n        return _sock.local_address();\n    }\nprivate:\n\n    shared_ptr<server_credentials> _creds;\n    server_socket _sock;\n};\n\nclass tls_socket_impl : public net::socket_impl {\n    shared_ptr<certificate_credentials> _cred;\n    tls_options _options;\n    ::seastar::socket _socket;\npublic:\n    tls_socket_impl(shared_ptr<certificate_credentials> cred, tls_options options)\n            : _cred(cred), _options(std::move(options)), _socket(make_socket()) {\n    }\n    virtual future<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) override {\n        return _socket.connect(sa, local, proto).then([cred = std::move(_cred), options = std::move(_options)](connected_socket s) mutable {\n            return wrap_client(cred, std::move(s), std::move(options));\n        });\n    }\n    void set_reuseaddr(bool reuseaddr) override {\n      _socket.set_reuseaddr(reuseaddr);\n    }\n    bool get_reuseaddr() const override {\n      return _socket.get_reuseaddr();\n    }\n    virtual void shutdown() override {\n        _socket.shutdown();\n    }\n};\n\n}\n\ndata_source tls::tls_connected_socket_impl::source() {\n    return data_source(std::make_unique<source_impl>(_session));\n}\n\ndata_sink tls::tls_connected_socket_impl::sink() {\n    return data_sink(std::make_unique<sink_impl>(_session));\n}\n\n\nfuture<connected_socket> tls::connect(shared_ptr<certificate_credentials> cred, socket_address sa, sstring name) {\n    tls_options options{.server_name = std::move(name)};\n    return connect(std::move(cred), std::move(sa), std::move(options));\n}\n\nfuture<connected_socket> tls::connect(shared_ptr<certificate_credentials> cred, socket_address sa, socket_address local, sstring name) {\n    tls_options options{.server_name = std::move(name)};\n    return connect(std::move(cred), std::move(sa), std::move(local), std::move(options));\n}\n\nfuture<connected_socket> tls::connect(shared_ptr<certificate_credentials> cred, socket_address sa, tls_options options) {\n    return engine().connect(sa).then([cred = std::move(cred), options = std::move(options)](connected_socket s) mutable {\n        return wrap_client(std::move(cred), std::move(s), std::move(options));\n    });\n}\n\nfuture<connected_socket> tls::connect(shared_ptr<certificate_credentials> cred, socket_address sa, socket_address local, tls_options options) {\n    return engine().connect(sa, local).then([cred = std::move(cred), options = std::move(options)](connected_socket s) mutable {\n        return wrap_client(std::move(cred), std::move(s), std::move(options));\n    });\n}\n\nsocket tls::socket(shared_ptr<certificate_credentials> cred, sstring name) {\n    tls_options options{.server_name = std::move(name)};\n    return tls::socket(std::move(cred), std::move(options));\n}\n\nsocket tls::socket(shared_ptr<certificate_credentials> cred, tls_options options) {\n    return ::seastar::socket(std::make_unique<tls_socket_impl>(std::move(cred), std::move(options)));\n}\n\nfuture<connected_socket> tls::wrap_client(shared_ptr<certificate_credentials> cred, connected_socket&& s, sstring name) {\n    tls_options options{.server_name = std::move(name)};\n    return wrap_client(std::move(cred), std::move(s), std::move(options));\n}\n\nfuture<connected_socket> tls::wrap_client(shared_ptr<certificate_credentials> cred, connected_socket&& s, tls_options options) {\n    session::session_ref sess(make_lw_shared<session>(session::type::CLIENT, std::move(cred), std::move(s),  options));\n    connected_socket sock(std::make_unique<tls_connected_socket_impl>(std::move(sess)));\n    return make_ready_future<connected_socket>(std::move(sock));\n}\n\nfuture<connected_socket> tls::wrap_server(shared_ptr<server_credentials> cred, connected_socket&& s) {\n    session::session_ref sess(make_lw_shared<session>(session::type::SERVER, std::move(cred), std::move(s)));\n    connected_socket sock(std::make_unique<tls_connected_socket_impl>(std::move(sess)));\n    return make_ready_future<connected_socket>(std::move(sock));\n}\n\nserver_socket tls::listen(shared_ptr<server_credentials> creds, socket_address sa, listen_options opts) {\n    return listen(std::move(creds), seastar::listen(sa, opts));\n}\n\nserver_socket tls::listen(shared_ptr<server_credentials> creds, server_socket ss) {\n    server_socket ssls(std::make_unique<server_session>(creds, std::move(ss)));\n    return server_socket(std::move(ssls));\n}\n\nstatic tls::tls_connected_socket_impl* get_tls_socket(connected_socket& socket) {\n    auto impl = net::get_impl::maybe_get_ptr(socket);\n    if (impl == nullptr) {\n        // the socket is not yet created or moved from\n        throw std::system_error(ENOTCONN, std::system_category());\n    }\n    auto tls_impl = dynamic_cast<tls::tls_connected_socket_impl*>(impl);\n    if (!tls_impl) {\n        // bad cast here means that we're dealing with wrong socket type\n        throw std::invalid_argument(\"Not a TLS socket\");\n    }\n    return tls_impl;\n}\n\nfuture<std::optional<session_dn>> tls::get_dn_information(connected_socket& socket) {\n    return get_tls_socket(socket)->get_distinguished_name();\n}\n\nfuture<std::vector<tls::subject_alt_name>> tls::get_alt_name_information(connected_socket& socket, std::unordered_set<subject_alt_name_type> types) {\n    return get_tls_socket(socket)->get_alt_name_information(std::move(types));\n}\n\nfuture<std::vector<tls::certificate_data>> tls::get_peer_certificate_chain(connected_socket& socket) {\n    return get_tls_socket(socket)->get_peer_certificate_chain();\n}\n\nfuture<bool> tls::check_session_is_resumed(connected_socket& socket) {\n    return get_tls_socket(socket)->check_session_is_resumed();\n}\n\nfuture<tls::session_data> tls::get_session_resume_data(connected_socket& socket) {\n    return get_tls_socket(socket)->get_session_resume_data();\n}\n\nfuture<std::optional<sstring>> tls::get_selected_alpn_protocol(connected_socket& socket) {\n    return get_tls_socket(socket)->get_selected_alpn_protocol();\n}\n\nfuture<sstring> tls::get_cipher_suite(connected_socket& socket) {\n    return get_tls_socket(socket)->get_cipher_suite();\n}\n\nfuture<sstring> tls::get_protocol_version(connected_socket& socket) {\n    return get_tls_socket(socket)->get_protocol_version();\n}\nfuture<> tls::force_rehandshake(connected_socket& socket) {\n    auto s = get_tls_socket(socket);\n    if (!s) {\n        return make_ready_future<>();\n    }\n    return s->force_rehandshake();\n}\n\n\nstd::string_view tls::format_as(subject_alt_name_type type) {\n    switch (type) {\n        case subject_alt_name_type::dnsname:\n            return \"DNS\";\n        case subject_alt_name_type::rfc822name:\n            return \"EMAIL\";\n        case subject_alt_name_type::uri:\n            return \"URI\";\n        case subject_alt_name_type::ipaddress:\n            return \"IP\";\n        case subject_alt_name_type::othername:\n            return \"OTHERNAME\";\n        case subject_alt_name_type::dn:\n            return \"DIRNAME\";\n        default:\n            return \"UNKNOWN\";\n    }\n}\n\nstd::ostream& tls::operator<<(std::ostream& os, subject_alt_name_type type) {\n    return os << format_as(type);\n}\n\nstd::ostream& tls::operator<<(std::ostream& os, const subject_alt_name::value_type& v) {\n    fmt::print(os, \"{}\", v);\n    return os;\n}\n\nstd::ostream& tls::operator<<(std::ostream& os, const subject_alt_name& a) {\n    fmt::print(os, \"{}\", a);\n    return os;\n}\n\n\n}\n\nconst int seastar::tls::ERROR_UNKNOWN_COMPRESSION_ALGORITHM = GNUTLS_E_UNKNOWN_COMPRESSION_ALGORITHM;\nconst int seastar::tls::ERROR_UNKNOWN_CIPHER_TYPE = GNUTLS_E_UNKNOWN_CIPHER_TYPE;\nconst int seastar::tls::ERROR_INVALID_SESSION = GNUTLS_E_INVALID_SESSION;\nconst int seastar::tls::ERROR_UNEXPECTED_HANDSHAKE_PACKET = GNUTLS_E_UNEXPECTED_HANDSHAKE_PACKET;\nconst int seastar::tls::ERROR_UNKNOWN_CIPHER_SUITE = GNUTLS_E_UNKNOWN_CIPHER_SUITE;\nconst int seastar::tls::ERROR_UNKNOWN_ALGORITHM = GNUTLS_E_UNKNOWN_ALGORITHM;\nconst int seastar::tls::ERROR_UNSUPPORTED_SIGNATURE_ALGORITHM = GNUTLS_E_UNSUPPORTED_SIGNATURE_ALGORITHM;\nconst int seastar::tls::ERROR_SAFE_RENEGOTIATION_FAILED = GNUTLS_E_SAFE_RENEGOTIATION_FAILED;\nconst int seastar::tls::ERROR_UNSAFE_RENEGOTIATION_DENIED = GNUTLS_E_UNSAFE_RENEGOTIATION_DENIED;\nconst int seastar::tls::ERROR_UNKNOWN_SRP_USERNAME = GNUTLS_E_UNKNOWN_SRP_USERNAME;\nconst int seastar::tls::ERROR_PREMATURE_TERMINATION = GNUTLS_E_PREMATURE_TERMINATION;\nconst int seastar::tls::ERROR_PUSH = GNUTLS_E_PUSH_ERROR;\nconst int seastar::tls::ERROR_PULL = GNUTLS_E_PULL_ERROR;\nconst int seastar::tls::ERROR_UNEXPECTED_PACKET = GNUTLS_E_UNEXPECTED_PACKET;\nconst int seastar::tls::ERROR_UNSUPPORTED_VERSION = GNUTLS_E_UNSUPPORTED_VERSION_PACKET;\nconst int seastar::tls::ERROR_NO_CIPHER_SUITES = GNUTLS_E_NO_CIPHER_SUITES;\nconst int seastar::tls::ERROR_DECRYPTION_FAILED = GNUTLS_E_DECRYPTION_FAILED;\nconst int seastar::tls::ERROR_MAC_VERIFY_FAILED = GNUTLS_E_MAC_VERIFY_FAILED;\n"
  },
  {
    "path": "src/net/udp.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/net/ip.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/net/inet_address.hh>\n\nnamespace seastar {\n\nusing namespace net;\n\nnamespace net {\nnamespace ipv4_udp_impl {\n\nstatic inline\nipv4_addr\nto_ipv4_addr(ipv4_address a, uint16_t port) {\n    return {a.ip, port};\n}\n\nclass native_datagram : public datagram_impl {\nprivate:\n    ipv4_addr _src;\n    ipv4_addr _dst;\n    std::vector<temporary_buffer<char>> _bufs;\npublic:\n    native_datagram(ipv4_address src, ipv4_address dst, packet _p)\n    {\n        udp_hdr* hdr = _p.get_header<udp_hdr>();\n        auto h = ntoh(*hdr);\n        _p.trim_front(sizeof(*hdr));\n        _src = to_ipv4_addr(src, h.src_port);\n        _dst = to_ipv4_addr(dst, h.dst_port);\n        _bufs = _p.release();\n    }\n\n    virtual socket_address get_src() override {\n        return _src;\n    };\n\n    virtual socket_address get_dst() override {\n        return _dst;\n    };\n\n    virtual uint16_t get_dst_port() override {\n        return _dst.port;\n    }\n\n    virtual std::span<temporary_buffer<char>> get_buffers() override {\n        return _bufs;\n    }\n};\n\nclass native_channel : public datagram_channel_impl {\nprivate:\n    ipv4_udp& _proto;\n    ipv4_udp::registration _reg;\n    bool _closed;\n    lw_shared_ptr<udp_channel_state> _state;\n\npublic:\n    native_channel(ipv4_udp &proto, ipv4_udp::registration reg, lw_shared_ptr<udp_channel_state> state)\n            : _proto(proto)\n            , _reg(reg)\n            , _closed(false)\n            , _state(state)\n    {\n    }\n\n    ~native_channel()\n    {\n        if (!_closed)\n            close();\n    }\n\n    socket_address local_address() const override {\n        return socket_address(_proto.inet().host_address(), _reg.port());\n    }\n\n    virtual future<datagram> receive() override {\n        return _state->_queue.pop_eventually();\n    }\n\n    virtual future<> send(const socket_address& dst, const char* msg) override {\n        temporary_buffer<char> buf(const_cast<char *>(msg), strlen(msg), deleter());\n        return send(dst, std::span(&buf, 1));\n    }\n\n    virtual future<> send(const socket_address& dst, std::span<temporary_buffer<char>> bufs) override {\n        auto p = net::packet(bufs);\n        auto len = p.len();\n        return _state->wait_for_send_buffer(len).then([this, dst, p = std::move(p), len] () mutable {\n            p = packet(std::move(p), make_deleter([s = _state, len] { s->complete_send(len); }));\n            _proto.send(_reg.port(), dst, std::move(p));\n        });\n    }\n\n    virtual bool is_closed() const override {\n        return _closed;\n    }\n\n    virtual void shutdown_input() override {\n        _state->_queue.abort(std::make_exception_ptr(std::system_error(EBADF, std::system_category())));\n    }\n\n    virtual void shutdown_output() override {\n        _state->_queue.abort(std::make_exception_ptr(std::system_error(EPIPE, std::system_category())));\n    }\n\n    virtual void close() override {\n        _reg.unregister();\n        _closed = true;\n    }\n};\n\n} /* namespace ipv4_udp_impl */\n\nusing namespace net::ipv4_udp_impl;\n\nconst int ipv4_udp::default_queue_size = 1024;\n\nipv4_udp::ipv4_udp(ipv4& inet)\n    : _inet(inet)\n{\n    _inet.register_packet_provider([this] {\n        std::optional<ipv4_traits::l4packet> l4p;\n        if (!_packetq.empty()) {\n            l4p = std::move(_packetq.front());\n            _packetq.pop_front();\n        }\n        return l4p;\n    });\n}\n\nbool ipv4_udp::forward(forward_hash& out_hash_data, packet& p, size_t off)\n{\n    auto uh = p.get_header<udp_hdr>(off);\n\n    if (uh) {\n        out_hash_data.push_back(uh->src_port);\n        out_hash_data.push_back(uh->dst_port);\n    }\n    return true;\n}\n\nvoid ipv4_udp::received(packet p, ipv4_address from, ipv4_address to)\n{\n    datagram dgram(std::make_unique<native_datagram>(from, to, std::move(p)));\n\n    auto chan_it = _channels.find(dgram.get_dst_port());\n    if (chan_it != _channels.end()) {\n        auto chan = chan_it->second;\n        chan->_queue.push(std::move(dgram));\n    }\n}\n\nvoid ipv4_udp::send(uint16_t src_port, ipv4_addr dst, packet &&p)\n{\n    auto src = _inet.host_address();\n    auto hdr = p.prepend_header<udp_hdr>();\n    hdr->src_port = src_port;\n    hdr->dst_port = dst.port;\n    hdr->len = p.len();\n    *hdr = hton(*hdr);\n\n    offload_info oi;\n    checksummer csum;\n    ipv4_traits::udp_pseudo_header_checksum(csum, src, dst, p.len());\n    bool needs_frag = ipv4::needs_frag(p, ip_protocol_num::udp, _inet.hw_features());\n    if (_inet.hw_features().tx_csum_l4_offload && !needs_frag) {\n        hdr->cksum = ~csum.get();\n        oi.needs_csum = true;\n    } else {\n        csum.sum(p);\n        hdr->cksum = csum.get();\n        oi.needs_csum = false;\n    }\n    oi.protocol = ip_protocol_num::udp;\n    p.set_offload_info(oi);\n\n    // FIXME: future is discarded\n    (void)_inet.get_l2_dst_address(dst).then([this, dst, p = std::move(p)] (ethernet_address e_dst) mutable {\n        _packetq.emplace_back(ipv4_traits::l4packet{dst, std::move(p), e_dst, ip_protocol_num::udp});\n    });\n}\n\nuint16_t ipv4_udp::next_port(uint16_t port) {\n    return (port == std::numeric_limits<decltype(port)>::max()) ? min_anonymous_port : port + 1;\n}\n\nudp_channel\nipv4_udp::make_channel(ipv4_addr addr) {\n    if (!is_ip_unspecified(addr)) {\n        throw std::runtime_error(\"Binding to specific IP not supported yet\");\n    }\n\n    uint16_t bind_port;\n\n    if (!is_port_unspecified(addr)) {\n        if (_channels.count(addr.port)) {\n            throw std::runtime_error(\"Address already in use\");\n        }\n        bind_port = addr.port;\n    } else {\n        auto starting_port = _next_anonymous_port;\n        while (_channels.count(_next_anonymous_port)) {\n            _next_anonymous_port = next_port(_next_anonymous_port);\n            if (starting_port == _next_anonymous_port) {\n                throw std::runtime_error(\"No free port\");\n            }\n        }\n\n        bind_port = _next_anonymous_port;\n        _next_anonymous_port = next_port(_next_anonymous_port);\n    }\n\n    auto chan_state = make_lw_shared<udp_channel_state>(_queue_size);\n    _channels[bind_port] = chan_state;\n    return udp_channel(std::make_unique<native_channel>(*this, registration(*this, bind_port), chan_state));\n}\n\n} /* namespace net */\n\n}\n\n"
  },
  {
    "path": "src/net/unix_address.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Red Hat, Inc.\n */\n/*! \\file\n  \\brief unix-domain address structures, to be used for creating socket_address-es for unix-domain\n         sockets.\n\n  Note that the path in a unix-domain address may start with a null character.\n*/\n\n#include <ostream>\n#include <seastar/net/socket_defs.hh>\n#include <cassert>\n\nnamespace seastar {\n\nstd::ostream& operator<<(std::ostream& os, const unix_domain_addr& addr) {\n    if (addr.path_length() == 0) {\n        return os << \"{unnamed}\";\n    }\n    if (addr.name[0]) {\n        // regular (filesystem-namespace) path\n        return os << addr.name;\n    }\n\n    os << '@';\n    const char* src = addr.path_bytes() + 1;\n\n    for (auto k = addr.path_length(); --k > 0; src++) {\n        os << (std::isprint(*src) ? *src : '_');\n    }\n    return os;\n}\n\n} // namespace seastar\n\nsize_t std::hash<seastar::unix_domain_addr>::operator()(const seastar::unix_domain_addr& a) const {\n    return std::hash<std::string>()(a.name);\n}\n"
  },
  {
    "path": "src/net/virtio.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <atomic>\n#include <algorithm>\n#include <cstring>\n#include <memory>\n#include <queue>\n#include <string>\n#include <vector>\n#include <fcntl.h>\n#include <seastar/net/virtio-interface.hh>\n#include <linux/vhost.h>\n#include <linux/if_tun.h>\n#include <net/if.h>\n#include <seastar/util/assert.hh>\n\n#include <seastar/net/virtio.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/internal/pollable_fd.hh>\n#include <seastar/core/internal/poll.hh>\n#include \"core/vla.hh\"\n#include <seastar/core/reactor.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/util/function_input_iterator.hh>\n#include <seastar/util/transform_iterator.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/const.hh>\n#include <seastar/net/native-stack.hh>\n\nnamespace seastar {\n\nusing namespace net;\n\nnamespace virtio {\n\nusing phys = uint64_t;\n\nphys virt_to_phys(void* p) {\n    return reinterpret_cast<uintptr_t>(p);\n}\n\nclass device : public net::device {\nprivate:\n    net::hw_features _hw_features;\n    uint64_t _features;\n\nprivate:\n    uint64_t setup_features(const net::virtio_options& opts, const program_options::value<std::string>& lro) {\n        int64_t seastar_supported_features = VIRTIO_RING_F_INDIRECT_DESC | VIRTIO_NET_F_MRG_RXBUF;\n\n        if (!(opts.event_index && opts.event_index.get_value() == \"off\")) {\n            seastar_supported_features |= VIRTIO_RING_F_EVENT_IDX;\n        }\n        if (!(opts.csum_offload && opts.csum_offload.get_value() == \"off\")) {\n            seastar_supported_features |= VIRTIO_NET_F_CSUM | VIRTIO_NET_F_GUEST_CSUM;\n            _hw_features.tx_csum_l4_offload = true;\n            _hw_features.rx_csum_offload = true;\n        } else {\n            _hw_features.tx_csum_l4_offload = false;\n            _hw_features.rx_csum_offload = false;\n        }\n        if (!(opts.tso && opts.tso.get_value() == \"off\")) {\n            seastar_supported_features |= VIRTIO_NET_F_HOST_TSO4;\n            _hw_features.tx_tso = true;\n        } else {\n            _hw_features.tx_tso = false;\n        }\n\n        if (!(lro && lro.get_value() == \"off\")) {\n            seastar_supported_features |= VIRTIO_NET_F_GUEST_TSO4;\n            _hw_features.rx_lro = true;\n        } else {\n            _hw_features.rx_lro = false;\n        }\n\n        if (!(opts.ufo && opts.ufo.get_value() == \"off\")) {\n            seastar_supported_features |= VIRTIO_NET_F_HOST_UFO;\n            seastar_supported_features |= VIRTIO_NET_F_GUEST_UFO;\n            _hw_features.tx_ufo = true;\n        } else {\n            _hw_features.tx_ufo = false;\n        }\n\n        seastar_supported_features |= VIRTIO_NET_F_MAC;\n        return seastar_supported_features;\n    }\n\npublic:\n    device(const virtio_options& opts, const program_options::value<std::string>& lro)\n       : _features(setup_features(opts, lro))\n       {}\n    ethernet_address hw_address() override {\n        return { 0x12, 0x23, 0x34, 0x56, 0x67, 0x78 };\n    }\n\n    net::hw_features hw_features() override {\n        return _hw_features;\n    }\n\n    uint64_t features() {\n        return _features;\n    }\n\n    virtual std::unique_ptr<net::qp> init_local_queue(const program_options::option_group& opts, uint16_t qid) override;\n};\n\n/* The virtio_notifier class determines how to do host-to-guest and guest-to-\n * host notifications. We have two different implementations - one for vhost\n * (where both notifications occur through eventfds) and one for an assigned\n * virtio device from OSv.\n */\nclass notifier {\npublic:\n    // Notify the host\n    virtual void notify() = 0;\n    // Do whatever it takes to wake wait(). A notifier does not need to\n    // implement this function if wait() waits for an external even which is\n    // generated by an external process (e.g., virtio_notifier_host doesn't\n    // need to implement this).\n    virtual void wake_wait() {\n        abort();\n    }\n    virtual ~notifier() {\n    }\n};\n\nclass notifier_vhost : public notifier {\nprivate:\n    writeable_eventfd _kick;\npublic:\n    virtual void notify() override {\n        _kick.signal(1);\n    }\n    notifier_vhost(writeable_eventfd &&kick)\n        : _kick(std::move(kick)) {}\n};\n\n\nstruct ring_config {\n    char* descs;\n    char* avail;\n    char* used;\n    unsigned size;\n    bool event_index;\n    bool indirect;\n    bool mergable_buffers;\n};\n\nstruct buffer {\n    phys addr;\n    uint32_t len;\n    bool writeable;\n};\n\n// The 'buffer_chain' concept, used in vring, is a container of buffers, as in:\n//\n//   using buffer_chain = std::vector<buffer>;\n//\n// The 'Completion' concept is a functor with the signature:\n//\n//     void (buffer_chain&, size_t len);\n//\ntemplate <typename BufferChain, typename Completion>\nclass vring {\nprivate:\n    class desc {\n    public:\n        struct flags {\n            // This marks a buffer as continuing via the next field.\n            uint16_t has_next : 1;\n            // This marks a buffer as write-only (otherwise read-only).\n            uint16_t writeable : 1;\n            // This means the buffer contains a list of buffer descriptors.\n            uint16_t indirect : 1;\n        };\n\n        phys get_paddr();\n        uint32_t get_len() { return _len; }\n        uint16_t next_idx() { return _next; }\n\n        phys _paddr;\n        uint32_t _len;\n        flags _flags;\n        uint16_t _next;\n    };\n\n    // Guest to host\n    struct avail_layout {\n        struct flags {\n            // Mark that we do not need an interrupt for consuming a descriptor\n            // from the ring. Unreliable so it's simply an optimization\n            uint16_t no_interrupts : 1;\n        };\n\n        std::atomic<uint16_t> _flags;\n\n        // Where we put the next descriptor\n        std::atomic<uint16_t> _idx;\n        // There may be no more entries than the queue size read from device\n        uint16_t _ring[];\n        // used event index is an optimization in order to get an interrupt from the host\n        // only when the value reaches this number\n        // The location of this field is places after the variable length ring array,\n        // that's why we cannot fully define it within the struct and use a function accessor\n        //std::atomic<uint16_t> used_event;\n    };\n\n    struct used_elem {\n        // Index of start of used _desc chain. (uint32_t for padding reasons)\n        uint32_t _id;\n        // Total length of the descriptor chain which was used (written to)\n        uint32_t _len;\n    };\n\n    // Host to guest\n    struct used_layout {\n        enum {\n            // The Host advise the Guest: don't kick me when\n            // you add a buffer.  It's unreliable, so it's simply an\n            // optimization. Guest will still kick if it's out of buffers.\n            no_notify = 1\n        };\n\n        // Using std::atomic since it being changed by the host\n        std::atomic<uint16_t> _flags;\n        // Using std::atomic in order to have memory barriers for it\n        std::atomic<uint16_t> _idx;\n        used_elem _used_elements[];\n        // avail event index is an optimization kick the host only when the value reaches this number\n        // The location of this field is places after the variable length ring array,\n        // that's why we cannot fully define it within the struct and use a function accessor\n        //std::atomic<uint16_t> avail_event;\n    };\n\n    struct avail {\n        explicit avail(ring_config conf);\n        avail_layout* _shared;\n        uint16_t _head = 0;\n        uint16_t _avail_added_since_kick = 0;\n    };\n    struct used {\n        explicit used(ring_config conf);\n        used_layout* _shared;\n        uint16_t _tail = 0;\n    };\nprivate:\n    ring_config _config;\n    Completion _complete;\n    std::unique_ptr<notifier> _notifier;\n    std::unique_ptr<BufferChain[]> _buffer_chains;\n    desc* _descs;\n    avail _avail;\n    used _used;\n    std::atomic<uint16_t>* _avail_event;\n    std::atomic<uint16_t>* _used_event;\n    semaphore _available_descriptors = { 0 };\n    int _free_head = -1;\n    int _free_last = -1;\n    reactor::poller _poller;\npublic:\n\n    explicit vring(ring_config conf, Completion complete);\n    void set_notifier(std::unique_ptr<notifier> notifier) {\n        _notifier = std::move(notifier);\n    }\n    const ring_config& getconfig() {\n        return _config;\n    }\n    void wake_notifier_wait() {\n        _notifier->wake_wait();\n    }\n\n    // start the queue\n    void run();\n\n    // wait for the used ring to have at least @nr buffers\n    future<> on_used(size_t nr);\n\n    // Total number of descriptors in ring\n    int size() { return _config.size; }\n\n    template <typename Iterator>\n    void post(Iterator begin, Iterator end);\n\n    semaphore& available_descriptors() { return _available_descriptors; }\nprivate:\n    bool notifications_disabled() {\n        return (_used._shared->_flags.load(std::memory_order_relaxed) & VRING_USED_F_NO_NOTIFY) != 0;\n    }\n\n    void kick() {\n        bool need_kick = true;\n        // Make sure we see the fresh _idx value writen before kick.\n        std::atomic_thread_fence(std::memory_order_seq_cst);\n        if (_config.event_index) {\n            uint16_t avail_idx = _avail._shared->_idx.load(std::memory_order_relaxed);\n            uint16_t avail_event = _avail_event->load(std::memory_order_relaxed);\n            need_kick = (uint16_t)(avail_idx - avail_event - 1) < _avail._avail_added_since_kick;\n        } else {\n            if (notifications_disabled())\n                return;\n        }\n        if (need_kick || (_avail._avail_added_since_kick >= (uint16_t)(~0) / 2)) {\n            _notifier->notify();\n            _avail._avail_added_since_kick = 0;\n        }\n    }\n\n    bool do_complete();\n    size_t mask() { return size() - 1; }\n    size_t masked(size_t idx) { return idx & mask(); }\n    size_t available();\n    unsigned allocate_desc();\n    void setup();\n};\n\ntemplate <typename BufferChain, typename Completion>\nvring<BufferChain, Completion>::avail::avail(ring_config conf)\n    : _shared(reinterpret_cast<avail_layout*>(conf.avail)) {\n}\n\ntemplate <typename BufferChain, typename Completion>\nvring<BufferChain, Completion>::used::used(ring_config conf)\n    : _shared(reinterpret_cast<used_layout*>(conf.used)) {\n}\n\ntemplate <typename BufferChain, typename Completion>\ninline\nunsigned\nvring<BufferChain, Completion>::allocate_desc() {\n    SEASTAR_ASSERT(_free_head != -1);\n    auto desc = _free_head;\n    if (desc == _free_last) {\n        _free_last = _free_head = -1;\n    } else {\n        _free_head = _descs[desc]._next;\n    }\n    return desc;\n}\n\ntemplate <typename BufferChain, typename Completion>\nvring<BufferChain, Completion>::vring(ring_config conf, Completion complete)\n    : _config(conf)\n    , _complete(complete)\n    , _buffer_chains(new BufferChain[_config.size])\n    , _descs(reinterpret_cast<desc*>(conf.descs))\n    , _avail(conf)\n    , _used(conf)\n    , _avail_event(reinterpret_cast<std::atomic<uint16_t>*>(&_used._shared->_used_elements[conf.size]))\n    , _used_event(reinterpret_cast<std::atomic<uint16_t>*>(&_avail._shared->_ring[conf.size]))\n    , _poller(reactor::poller::simple([this] {\n        return do_complete();\n    }))\n{\n    setup();\n}\n\ntemplate <typename BufferChain, typename Completion>\nvoid vring<BufferChain, Completion>::setup() {\n    for (unsigned i = 0; i < _config.size; ++i) {\n        _descs[i]._next = i + 1;\n    }\n    _free_head = 0;\n    _free_last = _config.size - 1;\n    _available_descriptors.signal(_config.size);\n}\n\n// Iterator: points at a buffer_chain\ntemplate <typename BufferChain, typename Completion>\ntemplate <typename Iterator>\nvoid vring<BufferChain, Completion>::post(Iterator begin, Iterator end) {\n    for (auto bci = begin; bci!= end; ++bci) {\n        auto&& bc = *bci;\n        desc pseudo_head = {};\n        desc* prev = &pseudo_head;\n        for (auto i = bc.begin(); i != bc.end(); ++i) {\n            unsigned desc_idx = allocate_desc();\n            prev->_flags.has_next = true;\n            prev->_next = desc_idx;\n            desc &d = _descs[desc_idx];\n            d._flags = {};\n            auto&& b = *i;\n            d._flags.writeable = b.writeable;\n            d._paddr = b.addr;\n            d._len = b.len;\n            prev = &d;\n        }\n        auto desc_head = pseudo_head._next;\n        _buffer_chains[desc_head] = std::move(bc);\n        _avail._shared->_ring[masked(_avail._head++)] = desc_head;\n        _avail._avail_added_since_kick++;\n    }\n    _avail._shared->_idx.store(_avail._head, std::memory_order_release);\n    kick();\n}\n\ntemplate <typename BufferChain, typename Completion>\nbool vring<BufferChain, Completion>::do_complete() {\n    auto used_head = _used._shared->_idx.load(std::memory_order_acquire);\n    auto count = _used._tail - used_head;\n    _complete.bunch(count);\n    while (used_head != _used._tail) {\n        auto ue = _used._shared->_used_elements[masked(_used._tail++)];\n        _complete(std::move(_buffer_chains[ue._id]), ue._len);\n        auto id = ue._id;\n        if (_free_last != -1) {\n            _descs[_free_last]._next = id;\n        } else {\n            _free_head = id;\n        }\n        while (true) {\n            auto& d = _descs[id];\n            if (!d._flags.has_next) {\n                break;\n            }\n            id = d._next;\n        }\n        _free_last = id;\n    }\n    return count;\n}\n\nclass qp : public net::qp {\nprotected:\n    struct net_hdr {\n        uint8_t needs_csum : 1;\n        uint8_t flags_reserved : 7;\n        enum { gso_none = 0, gso_tcpv4 = 1, gso_udp = 3, gso_tcpv6 = 4, gso_ecn = 0x80 };\n        uint8_t gso_type;\n        uint16_t hdr_len;\n        uint16_t gso_size;\n        uint16_t csum_start;\n        uint16_t csum_offset;\n    };\n    struct net_hdr_mrg : net_hdr {\n        uint16_t num_buffers;\n    };\n    class txq {\n        static buffer fragment_to_buffer(fragment f) {\n            buffer b;\n            b.addr = virt_to_phys(f.base);\n            b.len = f.size;\n            b.writeable = false;\n            return b;\n        };\n        struct packet_as_buffer_chain {\n            packet p;\n            auto begin() {\n                return make_transform_iterator(p.fragments().begin(), fragment_to_buffer);\n            }\n            auto end() {\n                return make_transform_iterator(p.fragments().end(), fragment_to_buffer);\n            }\n        };\n        struct complete {\n            txq& q;\n            void operator()(packet_as_buffer_chain&& bc, size_t len) {\n                // move the packet here, to be destroyed on scope exit\n                auto p = std::move(bc.p);\n                q._ring.available_descriptors().signal(p.nr_frags());\n            }\n            void bunch(uint64_t c) {}\n        };\n        qp& _dev;\n        vring<packet_as_buffer_chain, complete> _ring;\n        std::vector<packet_as_buffer_chain> _packets;\n    public:\n        txq(qp& dev, ring_config config);\n        void set_notifier(std::unique_ptr<notifier> notifier) {\n            _ring.set_notifier(std::move(notifier));\n        }\n        const ring_config& getconfig() {\n            return _ring.getconfig();\n        }\n        void wake_notifier_wait() {\n            _ring.wake_notifier_wait();\n        }\n        uint32_t post(circular_buffer<packet>& p);\n    };\n    class rxq  {\n        struct buffer_and_virt : buffer {\n            std::unique_ptr<char[], free_deleter> buf;\n        };\n        using single_buffer = std::array<buffer_and_virt, 1>;\n        struct complete {\n            rxq& q;\n            void operator()(single_buffer&& bc, size_t len) {\n                q.complete_buffer(std::move(bc), len);\n            }\n            void bunch(uint64_t c) {\n                q.update_rx_count(c);\n            }\n        };\n        qp& _dev;\n        vring<single_buffer, complete> _ring;\n        unsigned _remaining_buffers = 0;\n        std::vector<fragment> _fragments;\n        std::vector<std::unique_ptr<char[], free_deleter>> _buffers;\n    public:\n        rxq(qp& _if, ring_config config);\n        void set_notifier(std::unique_ptr<notifier> notifier) {\n            _ring.set_notifier(std::move(notifier));\n        }\n        const ring_config& getconfig() {\n            return _ring.getconfig();\n        }\n        void run() {\n            // FIXME: future is discarded\n            // At least catch errors and warn about them.\n            (void)keep_doing([this] { return prepare_buffers(); });\n        }\n        void wake_notifier_wait() {\n            _ring.wake_notifier_wait();\n        }\n        void update_rx_count(uint64_t c) {\n            _dev._stats.rx.good.update_pkts_bunch(c);\n        }\n    private:\n        future<> prepare_buffers();\n        void complete_buffer(single_buffer&& b, size_t len);\n        void debug_mode_adjust_fragments();\n    };\nprotected:\n    device* _dev;\n    size_t _header_len;\n    std::unique_ptr<char[], free_deleter> _txq_storage;\n    std::unique_ptr<char[], free_deleter> _rxq_storage;\n    txq _txq;\n    rxq _rxq;\nprotected:\n    ring_config txq_config(size_t txq_ring_size);\n    ring_config rxq_config(size_t rxq_ring_size);\n    void common_config(ring_config& r);\n    size_t vring_storage_size(size_t ring_size);\npublic:\n    explicit qp(device* dev, size_t rx_ring_size, size_t tx_ring_size);\n    virtual future<> send(packet p) override {\n        abort();\n    }\n    virtual uint32_t send(circular_buffer<packet>& p) override;\n    virtual void rx_start() override;\n    friend class rxq;\n};\n\nqp::txq::txq(qp& dev, ring_config config)\n    : _dev(dev), _ring(config, complete{*this}) {\n}\n\nuint32_t\nqp::txq::post(circular_buffer<packet>& pb) {\n    uint64_t bytes = 0, nr_frags = 0;\n\n    _packets.clear();\n\n    while (!pb.empty() && pb.front().nr_frags() + 1 <= _ring.available_descriptors().current()) {\n        net_hdr_mrg vhdr = {};\n        auto p = std::move(pb.front());\n\n        bytes    += p.len();\n        nr_frags += p.nr_frags();\n\n        pb.pop_front();\n        // Handle TCP checksum offload\n        auto oi = p.get_offload_info();\n        if (_dev._dev->hw_features().tx_csum_l4_offload) {\n            auto eth_hdr_len = sizeof(eth_hdr);\n            auto ip_hdr_len = oi.ip_hdr_len;\n            auto mtu = _dev._dev->hw_features().mtu;\n            if (oi.protocol == ip_protocol_num::tcp) {\n                auto tcp_hdr_len = oi.tcp_hdr_len;\n                if (oi.needs_csum) {\n                    vhdr.needs_csum = 1;\n                    vhdr.csum_start = eth_hdr_len + ip_hdr_len;\n                    // TCP checksum filed's offset within the TCP header is 16 bytes\n                    vhdr.csum_offset = 16;\n                }\n                if (oi.tso_seg_size) {\n                    // IPv4 TCP TSO\n                    vhdr.gso_type = net_hdr::gso_tcpv4;\n                    // Sum of Ethernet, IP and TCP header size\n                    vhdr.hdr_len = eth_hdr_len + ip_hdr_len + tcp_hdr_len;\n                    // Maximum segment size of packet after the offload\n                    vhdr.gso_size = oi.tso_seg_size;\n                }\n            } else if (oi.protocol == ip_protocol_num::udp) {\n                auto udp_hdr_len = oi.udp_hdr_len;\n                if (oi.needs_csum) {\n                    vhdr.needs_csum = 1;\n                    vhdr.csum_start = eth_hdr_len + ip_hdr_len;\n                    // UDP checksum filed's offset within the UDP header is 6 bytes\n                    vhdr.csum_offset = 6;\n                }\n                if (_dev._dev->hw_features().tx_ufo && p.len() > mtu + eth_hdr_len) {\n                    vhdr.gso_type = net_hdr::gso_udp;\n                    vhdr.hdr_len = eth_hdr_len + ip_hdr_len + udp_hdr_len;\n                    vhdr.gso_size = mtu - ip_hdr_len - udp_hdr_len;\n                }\n            }\n        }\n        // prepend virtio-net header\n        packet q = packet(fragment{reinterpret_cast<char*>(&vhdr), _dev._header_len},\n                std::move(p));\n        auto fut = _ring.available_descriptors().wait(q.nr_frags());\n        SEASTAR_ASSERT(fut.available()); // how it cannot?\n        _packets.emplace_back(packet_as_buffer_chain{ std::move(q) });\n    }\n    _ring.post(_packets.begin(), _packets.end());\n\n    _dev._stats.tx.good.update_frags_stats(nr_frags, bytes);\n\n    return _packets.size();\n}\n\nqp::rxq::rxq(qp& dev, ring_config config)\n    : _dev(dev), _ring(config, complete{*this}) {\n}\n\nfuture<>\nqp::rxq::prepare_buffers() {\n    auto& available = _ring.available_descriptors();\n    return available.wait(1).then([this, &available] {\n        unsigned count = 1;\n        auto opportunistic = available.current();\n        if (available.try_wait(opportunistic)) {\n            count += opportunistic;\n        }\n        auto make_buffer_chain = [] {\n            single_buffer bc;\n            std::unique_ptr<char[], free_deleter> buf(reinterpret_cast<char*>(malloc(4096)));\n            buffer_and_virt& b = bc[0];\n            b.addr = virt_to_phys(buf.get());\n            b.len = 4096;\n            b.writeable = true;\n            b.buf = std::move(buf);\n            return bc;\n        };\n        auto start = make_function_input_iterator(make_buffer_chain, 0U);\n        auto finish = make_function_input_iterator(make_buffer_chain, count);\n        _ring.post(start, finish);\n    });\n}\n\nvoid\nqp::rxq::debug_mode_adjust_fragments() {\n#ifdef SEASTAR_DEBUG\n    // For debug mode, reallocate last fragment to detect buffer overruns\n    auto last = _fragments.back();\n    auto sz = last.size;\n    std::unique_ptr<char[], free_deleter> buf(reinterpret_cast<char*>(malloc(sz)));\n    if (!buf) {\n        throw std::bad_alloc();\n    }\n    std::copy_n(last.base, sz, buf.get());\n    _fragments.back() = { buf.get(), sz };\n    _buffers.back() = std::move(buf);\n#endif\n}\n\nvoid\nqp::rxq::complete_buffer(single_buffer&& bc, size_t len) {\n    auto&& sb = bc[0];\n    auto&& buf = sb.buf;\n    auto frag_buf = buf.get();\n    auto frag_len = len;\n    // First buffer\n    if (_remaining_buffers == 0) {\n        auto hdr = reinterpret_cast<net_hdr_mrg*>(frag_buf);\n        SEASTAR_ASSERT(hdr->num_buffers >= 1);\n        _remaining_buffers = hdr->num_buffers;\n        frag_buf += _dev._header_len;\n        frag_len -= _dev._header_len;\n        _fragments.clear();\n        _buffers.clear();\n    };\n\n    // Append current buffer\n    _fragments.emplace_back(fragment{frag_buf, frag_len});\n    _buffers.push_back(std::move(buf));\n    _remaining_buffers--;\n\n    // Last buffer\n    if (_remaining_buffers == 0) {\n        debug_mode_adjust_fragments();\n        deleter del;\n        if (_buffers.size() == 1) {\n            del = make_free_deleter(_buffers[0].release());\n            _buffers.clear();\n        } else {\n            del = make_object_deleter(std::move(_buffers));\n        }\n        packet p(_fragments.begin(), _fragments.end(), std::move(del));\n\n        _dev._stats.rx.good.update_frags_stats(p.nr_frags(), p.len());\n\n        _dev._dev->l2receive(std::move(p));\n\n\n        _ring.available_descriptors().signal(_fragments.size());\n    }\n}\n\n// Allocate and zero-initialize a buffer which is page-aligned and can be\n// used for virt_to_phys (i.e., physically contiguous).\nstatic std::unique_ptr<char[], free_deleter> virtio_buffer(size_t size) {\n    void* ret;\n    auto r = posix_memalign(&ret, 4096, size);\n    SEASTAR_ASSERT(r == 0);\n    bzero(ret, size);\n    return std::unique_ptr<char[], free_deleter>(reinterpret_cast<char*>(ret));\n}\n\nqp::qp(device* dev, size_t rx_ring_size, size_t tx_ring_size)\n    : _dev(dev)\n    , _txq_storage(virtio_buffer(vring_storage_size(tx_ring_size)))\n    , _rxq_storage(virtio_buffer(vring_storage_size(rx_ring_size)))\n    , _txq(*this, txq_config(tx_ring_size))\n    , _rxq(*this, rxq_config(rx_ring_size)) {\n}\n\nsize_t qp::vring_storage_size(size_t ring_size) {\n    // overestimate, but not by much.\n    return 3 * 4096 + ring_size * (16 + 2 + 8);\n}\n\nvoid qp::common_config(ring_config& r) {\n    r.avail = r.descs + 16 * r.size;\n    r.used = align_up(r.avail + 2 * r.size + 6, 4096);\n    r.event_index = (_dev->features() & VIRTIO_RING_F_EVENT_IDX) != 0;\n    r.indirect = false;\n}\n\nring_config qp::txq_config(size_t tx_ring_size) {\n    ring_config r;\n    r.size = tx_ring_size;\n    r.descs = _txq_storage.get();\n    r.mergable_buffers = false;\n    common_config(r);\n    return r;\n}\n\nring_config qp::rxq_config(size_t rx_ring_size) {\n    ring_config r;\n    r.size = rx_ring_size;\n    r.descs = _rxq_storage.get();\n    r.mergable_buffers = true;\n    common_config(r);\n    return r;\n}\n\nvoid\nqp::rx_start() {\n    _rxq.run();\n}\n\nuint32_t\nqp::send(circular_buffer<packet>& p) {\n    return _txq.post(p);\n}\n\nclass qp_vhost : public qp {\nprivate:\n    // The vhost file descriptor needs to remain open throughout the life of\n    // this driver, as as soon as we close it, vhost stops servicing us.\n    file_desc _vhost_fd;\npublic:\n    qp_vhost(device* dev, const native_stack_options& opts);\n};\n\nstatic size_t config_ring_size(const virtio_options& opts) {\n    if (opts.event_index) {\n        return opts.virtio_ring_size.get_value();\n    } else {\n        return 256;\n    }\n}\n\nqp_vhost::qp_vhost(device *dev, const native_stack_options& opts)\n    : qp(dev, config_ring_size(opts.virtio_opts), config_ring_size(opts.virtio_opts))\n    , _vhost_fd(file_desc::open(\"/dev/vhost-net\", O_RDWR))\n{\n    auto tap_device = opts.tap_device.get_value();\n    int64_t vhost_supported_features;\n    _vhost_fd.ioctl(VHOST_GET_FEATURES, vhost_supported_features);\n    vhost_supported_features &= _dev->features();\n    _vhost_fd.ioctl(VHOST_SET_FEATURES, vhost_supported_features);\n    if (vhost_supported_features & VIRTIO_NET_F_MRG_RXBUF) {\n        _header_len = sizeof(net_hdr_mrg);\n    } else {\n        _header_len = sizeof(net_hdr);\n    }\n\n    // Open and set up the tap device, which we'll tell vhost to use.\n    // Note that the tap_fd we open here will be closed at the end of\n    // this function. It appears that this is fine - i.e., after we pass\n    // this fd to VHOST_NET_SET_BACKEND, the Linux kernel keeps the reference\n    // to it and it's fine to close the file descriptor.\n    file_desc tap_fd(file_desc::open(\"/dev/net/tun\", O_RDWR | O_NONBLOCK));\n    SEASTAR_ASSERT(tap_device.size() + 1 <= IFNAMSIZ);\n    ifreq ifr = {};\n    ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE | IFF_VNET_HDR;\n    strcpy(ifr.ifr_ifrn.ifrn_name, tap_device.c_str());\n    tap_fd.ioctl(TUNSETIFF, ifr);\n    unsigned int offload = 0;\n    auto hw_features = _dev->hw_features();\n    if (hw_features.tx_csum_l4_offload && hw_features.rx_csum_offload) {\n        offload = TUN_F_CSUM;\n        if (hw_features.tx_tso) {\n            offload |= TUN_F_TSO4;\n        }\n        if (hw_features.tx_ufo) {\n            offload |= TUN_F_UFO;\n        }\n    }\n    tap_fd.ioctl(TUNSETOFFLOAD, offload);\n    tap_fd.ioctl(TUNSETVNETHDRSZ, _header_len);\n\n    // Additional vhost setup:\n    _vhost_fd.ioctl(VHOST_SET_OWNER);\n    auto mem_table = make_struct_with_vla(&vhost_memory::regions, 1);\n    mem_table->nregions = 1;\n    auto& region = mem_table->regions[0];\n    region.guest_phys_addr = 0;\n    region.memory_size = (size_t(1) << 47) - 4096;\n    region.userspace_addr = 0;\n    region.flags_padding = 0;\n    _vhost_fd.ioctl(VHOST_SET_MEM_TABLE, *mem_table);\n    vhost_vring_state vvs0 = { 0, _rxq.getconfig().size };\n    _vhost_fd.ioctl(VHOST_SET_VRING_NUM, vvs0);\n    vhost_vring_state vvs1 = { 1, _txq.getconfig().size };\n    _vhost_fd.ioctl(VHOST_SET_VRING_NUM, vvs1);\n    auto tov = [](char* x) { return reinterpret_cast<uintptr_t>(x); };\n\n    _vhost_fd.ioctl(VHOST_SET_VRING_ADDR, vhost_vring_addr{\n        0, 0, tov(_rxq.getconfig().descs), tov(_rxq.getconfig().used),\n        tov(_rxq.getconfig().avail), 0\n    });\n    _vhost_fd.ioctl(VHOST_SET_VRING_ADDR, vhost_vring_addr{\n        1, 0, tov(_txq.getconfig().descs), tov(_txq.getconfig().used),\n        tov(_txq.getconfig().avail), 0\n    });\n\n    readable_eventfd _txq_notify;\n    writeable_eventfd _txq_kick;\n    readable_eventfd _rxq_notify;\n    writeable_eventfd _rxq_kick;\n    _vhost_fd.ioctl(VHOST_SET_VRING_KICK, vhost_vring_file{0, _rxq_kick.get_read_fd()});\n    _vhost_fd.ioctl(VHOST_SET_VRING_CALL, vhost_vring_file{0, _rxq_notify.get_write_fd()});\n    _vhost_fd.ioctl(VHOST_SET_VRING_KICK, vhost_vring_file{1, _txq_kick.get_read_fd()});\n    _vhost_fd.ioctl(VHOST_SET_VRING_CALL, vhost_vring_file{1, _txq_notify.get_write_fd()});\n    _rxq.set_notifier(std::make_unique<notifier_vhost>(std::move(_rxq_kick)));\n    _txq.set_notifier(std::make_unique<notifier_vhost>(std::move(_txq_kick)));\n\n    _vhost_fd.ioctl(VHOST_NET_SET_BACKEND, vhost_vring_file{0, tap_fd.get()});\n    _vhost_fd.ioctl(VHOST_NET_SET_BACKEND, vhost_vring_file{1, tap_fd.get()});\n}\n\nstd::unique_ptr<net::qp> device::init_local_queue(const program_options::option_group& opts, uint16_t qid) {\n    static bool called = false;\n    SEASTAR_ASSERT(!qid);\n    SEASTAR_ASSERT(!called);\n    called = true;\n\n    auto net_opts = dynamic_cast<const net::native_stack_options*>(&opts);\n    SEASTAR_ASSERT(net_opts);\n    return std::make_unique<qp_vhost>(this, *net_opts);\n}\n\n}\n\nnet::virtio_options::virtio_options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"Virtio net options\")\n    , event_index(*this, \"event-index\",\n                \"on\",\n                \"Enable event-index feature (on / off)\")\n    , csum_offload(*this, \"csum-offload\",\n                \"on\",\n                \"Enable checksum offload feature (on / off)\")\n    , tso(*this, \"tso\",\n                \"on\",\n                \"Enable TCP segment offload feature (on / off)\")\n    , ufo(*this, \"ufo\",\n                \"on\",\n                \"Enable UDP fragmentation offload feature (on / off)\")\n    , virtio_ring_size(*this, \"virtio-ring-size\",\n                256,\n                \"Virtio ring size (must be power-of-two)\")\n{\n}\n\nstd::unique_ptr<net::device> create_virtio_net_device(const virtio_options& opts, const program_options::value<std::string>& lro) {\n    return std::make_unique<virtio::device>(opts, lro);\n}\n\n}\n"
  },
  {
    "path": "src/proto/metrics2.proto",
    "content": "// Copyright 2013 Prometheus Team\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\n// This is copied and lightly edited from\n// github.com/prometheus/client_model/io/prometheus/client/metrics.proto\n// and finally converted to proto3 syntax to make it usable for the\n// gogo-protobuf approach taken within prometheus/prometheus.\n\nsyntax = \"proto3\";\n\npackage io.prometheus.client;\noption go_package = \"io_prometheus_client\";\n\n//import \"google/protobuf/gogo.proto\";\nimport \"google/protobuf/timestamp.proto\";\n\nmessage LabelPair {\n  string name  = 1;\n  string value = 2;\n}\n\nenum MetricType {\n  // COUNTER must use the Metric field \"counter\".\n  COUNTER = 0;\n  // GAUGE must use the Metric field \"gauge\".\n  GAUGE = 1;\n  // SUMMARY must use the Metric field \"summary\".\n  SUMMARY = 2;\n  // UNTYPED must use the Metric field \"untyped\".\n  UNTYPED = 3;\n  // HISTOGRAM must use the Metric field \"histogram\".\n  HISTOGRAM = 4;\n  // GAUGE_HISTOGRAM must use the Metric field \"histogram\".\n  GAUGE_HISTOGRAM = 5;\n}\n\nmessage Gauge {\n  double value = 1;\n}\n\nmessage Counter {\n  double   value    = 1;\n  Exemplar exemplar = 2;\n}\n\nmessage Quantile {\n  double quantile = 1;\n  double value    = 2;\n}\n\nmessage Summary {\n  uint64            sample_count = 1;\n  double            sample_sum   = 2;\n  repeated Quantile quantile     = 3 ;// [(gogoproto.nullable) = false];\n}\n\nmessage Untyped {\n  double value = 1;\n}\n\nmessage Histogram {\n  uint64 sample_count       = 1;\n  double sample_count_float = 4; // Overrides sample_count if > 0.\n  double sample_sum         = 2;\n  // Buckets for the conventional histogram.\n  repeated Bucket bucket = 3 ;// [(gogoproto.nullable) = false]; // Ordered in increasing order of upper_bound, +Inf bucket is optional.\n\n  // Everything below here is for native histograms (also known as sparse histograms).\n  // Native histograms are an experimental feature without stability guarantees.\n\n  // schema defines the bucket schema. Currently, valid numbers are -4 <= n <= 8.\n  // They are all for base-2 bucket schemas, where 1 is a bucket boundary in each case, and\n  // then each power of two is divided into 2^n logarithmic buckets.\n  // Or in other words, each bucket boundary is the previous boundary times 2^(2^-n).\n  // In the future, more bucket schemas may be added using numbers < -4 or > 8.\n  sint32 schema           = 5;\n  double zero_threshold   = 6; // Breadth of the zero bucket.\n  uint64 zero_count       = 7; // Count in zero bucket.\n  double zero_count_float = 8; // Overrides sb_zero_count if > 0.\n\n  // Negative buckets for the native histogram.\n  repeated BucketSpan negative_span = 9 ;// [(gogoproto.nullable) = false];\n  // Use either \"negative_delta\" or \"negative_count\", the former for\n  // regular histograms with integer counts, the latter for float\n  // histograms.\n  repeated sint64 negative_delta = 10; // Count delta of each bucket compared to previous one (or to zero for 1st bucket).\n  repeated double negative_count = 11; // Absolute count of each bucket.\n\n  // Positive buckets for the native histogram.\n  repeated BucketSpan positive_span = 12 ;// [(gogoproto.nullable) = false];\n  // Use either \"positive_delta\" or \"positive_count\", the former for\n  // regular histograms with integer counts, the latter for float\n  // histograms.\n  repeated sint64 positive_delta = 13; // Count delta of each bucket compared to previous one (or to zero for 1st bucket).\n  repeated double positive_count = 14; // Absolute count of each bucket.\n}\n\nmessage Bucket {\n  uint64   cumulative_count       = 1; // Cumulative in increasing order.\n  double   cumulative_count_float = 4; // Overrides cumulative_count if > 0.\n  double   upper_bound            = 2; // Inclusive.\n  Exemplar exemplar               = 3;\n}\n\n// A BucketSpan defines a number of consecutive buckets in a native\n// histogram with their offset. Logically, it would be more\n// straightforward to include the bucket counts in the Span. However,\n// the protobuf representation is more compact in the way the data is\n// structured here (with all the buckets in a single array separate\n// from the Spans).\nmessage BucketSpan {\n  sint32 offset = 1; // Gap to previous span, or starting point for 1st span (which can be negative).\n  uint32 length = 2; // Length of consecutive buckets.\n}\n\nmessage Exemplar {\n  repeated LabelPair        label     = 1 ;// [(gogoproto.nullable) = false];\n  double                    value     = 2;\n  google.protobuf.Timestamp timestamp = 3; // OpenMetrics-style.\n}\n\nmessage Metric {\n  repeated LabelPair label        = 1 ;// [(gogoproto.nullable) = false];\n  Gauge              gauge        = 2;\n  Counter            counter      = 3;\n  Summary            summary      = 4;\n  Untyped            untyped      = 5;\n  Histogram          histogram    = 7;\n  int64              timestamp_ms = 6;\n}\n\nmessage MetricFamily {\n  string          name   = 1;\n  string          help   = 2;\n  MetricType      type   = 3;\n  repeated Metric metric = 4 ;// [(gogoproto.nullable) = false];\n}\n"
  },
  {
    "path": "src/rpc/lz4_compressor.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 Scylladb, Ltd.\n */\n\n#include <seastar/rpc/lz4_compressor.hh>\n#include <seastar/core/byteorder.hh>\n#include <lz4.h>\n\nnamespace seastar {\n\nnamespace rpc {\n\nsstring lz4_compressor::name() const {\n    return factory{}.supported();\n}\n\nconst sstring& lz4_compressor::factory::supported() const {\n    const static sstring name = \"LZ4\";\n    return name;\n}\n\nstd::unique_ptr<rpc::compressor> lz4_compressor::factory::negotiate(sstring feature, bool is_server) const {\n    return feature == supported() ? std::make_unique<lz4_compressor>() : nullptr;\n}\n\n// Reusable contiguous buffers needed for LZ4 compression and decompression functions.\nclass reusable_buffer {\n    static constexpr size_t chunk_size = 128 * 1024;\n    static_assert(snd_buf::chunk_size == chunk_size, \"snd_buf::chunk_size == chunk_size\");\n\n    std::unique_ptr<char[]> _data;\n    size_t _size;\nprivate:\n    void reserve(size_t n) {\n        if (_size < n) {\n            _data.reset();\n            // Not using std::make_unique to avoid value-initialisation.\n            _data = std::unique_ptr<char[]>(new char[n]);\n            _size = n;\n        }\n    }\npublic:\n    // Returns a pointer to a contiguous buffer containing all data stored in input.\n    // The pointer remains valid until next call to this.\n    const char* prepare(const std::variant<std::vector<temporary_buffer<char>>, temporary_buffer<char>>& input, size_t size) {\n        if (const auto single = std::get_if<temporary_buffer<char>>(&input)) {\n            return single->get();\n        }\n        reserve(size);\n        auto dst = _data.get();\n        for (const auto& fragment : std::get<std::vector<temporary_buffer<char>>>(input)) {\n            dst = std::copy_n(fragment.begin(), fragment.size(), dst);\n        }\n        return _data.get();\n    }\n\n    // Calls function fn passing to it a pointer to a temporary contigiuous max_size\n    // buffer.\n    // fn is supposed to return the actual size of the data.\n    // with_reserved() returns an Output object (snd_buf or rcv_buf or compatible),\n    // containing data that was written to the temporary buffer.\n    // Output should be either snd_buf or rcv_buf.\n    template<typename Output, typename Function>\n    requires requires (Function fn, char* ptr) {\n        { fn(ptr) } -> std::convertible_to<size_t>;\n    } && (std::is_same_v<Output, snd_buf> || std::is_same_v<Output, rcv_buf>)\n    Output with_reserved(size_t max_size, Function&& fn) {\n        if (max_size <= chunk_size) {\n            auto dst = temporary_buffer<char>(max_size);\n            size_t dst_size = fn(dst.get_write());\n            dst.trim(dst_size);\n            return Output(std::move(dst));\n        }\n\n        reserve(max_size);\n        size_t dst_size = fn(_data.get());\n        if (dst_size <= chunk_size) {\n            return Output(temporary_buffer<char>(_data.get(), dst_size));\n        }\n\n        auto left = dst_size;\n        auto pos = _data.get();\n        std::vector<temporary_buffer<char>> buffers;\n        while (left) {\n            auto this_size = std::min(left, chunk_size);\n            buffers.emplace_back(this_size);\n            std::copy_n(pos, this_size, buffers.back().get_write());\n            pos += this_size;\n            left -= this_size;\n        }\n        return Output(std::move(buffers), dst_size);\n    }\n\n    void clear() noexcept {\n        _data.reset();\n        _size = 0;\n    }\n};\n\nstatic thread_local reusable_buffer reusable_buffer_compressed_data;\nstatic thread_local reusable_buffer reusable_buffer_decompressed_data;\nstatic thread_local size_t buffer_use_count = 0;\nstatic constexpr size_t drop_buffers_trigger = 100'000;\n\nstatic void after_buffer_use() noexcept {\n    if (buffer_use_count++ == drop_buffers_trigger) {\n        reusable_buffer_compressed_data.clear();\n        reusable_buffer_decompressed_data.clear();\n        buffer_use_count = 0;\n    }\n}\n\nsnd_buf lz4_compressor::compress(size_t head_space, snd_buf data) {\n    head_space += 4;\n    auto dst_size = head_space + LZ4_compressBound(data.size);\n    auto dst = reusable_buffer_compressed_data.with_reserved<snd_buf>(dst_size, [&] (char* dst) {\n        auto src_size = data.size;\n        auto src = reusable_buffer_decompressed_data.prepare(data.bufs, data.size);\n\n        auto size = LZ4_compress_default(src, dst + head_space, src_size, LZ4_compressBound(src_size));\n        if (size == 0) {\n            throw std::runtime_error(\"RPC frame LZ4 compression failure\");\n        }\n        write_le<uint32_t>(dst + (head_space - 4), src_size);\n        return size + head_space;\n    });\n    after_buffer_use();\n    return dst;\n}\n\nrcv_buf lz4_compressor::decompress(rcv_buf data) {\n    if (data.size < 4) {\n        return rcv_buf();\n    } else {\n        auto src_size = data.size;\n        auto src = reusable_buffer_decompressed_data.prepare(data.bufs, data.size);\n\n        auto dst_size = read_le<uint32_t>(src);\n        if (!dst_size) {\n            throw std::runtime_error(\"RPC frame LZ4 decompression failure: decompressed size cannot be zero\");\n        }\n        src += sizeof(uint32_t);\n        src_size -= sizeof(uint32_t);\n\n        auto dst = reusable_buffer_compressed_data.with_reserved<rcv_buf>(dst_size, [&] (char* dst) {\n            if (LZ4_decompress_safe(src, dst, src_size, dst_size) < 0) {\n                throw std::runtime_error(\"RPC frame LZ4 decompression failure\");\n            }\n            return dst_size;\n        });\n        after_buffer_use();\n        return dst;\n    }\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/rpc/lz4_fragmented_compressor.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Scylladb, Ltd.\n */\n\n#include <seastar/rpc/lz4_fragmented_compressor.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/util/assert.hh>\n\n#include <lz4.h>\n// LZ4_DECODER_RING_BUFFER_SIZE macro is introduced since v1.8.2\n// To work with previous lz4 release, copied the definition in lz4 here\n#ifndef LZ4_DECODER_RING_BUFFER_SIZE\n#define LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize) (65536 + 14 + (maxBlockSize))\n#endif\n\nnamespace seastar {\nnamespace rpc {\n\nsstring lz4_fragmented_compressor::name() const {\n    return factory{}.supported();\n}\n\nconst sstring& lz4_fragmented_compressor::factory::supported() const {\n    const static sstring name = \"LZ4_FRAGMENTED\";\n    return name;\n}\n\nstd::unique_ptr<rpc::compressor> lz4_fragmented_compressor::factory::negotiate(sstring feature, bool is_server) const {\n    return feature == supported() ? std::make_unique<lz4_fragmented_compressor>() : nullptr;\n}\n\n// Compressed message format:\n// The message consists of one or more data chunks each preceeded by a 4 byte header.\n// The value of the header detrmines the size of the chunk:\n// - most significant bit is cleared: intermediate chunk, 31 least significant bits\n//   contain the compressed size of the chunk (i.e. how it appears on wire), the\n//   decompressed size is 32 kB.\n// - most significant bit is set: last chunk, 31 least significant bits contain the\n//   decompressed size of the chunk, the compressed size is the remaining part of\n//   the message.\n// Compression and decompression is done using LZ4 streaming interface. Each chunk\n// depends on the one that precedes it.\n// All metadata is little-endian.\n\nstatic constexpr uint32_t last_chunk_flag = uint32_t(1) << 31;\nstatic constexpr size_t chunk_header_size = sizeof(uint32_t);\nstatic constexpr size_t chunk_size = 32 * 1024;\n\nnamespace {\n\nstruct compression_stream_deleter {\n    void operator()(LZ4_stream_t* stream) const noexcept {\n        LZ4_freeStream(stream);\n    }\n};\n\nstruct decompression_stream_deleter {\n    void operator()(LZ4_streamDecode_t* stream) const noexcept {\n        LZ4_freeStreamDecode(stream);\n    }\n};\n\n}\n\nsnd_buf lz4_fragmented_compressor::compress(size_t head_space, snd_buf data) {\n    static thread_local auto stream = std::unique_ptr<LZ4_stream_t, compression_stream_deleter>(LZ4_createStream());\n    static_assert(chunk_size <= snd_buf::chunk_size, \"chunk_size <= snd_buf::chunk_size\");\n\n    LZ4_resetStream(stream.get());\n\n    auto size_left = data.size;\n    auto src = std::get_if<temporary_buffer<char>>(&data.bufs);\n    if (!src) {\n        src = std::get<std::vector<temporary_buffer<char>>>(data.bufs).data();\n    }\n\n    auto single_chunk_size = LZ4_COMPRESSBOUND(size_left) + head_space + chunk_header_size;\n    if (single_chunk_size <= chunk_size && size_left <= chunk_size && src->size() == size_left) {\n        // faster path for small messages\n        auto dst = temporary_buffer<char>(single_chunk_size);\n        auto header = dst.get_write() + head_space;\n        auto compressed_data = header + chunk_header_size;\n        auto compressed_size = LZ4_compress_fast_continue(stream.get(), src->get(), compressed_data, size_left, LZ4_COMPRESSBOUND(size_left), 0);\n        write_le(header, last_chunk_flag | size_left);\n        dst.trim(head_space + chunk_header_size + compressed_size);\n        return snd_buf(std::move(dst));\n    }\n\n    static constexpr size_t chunk_compress_bound = LZ4_COMPRESSBOUND(chunk_size);\n    static constexpr size_t chunk_maximum_compressed_size = chunk_compress_bound + chunk_header_size;\n    static_assert(chunk_maximum_compressed_size < snd_buf::chunk_size, \"chunk_maximum_compressed_size is too large\");\n\n    std::vector<temporary_buffer<char>> dst_buffers;\n    size_t dst_offset = head_space;\n\n    dst_buffers.emplace_back(std::max<size_t>(head_space, snd_buf::chunk_size));\n\n    // Intermediate chunks\n    size_t total_compressed_size = 0;\n    auto src_left = data.size;\n    size_t src_current_offset = 0;\n\n    // Advance offset in the current source fragment, move to the next fragment if needed.\n    auto advance_src = [&] (size_t n) {\n        src_current_offset += n;\n        if (src_current_offset >= src->size()) {\n            ++src;\n            src_current_offset = 0;\n        }\n        src_left -= n;\n    };\n\n    // Input chunks do not have to be multiplies of chunk_size.\n    // We handle such cases by reassembling a chunk in this temporary buffer.\n    // Note that this is similar to the ring buffer compression case in docs,\n    // we need to ensure that a suitable amount of (maybe) previous data is\n    // stable in this or input buffer, thus make the temp buffer\n    // LZ4_DECODER_RING_BUFFER_SIZE(chunk_size) large, and treat it as a ring.\n    static constexpr auto lin_buf_size = LZ4_DECODER_RING_BUFFER_SIZE(chunk_size);\n    static thread_local char temporary_chunk_data[lin_buf_size];\n    size_t lin_off = 0;\n\n    auto maybe_linearize = [&](size_t size) {\n        auto src_ptr = src->get() + src_current_offset;\n        if (src->size() - src_current_offset < size) {\n            auto left = size;\n            SEASTAR_ASSERT(lin_buf_size > size);\n            if (lin_buf_size - lin_off < size) {\n                lin_off = 0;\n            }\n            auto tmp = temporary_chunk_data + std::exchange(lin_off, lin_off + size);\n            src_ptr = tmp;\n            while (left) {\n                auto this_size = std::min(src->size() - src_current_offset, left);\n                tmp = std::copy_n(src->get() + src_current_offset, this_size, tmp);\n                left -= this_size;\n                advance_src(this_size);\n            }\n        } else {\n            advance_src(chunk_size);\n            lin_off = 0;\n        }\n        return src_ptr;\n    };\n\n    while (src_left > chunk_size) {\n        // Check if we can fit another chunk in the current destination fragment.\n        // If not allocate a new one.\n        if (dst_offset + chunk_maximum_compressed_size > dst_buffers.back().size()) {\n            dst_buffers.back().trim(dst_offset);\n            dst_buffers.emplace_back(snd_buf::chunk_size);\n            dst_offset = 0;\n        }\n\n        // Check if there is at least a contiguous chunk_size of data in the current\n        // source fragment. If not, linearise that into temporary_chunk_data.\n        auto src_ptr = maybe_linearize(chunk_size);\n        auto header = dst_buffers.back().get_write() + dst_offset;\n        auto dst = header + chunk_header_size;\n\n        auto compressed_size = LZ4_compress_fast_continue(stream.get(), src_ptr, dst, chunk_size, chunk_compress_bound, 0);\n        total_compressed_size += compressed_size + chunk_header_size;\n\n        dst_offset += compressed_size + chunk_header_size;\n        write_le<uint32_t>(header, compressed_size);\n    }\n\n    // Last chunk\n    auto last_chunk_compress_bound = LZ4_COMPRESSBOUND(src_left);\n    auto last_chunk_maximum_compressed_size = last_chunk_compress_bound + chunk_header_size;\n\n    // Check if we can fit the last chunk in the current destination fragment. Allocate a new one if not.\n    if (dst_offset + last_chunk_maximum_compressed_size > dst_buffers.back().size()) {\n        dst_buffers.back().trim(dst_offset);\n        dst_buffers.emplace_back(snd_buf::chunk_size);\n        dst_offset = 0;\n    }\n    auto header = dst_buffers.back().get_write() + dst_offset;\n    auto dst = header + chunk_header_size;\n\n    // Check if all remaining source data is contiguous. If not linearise it into temporary_chunk_data.\n    auto rem = src_left;\n    auto src_ptr = maybe_linearize(src_left);\n\n    auto compressed_size = LZ4_compress_fast_continue(stream.get(), src_ptr, dst, rem, last_chunk_compress_bound, 0);\n    dst_offset += compressed_size + chunk_header_size;\n    write_le<uint32_t>(header, last_chunk_flag | rem);\n    total_compressed_size += compressed_size + chunk_header_size + head_space;\n\n    auto& last = dst_buffers.back();\n    last.trim(dst_offset);\n\n    if (dst_buffers.size() == 1) {\n        return snd_buf(std::move(dst_buffers.front()));\n    }\n    return snd_buf(std::move(dst_buffers), total_compressed_size);\n}\n\nrcv_buf lz4_fragmented_compressor::decompress(rcv_buf data) {\n    if (data.size < 4) {\n        return rcv_buf();\n    }\n\n    static thread_local auto stream = std::unique_ptr<LZ4_streamDecode_t, decompression_stream_deleter>(LZ4_createStreamDecode());\n\n    if (!LZ4_setStreamDecode(stream.get(), nullptr, 0)) {\n        throw std::runtime_error(\"RPC frame LZ4_FRAGMENTED decompression failed to reset state\");\n    }\n\n    auto src = std::get_if<temporary_buffer<char>>(&data.bufs);\n    size_t src_left = data.size;\n    size_t src_offset = 0;\n\n    // Prepare source data. Returns pointer to n contiguous bytes of source data.\n    // Avoids copy if possible, otherwise uses dst as a temporary storage.\n    auto copy_src = [&] (char* dst, size_t n) -> const char* {\n        // Fast path, no need to copy anything.\n        if (src->size() - src_offset >= n) {\n            auto ptr = src->get() + src_offset;\n            src_left -= n;\n            src_offset += n;\n            return ptr;\n        }\n\n        // Need to linearise source chunk into dst.\n        auto ptr = dst;\n        src_left -= n;\n        while (n) {\n            if (src_offset == src->size()) {\n                ++src;\n                src_offset = 0;\n            }\n            auto this_size = std::min(n, src->size() - src_offset);\n            std::copy_n(src->get() + src_offset, this_size, dst);\n            n -= this_size;\n            dst += this_size;\n            src_offset += this_size;\n        }\n        return ptr;\n    };\n\n    // Read, possibly fragmented, header.\n    auto read_header = [&] {\n        uint32_t header_value;\n        auto ptr = copy_src(reinterpret_cast<char*>(&header_value), chunk_header_size);\n        if (ptr != reinterpret_cast<char*>(&header_value)) {\n            std::copy_n(ptr, sizeof(uint32_t), reinterpret_cast<char*>(&header_value));\n        }\n        return le_to_cpu(header_value);\n    };\n\n    if (src) {\n        auto header = read_le<uint32_t>(src->get());\n        if (header & last_chunk_flag) {\n            // faster path for small messages: single chunk in a single buffer\n            header &= ~last_chunk_flag;\n            src_offset += chunk_header_size;\n            src_left -= chunk_header_size;\n            auto dst = temporary_buffer<char>(header);\n            if (LZ4_decompress_safe_continue(stream.get(), src->get() + src_offset, dst.get_write(), src_left, header) < 0) {\n                throw std::runtime_error(\"RPC frame LZ4_FRAGMENTED decompression failure (short)\");\n            }\n            return rcv_buf(std::move(dst));\n        }\n        // not eligible for fast path: multiple chunks in a single buffer\n    } else {\n        // not eligible for fast path: multiple buffers\n        src = std::get<std::vector<temporary_buffer<char>>>(data.bufs).data();\n    }\n\n    // Let's be a bit paranoid and not assume that the remote has the same\n    // LZ4_COMPRESSBOUND as we do and allow any compressed chunk size.\n    static thread_local auto chunk_buffer = temporary_buffer<char>(LZ4_COMPRESSBOUND(chunk_size));\n\n    std::vector<temporary_buffer<char>> dst_buffers;\n    size_t total_size = 0;\n\n    // Decompressing requires either dest to be fully split or\n    // \"preserved\" in 64KB or larger, depending on how it was\n    // compressed. If not, decompression will fail, typically\n    // on text-like constructs. Making our dest buffers 64K\n    // ensures we retain a suitable dictionary region for all\n    // passes.\n    constexpr auto buf_size = 64 * 1024;\n    size_t dst_offset = 0;\n\n    auto get_dest = [&](size_t size) {\n        if (dst_buffers.empty()) {\n            dst_buffers.emplace_back(buf_size);\n        }\n        if (dst_buffers.back().size() - dst_offset < size) {\n            dst_buffers.back().trim(dst_offset);\n            dst_buffers.emplace_back(buf_size);\n            dst_offset = 0;\n        }\n        return dst_buffers.back().get_write() + std::exchange(dst_offset, dst_offset + size);\n    };\n\n    // Intermediate chunks\n    uint32_t header_value = read_header();\n    while (!(header_value & last_chunk_flag)) {\n        total_size += chunk_size;\n        if (chunk_buffer.size() < header_value) {\n            chunk_buffer = temporary_buffer<char>(header_value);\n        }\n        auto src_ptr = copy_src(chunk_buffer.get_write(), header_value);\n        auto dst = get_dest(chunk_size);\n        if (LZ4_decompress_safe_continue(stream.get(), src_ptr, /*dst_buffers.back().get_write()*/dst, header_value, chunk_size) < 0) {\n            throw std::runtime_error(format(\"RPC frame LZ4_FRAGMENTED decompression failure (long, at {} bytes)\", total_size - chunk_size));\n        }\n        header_value = read_header();\n    }\n\n    // Last chunk\n    header_value &= ~last_chunk_flag;\n    total_size += header_value;\n    auto dst = get_dest(header_value);\n    if (chunk_buffer.size() < src_left) {\n        chunk_buffer = temporary_buffer<char>(src_left);\n    }\n    auto last_chunk_compressed_size = src_left;\n    auto src_ptr = copy_src(chunk_buffer.get_write(), src_left);\n    if (LZ4_decompress_safe_continue(stream.get(), src_ptr, /*dst_buffers.back().get_write()*/dst, last_chunk_compressed_size, header_value) < 0) {\n        throw std::runtime_error(format(\"RPC frame LZ4_FRAGMENTED decompression failure (long, last frame, at {} bytes)\", total_size - header_value));\n    }\n\n    dst_buffers.back().trim(dst_offset);\n\n    if (dst_buffers.size() == 1) {\n        return rcv_buf(std::move(dst_buffers.front()));\n    }\n    return rcv_buf(std::move(dst_buffers), total_size);\n}\n\n}\n}\n"
  },
  {
    "path": "src/rpc/rpc.cc",
    "content": "#include <seastar/rpc/rpc.hh>\n#include <seastar/rpc/multi_algo_compressor_factory.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/switch_to.hh>\n#include <seastar/util/memory-data-source.hh>\n#include <seastar/util/assert.hh>\n#include <boost/range/adaptor/map.hpp>\n#include <boost/range/adaptor/transformed.hpp>\n#include <boost/algorithm/string.hpp>\n#include <boost/range/numeric.hpp>\n#include <fmt/ostream.h>\n\ntemplate <> struct fmt::formatter<seastar::rpc::streaming_domain_type> : fmt::ostream_formatter {};\n\nnamespace seastar {\n\nnamespace rpc {\n\nvoid logger::operator()(const client_info& info, id_type msg_id, const sstring& str) const {\n    log(fmt::format(\"client {} msg_id {}:  {}\", info.addr, msg_id, str));\n}\n\nvoid logger::operator()(const client_info& info, id_type msg_id, log_level level, std::string_view str) const {\n    log(level, \"client {} msg_id {}:  {}\", info.addr, msg_id, str);\n}\n\nvoid logger::operator()(const client_info& info, const sstring& str) const {\n    (*this)(info.addr, str);\n}\n\nvoid logger::operator()(const client_info& info, log_level level, std::string_view str) const {\n    (*this)(info.addr, level, str);\n}\n\nvoid logger::operator()(const socket_address& addr, const sstring& str) const {\n    log(fmt::format(\"client {}: {}\", addr, str));\n}\n\nvoid logger::operator()(const socket_address& addr, log_level level, std::string_view str) const {\n    log(level, \"client {}: {}\", addr, str);\n}\n\nno_wait_type no_wait;\n\nsnd_buf::snd_buf(size_t size_) : size(size_) {\n    if (size <= chunk_size) {\n        bufs = temporary_buffer<char>(size);\n    } else {\n        std::vector<temporary_buffer<char>> v;\n        v.reserve(align_up(size_t(size), chunk_size) / chunk_size);\n        while (size_) {\n            v.emplace_back(std::min(chunk_size, size_));\n            size_ -= v.back().size();\n        }\n        bufs = std::move(v);\n    }\n}\n\nsnd_buf::snd_buf(snd_buf&&) noexcept = default;\nsnd_buf& snd_buf::operator=(snd_buf&&) noexcept = default;\n\ntemporary_buffer<char>& snd_buf::front() {\n    auto* one = std::get_if<temporary_buffer<char>>(&bufs);\n    if (one) {\n        return *one;\n    } else {\n        return std::get<std::vector<temporary_buffer<char>>>(bufs).front();\n    }\n}\n\nnamespace internal {\n\n// Make a copy of a remote buffer. No data is actually copied, only pointers and\n// a deleter of a new buffer takes care of deleting the original buffer\nsnd_buf make_shard_local_buffer_copy(snd_buf* org, std::function<deleter(snd_buf*)> make_deleter) {\n    snd_buf buf(org->size);\n    auto* one = std::get_if<temporary_buffer<char>>(&org->bufs);\n\n    if (one) {\n        buf.bufs = temporary_buffer<char>(one->get_write(), one->size(), make_deleter(org));\n    } else {\n        auto& orgbufs = std::get<std::vector<temporary_buffer<char>>>(org->bufs);\n        std::vector<temporary_buffer<char>> newbufs;\n        newbufs.reserve(orgbufs.size());\n        auto d = make_deleter(org);\n        for (auto&& b : orgbufs) {\n            newbufs.emplace_back(b.get_write(), b.size(), d.share());\n        }\n        buf.bufs = std::move(newbufs);\n    }\n\n    return buf;\n}\n\n// Make a copy of a remote buffer. No data is actually copied, only pointers and\n// a deleter of a new buffer takes care of deleting the original buffer\nrcv_buf make_shard_local_buffer_copy(foreign_ptr<std::unique_ptr<rcv_buf>> org) {\n    if (org.get_owner_shard() == this_shard_id()) {\n        return std::move(*org);\n    }\n    rcv_buf buf(org->size);\n    auto* one = std::get_if<temporary_buffer<char>>(&org->bufs);\n\n    if (one) {\n        buf.bufs = temporary_buffer<char>(one->get_write(), one->size(), make_object_deleter(std::move(org)));\n    } else {\n        auto& orgbufs = std::get<std::vector<temporary_buffer<char>>>(org->bufs);\n        std::vector<temporary_buffer<char>> newbufs;\n        newbufs.reserve(orgbufs.size());\n        deleter d = make_object_deleter(std::move(org));\n        for (auto&& b : orgbufs) {\n            newbufs.emplace_back(b.get_write(), b.size(), d.share());\n        }\n        buf.bufs = std::move(newbufs);\n    }\n\n    return buf;\n}\n\n} // namespace internal\n\nstatic void log_exception(connection& c, log_level level, const char* log, std::exception_ptr eptr) {\n    const char* s;\n    try {\n        std::rethrow_exception(eptr);\n    } catch (std::exception& ex) {\n        s = ex.what();\n    } catch (...) {\n        s = \"unknown exception\";\n    }\n    auto formatted = format(\"{}: {}\", log, s);\n    c.get_logger()(c.peer_address(), level, std::string_view(formatted.data(), formatted.size()));\n}\n\nsnd_buf connection::compress(snd_buf buf) {\n    if (_compressor) {\n        buf = _compressor->compress(4, std::move(buf));\n        static_assert(snd_buf::chunk_size >= 4, \"send buffer chunk size is too small\");\n        write_le<uint32_t>(buf.front().get_write(), buf.size - 4);\n        return buf;\n    }\n    return buf;\n}\n\nfuture<> connection::send_buffer(snd_buf buf) {\n    auto* b = std::get_if<temporary_buffer<char>>(&buf.bufs);\n    if (b) {\n        return _connected->write_buf.write(std::move(*b));\n    } else {\n        return do_with(std::move(std::get<std::vector<temporary_buffer<char>>>(buf.bufs)),\n                [this] (std::vector<temporary_buffer<char>>& ar) {\n            return do_for_each(ar.begin(), ar.end(), [this] (auto& b) {\n                return _connected->write_buf.write(std::move(b));\n            });\n        });\n    }\n}\n\nfuture<> connection::send_entry(outgoing_entry& d) noexcept {\n    return futurize_invoke([this, &d] {\n        if (d.buf.size && _propagate_timeout) {\n            static_assert(snd_buf::chunk_size >= sizeof(uint64_t), \"send buffer chunk size is too small\");\n            if (_timeout_negotiated) {\n                auto expire = d.t.get_timeout();\n                uint64_t left = 0;\n                if (expire != typename timer<rpc_clock_type>::time_point()) {\n                    left = std::chrono::duration_cast<std::chrono::milliseconds>(expire - timer<rpc_clock_type>::clock::now()).count();\n                }\n                write_le<uint64_t>(d.buf.front().get_write(), left);\n            } else {\n                d.buf.front().trim_front(sizeof(uint64_t));\n                d.buf.size -= sizeof(uint64_t);\n            }\n        }\n        auto buf = compress(std::move(d.buf));\n        return send_buffer(std::move(buf)).then([this] {\n            _stats.sent_messages++;\n            return _connected->write_buf.flush();\n        });\n    });\n}\n\nvoid connection::set_negotiated() noexcept {\n    _negotiated->set_value();\n    _negotiated = std::nullopt;\n}\n\nfuture<> connection::stop_send_loop(std::exception_ptr ex) {\n    _error = true;\n    if (_connected) {\n        _connected->fd.shutdown_output();\n    }\n    if (ex == nullptr) {\n        ex = std::make_exception_ptr(closed_error());\n    }\n    while (!_outgoing_queue.empty()) {\n        auto it = std::prev(_outgoing_queue.end());\n        // Cancel all but front entry normally. The front entry is sitting in the\n        // send_entry() and cannot be withdrawn, except when _negotiated is still\n        // engaged. In the latter case when it will be aborted below the entry's\n        // continuation will not be called and its done promise will not resolve\n        // the _outgoing_queue_ready, so do it here\n        if (it != _outgoing_queue.begin()) {\n            withdraw(it, ex);\n        } else {\n            if (_negotiated) {\n                it->done.set_exception(ex);\n            }\n            break;\n        }\n    }\n    if (_negotiated) {\n        _negotiated->set_exception(ex);\n    }\n    return when_all(std::move(_outgoing_queue_ready), std::move(_sink_closed_future)).then([this] (std::tuple<future<>, future<bool>> res){\n        // _outgoing_queue_ready might be exceptional if queue drain or\n        // _negotiated abortion set it such\n        std::get<0>(res).ignore_ready_future();\n        // _sink_closed_future is never exceptional\n        bool sink_closed = std::get<1>(res).get();\n        return _connected && !sink_closed ? _connected->write_buf.close() : make_ready_future();\n    });\n}\n\nvoid connection::set_socket(connected_socket&& fd) {\n    if (_connected.has_value()) {\n        throw std::runtime_error(\"already connected\");\n    }\n    _connected.emplace(std::move(fd));\n}\n\nfuture<> connection::send_negotiation_frame(feature_map features) {\n    auto negotiation_frame_feature_record_size = [] (const feature_map::value_type& e) {\n        return 8 + e.second.size();\n    };\n    auto extra_len = boost::accumulate(\n        features | boost::adaptors::transformed(negotiation_frame_feature_record_size), uint32_t(0));\n    temporary_buffer<char> reply(sizeof(negotiation_frame) + extra_len);\n    auto p = reply.get_write();\n    p = std::copy_n(rpc_magic, 8, p);\n    write_le<uint32_t>(p, extra_len);\n    p += 4;\n    for (auto&& e : features) {\n        write_le<uint32_t>(p, static_cast<uint32_t>(e.first));\n        p += 4;\n        write_le<uint32_t>(p, e.second.size());\n        p += 4;\n        p = std::copy_n(e.second.begin(), e.second.size(), p);\n    }\n    return _connected->write_buf.write(std::move(reply)).then([this] {\n        _stats.sent_messages++;\n        return _connected->write_buf.flush();\n    });\n}\n\nvoid connection::withdraw(outgoing_entry::container_t::iterator it, std::exception_ptr ex) {\n    SEASTAR_ASSERT(it != _outgoing_queue.end());\n\n    auto pit = std::prev(it);\n    // Previous entry's (pit's) done future will schedule current entry (it)\n    // continuation. Similarly, it.done will schedule next entry continuation\n    // or will resolve _outgoing_queue_ready future.\n    //\n    // To withdraw \"it\" we need to do two things:\n    // - make pit.done resolve it->next (some time later)\n    // - resolve \"it\"'s continuation right now\n    //\n    // The latter is achieved by resolving pit.done immediatelly, the former\n    // by moving it.done into pit.done. For simplicity (verging on obscurity?)\n    // both done's are just swapped and \"it\" resolves its new promise\n\n    std::swap(it->done, pit->done);\n    it->uncancellable();\n    it->unlink();\n    if (ex == nullptr) {\n        it->done.set_value();\n    } else {\n        it->done.set_exception(ex);\n    }\n}\n\nfuture<> connection::send(snd_buf buf, std::optional<rpc_clock_type::time_point> timeout, cancellable* cancel) {\n    if (!_error) {\n        if (timeout && *timeout <= rpc_clock_type::now()) {\n            return make_ready_future<>();\n        }\n\n        auto p = std::make_unique<outgoing_entry>(std::move(buf));\n        auto& d = *p;\n        _outgoing_queue.push_back(d);\n        _outgoing_queue_size++;\n        auto deleter = [this, it = _outgoing_queue.iterator_to(d)] {\n            // Front entry is most likely (unless _negotiated is unresolved, check enqueue_zero_frame()) sitting\n            // inside send_entry() continuations and thus it cannot be cancelled.\n            if (it != _outgoing_queue.begin()) {\n                withdraw(it);\n            }\n        };\n\n        if (timeout) {\n            auto& t = d.t;\n            t.set_callback(deleter);\n            t.arm(timeout.value());\n        }\n        if (cancel) {\n            cancel->cancel_send = std::move(deleter);\n            cancel->send_back_pointer = &d.pcancel;\n            d.pcancel = cancel;\n        }\n\n        // New entry should continue (do its .then() lambda) after _outgoing_queue_ready\n        // resolves. Next entry will need to do the same after this entry's done resolves.\n        // Thus -- replace _outgoing_queue_ready with d's future and chain its continuation\n        // on ..._ready's old value.\n        return std::exchange(_outgoing_queue_ready, d.done.get_future()).then([this, p = std::move(p)] () mutable {\n            _outgoing_queue_size--;\n            if (__builtin_expect(!p->is_linked(), false)) {\n                // If withdrawn the entry is unlinked and this lambda is fired right at once\n                return make_ready_future<>();\n            }\n\n            p->uncancellable();\n            return send_entry(*p).then_wrapped([this, p = std::move(p)] (auto f) mutable {\n                if (f.failed()) {\n                    f.ignore_ready_future();\n                    abort();\n                }\n                p->done.set_value();\n            });\n        });\n    } else {\n        return make_exception_future<>(closed_error());\n    }\n}\n\nvoid connection::abort() {\n    if (!_error) {\n        _error = true;\n        _connected->fd.shutdown_input();\n    }\n}\n\nfuture<> connection::stop() noexcept {\n    try {\n        abort();\n    } catch (...) {\n        log_exception(*this, log_level::error, \"fail to shutdown connection while stopping\", std::current_exception());\n    }\n    return _stopped.get_future();\n}\n\ntemplate<typename Connection>\nstatic bool verify_frame(Connection& c, temporary_buffer<char>& buf, size_t expected, const char* log) {\n    if (buf.size() != expected) {\n        if (buf.size() != 0) {\n            c.get_logger()(c.peer_address(), log);\n        }\n        return false;\n    }\n    return true;\n}\n\ntemplate<typename Connection>\nstatic\nfuture<feature_map>\nreceive_negotiation_frame(Connection& c, input_stream<char>& in) {\n    return in.read_exactly(sizeof(negotiation_frame)).then([&c, &in] (temporary_buffer<char> neg) {\n        if (!verify_frame(c, neg, sizeof(negotiation_frame), \"unexpected eof during negotiation frame\")) {\n            return make_exception_future<feature_map>(closed_error());\n        }\n        negotiation_frame frame;\n        std::copy_n(neg.get_write(), sizeof(frame.magic), frame.magic);\n        frame.len = read_le<uint32_t>(neg.get_write() + 8);\n        if (std::memcmp(frame.magic, rpc_magic, sizeof(frame.magic)) != 0) {\n            c.get_logger()(c.peer_address(), format(\"wrong protocol magic: {:02x}{:02x}{:02x}{:02x}{:02x}{:02x}{:02x}{:02x}\",\n                frame.magic[0], frame.magic[1], frame.magic[2], frame.magic[3], frame.magic[4], frame.magic[5], frame.magic[6], frame.magic[7]));\n            return make_exception_future<feature_map>(closed_error());\n        }\n        auto len = frame.len;\n        return in.read_exactly(len).then([&c, len] (temporary_buffer<char> extra) {\n            if (extra.size() != len) {\n                c.get_logger()(c.peer_address(), \"unexpected eof during negotiation frame\");\n                return make_exception_future<feature_map>(closed_error());\n            }\n            feature_map map;\n            auto p = extra.get();\n            auto end = p + extra.size();\n            while (p != end) {\n                if (end - p < 8) {\n                    c.get_logger()(c.peer_address(), \"bad feature data format in negotiation frame\");\n                    return make_exception_future<feature_map>(closed_error());\n                }\n                auto feature = static_cast<protocol_features>(read_le<uint32_t>(p));\n                auto f_len = read_le<uint32_t>(p + 4);\n                p += 8;\n                if (f_len > end - p) {\n                    c.get_logger()(c.peer_address(), \"buffer underflow in feature data in negotiation frame\");\n                    return make_exception_future<feature_map>(closed_error());\n                }\n                auto data = sstring(p, f_len);\n                p += f_len;\n                map.emplace(feature, std::move(data));\n            }\n            return make_ready_future<feature_map>(std::move(map));\n        });\n    });\n}\n\ninline future<rcv_buf>\nread_rcv_buf(input_stream<char>& in, uint32_t size) {\n    return in.read_up_to(size).then([&, size] (temporary_buffer<char> data) mutable {\n        rcv_buf rb(size);\n        if (data.size() == 0) {\n            return make_ready_future<rcv_buf>(rcv_buf());\n        } else if (data.size() == size) {\n            rb.bufs = std::move(data);\n            return make_ready_future<rcv_buf>(std::move(rb));\n        } else {\n            size -= data.size();\n            std::vector<temporary_buffer<char>> v;\n            v.push_back(std::move(data));\n            rb.bufs = std::move(v);\n            return do_with(std::move(rb), std::move(size), [&in] (rcv_buf& rb, uint32_t& left) {\n                return repeat([&] () {\n                    return in.read_up_to(left).then([&] (temporary_buffer<char> data) {\n                        if (!data.size()) {\n                            rb.size -= left;\n                            return stop_iteration::yes;\n                        } else {\n                            left -= data.size();\n                            std::get<std::vector<temporary_buffer<char>>>(rb.bufs).push_back(std::move(data));\n                            return left ? stop_iteration::no : stop_iteration::yes;\n                        }\n                    });\n                }).then([&rb] {\n                    return std::move(rb);\n                });\n            });\n        }\n    });\n}\n\ntemplate<typename FrameType>\nfuture<typename FrameType::return_type>\nconnection::read_frame(socket_address info, input_stream<char>& in) {\n    auto header_size = FrameType::header_size();\n    return in.read_exactly(header_size).then([this, header_size, info, &in] (temporary_buffer<char> header) {\n        if (header.size() != header_size) {\n            if (header.size() != 0) {\n                _logger(info, format(\"unexpected eof on a {} while reading header: expected {:d} got {:d}\", FrameType::role(), header_size, header.size()));\n            }\n            return make_ready_future<typename FrameType::return_type>(FrameType::empty_value());\n        }\n        auto [size, h] = FrameType::decode_header(header.get());\n        if (!size) {\n            return make_ready_future<typename FrameType::return_type>(FrameType::make_value(h, rcv_buf()));\n        } else {\n            return read_rcv_buf(in, size).then([this, info, h = std::move(h), size] (rcv_buf rb) {\n                if (rb.size != size) {\n                    _logger(info, format(\"unexpected eof on a {} while reading data: expected {:d} got {:d}\", FrameType::role(), size, rb.size));\n                    return make_ready_future<typename FrameType::return_type>(FrameType::empty_value());\n                } else {\n                    return make_ready_future<typename FrameType::return_type>(FrameType::make_value(h, std::move(rb)));\n                }\n            });\n        }\n    });\n}\n\ntemplate<typename FrameType>\nfuture<typename FrameType::return_type>\nconnection::read_frame_compressed(socket_address info, std::unique_ptr<compressor>& compressor, input_stream<char>& in) {\n    if (compressor) {\n        return in.read_exactly(4).then([this, info, &in, &compressor] (temporary_buffer<char> compress_header) {\n            if (compress_header.size() != 4) {\n                if (compress_header.size() != 0) {\n                    _logger(info, format(\"unexpected eof on a {} while reading compression header: expected 4 got {:d}\", FrameType::role(), compress_header.size()));\n                }\n                return make_ready_future<typename FrameType::return_type>(FrameType::empty_value());\n            }\n            auto ptr = compress_header.get();\n            auto size = read_le<uint32_t>(ptr);\n            return read_rcv_buf(in, size).then([this, size, &compressor, info, &in] (rcv_buf compressed_data) {\n                if (compressed_data.size != size) {\n                    _logger(info, format(\"unexpected eof on a {} while reading compressed data: expected {:d} got {:d}\", FrameType::role(), size, compressed_data.size));\n                    return make_ready_future<typename FrameType::return_type>(FrameType::empty_value());\n                }\n                auto eb = compressor->decompress(std::move(compressed_data));\n                if (eb.size == 0) {\n                    // Empty frames might be sent as means of communication between the compressors, and should be skipped by the RPC layer.\n                    // We skip the empty frame here. We recursively restart the function, as if the empty frame didn't happen.\n                    // The yield() is here to limit the stack depth of the recursion to 1.\n                    return yield().then([this, info, &in, &compressor] { return read_frame_compressed<FrameType>(info, compressor, in); });\n                }\n                auto source = std::visit([] (auto&& b) { return util::as_input_stream(std::move(b)); }, eb.bufs);\n                return do_with(std::move(source), [this, info] (input_stream<char>& in) {\n                    return read_frame<FrameType>(info, in);\n                });\n            });\n        });\n    } else {\n        return read_frame<FrameType>(info, in);\n    }\n}\n\nstruct stream_frame {\n    using opt_buf_type = std::optional<rcv_buf>;\n    using return_type = opt_buf_type;\n    struct header_type {\n        bool eos;\n    };\n    static size_t header_size() {\n        return 4;\n    }\n    static const char* role() {\n        return \"stream\";\n    }\n    static auto empty_value() {\n        return std::nullopt;\n    }\n    static std::pair<uint32_t, header_type> decode_header(const char* ptr) {\n        auto size = read_le<uint32_t>(ptr);\n        return size != -1U ? std::make_pair(size, header_type{false}) : std::make_pair(0U, header_type{true});\n    }\n    static auto make_value(const header_type& t, rcv_buf data) {\n        if (t.eos) {\n            data.size = -1U;\n        }\n        return data;\n    }\n};\n\nfuture<std::optional<rcv_buf>>\nconnection::read_stream_frame_compressed(input_stream<char>& in) {\n    return read_frame_compressed<stream_frame>(peer_address(), _compressor, in);\n}\n\nfuture<> connection::stream_close() {\n    auto f = make_ready_future<>();\n    if (!error()) {\n        promise<bool> p;\n        _sink_closed_future = p.get_future();\n        // stop_send_loop(), which also calls _write_buf.close(), and this code can run in parallel.\n        // Use _sink_closed_future to serialize them and skip second call to close()\n        f = _connected->write_buf.close().finally([p = std::move(p)] () mutable { p.set_value(true);});\n    }\n    return f.finally([this] () mutable { return stop(); });\n}\n\nfuture<> connection::stream_process_incoming(rcv_buf&& buf) {\n    // we do not want to dead lock on huge packets, so let them in\n    // but only one at a time\n    auto size = std::min(size_t(buf.size), max_stream_buffers_memory);\n    return get_units(_stream_sem, size).then([this, buf = std::move(buf)] (semaphore_units<>&& su) mutable {\n        buf.su = std::move(su);\n        return _stream_queue.push_eventually(std::move(buf));\n    });\n}\n\nfuture<> connection::handle_stream_frame() {\n    return read_stream_frame_compressed(_connected->read_buf).then([this] (std::optional<rcv_buf> data) {\n        if (!data) {\n            _error = true;\n            return make_ready_future<>();\n        }\n        return stream_process_incoming(std::move(*data));\n    });\n}\n\nfuture<> connection::stream_receive(circular_buffer<foreign_ptr<std::unique_ptr<rcv_buf>>>& bufs) {\n    return _stream_queue.not_empty().then([this, &bufs] {\n        bool eof = !_stream_queue.consume([&bufs] (rcv_buf&& b) {\n            if (b.size == -1U) { // max fragment length marks an end of a stream\n                return false;\n            } else {\n                bufs.push_back(make_foreign(std::make_unique<rcv_buf>(std::move(b))));\n                return true;\n            }\n        });\n        if (eof && !bufs.empty()) {\n            SEASTAR_ASSERT(_stream_queue.empty());\n            _stream_queue.push(rcv_buf(-1U)); // push eof marker back for next read to notice it\n        }\n    });\n}\n\nvoid connection::register_stream(connection_id id, xshard_connection_ptr c) {\n    _streams.emplace(id, std::move(c));\n}\n\nxshard_connection_ptr connection::get_stream(connection_id id) const {\n    auto it = _streams.find(id);\n    if (it == _streams.end()) {\n        throw std::logic_error(format(\"rpc stream id {} not found\", id).c_str());\n    }\n    return it->second;\n}\n\n// The request frame is\n//   le64 optional timeout (see request_frame_with_timeout below)\n//   le64 message type a.k.a. verb ID\n//   le64 message ID\n//   le32 payload length\n//   ...  payload\nstruct request_frame {\n    using opt_buf_type = std::optional<rcv_buf>;\n    using return_type = std::tuple<std::optional<uint64_t>, uint64_t, int64_t, opt_buf_type>;\n    using header_type = std::tuple<std::optional<uint64_t>, uint64_t, int64_t>;\n    static constexpr size_t raw_header_size = sizeof(uint64_t) + sizeof(int64_t) + sizeof(uint32_t);\n    static size_t header_size() {\n        static_assert(request_frame_headroom >= raw_header_size);\n        return raw_header_size;\n    }\n    static const char* role() {\n        return \"server\";\n    }\n    static auto empty_value() {\n        return std::make_tuple(std::nullopt, uint64_t(0), 0, std::nullopt);\n    }\n    static std::pair<size_t, header_type> decode_header(const char* ptr) {\n        auto type = read_le<uint64_t>(ptr);\n        auto msgid = read_le<int64_t>(ptr + 8);\n        auto size = read_le<uint32_t>(ptr + 16);\n        return std::make_pair(size, std::make_tuple(std::nullopt, type, msgid));\n    }\n    static void encode_header(uint64_t type, int64_t msg_id, snd_buf& buf, size_t off) {\n        auto p = buf.front().get_write() + off;\n        write_le<uint64_t>(p, type);\n        write_le<int64_t>(p + 8, msg_id);\n        write_le<uint32_t>(p + 16, buf.size - raw_header_size - off);\n    }\n    static auto make_value(const header_type& t, rcv_buf data) {\n        return std::make_tuple(std::get<0>(t), std::get<1>(t), std::get<2>(t), std::move(data));\n    }\n};\n\n// This frame is used if protocol_features.TIMEOUT was negotiated\nstruct request_frame_with_timeout : request_frame {\n    using super = request_frame;\n    static constexpr size_t raw_header_size = sizeof(uint64_t) + request_frame::raw_header_size;\n    static size_t header_size() {\n        static_assert(request_frame_headroom >= raw_header_size);\n        return raw_header_size;\n    }\n    static std::pair<uint32_t, typename super::header_type> decode_header(const char* ptr) {\n        auto h = super::decode_header(ptr + 8);\n        std::get<0>(h.second) = read_le<uint64_t>(ptr);\n        return h;\n    }\n    static void encode_header(uint64_t type, int64_t msg_id, snd_buf& buf) {\n        static_assert(snd_buf::chunk_size >= raw_header_size, \"send buffer chunk size is too small\");\n        // expiration timer is encoded later\n        request_frame::encode_header(type, msg_id, buf, 8);\n    }\n};\n\nfuture<> client::request(uint64_t type, int64_t msg_id, snd_buf buf, std::optional<rpc_clock_type::time_point> timeout, cancellable* cancel) {\n    request_frame_with_timeout::encode_header(type, msg_id, buf);\n    return send(std::move(buf), timeout, cancel);\n}\n\nvoid\nclient::negotiate(feature_map provided) {\n    // record features returned here\n    for (auto&& e : provided) {\n        auto id = e.first;\n        switch (id) {\n        // supported features go here\n        case protocol_features::COMPRESS:\n            if (_options.compressor_factory) {\n                _compressor = _options.compressor_factory->negotiate(e.second, false, [this] { return send({}); });\n            }\n            if (!_compressor) {\n                throw std::runtime_error(format(\"RPC server responded with compression {} - unsupported\", e.second));\n            }\n            break;\n        case protocol_features::TIMEOUT:\n            _timeout_negotiated = true;\n            break;\n            case protocol_features::HANDLER_DURATION:\n            _handler_duration_negotiated = true;\n            break;\n        case protocol_features::CONNECTION_ID: {\n            _id = deserialize_connection_id(e.second);\n            break;\n        }\n        default:\n            // nothing to do\n            ;\n        }\n    }\n}\n\nfuture<> client::negotiate_protocol(feature_map features) {\n    return send_negotiation_frame(std::move(features)).then([this] {\n        return receive_negotiation_frame(*this, _connected->read_buf).then([this] (feature_map features) {\n            return negotiate(std::move(features));\n        });\n    });\n}\n\n// The response frame is\n//   le64 message ID\n//   le32 payload size\n//   ...  payload\nstruct response_frame {\n    using opt_buf_type = std::optional<rcv_buf>;\n    using return_type = std::tuple<int64_t, std::optional<uint32_t>, opt_buf_type>;\n    using header_type = std::tuple<int64_t, std::optional<uint32_t>>;\n    static constexpr size_t raw_header_size = sizeof(int64_t) + sizeof(uint32_t);\n    static size_t header_size() {\n        static_assert(response_frame_headroom >= raw_header_size);\n        return raw_header_size;\n    }\n    static const char* role() {\n        return \"client\";\n    }\n    static auto empty_value() {\n        return std::make_tuple(0, std::nullopt, std::nullopt);\n    }\n    static std::pair<uint32_t, header_type> decode_header(const char* ptr) {\n        auto msgid = read_le<int64_t>(ptr);\n        auto size = read_le<uint32_t>(ptr + 8);\n        return std::make_pair(size, std::make_tuple(msgid, std::nullopt));\n    }\n    static void encode_header(int64_t msg_id, snd_buf& data, size_t header_size = raw_header_size) {\n        static_assert(snd_buf::chunk_size >= raw_header_size, \"send buffer chunk size is too small\");\n        auto p = data.front().get_write();\n        write_le<int64_t>(p, msg_id);\n        write_le<uint32_t>(p + 8, data.size - header_size);\n    }\n    static auto make_value(const header_type& t, rcv_buf data) {\n        return std::make_tuple(std::get<0>(t), std::get<1>(t), std::move(data));\n    }\n};\n\n// The response frame is\n//   le64 message ID\n//   le32 payload size\n//   le32 handler duration\n//   ...  payload\nstruct response_frame_with_handler_time : public response_frame {\n    using super = response_frame;\n    static constexpr size_t raw_header_size = super::raw_header_size + sizeof(uint32_t);\n    static size_t header_size() {\n        static_assert(response_frame_headroom >= raw_header_size);\n        return raw_header_size;\n    }\n    static std::pair<uint32_t, header_type> decode_header(const char* ptr) {\n        auto p = super::decode_header(ptr);\n        auto ht = read_le<uint32_t>(ptr + 12);\n        if (ht != -1U) {\n            std::get<1>(p.second) = ht;\n        }\n        return p;\n    }\n    static uint32_t encode_handler_duration(const std::optional<rpc_clock_type::duration>& ht) noexcept {\n        if (ht.has_value()) {\n            std::chrono::microseconds us = std::chrono::duration_cast<std::chrono::microseconds>(*ht);\n            if (us.count() < std::numeric_limits<uint32_t>::max()) {\n                return us.count();\n            }\n        }\n        return -1U;\n    }\n    static void encode_header(int64_t msg_id, std::optional<rpc_clock_type::duration> ht, snd_buf& data) {\n        static_assert(snd_buf::chunk_size >= raw_header_size);\n        auto p = data.front().get_write();\n        super::encode_header(msg_id, data, raw_header_size);\n        write_le<uint32_t>(p + 12, encode_handler_duration(ht));\n    }\n};\n\nfuture<response_frame::return_type>\nclient::read_response_frame_compressed(input_stream<char>& in) {\n    if (_handler_duration_negotiated) {\n        return read_frame_compressed<response_frame_with_handler_time>(_server_addr, _compressor, in);\n    } else {\n        return read_frame_compressed<response_frame>(_server_addr, _compressor, in);\n    }\n}\n\nstats client::get_stats() const {\n    stats res = _stats;\n    res.wait_reply = incoming_queue_length();\n    res.pending = outgoing_queue_length();\n    return res;\n}\n\nvoid client::wait_for_reply(id_type id, std::unique_ptr<reply_handler_base>&& h, std::optional<rpc_clock_type::time_point> timeout, cancellable* cancel) {\n    if (timeout) {\n        h->t.set_callback(std::bind(std::mem_fn(&client::wait_timed_out), this, id));\n        h->t.arm(timeout.value());\n    }\n    if (cancel) {\n        cancel->cancel_wait = [this, id] {\n            _outstanding[id]->cancel();\n            _outstanding.erase(id);\n        };\n        h->pcancel = cancel;\n        cancel->wait_back_pointer = &h->pcancel;\n    }\n    _outstanding.emplace(id, std::move(h));\n}\nvoid client::wait_timed_out(id_type id) {\n    _stats.timeout++;\n    _outstanding[id]->timeout();\n    _outstanding.erase(id);\n}\n\nfuture<> client::stop() noexcept {\n    _error = true;\n    try {\n        _socket.shutdown();\n    } catch(...) {\n        log_exception(*this, log_level::error, \"fail to shutdown connection while stopping\", std::current_exception());\n    }\n    return _stopped.get_future();\n}\n\nvoid client::abort_all_streams() {\n    while (!_streams.empty()) {\n        auto&& s = _streams.begin();\n        SEASTAR_ASSERT(s->second->get_owner_shard() == this_shard_id()); // abort can be called only locally\n        s->second->get()->abort();\n        _streams.erase(s);\n    }\n}\n\nvoid client::deregister_this_stream() {\n    if (_parent) {\n        _parent->_streams.erase(_id);\n    }\n}\n\n// This is the enlightened copy of the connection::send() method. Its intention is to\n// keep a dummy entry in front of the queue while connect+negotiate is happenning so\n// that all subsequent entries could abort on timeout or explicit cancellation.\nvoid client::enqueue_zero_frame() {\n    if (_error) {\n        return;\n    }\n\n    auto p = std::make_unique<outgoing_entry>(snd_buf(0));\n    auto& d = *p;\n    _outgoing_queue.push_back(d);\n\n    // Make it in the background. Even if the client is stopped it will pick\n    // up all the entries hanging around\n    (void)std::exchange(_outgoing_queue_ready, d.done.get_future()).then_wrapped([p = std::move(p)] (auto f) mutable {\n        if (f.failed()) {\n            f.ignore_ready_future();\n        } else {\n            p->done.set_value();\n        }\n    });\n}\n\nstruct client::metrics::domain {\n    metrics::domain_list_t list;\n    stats dead;\n    seastar::metrics::metric_groups metric_groups;\n\n    static thread_local std::unordered_map<sstring, domain> all;\n    static domain& find_or_create(sstring name);\n\n    stats::counter_type count_all(stats::counter_type stats::* field) noexcept {\n        stats::counter_type res = dead.*field;\n        for (const auto& m : list) {\n            res += m._c._stats.*field;\n        }\n        return res;\n    }\n\n    size_t count_all_fn(size_t (client::*fn)(void) const) noexcept {\n        size_t res = 0;\n        for (const auto& m : list) {\n            res += (m._c.*fn)();\n        }\n        return res;\n    }\n\n    domain(sstring name)\n    {\n        namespace sm = seastar::metrics;\n        auto domain_l = sm::label(\"domain\")(name);\n\n        metric_groups.add_group(\"rpc_client\", {\n            sm::make_gauge(\"count\", [this] { return list.size(); },\n            sm::description(\"Total number of clients\"), { domain_l }),\n            sm::make_counter(\"sent_messages\", std::bind(&domain::count_all, this, &stats::sent_messages),\n            sm::description(\"Total number of messages sent\"), { domain_l }),\n            sm::make_counter(\"replied\", std::bind(&domain::count_all, this, &stats::replied),\n            sm::description(\"Total number of responses received\"), { domain_l }),\n            sm::make_counter(\"exception_received\", std::bind(&domain::count_all, this, &stats::exception_received),\n            sm::description(\"Total number of exceptional responses received\"), { domain_l }).set_skip_when_empty(),\n            sm::make_counter(\"timeout\", std::bind(&domain::count_all, this, &stats::timeout),\n            sm::description(\"Total number of timeout responses\"), { domain_l }).set_skip_when_empty(),\n            sm::make_counter(\"delay_samples\", std::bind(&domain::count_all, this, &stats::delay_samples),\n            sm::description(\"Total number of delay samples\"), { domain_l }),\n            sm::make_counter(\"delay_total\", [this] () -> double {\n                std::chrono::duration<double> res(0);\n                for (const auto& m : list) {\n                    res += m._c._stats.delay_total;\n                }\n                return res.count();\n            }, sm::description(\"Total delay in seconds\"), { domain_l }),\n            sm::make_gauge(\"pending\", std::bind(&domain::count_all_fn, this, &client::outgoing_queue_length),\n            sm::description(\"Number of queued outbound messages\"), { domain_l }),\n            sm::make_gauge(\"wait_reply\", std::bind(&domain::count_all_fn, this, &client::incoming_queue_length),\n            sm::description(\"Number of replies waiting for\"), { domain_l }),\n        });\n    }\n};\n\nthread_local std::unordered_map<sstring, client::metrics::domain> client::metrics::domain::all;\n\nclient::metrics::domain& client::metrics::domain::find_or_create(sstring name) {\n    auto i = all.try_emplace(name, name);\n    return i.first->second;\n}\n\nclient::metrics::metrics(const client& c)\n        : _c(c)\n        , _domain(domain::find_or_create(_c._options.metrics_domain))\n{\n    _domain.list.push_back(*this);\n}\n\nclient::metrics::~metrics() {\n    _domain.dead.replied += _c._stats.replied;\n    _domain.dead.exception_received += _c._stats.exception_received;\n    _domain.dead.sent_messages += _c._stats.sent_messages;\n    _domain.dead.timeout += _c._stats.timeout;\n    _domain.dead.delay_samples += _c._stats.delay_samples;\n    _domain.dead.delay_total += _c._stats.delay_total;\n}\n\nclient::client(const logger& l, void* s, client_options ops, socket socket, const socket_address& addr, const socket_address& local)\n        : rpc::connection(l, s), _socket(std::move(socket)), _server_addr(addr), _local_addr(local), _options(ops), _metrics(*this)\n{\n    _socket.set_reuseaddr(ops.reuseaddr);\n    // Run client in the background.\n    // Communicate result via _stopped.\n    // The caller has to call client::stop() to synchronize.\n    (void)loop(ops, addr, local);\n    enqueue_zero_frame();\n}\n\nfuture<> client::loop(client_options ops, const socket_address& addr, const socket_address& local) {\n    std::exception_ptr ep;\n    try {\n        connected_socket fd = co_await _socket.connect(addr, local);\n        fd.set_nodelay(ops.tcp_nodelay);\n        if (ops.keepalive) {\n            fd.set_keepalive(true);\n            fd.set_keepalive_parameters(ops.keepalive.value());\n        }\n        set_socket(std::move(fd));\n\n        feature_map features;\n        if (_options.compressor_factory) {\n            features[protocol_features::COMPRESS] = _options.compressor_factory->supported();\n        }\n        if (_options.send_timeout_data) {\n            features[protocol_features::TIMEOUT] = \"\";\n        }\n        if (_options.send_handler_duration) {\n            features[protocol_features::HANDLER_DURATION] = \"\";\n        }\n        if (_options.stream_parent) {\n            features[protocol_features::STREAM_PARENT] = serialize_connection_id(_options.stream_parent);\n        }\n        if (!_options.isolation_cookie.empty()) {\n            features[protocol_features::ISOLATION] = _options.isolation_cookie;\n        }\n\n        co_await negotiate_protocol(std::move(features));\n\n        _propagate_timeout = !is_stream();\n        set_negotiated();\n        while (!_connected->read_buf.eof() && !_error) {\n            if (is_stream()) {\n                co_await handle_stream_frame();\n                continue;\n            }\n            auto&& [msg_id, ht, data] = co_await read_response_frame_compressed(_connected->read_buf);\n            auto it = _outstanding.find(std::abs(msg_id));\n            if (!data) {\n                _error = true;\n            } else if (it != _outstanding.end()) {\n                auto handler = std::move(it->second);\n                _outstanding.erase(it);\n                (*handler)(*this, msg_id, std::move(data.value()));\n                if (ht) {\n                    _stats.delay_samples++;\n                    _stats.delay_total += (rpc_clock_type::now() - handler->start) - std::chrono::microseconds(*ht);\n                }\n            } else if (msg_id < 0) {\n                try {\n                    std::rethrow_exception(unmarshal_exception(data.value()));\n                } catch(const unknown_verb_error& ex) {\n                    // if this is unknown verb exception with unknown id ignore it\n                    // can happen if unknown verb was used by no_wait client\n                    get_logger()(peer_address(), format(\"unknown verb exception {:d} ignored\", ex.type));\n                } catch(...) {\n                    // We've got error response but handler is no longer waiting, could be timed out.\n                    log_exception(*this, log_level::info, \"ignoring error response\", std::current_exception());\n                }\n            } else {\n                // we get a reply for a message id not in _outstanding\n                // this can happened if the message id is timed out already\n                get_logger()(peer_address(), log_level::debug, \"got a reply for an expired message id\");\n            }\n        }\n    } catch (...) {\n        ep = std::current_exception();\n        if (_connected) {\n            if (is_stream()) {\n                log_exception(*this, log_level::error, \"client stream connection dropped\", ep);\n            } else {\n                log_exception(*this, log_level::error, \"client connection dropped\", ep);\n            }\n        } else {\n            if (is_stream()) {\n                log_exception(*this, log_level::debug, \"stream fail to connect\", ep);\n            } else {\n                log_exception(*this, log_level::debug, \"fail to connect\", ep);\n            }\n        }\n    }\n    if (is_stream() && (ep || _error)) {\n        _stream_queue.abort(std::make_exception_ptr(stream_closed()));\n    }\n    _error = true;\n    future<> f = co_await coroutine::as_future(stop_send_loop(ep));\n    f.ignore_ready_future();\n    _outstanding.clear();\n    if (is_stream()) {\n        deregister_this_stream();\n    } else {\n        abort_all_streams();\n    }\n    if (_compressor) {\n        co_await _compressor->close();\n    }\n    _stopped.set_value();\n}\n\nclient::client(const logger& l, void* s, const socket_address& addr, const socket_address& local)\n        : client(l, s, client_options{}, make_socket(), addr, local)\n{}\n\nclient::client(const logger& l, void* s, client_options options, const socket_address& addr, const socket_address& local)\n        : client(l, s, options, make_socket(), addr, local)\n{}\n\nclient::client(const logger& l, void* s, socket socket, const socket_address& addr, const socket_address& local)\n        : client(l, s, client_options{}, std::move(socket), addr, local)\n{}\n\n\nfuture<feature_map>\nserver::connection::negotiate(feature_map requested) {\n    feature_map ret;\n    future<> f = make_ready_future<>();\n    for (auto&& e : requested) {\n        auto id = e.first;\n        switch (id) {\n            // supported features go here\n        case protocol_features::COMPRESS: {\n            if (get_server()._options.compressor_factory) {\n                _compressor = get_server()._options.compressor_factory->negotiate(e.second, true, [this] { return send({}); });\n                if (_compressor) {\n                    ret[protocol_features::COMPRESS] = _compressor->name();\n                }\n            }\n            break;\n        }\n        case protocol_features::TIMEOUT:\n            _timeout_negotiated = true;\n            ret[protocol_features::TIMEOUT] = \"\";\n            break;\n        case protocol_features::HANDLER_DURATION:\n            _handler_duration_negotiated = true;\n            ret[protocol_features::HANDLER_DURATION] = \"\";\n            break;\n        case protocol_features::STREAM_PARENT: {\n            if (!get_server()._options.streaming_domain) {\n                f = f.then([] {\n                    return make_exception_future<>(std::runtime_error(\"streaming is not configured for the server\"));\n                });\n            } else {\n                _parent_id = deserialize_connection_id(e.second);\n                _is_stream = true;\n                // remove stream connection from rpc connection list\n                get_server()._conns.erase(get_connection_id());\n                f = f.then([this, c = shared_from_this()] () mutable {\n                    return smp::submit_to(_parent_id.shard(), [this, c = make_foreign(static_pointer_cast<rpc::connection>(c))] () mutable {\n                        auto sit = _servers.find(*get_server()._options.streaming_domain);\n                        if (sit == _servers.end()) {\n                            throw std::logic_error(format(\"Shard {:d} does not have server with streaming domain {}\", this_shard_id(), *get_server()._options.streaming_domain).c_str());\n                        }\n                        auto s = sit->second;\n                        auto it = s->_conns.find(_parent_id);\n                        if (it == s->_conns.end()) {\n                            throw std::logic_error(format(\"Unknown parent connection {} on shard {:d}\", _parent_id, this_shard_id()).c_str());\n                        }\n                        if (it->second->_error) {\n                            throw std::runtime_error(format(\"Parent connection {} is aborting on shard {:d}\", _parent_id, this_shard_id()).c_str());\n                        }\n                        auto id = c->get_connection_id();\n                        it->second->register_stream(id, make_lw_shared(std::move(c)));\n                    });\n                });\n            }\n            break;\n        }\n        case protocol_features::ISOLATION: {\n            auto&& isolation_cookie = e.second;\n            struct isolation_function_visitor {\n                isolation_function_visitor(const sstring& isolation_cookie)\n                        : _isolation_cookie(isolation_cookie) { }\n                future<isolation_config> operator() (resource_limits::syncronous_isolation_function f) const {\n                    return futurize_invoke(f, _isolation_cookie);\n                }\n                future<isolation_config> operator() (resource_limits::asyncronous_isolation_function f) const {\n                    return f(_isolation_cookie);\n                }\n                private:\n                sstring _isolation_cookie;\n            };\n\n            auto visitor = isolation_function_visitor(isolation_cookie);\n            f = f.then([visitor = std::move(visitor), this] () mutable {\n                return std::visit(visitor, get_server()._limits.isolate_connection).then([this] (isolation_config conf) {\n                    _isolation_config = conf;\n                });\n            });\n            ret.emplace(e);\n            break;\n        }\n        default:\n            // nothing to do\n            ;\n        }\n    }\n    if (get_server()._options.streaming_domain) {\n        ret[protocol_features::CONNECTION_ID] = serialize_connection_id(_id);\n    }\n    return f.then([ret = std::move(ret)] {\n        return ret;\n    });\n}\n\nfuture<>\nserver::connection::negotiate_protocol() {\n    return receive_negotiation_frame(*this, _connected->read_buf).then([this] (feature_map requested_features) {\n        return negotiate(std::move(requested_features)).then([this] (feature_map returned_features) {\n            return send_negotiation_frame(std::move(returned_features));\n        });\n    });\n}\n\nfuture<request_frame::return_type>\nserver::connection::read_request_frame_compressed(input_stream<char>& in) {\n    if (_timeout_negotiated) {\n        return read_frame_compressed<request_frame_with_timeout>(_info.addr, _compressor, in);\n    } else {\n        return read_frame_compressed<request_frame>(_info.addr, _compressor, in);\n    }\n}\n\nfuture<>\nserver::connection::respond(int64_t msg_id, snd_buf&& data, std::optional<rpc_clock_type::time_point> timeout, std::optional<rpc_clock_type::duration> handler_duration) {\n    if (_handler_duration_negotiated) {\n        response_frame_with_handler_time::encode_header(msg_id, handler_duration, data);\n    } else {\n        data.front().trim_front(sizeof(uint32_t));\n        data.size -= sizeof(uint32_t);\n        response_frame::encode_header(msg_id, data);\n    }\n    return send(std::move(data), timeout);\n}\n\nfuture<> server::connection::send_unknown_verb_reply(std::optional<rpc_clock_type::time_point> timeout, int64_t msg_id, uint64_t type) {\n    return wait_for_resources(28, timeout).then([this, timeout, msg_id, type] (auto permit) {\n        // send unknown_verb exception back\n        constexpr size_t unknown_verb_message_size = response_frame_headroom + 2 * sizeof(uint32_t) + sizeof(uint64_t);\n        snd_buf data(unknown_verb_message_size);\n        static_assert(snd_buf::chunk_size >= unknown_verb_message_size, \"send buffer chunk size is too small\");\n        auto p = data.front().get_write() + response_frame_headroom;\n        write_le<uint32_t>(p, uint32_t(exception_type::UNKNOWN_VERB));\n        write_le<uint32_t>(p + 4, uint32_t(8));\n        write_le<uint64_t>(p + 8, type);\n        try {\n            // Send asynchronously.\n            // This is safe since connection::stop() will wait for background work.\n            (void)with_gate(get_server()._reply_gate, [this, timeout, msg_id, data = std::move(data), permit = std::move(permit)] () mutable {\n                // workaround for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83268\n                auto c = shared_from_this();\n                return respond(-msg_id, std::move(data), timeout, std::nullopt).then([c = std::move(c), permit = std::move(permit)] {});\n            });\n        } catch(gate_closed_exception&) {/* ignore */}\n    });\n}\n\nfuture<> server::connection::process() {\n    // hold onto connection pointer until the while loop exits\n    auto conn_ptr = shared_from_this();\n    std::exception_ptr ep;\n    try {\n        co_await negotiate_protocol();\n        auto sg = _isolation_config ? _isolation_config->sched_group : current_scheduling_group();\n        co_await coroutine::switch_to(sg);\n        set_negotiated();\n        while (!_connected->read_buf.eof() && !_error) {\n            if (is_stream()) {\n                co_await handle_stream_frame();\n                continue;\n            }\n            auto [expire, type, msg_id, data] = co_await read_request_frame_compressed(_connected->read_buf);\n            if (!data) {\n                _error = true;\n                continue;\n            } else {\n                std::optional<rpc_clock_type::time_point> timeout;\n                if (expire && *expire) {\n                    timeout = relative_timeout_to_absolute(std::chrono::milliseconds(*expire));\n                }\n                auto h = get_server()._proto.get_handler(type);\n                if (!h) {\n                    co_await send_unknown_verb_reply(timeout, msg_id, type);\n                    continue;\n                }\n\n                // If the new method of per-connection scheduling group was used, honor it.\n                // Otherwise, use the old per-handler scheduling group.\n                auto sg = _isolation_config ? _isolation_config->sched_group : h->handler.sg;\n                if (sg == current_scheduling_group()) {\n                    co_await h->handler.func(shared_from_this(), timeout, msg_id, std::move(data.value()), std::move(h->holder));\n                    continue;\n                }\n                co_await with_scheduling_group(sg, [this, timeout, msg_id, &h = h->handler, data = std::move(data.value()), guard = std::move(h->holder)] () mutable {\n                    return h.func(shared_from_this(), timeout, msg_id, std::move(data), std::move(guard));\n                });\n            }\n        }\n    } catch (...) {\n        ep = std::current_exception();\n        log_exception(*this, log_level::error,\n            format(\"server{} connection dropped\", is_stream() ? \" stream\" : \"\").c_str(), ep);\n    }\n    _connected->fd.shutdown_input();\n    if (is_stream() && (ep || _error)) {\n        _stream_queue.abort(std::make_exception_ptr(stream_closed()));\n    }\n    _error = true;\n    future<> f = co_await coroutine::as_future(stop_send_loop(ep));\n    f.ignore_ready_future();\n    get_server()._conns.erase(get_connection_id());\n    if (is_stream()) {\n        co_await deregister_this_stream();\n    } else {\n        co_await abort_all_streams();\n    }\n    if (_compressor) {\n        co_await _compressor->close();\n    }\n    _stopped.set_value();\n}\n\nserver::connection::connection(server& s, connected_socket&& fd, socket_address&& addr, const logger& l, void* serializer, connection_id id)\n        : rpc::connection(std::move(fd), l, serializer, id)\n        , _info{.addr{std::move(addr)}, .server{s}, .conn_id{id}} {\n}\n\nfuture<> server::connection::deregister_this_stream() {\n    if (!get_server()._options.streaming_domain) {\n        return make_ready_future<>();\n    }\n    return smp::submit_to(_parent_id.shard(), [this] () mutable {\n        auto sit = server::_servers.find(*get_server()._options.streaming_domain);\n        if (sit != server::_servers.end()) {\n            auto s = sit->second;\n            auto it = s->_conns.find(_parent_id);\n            if (it != s->_conns.end()) {\n                it->second->_streams.erase(get_connection_id());\n            }\n        }\n    });\n}\n\nfuture<> server::connection::abort_all_streams() {\n    return parallel_for_each(_streams | boost::adaptors::map_values, [] (xshard_connection_ptr s) {\n        return smp::submit_to(s->get_owner_shard(), [s] {\n            s->get()->abort();\n        });\n    }).then([this] {\n        _streams.clear();\n    });\n}\n\nthread_local std::unordered_map<streaming_domain_type, server*> server::_servers;\n\nserver::server(protocol_base* proto, const socket_address& addr, resource_limits limits)\n        : server(proto, seastar::listen(addr, listen_options{true}), limits, server_options{})\n{}\n\nserver::server(protocol_base* proto, server_options opts, const socket_address& addr, resource_limits limits)\n        : server(proto, seastar::listen(addr, listen_options{true, opts.load_balancing_algorithm}), limits, opts)\n{}\n\nserver::server(protocol_base* proto, server_socket ss, resource_limits limits, server_options opts)\n        : _proto(*proto), _ss(std::move(ss)), _limits(limits), _resources_available(limits.max_memory), _options(opts)\n{\n    if (_options.streaming_domain) {\n        if (_servers.find(*_options.streaming_domain) != _servers.end()) {\n            throw std::runtime_error(format(\"An RPC server with the streaming domain {} is already exist\", *_options.streaming_domain));\n        }\n        _servers[*_options.streaming_domain] = this;\n    }\n    accept();\n}\n\nserver::server(protocol_base* proto, server_options opts, server_socket ss, resource_limits limits)\n        : server(proto, std::move(ss), limits, opts)\n{}\n\nvoid server::accept() {\n    // Run asynchronously in background.\n    // Communicate result via __ss_stopped.\n    // The caller has to call server::stop() to synchronize.\n    (void)keep_doing([this] () mutable {\n        return _ss.accept().then([this] (accept_result ar) mutable {\n            if (_options.filter_connection && !_options.filter_connection(ar.remote_address)) {\n                return;\n            }\n            auto fd = std::move(ar.connection);\n            auto addr = std::move(ar.remote_address);\n            fd.set_nodelay(_options.tcp_nodelay);\n            connection_id id = _options.streaming_domain\n                    ? connection_id::make_id(_next_client_id++, uint16_t(this_shard_id()))\n                    : connection_id::make_invalid_id(_next_client_id++);\n            auto conn = _proto.make_server_connection(*this, std::move(fd), std::move(addr), id);\n            auto r = _conns.emplace(id, conn);\n            SEASTAR_ASSERT(r.second);\n            // Process asynchronously in background.\n            (void)conn->process();\n        });\n    }).then_wrapped([this] (future<>&& f){\n        try {\n            f.get();\n            SEASTAR_ASSERT(false);\n        } catch (...) {\n            _ss_stopped.set_value();\n        }\n    });\n}\n\nfuture<> server::shutdown() {\n    if (_shutdown) {\n        return make_ready_future<>();\n    }\n\n    _ss.abort_accept();\n    _resources_available.broken();\n    if (_options.streaming_domain) {\n        _servers.erase(*_options.streaming_domain);\n    }\n    return _ss_stopped.get_future().then([this] {\n        return parallel_for_each(_conns | boost::adaptors::map_values, [] (shared_ptr<connection> conn) {\n            return conn->stop();\n        });\n    }).finally([this] {\n        _shutdown = true;\n    });\n}\n\nfuture<> server::stop() {\n    return when_all(\n        shutdown(),\n        _reply_gate.close()\n    ).discard_result();\n}\n\nvoid server::abort_connection(connection_id id) {\n    auto it = _conns.find(id);\n    if (it == _conns.end()) {\n        return;\n    }\n    try {\n        it->second->abort();\n    } catch (...) {\n        log_exception(*it->second, log_level::error,\n            \"fail to shutdown connection on user request\", std::current_exception());\n        }\n    }\n\n    std::ostream& operator<<(std::ostream& os, const connection_id& id) {\n        fmt::print(os, \"{:x}\", id.id());\n        return os;\n    }\n\n    std::ostream& operator<<(std::ostream& os, const streaming_domain_type& domain) {\n        fmt::print(os, \"{:d}\", domain._id);\n        return os;\n    }\n\n    isolation_config default_isolate_connection(sstring isolation_cookie) {\n        return isolation_config{};\n    }\n\n    multi_algo_compressor_factory::multi_algo_compressor_factory(std::vector<const rpc::compressor::factory*> factories)\n            : _factories(std::move(factories)) {\n        _features =  boost::algorithm::join(_factories | boost::adaptors::transformed(std::mem_fn(&rpc::compressor::factory::supported)), sstring(\",\"));\n    }\n\n    std::unique_ptr<compressor>\n    multi_algo_compressor_factory::negotiate(sstring feature, bool is_server, std::function<future<>()> send_empty_frame) const {\n        std::vector<sstring> names;\n        boost::split(names, feature, boost::is_any_of(\",\"));\n        std::unique_ptr<compressor> c;\n        if (is_server) {\n            for (auto&& n : names) {\n                for (auto&& f : _factories) {\n                    if ((c = f->negotiate(n, is_server, send_empty_frame))) {\n                        return c;\n                    }\n                }\n            }\n        } else {\n            for (auto&& f : _factories) {\n                for (auto&& n : names) {\n                    if ((c = f->negotiate(n, is_server, send_empty_frame))) {\n                        return c;\n                    }\n                }\n            }\n        }\n        return nullptr;\n    }\n\n}\n\n}\n"
  },
  {
    "path": "src/seastar.cppm",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2019 ScyllaDB\n */\n\n// we could split the subsystems into multiple module partitions for a cleaner\n// structure of the module, but the dependencies between Seastar subsystems\n// form a cylic graph, if we split the sources at the boundary of the\n// subdirectory the header files are located. for instance:\n//   core/future => util/backtrace => core/sstring.\n//\n// one way to address this circular dependency problem by breaking some\n// subsystems into smaller pieces at the expense of creasomg the complexity\n// level of the module structure. as each partition has\n// - its own source file\n// - an entry in CMakeLists.txt\n// - one or more cross partition import / export clause when it is used / exposed\n//\n// a simpler alternative is to put all headers into a the same purview of\n// the \"seastar\" module. but this slows down the build speed of Seastar itself,\n// as each time when we modify any of the header file, the whole module is\n// recompiled. but this should fine at this moment, as the majority Seastar\n// developers are not supposed to build Seastar as a C++ module, which is, in\n// general, built for a single time to be consumed by Seastar applications.\n\nmodule;\n\n#include <seastar/util/std-compat.hh>\n#include <seastar/core/abortable_fifo.hh>\n#include <seastar/core/abort_on_ebadf.hh>\n#include <seastar/core/abort_on_expiry.hh>\n#include <seastar/core/abort_source.hh>\n#include <seastar/core/alien.hh>\n#include <seastar/core/align.hh>\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/bitops.hh>\n#include <seastar/core/bitset-iter.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/cacheline.hh>\n#include <seastar/core/checked_ptr.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/circular_buffer_fixed_capacity.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/deleter.hh>\n#include <seastar/core/disk_params.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/enum.hh>\n#include <seastar/core/exception_hacks.hh>\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/expiring_fifo.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/file-types.hh>\n#include <seastar/core/fsnotify.hh>\n#include <seastar/core/fsqual.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/idle_cpu_handler.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/iostream-impl.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/io_priority_class.hh>\n#include <seastar/core/layered_file.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/make_task.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/metrics_types.hh>\n#include <seastar/core/pipe.hh>\n#include <seastar/core/polymorphic_temporary_buffer.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/preempt.hh>\n#include <seastar/core/prefetch.hh>\n#include <seastar/core/print.hh>\n// #include <seastar/core/prometheus.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/ragel.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/reactor_config.hh>\n#include <seastar/core/relabel_config.hh>\n#include <seastar/core/report_exception.hh>\n#include <seastar/core/resource.hh>\n#include <seastar/core/rwlock.hh>\n#include <seastar/core/scattered_message.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/scheduling_specific.hh>\n#include <seastar/core/scollectd.hh>\n#include <seastar/core/scollectd_api.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/shared_future.hh>\n#include <seastar/core/shared_mutex.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/shared_ptr_debug_helper.hh>\n#include <seastar/core/shared_ptr_incomplete.hh>\n#include <seastar/core/signal.hh>\n#include <seastar/core/simple-stream.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/smp_options.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/core/stall_sampler.hh>\n#include <seastar/core/task.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/transfer.hh>\n#include <seastar/core/unaligned.hh>\n#include <seastar/core/units.hh>\n#include <seastar/core/vector-data-sink.hh>\n#include <seastar/core/weak_ptr.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/when_any.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/with_timeout.hh>\n\n#include <seastar/coroutine/all.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/exception.hh>\n#include <seastar/coroutine/generator.hh>\n#include <seastar/coroutine/maybe_yield.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n#include <seastar/coroutine/switch_to.hh>\n\n#include <seastar/util/alloc_failure_injector.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/conversions.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/file.hh>\n#include <seastar/util/log-cli.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/optimized_optional.hh>\n#include <seastar/util/print_safe.hh>\n#include <seastar/util/process.hh>\n#include <seastar/util/read_first_line.hh>\n#include <seastar/util/short_streams.hh>\n#include <seastar/util/memory-data-source.hh>\n#include <seastar/util/memory-data-sink.hh>\n\n#include <seastar/net/arp.hh>\n#include <seastar/net/packet.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/ip_checksum.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/net/ipv4_address.hh>\n#include <seastar/net/native-stack.hh>\n#include <seastar/net/posix-stack.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/net/tcp.hh>\n#include <seastar/net/udp.hh>\n#include <seastar/net/tls.hh>\n\n#include <seastar/http/common.hh>\n#include <seastar/http/client.hh>\n#include <seastar/http/exception.hh>\n#include <seastar/http/file_handler.hh>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/json_path.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/http/response_parser.hh>\n#include <seastar/http/request.hh>\n#include <seastar/http/routes.hh>\n#include <seastar/http/transformers.hh>\n\n#include <seastar/json/formatter.hh>\n#include <seastar/json/json_elements.hh>\n\nexport module seastar;\n\nexport namespace seastar {\n\n// Core types and utilities\nusing seastar::app_template;\nusing seastar::logger;\nusing seastar::log_level;\nusing seastar::future;\nusing seastar::promise;\nusing seastar::shared_future;\nusing seastar::make_ready_future;\nusing seastar::make_exception_future;\nusing seastar::thread;\nusing seastar::thread_attributes;\nusing seastar::reactor;\nusing seastar::engine;\nusing seastar::smp;\nusing seastar::shard_id;\nusing seastar::this_shard_id;\n\n// Memory management\nusing seastar::temporary_buffer;\nusing seastar::deleter;\nusing seastar::shared_ptr;\nusing seastar::lw_shared_ptr;\nusing seastar::make_shared;\nusing seastar::make_lw_shared;\nusing seastar::weak_ptr;\nusing seastar::enable_shared_from_this;\nusing seastar::enable_lw_shared_from_this;\nusing seastar::foreign_ptr;\nusing seastar::make_foreign;\n\n// Synchronization primitives\nusing seastar::semaphore;\nusing seastar::broken_semaphore;\nusing seastar::semaphore_timed_out;\nusing seastar::gate;\nusing seastar::gate_closed_exception;\nusing seastar::condition_variable;\nusing seastar::broken_condition_variable;\nusing seastar::rwlock;\nusing seastar::shared_mutex;\nusing seastar::abort_source;\nusing seastar::abort_requested_exception;\n\n// Containers\nusing seastar::circular_buffer;\nusing seastar::circular_buffer_fixed_capacity;\nusing seastar::chunked_fifo;\nusing seastar::queue;\nusing seastar::sstring;\n\n// Scheduling\nusing seastar::scheduling_group;\nusing seastar::task;\n\n// Timers and clocks\nusing seastar::timer;\nusing seastar::lowres_clock;\nusing seastar::lowres_system_clock;\nusing seastar::manual_clock;\nusing seastar::steady_clock_type;\n\n// Sharding\nusing seastar::sharded;\n\n// I/O\nusing seastar::file;\nusing seastar::file_open_options;\nusing seastar::open_flags;\nusing seastar::directory_entry_type;\nusing seastar::file_handle;\nusing seastar::input_stream;\nusing seastar::output_stream;\nusing seastar::data_sink;\nusing seastar::data_source;\n\n// Networking\nusing seastar::socket;\nusing seastar::server_socket;\nusing seastar::connected_socket;\nusing seastar::socket_address;\nusing seastar::ipv4_addr;\nusing seastar::ipv6_addr;\nusing seastar::transport;\n\n// Metrics\nusing seastar::metrics::metric_groups;\nusing seastar::metrics::metric_group_definition;\nusing seastar::metrics::label;\nusing seastar::metrics::label_instance;\n\n// Utility functions\nusing seastar::do_with;\nusing seastar::do_for_each;\nusing seastar::do_until;\nusing seastar::repeat;\nusing seastar::repeat_until_value;\nusing seastar::parallel_for_each;\nusing seastar::map_reduce;\nusing seastar::when_all;\nusing seastar::when_all_succeed;\nusing seastar::sleep;\nusing seastar::yield;\n\n// Exception types\nusing seastar::nested_exception;\nusing seastar::timed_out_error;\n\n// Resource management\nusing seastar::with_scheduling_group;\nusing seastar::with_timeout;\n\n// Print utilities\nusing seastar::format;\n\n}\n\nexport namespace seastar::net {\n\nusing seastar::net::packet;\nusing seastar::net::fragment;\nusing seastar::net::ethernet_address;\nusing seastar::net::ipv4_address;\nusing seastar::net::ipv6_address;\nusing seastar::net::inet_address;\nusing seastar::net::ipv4_tcp;\nusing seastar::net::ipv4_udp;\nusing seastar::net::udp_channel;\nusing seastar::net::datagram;\nusing seastar::net::get_impl;\n\n}\n\nexport namespace seastar::http {\n\nusing seastar::http::reply;\nusing seastar::http::request;\n\n}\n\nexport namespace seastar::util {\n\nusing seastar::util::basic_memory_data_source;\nusing seastar::util::temporary_buffer_data_source;\nusing seastar::util::basic_memory_data_sink;\n\n}\n\nexport namespace std {\n\nusing std::hash;\n\n// Expose hash specializations for seastar types that have them\ntemplate <>\nstruct hash<seastar::socket_address>;\n\ntemplate <>\nstruct hash<seastar::ipv4_addr>;\n\ntemplate <>\nstruct hash<seastar::net::inet_address>;\n\n}\n"
  },
  {
    "path": "src/testing/entry_point.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/testing/entry_point.hh>\n#include <seastar/testing/seastar_test.hh>\n#include <seastar/testing/test_runner.hh>\n\nnamespace seastar {\n\nnamespace testing {\n\nstatic bool init_unit_test_suite() {\n    auto&& ts = boost::unit_test::framework::master_test_suite();\n    return global_test_runner().start(ts.argc, ts.argv);\n}\n\nstatic void dummy_handler(int) {\n    // This handler should have been replaced.\n    _exit(1);\n}\n\nstatic void install_dummy_handler(int sig) {\n    struct sigaction sa {};\n    sa.sa_handler = dummy_handler;\n    sigaction(sig, &sa, nullptr);\n}\n\nint entry_point(int argc, char** argv) {\n#ifndef SEASTAR_ASAN_ENABLED\n    // Before we call into boost, install some dummy signal\n    // handlers. This seems to be the only way to stop boost from\n    // installing its own handlers, which disables our backtrace\n    // printer. The real handler will be installed when the reactor is\n    // constructed.\n    // If we are using ASAN, it has already installed a signal handler\n    // that does its own stack printing.\n    for (int sig : {SIGSEGV, SIGABRT}) {\n        install_dummy_handler(sig);\n    }\n#else\n    (void)install_dummy_handler;\n#endif\n\n    const int boost_exit_code = ::boost::unit_test::unit_test_main(&init_unit_test_suite, argc, argv);\n    const int seastar_exit_code = seastar::testing::global_test_runner().finalize();\n    if (boost_exit_code) {\n        return boost_exit_code;\n    }\n    return seastar_exit_code;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/testing/random.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#include <random>\n#include <seastar/testing/test_runner.hh>\n\nnamespace seastar {\n\nnamespace testing {\n\nthread_local std::default_random_engine local_random_engine;\n\n} // namespace testing\n\n} // namespace seastar\n"
  },
  {
    "path": "src/testing/seastar_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <thread>\n#include <iostream>\n\n#include <boost/test/execution_monitor.hpp>\n#include <boost/exception/diagnostic_information.hpp>\n#include <boost/core/type_name.hpp>\n\n#include <seastar/testing/entry_point.hh>\n#include <seastar/testing/seastar_test.hh>\n#include <seastar/testing/test_fixture.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/on_internal_error.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/testing/on_internal_error.hh>\n\nnamespace seastar {\n\nnamespace testing {\n\n// #3165 - build a message for a possibly nested exception chain.\nstatic boost::execution_exception::location\nadd_exception_message(const std::exception* ep, bool rec, char *out, char *end) {\n    auto pfx = rec ? \"; Caused by \" : \"\";\n\n    if (ep) {\n        out = fmt::format_to_n(out, end - out, \"{}{}: {}\", pfx, seastar::pretty_type_name(typeid(*ep)), ep->what()).out;\n    } else {\n        auto tp = abi::__cxa_current_exception_type();\n        out = fmt::format_to_n(out, out - end, \"{}{}\", pfx, tp ? seastar::pretty_type_name(*tp) : \"<unknown>\").out;\n    }\n\n    boost::execution_exception::location loc;\n    if (auto* be = dynamic_cast<const boost::exception*>(ep)) {\n        auto sloc = boost::exception_detail::get_exception_throw_location(*be);\n        if (rec) {\n            out = fmt::format_to_n(out, end - out, \"({}:{}:{})\", sloc.file_name(), sloc.function_name(), sloc.line()).out;\n        }\n        loc = boost::execution_exception::location(sloc.file_name(), sloc.line(), sloc.function_name());\n    }\n\n    *out = 0; // see initial invoke below. always valid\n\n    if (ep) {\n        try {\n            std::rethrow_if_nested(*ep);\n        } catch (std::exception& e) {\n            add_exception_message(&e, true, out, end);\n        } catch (...) {\n            add_exception_message(nullptr, true, out, end);\n        }\n    }\n\n    return loc;\n}\n\n[[noreturn]]\nstatic void repackage_exception_and_rethrow(const std::exception* ep) {\n    // Note: using a static buffer for formatting, same as boost::test code,\n    // so we make it less prone to fail in failure handling for OOM\n    // situations etc.\n    static const int REPORT_ERROR_BUFFER_SIZE = 4096;\n    static char buf[REPORT_ERROR_BUFFER_SIZE];\n\n    auto loc = add_exception_message(ep, false, buf, buf + sizeof(buf) - 1);\n    boost:: BOOST_TEST_I_THROW(boost::execution_exception(boost::execution_exception::cpp_exception_error, buf, loc));\n}\n\nvoid seastar_test::run() {\n    // HACK: please see https://github.com/cloudius-systems/seastar/issues/10\n    BOOST_REQUIRE(true);\n\n    // HACK: please see https://github.com/cloudius-systems/seastar/issues/10\n    boost::program_options::variables_map()[\"dummy\"];\n\n    set_abort_on_internal_error(true);\n\n    global_test_runner().run_sync([this] {\n        // #3165 - do exception catch here already, and package\n        // the info into an execution_exception, potentially including\n        // nestedness etc.\n        try {\n            return run_test_case();\n        } catch (std::exception& e) {\n            repackage_exception_and_rethrow(&e);\n        } catch (...) {\n            repackage_exception_and_rethrow(nullptr);\n        }\n    });\n}\n\nseastar_test::seastar_test(const char* test_name, const char* test_file, int test_line)\n    : seastar_test(test_name, test_file, test_line, boost::unit_test::decorator::collector_t::instance()) {}\n\nseastar_test::seastar_test(const char* test_name, const char* test_file, int test_line,\n                           boost::unit_test::decorator::collector_t& decorators)\n    : _test_file{test_file} {\n    auto test = boost::unit_test::make_test_case([this] { run(); }, test_name, test_file, test_line);\n    decorators.store_in(*test);\n    decorators.reset();\n    boost::unit_test::framework::current_auto_test_suite().add(test);\n}\n\nconst std::string& seastar_test::get_name() {\n    const auto& current_test = boost::unit_test::framework::current_test_unit();\n    return current_test.p_name.get();\n}\n\nnamespace exception_predicate {\n\nstd::function<bool(const std::exception&)> message_equals(std::string_view expected_message) {\n    return [expected_message] (const std::exception& e) {\n        std::string error = e.what();\n        if (error == expected_message) {\n            return true;\n        } else {\n            std::cerr << \"Expected \\\"\" << expected_message << \"\\\" but got \\\"\" << error << '\"' << std::endl;\n            return false;\n        }\n    };\n}\n\nstd::function<bool(const std::exception&)> message_contains(std::string_view expected_message) {\n    return [expected_message] (const std::exception& e) {\n        std::string error = e.what();\n        if (error.find(expected_message.data()) != std::string::npos) {\n            return true;\n        } else {\n            std::cerr << \"Expected \\\"\" << expected_message << \"\\\" but got \\\"\" << error << '\"' << std::endl;\n            return false;\n        }\n    };\n}\n\n} // exception_predicate\n\nscoped_no_abort_on_internal_error::scoped_no_abort_on_internal_error() noexcept\n    : _prev(set_abort_on_internal_error(false))\n{\n}\n\nscoped_no_abort_on_internal_error::~scoped_no_abort_on_internal_error() {\n    set_abort_on_internal_error(_prev);\n}\n\nvoid detail::warn_teardown_exception(const sstring& name, std::exception_ptr e) {\n    std::cerr << \"Warning! Exception in fixture \" << name << \"::teardown. \" << e << std::endl;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/testing/test_runner.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <iostream>\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nnamespace testing {\n\nstatic test_runner instance;\n\nstruct stop_execution : public std::exception {};\n\ntest_runner::~test_runner() {\n    finalize();\n}\n\nbool\ntest_runner::start(int ac, char** av) {\n    bool expected = false;\n    if (!_started.compare_exchange_strong(expected, true, std::memory_order_acquire)) {\n        return true;\n    }\n\n    // Don't interfere with seastar signal handling\n    sigset_t mask;\n    sigfillset(&mask);\n    for (auto sig : { SIGSEGV }) {\n        sigdelset(&mask, sig);\n    }\n    auto r = ::pthread_sigmask(SIG_BLOCK, &mask, NULL);\n    if (r) {\n        std::cerr << \"Error blocking signals. Aborting.\" << std::endl;\n        abort();\n    }\n\n    _st_args = std::make_unique<start_thread_args>(ac, av);\n    return true;\n}\n\nbool test_runner::start_thread(int ac, char** av) {\n    auto init_outcome = std::make_shared<exchanger<bool>>();\n\n    namespace bpo = boost::program_options;\n    _thread = std::make_unique<posix_thread>([this, ac, av, init_outcome]() mutable {\n        app_template app;\n        app.add_options()\n            (\"random-seed\", bpo::value<unsigned>(), \"Random number generator seed\")\n            (\"fail-on-abandoned-failed-futures\", bpo::value<bool>()->default_value(true), \"Fail the test if there are any abandoned failed futures\");\n        // We guarantee that only one thread is running.\n        // We only read this after that one thread is joined, so this is safe.\n        _exit_code = app.run(ac, av, [this, &app, init_outcome = init_outcome.get()] {\n            init_outcome->give(true);\n            auto init = [&app] {\n                auto conf_seed = app.configuration()[\"random-seed\"];\n                auto seed = conf_seed.empty() ? std::random_device()():  conf_seed.as<unsigned>();\n                std::cout << \"random-seed=\" << seed << std::endl;\n                return smp::invoke_on_all([seed] {\n                    auto local_seed = seed + this_shard_id();\n                    local_random_engine.seed(local_seed);\n                });\n            };\n\n            return init().then([this] {\n              return do_until([this] { return _done; }, [this] {\n                // this will block the reactor briefly, but we don't care\n                try {\n                    auto func = _task.take();\n                    return func();\n                } catch (const stop_execution&) {\n                    _done = true;\n                    return make_ready_future<>();\n                }\n              }).or_terminate();\n            }).then([&app] {\n                if (engine().abandoned_failed_futures()) {\n                    std::cerr << \"*** \" << engine().abandoned_failed_futures() << \" abandoned failed future(s) detected\" << std::endl;\n                    if (app.configuration()[\"fail-on-abandoned-failed-futures\"].as<bool>()) {\n                        std::cerr << \"Failing the test because fail was requested by --fail-on-abandoned-failed-futures\" << std::endl;\n                        return 3;\n                    }\n                }\n                return 0;\n            });\n        });\n        init_outcome->give(false);\n    });\n\n    return init_outcome->take();\n}\n\nvoid\ntest_runner::run_sync(std::function<future<>()> task) {\n    if (_st_args) {\n        start_thread_args sa = *_st_args;\n        _st_args.reset();\n        if (!start_thread(sa.ac, sa.av)) {\n            // something bad happened when starting the reactor or app, and\n            // the _thread has exited before taking any task. but we need to\n            // move on. let's report this bad news with exit code\n            _done = true;\n        }\n    }\n    if (_done) {\n        // we failed to start the worker reactor, so we cannot send the task to\n        // it.\n        return;\n    }\n\n    exchanger<std::exception_ptr> e;\n    _task.give([task = std::move(task), &e] {\n        SEASTAR_ASSERT(engine_is_ready());\n        try {\n            return task().then_wrapped([&e](auto&& f) {\n                try {\n                    f.get();\n                    e.give({});\n                } catch (...) {\n                    e.give(std::current_exception());\n                }\n            });\n        } catch (...) {\n            e.give(std::current_exception());\n            return make_ready_future<>();\n        }\n    });\n    auto maybe_exception = e.take();\n    if (maybe_exception) {\n        std::rethrow_exception(maybe_exception);\n    }\n}\n\nint test_runner::finalize() {\n    if (_thread) {\n        _task.interrupt(stop_execution());\n        _thread->join();\n        _thread = nullptr;\n    }\n    return _exit_code;\n}\n\ntest_runner& global_test_runner() {\n    return instance;\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/util/alloc_failure_injector.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 ScyllaDB\n */\n\n#include <seastar/util/alloc_failure_injector.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/defer.hh>\n\nnamespace seastar {\nnamespace memory {\n\nstatic logger log(\"failure_injector\");\n\nthread_local alloc_failure_injector the_alloc_failure_injector;\n\nvoid alloc_failure_injector::fail() {\n    _failed = true;\n    cancel();\n    if (log.is_enabled(log_level::trace)) {\n        log.trace(\"Failing at {}\", current_backtrace());\n    }\n    _on_alloc_failure();\n}\n\nvoid alloc_failure_injector::run_with_callback(noncopyable_function<void()> callback, noncopyable_function<void()> to_run) {\n    auto restore = defer([this, prev = std::exchange(_on_alloc_failure, std::move(callback))] () mutable noexcept {\n        _on_alloc_failure = std::move(prev);\n    });\n    to_run();\n}\n\nvoid with_allocation_failures(noncopyable_function<void()> func) {\n    auto& injector = memory::local_failure_injector();\n    uint64_t i = 0;\n    do {\n        try {\n            injector.fail_after(i++);\n            func();\n            injector.cancel();\n        } catch (const std::bad_alloc&) {\n            // expected\n        }\n    } while (injector.failed());\n}\n\n}\n}\n"
  },
  {
    "path": "src/util/backtrace.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 ScyllaDB\n */\n\n#include <link.h>\n#include <sys/types.h>\n#include <unistd.h>\n#include <algorithm>\n#include <cstddef>\n#include <cerrno>\n#include <cstring>\n#include <iostream>\n#include <source_location>\n#include <variant>\n#include <vector>\n#include <boost/container/static_vector.hpp>\n#include <fmt/ostream.h>\n#include <fmt/ranges.h>\n\n#include <seastar/util/backtrace.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/reactor.hh>\n\nnamespace seastar {\n\nstatic int dl_iterate_phdr_callback(struct dl_phdr_info *info, size_t size, void *data)\n{\n    std::size_t total_size{0};\n    for (int i = 0; i < info->dlpi_phnum; i++) {\n        const auto hdr = info->dlpi_phdr[i];\n\n        // Only account loadable segments\n        if (hdr.p_type == PT_LOAD) {\n            total_size += hdr.p_memsz;\n        }\n    }\n\n    reinterpret_cast<std::vector<shared_object>*>(data)->push_back({info->dlpi_name, info->dlpi_addr, info->dlpi_addr + total_size});\n\n    return 0;\n}\n\nstatic std::vector<shared_object> enumerate_shared_objects() {\n    std::vector<shared_object> shared_objects;\n    dl_iterate_phdr(dl_iterate_phdr_callback, &shared_objects);\n\n    return shared_objects;\n}\n\nstatic const std::vector<shared_object> shared_objects{enumerate_shared_objects()};\nstatic const shared_object uknown_shared_object{\"\", 0, std::numeric_limits<uintptr_t>::max()};\n\nbool operator==(const frame& a, const frame& b) noexcept {\n    return a.so == b.so && a.addr == b.addr;\n}\n\nframe decorate(uintptr_t addr) noexcept {\n    // If the shared-objects are not enumerated yet, or the enumeration\n    // failed return the addr as-is with a dummy shared-object.\n    if (shared_objects.empty()) {\n        return {&uknown_shared_object, addr};\n    }\n\n    auto it = std::find_if(shared_objects.begin(), shared_objects.end(), [&] (const shared_object& so) {\n        return addr >= so.begin && addr < so.end;\n    });\n\n    // Unidentified addresses are assumed to originate from the executable.\n    auto& so = it == shared_objects.end() ? shared_objects.front() : *it;\n    return {&so, addr - so.begin};\n}\n\nsimple_backtrace current_backtrace_tasklocal() noexcept {\n    simple_backtrace::vector_type v;\n    backtrace([&] (frame f) {\n        if (v.size() < v.capacity()) {\n            v.emplace_back(std::move(f));\n        }\n    });\n    return simple_backtrace(std::move(v));\n}\n\nsize_t simple_backtrace::calculate_hash() const noexcept {\n    size_t h = 0;\n    for (auto f : _frames) {\n        h = ((h << 5) - h) ^ (f.so->begin + f.addr);\n    }\n    return h;\n}\n\nstd::ostream& operator<<(std::ostream& out, const frame& f) {\n    fmt::print(out, \"{}\", f);\n    return out;\n}\n\nstd::ostream& operator<<(std::ostream& out, const simple_backtrace& b) {\n    fmt::print(out, \"{}\", b);\n    return out;\n}\n\nstd::ostream& operator<<(std::ostream& out, const tasktrace& b) {\n    fmt::print(out, \"{}\", b);\n    return out;\n}\n\nstd::ostream& operator<<(std::ostream& out, const task_entry& e) {\n    fmt::print(out, \"{}\", e);\n    return out;\n}\n\ntasktrace current_tasktrace() noexcept {\n    auto main = current_backtrace_tasklocal();\n\n    tasktrace::vector_type prev;\n    size_t hash = 0;\n    if (local_engine && g_current_context) {\n        task* tsk = nullptr;\n\n        thread_context* thread = thread_impl::get();\n        if (thread) {\n            tsk = thread->waiting_task();\n        } else {\n            tsk = local_engine->current_task();\n        }\n\n        while (tsk && prev.size() < prev.max_size()) {\n            shared_backtrace bt = tsk->get_backtrace();\n            hash *= 31;\n            if (bt) {\n                hash ^= bt->hash();\n                prev.push_back({ bt, tsk->get_resume_point() });\n            } else {\n                const std::type_info& ti = typeid(*tsk);\n                prev.push_back({ task_entry(ti), tsk->get_resume_point() });\n                hash ^= ti.hash_code();\n            }\n            tsk = tsk->waiting_task();\n        }\n    }\n\n    return tasktrace(std::move(main), std::move(prev), hash, current_scheduling_group());\n}\n\nsaved_backtrace current_backtrace() noexcept {\n    return current_tasktrace();\n}\n\ntasktrace::tasktrace(simple_backtrace main, tasktrace::vector_type prev, size_t prev_hash, scheduling_group sg) noexcept\n    : _main(std::move(main))\n    , _prev(std::move(prev))\n    , _sg(sg)\n    , _hash(_main.hash() * 31 ^ prev_hash)\n{ }\n\nbool tasktrace::operator==(const tasktrace& o) const noexcept {\n    return _hash == o._hash && _main == o._main && _prev == o._prev;\n}\n\ntasktrace::~tasktrace() {}\n\n} // namespace seastar\n\nnamespace fmt {\n\nauto formatter<seastar::frame>::format(const seastar::frame& f, format_context& ctx) const\n    -> decltype(ctx.out()) {\n    auto out = ctx.out();\n    if (!f.so->name.empty()) {\n        out = fmt::format_to(out, \"{}+\", f.so->name);\n    }\n    return fmt::format_to(out, \"0x{:x}\", f.addr);\n}\n\nauto formatter<seastar::simple_backtrace>::format(const seastar::simple_backtrace& b, format_context& ctx) const\n    -> decltype(ctx.out()) {\n    return fmt::format_to(ctx.out(), \"{}\", fmt::join(b._frames, \" \"));\n}\n\nauto formatter<seastar::tasktrace>::format(const seastar::tasktrace& b, format_context& ctx) const\n    -> decltype(ctx.out()) {\n    auto out = ctx.out();\n    out = fmt::format_to(out, \"{}\", b._main);\n    for(const auto & [ e, resume_loc ] : b._prev) {\n        out = fmt::format_to(out,  \"\\n   --------\");\n\n        if (resume_loc.file_name()[0] != 0) {\n            out = fmt::format_to(out, \"\\n   {}:{}:{}\", resume_loc.file_name(), resume_loc.line(), resume_loc.column());\n        }\n        out = std::visit(seastar::make_visitor(\n            [&] (const seastar::shared_backtrace& sb) {\n                return fmt::format_to(out,  \"\\n{}\", sb);\n            },\n            [&] (const seastar::task_entry& f) {\n                return fmt::format_to(out,  \"\\n   {}\", f);\n            }), e);\n    }\n    return out;\n}\n\nauto formatter<seastar::task_entry>::format(const seastar::task_entry& e, format_context& ctx) const\n    -> decltype(ctx.out()) {\n    return fmt::format_to(ctx.out(), \"{}\", seastar::pretty_type_name(*e._task_type));\n}\n\n}\n"
  },
  {
    "path": "src/util/conversions.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n\n#include <boost/algorithm/string.hpp>\n#include <boost/lexical_cast.hpp>\n#include <cctype>\n\n#include <seastar/util/conversions.hh>\n#include <seastar/core/print.hh>\n\nnamespace seastar {\n\nstatic constexpr struct {\n    std::string_view suffix;\n    unsigned power;\n} suffixes[] = {\n    {\"k\", 10},\n    {\"K\", 10},\n    {\"M\", 20},\n    {\"G\", 30},\n    {\"T\", 40},\n};\n\nsize_t parse_memory_size(std::string_view s) {\n    for (std::string_view unit : {\"i\", \"iB\", \"B\"}) {\n        if (boost::algorithm::ends_with(s, unit)) {\n            s.remove_suffix(unit.size());\n            break;\n        }\n    }\n    size_t factor = 1;\n    for (auto [suffix, power] : suffixes) {\n        if (boost::algorithm::ends_with(s, suffix)) {\n            factor <<= power;\n            s.remove_suffix(suffix.size());\n            break;\n        }\n    }\n    return boost::lexical_cast<size_t>(s) * factor;\n}\n\n}\n\n"
  },
  {
    "path": "src/util/exceptions.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#include <seastar/util/exceptions.hh>\n\nnamespace seastar {\n\nstd::filesystem::filesystem_error make_filesystem_error(const std::string& what, std::filesystem::path path, int error) {\n    return std::filesystem::filesystem_error(what, std::move(path), std::error_code(error, std::system_category()));\n}\n\nstd::filesystem::filesystem_error make_filesystem_error(const std::string& what, std::filesystem::path path1, std::filesystem::path path2, int error) {\n    return std::filesystem::filesystem_error(what, std::move(path1), std::move(path1), std::error_code(error, std::system_category()));\n}\n\n} // namespace seastar\n"
  },
  {
    "path": "src/util/file.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n\n#include <cstdint>\n#include <list>\n#include <optional>\n#include <filesystem>\n#include <iostream>\n#include <list>\n#include <vector>\n#include <coroutine>\n#include <sys/statvfs.h>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/util/file.hh>\n\nnamespace seastar {\n\nnamespace fs = std::filesystem;\n\nfuture<> make_directory(std::string_view name, file_permissions permissions) noexcept {\n    return engine().make_directory(name, permissions);\n}\n\nfuture<> touch_directory(std::string_view name, file_permissions permissions) noexcept {\n    return engine().touch_directory(name, permissions);\n}\n\nfuture<> sync_directory(std::string_view name) noexcept {\n    return open_directory(name).then([] (file f) {\n        return do_with(std::move(f), [] (file& f) {\n            return f.flush().then([&f] () mutable {\n                return f.close();\n            });\n        });\n    });\n}\n\nstatic future<> do_recursive_touch_directory(std::string_view base_view, std::string_view name, file_permissions permissions) {\n    sstring base(base_view);\n    static const sstring::value_type separator = '/';\n\n    if (name.empty()) {\n        return make_ready_future<>();\n    }\n\n    size_t pos = std::min(name.find(separator), name.size() - 1);\n    base += sstring(name.substr(0 , pos + 1));\n    name = name.substr(pos + 1);\n    if (name.length() == 1 && name[0] == separator) {\n        name = {};\n    }\n    // use the optional permissions only for last component,\n    // other directories in the patch will always be created using the default_dir_permissions\n    auto f = name.empty() ? touch_directory(base, permissions) : touch_directory(base);\n    return f.then([base, name = sstring(name), permissions] {\n        return do_recursive_touch_directory(base, std::move(name), permissions);\n    }).then([base] {\n        // We will now flush the directory that holds the entry we potentially\n        // created. Technically speaking, we only need to touch when we did\n        // create. But flushing the unchanged ones should be cheap enough - and\n        // it simplifies the code considerably.\n        if (base.empty()) {\n            return make_ready_future<>();\n        }\n\n        return sync_directory(base);\n    });\n}\n\nfuture<> recursive_touch_directory(std::string_view name, file_permissions permissions) noexcept {\n    // If the name is empty,  it will be of the type a/b/c, which should be interpreted as\n    // a relative path. This means we have to flush our current directory\n    std::string_view base = \"\";\n    if (name[0] != '/' || name[0] == '.') {\n        base = \"./\";\n    }\n    return futurize_invoke(do_recursive_touch_directory, base, name, permissions);\n}\n\nfuture<> remove_file(std::string_view pathname) noexcept {\n    return engine().remove_file(pathname);\n}\n\nfuture<> rename_file(std::string_view old_pathname, std::string_view new_pathname) noexcept {\n    return engine().rename_file(old_pathname, new_pathname);\n}\n\nfuture<fs_type> file_system_at(std::string_view name) noexcept {\n    return engine().file_system_at(name);\n}\n\nfuture<uint64_t> fs_avail(std::string_view name) noexcept {\n    return engine().statvfs(name).then([] (struct statvfs st) {\n        return make_ready_future<uint64_t>(st.f_bavail * st.f_frsize);\n    });\n}\n\nfuture<uint64_t> fs_free(std::string_view name) noexcept {\n    return engine().statvfs(name).then([] (struct statvfs st) {\n        return make_ready_future<uint64_t>(st.f_bfree * st.f_frsize);\n    });\n}\n\nfuture<std::filesystem::space_info> file_system_space(std::string_view name) noexcept {\n    return engine().file_system_space(name);\n}\n\nfuture<stat_data> file_stat(std::string_view name, follow_symlink follow) noexcept {\n    return engine().file_stat(name, follow);\n}\n\nfuture<stat_data> file_stat(file& directory, std::string_view name, follow_symlink follow) noexcept {\n    return engine().file_stat(directory, name, follow);\n}\n\nfuture<std::optional<struct group_details>> getgrnam(std::string_view name) {\n    return engine().getgrnam(name);\n}\n\nfuture<> chown(std::string_view filepath, uid_t owner, gid_t group) {\n    return engine().chown(filepath, owner, group);\n}\n\nfuture<uint64_t> file_size(std::string_view name) noexcept {\n    return engine().file_size(name);\n}\n\nfuture<bool> file_accessible(std::string_view name, access_flags flags) noexcept {\n    return engine().file_accessible(name, flags);\n}\n\nfuture<bool> file_exists(std::string_view name) noexcept {\n    return engine().file_exists(name);\n}\n\nfuture<> link_file(std::string_view oldpath, std::string_view newpath) noexcept {\n    return engine().link_file(oldpath, newpath);\n}\n\nfuture<> chmod(std::string_view name, file_permissions permissions) noexcept {\n    return engine().chmod(name, permissions);\n}\n\nstatic future<> do_recursive_remove_directory(const fs::path path) noexcept {\n    struct work_entry {\n        fs::path path;\n        bool listed;\n    };\n\n    std::list<work_entry> work_queue;\n    work_queue.emplace_back(std::move(path), false);\n    while (!work_queue.empty()) {\n        auto ent = std::move(work_queue.back());\n        work_queue.pop_back();\n        if (ent.listed) {\n            co_await remove_file(ent.path.native());\n        } else {\n            work_queue.emplace_back(ent.path, true);\n            auto path = std::move(ent.path);\n            file dir = co_await open_directory(path.native());\n            co_await dir.list_directory([&] (directory_entry de) -> future<> {\n                fs::path sub_path = path / de.name.c_str();\n                bool is_dir = de.type && *de.type == directory_entry_type::directory;\n                work_queue.emplace_back(std::move(sub_path), !is_dir);\n                co_return;\n            }).done();\n            co_await dir.close();\n        }\n    }\n}\n\nfuture<> recursive_remove_directory(fs::path path) noexcept {\n    sstring parent;\n    try {\n        parent = (path / \"..\").native();\n    } catch (...) {\n        return current_exception_as_future();\n    }\n    return open_directory(std::move(parent)).then([path = std::move(path)] (file parent) mutable {\n        return do_with(std::move(parent), [path = std::move(path)] (file& parent) mutable {\n            return do_recursive_remove_directory(std::move(path)).then([&parent] {\n                return parent.flush().then([&parent] () mutable {\n                    return parent.close();\n                });\n            });\n        });\n    });\n}\n\nnamespace util {\n\nfuture<std::vector<temporary_buffer<char>>> read_entire_file(std::filesystem::path path) {\n    return with_file_input_stream(path, [] (input_stream<char>& in) {\n        return read_entire_stream(in);\n    });\n}\n\nfuture<sstring> read_entire_file_contiguous(std::filesystem::path path) {\n    return with_file_input_stream(path, [] (input_stream<char>& in) {\n        return read_entire_stream_contiguous(in);\n    });\n}\n\n} // namespace util\n\n} //namespace seastar\n"
  },
  {
    "path": "src/util/log.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n\n#include <iostream>\n#include <map>\n#include <memory>\n#include <regex>\n#include <string>\n#include <string_view>\n#include <system_error>\n#include <chrono>\n#include <algorithm>\n\n#include <fmt/core.h>\n#include <fmt/chrono.h>\n#include <fmt/color.h>\n#include <fmt/ostream.h>\n#include <boost/any.hpp>\n#include <boost/lexical_cast.hpp>\n#include <boost/program_options.hpp>\n#include <boost/range/adaptor/map.hpp>\n#include <cxxabi.h>\n#include <syslog.h>\n#include <unistd.h>\n\n\n#include <seastar/util/log.hh>\n#include <seastar/util/log-cli.hh>\n\n#include <seastar/util/internal/array_map.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/print.hh>\n\n\n#include \"core/program_options.hh\"\n\nusing namespace std::chrono_literals;\n\nstruct wrapped_log_level {\n    seastar::log_level level;\n};\n\nstatic const std::map<seastar::log_level, std::string_view> log_level_names = {\n        { seastar::log_level::trace, \"trace\" },\n        { seastar::log_level::debug, \"debug\" },\n        { seastar::log_level::info, \"info\" },\n        { seastar::log_level::warn, \"warn\" },\n        { seastar::log_level::error, \"error\" },\n};\n\nnamespace fmt {\ntemplate <> struct formatter<wrapped_log_level> {\n    using log_level = seastar::log_level;\n    static constexpr size_t nr_levels = static_cast<size_t>(log_level::trace) + 1;\n    static bool colored;\n\n    // format specifier not supported\n    template <typename ParseContext>\n    constexpr auto parse(ParseContext &ctx) { return ctx.begin(); }\n\n    template <typename FormatContext>\n    auto format(wrapped_log_level wll, FormatContext& ctx) const {\n        static seastar::internal::array_map<seastar::sstring, nr_levels> text = {\n            { int(log_level::debug), \"DEBUG\" },\n            { int(log_level::info),  \"INFO \" },\n            { int(log_level::trace), \"TRACE\" },\n            { int(log_level::warn),  \"WARN \" },\n            { int(log_level::error), \"ERROR\" },\n        };\n        int index = static_cast<int>(wll.level);\n        std::string_view name = text[index];\n        static seastar::internal::array_map<text_style, nr_levels> style = {\n            { int(log_level::debug), fg(terminal_color::green)  },\n            { int(log_level::info),  fg(terminal_color::white)  },\n            { int(log_level::trace), fg(terminal_color::blue)   },\n            { int(log_level::warn),  fg(terminal_color::yellow) },\n            { int(log_level::error), fg(terminal_color::red)    },\n        };\n        if (colored) {\n            return fmt::format_to(ctx.out(), \"{}\",\n                fmt::format(style[index], \"{}\", name));\n        }\n        return fmt::format_to(ctx.out(), \"{}\", name);\n    }\n};\nbool formatter<wrapped_log_level>::colored = true;\n\nauto formatter<seastar::log_level>::format(seastar::log_level level, format_context& ctx) const\n    -> decltype(ctx.out()) {\n    return fmt::format_to(ctx.out(), \"{}\", log_level_names.at(level));\n}\n\n}\n\nnamespace seastar {\n\nnamespace internal {\n\n[[noreturn]] void assert_fail(const char* msg, const char* file, int line, const char* func) {\n    fprintf(stderr, \"%s:%u: %s: Assertion `%s` failed.\\n\", file, line, func, msg);\n    std::fflush(stderr);\n    std::terminate();\n}\n\nvoid log_buf::free_buffer() noexcept {\n    if (_own_buf) {\n        delete[] _begin;\n    }\n}\n\nvoid log_buf::realloc_buffer_and_append(char c) noexcept {\n  if (_alloc_failure) {\n    // Already failed to reallocate once, don't try again\n    return;\n  }\n\n  try {\n    const auto old_size = size();\n    const auto new_size = old_size * 2;\n\n    auto new_buf = new char[new_size];\n    std::memcpy(new_buf, _begin, old_size);\n    free_buffer();\n\n    _begin = new_buf;\n    _current = _begin + old_size;\n    _end = _begin + new_size;\n    _own_buf = true;\n    *_current++ = c;\n  } catch (...) {\n    _alloc_failure = true;\n    std::string_view msg = \"(log buffer allocation failure)\";\n    auto can_copy = std::min(msg.size(), size_t(_current - _begin));\n    std::memcpy(_current - can_copy, msg.data(), can_copy);\n  }\n}\n\nlog_buf::log_buf()\n    : _begin(new char[512])\n    , _end(_begin + 512)\n    , _current(_begin)\n    , _own_buf(true)\n{\n}\n\nlog_buf::log_buf(char* external_buf, size_t size) noexcept\n    : _begin(external_buf)\n    , _end(_begin + size)\n    , _current(_begin)\n    , _own_buf(false)\n{\n}\n\nlog_buf::~log_buf() {\n    free_buffer();\n}\n\n} // namespace internal\n\nthread_local uint64_t logging_failures = 0;\n\nvoid validate(boost::any& v,\n              const std::vector<std::string>& values,\n              logger_timestamp_style* target_type, int) {\n    using namespace boost::program_options;\n    validators::check_first_occurrence(v);\n    auto s = validators::get_single_string(values);\n    if (s == \"none\") {\n        v = logger_timestamp_style::none;\n        return;\n    } else if (s == \"boot\") {\n        v = logger_timestamp_style::boot;\n        return;\n    } else if (s == \"real\") {\n        v = logger_timestamp_style::real;\n        return;\n    }\n    throw validation_error(validation_error::invalid_option_value);\n}\n\nstd::ostream& operator<<(std::ostream& os, logger_timestamp_style lts) {\n    switch (lts) {\n    case logger_timestamp_style::none: return os << \"none\";\n    case logger_timestamp_style::boot: return os << \"boot\";\n    case logger_timestamp_style::real: return os << \"real\";\n    default: abort();\n    }\n    return os;\n}\n\nvoid validate(boost::any& v,\n              const std::vector<std::string>& values,\n              logger_ostream_type* target_type, int) {\n    using namespace boost::program_options;\n    validators::check_first_occurrence(v);\n    auto s = validators::get_single_string(values);\n    if (s == \"none\") {\n        v = logger_ostream_type::none;\n        return;\n    } else if (s == \"stdout\") {\n        v = logger_ostream_type::cout;\n        return;\n    } else if (s == \"stderr\") {\n        v = logger_ostream_type::cerr;\n        return;\n    }\n    throw validation_error(validation_error::invalid_option_value);\n}\n\nstd::ostream& operator<<(std::ostream& os, logger_ostream_type lot) {\n    switch (lot) {\n    case logger_ostream_type::none: return os << \"none\";\n    case logger_ostream_type::cout: return os << \"stdout\";\n    case logger_ostream_type::cerr: return os << \"stderr\";\n    default: abort();\n    }\n    return os;\n}\n\nstatic internal::log_buf::inserter_iterator print_no_timestamp(internal::log_buf::inserter_iterator it) {\n    return it;\n}\n\nstatic internal::log_buf::inserter_iterator print_boot_timestamp(internal::log_buf::inserter_iterator it) {\n    auto n = std::chrono::steady_clock::now().time_since_epoch() / 1us;\n    return fmt::format_to(it, \"{:10d}.{:06d}\", n / 1000000, n % 1000000);\n}\n\nstatic internal::log_buf::inserter_iterator print_real_timestamp(internal::log_buf::inserter_iterator it) {\n    struct a_second {\n        time_t t;\n        std::array<char, 32> static_buf; // big enough to hold '2023-01-14 15:06:33'\n        internal::log_buf buf{static_buf.data(), static_buf.size()};\n    };\n    static thread_local a_second this_second;\n    using clock = std::chrono::system_clock;\n    auto n = clock::now();\n    auto t = clock::to_time_t(n);\n    if (this_second.t != t) {\n        this_second.t = t;\n        this_second.buf.clear();\n        std::tm tm_local;\n        if (!localtime_r(&t, &tm_local)) {\n            throw fmt::format_error(\"time_t value out of range\");\n        }\n        fmt::format_to(this_second.buf.back_insert_begin(), \"{:%F %T}\", tm_local);\n    }\n    auto ms = (n - clock::from_time_t(t)) / 1ms;\n    return fmt::format_to(it, \"{},{:03d}\", this_second.buf.view(), ms);\n}\n\nstatic internal::log_buf::inserter_iterator (*print_timestamp)(internal::log_buf::inserter_iterator) = print_no_timestamp;\n\n\nstd::ostream& operator<<(std::ostream& out, log_level level) {\n    return out << log_level_names.at(level);\n}\n\nstd::istream& operator>>(std::istream& in, log_level& level) {\n    sstring s;\n    in >> s;\n    if (!in) {\n        return in;\n    }\n    for (auto&& x : log_level_names) {\n        if (s == x.second) {\n            level = x.first;\n            return in;\n        }\n    }\n    in.setstate(std::ios::failbit);\n    return in;\n}\n\nstd::ostream* logger::_out = &std::cerr;\nstd::atomic<bool> logger::_ostream = { true };\nstd::atomic<bool> logger::_syslog = { false };\nunsigned logger::_shard_field_width = 1;\n#ifdef SEASTAR_BUILD_SHARED_LIBS\nthread_local bool logger::silent = false;\n#endif\n\nlogger::logger(sstring name) : _name(std::move(name)) {\n    global_logger_registry().register_logger(this);\n}\n\nlogger::logger(logger&& x) : _name(std::move(x._name)), _level(x._level.load(std::memory_order_relaxed)) {\n    global_logger_registry().moved(&x, this);\n}\n\nlogger::~logger() {\n    global_logger_registry().unregister_logger(this);\n}\n\nstatic thread_local std::array<char, 8192> static_log_buf;\n\nbool logger::rate_limit::check() {\n    const auto now = clock::now();\n    if (now < _next) {\n        ++_dropped_messages;\n        return false;\n    }\n    _next = now + _interval;\n    return true;\n}\n\nlogger::rate_limit::rate_limit(std::chrono::milliseconds interval)\n    : _interval(interval), _next(clock::now())\n{ }\n\nvoid\nlogger::do_log(log_level level, log_writer& writer) {\n    bool is_ostream_enabled = _ostream.load(std::memory_order_relaxed);\n    bool is_syslog_enabled = _syslog.load(std::memory_order_relaxed);\n    if(!is_ostream_enabled && !is_syslog_enabled) {\n      return;\n    }\n    auto print_once = [&] (internal::log_buf::inserter_iterator it) {\n      if (local_engine) {\n          it = fmt::format_to(it, \" [shard {:{}}:{}]\", this_shard_id(), _shard_field_width, current_scheduling_group().short_name());\n      }\n      it = fmt::format_to(it, \" {} - \", _name);\n      return writer(it);\n    };\n\n    // Mainly this protects us from re-entrance via malloc()'s\n    // oversized allocation warnings and failed allocation errors\n    silencer be_silent;\n\n    if (is_ostream_enabled) {\n        internal::log_buf buf(static_log_buf.data(), static_log_buf.size());\n        auto it = buf.back_insert_begin();\n        it = fmt::format_to(it, \"{} \", wrapped_log_level{level});\n        it = print_timestamp(it);\n        it = print_once(it);\n        *it++ = '\\n';\n        *_out << buf.view();\n        _out->flush();\n    }\n    if (is_syslog_enabled) {\n        internal::log_buf buf(static_log_buf.data(), static_log_buf.size());\n        auto it = buf.back_insert_begin();\n        it = print_once(it);\n        *it = '\\0';\n        static internal::array_map<int, 20> level_map = {\n                { int(log_level::debug), LOG_DEBUG },\n                { int(log_level::info), LOG_INFO },\n                { int(log_level::trace), LOG_DEBUG },  // no LOG_TRACE\n                { int(log_level::warn), LOG_WARNING },\n                { int(log_level::error), LOG_ERR },\n        };\n        // NOTE: syslog() can block, which will stall the reactor thread.\n        //       this should be rare (will have to fill the pipe buffer\n        //       before syslogd can clear it) but can happen.  If it does,\n        //       we'll have to implement some internal buffering (which\n        //       still means the problem can happen, just less frequently).\n        // syslog() interprets % characters, so send msg as a parameter\n        syslog(level_map[int(level)], \"%s\", buf.data());\n    }\n}\n\nvoid logger::failed_to_log(std::exception_ptr ex,\n                           fmt::string_view fmt,\n                           std::source_location loc) noexcept\n{\n    try {\n        lambda_log_writer writer([ex = std::move(ex), fmt, loc] (internal::log_buf::inserter_iterator it) {\n            it = fmt::format_to(it, \"{}:{} @{}: failed to log message\", loc.file_name(), loc.line(), loc.function_name());\n            if (fmt.size() > 0) {\n                it = fmt::format_to(it, \": fmt='{}'\", fmt);\n            }\n            return fmt::format_to(it, \": {}\", ex);\n        });\n        do_log(log_level::error, writer);\n    } catch (...) {\n        ++logging_failures;\n    }\n}\n\nvoid\nlogger::set_ostream(std::ostream& out) noexcept {\n    _out = &out;\n}\n\nvoid\nlogger::set_ostream_enabled(bool enabled) noexcept {\n    _ostream.store(enabled, std::memory_order_relaxed);\n}\n\nvoid\nlogger::set_syslog_enabled(bool enabled) noexcept {\n    _syslog.store(enabled, std::memory_order_relaxed);\n}\n\nvoid\nlogger::set_shard_field_width(unsigned width) noexcept {\n    _shard_field_width = width;\n}\n\nvoid\nlogger::set_with_color(bool enabled) noexcept {\n    fmt::formatter<wrapped_log_level>::colored = enabled;\n}\n\nbool logger::is_shard_zero() noexcept {\n    return this_shard_id() == 0;\n}\n\nvoid\nlogger_registry::set_all_loggers_level(log_level level) {\n    std::lock_guard<std::mutex> g(_mutex);\n    for (auto&& l : _loggers | boost::adaptors::map_values) {\n        l->set_level(level);\n    }\n}\n\nlog_level\nlogger_registry::get_logger_level(sstring name) const {\n    std::lock_guard<std::mutex> g(_mutex);\n    return _loggers.at(name)->level();\n}\n\nvoid\nlogger_registry::set_logger_level(sstring name, log_level level) {\n    std::lock_guard<std::mutex> g(_mutex);\n    _loggers.at(name)->set_level(level);\n}\n\nstd::vector<sstring>\nlogger_registry::get_all_logger_names() {\n    std::lock_guard<std::mutex> g(_mutex);\n    auto ret = _loggers | boost::adaptors::map_keys;\n    return std::vector<sstring>(ret.begin(), ret.end());\n}\n\nvoid\nlogger_registry::register_logger(logger* l) {\n    std::lock_guard<std::mutex> g(_mutex);\n    if (_loggers.find(l->name()) != _loggers.end()) {\n        throw std::runtime_error(format(\"Logger '{}' registered twice\", l->name()));\n    }\n    _loggers[l->name()] = l;\n}\n\nvoid\nlogger_registry::unregister_logger(logger* l) {\n    std::lock_guard<std::mutex> g(_mutex);\n    _loggers.erase(l->name());\n}\n\nvoid\nlogger_registry::moved(logger* from, logger* to) {\n    std::lock_guard<std::mutex> g(_mutex);\n    _loggers[from->name()] = to;\n}\n\nvoid apply_logging_settings(const logging_settings& s) {\n    global_logger_registry().set_all_loggers_level(s.default_level);\n\n    for (const auto& pair : s.logger_levels) {\n        try {\n            global_logger_registry().set_logger_level(pair.first, pair.second);\n        } catch (const std::out_of_range&) {\n            throw std::runtime_error(\n                        seastar::format(\"Unknown logger '{}'. Use --help-loggers to list available loggers.\",\n                                        pair.first));\n        }\n    }\n\n    logger_ostream_type logger_ostream = s.stdout_enabled ? s.logger_ostream : logger_ostream_type::none;\n    switch (logger_ostream) {\n    case logger_ostream_type::none:\n        logger::set_ostream_enabled(false);\n        break;\n    case logger_ostream_type::cout:\n        logger::set_ostream(std::cout);\n        logger::set_ostream_enabled(true);\n        break;\n    case logger_ostream_type::cerr:\n        logger::set_ostream(std::cerr);\n        logger::set_ostream_enabled(true);\n        break;\n    }\n    logger::set_syslog_enabled(s.syslog_enabled);\n    logger::set_with_color(s.with_color);\n\n    switch (s.stdout_timestamp_style) {\n    case logger_timestamp_style::none:\n        print_timestamp = print_no_timestamp;\n        break;\n    case logger_timestamp_style::boot:\n        print_timestamp = print_boot_timestamp;\n        break;\n    case logger_timestamp_style::real:\n        print_timestamp = print_real_timestamp;\n        break;\n    default:\n        break;\n    }\n}\n\nsstring pretty_type_name(const std::type_info& ti) {\n    int status;\n    std::unique_ptr<char[], void (*)(void*)> result(\n            abi::__cxa_demangle(ti.name(), 0, 0, &status), std::free);\n    return result.get() ? result.get() : ti.name();\n}\n\nlogger_registry& global_logger_registry() {\n    static logger_registry g_registry;\n    return g_registry;\n}\n\nsstring level_name(log_level level) {\n    return sstring(log_level_names.at(level));\n}\n\nnamespace log_cli {\n\nnamespace bpo = boost::program_options;\n\nlog_level parse_log_level(const sstring& s) {\n    try {\n        return boost::lexical_cast<log_level>(s.c_str());\n    } catch (const boost::bad_lexical_cast&) {\n        throw std::runtime_error(format(\"Unknown log level '{}'\", s));\n    }\n}\n\nvoid parse_map_associations(const std::string& v, std::function<void(std::string, std::string)> consume_key_value) {\n    static const std::regex colon(\":\");\n\n    std::sregex_token_iterator s(v.begin(), v.end(), colon, -1);\n    const std::sregex_token_iterator e;\n    while (s != e) {\n        const sstring p = std::string(*s++);\n\n        const auto i = p.find('=');\n        if (i == sstring::npos) {\n            throw bpo::invalid_option_value(p);\n        }\n\n        auto k = p.substr(0, i);\n        auto v = p.substr(i + 1, p.size());\n        consume_key_value(std::move(k), std::move(v));\n    };\n}\n\nbpo::options_description get_options_description() {\n    program_options::options_description_building_visitor descriptor;\n    options(nullptr).describe(descriptor);\n    return std::move(descriptor).get_options_description();\n}\n\noptions::options(program_options::option_group* parent_group)\n    : program_options::option_group(parent_group, \"Logging options\")\n    , default_log_level(*this, \"default-log-level\",\n             log_level::info,\n             \"Default log level for log messages. Valid values are trace, debug, info, warn, error.\"\n             )\n    , logger_log_level(*this, \"logger-log-level\",\n             log_level_map{},\n             \"Map of logger name to log level. The format is \\\"NAME0=LEVEL0[:NAME1=LEVEL1:...]\\\". \"\n             \"Valid logger names can be queried with --help-loggers. \"\n             \"Valid values for levels are trace, debug, info, warn, error. \"\n             \"This option can be specified multiple times.\"\n            )\n    , logger_stdout_timestamps(*this, \"logger-stdout-timestamps\", logger_timestamp_style::real,\n                    \"Select timestamp style for stdout logs: none|boot|real\")\n    , log_to_stdout(*this, \"log-to-stdout\", true, \"Send log output to output stream, as selected by --logger-ostream-type\")\n    , logger_ostream_type(*this, \"logger-ostream-type\", logger_ostream_type::cerr,\n            \"Send log output to: none|stdout|stderr\")\n    , log_to_syslog(*this, \"log-to-syslog\", false, \"Send log output to syslog.\")\n    , log_with_color(*this, \"log-with-color\", isatty(STDOUT_FILENO), \"Print colored tag prefix in log message written to ostream\")\n{\n}\n\nvoid print_available_loggers(std::ostream& os) {\n    auto names = global_logger_registry().get_all_logger_names();\n    // For quick searching by humans.\n    std::sort(names.begin(), names.end());\n\n    os << \"Available loggers:\\n\";\n\n    for (auto&& name : names) {\n        os << \"    \" << name << '\\n';\n    }\n}\n\nlogging_settings extract_settings(const boost::program_options::variables_map& vars) {\n    options opts(nullptr);\n    program_options::variables_map_extracting_visitor visitor(vars);\n    opts.mutate(visitor);\n    return extract_settings(opts);\n}\n\nlogging_settings extract_settings(const options& opts) {\n    return logging_settings{\n        opts.logger_log_level.get_value(),\n        opts.default_log_level.get_value(),\n        opts.log_to_stdout.get_value(),\n        opts.log_to_syslog.get_value(),\n        opts.log_with_color.get_value(),\n        opts.logger_stdout_timestamps.get_value(),\n        opts.logger_ostream_type.get_value(),\n    };\n}\n\n}\n\n}\n\nnamespace std {\nstd::ostream& operator<<(std::ostream& out, const std::exception_ptr& eptr) {\n    if (!eptr) {\n        out << \"<no exception>\";\n        return out;\n    }\n    try {\n        std::rethrow_exception(eptr);\n    } catch(...) {\n        auto tp = abi::__cxa_current_exception_type();\n        if (tp) {\n            out << seastar::pretty_type_name(*tp);\n        } else {\n            // This case shouldn't happen...\n            out << \"<unknown exception>\";\n        }\n        // Print more information on some familiar exception types\n        try {\n            throw;\n        } catch (const seastar::nested_exception& ne) {\n            out << fmt::format(\": {} (while cleaning up after {})\", ne.inner, ne.outer);\n        } catch (const std::system_error& e) {\n            out << \" (error \" << e.code() << \", \" << e.what() << \")\";\n        } catch (const std::exception& e) {\n            out << \" (\" << e.what() << \")\";\n        } catch (...) {\n            // no extra info\n        }\n\n        try {\n            throw;\n        } catch (const std::nested_exception& ne) {\n            out << \": \" << ne.nested_ptr();\n        } catch (...) {\n            // do nothing\n        }\n    }\n    return out;\n}\n\nstd::ostream& operator<<(std::ostream& out, const std::exception& e) {\n    return out << seastar::pretty_type_name(typeid(e)) << \" (\" << e.what() << \")\";\n}\n\nstd::ostream& operator<<(std::ostream& out, const std::system_error& e) {\n    return out << seastar::pretty_type_name(typeid(e)) << \" (error \" << e.code() << \", \" << e.what() << \")\";\n}\n\n}\n"
  },
  {
    "path": "src/util/process.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 Kefu Chai ( tchaikov@gmail.com )\n */\n\n#include <seastar/core/fstream.hh>\n#include <seastar/core/internal/buffer_allocator.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/polymorphic_temporary_buffer.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/util/process.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar::experimental {\n\nnamespace {\nclass pipe_data_source_impl final : public data_source_impl {\n    static constexpr std::size_t buffer_size = 8192;\n    struct buffer_allocator : public internal::buffer_allocator {\n        temporary_buffer<char> allocate_buffer() override {\n            return make_temporary_buffer<char>(memory::malloc_allocator, buffer_size);\n        }\n    };\n    pollable_fd _fd;\n    buffer_allocator _ba;\npublic:\n    explicit pipe_data_source_impl(pollable_fd fd)\n        : _fd(std::move(fd)) {}\n    static auto from_fd(file_desc&& fd) {\n        return std::make_unique<pipe_data_source_impl>(pollable_fd(std::move(fd)));\n    }\n    future<temporary_buffer<char>> get() override {\n        return _fd.read_some(&_ba);\n    }\n    future<> close() override {\n        _fd.close();\n        return make_ready_future();\n    }\n};\n\nclass pipe_data_sink_impl final : public data_sink_impl {\n    file_desc _fd;\n    io_queue& _io_queue;\n    const size_t _buffer_size;\npublic:\n    explicit pipe_data_sink_impl(file_desc&& fd)\n        : _fd(std::move(fd))\n        , _io_queue(engine().get_io_queue(0))\n        , _buffer_size(file_input_stream_options{}.buffer_size) {}\n    static auto from_fd(file_desc&& fd) {\n        return std::make_unique<pipe_data_sink_impl>(std::move(fd));\n    }\nprivate:\n    future<> do_put(temporary_buffer<char> buf) {\n        size_t buf_size = buf.size();\n        auto req = internal::io_request::make_write(_fd.get(), 0, buf.get(), buf_size, false);\n        return _io_queue.submit_io_write(buf_size, std::move(req), nullptr).then(\n            [this, buf = std::move(buf), buf_size] (size_t written) mutable {\n                if (written < buf_size) {\n                    buf.trim_front(written);\n                    return do_put(std::move(buf));\n                }\n                return make_ready_future();\n            });\n    }\n\npublic:\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        return data_sink_impl::fallback_put(bufs, [this] (temporary_buffer<char>&& buf) {\n            return do_put(std::move(buf));\n        });\n    }\n#else\n    using data_sink_impl::put;\n    future<> put(temporary_buffer<char> buf) override {\n        return do_put(std::move(buf));\n    }\n    future<> put(net::packet data) override {\n        return do_with(data.release(), [this] (std::vector<temporary_buffer<char>>& bufs) {\n            return do_for_each(bufs, [this] (temporary_buffer<char>& buf) {\n                return put(buf.share());\n            });\n        });\n    }\n#endif\n    future<> close() override {\n        _fd.close();\n        return make_ready_future();\n    }\n    size_t buffer_size() const noexcept override {\n        return _buffer_size;\n    }\n};\n}\n\nprocess::process(create_tag, pid_t pid, file_desc&& cin, file_desc&& cout, file_desc&& cerr)\n    : _pid(pid)\n    , _stdin(std::move(cin))\n    , _stdout(std::move(cout))\n    , _stderr(std::move(cerr)) {}\n\nfuture<process::wait_status> process::wait() {\n    return engine().waitpid(_pid).then([] (int wstatus) -> wait_status {\n        if (WIFEXITED(wstatus)) {\n            return wait_exited{WEXITSTATUS(wstatus)};\n        } else {\n            SEASTAR_ASSERT(WIFSIGNALED(wstatus));\n            return wait_signaled{WTERMSIG(wstatus)};\n        }\n    });\n}\n\nvoid process::terminate() {\n    engine().kill(_pid, SIGTERM);\n}\n\nvoid process::kill() {\n    engine().kill(_pid, SIGKILL);\n}\n\nfuture<process> process::spawn(const std::filesystem::path& pathname,\n                               spawn_parameters params) {\n    SEASTAR_ASSERT(!params.argv.empty());\n    return engine().spawn(pathname.native(), std::move(params.argv), std::move(params.env)).then_unpack(\n            [] (pid_t pid, file_desc stdin_pipe, file_desc stdout_pipe, file_desc stderr_pipe) {\n        return make_ready_future<process>(create_tag{}, pid, std::move(stdin_pipe), std::move(stdout_pipe), std::move(stderr_pipe));\n    });\n}\n\nfuture<process> process::spawn(const std::filesystem::path& pathname) {\n    return spawn(pathname, {{pathname.native()}, {}});\n}\n\noutput_stream<char> process::cin() {\n    return output_stream<char>(data_sink(pipe_data_sink_impl::from_fd(std::move(_stdin))));\n}\n\ninput_stream<char> process::cout() {\n    return input_stream<char>(data_source(pipe_data_source_impl::from_fd(std::move(_stdout))));\n}\n\ninput_stream<char> process::cerr() {\n    return input_stream<char>(data_source(pipe_data_source_impl::from_fd(std::move(_stderr))));\n}\n\n}\n"
  },
  {
    "path": "src/util/program-options.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.// Custom \"validator\" that gets called by the internals of Boost.Test. This allows for reading associations into an\n// unordered map and for multiple occurances of the option name to appear and be merged.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n\n#include <boost/any.hpp>\n#include <boost/intrusive/list.hpp>\n#include <boost/program_options.hpp>\n\n#include <seastar/util/program-options.hh>\n#include <seastar/util/log-cli.hh>\n\nnamespace bpo = boost::program_options;\n\nnamespace seastar {\n\nnamespace program_options {\n\nsstring get_or_default(const string_map& ss, const sstring& key, const sstring& def) {\n    const auto iter = ss.find(key);\n    if (iter != ss.end()) {\n        return iter->second;\n    }\n\n    return def;\n}\n\nvoid validate(boost::any& out, const std::vector<std::string>& in, string_map*, int) {\n    if (out.empty()) {\n        out = boost::any(string_map());\n    }\n\n    auto* ss = boost::any_cast<string_map>(&out);\n\n    for (const auto& s : in) {\n        log_cli::parse_map_associations(s, [&ss] (std::string k, std::string v) { (*ss)[std::move(k)] = std::move(v); });\n    }\n}\n\nstd::ostream& operator<<(std::ostream& os, const string_map& ss) {\n    int n = 0;\n\n    for (const auto& e : ss) {\n        if (n > 0) {\n            os << \":\";\n        }\n\n        os << e.first << \"=\" << e.second;\n        ++n;\n    }\n\n    return os;\n}\n\nstd::istream& operator>>(std::istream& is, string_map& ss) {\n    std::string str;\n    is >> str;\n\n    log_cli::parse_map_associations(str, [&ss] (std::string k, std::string v) { ss[std::move(k)] = std::move(v); });\n    return is;\n}\n\noption_group::option_group(option_group* parent, std::string name)\n    : _parent(parent), _used(true), _name(std::move(name)) {\n    if (_parent) {\n        _parent->_subgroups.push_back(*this);\n    }\n}\n\noption_group::option_group(option_group* parent, std::string name, unused)\n    : _parent(parent), _used(false), _name(std::move(name)) {\n    if (_parent) {\n        _parent->_subgroups.push_back(*this);\n    }\n}\n\noption_group::option_group(option_group&& o)\n    : _parent(o._parent), _used(o._used), _name(std::move(o._name))\n{\n    for (auto& val : o._values) {\n        val._group = this;\n    }\n    for (auto& grp : o._subgroups) {\n        grp._parent = this;\n    }\n    unlink();\n    if (_parent) {\n        _parent->_subgroups.push_back(*this);\n    }\n}\n\nvoid option_group::describe(options_descriptor& descriptor) const {\n    if (descriptor.visit_group_start(_name, _used)) {\n        for (auto& value : _values) {\n            value.describe(descriptor);\n        }\n        for (auto& grp : _subgroups) {\n            grp.describe(descriptor);\n        }\n    }\n    descriptor.visit_group_end();\n}\n\nvoid option_group::mutate(options_mutator& mutator) {\n    if (mutator.visit_group_start(_name, _used)) {\n        for (auto& value : _values) {\n            value.mutate(mutator);\n        }\n        for (auto& grp : _subgroups) {\n            grp.mutate(mutator);\n        }\n    }\n    mutator.visit_group_end();\n}\n\nbasic_value::basic_value(option_group& group, bool used, std::string name, std::string description)\n    : _group(&group), _used(used), _name(std::move(name)), _description(std::move(description))\n{\n    _group->_values.push_back(*this);\n}\n\nbasic_value::basic_value(basic_value&& o)\n    : _group(o._group), _used(o._used), _name(std::move(o._name)), _description(std::move(o._description))\n{\n    unlink();\n    _group->_values.push_back(*this);\n}\n\nvoid basic_value::describe(options_descriptor& descriptor) const {\n    if (descriptor.visit_value_metadata(_name, _description, _used)) {\n        do_describe(descriptor);\n    }\n}\n\nvoid basic_value::mutate(options_mutator& mutator) {\n    if (mutator.visit_value_metadata(_name, _used)) {\n        do_mutate(mutator);\n    }\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/util/read_first_line.cc",
    "content": "#include <seastar/core/posix.hh>\n#include <seastar/util/read_first_line.hh>\n\nnamespace seastar {\n\nsstring read_first_line(std::filesystem::path sys_file) {\n    auto file = file_desc::open(sys_file.string(), O_RDONLY | O_CLOEXEC);\n    sstring buf;\n    size_t n = 0;\n    do {\n        // try to avoid allocations\n        sstring tmp = uninitialized_string(8);\n        auto ret = file.read(tmp.data(), 8ul);\n        if (!ret) { // EAGAIN\n            continue;\n        }\n        n = *ret;\n        if (n > 0) {\n            buf += tmp;\n        }\n    } while (n != 0);\n    auto end = buf.find('\\n');\n    auto value = buf.substr(0, end);\n    file.close();\n    return value;\n}\n\n}\n"
  },
  {
    "path": "src/util/short_streams.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n\n#include <seastar/core/future.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/temporary_buffer.hh>\n\nnamespace seastar {\n\nnamespace util {\n\nfuture<std::vector<temporary_buffer<char>>> read_entire_stream(input_stream<char>& inp) {\n    using tmp_buf = temporary_buffer<char>;\n    using consumption_result_type = consumption_result<char>;\n    return do_with(std::vector<tmp_buf>(), [&inp] (std::vector<tmp_buf>& bufs) {\n        return inp.consume([&bufs] (tmp_buf buf) {\n            if (buf.empty()) {\n                return make_ready_future<consumption_result_type>(stop_consuming(std::move(buf)));\n            }\n            bufs.push_back(std::move(buf));\n            return make_ready_future<consumption_result_type>(continue_consuming());\n        }).then([&bufs] {\n            return std::move(bufs);\n        });\n    });\n}\n\nfuture<sstring> read_entire_stream_contiguous(input_stream<char>& inp) {\n    return read_entire_stream(inp).then([] (std::vector<temporary_buffer<char>> bufs) {\n        size_t total_size = 0;\n        for (auto&& buf : bufs) {\n            total_size += buf.size();\n        }\n        auto ret = uninitialized_string(total_size);\n        size_t pos = 0;\n        for (auto&& buf : bufs) {\n            std::copy(buf.begin(), buf.end(), ret.data() + pos);\n            pos += buf.size();\n        }\n        return ret;\n    });\n};\n\nfuture<> skip_entire_stream(input_stream<char>& inp) {\n    return inp.consume([] (temporary_buffer<char> tmp) {\n        return tmp.empty() ? make_ready_future<consumption_result<char>>(stop_consuming(temporary_buffer<char>()))\n                           : make_ready_future<consumption_result<char>>(continue_consuming());\n    });\n}\n\n}\n\n}\n"
  },
  {
    "path": "src/util/tmp_file.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2020 ScyllaDB\n */\n\n#include <iostream>\n#include <random>\n\n#include <seastar/core/seastar.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/exceptions.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/tmp_file.hh>\n#include <seastar/util/file.hh>\n\nnamespace seastar {\n\nnamespace fs = std::filesystem;\n\nstatic constexpr const char* default_tmp_name_template = \"XXXXXX.tmp\";\n\nstatic fs::path\ngenerate_tmp_name(const fs::path& path_template) {\n    fs::path parent = path_template.parent_path();\n    std::string filename = path_template.filename().native();\n    if (parent.empty()) {\n        parent = \".\";\n    }\n    auto pos = filename.find(\"XX\");\n    if (pos == std::string::npos) {\n        parent = path_template;\n        filename = default_tmp_name_template;\n        pos = filename.find(\"XX\");\n        SEASTAR_ASSERT(pos != std::string::npos);\n    }\n    auto end = filename.size();\n    static constexpr char charset[] = \"0123456789abcdef\";\n    static thread_local std::default_random_engine engine(std::random_device{}());\n    static thread_local std::uniform_int_distribution<int> dist(0, sizeof(charset) - 2);\n    while (pos < end && filename[pos] == 'X') {\n        filename[pos++] = charset[dist(engine)];\n    }\n    parent /= filename;\n    return parent;\n}\n\nstatic fs::path default_tmpdir_path;\n\nconst fs::path& default_tmpdir() {\n    if (default_tmpdir_path.empty()) {\n        auto TMPDIR = getenv(\"TMPDIR\");\n        default_tmpdir_path = TMPDIR ? TMPDIR : \"/tmp\";\n    }\n    return default_tmpdir_path;\n}\n\nvoid set_default_tmpdir(fs::path path) {\n    default_tmpdir_path = std::move(path);\n}\n\ntmp_file::tmp_file(tmp_file&& x) noexcept\n    : _path(std::move(x._path))\n    , _file(std::move(x._file))\n{\n    std::swap(_is_open, x._is_open);\n}\n\ntmp_file::~tmp_file() {\n    SEASTAR_ASSERT(!has_path());\n    SEASTAR_ASSERT(!is_open());\n}\n\nfuture<> tmp_file::open(fs::path path_template, open_flags oflags, file_open_options options) noexcept {\n    SEASTAR_ASSERT(!has_path());\n    SEASTAR_ASSERT(!is_open());\n    oflags |= open_flags::create | open_flags::exclusive;\n    fs::path path;\n    try {\n        path = generate_tmp_name(std::move(path_template));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n    return open_file_dma(path.native(), oflags, std::move(options)).then([this, path = std::move(path)] (file f) mutable {\n        _path = std::move(path);\n        _file = std::move(f);\n        _is_open = true;\n        return make_ready_future<>();\n    });\n}\n\nfuture<> tmp_file::close() noexcept {\n    if (!is_open()) {\n        return make_ready_future<>();\n    }\n    return _file.close().then([this] {\n        _is_open = false;\n    });\n}\n\nfuture<> tmp_file::remove() noexcept {\n    if (!has_path()) {\n        return make_ready_future<>();\n    }\n    return remove_file(get_path().native()).then([this] {\n        _path.clear();\n    });\n}\n\nfuture<tmp_file>\nmake_tmp_file(fs::path path_template, open_flags oflags, file_open_options options) noexcept {\n    return do_with(tmp_file(), [path_template = std::move(path_template), oflags, options = std::move(options)] (tmp_file& t) mutable {\n        return t.open(std::move(path_template), oflags, std::move(options)).then([&t] {\n            return make_ready_future<tmp_file>(std::move(t));\n        });\n    });\n}\n\ntmp_dir::~tmp_dir() {\n    SEASTAR_ASSERT(!has_path());\n}\n\nfuture<> tmp_dir::create(fs::path path_template, file_permissions create_permissions) noexcept {\n    SEASTAR_ASSERT(!has_path());\n    fs::path path;\n    try {\n        path = generate_tmp_name(std::move(path_template));\n    } catch (...) {\n        return current_exception_as_future();\n    }\n    return touch_directory(path.native(), create_permissions).then([this, path = std::move(path)] () mutable {\n        _path = std::move(path);\n        return make_ready_future<>();\n    });\n}\n\nfuture<> tmp_dir::remove() noexcept {\n    if (!has_path()) {\n        return make_ready_future<>();\n    }\n    return recursive_remove_directory(std::move(_path));\n}\n\nfuture<tmp_dir> make_tmp_dir(std::filesystem::path path_template, file_permissions create_permissions) noexcept {\n    return do_with(tmp_dir(), [path_template = std::move(path_template), create_permissions] (tmp_dir& t) mutable {\n        return t.create(std::move(path_template), create_permissions).then([&t] () mutable {\n            return make_ready_future<tmp_dir>(std::move(t));\n        });\n    });\n}\n\n} //namespace seastar\n"
  },
  {
    "path": "src/websocket/client.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <exception>\n#include <random>\n#include <seastar/core/future.hh>\n#include <seastar/coroutine/all.hh>\n#include <seastar/websocket/client.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/http/reply.hh>\n\nnamespace seastar::experimental::websocket {\n\nusing namespace std::string_view_literals;\n\n// refer https://datatracker.ietf.org/doc/html/rfc6455#section-1.3\nconstexpr auto magic_key_suffix_client = \"258EAFA5-E914-47DA-95CA-C5AB0DC85B11\"sv;\n\nstatic thread_local std::mt19937 rng(std::random_device{}());\n\nstatic sstring generate_websocket_key() {\n    char raw[16];\n    for (int i = 0; i < 16; i += 4) {\n        uint32_t val = rng();\n        std::memcpy(raw + i, &val, sizeof(val));\n    }\n    return sstring(encode_base64(std::string_view(raw, 16)));\n}\n\ntemplate <bool text_frame>\nclient_connection<text_frame>::client_connection(connected_socket&& fd, sstring resource,\n                                     sstring host, sstring subprotocol,\n                                     handler_t handler)\n    : basic_connection<true, text_frame>(std::move(fd))\n    , _resource(std::move(resource))\n    , _host(std::move(host))\n{\n    this->_subprotocol = std::move(subprotocol);\n    this->_handler = std::move(handler);\n}\n\ntemplate <bool text_frame>\nfuture<> client_connection<text_frame>::send_http_upgrade_request() {\n    _websocket_key = generate_websocket_key();\n    auto req = fmt::format(\n        \"GET {} HTTP/1.1\\r\\n\"\n        \"Host: {}\\r\\n\"\n        \"Upgrade: websocket\\r\\n\"\n        \"Connection: Upgrade\\r\\n\"\n        \"Sec-WebSocket-Key: {}\\r\\n\"\n        \"Sec-WebSocket-Version: 13\\r\\n\",\n        _resource, _host, _websocket_key);\n\n    if (!this->_subprotocol.empty()) {\n        req += fmt::format(\"Sec-WebSocket-Protocol: {}\\r\\n\", this->_subprotocol);\n    }\n    req += \"\\r\\n\";\n\n    co_await this->_write_buf.write(req);\n    co_await this->_write_buf.flush();\n}\n\ntemplate <bool text_frame>\nfuture<> client_connection<text_frame>::read_http_upgrade_response() {\n    _http_parser.init();\n    co_await this->_read_buf.consume(_http_parser);\n\n    if (_http_parser.eof()) {\n        throw websocket::exception(\"Connection closed during HTTP upgrade\");\n    }\n\n    auto resp = _http_parser.get_parsed_response();\n    if (!resp) {\n        throw websocket::exception(\"Failed to parse HTTP upgrade response\");\n    }\n\n    if (resp->_status != http::reply::status_type::switching_protocols) {\n        throw websocket::exception(fmt::format(\n            \"Server responded with status {} instead of 101\",\n            static_cast<int>(resp->_status)));\n    }\n\n    // Validate Sec-WebSocket-Accept\n    auto expected_accept = sha1_base64(\n        fmt::format(\"{}{}\", _websocket_key, magic_key_suffix_client));\n\n    sstring actual_accept = resp->get_header(\"Sec-WebSocket-Accept\");\n    // Trim leading whitespace\n    size_t start = 0;\n    while (start < actual_accept.size() && !std::isgraph(actual_accept[start])) {\n        ++start;\n    }\n    // No need to trim trailing whitespace, as parse guarantees there are none.\n    if (start != 0) {\n        actual_accept = sstring(actual_accept.data() + start, actual_accept.size() - start);\n    }\n\n    if (actual_accept != sstring(expected_accept)) {\n        throw websocket::exception(fmt::format(\n            \"Invalid Sec-WebSocket-Accept: expected '{}', got '{}'\",\n            expected_accept, actual_accept));\n    }\n\n    websocket_logger.debug(\"WebSocket client handshake completed\");\n}\n\ntemplate <bool text_frame>\nfuture<> client_connection<text_frame>::handshake() {\n    co_await send_http_upgrade_request();\n    co_await read_http_upgrade_response();\n}\n\ntemplate <bool text_frame>\nfuture<> client_connection<text_frame>::process() {\n    co_await coroutine::all(\n        [this] () -> future<> {\n            co_await this->_handler(this->_input, this->_output).handle_exception([this] (std::exception_ptr e) -> future<> {\n                co_await this->_read_buf.close();\n                std::rethrow_exception(e);\n            });\n        },\n        [this] () -> future<> {\n            while (!this->_done) {\n                co_await this->read_one();\n            }\n        },\n        [this] () {\n            return this->response_loop();\n        }\n    );\n}\n\ntemplate <bool text_frame>\nfuture<> client<text_frame>::connect(socket_address addr, sstring resource, sstring host,\n                         sstring subprotocol, handler_t handler) {\n    auto fd = co_await seastar::connect(addr);\n    _conn = std::make_unique<client_connection<text_frame>>(std::move(fd),\n        std::move(resource), std::move(host),\n        std::move(subprotocol), std::move(handler));\n\n    co_await _conn->handshake();\n    (void)try_with_gate(_task_gate, [this] () -> future<> {\n        try {\n            co_await _conn->process();\n        } catch (...) {\n            websocket_logger.debug(\"WebSocket client processing failed: {}\", std::current_exception());\n        }\n    }).handle_exception_type([] (const gate_closed_exception&) {});\n}\n\ntemplate <bool text_frame>\nfuture<> client<text_frame>::connect(socket_address addr,\n                         shared_ptr<tls::certificate_credentials> creds,\n                         sstring resource, sstring host,\n                         sstring subprotocol, handler_t handler) {\n    auto fd = co_await tls::connect(creds, addr, tls::tls_options{.server_name = host});\n    this->_conn = std::make_unique<client_connection<text_frame>>(std::move(fd),\n        std::move(resource), std::move(host),\n        std::move(subprotocol), std::move(handler));\n\n    co_await _conn->handshake();\n    (void)try_with_gate(_task_gate, [this] () -> future<> {\n        try {\n            co_await _conn->process();\n        } catch (...) {\n            websocket_logger.debug(\"WebSocket client processing failed: {}\", std::current_exception());\n        }\n    }).handle_exception_type([] (const gate_closed_exception&) {});\n}\n\ntemplate <bool text_frame>\nfuture<> client<text_frame>::close() {\n    if (_conn) {\n        co_await _conn->close(true).handle_exception([] (auto) {});\n        _conn->shutdown_input();\n    }\n    co_await _task_gate.close();\n}\n\ntemplate class client_connection<false>;\ntemplate class client_connection<true>;\n\ntemplate class client<true>;\ntemplate class client<false>;\n\n}\n"
  },
  {
    "path": "src/websocket/common.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2024 ScyllaDB\n */\n\n#include <seastar/core/future.hh>\n#include <seastar/websocket/common.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n#include <gnutls/crypto.h>\n#include <gnutls/gnutls.h>\n#include <random>\n#include <seastar/websocket/parser.hh>\n\nnamespace seastar::experimental::websocket {\n\nlogger websocket_logger(\"websocket\");\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::handle_ping(temporary_buffer<char> buff) {\n    return send_data(opcodes::PONG, std::move(buff));\n}\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::handle_pong() {\n    // TODO\n    return make_ready_future<>();\n}\n\nstatic thread_local std::mt19937 masking_rng{std::random_device{}()};\n\nstatic uint32_t generate_masking_key() {\n    return masking_rng();\n}\n\nstatic void apply_mask(char* data, size_t len, uint32_t masking_key) {\n    char mask_bytes[4];\n    write_be<uint32_t>(mask_bytes, masking_key);\n    for (size_t i = 0; i < len; ++i) {\n        data[i] ^= mask_bytes[i % 4];\n    }\n}\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::send_data(opcodes opcode, temporary_buffer<char> buff) {\n    char header[14] = {'\\x80', 0}; // max: 2 + 8 (extended len) + 4 (mask key)\n    size_t header_size = 2;\n\n    header[0] += opcode;\n\n    if ((126 <= buff.size()) && (buff.size() <= std::numeric_limits<uint16_t>::max())) {\n        header[1] = 0x7E;\n        write_be<uint16_t>(header + 2, buff.size());\n        header_size += sizeof(uint16_t);\n    } else if (std::numeric_limits<uint16_t>::max() < buff.size()) {\n        header[1] = 0x7F;\n        write_be<uint64_t>(header + 2, buff.size());\n        header_size += sizeof(uint64_t);\n    } else {\n        header[1] = uint8_t(buff.size());\n    }\n\n    if constexpr (is_client) {\n        // RFC 6455 §5.3: client frames must be masked\n        header[1] |= 0x80; // set mask bit\n        uint32_t masking_key = generate_masking_key();\n        write_be<uint32_t>(header + header_size, masking_key);\n        header_size += sizeof(uint32_t);\n        apply_mask(buff.get_write(), buff.size(), masking_key);\n    }\n\n    co_await _write_buf.write(header, header_size);\n    co_await _write_buf.write(std::move(buff));\n    co_await _write_buf.flush();\n}\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::response_loop() {\n    return do_until([this] {return _done;}, [this] {\n        // FIXME: implement error handling\n        return _output_buffer.pop_eventually().then([this] (\n                temporary_buffer<char> buf) {\n            if (!buf) {\n                return make_ready_future<>();\n            }\n            return send_data(text_frame ? opcodes::TEXT : opcodes::BINARY, std::move(buf));\n        });\n    }).finally([this]() {\n        return _write_buf.close();\n    });\n}\n\ntemplate <bool is_client, bool text_frame>\nvoid basic_connection<is_client, text_frame>::shutdown_input() {\n    _fd.shutdown_input();\n}\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::close(bool send_close) {\n    if (_half_close) {\n        return make_ready_future<>();\n    }\n    _half_close = true;\n    return [this, send_close]() {\n        if (send_close) {\n            return send_data(opcodes::CLOSE, temporary_buffer<char>(0));\n        } else {\n            return make_ready_future<>();\n        }\n    }().finally([this] {\n        _done = true;\n        return when_all_succeed(_input.close(), _output.close()).discard_result().finally([this] {\n            _fd.shutdown_output();\n        });\n    });\n}\n\ntemplate <bool is_client, bool text_frame>\nfuture<> basic_connection<is_client, text_frame>::read_one() {\n    return _read_buf.consume(_websocket_parser).then([this] () mutable {\n        if (_websocket_parser.is_valid()) {\n            if (_half_close) {\n                // When the connection enters _half_closed state, just wait for the close to complete.\n                return make_ready_future();\n            }\n            // FIXME: implement error handling\n            switch(_websocket_parser.opcode()) {\n            // We do not distinguish between these 3 types.\n            case opcodes::CONTINUATION:\n            case opcodes::TEXT:\n            case opcodes::BINARY:\n                return _input_buffer.push_eventually(_websocket_parser.result());\n            case opcodes::CLOSE:\n                websocket_logger.debug(\"Received close frame.\");\n                // datatracker.ietf.org/doc/html/rfc6455#section-5.5.1\n                return close(true);\n            case opcodes::PING:\n                websocket_logger.debug(\"Received ping frame.\");\n                return handle_ping(_websocket_parser.result());\n            case opcodes::PONG:\n                websocket_logger.debug(\"Received pong frame.\");\n                return handle_pong();\n            default:\n                // Invalid - do nothing.\n                ;\n            }\n        } else if (_websocket_parser.eof()) {\n            return close(false);\n        }\n        websocket_logger.debug(\"Reading from socket has failed.\");\n        return close(true);\n    });\n}\n\nstd::string sha1_base64(std::string_view source) {\n    unsigned char hash[20];\n    SEASTAR_ASSERT(sizeof(hash) == gnutls_hash_get_len(GNUTLS_DIG_SHA1));\n    if (int ret = gnutls_hash_fast(GNUTLS_DIG_SHA1, source.data(), source.size(), hash);\n        ret != GNUTLS_E_SUCCESS) {\n        throw websocket::exception(fmt::format(\"gnutls_hash_fast: {}\", gnutls_strerror(ret)));\n    }\n    return encode_base64(std::string_view(reinterpret_cast<const char*>(hash), sizeof(hash)));\n}\n\nstd::string encode_base64(std::string_view source) {\n    gnutls_datum_t src_data{\n        .data = reinterpret_cast<uint8_t*>(const_cast<char*>(source.data())),\n        .size = static_cast<unsigned>(source.size())\n    };\n    gnutls_datum_t encoded_data;\n    if (int ret = gnutls_base64_encode2(&src_data, &encoded_data); ret != GNUTLS_E_SUCCESS) {\n        throw websocket::exception(fmt::format(\"gnutls_base64_encode2: {}\", gnutls_strerror(ret)));\n    }\n    auto free_encoded_data = defer([&] () noexcept { gnutls_free(encoded_data.data); });\n    // base64_encoded.data is \"unsigned char *\"\n    return std::string(reinterpret_cast<const char*>(encoded_data.data), encoded_data.size);\n}\n\ntemplate class basic_connection<true, false>;\ntemplate class basic_connection<true, true>;\ntemplate class basic_connection<false, false>;\ntemplate class basic_connection<false, true>;\n\n}\n"
  },
  {
    "path": "src/websocket/parser.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <seastar/websocket/parser.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar::experimental::websocket {\n\nopcodes websocket_parser::opcode() const {\n    if (_header) {\n        return opcodes(_header->opcode);\n    } else {\n        return opcodes::INVALID;\n    }\n}\n\nwebsocket_parser::buff_t websocket_parser::result() {\n    return std::move(_result);\n}\n\nfuture<websocket_parser::consumption_result_t> websocket_parser::operator()(\n        temporary_buffer<char> data) {\n    if (data.size() == 0) {\n        // EOF\n        _cstate = connection_state::closed;\n        return websocket_parser::stop(std::move(data));\n    }\n    if (_state == parsing_state::flags_and_payload_data) {\n        if (_buffer.length() + data.size() >= 2) {\n            // _buffer.length() is less than 2 when entering this if body due to how\n            // the rest of code is structured. The else branch will never increase\n            // _buffer.length() to >=2 and other paths to this condition will always\n            // have buffer cleared.\n            SEASTAR_ASSERT(_buffer.length() < 2);\n\n            size_t hlen = _buffer.length();\n            _buffer.append(data.get(), 2 - hlen);\n            data.trim_front(2 - hlen);\n            _header = std::make_unique<frame_header>(_buffer.data());\n            _buffer = {};\n\n            // https://datatracker.ietf.org/doc/html/rfc6455#section-5.1\n            // Validate mask bit: server requires masked, client requires unmasked.\n            if ((_require_mask && !_header->masked) ||\n                (!_require_mask && _header->masked) ||\n                // RSVX must be 0\n                (_header->rsv1 | _header->rsv2 | _header->rsv3) ||\n                // Opcode must be known.\n                (!_header->is_opcode_known())) {\n                _cstate = connection_state::error;\n                return websocket_parser::stop(std::move(data));\n            }\n            _state = parsing_state::payload_length_and_mask;\n        } else {\n            _buffer.append(data.get(), data.size());\n            return websocket_parser::dont_stop();\n        }\n    }\n    if (_state == parsing_state::payload_length_and_mask) {\n        size_t const required_bytes = _header->get_rest_of_header_length();\n        if (_buffer.length() + data.size() >= required_bytes) {\n            if (_buffer.length() < required_bytes) {\n                size_t hlen = _buffer.length();\n                _buffer.append(data.get(), required_bytes - hlen);\n                data.trim_front(required_bytes - hlen);\n            }\n            _payload_length = _header->length;\n            char const *input = _buffer.data();\n            if (_header->length == 126) {\n                _payload_length = consume_be<uint16_t>(input);\n            } else if (_header->length == 127) {\n                _payload_length = consume_be<uint64_t>(input);\n            }\n\n            if (_header->masked) {\n                _masking_key = consume_be<uint32_t>(input);\n            }\n            _buffer = {};\n            _state = parsing_state::payload;\n        } else {\n            _buffer.append(data.get(), data.size());\n            return websocket_parser::dont_stop();\n        }\n    }\n    if (_state == parsing_state::payload) {\n        if (data.size() < remaining_payload_length()) {\n            // data has insufficient data to complete the frame - consume data.size() bytes\n            if (_result.empty()) {\n                _result = temporary_buffer<char>(remaining_payload_length());\n                _consumed_payload_length = 0;\n            }\n            std::copy(data.begin(), data.end(), _result.get_write() + _consumed_payload_length);\n            _consumed_payload_length += data.size();\n            return websocket_parser::dont_stop();\n        } else {\n            // data has sufficient data to complete the frame - consume remaining_payload_length()\n            auto consumed_bytes = remaining_payload_length();\n            if (_result.empty()) {\n                // Try to avoid memory copies in case when network packets contain one or more full\n                // websocket frames.\n                if (consumed_bytes == data.size()) {\n                    _result = std::move(data);\n                    data = temporary_buffer<char>(0);\n                } else {\n                    _result = data.share();\n                    _result.trim(consumed_bytes);\n                    data.trim_front(consumed_bytes);\n                }\n            } else {\n                std::copy(data.begin(), data.begin() + consumed_bytes,\n                          _result.get_write() + _consumed_payload_length);\n                data.trim_front(consumed_bytes);\n            }\n            if (_header->masked) {\n                remove_mask(_result, _payload_length);\n            }\n            _consumed_payload_length = 0;\n            _state = parsing_state::flags_and_payload_data;\n            return websocket_parser::stop(std::move(data));\n        }\n    }\n    _cstate = connection_state::error;\n    return websocket_parser::stop(std::move(data));\n}\n\n}\n"
  },
  {
    "path": "src/websocket/server.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#include <seastar/websocket/server.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/log.hh>\n#include <seastar/http/request.hh>\n\nnamespace seastar::experimental::websocket {\n\nusing namespace std::string_view_literals;\n\nconstexpr auto magic_key_suffix = \"258EAFA5-E914-47DA-95CA-C5AB0DC85B11\"sv;\nconstexpr auto http_upgrade_reply_template =\n    \"HTTP/1.1 101 Switching Protocols\\r\\n\"\n    \"Upgrade: websocket\\r\\n\"\n    \"Connection: Upgrade\\r\\n\"\n    \"Sec-WebSocket-Version: 13\\r\\n\"\n    \"Sec-WebSocket-Accept: \"sv;\n\nvoid server::listen(socket_address addr, listen_options lo) {\n    _listeners.push_back(seastar::listen(addr, lo));\n    accept(_listeners.back());\n}\nvoid server::listen(socket_address addr) {\n    listen_options lo;\n    lo.reuse_address = true;\n    return listen(addr, lo);\n}\n\nvoid server::accept(server_socket &listener) {\n    (void)try_with_gate(_task_gate, [this, &listener]() {\n        return repeat([this, &listener]() {\n            return accept_one(listener);\n        });\n    }).handle_exception_type([](const gate_closed_exception &e) {});\n}\n\nfuture<stop_iteration> server::accept_one(server_socket &listener) {\n    return listener.accept().then([this](accept_result ar) {\n        auto conn = std::make_unique<server_connection>(*this, std::move(ar.connection));\n        (void)try_with_gate(_task_gate, [conn = std::move(conn)]() mutable {\n            return conn->process().finally([conn = std::move(conn)] {\n                websocket_logger.debug(\"Connection is finished\");\n            });\n        }).handle_exception_type([](const gate_closed_exception &e) {});\n        return make_ready_future<stop_iteration>(stop_iteration::no);\n    }).handle_exception_type([](const std::system_error &e) {\n        // We expect a ECONNABORTED when server::stop is called,\n        // no point in warning about that.\n        if (e.code().value() != ECONNABORTED) {\n            websocket_logger.error(\"accept failed: {}\", e);\n        }\n        return make_ready_future<stop_iteration>(stop_iteration::yes);\n    }).handle_exception([](std::exception_ptr ex) {\n        websocket_logger.info(\"accept failed: {}\", ex);\n        return make_ready_future<stop_iteration>(stop_iteration::yes);\n    });\n}\n\nfuture<> server::stop() {\n    for (auto&& l : _listeners) {\n        l.abort_accept();\n    }\n\n    for (auto&& c : _connections) {\n        c.shutdown_input();\n    }\n\n    return _task_gate.close().finally([this] {\n        return parallel_for_each(_connections, [] (server_connection& conn) {\n            return conn.close(true).handle_exception([] (auto ignored) {});\n        });\n    });\n}\n\nserver_connection::~server_connection() {\n    _server._connections.erase(_server._connections.iterator_to(*this));\n}\n\nvoid server_connection::on_new_connection() {\n    _server._connections.push_back(*this);\n}\n\nfuture<> server_connection::process() {\n    return when_all_succeed(read_loop(), response_loop()).discard_result().handle_exception([] (const std::exception_ptr& e) {\n        websocket_logger.debug(\"Processing failed: {}\", e);\n    });\n}\n\nfuture<> server_connection::read_http_upgrade_request() {\n    _http_parser.init();\n    co_await _read_buf.consume(_http_parser);\n\n    if (_http_parser.eof()) {\n        _done = true;\n        co_return;\n    }\n    std::unique_ptr<http::request> req = _http_parser.get_parsed_request();\n    if (_http_parser.failed()) {\n        throw websocket::exception(\"Incorrect upgrade request\");\n    }\n\n    sstring upgrade_header = req->get_header(\"Upgrade\");\n    if (upgrade_header != \"websocket\") {\n        throw websocket::exception(\"Upgrade header missing\");\n    }\n\n    sstring subprotocol = req->get_header(\"Sec-WebSocket-Protocol\");\n\n    if (!_server.is_handler_registered(subprotocol)) {\n        throw websocket::exception(\"Subprotocol not supported.\");\n    }\n    this->_handler = this->_server._handlers[subprotocol];\n    this->_subprotocol = subprotocol;\n    websocket_logger.debug(\"Sec-WebSocket-Protocol: {}\", subprotocol);\n\n    sstring sec_key = req->get_header(\"Sec-Websocket-Key\");\n    sstring sec_version = req->get_header(\"Sec-Websocket-Version\");\n\n    auto sha1_input = fmt::format(\"{}{}\", sec_key, magic_key_suffix);\n\n    websocket_logger.debug(\"Sec-Websocket-Key: {}, Sec-Websocket-Version: {}\", sec_key, sec_version);\n\n    std::string sha1_output = sha1_base64(sha1_input);\n    websocket_logger.debug(\"SHA1 output: {} of size {}\", sha1_output, sha1_output.size());\n\n    co_await _write_buf.write(http_upgrade_reply_template.data(), http_upgrade_reply_template.size());\n    co_await _write_buf.write(sha1_output);\n    if (!_subprotocol.empty()) {\n        co_await _write_buf.write(\"\\r\\nSec-WebSocket-Protocol: \", 26);\n        co_await _write_buf.write(_subprotocol);\n    }\n    co_await _write_buf.write(\"\\r\\n\\r\\n\", 4);\n    co_await _write_buf.flush();\n}\n\nfuture<> server_connection::read_loop() {\n    return read_http_upgrade_request().then([this] {\n        return when_all_succeed(\n            _handler(_input, _output).handle_exception([this] (std::exception_ptr e) mutable {\n                return _read_buf.close().then([e = std::move(e)] () mutable {\n                    return make_exception_future<>(std::move(e));\n                });\n            }),\n            do_until([this] {return _done;}, [this] {return read_one();})\n        ).discard_result();\n    }).finally([this] {\n        return _read_buf.close();\n    });\n}\n\nbool server::is_handler_registered(std::string const& name) {\n    return _handlers.find(name) != _handlers.end();\n}\n\nvoid server::register_handler(const std::string& name, handler_t handler) {\n    _handlers[name] = handler;\n}\n\n}\n"
  },
  {
    "path": "test.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\nimport argparse\nimport subprocess\nimport seastar_cmake\n\nif __name__ == \"__main__\":\n\n    parser = argparse.ArgumentParser(description=\"Seastar test runner\")\n    parser.add_argument('--fast',  action=\"store_true\", help=\"Run only fast tests\")\n    parser.add_argument('--name',  action=\"store\", help=\"Run only test whose name contains given string\")\n    parser.add_argument('--mode', choices=seastar_cmake.SUPPORTED_MODES, help=\"Run only tests for given build mode\")\n    parser.add_argument('--build-root', action='store', default=seastar_cmake.DEFAULT_BUILD_ROOT, type=str,\n                        help=\"The name of the build root build directoy: \"\n                        \"using a different name allows multiple configurations to co-exist in the same repository\")\n    parser.add_argument('--timeout', action=\"store\",default=\"300\",type=int, help=\"timeout value for test execution\")\n    parser.add_argument('--jenkins', action=\"store\",help=\"jenkins output file prefix\")\n    parser.add_argument('--smp', '-c', action=\"store\",default='2',type=int,help=\"Number of threads for multi-core tests\")\n    parser.add_argument('--verbose', '-v', action = 'store_true', default = False,\n                        help = 'Verbose reporting')\n    parser.add_argument('--offline', action=\"store_true\", default = False,\n                        help=\"Disable tests accessing internet\")\n    parser.add_argument('ctest_forward', nargs='*', help=\"These parameters will be passed directly to ctest\")\n    args = parser.parse_args()\n\n    MODES = [args.mode] if args.mode else seastar_cmake.SUPPORTED_MODES\n\n    def run_tests(mode):\n        BUILD_PATH = seastar_cmake.build_path(mode, args.build_root)\n\n        # For convenience.\n        tr = seastar_cmake.translate_arg\n\n        TRANSLATED_CMAKE_ARGS = [\n            tr(args.timeout, 'TEST_TIMEOUT'),\n            tr(args.fast, 'EXECUTE_ONLY_FAST_TESTS'),\n            tr(args.smp, 'UNIT_TEST_SMP'),\n            tr(not args.offline, 'ENABLE_TESTS_ACCESSING_INTERNET'),\n            tr(args.jenkins, 'JENKINS', value_when_none=''),\n        ]\n\n        # Modify the existing build by pointing to the build directory.\n        CMAKE_ARGS = ['cmake', BUILD_PATH] + TRANSLATED_CMAKE_ARGS\n        print(CMAKE_ARGS)\n        subprocess.check_call(CMAKE_ARGS, shell=False, cwd=seastar_cmake.ROOT_PATH)\n\n        TRANSLATED_CTEST_ARGS = ['--output-on-failure']\n        if args.verbose:\n            TRANSLATED_CTEST_ARGS += ['--verbose']\n        if args.name:\n            TRANSLATED_CTEST_ARGS += ['-R', args.name]\n\n        CTEST_ARGS = ['ctest', BUILD_PATH] + TRANSLATED_CTEST_ARGS + args.ctest_forward\n        print(CTEST_ARGS)\n        subprocess.check_call(CTEST_ARGS, shell=False, cwd=BUILD_PATH)\n\n    for mode in MODES:\n        run_tests(mode)\n"
  },
  {
    "path": "tests/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\nfunction (seastar_jenkins_arguments test_name var)\n  string (TOLOWER ${CMAKE_BUILD_TYPE} mode)\n  set (output_file \"${CMAKE_SOURCE_DIR}/${Seastar_JENKINS}.${mode}.${test_name}_test.boost.xml\")\n\n  set (${var}\n    --output_format=XML\n    --log_level=all\n    --report_level=no\n    --log_sink=${output_file}\n    PARENT_SCOPE)\nendfunction ()\n\n# Performance tests.\nadd_subdirectory (perf)\n\n# Unit tests.\nadd_subdirectory (unit)\n\n# Fuzz tests.\nadd_subdirectory (fuzz)\n"
  },
  {
    "path": "tests/fuzz/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2026 Redpanda Data.\n#\n\n#\n# sstring fuzzer (only when using custom sstring, not std::string)\n#\nif (Seastar_SSTRING)\n  seastar_add_test (sstring\n    KIND FUZZ\n    SOURCES sstring_fuzz.cc\n    RUN_ARGS -max_len=100)\nendif ()\n"
  },
  {
    "path": "tests/fuzz/sstring_fuzz.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2026 Redpanda Data.\n */\n\n/**\n * Differential Fuzzer for seastar::sstring\n *\n * This fuzzer uses std::string as an oracle to find correctness bugs in sstring.\n * The fuzzer interprets the random input as a sequence of operations (opcodes),\n * applying each operation to both an sstring and a std::string in parallel.\n * After each operation, we verify that both strings remain equivalent.\n *\n * Architecture:\n * - NUM_SLOTS string pairs (sstring + std::string) for multi-string operations\n * - Each byte from fuzz input is interpreted as an opcode + parameters\n * - FuzzReader consumes bytes and generates deterministic test data from seeds\n * - Operations cover construction, mutation, search, comparison, and iteration\n *\n * The fuzzer is designed to exercise both the internal (small string optimization)\n * and external (heap-allocated) storage paths in sstring by using a mix of small\n * and large strings generated from the fuzz input.\n */\n\n#include <seastar/core/sstring.hh>\n#include <seastar/util/assert.hh>\n#include <cstddef>\n#include <cstdint>\n#include <cstring>\n#include <optional>\n#include <random>\n#include <string>\n#include <string_view>\n#include <type_traits>\n#include <typeindex>\n\nusing seastar::sstring;\n\n// Helpers for C++20 compatibility (std::string::contains added in C++23)\nbool contains(const std::string& str, std::string_view sv) {\n    return str.find(sv) != std::string::npos;\n}\n\nbool contains(const std::string& str, char c) {\n    return str.find(c) != std::string::npos;\n}\n\n// Command opcodes for the interpreted fuzzer\nenum class Op : uint8_t {\n    // Construction / Assignment\n    ASSIGN_FROM_DATA = 0,   // Assign from inline data\n    ASSIGN_COPY,            // Copy assignment from other slot\n    ASSIGN_MOVE,            // Move assignment from other slot\n    CLEAR,                  // Clear the string\n\n    // Append operations\n    APPEND_DATA,            // Append inline data\n    APPEND_SSTRING,         // Append from another slot using +=\n    APPEND_CHAR_N,          // Append N copies of character using resize\n\n    // Size operations\n    CHECK_SIZE,             // Verify size matches\n    CHECK_EMPTY,            // Verify empty matches\n    RESIZE,                 // Resize to N with fill char\n    RESIZE_SHRINK,          // Resize smaller (relative to current size)\n\n    // Search operations\n    FIND_CHAR,              // Find character\n    FIND_CHAR_POS,          // Find character starting at position\n    FIND_SUBSTR,            // Find substring\n    FIND_LAST_OF,           // Find last occurrence of character\n\n    // Substring operations\n    SUBSTR,                 // Extract substring\n    ERASE,                  // Erase range using iterators\n\n    // Comparison\n    COMPARE,                // Compare with other slot\n    COMPARE_SUBSTR,         // Compare substring\n\n    // Predicates\n    STARTS_WITH,            // Check prefix\n    ENDS_WITH,              // Check suffix\n    CONTAINS,               // Check contains\n\n    // Access\n    AT,                     // Bounds-checked access\n    FRONT,                  // Access front element\n    BACK,                   // Access back element\n\n    // Multi-string operations\n    SWAP,                   // Swap two slots\n    CONCAT,                 // Concatenate two slots\n\n    // Length operations\n    LENGTH,                 // Check length() == size()\n\n    // Equality/comparison operators\n    EQUAL,                  // operator==\n    NOT_EQUAL,              // operator!=\n    LESS_THAN,              // operator<\n\n    // Additional construction\n    CONSTRUCT_FILL,         // Construct with (size, char)\n\n    // Additional modification\n    REPLACE,                // replace(pos, n1, s, n2)\n    INSERT,                 // insert(p, beg, end)\n\n    // Additional access\n    SUBSCRIPT,              // operator[] unchecked access\n\n    // Additional predicates (char variants)\n    STARTS_WITH_CHAR,       // starts_with(char)\n    ENDS_WITH_CHAR,         // ends_with(char)\n    CONTAINS_CHAR,          // contains(char)\n\n    // Additional find operations\n    FIND_SSTRING,           // find(sstring, pos)\n    FIND_LAST_OF_POS,       // find_last_of(char, pos)\n\n    // Additional comparisons\n    GREATER_THAN,           // operator>\n    LESS_EQUAL,             // operator<=\n    GREATER_EQUAL,          // operator>=\n\n    // Conversion/access\n    TO_STD_STRING,          // operator std::string()\n    C_STR,                  // c_str() returns correct data\n    DATA,                   // data() returns correct data\n\n    // Iterator operations\n    ITERATOR_DISTANCE,      // end() - begin() == size()\n    REVERSE_ITERATE,        // iterate backwards and check\n\n    // Additional construction\n    CONSTRUCT_ITERATOR,     // construct from iterators\n\n    // Raw data operations (bytes read directly from fuzz input)\n    ASSIGN_RAW,             // Assign from raw fuzz input bytes\n    APPEND_RAW,             // Append raw fuzz input bytes\n\n    // Exception testing (out-of-bounds operations)\n    AT_THROWING,            // at() with invalid position (must throw)\n    SUBSTR_THROWING,        // substr() with invalid position (must throw)\n    COMPARE_SUBSTR_THROWING, // compare() with invalid position (must throw)\n    REPLACE_THROWING,       // replace() with invalid position (must throw)\n    INSERT_THROWING,        // insert() with invalid iterator (must throw)\n\n    OP_COUNT\n};\n\n// Number of string slots available for operations\nconstexpr size_t NUM_SLOTS = 4;\n\n// Maximum size for any single string operation\n// Making this too large reduces the effectiveness of the fuzzer at finding\n// certain types of bugs, since most strings will be long so a bug like\n// \"the final byte is skipped in ==\" is hard to catch since we are unlikely\n// to reach that scenario with a longer string. However, we still want it\n// be large enough to trigger all the code paths such as heap allocations,\n// so this should be more than 2x the SSO limit.\nconstexpr size_t MAX_STRING_SIZE = 50;\n\n// Knuth's MMIX LCG for deterministic data generation\n// a = 6364136223846793005, c = 1442695040888963407, m = 2^64\nusing Rng = std::linear_congruential_engine<uint64_t, 6364136223846793005ULL, 1442695040888963407ULL, 0ULL>;\n\n// Reader for consuming bytes from the fuzz input\nclass FuzzReader {\n    const uint8_t* data_;\n    size_t remaining_;\n\npublic:\n    FuzzReader(const uint8_t* data, size_t size)\n        : data_(data), remaining_(size) {}\n\n    bool empty() const { return remaining_ == 0; }\n\n    uint8_t read_u8() {\n        if (remaining_ == 0) return 0;\n        remaining_--;\n        return *data_++;\n    }\n\n    uint16_t read_u16() {\n        uint16_t lo = read_u8();\n        uint16_t hi = read_u8();\n        return lo | (hi << 8);\n    }\n\n    uint32_t read_u32() {\n        uint32_t b0 = read_u8();\n        uint32_t b1 = read_u8();\n        uint32_t b2 = read_u8();\n        uint32_t b3 = read_u8();\n        return b0 | (b1 << 8) | (b2 << 16) | (b3 << 24);\n    }\n\n    uint64_t read_u64() {\n        uint64_t lo = read_u32();\n        uint64_t hi = read_u32();\n        return lo | (hi << 32);\n    }\n\n    size_t read_size(size_t max) {\n        if (max == 0) return 0;\n        if (max <= 255) {\n            return read_u8() % (max + 1);\n        }\n        return read_u16() % (max + 1);\n    }\n\n    // Generate deterministic data from size and seed\n    // Low entropy mode: characters alternate between two values\n    // High entropy mode: full byte range\n    std::string read_generated_data(size_t max_len) {\n        size_t len = read_size(max_len);\n        uint64_t seed = read_u64();\n        int mode = seed % 3;  // 0 = zeros, 1 = low entropy, 2 = high entropy\n\n        std::string result;\n        result.reserve(len);\n\n        if (mode == 0) {\n            result.resize(len, '\\0');\n        } else if (mode == 1) {\n            // Low entropy: only 2 possible character values\n            // Helps exercise comparisons\n            Rng rng(seed | 1);  // Ensure seed is never 0\n            for (size_t i = 0; i < len; i++) {\n                result.push_back((rng() & 1) ? 'a' : 'b');\n            }\n        } else {\n            // High entropy: full byte range\n            Rng rng(seed | 1);  // Ensure seed is never 0\n            for (size_t i = 0; i < len; i++) {\n                result.push_back(static_cast<char>(rng()));\n            }\n        }\n        return result;\n    }\n\n    char read_char() {\n        return static_cast<char>(read_u8());\n    }\n\n    // Read raw bytes directly from fuzz input\n    std::string read_raw_data(size_t max_len) {\n        size_t len = read_size(max_len);\n        std::string result;\n        result.reserve(len);\n        for (size_t i = 0; i < len; i++) {\n            result.push_back(static_cast<char>(read_u8()));\n        }\n        return result;\n    }\n};\n\n// Verify that sstring and std::string are equivalent\nvoid verify_equal(const sstring& ss, const std::string& rs) {\n    SEASTAR_ASSERT(ss.size() == rs.size());\n    SEASTAR_ASSERT(ss.empty() == rs.empty());\n    SEASTAR_ASSERT(ss.size() == 0 || std::memcmp(ss.data(), rs.data(), ss.size()) == 0);\n}\n\n// Verify both implementations throw the same exception type\n// Both MUST throw - this is for testing operations that should fail\n// Op is a generic lambda that works on both sstring and std::string\ntemplate<typename Op>\nvoid both_throw(sstring& ss, std::string& rs, Op&& op) {\n    std::optional<std::type_index> ss_exception_type;\n    std::optional<std::type_index> rs_exception_type;\n\n    try {\n        op(ss);\n    } catch (const std::exception& e) {\n        ss_exception_type = std::type_index(typeid(e));\n    } catch (...) {\n        ss_exception_type = std::type_index(typeid(void));\n    }\n\n    try {\n        op(rs);\n    } catch (const std::exception& e) {\n        rs_exception_type = std::type_index(typeid(e));\n    } catch (...) {\n        rs_exception_type = std::type_index(typeid(void));\n    }\n\n    // Both must throw\n    SEASTAR_ASSERT(ss_exception_type.has_value());\n    SEASTAR_ASSERT(rs_exception_type.has_value());\n    // Exception types must match\n    SEASTAR_ASSERT(ss_exception_type == rs_exception_type);\n}\n\n// Execute the fuzzer program\nvoid execute_program(FuzzReader& reader) {\n    // The string slots under test\n    sstring slots[NUM_SLOTS];\n    std::string rss[NUM_SLOTS];\n\n    while (!reader.empty()) {\n        Op op = static_cast<Op>(reader.read_u8() % static_cast<uint8_t>(Op::OP_COUNT));\n        uint8_t slot = reader.read_u8() % NUM_SLOTS;\n\n        sstring& ss = slots[slot];\n        std::string& rs = rss[slot];\n\n        switch (op) {\n            case Op::ASSIGN_FROM_DATA: {\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                ss = sstring(data.data(), data.size());\n                rs = std::move(data);\n                break;\n            }\n\n            case Op::ASSIGN_COPY: {\n                uint8_t src = reader.read_u8() % NUM_SLOTS;\n                ss = slots[src];\n                rs = rss[src];\n                break;\n            }\n\n            case Op::ASSIGN_MOVE: {\n                uint8_t src = reader.read_u8() % NUM_SLOTS;\n                if (src != slot) {\n                    ss = std::move(slots[src]);\n                    rs = std::move(rss[src]);\n                    // Reset moved-from state to known empty\n                    slots[src] = sstring();\n                    rss[src] = std::string();\n                }\n                break;\n            }\n\n            case Op::CLEAR: {\n                ss = sstring();\n                rs.clear();\n                break;\n            }\n\n            case Op::APPEND_DATA: {\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                if (ss.size() + data.size() <= MAX_STRING_SIZE) {\n                    ss.append(data.data(), data.size());\n                    rs.append(data.data(), data.size());\n                }\n                break;\n            }\n\n            case Op::APPEND_SSTRING: {\n                uint8_t src = reader.read_u8() % NUM_SLOTS;\n                if (ss.size() + slots[src].size() <= MAX_STRING_SIZE) {\n                    ss += slots[src];\n                    rs += rss[src];\n                }\n                break;\n            }\n\n            case Op::APPEND_CHAR_N: {\n                char c = reader.read_char();\n                size_t max_count = MAX_STRING_SIZE - ss.size();\n                size_t count = reader.read_size(max_count);\n                // sstring doesn't have append(count, char), use resize instead\n                size_t old_size = ss.size();\n                ss.resize(old_size + count, c);\n                rs.resize(old_size + count, c);\n                break;\n            }\n\n            case Op::CHECK_SIZE: {\n                SEASTAR_ASSERT(ss.size() == rs.size());\n                break;\n            }\n\n            case Op::CHECK_EMPTY: {\n                SEASTAR_ASSERT(ss.empty() == rs.empty());\n                break;\n            }\n\n            case Op::RESIZE: {\n                size_t new_size = reader.read_size(MAX_STRING_SIZE);\n                if (new_size <= MAX_STRING_SIZE) {\n                    char fill = reader.read_char();\n                    ss.resize(new_size, fill);\n                    rs.resize(new_size, fill);\n                }\n                break;\n            }\n\n            case Op::RESIZE_SHRINK: {\n                size_t new_size = reader.read_size(ss.size());\n                ss.resize(new_size);\n                rs.resize(new_size);\n                break;\n            }\n\n            case Op::FIND_CHAR: {\n                char c = reader.read_char();\n                size_t r1 = ss.find(c);\n                size_t r2 = rs.find(c);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::FIND_CHAR_POS: {\n                char c = reader.read_char();\n                size_t pos = reader.read_size(ss.size() + 10);  // Allow some out of bounds\n                size_t r1 = ss.find(c, pos);\n                size_t r2 = rs.find(c, pos);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::FIND_SUBSTR: {\n                uint8_t src = reader.read_u8() % NUM_SLOTS;\n                std::string_view sv(rss[src]);\n                size_t r1 = ss.find(sv);\n                size_t r2 = rs.find(rss[src]);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::FIND_LAST_OF: {\n                char c = reader.read_char();\n                size_t r1 = ss.find_last_of(c);\n                size_t r2 = rs.find_last_of(c);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::SUBSTR: {\n                size_t pos = reader.read_size(ss.size());\n                size_t max_len = ss.size() >= pos ? ss.size() - pos : 0;\n                size_t len = reader.read_size(max_len);\n                sstring sub1 = ss.substr(pos, len);\n                std::string sub2 = rs.substr(pos, len);\n                verify_equal(sub1, sub2);\n                break;\n            }\n\n            case Op::ERASE: {\n                if (!ss.empty()) {\n                    size_t pos = reader.read_size(ss.size() - 1);\n                    size_t max_len = ss.size() - pos;\n                    size_t len = reader.read_size(max_len);\n                    // sstring::erase uses iterators\n                    ss.erase(ss.begin() + pos, ss.begin() + pos + len);\n                    rs.erase(pos, len);\n                }\n                break;\n            }\n\n            case Op::COMPARE: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                // Compare sign only, not magnitude\n                std::string_view sv(rss[other]);\n                int r1 = ss.compare(sv);\n                int r2 = rs.compare(rss[other]);\n                SEASTAR_ASSERT((r1 < 0) == (r2 < 0) && (r1 == 0) == (r2 == 0));\n                break;\n            }\n\n            case Op::COMPARE_SUBSTR: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                size_t pos = reader.read_size(ss.size());\n                size_t max_len = ss.size() >= pos ? ss.size() - pos : 0;\n                size_t len = reader.read_size(max_len);\n                std::string_view sv(rss[other]);\n                int r1 = ss.compare(pos, len, sv);\n                int r2 = rs.compare(pos, len, rss[other]);\n                SEASTAR_ASSERT((r1 < 0) == (r2 < 0) && (r1 == 0) == (r2 == 0));\n                break;\n            }\n\n            case Op::STARTS_WITH: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                std::string_view sv(rss[other]);\n                bool r1 = ss.starts_with(sv);\n                bool r2 = rs.starts_with(sv);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::ENDS_WITH: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                std::string_view sv(rss[other]);\n                bool r1 = ss.ends_with(sv);\n                bool r2 = rs.ends_with(sv);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::CONTAINS: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                std::string_view sv(rss[other]);\n                bool r1 = ss.contains(sv);\n                bool r2 = contains(rs, sv);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::AT: {\n                if (!ss.empty()) {\n                    size_t pos = reader.read_size(ss.size() - 1);\n                    char c1 = ss.at(pos);\n                    char c2 = rs.at(pos);\n                    SEASTAR_ASSERT(c1 == c2);\n                }\n                break;\n            }\n\n            case Op::FRONT: {\n                if (!ss.empty()) {\n                    SEASTAR_ASSERT(ss.front() == rs.front());\n                }\n                break;\n            }\n\n            case Op::BACK: {\n                if (!ss.empty()) {\n                    SEASTAR_ASSERT(ss.back() == rs.back());\n                }\n                break;\n            }\n\n            case Op::SWAP: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                ss.swap(slots[other]);\n                rs.swap(rss[other]);\n                break;\n            }\n\n            case Op::CONCAT: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                if (ss.size() + slots[other].size() <= MAX_STRING_SIZE) {\n                    ss = ss + slots[other];\n                    rs = rs + rss[other];\n                }\n                break;\n            }\n\n            case Op::LENGTH: {\n                // Verify length() == size()\n                SEASTAR_ASSERT(ss.length() == ss.size());\n                SEASTAR_ASSERT(ss.length() == rs.length());\n                break;\n            }\n\n            case Op::EQUAL: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                bool r1 = (ss == slots[other]);\n                bool r2 = (rs == rss[other]);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::NOT_EQUAL: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                bool r1 = (ss != slots[other]);\n                bool r2 = (rs != rss[other]);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::LESS_THAN: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                bool r1 = (ss < slots[other]);\n                bool r2 = (rs < rss[other]);\n                SEASTAR_ASSERT(r1 == r2);\n                break;\n            }\n\n            case Op::CONSTRUCT_FILL: {\n                size_t len = reader.read_size(MAX_STRING_SIZE);\n                char c = reader.read_char();\n                ss = sstring(len, c);\n                rs = std::string(len, c);\n                break;\n            }\n\n            case Op::REPLACE: {\n                size_t pos = reader.read_size(ss.size());\n                size_t max_n1 = ss.size() >= pos ? ss.size() - pos : 0;\n                size_t n1 = reader.read_size(max_n1);\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                size_t n2 = data.size();\n                // Check resulting size won't exceed max\n                if (ss.size() - n1 + n2 <= MAX_STRING_SIZE) {\n                    ss.replace(pos, n1, data.data(), n2);\n                    rs.replace(pos, n1, data.data(), n2);\n                }\n                break;\n            }\n\n            case Op::INSERT: {\n                size_t pos = reader.read_size(ss.size());\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                if (ss.size() + data.size() <= MAX_STRING_SIZE) {\n                    ss.insert(ss.begin() + pos, data.begin(), data.end());\n                    rs.insert(rs.begin() + pos, data.begin(), data.end());\n                }\n                break;\n            }\n\n            case Op::SUBSCRIPT: {\n                if (!ss.empty()) {\n                    size_t pos = reader.read_size(ss.size() - 1);\n                    SEASTAR_ASSERT(ss[pos] == rs[pos]);\n                }\n                break;\n            }\n\n            case Op::STARTS_WITH_CHAR: {\n                char c = reader.read_char();\n                SEASTAR_ASSERT(ss.starts_with(c) == rs.starts_with(c));\n                break;\n            }\n\n            case Op::ENDS_WITH_CHAR: {\n                char c = reader.read_char();\n                SEASTAR_ASSERT(ss.ends_with(c) == rs.ends_with(c));\n                break;\n            }\n\n            case Op::CONTAINS_CHAR: {\n                char c = reader.read_char();\n                SEASTAR_ASSERT(ss.contains(c) == contains(rs, c));\n                break;\n            }\n\n            case Op::FIND_SSTRING: {\n                uint8_t src = reader.read_u8() % NUM_SLOTS;\n                size_t pos = reader.read_size(ss.size() + 10);\n                SEASTAR_ASSERT(ss.find(slots[src], pos) == rs.find(rss[src], pos));\n                break;\n            }\n\n            case Op::FIND_LAST_OF_POS: {\n                char c = reader.read_char();\n                size_t pos = reader.read_size(ss.size() + 10);\n                SEASTAR_ASSERT(ss.find_last_of(c, pos) == rs.find_last_of(c, pos));\n                break;\n            }\n\n            case Op::GREATER_THAN: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                SEASTAR_ASSERT((ss > slots[other]) == (rs > rss[other]));\n                break;\n            }\n\n            case Op::LESS_EQUAL: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                SEASTAR_ASSERT((ss <= slots[other]) == (rs <= rss[other]));\n                break;\n            }\n\n            case Op::GREATER_EQUAL: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                SEASTAR_ASSERT((ss >= slots[other]) == (rs >= rss[other]));\n                break;\n            }\n\n            case Op::TO_STD_STRING: {\n                std::string converted = static_cast<std::string>(ss);\n                SEASTAR_ASSERT(converted == rs);\n                break;\n            }\n\n            case Op::C_STR: {\n                // Use strcmp to verify null termination is correct\n                SEASTAR_ASSERT(std::strcmp(ss.c_str(), rs.c_str()) == 0);\n                break;\n            }\n\n            case Op::DATA: {\n                SEASTAR_ASSERT(ss.size() == rs.size());\n                SEASTAR_ASSERT(std::memcmp(ss.data(), rs.data(), ss.size()) == 0);\n                break;\n            }\n\n            case Op::ITERATOR_DISTANCE: {\n                auto dist = ss.end() - ss.begin();\n                SEASTAR_ASSERT(static_cast<size_t>(dist) == ss.size());\n                SEASTAR_ASSERT(static_cast<size_t>(dist) == rs.size());\n                break;\n            }\n\n            case Op::REVERSE_ITERATE: {\n                // Iterate backwards and compare\n                auto it1 = ss.end();\n                auto it2 = rs.end();\n                while (it1 != ss.begin()) {\n                    --it1;\n                    --it2;\n                    SEASTAR_ASSERT(*it1 == *it2);\n                }\n                break;\n            }\n\n            case Op::CONSTRUCT_ITERATOR: {\n                // Construct from rs's iterators\n                if (rs.size() <= MAX_STRING_SIZE) {\n                    ss = sstring(rs.begin(), rs.end());\n                    // rs stays the same, s should now equal rs\n                }\n                break;\n            }\n\n            case Op::ASSIGN_RAW: {\n                auto data = reader.read_raw_data(MAX_STRING_SIZE);\n                ss = sstring(data.data(), data.size());\n                rs = std::move(data);\n                break;\n            }\n\n            case Op::APPEND_RAW: {\n                auto data = reader.read_raw_data(MAX_STRING_SIZE);\n                if (ss.size() + data.size() <= MAX_STRING_SIZE) {\n                    ss.append(data.data(), data.size());\n                    rs.append(data.data(), data.size());\n                }\n                break;\n            }\n\n            case Op::AT_THROWING: {\n                // Force position >= size() to guarantee exception\n                size_t pos = ss.size() + reader.read_size(10);\n                both_throw(ss, rs, [pos](auto& s) { (void)s.at(pos); });\n                break;\n            }\n\n            case Op::SUBSTR_THROWING: {\n                // Force from > size() to guarantee exception\n                size_t from = ss.size() + 1 + reader.read_size(10);\n                size_t len = reader.read_size(MAX_STRING_SIZE);\n                both_throw(ss, rs, [from, len](auto& s) { (void)s.substr(from, len); });\n                break;\n            }\n\n            case Op::COMPARE_SUBSTR_THROWING: {\n                uint8_t other = reader.read_u8() % NUM_SLOTS;\n                // Force pos > size() to guarantee exception\n                size_t pos = ss.size() + 1 + reader.read_size(10);\n                size_t len = reader.read_size(MAX_STRING_SIZE);\n                // Work around libstdc++ bug: string_view overload of compare()\n                // has incorrect noexcept, causing terminate() instead of throwing.\n                // https://gcc.gnu.org/bugzilla/show_bug.cgi?id=123991\n                both_throw(ss, rs, [pos, len, &slots, &rss, other](auto& s) {\n                    if constexpr (std::is_same_v<std::decay_t<decltype(s)>, sstring>) {\n                        (void)s.compare(pos, len, std::string_view(slots[other]));\n                    } else {\n                        (void)s.compare(pos, len, rss[other]);\n                    }\n                });\n                break;\n            }\n\n            case Op::REPLACE_THROWING: {\n                // Force pos > size() to guarantee exception\n                size_t pos = ss.size() + 1 + reader.read_size(10);\n                size_t n1 = reader.read_size(MAX_STRING_SIZE);\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                both_throw(ss, rs, [pos, n1, &data](auto& s) {\n                    s.replace(pos, n1, data.data(), data.size());\n                });\n                break;\n            }\n\n            case Op::INSERT_THROWING: {\n                // Force position > size() to guarantee exception\n                size_t pos = ss.size() + 1 + reader.read_size(10);\n                auto data = reader.read_generated_data(MAX_STRING_SIZE);\n                both_throw(ss, rs, [pos, &data](auto& s) {\n                    s.insert(s.begin() + pos, data.begin(), data.end());\n                });\n                break;\n            }\n\n            default:\n                SEASTAR_ASSERT(false);  // Invalid op\n            }\n\n        // Verify consistency after every operation\n        verify_equal(ss, rs);\n    }\n\n    // Final verification of all slots\n    for (size_t i = 0; i < NUM_SLOTS; i++) {\n        verify_equal(slots[i], rss[i]);\n    }\n}\n\nextern \"C\" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {\n    // Skip empty inputs\n    if (size == 0) {\n        return 0;\n    }\n\n    FuzzReader reader(data, size);\n    execute_program(reader);\n\n    return 0;\n}\n"
  },
  {
    "path": "tests/manual/iosched.py",
    "content": "#!/bin/env python3\n\nimport os.path\nimport yaml\nimport shutil\nimport subprocess\nimport argparse\nfrom functools import reduce\n\n# Default task-quota-ms is 0.5ms\n# IO subsystem sets its latency goal to 1.5 of the above\n# Test adds 5% to compensate for potential glitches\ndefault_latency_goal = 0.5 * 1.5 * 1.05 / 1000\n\nparser = argparse.ArgumentParser(description='IO scheduler tester')\nparser.add_argument('--directory', help='Directory to run on', default='/mnt')\nparser.add_argument('--seastar-build-dir', help='Path to seastar build directory', default='./build/dev/', dest='bdir')\nparser.add_argument('--duration', help='One run duration', default=60)\nparser.add_argument('--latency-goal', help='Target latency the scheduler should meet', default=default_latency_goal)\nargs = parser.parse_args()\n\nclass iotune:\n    def __init__(self, args):\n        self._iotune = args.bdir + '/apps/iotune/iotune'\n        self._dir = args.directory\n\n    def ensure_io_properties(self):\n        if os.path.exists('io_properties.yaml'):\n            print('Using existing io_properties file')\n        else:\n            print('Running iotune')\n            subprocess.check_call([self._iotune, '--evaluation-directory', self._dir, '-c1', '--properties-file', 'io_properties.yaml'])\n\nclass ioinfo:\n    def __init__(self, args):\n        self._ioinfo = args.bdir + '/apps/io_tester/ioinfo'\n        self._dir = args.directory\n        res = subprocess.check_output([self._ioinfo, '--directory', self._dir, '--io-properties-file', 'io_properties.yaml'])\n        self._info = yaml.safe_load(res)\n\n    def max_read_length(self):\n        return min(self._info['disk_read_max_length'] / 1024, 128)\n\n    def min_write_length(self):\n        return 4\n\n    def max_write_length(self):\n        return min(self._info['disk_write_max_length'] / 1024, 64)\n\n\nclass job:\n    def __init__(self, typ, req_size_kb, prl):\n        self._typ = typ\n        self._req_size = req_size_kb\n        self._prl = prl\n        self._shares = 100\n\n    def prl(self):\n        return self._prl\n\n    def rqsz(self):\n        return self._req_size\n\n    def to_conf_entry(self, name):\n        return {\n            'name': name,\n            'shards': 'all',\n            'type': self._typ,\n            'shard_info': {\n                'parallelism': self._prl,\n                'reqsize': f'{self._req_size}kB',\n                'shares': self._shares\n            }\n        }\n\nclass io_tester:\n    def __init__(self, args):\n        self._jobs = []\n        self._duration = args.duration\n        self._io_tester = args.bdir + '/apps/io_tester/io_tester'\n        self._dir = args.directory\n        self._use_fraction = 0.8\n\n    def add_job(self, name, job):\n        self._jobs.append(job.to_conf_entry(name))\n\n    def _setup_data_sizes(self):\n        du = shutil.disk_usage(self._dir)\n        one_job_space_mb = int(du.free * self._use_fraction / len(self._jobs) / (100*1024*1024)) * 100 # round down to 100MB\n        for j in self._jobs:\n            j['data_size'] = f'{one_job_space_mb}MB'\n\n    def run(self):\n        if not self._jobs:\n            raise 'Empty jobs'\n\n        self._setup_data_sizes()\n        yaml.dump(self._jobs, open('conf.yaml', 'w'))\n        self._proc = subprocess.Popen([self._io_tester, '--storage', self._dir, '-c1', '--poll-mode', '--conf', 'conf.yaml', '--duration', f'{self._duration}', '--io-properties-file', 'io_properties.yaml'], stdout=subprocess.PIPE)\n        res = self._proc.communicate()\n        res = res[0].split(b'---\\n')[1]\n        return yaml.safe_load(res)[0]\n\n\ndef run_jobs(jobs, args):\n    iot = io_tester(args)\n    names = []\n    for j in jobs:\n        iot.add_job(j, jobs[j])\n        names.append(j)\n\n    results = iot.run()\n    statuses = []\n\n    # For now the only criteria is that requests don't spend\n    # more time in disk than the IO latency goal is\n    for n in names:\n        res = results[n]['stats']\n        total_ops = res['io_queue_total_operations']\n        max_in_disk = args.latency_goal * total_ops\n\n        status = {\n            'meets_goal': res['io_queue_total_exec_sec'] <= max_in_disk,\n            'queues': res['io_queue_total_delay_sec'] > total_ops * 0.0005\n        }\n        statuses.append(status)\n\n        label = ''\n        average_exec = '%.3f' % ((res['io_queue_total_exec_sec'] / total_ops) * 1000)\n        average_inq = '%.3f' % ((res['io_queue_total_delay_sec'] / total_ops) * 1000)\n        average_lat = '%.3f' % (results[n]['latencies']['average'] / 1000)\n        label += '' if status['meets_goal'] else '  no-goal'\n        label += ' saturated' if status['queues'] else ''\n        print(f'\\t{n:16}  exec={average_exec}ms  queue={average_inq}ms  latency={average_lat}ms{label}')\n\n    return statuses\n\ndef check_goal(statuses):\n    return reduce(lambda a, b: a and b, map(lambda x: x['meets_goal'], statuses))\n\ndef check_saturated(statuses):\n    return reduce(lambda a, b: a or b, map(lambda x: x['queues'], statuses))\n\ndef run_saturation_test(name, get_jobs, args):\n    print(f'** running {name} saturation test')\n    prl = 4\n    meets_goal = True\n    saturated = False\n    while True:\n        st = run_jobs(get_jobs(prl), args)\n        saturated = check_saturated(st)\n        meets_goal &= check_goal(st)\n\n        if saturated or prl > 8000:\n            break\n        else:\n            prl *= 4\n\n    for p in [ prl * 2, prl * 3 ]:\n        st = run_jobs(get_jobs(p), args)\n        meets_goal &= check_goal(st)\n\n    if meets_goal:\n        test_statuses[name] = 'PASS'\n        print(\"PASS\")\n    elif saturated:\n        test_statuses[name] = 'FAIL'\n        print(\"FAIL -- doesn't meet goal\")\n    else:\n        test_statuses[name] = 'FAIL (*)'\n        print(\"FAIL -- doesn't meet goal, no saturation\")\n\ndef run_base_test(name, get_jobs, args):\n    print(f'* running {name} test')\n    st = run_jobs(get_jobs(1), args)\n    if check_goal(st):\n        return True\n\n    test_statuses[name] = 'SKIP'\n    print(\"SKIP -- doesn't meet goal, no concurrency\")\n    return False\n\ndef run_pure_test(name, get_jobs, args):\n    if run_base_test(name, get_jobs, args):\n        run_saturation_test(name, get_jobs, args)\n\ndef run_mixed_tests(name, get_jobs, args):\n    if run_base_test(name, lambda p: get_jobs(p, p), args):\n        run_saturation_test(name + ' symmetrical', lambda p: get_jobs(p, p), args)\n        run_saturation_test(name + ' one-write', lambda p: get_jobs(p, 1), args)\n        run_saturation_test(name + ' one-read', lambda p: get_jobs(1, p), args)\n\n\ndef get_read_job(p, rqsz):\n    j = job('randread', rqsz, p)\n    return { f'{rqsz}k_x{p}': j }\n\ndef get_write_job(p, rqsz):\n    j = job('seqwrite', rqsz, p)\n    return { f'{rqsz}k_x{p}': j }\n\ndef get_mixed_jobs(rp, rqsz, wp, wqsz):\n    rj = job('randread', rqsz, rp)\n    wj = job('seqwrite', wqsz, wp)\n    return { f'r_{rqsz}k_x{rp}': rj, f'w_{wqsz}k_x{wp}': wj }\n\niotune(args).ensure_io_properties()\nioinf = ioinfo(args)\n\ntest_statuses = {}\n\nrun_pure_test('pure read', lambda p: get_read_job(p, ioinf.max_read_length()), args)\nrun_pure_test('pure small write', lambda p: get_write_job(p, ioinf.min_write_length()), args)\nrun_pure_test('pure large write', lambda p: get_write_job(p, ioinf.max_write_length()), args)\nrun_mixed_tests('mixed read / small write', lambda rp, wp: get_mixed_jobs(rp, ioinf.max_read_length(), wp, ioinf.min_write_length()), args)\nrun_mixed_tests('mixed read / large write', lambda rp, wp: get_mixed_jobs(rp, ioinf.max_read_length(), wp, ioinf.max_write_length()), args)\n\nfor tn in test_statuses:\n    print(f'{tn:40}: {test_statuses[tn]}')\n"
  },
  {
    "path": "tests/manual/iosched_reproducers/one_cpu_starved_shard_can_still_saturate_io.sh",
    "content": "#!/usr/bin/env bash\n\n# Test scenario:\n# A single CPU-starved shard has a batch IO job.\n# Goal: it should be able to utilize the entire bandwidth of the disk,\n# despite the rare polls.\n\nif [ $# -ne 1 ]; then\n    echo \"Usage: $0 IO_TESTER_EXECUTABLE\" >&2\n    exit 1\nfi\n\n\"$1\" --smp=7 --storage=/dev/null --conf=<(cat <<'EOF'\n- name: tablet-streaming\n  data_size: 1GB\n  shards: [0]\n  type: seqread\n  shard_info:\n    parallelism: 50\n    reqsize: 128kB\n    shares: 200\n- name: cpuhog\n  type: cpu\n  shards: [0]\n  shard_info:\n    parallelism: 1\n    execution_time: 550us\n\nEOF\n) --io-properties-file=<(cat <<'EOF'\n# i4i.2xlarge\ndisks:\n- mountpoint: /dev\n  read_bandwidth: 1542559872\n  read_iops: 218786\n  write_bandwidth: 1130867072\n  write_iops: 121499\nEOF\n)\n"
  },
  {
    "path": "tests/manual/iosched_reproducers/one_cpu_starved_shard_has_reasonable_fairness.sh",
    "content": "#!/usr/bin/env bash\n\n# Test scenario:\n# all shards contend for IO, but one shard is additionally CPU-starved\n# and polls rarely.\n# Goal: it should still be getting a reasonably fair share of disk bandwidth.\n\nif [ $# -ne 1 ]; then\n    echo \"Usage: $0 IO_TESTER_EXECUTABLE\" >&2\n    exit 1\nfi\n\n\"$1\" --smp=7 --storage=/dev/null --conf=<(cat <<'EOF'\n- name: tablet-streaming\n  data_size: 1GB\n  shards: all\n  type: seqread\n  shard_info:\n    parallelism: 50\n    reqsize: 128kB\n    shares: 200\n- name: cpuhog\n  type: cpu\n  shards: [0]\n  shard_info:\n    parallelism: 1\n    execution_time: 550us\n\nEOF\n) --io-properties-file=<(cat <<'EOF'\n# i4i.2xlarge\ndisks:\n- mountpoint: /dev\n  read_bandwidth: 1542559872\n  read_iops: 218786\n  write_bandwidth: 1130867072\n  write_iops: 121499\nEOF\n) --duration=2\n"
  },
  {
    "path": "tests/manual/iosched_reproducers/scylla_tablet_migration.sh",
    "content": "#!/usr/bin/env bash\n\n# Test scenario:\n# Simulation of a ScyllaDB workload which prompted some changes to the IO scheduler:\n# database queries concurrent with tablet streaming.\n#\n# All 7 shards are running a low-priority (200 shares) batch IO workload\n# and a high-priority (1000 shares), moderate-bandwidth, interactive workload.\n#\n# The interactive workload requires about 30% of the node's\n# total bandwidth (as measured in tokens), in small random reads.\n# The batch workload does large sequential reads and wants to utilize all\n# spare bandwidth.\n#\n# This workload is almost symmetric across shards, but is slightly skewed\n# and shard 0 is slightly more loaded. But even on this shard, the workload\n# doesn't need more than 35% of the fair bandwidth of this shard.\n#\n# Due to the distribution of shares across IO classes, the user expects that\n# the interactive workload should be guaranteed (1000 / (1000 + 200)) == ~84% of\n# the disk bandwidth on each shard. So if it's only asking for less than 35%,\n# the lower-priority job shouldn't disturb it.\n#\n# But before the relevant IO scheduler changes, this goal wasn't met,\n# and the interactive workload on shard 0 was instead starved for IO\n# by the low-priority workloads on other shards.\n\nif [ $# -ne 1 ]; then\n    echo \"Usage: $0 IO_TESTER_EXECUTABLE\" >&2\n    exit 1\nfi\n\n\"$1\" --smp=7 --storage=/dev/null --conf=<(cat <<'EOF'\n- name: tablet-streaming\n  data_size: 1GB\n  shards: all\n  type: seqread\n  shard_info:\n    parallelism: 50\n    reqsize: 128kB\n    shares: 200\n- name: cassandra-stress\n  shards: all\n  type: randread\n  data_size: 1GB\n  shard_info:\n    parallelism: 100\n    reqsize: 1536\n    shares: 1000\n    rps: 75\n  options:\n    pause_distribution: poisson\n    sleep_type: steady\n- name: cassandra-stress-slight-imbalance\n  shards: [0]\n  type: randread\n  data_size: 1GB\n  shard_info:\n    parallelism: 100\n    reqsize: 1536\n    class: cassandra-stress\n    rps: 10\n  options:\n    pause_distribution: poisson\n    sleep_type: steady\n\nEOF\n) --io-properties-file=<(cat <<'EOF'\n# i4i.2xlarge\ndisks:\n- mountpoint: /dev\n  read_bandwidth: 1542559872\n  read_iops: 218786\n  write_bandwidth: 1130867072\n  write_iops: 121499\nEOF\n)\n"
  },
  {
    "path": "tests/manual/iosched_reproducers/tau_nemesis.sh",
    "content": "#!/usr/bin/env bash\n\n# There is a `tau` mechanism in `fair_queue` which lets newly-activated\n# IO classes to monopolize the shard's IO queue for a while.\n#\n# This isn't very useful and can result in major performance problems,\n# as this test illustrates. The `highprio` workload could have tail latency\n# of about 2 milliseconds, but the `bursty_lowprio` is allowed by `tau` to butt in\n# periodically and preempt `highprio` for ~30ms, bringing its tail latency\n# to that threshold.\n\nif [ $# -ne 1 ]; then\n    echo \"Usage: $0 IO_TESTER_EXECUTABLE\" >&2\n    exit 1\nfi\n\n\"$1\" --smp=7 --storage=/dev/null --conf=<(cat <<'EOF'\n- name: filler\n  data_size: 1GB\n  shards: all\n  type: seqread\n  shard_info:\n    parallelism: 10\n    reqsize: 128kB\n    shares: 10\n- name: bursty_lowprio\n  data_size: 1GB\n  shards: all\n  type: seqread\n  shard_info:\n    parallelism: 1\n    reqsize: 128kB\n    shares: 100\n    batch: 50\n    rps: 8\n- name: highprio\n  shards: all\n  type: randread\n  data_size: 1GB\n  shard_info:\n    parallelism: 100\n    reqsize: 1536\n    shares: 1000\n    rps: 50\n  options:\n    pause_distribution: poisson\n    sleep_type: steady\nEOF\n) --io-properties-file=<(cat <<'EOF'\n# i4i.2xlarge\ndisks:\n- mountpoint: /dev\n  read_bandwidth: 1542559872\n  read_iops: 218786\n  write_bandwidth: 1130867072\n  write_iops: 121499\nEOF\n)\n"
  },
  {
    "path": "tests/manual/rl-iosched.py",
    "content": "#!/bin/env python3\n\nimport yaml\nimport time\nimport subprocess\nimport multiprocessing\nimport argparse\nimport shutil\nimport os\nimport json\n\nt_parser = argparse.ArgumentParser(description='IO scheduler tester')\nt_parser.add_argument('--directory', help='Directory to run on', default='/mnt')\nt_parser.add_argument('--seastar-build-dir', help='Path to seastar build directory', default='./build/dev/', dest='bdir')\nt_parser.add_argument('--duration', help='One run duration', type=int, default=60)\nt_parser.add_argument('--shards', help='Number of shards to use', type=int)\n\nsub_parser = t_parser.add_subparsers(help='Use --help for the list of tests')\nsub_parser.required = True\n\nparser = sub_parser.add_parser('mixed', help='Run mixed test')\nparser.set_defaults(test_name='mixed')\nparser.add_argument('--read-reqsize', help='Size of a read request in kbytes', type=int, default=4)\nparser.add_argument('--read-fibers', help='Number of reading fibers', type=int, default=5)\nparser.add_argument('--read-shares', help='Shares for read workload', type=int, default=2500)\nparser.add_argument('--write-reqsize', help='Size of a write request in kbytes', type=int, default=64)\nparser.add_argument('--write-fibers', help='Number of writing fibers', type=int, default=2)\nparser.add_argument('--write-shares', help='Shares for write workload', type=int, default=100)\nparser.add_argument('--sleep-type', help='The io_tester conf.options.sleep_type option', default='busyloop')\nparser.add_argument('--pause-dist', help='The io_tester conf.option.pause_distribution option', default='uniform')\n\nparser = sub_parser.add_parser('limits', help='Run limits test')\nparser.set_defaults(test_name='limits')\nparser.add_argument('--bw-reqsize', help='Bandwidth job request size in kbytes', type=int, default=64)\nparser.add_argument('--iops-reqsize', help='IOPS job request size in kbytes', type=int, default=4)\nparser.add_argument('--limit-ratio', help='Bandidth/IOPS fraction to test', type=int, default=2)\nparser.add_argument('--parallelism', help='IO job parallelism', type=int, default=100)\n\nparser = sub_parser.add_parser('isolation', help='Run isolation test')\nparser.set_defaults(test_name='isolation')\nparser.add_argument('--instances', help='Number of instances to isolate from each other', type=int, default=3)\nparser.add_argument('--parallelism', help='IO job parallelism', type=int, default=100)\n\nparser = sub_parser.add_parser('class_limit', help='Run class bandwidth limit test')\nparser.set_defaults(test_name='class_limit')\nparser.add_argument('--parallelism', help='IO job parallelism', type=int, default=10)\n\nargs = t_parser.parse_args()\n\n\nclass iotune:\n    def __init__(self, args):\n        self._iotune = args.bdir + '/apps/iotune/iotune'\n        self._dir = args.directory\n\n    def ensure_io_properties(self):\n        if os.path.exists('io_properties.yaml'):\n            print('Using existing io_properties file')\n        else:\n            print('Running iotune')\n            subprocess.check_call([self._iotune, '--evaluation-directory', self._dir, '--properties-file', 'io_properties.yaml'])\n\n\nclass job:\n    def __init__(self, typ, req_size_kb, shares = 100, prl = None, rps = None, bandwidth_mb = None):\n        self._typ = typ\n        self._req_size = req_size_kb\n        self._prl = prl\n        self._rps = rps\n        self._shares = shares\n        self._bandwidth = bandwidth_mb\n\n    def prl(self):\n        return self._prl\n\n    def rqsz(self):\n        return self._req_size\n\n    def to_conf_entry(self, name, options):\n        ret = {\n            'name': name,\n            'shards': 'all',\n            'type': self._typ,\n            'shard_info': {\n                'reqsize': f'{self._req_size}kB',\n                'shares': self._shares,\n            },\n        }\n        if options is not None:\n            ret['options'] = options\n        if self._prl is not None:\n            ret['shard_info']['parallelism'] = int(self._prl)\n        if self._rps is not None:\n            ret['shard_info']['rps'] = int(self._rps)\n        if self._bandwidth is not None:\n            ret['shard_info']['bandwidth'] = f'{int(self._bandwidth)}MB'\n        return ret\n\n\nclass io_tester:\n    def __init__(self, args, opts = None, ioprop = 'io_properties.yaml', groups = None):\n        self._jobs = []\n        self._io_tester = args.bdir + '/apps/io_tester/io_tester'\n        self._dir = args.directory\n        self._use_fraction = 0.8\n        self._max_data_size_gb = 8\n        self._job_options = opts\n        self._io_tester_args = [\n            '--io-properties-file', ioprop,\n            '--storage', self._dir,\n            '--duration', f'{args.duration}',\n        ]\n        if args.shards is not None:\n            self._io_tester_args += [ f'-c{args.shards}' ]\n        if groups is not None:\n            self._io_tester_args += [ '--num-io-groups', f'{groups}' ]\n\n    def add_job(self, name, job):\n        self._jobs.append(job.to_conf_entry(name, self._job_options))\n\n    def _setup_data_sizes(self):\n        du = shutil.disk_usage(self._dir)\n        one_job_space_mb = int(du.free * self._use_fraction / len(self._jobs) / (100*1024*1024)) * 100 # round down to 100MB\n        if one_job_space_mb > self._max_data_size_gb * 1024:\n            one_job_space_mb = self._max_data_size_gb * 1024\n        for j in self._jobs:\n            j['data_size'] = f'{one_job_space_mb}MB'\n\n    def names(self):\n        return [ j.name for j in _jobs ]\n\n    def run(self):\n        if not self._jobs:\n            raise 'Empty jobs'\n\n        self._setup_data_sizes()\n        yaml.Dumper.ignore_aliases = lambda *args : True\n        yaml.dump(self._jobs, open('conf.yaml', 'w'))\n        res = subprocess.check_output([self._io_tester, '--conf', 'conf.yaml', *self._io_tester_args], stderr=subprocess.PIPE)\n        res = res.split(b'---\\n')[1]\n        res = yaml.safe_load(res)\n\n        ret = {}\n        for shard_res in res:\n            for wl in shard_res:\n                if wl == 'shard':\n                    continue\n\n                if wl not in ret:\n                    ret[wl] = {\n                            'throughput': 0.0,\n                            'IOPS': 0.0,\n                            'latencies': {\n                                'p0.95': 0,\n                            },\n                            'stats': {\n                                'total_requests': 0,\n                                'io_queue_total_operations': 0,\n                                'io_queue_total_exec_sec': 0,\n                                'io_queue_total_delay_sec': 0,\n                            },\n                    }\n\n                ret[wl]['throughput'] += shard_res[wl]['throughput']\n                ret[wl]['IOPS'] += shard_res[wl]['IOPS']\n                ret[wl]['latencies']['p0.95'] += shard_res[wl]['latencies']['p0.95']\n                ret[wl]['stats']['total_requests'] += shard_res[wl]['stats']['total_requests']\n                ret[wl]['stats']['io_queue_total_operations'] += shard_res[wl]['stats']['io_queue_total_operations']\n                ret[wl]['stats']['io_queue_total_delay_sec'] += shard_res[wl]['stats']['io_queue_total_delay_sec']\n                ret[wl]['stats']['io_queue_total_exec_sec'] += shard_res[wl]['stats']['io_queue_total_exec_sec']\n\n        for wl in ret:\n            ret[wl]['latencies']['p0.95'] /= len(res)\n\n        return ret\n\n\nall_tests = {}\n# decorator to add test functions by names\ndef test_name(name):\n    def add_test(name, fn):\n        all_tests[name] = fn\n    return lambda x : add_test(name, x)\n\n\niot = iotune(args)\niot.ensure_io_properties()\n\nioprop = yaml.safe_load(open('io_properties.yaml'))\n\nfor prop in ioprop['disks']:\n    if prop['mountpoint'] == args.directory:\n        ioprop = prop\n        break\nelse:\n    raise 'Cannot find required mountpoint in io-properties'\n\n\ndef mixed_show_stat_header():\n    print('-' * 20 + '8<' + '-' * 20)\n    print('name           througput(kbs) iops lat95(us) queue-time(us) execution-time(us) K(bw) K(iops) K()')\n\n\ndef mixed_run_and_show_results(m, ioprop):\n    def xtimes(st, nm):\n        return (st[nm] * 1000000) / st[\"io_queue_total_operations\"]\n\n    res = m.run()\n    for name in res:\n        st = res[name]\n        throughput = st['throughput']\n        iops = st['IOPS']\n        lats = st['latencies']\n        stats = st['stats']\n\n        if name.startswith('read'):\n            k_bw = throughput / (ioprop['read_bandwidth']/1000)\n            k_iops = iops / ioprop['read_iops']\n        else:\n            k_bw = throughput / (ioprop['write_bandwidth']/1000)\n            k_iops = iops / ioprop['write_iops']\n\n        print(f'{name:20} {int(throughput):7} {int(iops):5} {lats[\"p0.95\"]:.1f} {xtimes(stats, \"io_queue_total_delay_sec\"):.1f} {xtimes(stats, \"io_queue_total_exec_sec\"):.1f} {k_bw:.3f} {k_iops:.3f} {k_bw + k_iops:.3f}')\n\n\n@test_name('mixed')\ndef run_mixed_test(args, ioprop):\n    nr_cores = args.shards\n    if nr_cores is None:\n        nr_cores = multiprocessing.cpu_count()\n\n    read_rps_per_shard = int(ioprop['read_iops'] / nr_cores * 0.5)\n    read_rps = read_rps_per_shard / args.read_fibers\n\n    options = {\n        'sleep_type': args.sleep_type,\n        'pause_distribution': args.pause_dist,\n    }\n\n    print(f'Read RPS:{read_rps} fibers:{args.read_fibers}')\n\n    mixed_show_stat_header()\n\n    m = io_tester(args, opts = options)\n    m.add_job(f'read_rated_{args.read_shares}', job('randread', args.read_reqsize, shares = args.read_shares, prl = args.read_fibers, rps = read_rps))\n    mixed_run_and_show_results(m, ioprop)\n\n    m = io_tester(args, opts = options)\n    m.add_job(f'write_{args.write_shares}', job('seqwrite', args.write_reqsize, shares = args.write_shares, prl = args.write_fibers))\n    m.add_job(f'read_rated_{args.read_shares}', job('randread', args.read_reqsize, shares = args.read_shares, prl = args.read_fibers, rps = read_rps))\n    mixed_run_and_show_results(m, ioprop)\n\n\ndef limits_make_ioprop(name, ioprop, changes):\n    nprop = {}\n    for k in ioprop:\n        nprop[k] = ioprop[k] if k not in changes else changes[k]\n    yaml.dump({'disks': [ nprop ]}, open(name, 'w'))\n\n\ndef limits_show_stat_header():\n    print('-' * 20 + '8<' + '-' * 20)\n    print('name           througput(kbs) iops')\n\n\ndef limits_run_and_show_results(m):\n    res = m.run()\n    for name in res:\n        st = res[name]\n        throughput = st['throughput']\n        iops = st['IOPS']\n        print(f'{name:20} {int(throughput):7} {int(iops):5}')\n\n\n@test_name('limits')\ndef run_limits_test(args, ioprop):\n    disk_bw = ioprop['read_bandwidth']\n    disk_iops = ioprop['read_iops']\n\n    print(f'Target bandwidth {disk_bw} -> {int(disk_bw / args.limit_ratio)}, IOPS {disk_iops} -> {int(disk_iops / args.limit_ratio)}')\n    limits_show_stat_header()\n\n    m = io_tester(args)\n    m.add_job(f'reads_bw', job('seqread', args.bw_reqsize, prl = args.parallelism))\n    limits_run_and_show_results(m)\n\n    limits_make_ioprop('ioprop_1.yaml', ioprop, { 'read_bandwidth': int(disk_bw / args.limit_ratio) })\n    m = io_tester(args, ioprop = 'ioprop_1.yaml')\n    m.add_job(f'reads_lim_bw', job('seqread', args.bw_reqsize, prl = args.parallelism))\n    limits_run_and_show_results(m)\n\n    m = io_tester(args)\n    m.add_job(f'reads_iops', job('randread', args.iops_reqsize, prl = args.parallelism))\n    limits_run_and_show_results(m)\n\n    limits_make_ioprop('ioprop_2.yaml', ioprop, { 'read_iops': int(disk_iops / args.limit_ratio) })\n    m = io_tester(args, ioprop = 'ioprop_2.yaml')\n    m.add_job(f'reads_lim_iops', job('randread', args.iops_reqsize, prl = args.parallelism))\n    limits_run_and_show_results(m)\n\n\ndef isolation_show_stat_header():\n    print('-' * 20 + '8<' + '-' * 20)\n    print('name           iops  lat95(us)')\n\n\ndef isolation_run_and_show_results(m):\n    res = m.run()\n    for name in res:\n        st = res[name]\n        iops = st['IOPS']\n        lats = st['latencies']\n\n        print(f'{name:20} {int(iops):5} {lats[\"p0.95\"]:.1f}')\n\n\n@test_name('isolation')\ndef run_isolation_test(args, ioprop):\n    if args.shards is not None and args.shards < args.instances:\n        raise f'Number of shards ({args.shards}) should be more than the number of instances ({args.instances})'\n\n    isolation_show_stat_header()\n\n    m = io_tester(args)\n    m.add_job('reads', job('randread', 4, prl = args.parallelism))\n    isolation_run_and_show_results(m)\n\n    m = io_tester(args, groups = args.instances)\n    m.add_job('reads_groups', job('randread', 4, prl = args.parallelism))\n    isolation_run_and_show_results(m)\n\n\ndef class_limit_show_stat_header():\n    print('-' * 20 + '8<' + '-' * 20)\n    print('name           througput(kbs)')\n\n\ndef class_limit_run_and_show_results(m):\n    res = m.run()\n    for name in res:\n        st = res[name]\n        throughput = st['throughput']\n\n        print(f'{name:20} {int(throughput):10}')\n\n\n@test_name('class_limit')\ndef run_class_limit_test(args, ioprop):\n\n    class_limit_show_stat_header()\n\n    for bw in [ 150, 200 ]:\n        m = io_tester(args)\n        m.add_job(f'reads_{bw}', job('randread', 128, prl = args.parallelism, bandwidth_mb = bw))\n        class_limit_run_and_show_results(m)\n\n    for bw in [ 50, 100 ]:\n        m = io_tester(args)\n        m.add_job(f'reads_a_{bw}', job('randread', 128, prl = args.parallelism, bandwidth_mb = bw))\n        m.add_job(f'reads_b_{bw}', job('randread', 128, prl = args.parallelism, bandwidth_mb = bw))\n        class_limit_run_and_show_results(m)\n\n    disk_bw = ioprop['read_bandwidth'] / (1024 * 1024)\n\n    m = io_tester(args)\n    m.add_job(f'reads_a_unb', job('randread', 128, prl = args.parallelism))\n    m.add_job(f'reads_b_unb', job('randread', 128, prl = args.parallelism))\n    class_limit_run_and_show_results(m)\n\n    m = io_tester(args)\n    m.add_job(f'reads_a_unb', job('randread', 128, prl = args.parallelism))\n    m.add_job(f'reads_b_{int(disk_bw/4)}', job('randread', 128, prl = args.parallelism, bandwidth_mb = int(disk_bw/4)))\n    class_limit_run_and_show_results(m)\n\n    m = io_tester(args)\n    m.add_job(f'reads_a_unb', job('randread', 128, prl = args.parallelism))\n    m.add_job(f'reads_b_{int(disk_bw*2/3)}', job('randread', 128, prl = args.parallelism, bandwidth_mb = int(disk_bw*2/3)))\n    class_limit_run_and_show_results(m)\n\n\nprint(f'=== Running {args.test_name} ===')\nall_tests[args.test_name](args, ioprop)\n"
  },
  {
    "path": "tests/perf/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\n# Logical target for all perf tests.\nadd_custom_target (perf_tests)\n\nmacro (seastar_add_test name)\n  set (args ${ARGN})\n\n  cmake_parse_arguments (\n    parsed_args\n    \"NO_SEASTAR_PERF_TESTING_LIBRARY\"\n    \"\"\n    \"SOURCES\"\n    ${args})\n\n  set (target test_perf_${name})\n  add_executable (${target} ${parsed_args_SOURCES})\n\n  if (parsed_args_NO_SEASTAR_PERF_TESTING_LIBRARY)\n    set (libraries seastar_private)\n  else ()\n    set (libraries\n      seastar_private\n      seastar_perf_testing)\n  endif ()\n\n  target_link_libraries (${target}\n    PRIVATE ${libraries})\n\n  target_include_directories (${target}\n    PRIVATE\n      ${CMAKE_CURRENT_SOURCE_DIR}\n      ${Seastar_SOURCE_DIR}/src)\n\n  set_target_properties (${target}\n    PROPERTIES\n      OUTPUT_NAME ${name}_perf)\n\n  add_dependencies (perf_tests ${target})\n  set (${name}_test ${target})\nendmacro ()\n\nseastar_add_test (fstream\n  SOURCES fstream_perf.cc\n  NO_SEASTAR_PERF_TESTING_LIBRARY)\n\nseastar_add_test (fair_queue\n  SOURCES fair_queue_perf.cc)\n\nseastar_add_test (shared_token_bucket\n  SOURCES shared_token_bucket.cc)\n\nseastar_add_test (future_util\n  SOURCES future_util_perf.cc)\n\nseastar_add_test (rpc\n  SOURCES rpc_perf.cc)\n\nseastar_add_test (smp_submit_to\n  SOURCES smp_submit_to_perf.cc\n  NO_SEASTAR_PERF_TESTING_LIBRARY)\n\nseastar_add_test (thread_context_switch\n  SOURCES thread_context_switch_test.cc\n  NO_SEASTAR_PERF_TESTING_LIBRARY)\n\nseastar_add_test (coroutine\n  SOURCES coroutine_perf.cc)\n\nseastar_add_test (allocator\n  SOURCES allocator_perf.cc)\n\nseastar_add_test (container\n  SOURCES container_perf.cc)\n\nseastar_add_test (http_client\n  SOURCES http_client_perf.cc linux_perf_event.cc\n  NO_SEASTAR_PERF_TESTING_LIBRARY)\n\nseastar_add_test (perf_tests\n  SOURCES perf_tests_perf.cc)\n\nseastar_add_test (metrics\n  SOURCES metrics_perf.cc)\n"
  },
  {
    "path": "tests/perf/allocator_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB\n */\n\n#include <seastar/testing/perf_tests.hh>\n\n#include <seastar/core/memory.hh>\n#include <sys/mman.h>\n\nstruct alloc_bench {\n\n    static constexpr size_t small_alloc_size = 8;\n    static constexpr size_t large_alloc_size = 32000;\n    static constexpr size_t MAX_POINTERS = 1000;\n\n    enum alloc_test_flags : uint8_t {\n        MEASURE_ALLOC = 1,\n        MEASURE_FREE  = 2\n    };\n\n    static constexpr alloc_test_flags MEASURE_BOTH = (alloc_test_flags)(MEASURE_ALLOC | MEASURE_FREE);\n\n    using alloc_fn = void * (size_t);\n    using free_fn = void (void *);\n\n    struct c_funcs {\n        static constexpr alloc_fn * const alloc = std::malloc;\n        static constexpr free_fn * const free = std::free;\n    };\n\n    struct cpp_operator_funcs {\n        static void * alloc(size_t size) { return operator new(size); }\n        static void free(void *p){ operator delete(p); }\n    };\n\n    struct cpp_array_funcs {\n        static void * alloc(size_t size) { return new char[size]; }\n        static void free(void *p){ delete [] static_cast<char *>(p); }\n    };\n\n    template <size_t alloc_size, alloc_test_flags F, typename A>\n    size_t alloc_test() {\n        // in some cases we want to measure allocation, on deallocation or\n        // both, and this helper facilitates that with a minimum of fuss\n        // it always executes fn on every pointer in the array, but only\n        // measures when should_measure is true\n        auto run_maybe_measure = [this](bool should_measure, auto fn) {\n            if (should_measure) {\n                perf_tests::start_measuring_time();\n            }\n            for (auto &p : _pointers) {\n                fn(p);\n            }\n            if (should_measure) {\n                perf_tests::stop_measuring_time();\n            }\n        };\n\n        run_maybe_measure(F & MEASURE_ALLOC, [](auto& p) { p = A::alloc(alloc_size); });\n        run_maybe_measure(F & MEASURE_FREE, [](auto& p) { A::free(p); });\n        return _pointers.size();\n    }\n\n    std::array<void *, MAX_POINTERS> _pointers{};\n};\n\nPERF_TEST_F(alloc_bench, malloc_only)      { return alloc_test<small_alloc_size, MEASURE_ALLOC, c_funcs>(); }\nPERF_TEST_F(alloc_bench, free_only)        { return alloc_test<small_alloc_size, MEASURE_FREE, c_funcs>(); }\nPERF_TEST_F(alloc_bench, malloc_free)      { return alloc_test<small_alloc_size, MEASURE_BOTH, c_funcs>(); }\n\nPERF_TEST_F(alloc_bench, op_new_only)      { return alloc_test<small_alloc_size, MEASURE_ALLOC, cpp_operator_funcs>(); }\nPERF_TEST_F(alloc_bench, op_delete_only)   { return alloc_test<small_alloc_size, MEASURE_FREE, cpp_operator_funcs>(); }\nPERF_TEST_F(alloc_bench, op_new_delete)    { return alloc_test<small_alloc_size, MEASURE_BOTH, cpp_operator_funcs>(); }\n\nPERF_TEST_F(alloc_bench, new_array_only)   { return alloc_test<small_alloc_size, MEASURE_ALLOC, cpp_array_funcs>(); }\nPERF_TEST_F(alloc_bench, delete_array_only){ return alloc_test<small_alloc_size, MEASURE_FREE, cpp_array_funcs>(); }\nPERF_TEST_F(alloc_bench, array_new_delete) { return alloc_test<small_alloc_size, MEASURE_BOTH, cpp_array_funcs>(); }\n\nPERF_TEST_F(alloc_bench, alloc_only_large) { return alloc_test<large_alloc_size, MEASURE_ALLOC, c_funcs>(); }\nPERF_TEST_F(alloc_bench, free_only_large ) { return alloc_test<large_alloc_size, MEASURE_FREE, c_funcs>(); }\nPERF_TEST_F(alloc_bench, alloc_free_large) { return alloc_test<large_alloc_size, MEASURE_BOTH, c_funcs>(); }\n\n// this test doesn't serve much value. It should take about 10 times as the\n// single alloc test above. If not, something is wrong.\nPERF_TEST_F(alloc_bench, single_alloc_and_free_small_many)\n{\n    const std::size_t allocs = 10;\n    static_assert(allocs <= MAX_POINTERS);\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        auto ptr = malloc(10);\n        perf_tests::do_not_optimize(ptr);\n        _pointers[i] = ptr;\n    }\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        free(_pointers[i]);\n    }\n}\n\n// Allocate from more than one page. Should also not suffer a perf penalty\n// As we should have at least min free of 50 objects (hard coded internally)\nPERF_TEST_F(alloc_bench, single_alloc_and_free_small_many_cross_page)\n{\n    const std::size_t alloc_size = 1024;\n    const std::size_t allocs = (seastar::memory::page_size / alloc_size) + 1;\n\n    static_assert(allocs <= MAX_POINTERS);\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        auto ptr = malloc(alloc_size);\n        perf_tests::do_not_optimize(ptr);\n        _pointers[i] = ptr;\n    }\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        free(_pointers[i]);\n    }\n}\n\n// Include an allocation in the benchmark that will require going to the large pool\n// for more data for the small pool\nPERF_TEST_F(alloc_bench, single_alloc_and_free_small_many_cross_page_alloc_more)\n{\n    const std::size_t alloc_size = 1024;\n    const std::size_t allocs = 101; // at 1024 alloc size we will have a _max_free of 100 objects\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        auto ptr = malloc(alloc_size);\n        perf_tests::do_not_optimize(ptr);\n        _pointers[i] = ptr;\n    }\n\n    for (std::size_t i = 0; i < allocs; ++i) {\n        free(_pointers[i]);\n    }\n}\n\ntemplate <typename DistType>\nstatic size_t dist_bench() {\n    std::random_device rd_device;\n    std::mt19937_64 random_gen(rd_device());\n\n    constexpr size_t ITERS = 10000;\n    size_t rate = 3'000'000;\n    perf_tests::do_not_optimize(rate);\n\n    DistType dist(rate);\n\n    typename DistType::result_type sum = 0;\n\n    for (size_t i = 0; i < ITERS; i++) {\n        sum += dist(random_gen);\n    }\n\n    perf_tests::do_not_optimize(sum);\n\n    return ITERS;\n}\n\nPERF_TEST(random_sampling, exp_dist) {\n    return dist_bench<std::exponential_distribution<double>>();\n}\n\nPERF_TEST(random_sampling, geo_dist) {\n    return dist_bench<std::geometric_distribution<size_t>>();\n}\n"
  },
  {
    "path": "tests/perf/container_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB Ltd.\n */\n\n\n#include <boost/container/deque.hpp>\n#include <boost/container/options.hpp>\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/circular_buffer.hh>\n\nusing trivial_elem = int;\n\nstatic constexpr size_t big_size = 10000;\nstatic constexpr size_t small_size = 3;\n\n// test should do this many iterations, at least, inside the \"measured region\"\n// to defray the cost of the start/stop measuring time pairs, which are expensive\nstatic constexpr size_t total_iters = 10000;\n\n\nstruct nontrivial_elem {\n    nontrivial_elem(int x) : x{x} { perf_tests::do_not_optimize(this->x); }\n    nontrivial_elem(const nontrivial_elem&) = default;\n    nontrivial_elem& operator=(const nontrivial_elem&) = default;\n    ~nontrivial_elem() {\n        perf_tests::do_not_optimize(x);\n    }\n    int x;\n};\n\nstruct fifo_traits {\n    template <typename T>\n    using type = chunked_fifo<T>;\n};\n\nstruct circ_traits {\n    template <typename T>\n    using type = circular_buffer<T>;\n};\n\nstruct boost_deque_traits {\n    using deque_opts = boost::container::deque_options_t<boost::container::block_bytes<16384u>>;\n\n    template <typename T>\n    using type = boost::container::deque<T, void, deque_opts>;\n};\n\ntemplate <typename C>\n    requires requires(C c) { c.reserve(0); }\nvoid reserve(C& c, size_t size) {\n    c.reserve(size);\n}\n\ntemplate <typename C>\nvoid reserve(C& c, size_t size) {}\n\ntemplate <typename Traits, typename T>\nauto make_n(size_t size) {\n    using container = Traits::template type<T>;\n\n    perf_tests::do_not_optimize(size);\n\n    container c;\n    reserve(c, size);\n\n    for (size_t i = 0; i < size; i++) {\n        auto e = trivial_elem(i);\n        c.push_back(e);\n    }\n\n    perf_tests::do_not_optimize(c);\n\n    return c;\n}\n\nstruct container_perf {};\n\n\nsize_t outer_loops(size_t inner_size) {\n    return total_iters / inner_size + 1;\n}\n\ntemplate <typename Traits>\nsize_t iteration_bench(size_t size) {\n    auto c = make_n<Traits, trivial_elem>(size);\n    auto outer = outer_loops(size);\n\n    perf_tests::start_measuring_time();\n    for (size_t o = 0; o < outer; o++) {\n        for (auto& e : c) {\n            perf_tests::do_not_optimize(e);\n        }\n    }\n    perf_tests::stop_measuring_time();\n    return outer * size;\n}\n\ntemplate <typename Traits>\nsize_t index_bench(size_t size) {\n    auto c = make_n<Traits, trivial_elem>(size);\n    auto outer = outer_loops(size);\n\n    perf_tests::start_measuring_time();\n    for (size_t o = 0; o < outer; o++) {\n        for (size_t i = 0; i < size; i++) {\n            perf_tests::do_not_optimize(c[i]);\n        }\n    }\n    perf_tests::stop_measuring_time();\n    return size * outer;\n}\n\ntemplate <typename Traits, typename T>\nsize_t clear_bench(size_t size) {\n    auto c = make_n<Traits, T>(size);\n\n    perf_tests::start_measuring_time();\n    c.clear();\n    perf_tests::stop_measuring_time();\n\n    return size;\n}\n\ntemplate <typename Traits, typename T>\nsize_t erase_bench(size_t size) {\n    auto c = make_n<Traits, T>(size);\n\n    perf_tests::start_measuring_time();\n    c.erase(c.begin(), c.begin() + size / 2);\n    perf_tests::stop_measuring_time();\n\n    perf_tests::do_not_optimize(c);\n\n    return size / 2;\n}\n\ntemplate <typename Traits, typename T>\nsize_t erase_front_bench(size_t size) {\n    auto c = make_n<Traits, T>(size);\n\n    perf_tests::start_measuring_time();\n    c.pop_front_n(size / 2);\n    perf_tests::stop_measuring_time();\n\n    perf_tests::do_not_optimize(c);\n\n    return size / 2;\n}\n\nPERF_TEST_F(container_perf, erase_trivial_chunked_fifo) {\n    return erase_front_bench<fifo_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, erase_trivial_circular_buffer) {\n    return erase_bench<circ_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, erase_trivial_boost_deque) {\n    return erase_bench<boost_deque_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_nontrivial_chunked_fifo) {\n    return clear_bench<fifo_traits, nontrivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_nontrivial_circular_buffer) {\n    return clear_bench<circ_traits, nontrivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_nontrivial_boost_deque) {\n    return clear_bench<boost_deque_traits, nontrivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_trivial_chunked_fifo) {\n    return clear_bench<fifo_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_trivial_circular_buffer) {\n    return clear_bench<circ_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, clear_trivial_boost_deque) {\n    return clear_bench<boost_deque_traits, trivial_elem>(big_size);\n}\n\nPERF_TEST_F(container_perf, iter_big_chunked_fifo) {\n    return iteration_bench<fifo_traits>(big_size);\n}\n\nPERF_TEST_F(container_perf, iter_big_circular_buffer) {\n    return iteration_bench<circ_traits>(big_size);\n}\n\nPERF_TEST_F(container_perf, iter_big_boost_deque) {\n    return iteration_bench<boost_deque_traits>(big_size);\n}\n\nPERF_TEST_F(container_perf, index_big_circular_buffer) {\n    return index_bench<circ_traits>(big_size);\n}\n\nPERF_TEST_F(container_perf, index_big_boost_deque) {\n    return index_bench<boost_deque_traits>(big_size);\n}\n\nPERF_TEST_F(container_perf, iter_small_chunked_fifo) {\n    return iteration_bench<fifo_traits>(small_size);\n}\n\nPERF_TEST_F(container_perf, iter_small_circular_buffer) {\n    return iteration_bench<circ_traits>(small_size);\n}\n\nPERF_TEST_F(container_perf, iter_small_boost_deque) {\n    return iteration_bench<boost_deque_traits>(small_size);\n}\n\n"
  },
  {
    "path": "tests/perf/coroutine_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022-present ScyllaDB\n */\n\n#include <seastar/testing/perf_tests.hh>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/maybe_yield.hh>\n#include <seastar/coroutine/generator.hh>\n#include <seastar/core/circular_buffer_fixed_capacity.hh>\n#include <seastar/util/later.hh>\n#include <vector>\n\nstruct coroutine_test {\n};\n\nPERF_TEST_C(coroutine_test, empty)\n{\n    co_return;\n}\n\nPERF_TEST_C(coroutine_test, without_preemption_check)\n{\n    co_await coroutine::without_preemption_check(make_ready_future<>());\n}\n\nPERF_TEST_C(coroutine_test, ready)\n{\n    co_await make_ready_future<>();\n}\n\nPERF_TEST_C(coroutine_test, maybe_yield)\n{\n    co_await coroutine::maybe_yield();\n}\n\n// Benchmark unbuffered generator: one suspension per element\nPERF_TEST_C(coroutine_test, unbuffered_generator)\n{\n    constexpr int count = 100;\n\n    auto gen = []() -> coroutine::experimental::generator<int> {\n        for (int i = 0; i < count; ++i) {\n            co_yield i;\n        }\n    }();\n\n    int sum = 0;\n    while (auto val = co_await gen()) {\n        sum += *val;\n    }\n    perf_tests::do_not_optimize(sum);\n}\n\n// Benchmark buffered generator: amortized suspension overhead\nPERF_TEST_C(coroutine_test, buffered_generator)\n{\n    constexpr int count = 100;\n    constexpr int buffer_size = 16;\n\n    auto gen = []() -> coroutine::experimental::generator<int, int, circular_buffer_fixed_capacity<int, buffer_size>> {\n        for (int i = 0; i < count; ++i) {\n            co_yield i;\n        }\n    }();\n\n    int sum = 0;\n    while (auto val = co_await gen()) {\n        sum += *val;\n    }\n    perf_tests::do_not_optimize(sum);\n}\n"
  },
  {
    "path": "tests/perf/fair_queue_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB Ltd.\n */\n\n\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/fair_queue.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/when_all.hh>\n#include <ranges>\n\nstatic constexpr fair_queue::class_id cid = 0;\n\nstruct local_fq_and_class {\n    seastar::fair_queue fq;\n    seastar::fair_queue sfq;\n    unsigned executed = 0;\n\n    seastar::fair_queue& queue(bool local) noexcept { return local ? fq : sfq; }\n\n    local_fq_and_class()\n        : fq(seastar::fair_queue::config())\n        , sfq(seastar::fair_queue::config())\n    {\n        fq.register_priority_class(cid, 1);\n        sfq.register_priority_class(cid, 1);\n    }\n\n    ~local_fq_and_class() {\n        fq.unregister_priority_class(cid);\n        sfq.unregister_priority_class(cid);\n    }\n};\n\nstruct local_fq_entry {\n    seastar::fair_queue_entry ent;\n    std::function<void()> submit;\n\n    template <typename Func>\n    local_fq_entry(fair_queue_entry::capacity_t cap, Func&& f)\n        : ent(cap)\n        , submit(std::move(f)) {}\n};\n\nstruct perf_fair_queue {\n\n    static constexpr unsigned requests_to_dispatch = 1000;\n\n    seastar::sharded<local_fq_and_class> local_fq;\n\n    perf_fair_queue()\n    {\n        local_fq.start().get();\n    }\n\n    ~perf_fair_queue() {\n        local_fq.stop().get();\n    }\n\n    future<> test(bool local);\n};\n\nfuture<> perf_fair_queue::test(bool loc) {\n\n    auto invokers = local_fq.invoke_on_all([loc] (local_fq_and_class& local) {\n        return parallel_for_each(std::views::iota(0u, requests_to_dispatch), [&local, loc] (unsigned dummy) {\n            auto cap = fair_queue_entry::capacity_t(1);\n            auto req = std::make_unique<local_fq_entry>(cap, [&local, loc, cap] {\n                local.executed++;\n                local.queue(loc).notify_request_finished(cap);\n            });\n            local.queue(loc).queue(cid, req->ent);\n            req.release();\n            return make_ready_future<>();\n        });\n    });\n\n    auto collectors = local_fq.invoke_on_all([loc] (local_fq_and_class& local) {\n        // Zeroing this counter must be here, otherwise should the collectors win the\n        // execution order in when_all_succeed(), the do_until()'s stopping callback\n        // would return true immediately and the queue would not be dispatched.\n        //\n        // At the same time, although this counter is incremented by the lambda from\n        // invokers, it's not called until the fq.dispatch_requests() is, so there's no\n        // opposite problem if zeroing it here.\n        local.executed = 0;\n\n        return do_until([&local] { return local.executed == requests_to_dispatch; }, [&local, loc] {\n            auto& q = local.queue(loc);\n            auto* req = q.top();\n            if (req == nullptr) {\n                return make_ready_future<>();\n            }\n\n            q.pop_front();\n            local_fq_entry* le = boost::intrusive::get_parent_from_member(req, &local_fq_entry::ent);\n            le->submit();\n            delete le;\n            return make_ready_future<>();\n        });\n    });\n\n    return when_all_succeed(std::move(invokers), std::move(collectors)).discard_result();\n}\n\nPERF_TEST_F(perf_fair_queue, contended_local)\n{\n    return test(true);\n}\nPERF_TEST_F(perf_fair_queue, contended_shared)\n{\n    return test(false);\n}\n"
  },
  {
    "path": "tests/perf/fstream_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#include <seastar/core/fstream.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <fmt/printf.h>\n#include <string>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nint main(int ac, char** av) {\n    app_template at;\n    namespace bpo = boost::program_options;\n    at.add_options()\n            (\"concurrency\", bpo::value<unsigned>()->default_value(1), \"Write operations to issue in parallel\")\n            (\"buffer-size\", bpo::value<size_t>()->default_value(4096), \"Write buffer size\")\n            (\"total-ops\", bpo::value<unsigned>()->default_value(100000), \"Total write operations to issue\")\n            (\"sloppy-size\", bpo::value<bool>()->default_value(false), \"Enable the sloppy-size optimization\")\n            ;\n    return at.run(ac, av, [&at] {\n        auto concurrency = at.configuration()[\"concurrency\"].as<unsigned>();\n        auto buffer_size = at.configuration()[\"buffer-size\"].as<size_t>();\n        auto total_ops = at.configuration()[\"total-ops\"].as<unsigned>();\n        auto sloppy_size = at.configuration()[\"sloppy-size\"].as<bool>();\n        file_open_options foo;\n        foo.sloppy_size = sloppy_size;\n        return open_file_dma(\n                \"testfile.tmp\", open_flags::wo | open_flags::create | open_flags::exclusive,\n                foo).then([=] (file f) {\n            file_output_stream_options foso;\n            foso.buffer_size = buffer_size;\n            foso.preallocation_size = 32 << 20;\n            foso.write_behind = concurrency;\n            return make_file_output_stream(f, foso).then([=] (output_stream<char>&& os) {\n                return do_with(std::move(os), std::move(f), unsigned(0), [=] (output_stream<char>& os, file& f, unsigned& completed) {\n                    auto start = std::chrono::steady_clock::now();\n                    return repeat([=, &os, &completed] {\n                        if (completed == total_ops) {\n                            return make_ready_future<stop_iteration>(stop_iteration::yes);\n                        }\n                        std::string buf(buffer_size, '\\0');\n                        return os.write(buf).then([&completed] {\n                            ++completed;\n                            return stop_iteration::no;\n                        });\n                    }).then([=, &os] {\n                        auto end = std::chrono::steady_clock::now();\n                        using fseconds = std::chrono::duration<float, std::ratio<1, 1>>;\n                        auto iops = total_ops / std::chrono::duration_cast<fseconds>(end - start).count();\n                        fmt::print(\"{:10} {:10} {:10} {:12}\\n\", \"bufsize\", \"ops\", \"iodepth\", \"IOPS\");\n                        fmt::print(\"{:10d} {:10d} {:10d} {:12.0f}\\n\", buffer_size, total_ops, concurrency, iops);\n                        return os.flush();\n                    }).then([&os] {\n                        return os.close();\n                    });\n                });\n            });\n        });\n    });\n}\n"
  },
  {
    "path": "tests/perf/future_util_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#include <boost/range.hpp>\n#include <boost/range/irange.hpp>\n\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/util/later.hh>\n\nstruct parallel_for_each {\n    std::vector<int> empty_range;\n    std::vector<int> range;\n    int value;\n\n    static constexpr int max_range_size = 100;\n\n    parallel_for_each()\n        : empty_range()\n        , range(boost::copy_range<std::vector<int>>(boost::irange(1, max_range_size)))\n    { }\n};\n\nPERF_TEST_F(parallel_for_each, empty)\n{\n    return seastar::parallel_for_each(empty_range, [] (int) -> future<> {\n        abort();\n    });\n}\n\n[[gnu::noinline]]\nfuture<> immediate(int v, int& vs)\n{\n    vs += v;\n    return make_ready_future<>();\n}\n\nstruct chain {\n    static constexpr int scale = 32;\n    int value = 0;\n\n    [[gnu::noinline]]\n    future<> do_then(future<> w, int depth = scale) {\n        if (depth > 0) {\n            return do_then(std::move(w), depth - 1).then([this] {\n                value += 1;\n                perf_tests::do_not_optimize(value);\n                return make_ready_future<>();\n            });\n        } else {\n            return w;\n        }\n    }\n\n    [[gnu::noinline]]\n    future<> do_await(future<> w, int depth = scale) {\n        if (depth > 0) {\n            co_await do_await(std::move(w), depth - 1);\n            value += 1;\n            perf_tests::do_not_optimize(value);\n        } else {\n            co_await std::move(w);\n        }\n    }\n\n    [[gnu::noinline]]\n    future<std::nullopt_t> do_await_as_future(future<std::nullopt_t> w, int depth = scale) {\n        if (depth > 0) {\n            auto fut = co_await coroutine::as_future<std::nullopt_t>(do_await_as_future(std::move(w), depth - 1));\n            if (fut.failed()) {\n                co_await seastar::coroutine::return_exception_ptr(fut.get_exception());\n            }\n            value += 1;\n            perf_tests::do_not_optimize(value);\n            co_return fut.get();\n        } else {\n            auto fut = co_await coroutine::as_future<std::nullopt_t>(std::move(w));\n            if (fut.failed()) {\n                co_await seastar::coroutine::return_exception_ptr(fut.get_exception());\n            }\n            co_return fut.get();\n        }\n    }\n};\n\nPERF_TEST_F(chain, then_value)\n{\n    promise<> p;\n    auto f = do_then(p.get_future());\n    p.set_value();\n    return f.then([] { return scale; });\n}\n\nPERF_TEST_F(chain, await_value)\n{\n    promise<> p;\n    auto f = do_await(p.get_future());\n    p.set_value();\n    return f.then([] { return scale; });\n}\n\nPERF_TEST_F(chain, await_value_as_future)\n{\n    promise<std::nullopt_t> p;\n    auto f = do_await_as_future(p.get_future());\n    p.set_value(std::nullopt);\n    return f.then([](std::nullopt_t) { return scale; });\n}\n\nPERF_TEST_F(chain, then_exception)\n{\n    promise<> p;\n    auto f = do_then(p.get_future());\n    p.set_exception(std::runtime_error(\"\"));\n    return f.then_wrapped([] (auto x) {\n        x.ignore_ready_future();\n        return scale;\n    });\n}\n\nPERF_TEST_F(chain, await_exception)\n{\n    promise<> p;\n    auto f = do_await(p.get_future());\n    p.set_exception(std::runtime_error(\"\"));\n    return f.then_wrapped([] (auto x) {\n        x.ignore_ready_future();\n        return scale;\n    });\n}\n\nPERF_TEST_F(chain, await_exception_as_future)\n{\n    promise<std::nullopt_t> p;\n    auto f = do_await_as_future(p.get_future());\n    p.set_exception(std::runtime_error(\"\"));\n    return f.then_wrapped([] (auto x) {\n        x.ignore_ready_future();\n        return scale;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, immediate_1)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 1;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 1;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, immediate_2)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 2;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 2;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, immediate_10)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 10;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 10;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, immediate_100)\n{\n    return seastar::parallel_for_each(range, [this] (int v) {\n        return immediate(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return range.size();\n    });\n}\n\n[[gnu::noinline]]\nfuture<> suspend(int v, int& vs)\n{\n    vs += v;\n    return yield();\n}\n\nPERF_TEST_F(parallel_for_each, suspend_1)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 1;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 1;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, suspend_2)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 2;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 2;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, suspend_10)\n{\n    auto&& begin = range.begin();\n    auto&& end = begin + 10;\n    return seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return 10;\n    });\n}\n\nPERF_TEST_F(parallel_for_each, suspend_100)\n{\n    return seastar::parallel_for_each(range, [this] (int v) {\n        return suspend(v, value);\n    }).then([this] {\n        perf_tests::do_not_optimize(value);\n        return range.size();\n    });\n}\n\nPERF_TEST_C(parallel_for_each, cor_empty)\n{\n    co_await seastar::parallel_for_each(empty_range, [] (int) -> future<> {\n        abort();\n    });\n}\n\nPERF_TEST_CN(parallel_for_each, cor_immediate_1)\n{\n    constexpr size_t n = 1;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_immediate_2)\n{\n    constexpr size_t n = 2;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_immediate_10)\n{\n    constexpr size_t n = 10;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_immediate_100)\n{\n    co_await seastar::parallel_for_each(range, [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return range.size();\n}\n\nPERF_TEST_CN(parallel_for_each, cor_suspend_1)\n{\n    constexpr size_t n = 1;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_suspend_2)\n{\n    constexpr size_t n = 2;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_suspend_10)\n{\n    constexpr size_t n = 10;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_suspend_100)\n{\n    co_await seastar::parallel_for_each(range, [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return range.size();\n}\n\nPERF_TEST_C(parallel_for_each, cor_pfe_empty)\n{\n    co_await seastar::coroutine::parallel_for_each(empty_range, [] (int) -> future<> {\n        abort();\n    });\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_immediate_1)\n{\n    constexpr size_t n = 1;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_immediate_2)\n{\n    constexpr size_t n = 2;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_immediate_10)\n{\n    constexpr size_t n = 10;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_immediate_100)\n{\n    co_await seastar::coroutine::parallel_for_each(range, [this] (int v) {\n        return immediate(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return range.size();\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_suspend_1)\n{\n    constexpr size_t n = 1;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_suspend_2)\n{\n    constexpr size_t n = 2;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_suspend_10)\n{\n    constexpr size_t n = 10;\n    auto&& begin = range.begin();\n    auto&& end = begin + n;\n    co_await seastar::coroutine::parallel_for_each(std::move(begin), std::move(end), [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return n;\n}\n\nPERF_TEST_CN(parallel_for_each, cor_pfe_suspend_100)\n{\n    co_await seastar::coroutine::parallel_for_each(range, [this] (int v) {\n        return suspend(v, value);\n    });\n    perf_tests::do_not_optimize(value);\n    co_return range.size();\n}\n\nstruct sched {\n    bool once = false;\n    seastar::scheduling_supergroup s1;\n    seastar::scheduling_supergroup s2;\n    seastar::scheduling_group g1;\n    seastar::scheduling_group g2;\n\n    class dummy final : public seastar::task {\n        seastar::task* peer = nullptr;\n    public:\n        dummy(scheduling_group sg, seastar::task* p) noexcept : task(sg), peer(p) {}\n\n        virtual void run_and_dispose() noexcept override {\n            seastar::schedule(peer);\n        }\n\n        task* waiting_task() noexcept override { return nullptr; }\n    };\n\n    class switcher final : public seastar::task {\n        std::unique_ptr<dummy> bounce;\n        unsigned iterations = 0;\n\n    public:\n        promise<> done;\n\n        switcher(scheduling_group sg1, scheduling_group sg2, unsigned it) noexcept\n            : task(sg1)\n            , bounce(std::make_unique<dummy>(sg2, this))\n            , iterations(it)\n        {}\n\n        virtual void run_and_dispose() noexcept override {\n            if (--iterations > 0) {\n                seastar::schedule(bounce.get());\n                return;\n            }\n\n            done.set_value();\n            delete this;\n        }\n\n        task* waiting_task() noexcept override { return nullptr; }\n    };\n};\n\nPERF_TEST_CN(sched, context_switch)\n{\n    if (!once) {\n        g1 = co_await create_scheduling_group(\"g1\", 100);\n        g2 = co_await create_scheduling_group(\"g2\", 100);\n        once = true;\n    }\n\n    auto* t = new sched::switcher(g1, g2, 1000);\n    auto f = t->done.get_future();\n    perf_tests::start_measuring_time();\n    schedule(t);\n    co_await std::move(f);\n    perf_tests::stop_measuring_time();\n    co_return 1000;\n}\n\n\nPERF_TEST_CN(sched, context_switch_x2)\n{\n    if (!once) {\n        s1 = co_await create_scheduling_supergroup(100);\n        s2 = co_await create_scheduling_supergroup(100);\n        g1 = co_await create_scheduling_group(\"g1\", \"g1\", 100, s1);\n        g2 = co_await create_scheduling_group(\"g2\", \"g2\", 100, s2);\n        once = true;\n    }\n\n    auto* t = new sched::switcher(g1, g2, 1000);\n    auto f = t->done.get_future();\n    perf_tests::start_measuring_time();\n    schedule(t);\n    co_await std::move(f);\n    perf_tests::stop_measuring_time();\n    co_return 1000;\n}\n\nPERF_TEST_CN(sched, context_switch_x1_5)\n{\n    if (!once) {\n        s1 = co_await create_scheduling_supergroup(100);\n        g1 = co_await create_scheduling_group(\"g1\", 100);\n        g2 = co_await create_scheduling_group(\"g2\", \"g2\", 100, s1);\n        once = true;\n    }\n\n    auto* t = new sched::switcher(g1, g2, 1000);\n    auto f = t->done.get_future();\n    perf_tests::start_measuring_time();\n    schedule(t);\n    co_await std::move(f);\n    perf_tests::stop_measuring_time();\n    co_return 1000;\n}\n"
  },
  {
    "path": "tests/perf/http_client_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2024 ScyllaDB\n */\n\n/*\n * The test runs http::experimental::client against minimalistic (see below) server on\n * one shard using single \"in-memory\" connection.\n *\n * The client sents one request at-a-time, waiting for the server response before sending\n * the next one.\n *\n * The server is a fiber that runs on top of the raw connection, reads it up until double\n * CRLF and then responds back with the \"HTTP/1.1 200 OK host: test\" line. So it's not\n * http::server instance, but a lightweight mock.\n *\n * The connection is net::connected_socket wrapper over seastar::queue, not Linux socket.\n */\n\n#include <seastar/core/seastar.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/http/client.hh>\n#include <seastar/http/request.hh>\n#include <seastar/testing/linux_perf_event.hh>\n#include <../../tests/unit/loopback_socket.hh>\n#include <fmt/printf.h>\n#include <string>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nclass server {\n    seastar::server_socket _ss;\n    seastar::connected_socket _cs;\n    std::optional<seastar::input_stream<char>> _in;\n    std::optional<seastar::output_stream<char>> _out;\n    sstring _req;\n\n    future<> run_serve_loop() {\n        while (true) {\n            temporary_buffer<char> buf = co_await _in->read();\n            if (buf.empty()) {\n                co_return;\n            }\n\n            _req += sstring(buf.get(), buf.size());\n            if (_req.ends_with(\"\\r\\n\\r\\n\")) {\n                sstring r200(\"HTTP/1.1 200 OK\\r\\nHost: test\\r\\n\\r\\n\");\n                co_await _out->write(r200);\n                co_await _out->flush();\n                _req = \"\";\n            }\n        }\n    }\n\npublic:\n    server(loopback_connection_factory& lcf) : _ss(lcf.get_server_socket()) {}\n    future<> serve() {\n        return _ss.accept().then([this] (seastar::accept_result ar) {\n            _cs = std::move(ar.connection);\n            _in.emplace(_cs.input());\n            _out.emplace(_cs.output());\n            return run_serve_loop().finally([this] {\n                return when_all(_in->close(), _out->close()).discard_result();\n            });\n        });\n    }\n};\n\nclass loopback_http_factory : public http::experimental::connection_factory {\n    loopback_socket_impl lsi;\npublic:\n    explicit loopback_http_factory(loopback_connection_factory& f) : lsi(f) {}\n    virtual future<connected_socket> make(abort_source* as) override {\n        return lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr()));\n    }\n};\n\nclass client {\n    seastar::http::experimental::client _cln;\n    const unsigned _warmup_limit;\n    const unsigned _limit;\n    linux_perf_event _instructions;\n    linux_perf_event _cpu_cycles;\n\n    struct stats {\n        std::chrono::steady_clock::time_point ts;\n        uint64_t mallocs;\n        uint64_t tasks;\n        uint64_t instructions;\n        uint64_t cpu_cycles;\n    };\n\n    stats stats_snapshot() {\n        return stats {\n            .ts = std::chrono::steady_clock::now(),\n            .mallocs = memory::stats().mallocs(),\n            .tasks = engine().get_sched_stats().tasks_processed,\n            .instructions = _instructions.read(),\n            .cpu_cycles = _cpu_cycles.read(),\n        };\n    }\n\n    future<> make_requests(unsigned nr) {\n        for (unsigned i = 0; i < nr; i++) {\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            co_await _cln.make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                return make_ready_future<>();\n            }, http::reply::status_type::ok);\n        }\n    }\npublic:\n    client(loopback_connection_factory& lcf, unsigned ops, unsigned warmup)\n            : _cln(std::make_unique<loopback_http_factory>(lcf))\n            , _warmup_limit(warmup)\n            , _limit(ops)\n            , _instructions(linux_perf_event::user_instructions_retired())\n            , _cpu_cycles(linux_perf_event::user_cpu_cycles_retired())\n    {}\n    future<> work() {\n        fmt::print(\"Warming up with {} requests\\n\", _warmup_limit);\n        return make_requests(_warmup_limit).then([this] {\n            fmt::print(\"Warmup finished, making {} requests\\n\", _limit);\n            auto start_stats = stats_snapshot();\n            _instructions.enable();\n            _cpu_cycles.enable();\n            return make_requests(_limit).then([this, start_stats] {\n                _instructions.disable();\n                _cpu_cycles.disable();\n                auto end_stats = stats_snapshot();\n                auto delta = std::chrono::duration_cast<std::chrono::duration<double, std::micro>>(end_stats.ts - start_stats.ts) / _limit;\n                auto allocs = double(end_stats.mallocs - start_stats.mallocs) / _limit;\n                auto tasks = double(end_stats.tasks - start_stats.tasks) / _limit;\n                auto insns = (end_stats.instructions - start_stats.instructions) / _limit;\n                auto cycles = (end_stats.cpu_cycles - start_stats.cpu_cycles) / _limit;\n                fmt::print(\"Made {} requests, {:.3f} usec/op, {:.1f} allocs/op, {:.1f} tasks/op, {} insns/op, {} cycles/op\\n\", _limit,\n                        delta.count(), allocs, tasks, insns, cycles);\n            });\n        }).finally([this] {\n            return _cln.close();\n        });\n    }\n};\n\nint main(int ac, char** av) {\n    app_template at;\n    namespace bpo = boost::program_options;\n    at.add_options()\n            (\"total-ops\", bpo::value<unsigned>()->default_value(1000000), \"Total requests to make\")\n            (\"warmup-ops\", bpo::value<unsigned>()->default_value(10000), \"Requests to warm up\")\n            ;\n    return at.run(ac, av, [&at] {\n        auto total_ops = at.configuration()[\"total-ops\"].as<unsigned>();\n        auto warmup_ops = at.configuration()[\"warmup-ops\"].as<unsigned>();\n        return seastar::async([total_ops, warmup_ops] {\n            loopback_connection_factory lcf(1);\n            server srv(lcf);\n            client cln(lcf, total_ops, warmup_ops);\n            when_all(srv.serve(), cln.work()).discard_result().get();\n        });\n    });\n}\n"
  },
  {
    "path": "tests/perf/linux_perf_event.cc",
    "content": "/*\n * Copyright (C) 2021-present ScyllaDB\n */\n\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * This file was copied from Scylla (https://github.com/scylladb/scylla)\n */\n\n#include <seastar/testing/linux_perf_event.hh>\n#include <seastar/util/assert.hh>\n\n#include <linux/perf_event.h>\n#include <linux/hw_breakpoint.h>\n#include <sys/ioctl.h>\n#include <asm/unistd.h>\n\nlinux_perf_event::linux_perf_event(const struct ::perf_event_attr& attr, pid_t pid, int cpu, int group_fd, unsigned long flags) {\n    int ret = syscall(__NR_perf_event_open, &attr, pid, cpu, group_fd, flags);\n    if (ret != -1) {\n        _fd = ret; // ignore failures, can happen in constrained environments such as containers\n    }\n}\n\nlinux_perf_event::~linux_perf_event() {\n    if (_fd != -1) {\n        ::close(_fd);\n    }\n}\n\nlinux_perf_event&\nlinux_perf_event::operator=(linux_perf_event&& x) noexcept {\n    if (this != &x) {\n        if (_fd != -1) {\n            ::close(_fd);\n        }\n        _fd = std::exchange(x._fd, -1);\n    }\n    return *this;\n}\n\nuint64_t\nlinux_perf_event::read() {\n    if (_fd == -1) {\n        return 0;\n    }\n    uint64_t ret;\n    auto res = ::read(_fd, &ret, sizeof(ret));\n    SEASTAR_ASSERT(res == sizeof(ret) && \"read(2) failed on perf_event fd\");\n    return ret;\n}\n\nvoid\nlinux_perf_event::enable() {\n    if (_fd == -1) {\n        return;\n    }\n    ::ioctl(_fd, PERF_EVENT_IOC_ENABLE, 0);\n}\n\nvoid\nlinux_perf_event::disable() {\n    if (_fd == -1) {\n        return;\n    }\n    ::ioctl(_fd, PERF_EVENT_IOC_DISABLE, 0);\n}\n\nstatic linux_perf_event\nmake_linux_perf_event(unsigned config, pid_t pid = 0, int cpu = -1, int group_fd = -1, unsigned long flags = 0) {\n    return linux_perf_event(perf_event_attr{\n            .type = PERF_TYPE_HARDWARE,\n            .size = sizeof(struct perf_event_attr),\n            .config = config,\n            .disabled = 1,\n            .exclude_kernel = 1,\n            .exclude_hv = 1,\n#if defined(__x86_64__)\n            // exclude_idle is not supported on all architectures (e.g. aarch64)\n            // so enable it selectively only on architectures that support it.\n            .exclude_idle = 1,\n#endif\n            }, pid, cpu, group_fd, flags);\n}\n\nlinux_perf_event\nlinux_perf_event::user_instructions_retired() {\n    return make_linux_perf_event(PERF_COUNT_HW_INSTRUCTIONS);\n}\n\nlinux_perf_event\nlinux_perf_event::user_cpu_cycles_retired() {\n    return make_linux_perf_event(PERF_COUNT_HW_CPU_CYCLES);\n}\n"
  },
  {
    "path": "tests/perf/metrics_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <seastar/core/iostream.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/util/tmp_file.hh>\n\n#include <seastar/core/internal/estimated_histogram.hh>\n\n#include <ranges>\n#include <stdexcept>\n\nnamespace sm = seastar::metrics;\n\nnamespace {\n\nauto irange(size_t upper_bound) {\n    return std::ranges::views::iota(size_t(0), upper_bound);\n}\n\nvoid remove_existing_metrics() {\n    const auto& map = seastar::metrics::impl::get_value_map();\n\n    for (auto& family : map) {\n        auto name = family.first;\n        for (auto& series: family.second) {\n            seastar::metrics::impl::unregister_metric(series.second->get_id());\n        }\n    }\n\n    assert(seastar::metrics::impl::get_value_map().size() == 0);\n}\n}\n\n// just records the size of everything written\nstruct counting_data_sink_impl : public data_sink_impl {\n    counting_data_sink_impl(size_t buf_size) : buf_size{buf_size} {}\n\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        written += std::accumulate(bufs.begin(), bufs.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.size(); });\n        return make_ready_future<>();\n    }\n#else\n    virtual future<> put(net::packet data) override {\n        abort();\n    }\n\n    virtual future<> put(temporary_buffer<char> buf) override {\n        written += buf.size();\n        return make_ready_future<>();\n    }\n#endif\n\n    virtual future<> flush() override {\n        return make_ready_future<>();\n    }\n\n    virtual future<> close() override {\n        return make_ready_future<>();\n    }\n\n    virtual size_t buffer_size() const noexcept override {\n        return buf_size;\n    }\n\n    size_t written = 0;\n    size_t buf_size;\n};\n\nclass counting_data_sink : public data_sink {\npublic:\n    counting_data_sink()\n        : data_sink(std::make_unique<counting_data_sink_impl>(32000)) {}\n};\n\n\nnamespace seastar::prometheus::details {\n\nusing data_type = seastar::metrics::impl::data_type;\n\nstruct metrics_perf_fixture {\n\n    static constexpr uint64_t histo_min = 1, histo_max = 1000000;\n    using histo_type = sm::internal::approximate_exponential_histogram<histo_min, histo_max, 1>;\n    const int histo_buckets = histo_type{}.find_bucket_index(-1) + 1;\n\n    const filter_t always_true = [](auto& mi){ return true; };\n\n    template <typename COUNTER_TYPE = double>\n    seastar::future<size_t> run_metrics_bench(\n        size_t group_count, size_t families_per_group, size_t series_per_family, data_type type,\n        bool enable_aggregation = false, bool use_protobuf = false,\n        family_filter_t family_filter = [](std::string_view) { return true; }) {\n        using namespace seastar;\n        using namespace seastar::metrics;\n\n        remove_existing_metrics();\n\n        metric_groups perf_metrics;\n\n        const size_t series_count = group_count * families_per_group * series_per_family;\n\n        constexpr size_t name_length = 50;\n\n        sstring name_template = (sstring(\"gauge\") + sstring(name_length, '_')).substr(0, name_length - 3) + \"{}\";\n        std::string_view name_template_sv = name_template;\n\n        auto label_template = fmt::runtime(\"label_value_medium_long_{}\");\n\n        label label_0{\"some-long-label-name-this-happens-in-real-life\"};\n        label label_1{\"short\"};\n        label label_2(\"fixed-label\");\n\n        std::optional<histo_type> histogram;\n        if (type == data_type::HISTOGRAM) {\n            histogram = histo_type{};\n            for (double v = histo_min; v < histo_max; v *= 1.3) {\n                histogram->add(v);\n            }\n        }\n\n        for (auto group_id : irange(group_count)) {\n            std::vector<metric_definition> defs;\n            for (auto family_id : irange(families_per_group)) {\n                auto name = fmt::format(fmt::runtime(name_template_sv), family_id);\n                auto desc = fmt::format(fmt::runtime(name_template_sv), family_id);\n\n                for (auto label_id : irange(series_per_family)) {\n                    auto label0 = label_0(fmt::format(label_template, label_id));\n                    auto label1 = label_1(fmt::format(label_template, label_id));\n                    auto label2 = label_2(\"a fixed value\");\n                    auto l = label0;\n\n                    std::vector<label_instance> labels{label0, label1, label2};\n\n                    auto impl = [&] {\n                        if (type == data_type::COUNTER) {\n                            return\n                                make_counter(name, description(desc), labels, [] { return (COUNTER_TYPE)123.4; });\n                        } else if (type == data_type::HISTOGRAM) {\n                            return\n                                make_histogram(name, description(desc), labels,\n                                    [histogram]() { return histogram->to_metrics_histogram(); });\n                        }\n                        throw std::runtime_error(\"bad type\");\n                    }();\n\n                    if (enable_aggregation) {\n                       impl.aggregate({label_0, label_1});\n                    }\n\n                    defs.emplace_back(std::move(impl));\n\n                }\n            }\n            perf_metrics.add_group(fmt::format(\"group-{}\", group_id), defs);\n        }\n\n        prometheus::config config{};\n\n        using access = prometheus::details::test_access;\n\n        constexpr int iterations = 100;\n\n        perf_tests::start_measuring_time();\n        for ([[maybe_unused]] auto _: irange(iterations)) {\n            output_stream<char> out{counting_data_sink{}};\n            co_await access{}.write_body(config,\n                write_body_args{\n                    .filter = always_true,\n                    .family_filter = family_filter,\n                    .use_protobuf_format = use_protobuf,\n                    .show_help = true,\n                    .enable_aggregation = enable_aggregation\n                },\n                std::move(out));\n        }\n        perf_tests::stop_measuring_time();\n\n        // if histogram metrics are used there are N buckets per metric, plus 2 for count and sum\n        co_return series_count * iterations * (type == data_type::HISTOGRAM ? histo_buckets + 2 : 1);\n    }\n};\n\nPERF_TEST_CN(metrics_perf_fixture, test_few_metrics) {\n    // only 1 series added, but note there are some seastar metrics\n    // which are registered too, so this gives a very bloated time as\n    co_return co_await run_metrics_bench(1, 1, 1, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_large_families) {\n    // relatively few families, but many series in each, stresses\n    // \"per series\" work\n    co_return co_await run_metrics_bench(1, 1, 10000, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_large_families_int) {\n    // relatively few families, but many series in each, stresses\n    // \"per series\" work\n    co_return co_await run_metrics_bench<size_t>(1, 1, 10000, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_large_families_int_aggr) {\n    // relatively few families, but many series in each, stresses\n    // \"per series\" work\n    co_return co_await run_metrics_bench<size_t>(1, 1, 10000, data_type::COUNTER, true);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_many_families_int) {\n    // many families, each with only 1 series, stresses \"per family\"\n    // work\n    co_return co_await run_metrics_bench<size_t>(1, 10000, 1, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_many_families_int_aggr) {\n    // many families, each with only 1 series, stresses \"per family\"\n    // work\n    co_return co_await run_metrics_bench<size_t>(1, 10000, 1, data_type::COUNTER, true);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_middle_ground) {\n    // the goldilocks version\n    co_return co_await run_metrics_bench(1, 1000, 10, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_middle_ground_int) {\n    // the goldilocks version\n    co_return co_await run_metrics_bench<size_t>(1, 1000, 10, data_type::COUNTER);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_middle_ground_protobuf) {\n    // the goldilocks version, protobuf output\n    // note that protobuf perf is the same with int or float, so we don't both\n    // with the int variants\n    co_return co_await run_metrics_bench(1, 1000, 10, data_type::COUNTER, false, true);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_histogram) {\n    co_return co_await run_metrics_bench(1, 100, 10, data_type::HISTOGRAM);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_histogram_protobuf) {\n    co_return co_await run_metrics_bench(1, 100, 10, data_type::HISTOGRAM, false, true);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_histogram_aggr) {\n    co_return co_await run_metrics_bench(1, 100, 10, data_type::HISTOGRAM, true);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_name_filter_exact_match) {\n    // Many families but filter to only one using exact name match.\n    // Tests overhead of iterating families when most are filtered out.\n    auto filter = make_family_filter({name_filter{\"group-0_gauge______________________________________________0\", false}});\n    co_return co_await run_metrics_bench(1, 1000, 10, data_type::COUNTER, false, false, filter);\n}\n\nPERF_TEST_CN(metrics_perf_fixture, test_name_filter_many_no_match) {\n    // 100 exact name filters, none of which match any metrics.\n    // Tests overhead of checking many filters when none match.\n    std::vector<name_filter> filters;\n    for (int i = 0; i < 100; ++i) {\n        filters.emplace_back(fmt::format(\"group-{}_nomatch\", i), false);\n    }\n    auto filter = make_family_filter(std::move(filters));\n    co_return co_await run_metrics_bench(1, 1000, 10, data_type::COUNTER, false, false, filter);\n}\n\n}\n"
  },
  {
    "path": "tests/perf/perf-tests.md",
    "content": "# perf-tests\n\n`perf-tests` is a simple microbenchmarking framework. Its main purpose is to allow monitoring the impact that code changes have on performance.\n\n## Theory of operation\n\nThe framework performs each test in several runs. During a run the microbenchmark code is executed in a loop and the average time of an iteration is computed. The shown results are median, median absolute deviation, maximum and minimum value of all the runs.\n\n```\nsingle run iterations:    0\nsingle run duration:      1.000s\nnumber of runs:           5\n\ntest                            iterations      median         mad         min         max\ncombined.one_row                    745336   691.218ns     0.175ns   689.073ns   696.476ns\ncombined.single_active                7871    85.271us    76.185ns    85.145us   108.316us\n```\n\n`perf-tests` allows limiting the number of iterations or the duration of each run. In the latter case there is an additional dry run used to estimate how many iterations can be run in the specified time. The measured runs are limited by that number of iterations. This means that there is no overhead caused by timers and that each run consists of the same number of iterations.\n\n### Flags\n\n* `-i <n>` or `--iterations <n>` – limits the number of iterations in each run to no more than `n` (0 for unlimited)\n* `-d <t>` or `--duration <t>` – limits the duration of each run to no more than `t` seconds (0 for unlimited)\n* `-r <n>` or `--runs <n>` – the number of runs of each test to execute\n* `-t <regexs>` or `--tests <regexs>` – executes only tests which names match any regular expression in a comma-separated list `regexs`\n* `--list` – lists all available tests\n* `--overhead-threshold <percent>` – warn if measurement overhead exceeds this percentage (default: 10)\n* `--fail-on-high-overhead` – fail the test run if any test exceeds the overhead threshold\n\n## Example usage\n\n### Simple test\n\nPerformance tests are defined in a similar manner to unit tests. Macro `PERF_TEST(test_group, test_case)` allows specifying the name of the test and the group it belongs to. Microbenchmark can either return nothing or a future.\n\nCompiler may attempt to optimise too much of the test logic. A way of preventing this is passing the final result of all computations to a function `perf_tests::do_not_optimize()`. That function should introduce little to none overhead, but forces the compiler to actually compute the value.\n\n```c++\nPERF_TEST(example, simple1)\n{\n    auto v = compute_value();\n    perf_tests::do_not_optimize(v);\n}\n\nPERF_TEST(example, simple2)\n{\n    return compute_different_value().then([] (auto v) {\n        perf_tests::do_not_optimize(v);\n    });\n}\n```\n\n### Fixtures\n\nAs it is in case of unit tests, performance tests may benefit from using a fixture that would set up a proper environment. Such tests should use macro `PERF_TEST_F(test_group, test_case)`. The test itself will be a member function of a class derivative of `test_group`.\n\nThe constructor and destructor of a fixture are executed in a context of Seastar thread, but the actual test logic is not. The same instance of a fixture will be used all runs (and iterations) of a given test, but a unique fixture is created for each test. In the example below, exactly 2 fixture objects will be created, for `fixture1` and `fixture2` tests. If you want to share setup _between_ test cases, you can use static members as shown below.\n\n```c++\nclass example {\nprotected:\n    data_set _ds1;\n    data_set _ds2;\nprivate:\n    static data_set perpare_data_set();\npublic:\n    example()\n        : _ds1(prepare_data_set())\n        , _ds2(prepare_data_set())\n    { }\n};\n\nPERF_TEST_F(example, fixture1)\n{\n    auto r = do_something_with(_ds1);\n    perf_tests::do_not_optimize(r);\n}\n\nPERF_TEST_F(example, fixture2)\n{\n    auto r = do_something_with(_ds1, _ds2);\n    perf_tests::do_not_optimize(r);\n}\n```\n\n### Custom time measurement\n\nEven with fixtures it may be necessary to do some costly initialization during each iteration. Its impact can be reduced by specifying the exact part of the test that should be measured using functions `perf_tests::start_measuring_time()` and `perf_tests::stop_measuring_time()`.\n\n```c++\nPERF_TEST(example, custom_time_measurement2)\n{\n    auto data = prepare_data();\n    perf_tests::start_measuring_time();\n    do_something(std::move(data));\n    perf_tests::stop_measuring_time();\n}\n\nPERF_TEST(example, custom_time_measurement2)\n{\n    auto data = prepare_data();\n    perf_tests::start_measuring_time();\n    return do_something_else(std::move(data)).finally([] {\n        perf_tests::stop_measuring_time();\n    });\n}\n```\n\n#### Measurement overhead\n\nThe cost of starting and stopping the timers is substantial, about 1μs for a start/stop pair, due to the overhead of reading the performance counters in the kernel. When using manual time measurement with `start_measuring_time()`/`stop_measuring_time()`, you should ensure that the timed region is substantially longer than this overhead to reduce measurement error. A common approach is to use a loop inside the timed region and return the number of iterations from the test method.\n\nIf you do _not_ use these manual methods, the overhead is very low (a few instructions) as the test method is already called in such a loop by the framework, with the time manipulation methods outside that.\n\n##### Overhead column\n\nThe framework tracks and reports the estimated measurement overhead as a percentage in the `overhead` column. This is calculated by:\n\n1. Calibrating the cost of a single `start_measuring_time()`/`stop_measuring_time()` pair at startup\n2. Counting how many times these functions are called during each test run\n3. Computing `overhead% = (call_count × cost_per_call) / measured_runtime × 100`\n\nA high overhead percentage (e.g., >10%) indicates that the timing instrumentation is consuming a significant portion of the measured time, which reduces the accuracy of the results. This typically happens when:\n- The timed region is very short (comparable to the ~1μs overhead)\n- `start_measuring_time()`/`stop_measuring_time()` are called many times with little work between them\n\nTo reduce overhead, either:\n- Increase the amount of work in each timed region\n- Use an inner loop and return the iteration count, rather than calling start/stop for each iteration\n\n##### Overhead warnings\n\nBy default, a warning is printed if any test has median overhead exceeding 10%:\n\n```\nWARNING: test 'example.my_test' has high measurement overhead: 15.2% (threshold: 10.0%)\n```\n\nYou can adjust the threshold with `--overhead-threshold <percent>`, or fail the test run entirely when overhead is too high with `--fail-on-high-overhead`.\n"
  },
  {
    "path": "tests/perf/perf_tests.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#include <algorithm>\n#include <boost/program_options/value_semantic.hpp>\n#include <cmath>\n#include <optional>\n#include <ranges>\n#include <seastar/testing/perf_tests.hh>\n\n#include <cstdio>\n#include <fstream>\n#include <regex>\n#include <type_traits>\n#include <utility>\n#include <string_view>\n#include <array>\n\n#include <fmt/ostream.h>\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/util/later.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/reactor.hh>\n\n#include <signal.h>\n\nnamespace perf_tests::internal {\n    struct duration;\n}\ntemplate <> struct fmt::formatter<perf_tests::internal::duration> : fmt::ostream_formatter {};\n\nnamespace perf_tests {\nnamespace internal {\n\nusing namespace std::chrono_literals;\n\nnamespace {\n\n// We need to use signal-based timer instead of seastar ones so that\n// tests that do not suspend can be interrupted.\n// This causes no overhead though since the timer is used only in a dry run.\nclass signal_timer {\n    std::function<void()> _fn;\n    timer_t _timer;\npublic:\n    explicit signal_timer(std::function<void()> fn) : _fn(fn) {\n        sigevent se{};\n        se.sigev_notify = SIGEV_SIGNAL;\n        se.sigev_signo = SIGALRM;\n        se.sigev_value.sival_ptr = this;\n        auto ret = timer_create(CLOCK_MONOTONIC, &se, &_timer);\n        if (ret) {\n            throw std::system_error(ret, std::system_category());\n        }\n    }\n\n    ~signal_timer() {\n        timer_delete(_timer);\n    }\n\n    void arm(std::chrono::steady_clock::duration dt) {\n        time_t sec = std::chrono::duration_cast<std::chrono::seconds>(dt).count();\n        auto nsec = std::chrono::duration_cast<std::chrono::nanoseconds>(dt).count();\n        nsec -= std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::seconds(sec)).count();\n\n        itimerspec ts{};\n        ts.it_value.tv_sec = sec;\n        ts.it_value.tv_nsec = nsec;\n        auto ret = timer_settime(_timer, 0, &ts, nullptr);\n        if (ret) {\n            throw std::system_error(ret, std::system_category());\n        }\n    }\n\n    void cancel() {\n        itimerspec ts{};\n        auto ret = timer_settime(_timer, 0, &ts, nullptr);\n        if (ret) {\n            throw std::system_error(ret, std::system_category());\n        }\n    }\npublic:\n    static void init() {\n        struct sigaction sa{};\n        sa.sa_sigaction = &signal_timer::signal_handler;\n        sa.sa_flags = SA_SIGINFO;\n        auto ret = sigaction(SIGALRM, &sa, nullptr);\n        if (ret) {\n            throw std::system_error(ret, std::system_category());\n        }\n    }\nprivate:\n    static void signal_handler(int, siginfo_t* si, void*) {\n        auto t = static_cast<signal_timer*>(si->si_value.sival_ptr);\n        t->_fn();\n    }\n};\n\n}\n\nuint64_t perf_stats::perf_mallocs() {\n    return memory::stats().mallocs();\n}\n\nuint64_t perf_stats::perf_tasks_processed() {\n    return engine().get_sched_stats().tasks_processed;\n}\n\nperf_stats perf_stats::snapshot(linux_perf_event* instructions_retired_counter, linux_perf_event* cpu_cycles_retired_counter) {\n    return perf_stats(\n        perf_mallocs(),\n        perf_tasks_processed(),\n        instructions_retired_counter ? instructions_retired_counter->read() : 0,\n        cpu_cycles_retired_counter ? cpu_cycles_retired_counter->read() : 0\n    );\n}\n\nclass time_measurement {\n    clock_type::time_point _run_start_time;\n    clock_type::time_point _start_time;\n    clock_type::duration _total_time;\n\n    perf_stats _start_stats;\n    perf_stats _total_stats;\n\n    linux_perf_event _instructions_retired_counter;\n    linux_perf_event _cpu_cycles_retired_counter;\n\n    uint64_t _start_stop_count = 0;\n\n    // Calibrated cost of a single start_measuring_time()/stop_measuring_time() pair\n    clock_type::duration _start_stop_overhead{0};         // internal timing (used for overhead calculations)\n    clock_type::duration _start_stop_overhead_external{0}; // external clock measurement (for comparison)\n\npublic:\n    explicit time_measurement(bool perf_counters)\n        : _instructions_retired_counter(perf_counters ? linux_perf_event::user_instructions_retired() : linux_perf_event::always_zero())\n        , _cpu_cycles_retired_counter(perf_counters ? linux_perf_event::user_cpu_cycles_retired() : linux_perf_event::always_zero())\n    {}\n\n    void enable_counters() {\n        _instructions_retired_counter.enable();\n        _cpu_cycles_retired_counter.enable();\n    }\n\n    void disable_counters() {\n        _instructions_retired_counter.disable();\n        _cpu_cycles_retired_counter.disable();\n    }\n\n    void start_run() {\n        _total_time = { };\n        _total_stats = {};\n        _start_stop_count = 1;  // Start at 1 because we start timing here\n        auto t = clock_type::now();\n        _run_start_time = t;\n        _start_time = t;\n        _start_stats = perf_stats::snapshot(&_instructions_retired_counter, &_cpu_cycles_retired_counter);\n    }\n\n    performance_test::run_result stop_run() {\n        auto t = clock_type::now();\n        performance_test::run_result ret;\n        if (_start_time == _run_start_time) {\n            ret.duration = t - _start_time;\n            auto stats = perf_stats::snapshot(&_instructions_retired_counter, &_cpu_cycles_retired_counter);\n            ret.stats = stats - _start_stats;\n        } else {\n            ret.duration = _total_time;\n            ret.stats = _total_stats;\n        }\n        ret.start_stop_count = _start_stop_count;\n        return ret;\n    }\n\n    void start_iteration() {\n        ++_start_stop_count;\n        _start_time = clock_type::now();\n        _start_stats = perf_stats::snapshot(&_instructions_retired_counter, &_cpu_cycles_retired_counter);\n    }\n\n    void stop_iteration() {\n        auto t = clock_type::now();\n        _total_time += t - _start_time;\n        perf_stats stats = perf_stats::snapshot(&_instructions_retired_counter, &_cpu_cycles_retired_counter);\n        _total_stats += stats - _start_stats;\n    }\n\n    clock_type::duration start_stop_overhead() const { return _start_stop_overhead; }\n    clock_type::duration start_stop_overhead_external() const { return _start_stop_overhead_external; }\n\n    void calibrate_overhead() {\n        constexpr int CALIBRATION_LOOPS = 1000;\n\n        // Warm up\n        for (int i = 0; i < 100; i++) {\n            start_iteration();\n            stop_iteration();\n        }\n\n        // Use internal timing via start_run/stop_run (same as real tests)\n        enable_counters();\n        start_run();\n        for (int i = 0; i < CALIBRATION_LOOPS; i++) {\n            start_iteration();\n            stop_iteration();\n        }\n        auto result = stop_run();\n        disable_counters();\n\n        _start_stop_overhead = result.duration / CALIBRATION_LOOPS;\n\n        // Also measure with external clock for comparison\n        auto start = clock_type::now();\n        for (int i = 0; i < CALIBRATION_LOOPS; i++) {\n            start_iteration();\n            stop_iteration();\n        }\n        auto end = clock_type::now();\n\n        _start_stop_overhead_external = (end - start) / CALIBRATION_LOOPS;\n    }\n};\n\nstatic std::optional<time_measurement> measure_time;\n\nvoid performance_test::start_run() {\n    measure_time->enable_counters();\n    measure_time->start_run();\n}\n\nperformance_test::run_result performance_test::stop_run() {\n    auto ret = measure_time->stop_run();\n    measure_time->disable_counters();\n    return ret;\n}\n\nvoid time_measurement_start_iteration() {\n    measure_time->start_iteration();\n}\n\nvoid time_measurement_stop_iteration() {\n    measure_time->stop_iteration();\n}\n\nstruct config;\nstruct result;\n\nstruct result_printer {\n    virtual ~result_printer() = default;\n\n    virtual void print_configuration(const config&) = 0;\n    virtual void print_result(const result&) = 0;\n    virtual void print_summary(clock_type::duration /*total_duration*/) { }\n\n    void update_name_column_length(size_t length) {\n        _name_column_length = std::max(1ul, length);\n    }\n    size_t name_column_length() const {\n        return _name_column_length;\n    }\n\nprivate:\n    static constexpr size_t DEFAULT_NAME_COLUMN_LENGTH = 40;\n    size_t _name_column_length = DEFAULT_NAME_COLUMN_LENGTH;\n};\n\nstruct config {\n    uint64_t single_run_iterations;\n    std::chrono::nanoseconds single_run_duration;\n    unsigned number_of_runs;\n    std::vector<std::unique_ptr<result_printer>> printers;\n    unsigned random_seed = 0;\n    double overhead_threshold = 0.1;  // warn if overhead exceeds this ratio (e.g., 0.1 = 10%)\n    bool fail_on_high_overhead = false;  // fail the test run if overhead exceeds threshold\n};\n\n// absorbs a single metric across all runs and calculates summary statistics\n// on it\ntemplate <typename Wrapper = double>\nstruct float_stats {\n    std::vector<double> results_;\n\n    float_stats(size_t runs) {\n        results_.reserve(runs);\n    }\n\n    // wrap a double value in the Wrapper type, e.g. to distinguish durations\n    // from plain doubles\n    static Wrapper wrap(double value) {\n        return Wrapper{value};\n    }\n\n    // add a given value (over all iterations) and a number of iterations\n    // the recorded value will be value / iterations\n    void add(double value, double iterations) {\n        results_.push_back(value / iterations);\n    }\n\n    // summary stats\n    struct summary {\n        double avg, med, min, max, mad;\n    };\n\n    summary stats() const {\n        auto results = results_;                 // work on a copy\n        const size_t count = results.size();\n        assert(count);\n        const size_t mid = count / 2;\n        std::sort(results.begin(), results.end());\n        const double median = results[mid];\n        std::vector<double> diffs(results.begin(), results.end());\n        for (auto& d : diffs) { d = std::fabs(d - median); }\n        std::sort(diffs.begin(), diffs.end());\n        const double sum = std::accumulate(results.begin(), results.end(), 0.0);\n        return { sum / count, median, results.front(), results.back(), diffs[mid] };\n    }\n};\n\nstruct result {\n    result(size_t run_count) :\n        runtime{run_count},\n        allocs{run_count},\n        tasks{run_count},\n        inst{run_count},\n        cycles{run_count},\n        overhead{run_count}\n        {}\n\n    sstring test_name = \"\";\n\n    uint64_t total_iterations = 0;\n    unsigned runs = 0;\n\n    float_stats<duration> runtime;\n\n    float_stats<> allocs;\n    float_stats<> tasks;\n    float_stats<> inst;\n    float_stats<> cycles;\n    float_stats<> overhead;  // overhead percentage from start/stop timing calls\n};\n\nstruct duration {\n    double value;  // in nanoseconds\n\n    constexpr explicit duration(double nanos = 0) : value(nanos) {}\n\n    template <typename Rep, typename Period>\n    constexpr explicit duration(std::chrono::duration<Rep, Period> d)\n        : value(d / 1ns) {}\n};\n\nstruct scaled_duration {\n    double scaled;\n    std::string_view unit; // one of ns, us, ms, s\n};\n\nstruct text_options {\n    std::FILE *file = nullptr;\n    std::set<sstring> mad_columns;\n};\n\nstruct format_options {\n    size_t width = 0;      // 0 means 'unspecified/disabled'\n    int precision = -1;    // -1 triggers adaptive reduction logic\n};\n\n// compute the apparent width of a utf-8 string for terminal display.\n// The only multi-byte UTF-8 characters we use are ± (U+00B1) and µ (U+00B5),\n// both of which are 2 bytes but display as 1 column. All other characters are ASCII.\n// Use a concept so we accept either std::string_view directly or any type exposing\n// data()/size() (e.g. seastar::sstring) without creating multiple overloads.\nsize_t apparent_width(std::convertible_to<std::string_view> auto sv) {\n    using namespace std::string_view_literals;\n    std::string_view view{sv};\n\n    // Count occurrences of ± (UTF-8: 0xC2 0xB1) and µ (UTF-8: 0xC2 0xB5)\n    // Each is 2 bytes but displays as 1 column, so subtract 1 for each occurrence\n    auto count_subsequence = [](std::string_view haystack, std::string_view needle) {\n        size_t count = 0;\n        for (auto pos = haystack; !pos.empty(); ) {\n            auto found = std::ranges::search(pos, needle);\n            if (found.empty()) {\n                break;\n            }\n            ++count;\n            pos = std::string_view(found.end(), pos.end());\n        }\n        return count;\n    };\n\n    size_t multibyte_count = 0;\n    for (auto pattern : {\"\\xC2\\xB1\"sv, \"\\xC2\\xB5\"sv}) {\n        multibyte_count += count_subsequence(view, pattern);\n    }\n\n    return view.size() - multibyte_count;\n}\n\n// Definition of adaptive formatter (see forward declaration above).\nstatic sstring format_double_fit(double value, size_t width, size_t default_precision) {\n    if (!width || default_precision < 0) {\n        return sstring(\"\");\n    }\n\n    auto fits = [width](const sstring& s) {\n        return apparent_width(s) <= width;\n    };\n\n    auto make_hash_fallback = [width]() {\n        auto r = uninitialized_string(width);\n        for (size_t i = 0; i < width; ++i) {\n            r.data()[i] = '#';\n        }\n        return r;\n    };\n\n    if (default_precision > 18) { // more than ~18 digits rarely useful for double\n        default_precision = 18;\n    }\n\n    // 1 & 2: fixed format with decreasing precision.\n    for (int p = default_precision; p >= 0; --p) {\n        // Build a runtime format specification for fixed format.\n        auto fmt_spec = fmt::format(\"{{:>{}.{}f}}\", width, p);\n        auto s = fmt::format(fmt::runtime(fmt_spec), value);\n        if (fits(s)) {\n            return sstring(s);\n        }\n    }\n\n    // 3: scientific notation with decreasing precision.\n    // Note: scientific adds characters like e+NN which may help for very large/small values.\n    for (int p = default_precision; p >= 0; --p) {\n        auto fmt_spec = fmt::format(\"{{:>{}.{}e}}\", width, p);\n        auto s = fmt::format(fmt::runtime(fmt_spec), value);\n        if (fits(s)) {\n            return sstring(s);\n        }\n    }\n\n    // 4: last resort.\n    return make_hash_fallback();\n}\n\n\n// Given a value in nanoseconds, scale to the first unit whose value is < 1000.\n// Progression: ns -> us -> ms -> s.\nstatic inline scaled_duration calculate_units_and_scale(double nanoseconds) {\n    static const std::array units = {\n        \"ns\", \"µs\", \"ms\"\n    };\n\n    double scaled = nanoseconds;\n    for (auto unit : units) {\n        if (scaled < 1000.) {\n            return {scaled, unit};\n        }\n        scaled /= 1000.0;\n    }\n    return {scaled, \"s\"};\n}\n\ninline std::ostream& operator<<(std::ostream& os, duration d) {\n    auto sd = calculate_units_and_scale(d.value);\n    // fmt hasn't discovered unicode yet so we are stuck with 'us' not 'µs'\n    os << fmt::format(\"{:.3f}{}\", sd.scaled, sd.unit);\n    return os;\n}\n\n\n// Format a raw duration value expressed in nanoseconds, scaling to the most\n// suitable unit (ns, µs, ms, s) and fitting the numeric portion into the\n// provided width minus the unit suffix length using the same adaptive logic\n// as format_double_fit.\nstatic sstring format_duration_fit(double nanoseconds, size_t width, int precision) {\n    auto sd = calculate_units_and_scale(nanoseconds);\n    size_t unit_len = apparent_width(sd.unit);\n    size_t avail_width = (width > unit_len) ? (width - unit_len) : size_t(1);\n    auto num_part = format_double_fit(sd.scaled, avail_width, precision);\n    assert(apparent_width(num_part) == avail_width);\n    return fmt::format(\"{}{}\", num_part, sd.unit);\n}\n\nstruct printer {\n    format_options fopts{};\n    bool include_mad = false;\n\n    sstring operator()(duration t) const {\n        return format_duration_fit(t.value, fopts.width, fopts.precision);\n    }\n    sstring operator()(double d) const {\n        return format_double_fit(d, fopts.width, fopts.precision);\n    }\n    template <typename T>\n    sstring operator()(const float_stats<T>& t) const {\n        auto st = t.stats();\n        sstring ret = (*this)(T{st.med});\n        if (include_mad) {\n            double rel_med = st.mad / st.med * 100.;\n            if (rel_med != rel_med) { rel_med = 0; }\n            ret += fmt::format(\" \\u00B1{:5.2f}%\", rel_med);\n        }\n        return ret;\n    }\n};\n\ntemplate <typename T>\nstruct value_traits {\n    static constexpr T sample_value{};\n};\n\n\ntemplate <typename T>\nstruct value_traits<float_stats<T>> {\n    static const float_stats<T> sample_value;\n};\n\ntemplate<typename T>\nconst float_stats<T> value_traits<float_stats<T>>::sample_value = [] {\n    float_stats<T> f{1};\n    f.add(1., 1.);\n    return f;\n}();\n\n\n/**\n * A column object encapsulates the logic needed to print one\n * type of result value, usually as a column (or a json attribute).\n *\n * This allows all printers to share a common view of the available\n * columns.\n */\nstruct column {\n    static constexpr int default_width = 9;\n\n    using print_fn = std::function<void(const text_options&, const result&)>;\n    using header_fn = std::function<size_t(const text_options&)>;\n\n    template <typename F>\n    column(sstring header, int prec, F fn) : header{header}, fopts{default_width, prec} {\n        using result_t = std::invoke_result_t<F, const result&>;\n        using result_traits = value_traits<result_t>;\n\n        print_text = [=, fopts = fopts](const text_options& opts, const result& r) {\n            printer p{fopts, opts.mad_columns.contains(header)};\n            fmt::print(opts.file, \"{}\", p(fn(r)));\n        };\n\n        header_size = [=, fopts = fopts](const text_options& opts) {\n            printer p{fopts, opts.mad_columns.contains(header)};\n            auto v = p(result_traits::sample_value);\n            return apparent_width(std::string_view(v));\n        };\n\n        to_double = [fn](const result& r) {\n            return column::convert_to_double(fn(r));\n        };\n    }\n\n    static double convert_to_double(duration d) {\n        return d.value;\n    }\n\n    static double convert_to_double(double d) {\n        return d;\n    }\n\n    template <typename T>\n    static double convert_to_double(float_stats<T> s) {\n        return s.stats().med;\n    }\n\n    void print_header(text_options topts, const char* str = nullptr) const {\n        size_t width = header_size(topts); // calculate header size by actually formatting a representative value\n        fmt::print(topts.file, \"{:>{}}\", str ? str : header, width);\n    }\n\n    // column header\n    sstring header;\n\n    format_options fopts;\n\n    // used by stdout and md formats to print as text\n    print_fn print_text;\n\n    header_fn header_size;\n\n    // used by json format to extract double result\n    std::function<double(const result&)> to_double;\n};\n\nusing columns = std::vector<column>;\n\nstatic const std::vector<column> common_columns{\n    {\"allocs\"  ,  3, [](const result& r) { return r.allocs;              }},\n    {\"tasks\"   ,  3, [](const result& r) { return r.tasks;               }},\n    {\"inst\"    ,  2, [](const result& r) { return r.inst;                }},\n    {\"cycles\"  ,  1, [](const result& r) { return r.cycles;              }},\n    {\"overhead\",  3, [](const result& r) { return r.overhead;            }},\n};\n// json columns\nstatic const std::vector<column> json_columns = [] {\n    columns v{\n        {\"median\", 0, [](const result& r) { return duration { r.runtime.stats().med }; }},\n        {\"mad\"   , 0, [](const result& r) { return duration { r.runtime.stats().mad }; }},\n        {\"min\"   , 0, [](const result& r) { return duration { r.runtime.stats().min }; }},\n        {\"max\"   , 0, [](const result& r) { return duration { r.runtime.stats().max }; }},\n    };\n    v.insert(v.end(), common_columns.begin(), common_columns.end());\n    return v;\n}();\n\n// text columns\nstatic const columns text_columns = [] {\n    columns v{\n        {\"iters\"   , 0, [](const result& r) { return 1. * r.total_iterations / r.runs; }},\n        {\"runtime\" , 2, [](const result& r) { return r.runtime; }}\n    };\n    v.insert(v.end(), common_columns.begin(), common_columns.end());\n    return v;\n}();\n\nstruct delimiters {\n    const char* start_delim = \"\";\n    const char* middle_delim = \"  \";\n    const char* end_delim = \"\";\n};\n\nstruct text_printer : public result_printer {\n    text_printer(const columns& columns, const text_options& opts, delimiters delims = {}) :\n        columns_{columns},\n        opts{opts},\n        delims{delims}\n    {\n        assert(opts.file && delims.start_delim && delims.middle_delim && delims.end_delim);\n    }\n\n    void print_header_row(const char* first_col = \"test\", const char* header_override = nullptr) {\n        // start delimeter, and the top-left cell\n        fmt::print(opts.file, \"{}{:<{}}\", delims.start_delim, first_col, name_column_length());\n        // the header cells\n        for (auto& c : columns_) {\n            // middle delimeter\n            fmt::print(opts.file, \"{}\", delims.middle_delim);\n            // right align the text in result cells\n            c.print_header(opts, header_override);\n        }\n        // end delimeter\n        fmt::print(opts.file, \"{}\\n\", delims.end_delim);\n    }\n\n    virtual void print_result(const result& r) override {\n        fmt::print(opts.file, \"{}{:<{}}\", delims.start_delim, r.test_name, name_column_length());\n        for (auto& c : columns_) {\n            fmt::print(opts.file, \"{}\", delims.middle_delim);\n            c.print_text(opts, r);\n        }\n        fmt::print(opts.file, \"{}\\n\", delims.end_delim);\n    }\n\n    columns columns_;\n    text_options opts;\n    delimiters delims;\n};\n\n\n\nstruct stdout_printer : text_printer {\n\n    stdout_printer(const columns& columns, text_options options)\n        : text_printer(columns, options) {}\n\n    virtual void print_configuration(const config& c) override {\n        fmt::print(\"{:<25} {}\\n{:<25} {}\\n{:<25} {}\\n{:<25} {}\\n{:<25} {}\\n{:<25} {} ({})\\n\\n\",\n                \"single run iterations:\", c.single_run_iterations,\n                \"single run duration:\", duration { double(c.single_run_duration.count()) },\n                \"number of runs:\", c.number_of_runs,\n                \"number of cores:\", smp::count,\n                \"random seed:\", c.random_seed,\n                \"start/stop overhead:\", duration { measure_time->start_stop_overhead() },\n                duration { measure_time->start_stop_overhead_external() });\n\n        print_header_row();\n    }\n};\n\nclass json_printer final : public result_printer {\n    std::string _output_file;\n    std::unordered_map<std::string,\n                       std::unordered_map<std::string,\n                                          std::unordered_map<std::string, double>>> _root;\n    std::unordered_map<std::string, double> _summary;\npublic:\n    explicit json_printer(const std::string& file) : _output_file(file) { }\n\n    ~json_printer() {\n        std::ofstream out(_output_file);\n        out << \"{\\\"results\\\":\" << json::formatter::to_json(_root[\"results\"])\n            << \",\\\"summary\\\":\"  << json::formatter::to_json(_summary) << \"}\";\n    }\n\n    virtual void print_configuration(const config&) override { }\n\n    virtual void print_result(const result& r) override {\n        auto& result = _root[\"results\"][r.test_name];\n        result[\"runs\"] = r.runs;\n        result[\"total_iterations\"] = r.total_iterations;\n\n        for (auto& c : json_columns) {\n            result[c.header] = c.to_double(r);\n        }\n    }\n\n    virtual void print_summary(clock_type::duration total_duration) override {\n        _summary[\"total_runtime_s\"] = std::chrono::duration<double>(total_duration).count();\n    }\n};\n\nstruct markdown_printer final : public text_printer {\n    constexpr static delimiters md_delims{\"| \", \" | \", \" |\"};\n\n    explicit markdown_printer(const columns& columns, text_options options)\n        : text_printer(columns, options, md_delims) {}\n\n    ~markdown_printer() {\n        if (opts.file != stdout) {\n            std::fclose(opts.file);\n        }\n    }\n\n    void print_configuration(const config&) override {\n        // print the header row\n        print_header_row(\"test\", nullptr);\n        // then the divider row of all -\n        print_header_row(\"-\", \"-:\");\n    }\n};\n\nstatic std::vector<pre_run_hook> pre_run_hooks;\n\nint register_pre_run_hook(pre_run_hook hook) {\n    pre_run_hooks.emplace_back(std::move(hook));\n    return 0; // this return value just helps run this function as a global constructor\n}\n\nvoid performance_test::run_hooks() {\n    for (auto& hook : pre_run_hooks) {\n        hook(test_group(), test_case());\n    }\n}\n\nstatic std::FILE* maybe_open(const sstring& filename) {\n    FILE *ret = filename == \"-\" ? stdout : std::fopen(filename.c_str(), \"w\");\n    if (!ret) {\n        throw std::invalid_argument(fmt::format(\"unable to write to {}\", filename));\n    }\n    return ret;\n}\n\nvoid performance_test::do_run(const config& conf)\n{\n    _max_single_run_iterations = conf.single_run_iterations;\n    if (!_max_single_run_iterations) {\n        _max_single_run_iterations = std::numeric_limits<uint64_t>::max();\n    }\n\n    signal_timer tmr([this] {\n        _max_single_run_iterations.store(0, std::memory_order_relaxed);\n    });\n\n    // dry run, estimate the number of iterations\n    if (conf.single_run_duration.count()) {\n        // switch out of seastar thread\n        yield().then([&] {\n            tmr.arm(conf.single_run_duration);\n            return do_single_run().finally([&] {\n                tmr.cancel();\n                _max_single_run_iterations = _single_run_iterations;\n            });\n        }).get();\n    }\n\n    result r{conf.number_of_runs};\n\n    uint64_t total_iterations = 0;\n    for (auto i = 0u; i < conf.number_of_runs; i++) {\n        // switch out of seastar thread\n        yield().then([&] {\n            _single_run_iterations = 0;\n            return do_single_run().then([&] (run_result rr) {\n                clock_type::duration dt = rr.duration;\n                double ns = std::chrono::duration_cast<std::chrono::nanoseconds>(dt).count();\n\n                auto add = [this](auto& m, double value) {\n                    m.add(value, _single_run_iterations);\n                };\n\n                add(r.runtime, ns);\n\n                total_iterations += _single_run_iterations;\n\n                add(r.allocs, rr.stats.allocations);\n                add(r.tasks, rr.stats.tasks_executed);\n                add(r.inst, rr.stats.instructions_retired);\n                add(r.cycles, rr.stats.cpu_cycles_retired);\n\n                // Calculate overhead ratio from start/stop timing calls\n                auto overhead = rr.start_stop_count * measure_time->start_stop_overhead();\n                double overhead_ratio = dt.count() ? (1.0 * overhead / dt) : 0.;\n                r.overhead.add(overhead_ratio, 1.0);  // already per-run, not per-iteration\n            });\n        }).get();\n    }\n\n    r.test_name = name();\n    r.total_iterations = total_iterations;\n    r.runs = conf.number_of_runs;\n\n    for (auto& rp : conf.printers) {\n        rp->print_result(r);\n    }\n\n    // Check if overhead exceeds threshold and warn/fail\n    double median_overhead = r.overhead.stats().med;\n    if (median_overhead > conf.overhead_threshold) {\n        fmt::print(\"WARNING: test '{}' has high measurement overhead: {:.1f}% (threshold: {:.1f}%)\\n\",\n                   name(), median_overhead * 100, conf.overhead_threshold * 100);\n        if (conf.fail_on_high_overhead) {\n            throw std::runtime_error(fmt::format(\n                \"Test '{}' failed due to high measurement overhead: {:.1f}%\",\n                name(), median_overhead * 100));\n        }\n    }\n}\n\nvoid performance_test::run(const config& conf)\n{\n    set_up();\n    try {\n        do_run(conf);\n    } catch (...) {\n        tear_down();\n        throw;\n    }\n    tear_down();\n}\n\nstd::vector<std::unique_ptr<performance_test>>& all_tests()\n{\n    static std::vector<std::unique_ptr<performance_test>> tests;\n    return tests;\n}\n\nvoid performance_test::register_test(std::unique_ptr<performance_test> test)\n{\n    all_tests().emplace_back(std::move(test));\n}\n\nvoid run_all(const std::vector<std::string>& test_patterns, config& conf) {\n    std::vector<std::regex> regexes;\n    regexes.reserve(test_patterns.size());\n    for (auto& pat : test_patterns) {\n        regexes.emplace_back(pat);\n    }\n    auto match = [&regexes](const performance_test* t) {\n        if (regexes.empty()) { return true; }\n        for (auto& rgx : regexes) {\n            if (std::regex_match(t->name(), rgx)) { return true; }\n        }\n        return false;\n    };\n    size_t max_name_column_length = 0;\n    size_t matched_count = 0;\n    for (auto& t : all_tests()) {\n        if (match(t.get())) {\n            max_name_column_length = std::max(max_name_column_length, t->name().size());\n            ++matched_count;\n        }\n    }\n    if (!regexes.empty() && matched_count == 0) {\n        fmt::print(stderr, \"WARNING: no tests matched the given pattern(s):\");\n        for (auto& pat : test_patterns) {\n            fmt::print(stderr, \" '{}'\", pat);\n        }\n        fmt::print(stderr, \"\\n\");\n        throw std::runtime_error(\"no tests matched the given pattern(s)\");\n    }\n    for (auto& rp : conf.printers) {\n        rp->update_name_column_length(max_name_column_length);\n        rp->print_configuration(conf);\n    }\n    auto run_start = clock_type::now();\n    for (auto& t : all_tests()) {\n        if (match(t.get())) {\n            t->run(conf);\n        }\n    }\n    auto total_duration = clock_type::now() - run_start;\n    for (auto& rp : conf.printers) {\n        rp->print_summary(total_duration);\n    }\n}\n\n}\n}\n\nint main(int ac, char** av)\n{\n    using namespace perf_tests::internal;\n    namespace bpo = boost::program_options;\n\n    app_template app;\n    app.add_options()\n        (\"iterations,i\", bpo::value<size_t>()->default_value(0),\n            \"number of iterations in a single run\")\n        (\"duration,d\", bpo::value<double>()->default_value(1),\n            \"duration of a single run in seconds\")\n        (\"runs,r\", bpo::value<size_t>()->default_value(5), \"number of runs\")\n        (\"test,t\", bpo::value<std::vector<std::string>>(), \"tests to execute\")\n        (\"random-seed,S\", bpo::value<unsigned>()->default_value(0),\n            \"random number generator seed\")\n        (\"no-stdout\", \"do not print to stdout\")\n        (\"json-output\", bpo::value<std::string>(), \"output json file\")\n        (\"md-output\", bpo::value<std::string>(), \"output markdown file\")\n        (\"mad-columns\", bpo::value<std::string>()->default_value(\"runtime\"),\n            \"Either 'all' or comma-separated list of columns for which to show MAD as a percentage of median\")\n        (\"columns\", bpo::value<std::string>()->default_value(\"all\"),\n            \"comma separated list of column (by name) to include in text/md output, or 'all'\")\n        (\"list\", \"list available tests\")\n        (\"overhead-threshold\", bpo::value<double>()->default_value(0.1),\n            \"warn if overhead exceeds this ratio (default: 0.1 = 10%)\")\n        (\"fail-on-high-overhead\", \"fail the test run if any test exceeds the overhead threshold\")\n        (\"no-perf-counters\", \"disable hardware perf counters (inst/cycles)\")\n        ;\n\n    return app.run(ac, av, [&] {\n        return async([&] {\n            signal_timer::init();\n            bool use_counters = !app.configuration().count(\"no-perf-counters\");\n            measure_time.emplace(use_counters);\n            measure_time->calibrate_overhead();\n\n            config conf;\n            conf.single_run_iterations = app.configuration()[\"iterations\"].as<size_t>();\n            auto dur = std::chrono::duration<double>(app.configuration()[\"duration\"].as<double>());\n            conf.single_run_duration = std::chrono::duration_cast<std::chrono::nanoseconds>(dur);\n            conf.number_of_runs = app.configuration()[\"runs\"].as<size_t>();\n            conf.random_seed = app.configuration()[\"random-seed\"].as<unsigned>();\n            conf.overhead_threshold = app.configuration()[\"overhead-threshold\"].as<double>();\n            conf.fail_on_high_overhead = app.configuration().count(\"fail-on-high-overhead\") > 0;\n\n            std::vector<std::string> tests_to_run;\n            if (app.configuration().count(\"test\")) {\n                tests_to_run = app.configuration()[\"test\"].as<std::vector<std::string>>();\n            }\n\n            if (app.configuration().count(\"list\")) {\n                fmt::print(\"available tests:\\n\");\n                for (auto&& t : all_tests()) {\n                    fmt::print(\"\\t{}\\n\", t->name());\n                }\n                return;\n            }\n\n            // local helper to split comma-separated strings\n            auto split = [](const std::string& str) {\n                auto view = str | std::views::split(',') | std::views::transform([](const auto& sub) {\n                    return std::string(sub.begin(), sub.end());\n                });\n\n                return std::vector<std::string>{view.begin(), view.end()};\n            };\n\n            // calculate the effective column set\n            auto selected_cols_str = split(app.configuration()[\"columns\"].as<std::string>());\n            std::set<std::string> selected_set(selected_cols_str.begin(), selected_cols_str.end());\n\n            columns selected_columns;\n            for (const column& col : text_columns) {\n                if (selected_set.contains(\"all\") || selected_set.contains(col.header)) {\n                    selected_columns.emplace_back(col);\n                }\n            }\n\n            auto selected_mad_str = split(app.configuration()[\"mad-columns\"].as<std::string>());\n            if (std::ranges::find(selected_mad_str, \"all\") != selected_mad_str.end()) {\n                selected_mad_str.clear();\n                // add all column names\n                for (const column& col : selected_columns) {\n                    selected_mad_str.emplace_back(col.header);\n                }\n            }\n\n            text_options base_topts{\n                .file = nullptr,\n                .mad_columns{selected_mad_str.begin(), selected_mad_str.end()},\n            };\n\n            if (!app.configuration().count(\"no-stdout\")) {\n                auto topts = base_topts;\n                topts.file = stdout;\n                conf.printers.emplace_back(std::make_unique<stdout_printer>(selected_columns, topts));\n            }\n\n            if (app.configuration().count(\"json-output\")) {\n                conf.printers.emplace_back(std::make_unique<json_printer>(\n                    app.configuration()[\"json-output\"].as<std::string>()\n                ));\n            }\n\n            if (app.configuration().count(\"md-output\")) {\n                auto topts = base_topts;\n                topts.file = maybe_open(app.configuration()[\"md-output\"].as<std::string>());\n                conf.printers.emplace_back(\n                    std::make_unique<markdown_printer>(selected_columns, topts\n                ));\n            }\n\n            if (!conf.random_seed) {\n                conf.random_seed = std::random_device()();\n            }\n            smp::invoke_on_all([seed = conf.random_seed] {\n                auto local_seed = seed + this_shard_id();\n                testing::local_random_engine.seed(local_seed);\n            }).get();\n\n            run_all(tests_to_run, conf);\n        });\n    });\n}\n"
  },
  {
    "path": "tests/perf/perf_tests_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <random>\n#include <seastar/core/future.hh>\n#include <seastar/coroutine/maybe_yield.hh>\n#include <seastar/util/later.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/testing/perf_tests.hh>\n\n// Benchmarks that test raw overhead of almost empty perf tests\n// in all the basic variations.\n\nnamespace {\nvolatile int sink;\nconstexpr auto ITER_COUNT = 100;\nstruct fixture { };\nauto loop(size_t count = ITER_COUNT) {\n    for (size_t i = 0; i < count; i++) {\n        perf_tests::do_not_optimize(i);\n    }\n    return count;\n}\n}\n\nPERF_TEST(perf_tests, test_simple_1) { perf_tests::do_not_optimize(sink); }\n\nPERF_TEST(perf_tests, test_simple_n) { return loop(); }\n\n// do more work in 1 inner iteration to get a high instruction count to help\n// see the variability in the measurements\nPERF_TEST(perf_tests, test_simple_n_big) {\n    loop(10000000);\n    return 1;\n}\n\nPERF_TEST(perf_tests, test_ready_async_1) { return now(); }\n\nPERF_TEST(perf_tests, test_ready_async_n) { return as_ready_future(loop()); }\n\nPERF_TEST(perf_tests, test_unready_async_1) { return yield(); }\n\nPERF_TEST(perf_tests, test_unready_async_n) {\n    auto i = loop();\n    return yield().then([=] { return i; });\n};\n\n\nPERF_TEST(output_check, high_iters) {\n    return 10000;\n}\n\nPERF_TEST(output_check, no_runtime) {\n}\n\nPERF_TEST(output_check, low_runtime) {\n    loop(100);\n}\n\nPERF_TEST(output_check, high_runtime) {\n    loop(1000000);\n}\n\nPERF_TEST(output_check, high_runtime_allocs) {\n    for (size_t i = 0; i < 100000000; i++) {\n        auto* p = new int(i);\n        perf_tests::do_not_optimize(p);\n        delete p;\n    }\n}\n\nPERF_TEST(output_check, highly_variable_runtime) {\n    auto rand = [](size_t max) {\n        static thread_local std::default_random_engine gen(std::random_device{}());\n        return std::uniform_int_distribution<>(0, max)(gen);\n    };\n    size_t iters = rand(2) ? rand(10000000) : rand(100);\n    for (size_t i = 0; i < iters; i++) {\n        auto* p = new int(i);\n        perf_tests::do_not_optimize(p);\n        delete p;\n    }\n}\n\nPERF_TEST_F(fixture, test_fixture_1) { perf_tests::do_not_optimize(sink); }\n\nPERF_TEST_F(fixture, test_fixture_n) { return loop(); }\n\nPERF_TEST_C(fixture, test_coro_1) {\n    // without the next line, compiler will optimize away the coroutine nature of\n    // this function and compile/inline it as a regular function\n    co_await coroutine::maybe_yield();\n}\n\nPERF_TEST_CN(fixture, test_coro_n) {\n    co_await coroutine::maybe_yield();\n    co_return loop();\n}\n\nPERF_TEST(perf_tests, test_empty) { }\n\nPERF_TEST(perf_tests, test_timer_overhead) {\n    constexpr auto TIMER_LOOPS = 1000;\n    for (size_t i = 0; i < TIMER_LOOPS; i++) {\n        perf_tests::start_measuring_time();\n        perf_tests::stop_measuring_time();\n    }\n    return TIMER_LOOPS;\n}\n\n// Test to verify overhead measurement - this test intentionally has high overhead\n// because it does almost no work between start/stop calls\nPERF_TEST(overhead_check, high_overhead_test) {\n    constexpr size_t LOOPS = 1000;\n    for (size_t i = 0; i < LOOPS; i++) {\n        perf_tests::start_measuring_time();\n        perf_tests::do_not_optimize(i);\n        perf_tests::stop_measuring_time();\n    }\n    return LOOPS;\n}\n\n// The following tests run in order check that pre-run hooks are executed properly.\n\nstatic int hook_1_count = 0, hook_2_count = 1;\n\nPERF_PRE_RUN_HOOK([](const std::string& g, const std::string& c) {\n    ++hook_1_count;\n});\n\nPERF_PRE_RUN_HOOK([](const std::string& g, const std::string& c) {\n    ++hook_2_count;\n});\n\nPERF_TEST(hook_checker, hook_did_run_0) {\n    // check in a subsequent test that the hook ran\n    assert(hook_1_count > 0);\n    assert(hook_2_count > 0);\n}\n\n"
  },
  {
    "path": "tests/perf/rpc_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Scylladb, Ltd.\n */\n\n#include <random>\n\n#include <seastar/rpc/lz4_compressor.hh>\n#include <seastar/rpc/lz4_fragmented_compressor.hh>\n\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/testing/random.hh>\n\ntemplate<typename Compressor>\nstruct compression {\n    static constexpr size_t small_buffer_size = 128;\n    static constexpr size_t large_buffer_size = 16 * 1024 * 1024;\n\nprivate:\n    Compressor _compressor;\n\n    seastar::temporary_buffer<char> _small_buffer_random;\n    seastar::temporary_buffer<char> _small_buffer_zeroes;\n\n    std::vector<seastar::temporary_buffer<char>> _large_buffer_random;\n    std::vector<seastar::temporary_buffer<char>> _large_buffer_zeroes;\n\n    std::vector<seastar::temporary_buffer<char>> _small_compressed_buffer_random;\n    std::vector<seastar::temporary_buffer<char>> _small_compressed_buffer_zeroes;\n\n    std::vector<seastar::temporary_buffer<char>> _large_compressed_buffer_random;\n    std::vector<seastar::temporary_buffer<char>> _large_compressed_buffer_zeroes;\n\nprivate:\n    static seastar::rpc::rcv_buf get_rcv_buf(std::vector<temporary_buffer<char>>& input) {\n        if (input.size() == 1) {\n            return seastar::rpc::rcv_buf(input.front().share());\n        }\n        auto bufs = std::vector<temporary_buffer<char>>{};\n        auto total_size = std::accumulate(input.begin(), input.end(), size_t(0),\n            [&] (size_t n, temporary_buffer<char>& buf) {\n                bufs.emplace_back(buf.share());\n                return n + buf.size();\n            });\n        return seastar::rpc::rcv_buf(std::move(bufs), total_size);\n    }\n\n    static seastar::rpc::snd_buf get_snd_buf(std::vector<temporary_buffer<char>>& input) {\n        auto bufs = std::vector<temporary_buffer<char>>{};\n        auto total_size = std::accumulate(input.begin(), input.end(), size_t(0),\n            [&] (size_t n, temporary_buffer<char>& buf) {\n                bufs.emplace_back(buf.share());\n                return n + buf.size();\n            });\n        return seastar::rpc::snd_buf(std::move(bufs), total_size);\n    }\n    static seastar::rpc::snd_buf get_snd_buf(temporary_buffer<char>& input) {\n        return seastar::rpc::snd_buf(input.share());\n    }\n\npublic:\n    compression()\n        : _small_buffer_random(seastar::temporary_buffer<char>(small_buffer_size))\n        , _small_buffer_zeroes(seastar::temporary_buffer<char>(small_buffer_size))\n    {\n        auto& eng = testing::local_random_engine;\n        auto dist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n\n        std::generate_n(_small_buffer_random.get_write(), small_buffer_size, [&] { return dist(eng); });\n        for (auto i = 0u; i < large_buffer_size / seastar::rpc::snd_buf::chunk_size; i++) {\n            _large_buffer_random.emplace_back(seastar::rpc::snd_buf::chunk_size);\n            std::generate_n(_large_buffer_random.back().get_write(), seastar::rpc::snd_buf::chunk_size, [&] { return dist(eng); });\n            _large_buffer_zeroes.emplace_back(seastar::rpc::snd_buf::chunk_size);\n            std::fill_n(_large_buffer_zeroes.back().get_write(), seastar::rpc::snd_buf::chunk_size, 0);\n        }\n\n        auto rcv = _compressor.compress(0, seastar::rpc::snd_buf(_small_buffer_random.share()));\n        if (auto buffer = std::get_if<seastar::temporary_buffer<char>>(&rcv.bufs)) {\n            _small_compressed_buffer_random.emplace_back(std::move(*buffer));\n        } else {\n            _small_compressed_buffer_random\n                = std::move(std::get<std::vector<seastar::temporary_buffer<char>>>(rcv.bufs));\n        }\n\n        rcv = _compressor.compress(0, seastar::rpc::snd_buf(_small_buffer_zeroes.share()));\n        if (auto buffer = std::get_if<seastar::temporary_buffer<char>>(&rcv.bufs)) {\n            _small_compressed_buffer_zeroes.emplace_back(std::move(*buffer));\n        } else {\n            _small_compressed_buffer_zeroes\n                = std::move(std::get<std::vector<seastar::temporary_buffer<char>>>(rcv.bufs));\n        }\n\n        auto bufs = std::vector<temporary_buffer<char>>{};\n        for (auto&& b : _large_buffer_random) {\n            bufs.emplace_back(b.clone());\n        }\n        rcv = _compressor.compress(0, seastar::rpc::snd_buf(std::move(bufs), large_buffer_size));\n        if (auto buffer = std::get_if<seastar::temporary_buffer<char>>(&rcv.bufs)) {\n            _large_compressed_buffer_random.emplace_back(std::move(*buffer));\n        } else {\n            _large_compressed_buffer_random\n                = std::move(std::get<std::vector<seastar::temporary_buffer<char>>>(rcv.bufs));\n        }\n\n        bufs = std::vector<temporary_buffer<char>>{};\n        for (auto&& b : _large_buffer_zeroes) {\n            bufs.emplace_back(b.clone());\n        }\n        rcv = _compressor.compress(0, seastar::rpc::snd_buf(std::move(bufs), large_buffer_size));\n        if (auto buffer = std::get_if<seastar::temporary_buffer<char>>(&rcv.bufs)) {\n            _large_compressed_buffer_zeroes.emplace_back(std::move(*buffer));\n        } else {\n            _large_compressed_buffer_zeroes\n                = std::move(std::get<std::vector<seastar::temporary_buffer<char>>>(rcv.bufs));\n        }\n    }\n\n    Compressor& compressor() { return _compressor; }\n\n    seastar::rpc::snd_buf small_buffer_random() {\n        return get_snd_buf(_small_buffer_random);\n    }\n    seastar::rpc::snd_buf small_buffer_zeroes() {\n        return get_snd_buf(_small_buffer_zeroes);\n    }\n\n    seastar::rpc::snd_buf large_buffer_random() {\n        return get_snd_buf(_large_buffer_random);\n    }\n    seastar::rpc::snd_buf large_buffer_zeroes() {\n        return get_snd_buf(_large_buffer_zeroes);\n    }\n\n    seastar::rpc::rcv_buf small_compressed_buffer_random() {\n        return get_rcv_buf(_small_compressed_buffer_random);\n    }\n    seastar::rpc::rcv_buf small_compressed_buffer_zeroes() {\n        return get_rcv_buf(_small_compressed_buffer_zeroes);\n    }\n\n    seastar::rpc::rcv_buf large_compressed_buffer_random() {\n        return get_rcv_buf(_large_compressed_buffer_random);\n    }\n    seastar::rpc::rcv_buf large_compressed_buffer_zeroes() {\n        return get_rcv_buf(_large_compressed_buffer_zeroes);\n    }\n};\n\nusing lz4 = compression<seastar::rpc::lz4_compressor>;\n\nPERF_TEST_F(lz4, small_random_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, small_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4, small_zeroed_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, small_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4, large_random_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, large_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4, large_zeroed_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, large_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4, small_random_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(small_compressed_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4, small_zeroed_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(small_compressed_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4, large_random_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(large_compressed_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4, large_zeroed_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(large_compressed_buffer_zeroes())\n    );\n}\n\nusing lz4_fragmented = compression<seastar::rpc::lz4_fragmented_compressor>;\n\nPERF_TEST_F(lz4_fragmented, small_random_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, small_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, small_zeroed_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, small_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, large_random_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, large_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, large_zeroed_buffer_compress) {\n    perf_tests::do_not_optimize(\n        compressor().compress(0, large_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, small_random_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(small_compressed_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, small_zeroed_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(small_compressed_buffer_zeroes())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, large_random_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(large_compressed_buffer_random())\n    );\n}\n\nPERF_TEST_F(lz4_fragmented, large_zeroed_buffer_decompress) {\n    perf_tests::do_not_optimize(\n        compressor().decompress(large_compressed_buffer_zeroes())\n    );\n}\n"
  },
  {
    "path": "tests/perf/shared_token_bucket.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB Ltd.\n */\n\n\n#include <random>\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/shared_token_bucket.hh>\n\n// The test allows measuring if the shared_token_bucket<> allows the tokens\n// consumers to get tokens at the rate the bucket is configured with.\n//\n// Report example:\n//\n//     effective rate is 1016575.0t/s, [239668 ... 263522]\n//\n// The last line \"effective rate\" is the tokens-per-second rate all workers were\n// able to get. It should be equal to the configured rate (context::rate below)\n//\n// In braces there are minimal and maximum rate of individual shards. These numbers\n// should not differ to much from each other, if they do it means that the t.b.\n// is not fair\n\nusing clock_type = std::chrono::steady_clock;\n\n// The test uses uint64_t tokens with per-second time-measurement and can use\n// capped and non-capped token buckets.\nusing capped_token_bucket_t = internal::shared_token_bucket<uint64_t, std::ratio<1>, internal::capped_release::yes>;\nusing pure_token_bucket_t = internal::shared_token_bucket<uint64_t, std::ratio<1>, internal::capped_release::no>;\n\n// The test entry point calls map_reduce on the sharded<> workers set and this\n// is what each worker produces, so that the entry point could accumulate the\n// final result from\nstruct work_result {\n    uint64_t tokens;\n    uint64_t released;\n};\n\nstruct statistics {\n    uint64_t total = 0;\n    uint64_t min = std::numeric_limits<uint64_t>::max();\n    uint64_t max = std::numeric_limits<uint64_t>::min();\n};\n\nstatistics accumulate(statistics acc, const work_result& val) {\n    return statistics {\n        .total = acc.total + val.tokens,\n        .min = std::min(acc.min, val.tokens),\n        .max = std::max(acc.max, val.tokens),\n    };\n}\n\n// The worker itself. Has a reference on the shared token bucket and tries to\n// consume as many tokens as it can without artificial delays. Reports the number\n// of tokens grabbed from bucket while running\ntemplate <typename TokenBucket>\nstruct worker : public seastar::peering_sharded_service<worker<TokenBucket>> {\n    TokenBucket& tb;\n\n    // Capped bucket requires that t.b. user releases the tokens so that\n    // they could be replenished. Respectively, the worker keeps track of\n    // total number of grabbed tokens as well as the number of not-yet-released\n    // tokens and the total number of released tokens. The \"available\" number\n    // of tokens is used to decide if the worker can release them or not\n    uint64_t tokens = 0;\n    uint64_t available = 0;\n    uint64_t released = 0;\n\n    // Tokens are released in a timer. This mimics default reactor workflow\n    // when requests are reaped from the kernel each task-quota -- 0.5ms\n    static constexpr auto release_period = std::chrono::microseconds(500);\n    const uint64_t release_per_tick = 0;\n    clock_type::time_point last_release;\n    timer<> release_tokens;\n\n    std::optional<std::pair<int, uint64_t>> head;\n    // The IO-scheduler doesn't get more than this number of tokens per tick.\n    // The test tries to mimic this behavior\n    const uint64_t threshold;\n\n    // The number of tokens to grab at a time. It's a distribution to resemble\n    // IO queue that tries to grab different amount of tokens for requests of\n    // different sizes and direction\n    std::uniform_int_distribution<int> size;\n\n    // Per-tick statistics. Collected, but not reported by default\n    struct tick_data {\n        std::chrono::microseconds delay;\n        std::chrono::microseconds sleep;\n        uint64_t defic;\n        uint64_t tokens;\n        uint64_t total;\n\n        template <typename D1, typename D2>\n        tick_data(D1 dur, D2 slp, uint64_t def, uint64_t lt, uint64_t tt) noexcept\n                : delay(std::chrono::duration_cast<std::chrono::microseconds>(dur))\n                , sleep(std::chrono::duration_cast<std::chrono::microseconds>(slp))\n                , defic(def)\n                , tokens(lt)\n                , total(tt)\n        {}\n    };\n    std::deque<tick_data> ticks;\n\n    worker(TokenBucket& tb_) noexcept\n        : tb(tb_)\n        , release_per_tick(double(tb.rate()) / smp::count * std::chrono::duration_cast<std::chrono::duration<double>>(release_period).count())\n        , last_release(clock_type::now())\n        , release_tokens([this] { do_release(); })\n        , threshold(tb.limit() / smp::count)\n        , size(1, std::min<int>(threshold, 128))\n    {\n        release_tokens.arm_periodic(std::chrono::duration_cast<std::chrono::microseconds>(release_period));\n        fmt::print(\"{} worker, threshold {}, release-per-tick {}\\n\", this_shard_id(), threshold, release_per_tick);\n    }\n\n    void do_release(uint64_t tokens) {\n        available -= tokens;\n        released += tokens;\n        if constexpr (TokenBucket::is_capped == internal::capped_release::yes) {\n            tb.release(tokens);\n        }\n    }\n\n    void do_release() {\n        // Timer can fire later than programmed because of hogs not yielding in a timely\n        // manner. If that was an IO queue more requests would have been reaped from the\n        // kernel, so do the same here -- scale the number of releasable tokens proportionally\n        auto now = clock_type::now();\n        auto real_delay = std::chrono::duration_cast<std::chrono::duration<double>>(now - last_release);\n        last_release = now;\n        uint64_t to_release = real_delay.count() * release_per_tick / std::chrono::duration_cast<std::chrono::duration<double>>(release_period).count();\n        do_release(std::min(to_release, available));\n    }\n\n    future<work_result> work(std::function<future<>(std::chrono::duration<double> d)> do_sleep) {\n        SEASTAR_ASSERT(tokens == 0);\n        auto start = clock_type::now();\n        // Run for 1 second. The perf suite would restart this method several times\n        return do_until([end = start + std::chrono::seconds(1)] { return clock_type::now() >= end; },\n            [this, start, do_sleep = std::move(do_sleep)] {\n                uint64_t d = 0;\n                uint64_t l_tokens = 0;\n                int sz;\n\n                while (l_tokens < threshold) {\n                    if (head) {\n                        tb.replenish(clock_type::now());\n                        d = tb.deficiency(head->second);\n                        if (d > 0) {\n                            break;\n                        }\n                        sz = head->first;\n                        head.reset();\n                    } else {\n                        sz = size(testing::local_random_engine);\n                        auto h = tb.grab(sz);\n                        d = tb.deficiency(h);\n                        if (d > 0) {\n                            head = std::make_pair(sz, h);\n                            break;\n                        }\n                    }\n                    tokens += sz;\n                    l_tokens += sz;\n                    available += sz;\n                }\n\n                auto p = tb.duration_for(d);\n\n                ticks.emplace_back(clock_type::now() - start, p, d, l_tokens, tokens);\n                if (ticks.size() > 2048) {\n                    ticks.pop_front();\n                }\n\n                return do_sleep(p);\n            }\n        ).then([this, start] {\n            // Reports:\n            //  - shard-id\n            //  - total number of tokens and total time taken\n            //  - effective speed\n            //  - expected speed (token-bucket.rate() / smp::count)\n            //  - ticks -- the number of times the worker had change to grab tokens\n            //  - the info about tokens releasing\n            auto delay = std::chrono::duration_cast<std::chrono::duration<double>>(clock_type::now() - start).count();\n            fmt::print(\"{} {}t/{:.3f}s, speed is {:.1f}t/s goal {:.1f}t/s, {} ticks, released {} (accumulated {})\\n\", this_shard_id(), tokens, delay,\n                    double(tokens) / delay, double(tb.rate()) / smp::count, ticks.size(), released, available);\n            do_release(available);\n            work_result r {\n                .tokens = std::exchange(this->tokens, 0),\n                .released = std::exchange(this->released, 0),\n            };\n            return make_ready_future<work_result>(std::move(r));\n        });\n    }\n\n\n    // The below two are how worker waits for the token-bucket deficiency\n    // to disappear (i.e. -- when the requested number of tokens are replenished\n    //\n    // Two options -- poll infinitely or sleep for the estimated (by the\n    // bucket method) duration\n\n    future<work_result> work_sleeping() {\n        return work([] (std::chrono::duration<double> d) {\n            return seastar::sleep(std::chrono::duration_cast<std::chrono::microseconds>(d));\n        });\n    }\n\n    future<work_result> work_yielding() {\n        return work([] (std::chrono::duration<double>) {\n            return seastar::yield();\n        });\n    }\n\n    future<> print_and_clear_ticks() {\n        fmt::print(\"{} {} ticks\\n\", this_shard_id(), ticks.size());\n        std::chrono::microseconds p(0);\n        for (auto& td : ticks) {\n            fmt::print(\"  {:8} +{:5} us {:5}/{:5} def {:3} sleep {:5} us\\n\", td.delay.count(), (td.delay - p).count(), td.tokens, td.total, td.defic, td.sleep.count());\n            p = td.delay;\n        }\n        ticks.clear();\n        if (this_shard_id() == smp::count - 1) {\n            return make_ready_future<>();\n        }\n\n        return this->container().invoke_on(this_shard_id() + 1, &worker::print_and_clear_ticks);\n    }\n};\n\n// CPU hog that occupies CPU for \"busy\" duration, then sleeps for \"rest\" duration\n// The actual periods are randomized to be thus \"on average\"\nstruct hog {\n    std::exponential_distribution<double> busy;\n    std::exponential_distribution<double> rest;\n    std::optional<future<>> stopped;\n    bool keep_going = false;\n    uint64_t _iterations = 0;\n\n    template <typename T1, typename T2>\n    hog(T1 b, T2 r) noexcept\n        : busy(1.0 / std::chrono::duration_cast<std::chrono::duration<double>>(b).count())\n        , rest(1.0 / std::chrono::duration_cast<std::chrono::duration<double>>(r).count())\n    {}\n\n    void work() {\n        SEASTAR_ASSERT(!stopped.has_value());\n        keep_going = true;\n        stopped = do_until([this] { return !keep_going; },\n            [this] {\n                auto p = std::chrono::duration<double>(rest(testing::local_random_engine));\n                return seastar::sleep(std::chrono::duration_cast<std::chrono::microseconds>(p)).then([this] {\n                    _iterations++;\n                    auto until = clock_type::now() + std::chrono::duration<double>(busy(testing::local_random_engine));\n                    do {\n                    } while (clock_type::now() < until && keep_going);\n                });\n            }\n        );\n    }\n\n    future<> terminate() {\n        SEASTAR_ASSERT(stopped.has_value());\n        keep_going = false;\n        auto f = std::move(*stopped);\n        stopped.reset();\n        return f;\n    }\n};\n\ntemplate <typename TokenBucket>\nstruct context {\n    using worker_t = worker<TokenBucket>;\n    TokenBucket tb;\n    seastar::sharded<worker_t> w;\n    seastar::sharded<hog> h;\n\n    static constexpr uint64_t rate = 1000000;\n    static constexpr uint64_t limit = rate / 2000;\n    static constexpr uint64_t threshold = 1;\n\n    context() : tb(rate, limit, threshold)\n    {\n        w.start(std::ref(tb)).get();\n        h.start(std::chrono::microseconds(300), std::chrono::microseconds(100)).get();\n        fmt::print(\"Created tb {}t/s (limit {} threshold {})\\n\", tb.rate(), tb.limit(), tb.threshold());\n    }\n\n    ~context() {\n        h.stop().get();\n        w.stop().get();\n    }\n\n    template <typename Fn>\n    future<> run_workers(Fn&& fn) {\n        auto start = clock_type::now();\n        return w.map_reduce0(std::forward<Fn>(fn), statistics{}, accumulate).then([start] (statistics st) {\n            auto delay = std::chrono::duration_cast<std::chrono::duration<double>>(clock_type::now() - start).count();\n            fmt::print(\"effective rate is {:.1f}t/s, [{} ... {}]\\n\", st.total / delay, st.min, st.max);\n        });\n    }\n\n    future<> test_sleeping() {\n        fmt::print(\"---8<---\\n\");\n        return run_workers(&worker_t::work_sleeping);\n    }\n\n    future<> test_yielding() {\n        fmt::print(\"---8<---\\n\");\n        return run_workers(&worker_t::work_yielding);\n    }\n\n    future<> test_sleeping_with_hog() {\n        fmt::print(\"---8<---\\n\");\n        return h.invoke_on_all(&hog::work).then([this] {\n            return run_workers(&worker_t::work_sleeping).then([this] {\n                return h.invoke_on_all(&hog::terminate);\n            });\n        });\n    }\n};\n\nstruct perf_capped_context : public context<capped_token_bucket_t> {};\nstruct perf_pure_context : public context<pure_token_bucket_t> {};\n\n// There are 3 tests run over 2 types of buckets:\n//\n// - poll token bucket for tokens in case of deficiency\n// - sleep in case token bucket reports deficiency\n// - sleep on deficiency, but run CPU hogs in the background\n//\n// All tests are run with capped and non-capped (called pure) token buckets\n\nPERF_TEST_F(perf_capped_context, yielding_throughput) { return test_yielding(); }\nPERF_TEST_F(perf_capped_context, sleeping_throughput) { return test_sleeping(); }\nPERF_TEST_F(perf_capped_context, sleeping_throughput_with_hog) { return test_sleeping_with_hog(); }\n\nPERF_TEST_F(perf_pure_context, yielding_throughput) { return test_yielding(); }\nPERF_TEST_F(perf_pure_context, sleeping_throughput) { return test_sleeping(); }\nPERF_TEST_F(perf_pure_context, sleeping_throughput_with_hog) { return test_sleeping_with_hog(); }\n"
  },
  {
    "path": "tests/perf/smp_submit_to_perf.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#include <random>\n#include <ranges>\n#include <fmt/core.h>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/later.hh>\n\nusing namespace seastar;\nusing namespace std::chrono;\n\nclass thinker {\n    class poisson_process {\n        std::random_device _rd;\n        std::mt19937 _rng;\n        std::exponential_distribution<double> _exp;\n\n    public:\n        poisson_process(duration<double> period)\n                : _rng(_rd())\n                , _exp(1.0 / period.count())\n        {\n        }\n\n        duration<double> get() {\n            return duration<double>(_exp(_rng));\n        }\n    };\n\n    poisson_process _pause;\n    bool _stop;\n    future<> _done;\n\n    future<> start_thinking(unsigned concurrency) {\n        return parallel_for_each(std::views::iota(0u, concurrency), [this] (unsigned f) {\n            return do_until([this] { return _stop; }, [this] {\n                auto until = steady_clock::now() + _pause.get();\n                while (steady_clock::now() < until) {\n                    ; // do nothing\n                }\n                return make_ready_future<>();\n            });\n        });\n    }\n\npublic:\n    thinker(unsigned concurrency, microseconds think) noexcept\n        : _pause(think)\n        , _stop(false)\n        , _done(start_thinking(concurrency))\n    {\n        fmt::print(\"shard {} starts {}x{}us thinkers\\n\", this_shard_id(), concurrency, think.count());\n    }\n\n    future<> stop() {\n        _stop = true;\n        return std::move(_done);\n    }\n};\n\nenum class respond_type { ready, yield, io, timer };\n\nstatic respond_type parse_respond_type(std::string s) {\n    if (s == \"ready\") {\n        return respond_type::ready;\n    }\n    if (s == \"yield\") {\n        return respond_type::yield;\n    }\n    if (s == \"io\") {\n        return respond_type::io;\n    }\n    if (s == \"timer\") {\n        return respond_type::timer;\n    }\n\n    throw std::runtime_error(\"unknown respond type\");\n}\n\nclass worker {\n    const unsigned _to;\n\n    std::unique_ptr<thinker> _think;\n\n    uint64_t _total;\n    bool _stop;\n    future<> _done;\n\n    static unsigned my_target(unsigned targets) noexcept {\n        unsigned group_size = (smp::count + (targets - 1)) / targets;\n        unsigned group_no = this_shard_id() / group_size;\n        return group_size * group_no;\n    }\n\n    future<> start_working(unsigned concurrency, respond_type resp, microseconds tmo) {\n        return parallel_for_each(std::views::iota(0u, concurrency), [this, resp, tmo] (unsigned f) {\n            return do_until([this] { return _stop; }, [this, resp, tmo] {\n                return smp::submit_to(_to, [resp, tmo] {\n                    switch (resp) {\n                    case respond_type::ready:\n                        return make_ready_future<>();\n                    case respond_type::yield:\n                        return yield();\n                    case respond_type::io:\n                        return check_for_io_immediately();\n                    case respond_type::timer:\n                        return seastar::sleep<lowres_clock>(tmo);\n                    }\n\n                    __builtin_unreachable();\n                }).then([this] {\n                    _total++;\n                    return make_ready_future<>();\n                });\n            });\n        });\n    }\n\npublic:\n    struct config {\n        unsigned targets;\n        unsigned thinkers;\n        microseconds think;\n        respond_type respond;\n        microseconds respond_tmo;\n        unsigned concurrency;\n    };\n\n    worker(config cfg) noexcept\n        : _to(my_target(cfg.targets))\n        , _think(is_target() && (cfg.thinkers > 0) ? std::make_unique<thinker>(cfg.thinkers, cfg.think) : nullptr)\n        , _total(0)\n        , _stop(false)\n        , _done(start_working(cfg.concurrency, cfg.respond, cfg.respond_tmo))\n    {\n    }\n\n    future<> stop() {\n        if (_stop) {\n            return make_ready_future<>();\n        }\n\n        _stop = true;\n        return std::move(_done).then([this] {\n            return _think ? _think->stop() : make_ready_future<>();\n        });\n    }\n\n    bool is_target() const noexcept { return _to == this_shard_id(); }\n    uint64_t total() const noexcept { return _total; }\n};\n\nclass stats {\n    uint64_t _min = std::numeric_limits<uint64_t>::max();\n    uint64_t _max = std::numeric_limits<uint64_t>::min();\n    uint64_t _sum = 0;\n    unsigned _nr = 0;\n    const unsigned _norm;\n\npublic:\n\n    stats(unsigned norm) noexcept : _norm(norm) {}\n\n    void append(uint64_t val) noexcept {\n        _min = std::min(_min, val);\n        _max = std::max(_max, val);\n        _sum += val;\n        _nr++;\n    }\n\n    double min() const noexcept { return (double)_min / _norm; }\n    double max() const noexcept { return (double)_max / _norm; }\n    double avg() const noexcept { return _nr > 0 ? (double)_sum / _nr / _norm : 0.0; }\n    unsigned nr() const noexcept { return _nr; }\n};\n\nint main(int ac, char** av) {\n    app_template at;\n    namespace bpo = boost::program_options;\n    at.add_options()\n            (\"duration\", bpo::value<unsigned>()->default_value(32), \"time to run the test (seconds)\")\n            (\"targets\", bpo::value<unsigned>()->default_value(1), \"number of responder shards\")\n            (\"thinkers\", bpo::value<unsigned>()->default_value(0), \"thinker fibers to run in parallel on targets\")\n            (\"think\", bpo::value<unsigned>()->default_value(100), \"time (us) thinkers busyloop for\")\n            (\"respond\", bpo::value<std::string>()->default_value(\"ready\"), \"how to respond on target (ready, yield, io, timer)\")\n            (\"respond-timeout\", bpo::value<unsigned>()->default_value(1), \"the 'timer' respond timeout (us)\")\n            (\"concurrency\", bpo::value<unsigned>()->default_value(1), \"smp::submit_to operations to issue in parallel\")\n        ;\n\n    return at.run(ac, av, [&at] {\n        auto duration = seconds(at.configuration()[\"duration\"].as<unsigned>());\n        worker::config cfg;\n        cfg.targets = at.configuration()[\"targets\"].as<unsigned>();\n        cfg.thinkers = at.configuration()[\"thinkers\"].as<unsigned>();\n        cfg.think = microseconds(at.configuration()[\"think\"].as<unsigned>());\n        cfg.respond = parse_respond_type(at.configuration()[\"respond\"].as<std::string>());\n        cfg.respond_tmo = microseconds(at.configuration()[\"respond-timeout\"].as<unsigned>());\n        cfg.concurrency = at.configuration()[\"concurrency\"].as<unsigned>();\n\n        return async([cfg, duration] {\n            sharded<worker> workers;\n            auto start = steady_clock::now();\n\n            workers.start(cfg).get();\n            seastar::sleep(duration).get();\n            workers.invoke_on_all(&worker::stop).get();\n\n            auto real_duration = duration_cast<seconds>(steady_clock::now() - start);\n            fmt::print(\"took {}s (expected {}s)\\n\", real_duration.count(), duration.count());\n            stats st(real_duration.count()), st_targets(real_duration.count());\n            for (unsigned i = 0; i < smp::count; i++) {\n                workers.invoke_on(i, [&st, &st_targets] (worker& w) {\n                    if (w.is_target()) {\n                        st_targets.append(w.total());\n                    } else {\n                        st.append(w.total());\n                    }\n                }).get();\n            }\n            fmt::print(\"workers({:2}): min {:.1f} avg {:.1f} max {:.1f} op/s\\n\", st.nr(), st.min(), st.avg(), st.max());\n            fmt::print(\"targets({:2}): min {:.1f} avg {:.1f} max {:.1f} op/s\\n\", st_targets.nr(), st_targets.min(), st_targets.avg(), st_targets.max());\n\n            workers.stop().get();\n        });\n    });\n}\n"
  },
  {
    "path": "tests/perf/thread_context_switch_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/sleep.hh>\n#include <fmt/printf.h>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nclass context_switch_tester {\n    uint64_t _switches{0};\n    semaphore _s1{0};\n    semaphore _s2{0};\n    bool _done1{false};\n    bool _done2{false};\n    thread _t1{[this] { main1(); }};\n    thread _t2{[this] { main2(); }};\nprivate:\n    void main1() {\n        while (!_done1) {\n            _s1.wait().get();\n            ++_switches;\n            _s2.signal();\n        }\n        _done2 = true;\n    }\n    void main2() {\n        while (!_done2) {\n            _s2.wait().get();\n            ++_switches;\n            _s1.signal();\n        }\n    }\npublic:\n    void begin_measurement() {\n        _s1.signal();\n    }\n    future<uint64_t> measure() {\n        _done1 = true;\n        return _t1.join().then([this] {\n            return _t2.join();\n        }).then([this] {\n            return _switches;\n        });\n    }\n    future<> stop() {\n        return make_ready_future<>();\n    }\n};\n\nint main(int ac, char** av) {\n    static const auto test_time = 5s;\n    return app_template().run_deprecated(ac, av, [] {\n        auto dcstp = std::make_unique<sharded<context_switch_tester>>();\n        auto& dcst = *dcstp;\n        return dcst.start().then([&dcst] {\n            return dcst.invoke_on_all(&context_switch_tester::begin_measurement);\n        }).then([] {\n            return sleep(test_time);\n        }).then([&dcst] {\n            return dcst.map_reduce0(std::mem_fn(&context_switch_tester::measure), uint64_t(), std::plus<uint64_t>());\n        }).then([] (uint64_t switches) {\n            switches /= smp::count;\n            fmt::print(\"context switch time: {:5.1f} ns\\n\",\n                  double(std::chrono::duration_cast<std::chrono::nanoseconds>(test_time).count()) / switches);\n        }).then([&dcst] {\n            return dcst.stop();\n        }).then([dcstp = std::move(dcstp)] {\n            engine_exit(0);\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/CMakeLists.txt",
    "content": "#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2018 Scylladb, Ltd.\n#\n\n# Logical target for all unit tests.\nadd_custom_target (unit_tests)\n\nset (Seastar_UNIT_TEST_SMP\n  2\n  CACHE\n  STRING\n  \"Run unit tests with this many cores.\")\n\n# Check if the compiler supports -fsanitize=fuzzer (compile and link)\ninclude (CheckCXXSourceCompiles)\ninclude (CMakePushCheckState)\ncmake_push_check_state ()\nset (CMAKE_REQUIRED_FLAGS \"-fsanitize=fuzzer\")\nset (CMAKE_REQUIRED_LINK_OPTIONS \"-fsanitize=fuzzer\")\ncheck_cxx_source_compiles (\"\nextern \\\"C\\\" int LLVMFuzzerTestOneInput(const char*, long) { return 0; }\n\" Seastar_FUZZER_SUPPORTED)\ncmake_pop_check_state ()\n\nif (Seastar_FUZZER_SUPPORTED)\n  # Logical target for all fuzz tests.\n  add_custom_target (fuzz_tests)\n\n  # Maximum time in seconds for each fuzz test when run via ctest.\n  # Default to 1 minute in Fuzz mode, 1 second otherwise.\n  if (CMAKE_BUILD_TYPE STREQUAL \"Fuzz\")\n    set (default_fuzz_time 60)\n  else ()\n    set (default_fuzz_time 1)\n  endif ()\n  set (Seastar_FUZZ_MAX_TIME ${default_fuzz_time} CACHE STRING \"Maximum time in seconds for fuzz tests\")\nendif ()\n\n#\n# Define a new unit test with the given name.\n#\n# seastar_add_test (name\n#   [KIND {SEASTAR,BOOST,CUSTOM,FUZZ}]\n#   [SOURCES source1 source2 ... sourcen]\n#   [WORKING_DIRECTORY dir]\n#   [LIBRARIES library1 library2 ... libraryn]\n#   [RUN_ARGS arg1 arg2 ... argn])\n#\n# There are four kinds of test we support (the KIND parameter):\n#\n# - SEASTAR: Unit tests which use macros like `SEASTAR_TEST_CASE`\n# - BOOST: Unit tests which use macros like `BOOST_AUTO_TEST_CASE`\n# - CUSTOM: Custom tests which need to be specified\n# - FUZZ: Fuzz tests using libFuzzer (requires compiler support)\n#\n# SEASTAR and BOOST tests will have their output saved for interpretation by the Jenkins continuous integration service\n# if this is configured for the build.\n#\n# KIND can be omitted, in which case it is assumed to be SEASTAR.\n#\n# If SOURCES is provided, then the test files are first compiled into an executable which has the same name as the test\n# but with a suffix (\"_test\", or \"_fuzz\" for FUZZ tests).\n#\n# WORKING_DIRECTORY can be optionally provided to choose where the test is executed.\n#\n# If LIBRARIES is provided along with SOURCES, then the executable is additionally linked with these libraries.\n#\n# RUN_ARGS are optional additional arguments to pass to the executable. For SEASTAR tests, these come after `--`. For\n# CUSTOM tests with no SOURCES, this parameter can be used to specify the executable name as well as its arguments since\n# no executable is compiled. For FUZZ tests, these are passed directly to libFuzzer (e.g., -max_len=100).\n#\nfunction (seastar_add_test name)\n  set (test_kinds\n    SEASTAR\n    BOOST\n    CUSTOM\n    FUZZ)\n\n  cmake_parse_arguments (parsed_args\n    \"\"\n    \"WORKING_DIRECTORY;KIND\"\n    \"RUN_ARGS;SOURCES;LIBRARIES;DEPENDS\"\n    ${ARGN})\n\n  if (NOT parsed_args_KIND)\n    set (parsed_args_KIND SEASTAR)\n  elseif (NOT (parsed_args_KIND IN_LIST test_kinds))\n    message (FATAL_ERROR \"Invalid test kind. KIND must be one of ${test_kinds}\")\n  endif ()\n\n  # FUZZ tests require compiler support\n  if (parsed_args_KIND STREQUAL \"FUZZ\" AND NOT Seastar_FUZZER_SUPPORTED)\n    message (STATUS \"Skipping fuzz test ${name}: compiler does not support -fsanitize=fuzzer\")\n    return ()\n  endif ()\n\n  if (parsed_args_SOURCES)\n    # These may be unused.\n    seastar_jenkins_arguments (${name} jenkins_args)\n\n    #\n    # Each kind of test must populate the `args` and `libraries` lists.\n    #\n\n    set (libraries \"${parsed_args_LIBRARIES}\")\n\n    set (args \"\")\n    if (parsed_args_KIND STREQUAL \"SEASTAR\")\n      list (APPEND libraries\n        seastar_testing\n        seastar_private)\n\n      if (NOT (Seastar_JENKINS STREQUAL \"\"))\n        list (APPEND args ${jenkins_args})\n      endif ()\n\n      list (APPEND args -- -c ${Seastar_UNIT_TEST_SMP})\n    elseif (parsed_args_KIND STREQUAL \"BOOST\")\n      list (APPEND libraries\n        Boost::unit_test_framework\n        Boost::dynamic_linking\n        seastar_private)\n\n      if (NOT (Seastar_JENKINS STREQUAL \"\"))\n        list (APPEND args ${jenkins_args})\n      endif ()\n    elseif (parsed_args_KIND STREQUAL \"FUZZ\")\n      list (APPEND libraries seastar_private)\n      # Fuzz tests get -max_total_time automatically\n      list (APPEND args -max_total_time=${Seastar_FUZZ_MAX_TIME})\n    endif ()\n\n    list (APPEND args ${parsed_args_RUN_ARGS})\n\n    # Set up target name and output name based on test kind\n    if (parsed_args_KIND STREQUAL \"FUZZ\")\n      set (executable_target fuzz_${name})\n      set (output_name ${name}_fuzz)\n      set (test_group fuzz_tests)\n      set (test_name_prefix Seastar.fuzz)\n    else ()\n      set (executable_target test_unit_${name})\n      set (output_name ${name}_test)\n      set (test_group unit_tests)\n      set (test_name_prefix Seastar.unit)\n    endif ()\n\n    add_executable (${executable_target} ${parsed_args_SOURCES})\n\n    target_link_libraries (${executable_target}\n      PRIVATE ${libraries})\n\n    # FUZZ tests use libFuzzer which provides its own main\n    if (parsed_args_KIND STREQUAL \"FUZZ\")\n      target_compile_options (${executable_target}\n        PRIVATE -fsanitize=fuzzer)\n      target_link_options (${executable_target}\n        PRIVATE -fsanitize=fuzzer)\n    else ()\n      target_compile_definitions (${executable_target}\n        PRIVATE\n          SEASTAR_TESTING_MAIN\n          SEASTAR_TESTING_WITH_NETWORKING=$<BOOL:${Seastar_ENABLE_TESTS_ACCESSING_INTERNET}>)\n\n      if ((Seastar_STACK_GUARDS STREQUAL \"ON\") OR\n          ((Seastar_STACK_GUARDS STREQUAL \"DEFAULT\") AND\n           (CMAKE_BUILD_TYPE IN_LIST Seastar_STACK_GUARD_MODES)))\n        target_compile_definitions (${executable_target}\n          PRIVATE\n            SEASTAR_THREAD_STACK_GUARDS)\n      endif ()\n\n      if (Seastar_HEAP_PROFILING)\n        target_compile_definitions (${executable_target}\n          PRIVATE\n            SEASTAR_HEAPPROF)\n      endif ()\n    endif ()\n\n    target_include_directories (${executable_target}\n      PRIVATE\n        ${CMAKE_CURRENT_SOURCE_DIR}\n        ${Seastar_SOURCE_DIR}/src)\n\n    set_target_properties (${executable_target}\n      PROPERTIES\n        OUTPUT_NAME ${output_name})\n\n    add_dependencies (${test_group} ${executable_target})\n    set (forwarded_args COMMAND ${CMAKE_COMMAND} -E env \"${Seastar_TEST_ENVIRONMENT}\" $<TARGET_FILE:${executable_target}> ${args})\n  else ()\n    if (NOT (parsed_args_KIND STREQUAL \"CUSTOM\"))\n      message (FATAL_ERROR \"SOURCES are required for ${parsed_args_KIND} tests\")\n    endif ()\n\n    set (executable_target test_unit_${name})\n    set (test_name_prefix Seastar.unit)\n    set (forwarded_args COMMAND ${CMAKE_COMMAND} -E env \"${Seastar_TEST_ENVIRONMENT}\" ${parsed_args_RUN_ARGS})\n  endif ()\n\n  #\n  # We expect `forwarded_args` to be populated correctly at this point.\n  #\n\n  set (target ${executable_target}_run)\n\n  if (parsed_args_WORKING_DIRECTORY)\n    list (APPEND forwarded_args WORKING_DIRECTORY ${parsed_args_WORKING_DIRECTORY})\n  endif ()\n\n  if (parsed_args_DEPENDS)\n    add_dependencies(${executable_target} ${parsed_args_DEPENDS})\n  endif()\n\n  add_custom_target (${target}\n    ${forwarded_args}\n    USES_TERMINAL)\n\n  add_test (\n    NAME ${test_name_prefix}.${name}\n    COMMAND ${CMAKE_COMMAND} --build ${Seastar_BINARY_DIR} --target ${target})\n\n  set_tests_properties (${test_name_prefix}.${name}\n    PROPERTIES\n      TIMEOUT ${Seastar_TEST_TIMEOUT})\nendfunction ()\n\n#\n# Define a new custom unit test whose entry point is a Seastar application.\n#\n# seastar_add_app_test (name\n#   [SOURCES source1 source2 ... sourcen]\n#   [LIBRARIES library1 library2 ... libraryn]\n#   [RUN_ARGS arg1 arg2 ... argn])\n#\n# These kinds of tests are structured like Seastar applications.\n#\n# These tests always link against `seastar_private` and are always invoked with\n# `-c ${Seastar_UNIT_TEST_SMP}`.\n#\nfunction (seastar_add_app_test name)\n  cmake_parse_arguments (parsed_args\n    \"\"\n    \"\"\n    \"RUN_ARGS;SOURCES;LIBRARIES\"\n    ${ARGN})\n\n  seastar_add_test (${name}\n    KIND CUSTOM\n    SOURCES ${parsed_args_SOURCES}\n    LIBRARIES\n      seastar_private\n      ${parsed_args_LIBRARIES}\n    RUN_ARGS\n      -c ${Seastar_UNIT_TEST_SMP}\n      ${parsed_args_RUN_ARGS})\nendfunction ()\n\nfunction (prepend_each var prefix)\n  set (result \"\")\n\n  foreach (x ${ARGN})\n    list (APPEND result ${prefix}/${x})\n  endforeach ()\n\n  set (${var} ${result} PARENT_SCOPE)\nendfunction ()\n\nadd_custom_target (test_unit\n  COMMAND ctest --verbose -R Seastar.unit\n  USES_TERMINAL)\n\nseastar_add_test (abort_source\n  SOURCES abort_source_test.cc)\n\nseastar_add_test (alloc\n  SOURCES alloc_test.cc)\n\nif (NOT Seastar_EXECUTE_ONLY_FAST_TESTS)\n  set (allocator_test_args \"\")\nelse ()\n  if (CMAKE_BUILD_TYPE STREQUAL \"Debug\")\n    set (allocator_test_args --iterations 5)\n  else ()\n    set (allocator_test_args --time 0.1)\n  endif ()\nendif ()\n\nseastar_add_test (allocator\n  SOURCES allocator_test.cc\n  RUN_ARGS ${allocator_test_args})\n\nseastar_add_app_test (alien\n  SOURCES alien_test.cc)\n\nseastar_add_test (checked_ptr\n  SOURCES checked_ptr_test.cc)\n\nseastar_add_test (temporary_buffer\n  SOURCES temporary_buffer_test.cc)\n\nseastar_add_test (chunked_fifo\n  KIND BOOST\n  SOURCES chunked_fifo_test.cc)\n\nseastar_add_test (linux_perf_event\n  KIND BOOST\n  SOURCES linux_perf_event_test.cc ../perf/linux_perf_event.cc)\n\nseastar_add_test (chunk_parsers\n  SOURCES chunk_parsers_test.cc)\n\nseastar_add_test (circular_buffer\n  KIND BOOST\n  SOURCES circular_buffer_test.cc)\n\nseastar_add_test (circular_buffer_fixed_capacity\n  KIND BOOST\n  SOURCES circular_buffer_fixed_capacity_test.cc)\n\nseastar_add_test (condition_variable\n  SOURCES condition_variable_test.cc)\n\nseastar_add_test (connect\n  SOURCES connect_test.cc)\n\nseastar_add_test (content_source\n  SOURCES content_source_test.cc)\n\nseastar_add_test (coroutines\n  SOURCES coroutines_test.cc)\n\nseastar_add_test (generator\n  SOURCES generator_test.cc)\n\nseastar_add_test (defer\n  KIND BOOST\n  SOURCES defer_test.cc)\n\nseastar_add_test (deleter\n  KIND BOOST\n  SOURCES deleter_test.cc)\n\nseastar_add_test (directory\n  SOURCES directory_test.cc)\n\nseastar_add_test (distributed\n  SOURCES distributed_test.cc)\n\nseastar_add_test (dns\n  SOURCES dns_test.cc)\n\nseastar_add_test (execution_stage\n  SOURCES execution_stage_test.cc)\n\nseastar_add_test (expiring_fifo\n  SOURCES expiring_fifo_test.cc)\n\nseastar_add_test (abortable_fifo\n  SOURCES abortable_fifo_test.cc)\n\nseastar_add_test (io_queue\n  SOURCES io_queue_test.cc)\n\nseastar_add_test (fair_queue\n  SOURCES fair_queue_test.cc)\n\nseastar_add_test (file_io\n  SOURCES file_io_test.cc)\n\nseastar_add_test (file_utils\n  SOURCES\n    file_utils_test.cc\n    expected_exception.hh)\n\nseastar_add_test (foreign_ptr\n  SOURCES foreign_ptr_test.cc)\n\nseastar_add_test (fsnotifier\n  SOURCES fsnotifier_test.cc)\n\nseastar_add_test (fstream\n  SOURCES\n    fstream_test.cc\n    mock_file.hh)\n\nseastar_add_test (futures\n  SOURCES\n    futures_test.cc\n    expected_exception.hh)\n\nseastar_add_test (sharded\n  SOURCES sharded_test.cc)\n\nseastar_add_test (httpd\n  SOURCES\n    httpd_test.cc\n    loopback_socket.hh\n    memory-data-sink.hh)\n\nseastar_add_test (websocket\n  SOURCES websocket_test.cc)\n\nseastar_add_test (ipv6\n  SOURCES ipv6_test.cc)\n\nseastar_add_test (network_interface\n  SOURCES network_interface_test.cc)\n\nseastar_add_test (json_formatter\n  SOURCES\n    json_formatter_test.cc\n    memory-data-sink.hh)\n\nseastar_add_test (libc_wrapper\n  SOURCES libc_wrapper_test.cc)\n\nseastar_add_test (locking\n  SOURCES locking_test.cc)\n\nseastar_add_test (lowres_clock\n  SOURCES lowres_clock_test.cc)\n\nseastar_add_test (metrics\n  SOURCES metrics_test.cc)\n\nseastar_add_test (net_config\n  KIND BOOST\n  SOURCES net_config_test.cc)\n\nseastar_add_test (noncopyable_function\n  KIND BOOST\n  SOURCES noncopyable_function_test.cc)\n\nseastar_add_test (output_stream\n  SOURCES\n    output_stream_test.cc\n    memory-data-sink.hh)\n\nseastar_add_test (packet\n  KIND BOOST\n  SOURCES packet_test.cc)\n\nseastar_add_test (program_options\n  KIND BOOST\n  SOURCES program_options_test.cc)\n\nseastar_add_test (proxy_protocol\n  SOURCES proxy_protocol_test.cc)\n\nseastar_add_test (queue\n  SOURCES queue_test.cc)\n\nseastar_add_test (request_parser\n  SOURCES request_parser_test.cc)\n\nseastar_add_test (rpc\n  SOURCES\n    loopback_socket.hh\n    rpc_test.cc)\n\nseastar_add_test (semaphore\n  SOURCES\n    semaphore_test.cc\n    expected_exception.hh)\n\nseastar_add_test (shared_ptr\n  KIND BOOST\n  SOURCES shared_ptr_test.cc)\n\nseastar_add_test (signal\n  SOURCES signal_test.cc)\n\nseastar_add_test (simple_stream\n  KIND BOOST\n  SOURCES simple_stream_test.cc)\n\nseastar_add_test (slab\n  KIND BOOST\n  SOURCES slab_test.cc)\n\nseastar_add_app_test (smp\n  SOURCES smp_test.cc)\n\nseastar_add_test (app-template\n  KIND BOOST\n  SOURCES app-template_test.cc)\n\nseastar_add_test (socket\n  SOURCES socket_test.cc\n  # https://github.com/scylladb/seastar/issues/2302\n  RUN_ARGS --reactor-backend linux-aio)\n\nseastar_add_test (epoll\n  SOURCES epoll_test.cc\n  RUN_ARGS --reactor-backend epoll)\n\nseastar_add_test (sstring\n  KIND BOOST\n  SOURCES sstring_test.cc)\n\nseastar_add_test (stall_detector\n  SOURCES stall_detector_test.cc)\n\nseastar_add_test (stream_reader\n  SOURCES stream_reader_test.cc)\n\nseastar_add_test (thread\n  SOURCES thread_test.cc\n  LIBRARIES Valgrind::valgrind)\n\nseastar_add_test (scheduling_group\n  SOURCES scheduling_group_test.cc)\n\nseastar_add_test (scheduling_group_nesting\n  SOURCES scheduling_group_nesting_test.cc)\n\nseastar_add_test (timer\n  SOURCES timer_test.cc)\n\nseastar_add_test (uname\n  KIND BOOST\n  SOURCES uname_test.cc)\n\nseastar_add_test (source_location\n  KIND BOOST\n  SOURCES source_location_test.cc)\n\nseastar_add_test (shared_token_bucket\n  SOURCES shared_token_bucket_test.cc)\n\nseastar_add_test (prometheus_text\n  SOURCES prometheus_text_test.cc)\n\nseastar_add_test (prometheus_http\n  SOURCES\n    prometheus_http_test.cc)\n\nfunction(seastar_add_certgen name)\n  cmake_parse_arguments(CERT\n    \"\"\n    \"SUBJECT;SERVER;NAME;DOMAIN;COMMON;LOCALITY;ORG;WIDTH;STATE;COUNTRY;UNIT;EMAIL;DAYS;ALG\"\n    \"ALG_OPTS\"\n    ${ARGN}\n  )\n\n  if (NOT CERT_SERVER)\n    execute_process(COMMAND hostname\n      RESULT_VARIABLE CERT_SERVER\n    )\n  endif()\n  if (NOT CERT_DOMAIN)\n    execute_process(COMMAND dnsdomainname\n      RESULT_VARIABLE CERT_DOMAIN\n    )\n  endif()\n  if (NOT CERT_NAME)\n    set(CERT_NAME ${CERT_SERVER})\n  endif()\n  if (NOT CERT_COUNTRY)\n    set(CERT_COUNTRY SE)\n  endif()\n  if (NOT CERT_STATE)\n    set(CERT_STATE Stockholm)\n  endif()\n  if (NOT CERT_LOCALITY)\n    set(CERT_LOCALITY ${CERT_STATE})\n  endif()\n  if (NOT CERT_ORG)\n    set(CERT_ORG ${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_UNIT)\n    set(CERT_UNIT ${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_COMMON)\n    set(CERT_COMMON ${CERT_SERVER}.${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_EMAIL)\n    set(CERT_EMAIL postmaster@${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_ALT_EMAIL_1)\n    set(CERT_ALT_EMAIL_1 alt1@${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_ALT_EMAIL_2)\n    set(CERT_ALT_EMAIL_2 alt2@${CERT_DOMAIN})\n  endif()\n  if (NOT CERT_ALT_IP_1)\n    set(CERT_ALT_IP_1 127.0.0.1)\n  endif()\n  if (NOT CERT_ALT_DNS)\n    set(CERT_ALT_DNS ${CERT_COMMON})\n  endif()\n  if (NOT CERT_WIDTH)\n    set(CERT_WIDTH 4096)\n  endif()\n  if (NOT CERT_DAYS)\n    set(CERT_DAYS 3650)\n  endif()\n  if ((NOT CERT_ALG) AND (NOT CERT_ALG_OPTS))\n    set(CERT_ALG_OPTS -pkeyopt rsa_keygen_bits:${CERT_WIDTH})\n  endif()\n  if (NOT CERT_ALG)\n    set(CERT_ALG RSA)\n  endif()\n\n  set(CERT_PRIVKEY ${CERT_NAME}.key)\n  set(CERT_REQ ${CERT_NAME}.csr)\n  set(CERT_CERT ${CERT_NAME}.crt)\n  set(CERT_CERT_DER ${CERT_NAME}.crt.der)\n\n  set(CERT_CAPRIVKEY ca${CERT_NAME}.key)\n  set(CERT_CAROOT ca${CERT_NAME}.pem)\n\n  configure_file(\"${CMAKE_CURRENT_SOURCE_DIR}/cert.cfg.in\"\n    \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_NAME}.cfg\"\n  )\n\n  find_program(OPENSSL openssl)\n  if (NOT OPENSSL)\n    message(FATAL_ERROR \"openssl is required for performing tests!\")\n  endif ()\n\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_PRIVKEY}\"\n    COMMAND ${OPENSSL} genpkey -quiet -out ${CERT_PRIVKEY} -algorithm ${CERT_ALG} ${CERT_ALG_OPTS}\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_REQ}\"\n    COMMAND ${OPENSSL} req -new -key ${CERT_PRIVKEY} -out ${CERT_REQ} -config ${CERT_NAME}.cfg\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_PRIVKEY}\" \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_NAME}.cfg\"\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CAPRIVKEY}\"\n    COMMAND ${OPENSSL} genpkey -quiet -out ${CERT_CAPRIVKEY} -algorithm ${CERT_ALG} ${CERT_ALG_OPTS}\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CAROOT}\"\n    COMMAND ${OPENSSL} req -x509 -new -nodes -key ${CERT_CAPRIVKEY} -days ${CERT_DAYS} -config ${CERT_NAME}.cfg -out ${CERT_CAROOT}\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CAPRIVKEY}\" \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_NAME}.cfg\"\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n\n\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CERT}\"\n    COMMAND ${OPENSSL} x509 -req -in ${CERT_REQ} -CA ${CERT_CAROOT} -CAkey ${CERT_CAPRIVKEY} -CAcreateserial -out ${CERT_CERT} -days ${CERT_DAYS} -extensions req_ext -extfile ${CERT_NAME}.cfg\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_REQ}\"  \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CAROOT}\" \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_NAME}.cfg\"\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n\n  add_custom_command(OUTPUT \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CERT_DER}\"\n    COMMAND ${OPENSSL} x509 -in ${CERT_CERT} -out ${CERT_CERT_DER} -outform der\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CERT}\"\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}\n  )\n\n  add_custom_target(${name}\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/${CERT_CERT_DER}\"\n  )\nendfunction()\n\nfunction(seastar_sign_cert name)\n  # create and sign a cert with specified CN and optional CA\n  cmake_parse_arguments(CERT\n    \"\"\n    \"CA;SERIAL_NUM;COMMON_NAME\"\n    \"\"\n    ${ARGN})\n  set(cert ${name}.crt)\n  set(privkey ${name}.key)\n  set(extension \"subjectAltName = IP:127.0.0.1\")\n  set(subj \"/C=GB/ST=London/L=London/O=Redpanda Data/OU=Core/CN=${CERT_COMMON_NAME}\")\n\n  add_custom_command(OUTPUT ${privkey}\n    COMMAND ${OPENSSL} ecparam\n      -name prime256v1 -genkey -noout\n      -out ${privkey}\n    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n\n  if(DEFINED CERT_CA)\n    # create a request instead of a cert if CA is specified\n    set(req ${name}.csr)\n    add_custom_command(OUTPUT ${req}\n      COMMAND ${OPENSSL} req\n        -new -sha256\n        -key ${privkey}\n        -out ${req}\n        -subj ${subj}\n      DEPENDS ${privkey}\n      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n\n    # sign the cert with the request\n    set(ca_cert ${CERT_CA}.crt)\n    set(ca_privkey ${CERT_CA}.key)\n    add_custom_command(OUTPUT ${cert}\n      COMMAND ${OPENSSL} x509\n        -req -days 1000 -sha256\n        -set_serial ${CERT_SERIAL_NUM}\n        -in ${req}\n        -CA ${ca_cert}\n        -CAkey ${ca_privkey}\n        -out ${cert}\n      DEPENDS ${req} ${ca_cert} ${ca_privkey}\n      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n  else(DEFINED CERT_CA)\n    # create a cert if CA is not specified\n    add_custom_command(OUTPUT ${cert}\n      COMMAND ${OPENSSL} req\n        -new -x509 -sha256\n        -key ${privkey}\n        -out ${cert}\n        -subj ${subj}\n        -addext ${extension}\n      DEPENDS ${privkey}\n      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})\n  endif(DEFINED CERT_CA)\nendfunction(seastar_sign_cert)\n\nfunction(seastar_gen_mtls_certs)\n  find_program(OPENSSL openssl)\n  if (NOT OPENSSL)\n    message (FATAL_ERROR \"openssl is required for performing tests!\")\n  endif ()\n\n  # create a CA cert\n  set(cert_ca \"mtls_ca\")\n  seastar_sign_cert(${cert_ca}\n    COMMON_NAME \"redpanda.com\")\n  # server certificates\n  seastar_sign_cert(\"mtls_server\"\n    CA ${cert_ca}\n    SERIAL_NUM 1\n    COMMON_NAME \"redpanda.com\")\n  # client1 certificates\n  seastar_sign_cert(\"mtls_client1\"\n    CA ${cert_ca}\n    SERIAL_NUM 2\n    COMMON_NAME \"client1.org\")\n  # client2 certificates\n  seastar_sign_cert(\"mtls_client2\"\n    CA ${cert_ca}\n    SERIAL_NUM 3\n    COMMON_NAME \"client2.org\")\n\n  add_custom_target(mtls_certs\n    DEPENDS \"${CMAKE_CURRENT_BINARY_DIR}/mtls_client1.crt\" \"${CMAKE_CURRENT_BINARY_DIR}/mtls_client2.crt\" \"${CMAKE_CURRENT_BINARY_DIR}/mtls_server.crt\"\n  )\nendfunction()\n\nseastar_add_certgen(testcrt DOMAIN scylladb.org SERVER test)\nseastar_add_certgen(othercrt DOMAIN apa.org SERVER other)\nseastar_gen_mtls_certs()\n\nset (tls_certificate_files\n  tls-ca-bundle.pem\n)\n\nprepend_each (\n  in_tls_certificate_files\n  ${CMAKE_CURRENT_SOURCE_DIR}/\n  ${tls_certificate_files})\n\nprepend_each (\n  out_tls_certificate_files\n  ${CMAKE_CURRENT_BINARY_DIR}/\n  ${tls_certificate_files})\n\nadd_custom_command (\n  DEPENDS ${in_tls_certificate_files}\n  OUTPUT ${out_tls_certificate_files}\n  COMMAND ${CMAKE_COMMAND} -E copy ${in_tls_certificate_files} ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_custom_target(tls_files\n  DEPENDS ${out_tls_certificate_files}\n)\n\nadd_custom_command (\n  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/https-server.py\n  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/https-server.py\n  COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/https-server.py ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_custom_target (https_server\n  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/https-server.py)\n\nseastar_add_test (tls\n  DEPENDS tls_files testcrt othercrt mtls_certs https_server\n  SOURCES tls_test.cc\n  LIBRARIES Boost::filesystem\n  WORKING_DIRECTORY ${Seastar_BINARY_DIR})\n\nseastar_add_test (tuple_utils\n  KIND BOOST\n  SOURCES tuple_utils_test.cc)\n\nseastar_add_test (unix_domain\n  SOURCES unix_domain_test.cc)\n\nseastar_add_test (unwind\n  KIND BOOST\n  SOURCES unwind_test.cc)\n\nseastar_add_test (weak_ptr\n  KIND BOOST\n  SOURCES weak_ptr_test.cc)\n\nseastar_add_test (log_buf\n  SOURCES log_buf_test.cc)\n\nseastar_add_test (exception_logging\n  KIND BOOST\n  SOURCES exception_logging_test.cc)\n\nseastar_add_test (closeable\n  SOURCES\n    closeable_test.cc\n    expected_exception.hh)\n\nseastar_add_test (pipe\n  SOURCES pipe_test.cc)\n\nseastar_add_test (spawn\n  SOURCES spawn_test.cc)\n\nseastar_add_test (variant_utils\n  SOURCES variant_utils_test.cc)\n\nseastar_generate_swagger (\n  TARGET rest_api_httpd_swagger\n  VAR rest_api_httpd_swagger_files\n  IN_FILE ${CMAKE_CURRENT_SOURCE_DIR}/api.json\n  OUT_DIR ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_executable (rest_api_httpd\n  ${rest_api_httpd_swagger_files}\n  rest_api_httpd.cc)\ntarget_link_libraries (rest_api_httpd\n  PRIVATE seastar_private)\ntarget_include_directories (rest_api_httpd\n  PRIVATE ${CMAKE_CURRENT_BINARY_DIR})\n\nadd_dependencies (rest_api_httpd rest_api_httpd_swagger)\nadd_dependencies (unit_tests rest_api_httpd)\nadd_custom_target (test_unit_json2code_run\n  COMMAND ${CMAKE_COMMAND} -E env ${Seastar_TEST_ENVIRONMENT} ${CMAKE_CURRENT_SOURCE_DIR}/json2code_test.py --rest-api-httpd $<TARGET_FILE:rest_api_httpd>\n  USES_TERMINAL)\nadd_dependencies (test_unit_json2code_run rest_api_httpd)\nadd_test (\n  NAME Seastar.unit.json2code\n  COMMAND ${CMAKE_COMMAND} --build ${Seastar_BINARY_DIR} --target test_unit_json2code_run)\nset_tests_properties (Seastar.unit.json2code\n  PROPERTIES\n    TIMEOUT ${Seastar_TEST_TIMEOUT})\n\nadd_executable (metrics_tester\n  metrics_tester.cc)\ntarget_link_libraries (metrics_tester\n  PRIVATE\n    seastar_private\n    yaml-cpp::yaml-cpp)\n\nadd_dependencies (unit_tests metrics_tester)\nadd_custom_target (test_unit_prometheus_run\n  COMMAND ${CMAKE_COMMAND} -E env ${Seastar_TEST_ENVIRONMENT}\n    ${CMAKE_CURRENT_SOURCE_DIR}/prometheus_test.py\n    --exporter $<TARGET_FILE:metrics_tester>\n    --config ${CMAKE_CURRENT_SOURCE_DIR}/conf-example.yaml\n  USES_TERMINAL)\nadd_dependencies (test_unit_prometheus_run metrics_tester)\nadd_test (\n  NAME Seastar.unit.prometheus\n  COMMAND ${CMAKE_COMMAND} --build ${Seastar_BINARY_DIR} --target test_unit_prometheus_run)\nset_tests_properties (Seastar.unit.prometheus\n  PROPERTIES\n    TIMEOUT ${Seastar_TEST_TIMEOUT})\n\nseastar_add_test (gate\n  SOURCES\n    gate_test.cc)\n\nseastar_add_test (test_fixture\n  SOURCES\n    test_fixture_test.cc)\n"
  },
  {
    "path": "tests/unit/abort_source_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#include <exception>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/gate.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/abort_on_expiry.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/later.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_abort_source_notifies_subscriber) {\n    bool signalled = false;\n    auto as = abort_source();\n    auto st_opt = as.subscribe([&signalled] () noexcept {\n        signalled = true;\n    });\n    BOOST_REQUIRE_EQUAL(true, bool(st_opt));\n    as.request_abort();\n    BOOST_REQUIRE_EQUAL(true, signalled);\n    BOOST_REQUIRE_EQUAL(false, bool(st_opt));\n    BOOST_REQUIRE_THROW(as.check(), abort_requested_exception);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_abort_source_subscription_unregister) {\n    bool signalled = false;\n    auto as = abort_source();\n    auto st_opt = as.subscribe([&signalled] () noexcept {\n        signalled = true;\n    });\n    BOOST_REQUIRE_EQUAL(true, bool(st_opt));\n    st_opt = { };\n    as.request_abort();\n    BOOST_REQUIRE_EQUAL(false, signalled);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_abort_source_rejects_subscription) {\n    auto as = abort_source();\n    as.request_abort();\n    auto st_opt = as.subscribe([] () noexcept { });\n    BOOST_REQUIRE_EQUAL(false, bool(st_opt));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_sleep_abortable) {\n    auto as = std::make_unique<abort_source>();\n    auto f = sleep_abortable(100s, *as).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should have failed\");\n        } catch (const sleep_aborted& e) {\n            // expected\n        } catch (...) {\n            BOOST_FAIL(\"unexpected exception\");\n        }\n    });\n    as->request_abort();\n    return f.finally([as = std::move(as)] { });\n}\n\nSEASTAR_TEST_CASE(test_sleep_abortable_no_abort) {\n    auto as = std::make_unique<abort_source>();\n\n    // Check that the sleep completes as usual if the\n    // abort source doesn't fire.\n    auto f = sleep_abortable<manual_clock>(100s, *as);\n    manual_clock::advance(99s);\n    BOOST_REQUIRE(!f.available());\n    manual_clock::advance(101s);\n    return f.finally([as = std::move(as)] { });\n}\n\n\n\n// Verify that negative sleep does not sleep forever. It should not sleep\n// at all.\nSEASTAR_TEST_CASE(test_negative_sleep_abortable) {\n    return do_with(abort_source(), [] (abort_source& as) {\n        return sleep_abortable(-10s, as);\n    });\n}\n\nSEASTAR_TEST_CASE(test_request_abort_with_exception) {\n    abort_source as;\n    optimized_optional<abort_source::subscription> st_opt;\n    std::optional<std::exception_ptr> aborted_ex;\n    auto expected_message = \"expected\";\n\n    auto make_abort_source = [&] () {\n        as = abort_source();\n        st_opt = as.subscribe([&aborted_ex] (const std::optional<std::exception_ptr>& opt_ex) noexcept {\n            aborted_ex = opt_ex;\n        });\n        aborted_ex = std::nullopt;\n    };\n\n    make_abort_source();\n    auto ex = make_exception_ptr(std::runtime_error(expected_message));\n    as.request_abort_ex(ex);\n    BOOST_REQUIRE(aborted_ex.has_value());\n    bool caught_exception = false;\n    try {\n        std::rethrow_exception(*aborted_ex);\n    } catch (const std::runtime_error& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), expected_message);\n        caught_exception = true;\n    }\n    BOOST_REQUIRE(caught_exception);\n    BOOST_REQUIRE_THROW(as.check(), std::runtime_error);\n\n    make_abort_source();\n    as.request_abort_ex(make_exception_ptr(std::runtime_error(expected_message)));\n    BOOST_REQUIRE(aborted_ex.has_value());\n    caught_exception = false;\n    try {\n        std::rethrow_exception(*aborted_ex);\n    } catch (const std::runtime_error& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), expected_message);\n        caught_exception = true;\n    }\n    BOOST_REQUIRE(caught_exception);\n    BOOST_REQUIRE_THROW(as.check(), std::runtime_error);\n\n\n    make_abort_source();\n    as.request_abort_ex(std::runtime_error(expected_message));\n    BOOST_REQUIRE(aborted_ex.has_value());\n    caught_exception = false;\n    try {\n        std::rethrow_exception(*aborted_ex);\n    } catch (const std::runtime_error& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), expected_message);\n        caught_exception = true;\n    }\n    BOOST_REQUIRE(caught_exception);\n    BOOST_REQUIRE_THROW(as.check(), std::runtime_error);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_sleep_abortable_with_exception) {\n    abort_source as;\n    auto f = sleep_abortable(10s, as);\n    auto expected_message = \"expected\";\n    as.request_abort_ex(std::runtime_error(expected_message));\n\n    bool caught_exception = false;\n    try {\n        f.get();\n    } catch (const std::runtime_error& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), expected_message);\n        caught_exception = true;\n    }\n    BOOST_REQUIRE(caught_exception);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_destroy_with_moved_subscriptions) {\n    auto as = std::make_unique<abort_source>();\n    int aborted = 0;\n    auto sub1 = as->subscribe([&] () noexcept { ++aborted; });\n    auto sub2 = std::move(sub1);\n    optimized_optional<abort_source::subscription> sub3;\n    sub3 = std::move(sub2);\n    auto sub4 = as->subscribe([&] () noexcept { ++aborted; });\n    sub4 = std::move(sub3);\n    as.reset();\n    BOOST_REQUIRE_EQUAL(aborted, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_request_abort_twice) {\n    abort_source as;\n    as.request_abort_ex(std::runtime_error(\"\"));\n    as.request_abort();\n    BOOST_REQUIRE_THROW(as.check(), std::runtime_error);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_on_abort_call_after_abort) {\n    std::exception_ptr signalled_ex;\n    auto as = abort_source();\n    auto sub = as.subscribe([&] (const std::optional<std::exception_ptr>& ex) noexcept {\n        BOOST_REQUIRE(!signalled_ex);\n        signalled_ex = *ex;\n    });\n    BOOST_REQUIRE_EQUAL(bool(sub), true);\n    BOOST_REQUIRE(signalled_ex == nullptr);\n\n    // on_abort should trigger the subscribed callback\n    as.request_abort_ex(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE_EQUAL(bool(sub), false);\n    BOOST_REQUIRE(signalled_ex != nullptr);\n    BOOST_REQUIRE_THROW(std::rethrow_exception(signalled_ex), std::runtime_error);\n\n    // on_abort is single-shot\n    signalled_ex = nullptr;\n    sub->on_abort(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE(signalled_ex == nullptr);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_on_abort_call_before_abort) {\n    std::exception_ptr signalled_ex;\n    auto as = abort_source();\n    auto sub = as.subscribe([&] (const std::optional<std::exception_ptr>& ex) noexcept {\n        BOOST_REQUIRE(!signalled_ex);\n        signalled_ex = *ex;\n    });\n    BOOST_REQUIRE_EQUAL(bool(sub), true);\n    BOOST_REQUIRE(signalled_ex == nullptr);\n\n    // on_abort should trigger the subscribed callback\n    sub->on_abort(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE_EQUAL(bool(sub), false);\n    BOOST_REQUIRE(signalled_ex != nullptr);\n    BOOST_REQUIRE_THROW(std::rethrow_exception(signalled_ex), std::runtime_error);\n\n    // subscription is single-shot\n    signalled_ex = nullptr;\n    as.request_abort_ex(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE(signalled_ex == nullptr);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_subscribe_aborted_source) {\n    std::exception_ptr signalled_ex;\n    auto as = abort_source();\n    as.request_abort();\n    auto sub = as.subscribe([&] (const std::optional<std::exception_ptr>& ex) noexcept {\n        BOOST_REQUIRE(!signalled_ex);\n        signalled_ex = *ex;\n    });\n\n    // subscription is expected to evaluate to false\n    // if abort_source was already aborted\n    BOOST_REQUIRE_EQUAL(bool(sub), false);\n    BOOST_REQUIRE(signalled_ex == nullptr);\n\n    // on_abort should trigger the subscribed callback\n    // if abort_source was already aborted\n    sub->on_abort(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE(signalled_ex != nullptr);\n    BOOST_REQUIRE_THROW(std::rethrow_exception(signalled_ex), std::runtime_error);\n\n    // on_abort is single-shot\n    signalled_ex = nullptr;\n    sub->on_abort(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE(signalled_ex == nullptr);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_subscription_callback_lifetime) {\n    // The subscription callback function needs to be destroyed\n    // only when the subscription is destroyed.\n    bool callback_destroyed = false;\n    int callback_called = 0;\n    auto when_destroyed = deferred_action([&callback_destroyed] () noexcept { callback_destroyed = true; });\n    auto as = abort_source();\n    auto sub = std::make_unique<optimized_optional<abort_source::subscription>>(as.subscribe([&, when_destroyed = std::move(when_destroyed)] (const std::optional<std::exception_ptr>& ex) noexcept {\n        callback_called++;\n    }));\n    BOOST_REQUIRE_EQUAL(bool(sub), true);\n    BOOST_REQUIRE_EQUAL(bool(*sub), true);\n    BOOST_REQUIRE_EQUAL(callback_destroyed, false);\n    BOOST_REQUIRE_EQUAL(callback_called, 0);\n\n    // on_abort should trigger the subscribed callback\n    as.request_abort_ex(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE_EQUAL(bool(*sub), false);\n    BOOST_REQUIRE_EQUAL(callback_destroyed, false);\n    BOOST_REQUIRE_EQUAL(callback_called, 1);\n\n    // on_abort is single-shot\n    (*sub)->on_abort(std::make_exception_ptr(std::runtime_error(\"signaled\")));\n    BOOST_REQUIRE_EQUAL(callback_destroyed, false);\n    BOOST_REQUIRE_EQUAL(callback_called, 1);\n\n    sub.reset();\n    BOOST_REQUIRE_EQUAL(callback_destroyed, true);\n    BOOST_REQUIRE_EQUAL(callback_called, 1);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_abort_on_expiry) {\n    auto abort = abort_on_expiry<manual_clock>(manual_clock::now() + 1s);\n    std::exception_ptr ex;\n    int called = 0;\n    auto sub = abort.abort_source().subscribe([&] (const std::optional<std::exception_ptr>& ex_opt) noexcept {\n        called++;\n        if (ex_opt) {\n            ex = *ex_opt;\n        }\n    });\n    BOOST_REQUIRE(!abort.abort_source().abort_requested());\n    BOOST_REQUIRE(!called);\n    manual_clock::advance(1s);\n    yield().get();\n    BOOST_REQUIRE(abort.abort_source().abort_requested());\n    BOOST_REQUIRE_EQUAL(called, 1);\n    BOOST_REQUIRE(ex != nullptr);\n    BOOST_REQUIRE_THROW(std::rethrow_exception(ex), timed_out_error);\n}\n"
  },
  {
    "path": "tests/unit/abortable_fifo_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#include <chrono>\n#include <exception>\n\n#include <seastar/core/thread.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/abortable_fifo.hh>\n#include <seastar/core/abort_on_expiry.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/core/timed_out_error.hh>\n#include <seastar/util/later.hh>\n#include <boost/range/irange.hpp>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_no_abortable_operations) {\n    internal::abortable_fifo<int> fifo;\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n\n    fifo.push_back(1);\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    fifo.push_back(2);\n    fifo.push_back(3);\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 3u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 2);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_abortable_operations) {\n    std::vector<int> expired;\n    struct my_expiry {\n        std::vector<int>& e;\n        void operator()(int& v) noexcept { e.push_back(v); }\n    };\n\n    internal::abortable_fifo<int, my_expiry> fifo(my_expiry{expired});\n    abort_source as;\n\n    fifo.push_back(1, as);\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    as.request_abort();\n    yield().get();\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n    BOOST_REQUIRE_EQUAL(expired.size(), 1u);\n    BOOST_REQUIRE_EQUAL(expired[0], 1);\n\n    expired.clear();\n    as = abort_source();\n\n    fifo.push_back(1);\n    fifo.push_back(2, as);\n    fifo.push_back(3);\n\n    as.request_abort();\n    yield().get();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(expired.size(), 1u);\n    BOOST_REQUIRE_EQUAL(expired[0], 2);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n    expired.clear();\n\n    abort_source as1, as2;\n\n    fifo.push_back(1, as1);\n    fifo.push_back(2, as1);\n    fifo.push_back(3);\n    fifo.push_back(4, as2);\n\n    as1.request_abort();\n    yield().get();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(expired.size(), 2u);\n    std::sort(expired.begin(), expired.end());\n    BOOST_REQUIRE_EQUAL(expired[0], 1);\n    BOOST_REQUIRE_EQUAL(expired[1], 2);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 4);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n    expired.clear();\n\n    as = abort_source();\n\n    fifo.push_back(1);\n    fifo.push_back(2, as);\n    fifo.push_back(3, as);\n    fifo.push_back(4, as);\n\n    as.request_abort();\n    yield().get();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(expired.size(), 3u);\n    std::sort(expired.begin(), expired.end());\n    BOOST_REQUIRE_EQUAL(expired[0], 2);\n    BOOST_REQUIRE_EQUAL(expired[1], 3);\n    BOOST_REQUIRE_EQUAL(expired[2], 4);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n    expired.clear();\n    as = abort_source();\n\n    fifo.push_back(1);\n    fifo.push_back(2, as);\n    fifo.push_back(3, as);\n    fifo.push_back(4, as);\n    fifo.push_back(5);\n\n    as.request_abort();\n    yield().get();\n\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 5);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_abort_exception) {\n    struct entry {\n        int value;\n        std::exception_ptr ex;\n    };\n    std::vector<entry> expired;\n    struct my_expiry {\n        std::vector<entry>& e;\n        void operator()(int& v, const std::optional<std::exception_ptr>& ex) noexcept { e.push_back(entry{v, ex.value_or(nullptr)}); }\n    };\n\n    internal::abortable_fifo<int, my_expiry> fifo(my_expiry{expired});\n    auto aoe = abort_on_expiry<manual_clock>(manual_clock::now() + 1s);\n\n    fifo.push_back(1, aoe.abort_source());\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    manual_clock::advance(1s);\n    yield().get();\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n    BOOST_REQUIRE_EQUAL(expired.size(), 1u);\n    BOOST_REQUIRE_EQUAL(expired[0].value, 1);\n    BOOST_REQUIRE(expired[0].ex);\n    BOOST_REQUIRE_THROW(std::rethrow_exception(expired[0].ex), timed_out_error);\n}\n"
  },
  {
    "path": "tests/unit/alien_test.cc",
    "content": "// -*- mode:C++; tab-width:4; c-basic-offset:4; indent-tabs-mode:nil -*-\n/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 Red Hat\n */\n\n#include <future>\n#include <numeric>\n#include <iostream>\n#include <seastar/core/alien.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/posix.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/util/later.hh>\n#include <stdexcept>\n#include <ranges>\n#include <tuple>\n\n\nusing namespace seastar;\n\nenum {\n    ENGINE_READY = 24,\n    ALIEN_DONE   = 42,\n};\n\nint main(int argc, char** argv)\n{\n    // we need a protocol that both seastar and alien understand.\n    // and on which, a seastar future can wait.\n    int engine_ready_fd = eventfd(0, 0);\n    auto alien_done = file_desc::eventfd(0, 0);\n    seastar::app_template app;\n\n    // use the raw fd, because seastar engine want to take over the fds, if it\n    // polls on them.\n    auto zim = std::async([&app, engine_ready_fd,\n                           alien_done=alien_done.get()] {\n        eventfd_t result = 0;\n        // wait until the seastar engine is ready\n        int r = ::eventfd_read(engine_ready_fd, &result);\n        if (r < 0) {\n            throw std::runtime_error(\"failed to wait for seastar engine\");\n        }\n        if (result != ENGINE_READY) {\n            throw std::runtime_error(\"seastar failed to sent us the ready message\");\n        }\n        // test for alien::run_on()\n        std::promise<char> question;\n        auto answer = question.get_future();\n        alien::run_on(app.alien(), 0, [&question]() noexcept {\n            question.set_value('*');\n        });\n        // test for alien::submit_to(), which returns a std::future<int>\n        std::vector<std::future<int>> counts;\n        for (auto i : std::views::iota(0u, smp::count)) {\n            // send messages from alien.\n            counts.push_back(alien::submit_to(app.alien(), i, [i] {\n                return seastar::make_ready_future<int>(i);\n            }));\n        }\n        // test for alien::submit_to(), which returns a std::future<void>\n        alien::submit_to(app.alien(), 0, [] {\n            return seastar::make_ready_future<>();\n        }).wait();\n        int total = 0;\n        for (auto& count : counts) {\n            total += count.get();\n        }\n        // i am done. dismiss the engine\n        ::eventfd_write(alien_done, ALIEN_DONE);\n        return std::make_tuple(answer.get(), total);\n    });\n\n    eventfd_t result = 0;\n    app.run(argc, argv, [&] {\n        return seastar::now().then([engine_ready_fd] {\n            // engine ready!\n            ::eventfd_write(engine_ready_fd, ENGINE_READY);\n            return seastar::now();\n        }).then([alien_done = std::move(alien_done), &result]() mutable {\n            return do_with(seastar::pollable_fd(std::move(alien_done)), [&result] (pollable_fd& alien_done_fds) {\n                // check if alien has dismissed me.\n                return alien_done_fds.readable().then([&result, &alien_done_fds] {\n                    auto ret = alien_done_fds.get_file_desc().read(&result, sizeof(result));\n                    return make_ready_future<size_t>(*ret);\n                });\n            });\n        }).then([&result](size_t n) {\n            if (n != sizeof(result)) {\n                throw std::runtime_error(\"read from eventfd failed\");\n            }\n            if (result != ALIEN_DONE) {\n                throw std::logic_error(\"alien failed to dismiss me\");\n            }\n            return seastar::now();\n        }).handle_exception([](auto ep) {\n            std::cerr << \"Error: \" << ep << std::endl;\n        }).finally([] {\n            seastar::engine().exit(0);\n        });\n    });\n    auto [everything, total] = zim.get();\n    if (char expected = '*'; everything != '*') {\n        std::cerr << \"Bad everything: \" << everything << \" != \" << expected << std::endl;\n        return 1;\n    }\n    const auto shards = std::views::iota(0u, smp::count);\n    auto expected = std::accumulate(std::begin(shards), std::end(shards), 0);\n    if (total != expected) {\n        std::cerr << \"Bad total: \" << total << \" != \" << expected << std::endl;\n        return 1;\n    }\n}\n"
  },
  {
    "path": "tests/unit/alloc_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/memory.hh>\n#include <seastar/core/shard_id.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/testing/perf_tests.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/memory_diagnostics.hh>\n\n#include <memory>\n#include <new>\n#include <vector>\n#include <future>\n#include <iostream>\n\n#include <malloc.h>\n#include <stdlib.h>\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(alloc_almost_all_and_realloc_it_with_a_smaller_size) {\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n    auto all = memory::stats().total_memory();\n    auto reserve = size_t(0.02 * all);\n    auto to_alloc = all - (reserve + (10 << 20));\n    auto orig_to_alloc = to_alloc;\n    auto obj = malloc(to_alloc);\n    while (!obj) {\n        to_alloc *= 0.9;\n        obj = malloc(to_alloc);\n    }\n    BOOST_REQUIRE(to_alloc > orig_to_alloc / 4);\n    BOOST_REQUIRE(obj != nullptr);\n    auto obj2 = realloc(obj, to_alloc - (1 << 20));\n    BOOST_REQUIRE(obj == obj2);\n    free(obj2);\n#endif\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(malloc_0_and_free_it) {\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n    auto obj = malloc(0);\n    BOOST_REQUIRE(obj != nullptr);\n    free(obj);\n#endif\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(new_0) {\n\n    {\n        // new must always return a non-null pointer, even for 0 size\n        auto obj = operator new(0);\n        BOOST_REQUIRE(obj != nullptr);\n        operator delete(obj);\n    }\n\n    {\n        // same test but with a zero length array\n        auto obj = new char[0];\n        BOOST_REQUIRE(obj != nullptr);\n        delete [] obj;\n    }\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_live_objects_counter_with_cross_cpu_free) {\n    return smp::submit_to(1, [] {\n        auto ret = std::vector<std::unique_ptr<bool>>(1000000);\n        for (auto& o : ret) {\n            o = std::make_unique<bool>(false);\n        }\n        return ret;\n    }).then([] (auto&& vec) {\n        vec.clear(); // cause cross-cpu free\n        BOOST_REQUIRE(memory::stats().live_objects() < std::numeric_limits<size_t>::max() / 2);\n    });\n}\n\nSEASTAR_TEST_CASE(test_aligned_alloc) {\n    for (size_t align = sizeof(void*); align <= 65536; align <<= 1) {\n        for (size_t size = align; size <= align * 2; size <<= 1) {\n            void *p = aligned_alloc(align, size);\n            BOOST_REQUIRE(p != nullptr);\n            BOOST_REQUIRE((reinterpret_cast<uintptr_t>(p) % align) == 0);\n            ::memset(p, 0, size);\n            free(p);\n        }\n    }\n    return make_ready_future<>();\n}\n\n#ifdef __cpp_sized_deallocation\nSEASTAR_TEST_CASE(test_sized_delete) {\n    for (size_t size = 0; size <= 65536; size++) {\n        void *p0 = operator new(size), *p1 = operator new(size);\n        BOOST_REQUIRE(p0 != nullptr);\n        BOOST_REQUIRE(p1 != nullptr);\n        ::memset(p0, 1, size);\n        ::memset(p1, 2, size);\n        perf_tests::do_not_optimize(p0);\n        perf_tests::do_not_optimize(p1);\n        operator delete(p0, size);\n        operator delete(p1, size);\n    }\n    return make_ready_future<>();\n}\n#endif\n\nSEASTAR_TEST_CASE(test_temporary_buffer_aligned) {\n    for (size_t align = sizeof(void*); align <= 65536; align <<= 1) {\n        for (size_t size = align; size <= align * 2; size <<= 1) {\n            auto buf = temporary_buffer<char>::aligned(align, size);\n            void *p = buf.get_write();\n            BOOST_REQUIRE(p != nullptr);\n            BOOST_REQUIRE((reinterpret_cast<uintptr_t>(p) % align) == 0);\n            ::memset(p, 0, size);\n        }\n    }\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_memory_diagnostics) {\n    auto report = memory::generate_memory_diagnostics_report();\n#ifdef SEASTAR_DEFAULT_ALLOCATOR\n    BOOST_REQUIRE(report.length() == 0); // empty report with default allocator\n#else\n    // since the output format is unstructured text, not much\n    // to do except test that we get a non-empty string\n    BOOST_REQUIRE(report.length() > 0);\n    // useful while debugging diagnostics\n    // fmt::print(\"--------------------\\n{}--------------------\", report);\n#endif\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_cross_thread_realloc) {\n    // Tests that realloc seems to do the right thing with various sizes of\n    // buffer, including cases where the initial allocation is on another\n    // shard.\n    // Needs at least 2 shards to usefully test the cross-shard aspect but\n    // still passes when only 1 shard is used.\n    auto do_xshard_realloc = [](bool cross_shard, size_t initial_size, size_t realloc_size) {\n        BOOST_TEST_CONTEXT(\"cross_shard=\" << cross_shard << \", initial=\"\n                << initial_size << \", realloc_size=\" << realloc_size) {\n\n            auto other_shard = (this_shard_id() + cross_shard) % smp::count;\n\n            char *p = static_cast<char *>(malloc(initial_size));\n\n            // write some sentinels and check them after realloc\n            // x start of region\n            // y end of realloc'd region (if it falls within the initial size)\n            // z end of initial region\n            if (initial_size > 0) {\n                p[0] = 'x';\n                p[initial_size - 1] = 'z';\n                if (realloc_size > 0 && realloc_size <= initial_size) {\n                    p[realloc_size - 1] = 'y';\n                }\n            }\n            smp::submit_to(other_shard, [=] {\n                char* p2 = static_cast<char *>(realloc(p, realloc_size));\n                if (initial_size > 0 && realloc_size > 0) {\n                    BOOST_REQUIRE_EQUAL(p2[0], 'x');\n                    if (realloc_size <= initial_size) {\n                        BOOST_REQUIRE_EQUAL(p2[realloc_size - 1], 'y');\n                    }\n                    if (realloc_size > initial_size) {\n                        BOOST_REQUIRE_EQUAL(p2[initial_size - 1], 'z');\n                    }\n                }\n                free(p2);\n            }).get();\n        }\n    };\n\n    for (auto& cross_shard : {false, true}) {\n        do_xshard_realloc(cross_shard, 0, 0);\n        do_xshard_realloc(cross_shard, 0, 1);\n        do_xshard_realloc(cross_shard, 1, 0);\n        do_xshard_realloc(cross_shard, 50, 100);\n        do_xshard_realloc(cross_shard, 100, 50);\n        do_xshard_realloc(cross_shard, 100000, 500000);\n        do_xshard_realloc(cross_shard, 500000, 100000);\n    }\n}\n\n\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n\nstruct thread_alloc_info {\n    memory::statistics before;\n    memory::statistics after;\n    void *ptr;\n};\n\ntemplate <typename Func>\nthread_alloc_info run_with_stats(Func&& f) {\n    return std::async([&f](){\n        auto before = seastar::memory::stats();\n        void* ptr = f();\n        auto after = seastar::memory::stats();\n        return thread_alloc_info{before, after, ptr};\n    }).get();\n}\n\ntemplate <typename Func>\nvoid test_allocation_function(Func f) {\n    // alien alloc and free\n    auto alloc_info = run_with_stats(f);\n    auto free_info = std::async([p = alloc_info.ptr]() {\n        auto before = seastar::memory::stats();\n        free(p);\n        auto after = seastar::memory::stats();\n        return thread_alloc_info{before, after, nullptr};\n    }).get();\n\n    // there were mallocs\n    BOOST_REQUIRE(alloc_info.after.foreign_mallocs() - alloc_info.before.foreign_mallocs() > 0);\n    // mallocs balanced with frees\n    BOOST_REQUIRE(alloc_info.after.foreign_mallocs() - alloc_info.before.foreign_mallocs() == free_info.after.foreign_frees() - free_info.before.foreign_frees());\n\n    // alien alloc reactor free\n    auto info = run_with_stats(f);\n    auto before_cross_frees = memory::stats().foreign_cross_frees();\n    free(info.ptr);\n    BOOST_REQUIRE(memory::stats().foreign_cross_frees() - before_cross_frees == 1);\n\n    // reactor alloc, alien free\n    void *p = f();\n    auto alien_cross_frees = std::async([p]() {\n        auto frees_before = memory::stats().cross_cpu_frees();\n        free(p);\n        return memory::stats().cross_cpu_frees()-frees_before;\n    }).get();\n    BOOST_REQUIRE(alien_cross_frees == 1);\n}\n\nSEASTAR_TEST_CASE(test_foreign_function_use_glibc_malloc) {\n    test_allocation_function([]() ->void * { return malloc(1); });\n    test_allocation_function([]() { return realloc(NULL, 10); });\n    test_allocation_function([]() {\n        auto p = malloc(1);\n        return realloc(p, 1000);\n    });\n    test_allocation_function([]() { return aligned_alloc(4, 1024); });\n    return make_ready_future<>();\n}\n\n// So the compiler won't optimize the call to realloc(nullptr, size)\n// and call malloc directly.\nvoid* test_nullptr = nullptr;\n\nSEASTAR_TEST_CASE(test_realloc_nullptr) {\n    auto p0 = realloc(test_nullptr, 8);\n    BOOST_REQUIRE(p0 != nullptr);\n    BOOST_REQUIRE_EQUAL(realloc(p0, 0), nullptr);\n\n    p0 = realloc(test_nullptr, 0);\n    BOOST_REQUIRE(p0 != nullptr);\n    auto p1 = malloc(0);\n    BOOST_REQUIRE(p1 != nullptr);\n    free(p0);\n    free(p1);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_enable_abort_on_oom) {\n    bool original = seastar::memory::is_abort_on_allocation_failure();\n\n    seastar::memory::set_abort_on_allocation_failure(false);\n    BOOST_CHECK(!seastar::memory::is_abort_on_allocation_failure());\n\n    seastar::memory::set_abort_on_allocation_failure(true);\n    BOOST_CHECK(seastar::memory::is_abort_on_allocation_failure());\n\n    seastar::memory::set_abort_on_allocation_failure(original);\n\n    return make_ready_future<>();\n}\n\nvoid * volatile sink;\n\nSEASTAR_TEST_CASE(test_bad_alloc_throws) {\n    // test that a large allocation throws bad_alloc\n    auto stats = seastar::memory::stats();\n\n    // this allocation cannot be satisfied (at least when the seastar\n    // allocator is used, which it is for this test)\n    size_t size = stats.total_memory() * 2;\n\n    auto failed_allocs = [&stats]() {\n        return seastar::memory::stats().failed_allocations() - stats.failed_allocations();\n    };\n\n    // test that new throws\n    stats = seastar::memory::stats();\n    BOOST_REQUIRE_THROW(sink = operator new(size), std::bad_alloc);\n    BOOST_CHECK_EQUAL(failed_allocs(), 1);\n\n    // test that new[] throws\n    stats = seastar::memory::stats();\n    BOOST_REQUIRE_THROW(sink = new char[size], std::bad_alloc);\n    BOOST_CHECK_EQUAL(failed_allocs(), 1);\n\n    // test that huge malloc returns null\n    stats = seastar::memory::stats();\n    BOOST_REQUIRE_EQUAL(malloc(size), nullptr);\n    BOOST_CHECK_EQUAL(failed_allocs(), 1);\n\n    // test that huge realloc on nullptr returns null\n    stats = seastar::memory::stats();\n    BOOST_REQUIRE_EQUAL(realloc(nullptr, size), nullptr);\n    BOOST_CHECK_EQUAL(failed_allocs(), 1);\n\n    // test that huge realloc on an existing ptr returns null\n    stats = seastar::memory::stats();\n    void *p = malloc(1);\n    BOOST_REQUIRE(p);\n    void *p2 = realloc(p, size);\n    BOOST_REQUIRE_EQUAL(p2, nullptr);\n    BOOST_CHECK_EQUAL(failed_allocs(), 1);\n    free(p2 ?: p);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_diagnostics_failures) {\n    // test that an allocation failure is reflected in the diagnostics\n    auto stats = seastar::memory::stats();\n\n    size_t size = stats.total_memory() * 2; // cannot be satisfied\n\n    // we expect that the failure is immediately reflected in the diagnostics\n    try {\n        sink = operator new(size);\n    } catch (const std::bad_alloc&) {}\n\n    auto report = memory::generate_memory_diagnostics_report();\n\n    // +1 because we caused one additional hard failure from the allocation above\n    auto expected = fmt::format(\"Hard failures: {}\", stats.failed_allocations() + 1);\n\n    if (report.find(expected) == seastar::sstring::npos) {\n        BOOST_FAIL(fmt::format(\"Did not find expected message: {} in\\n{}\\n\", expected, report));\n    }\n\n    return seastar::make_ready_future();\n}\n\ntemplate <typename Func>\nrequires requires (Func fn) { fn(); }\nvoid check_function_allocation(const char* name, size_t expected_allocs, Func f) {\n    auto before = seastar::memory::stats();\n    f();\n    auto after = seastar::memory::stats();\n\n    BOOST_TEST_INFO(\"After function: \" << name);\n    BOOST_REQUIRE_EQUAL(expected_allocs, after.mallocs() - before.mallocs());\n}\n\nSEASTAR_TEST_CASE(test_diagnostics_allocation) {\n\n    check_function_allocation(\"empty\", 0, []{});\n\n    check_function_allocation(\"operator new\", 1, []{\n        // note that many pairs of malloc/free-alikes can just be optimized\n        // away, but not operator new(size_t), per the standard\n        void * volatile p = operator new(1);\n        operator delete(p);\n    });\n\n    // The meat of this test. Dump the diagnostics report to the log and ensure it\n    // doesn't allocate. Doing it lots is important because it may alloc only occasionally:\n    // a real example being the optimized timestamp logging which (used to) make an allocation\n    // only once a second.\n    check_function_allocation(\"log_memory_diagnostics_report\", 0, [&]{\n        for (int i = 0; i < 1000; i++) {\n            seastar::memory::internal::log_memory_diagnostics_report(log_level::info);\n        }\n    });\n\n    return seastar::make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_two_allocations_increase_total_bytes_allocated) {\n    auto before = seastar::memory::stats();\n\n    constexpr size_t size1 = 128;\n    constexpr size_t size2 = 256;\n\n    void* volatile p1 = operator new(size1);\n    void* volatile p2 = operator new(size2);\n\n    auto after = seastar::memory::stats();\n\n    BOOST_REQUIRE_EQUAL(after.total_bytes_allocated() - before.total_bytes_allocated(), size1 + size2);\n\n    operator delete(p2);\n    operator delete(p1);\n\n    return seastar::make_ready_future();\n}\n\n#ifdef SEASTAR_HEAPPROF\n\n// small wrapper to disincentivize gcc from unrolling the loop\n[[gnu::noinline]]\nchar* malloc_wrapper(size_t size) {\n    auto ret = static_cast<char*>(malloc(size));\n    *ret = 'c'; // to prevent compiler from considering this a dead allocation and optimizing it out\n    return ret;\n}\n\nnamespace seastar::memory {\nstd::ostream& operator<<(std::ostream& os, const allocation_site& site) {\n    os << \"allocation_site[count: \" << site.count << \", size: \" << site.size << \"]\";\n    return os;\n}\n}\n\nSEASTAR_TEST_CASE(test_sampled_profile_collection_small)\n{\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    std::size_t count = 100;\n    std::vector<volatile char*> ptrs(count);\n\n    seastar::memory::set_heap_profiling_sampling_rate(100);\n\n#ifdef __clang__\n    #pragma nounroll\n#endif\n    for (std::size_t i = 0; i < count / 2; ++i) {\n        ptrs[i] = malloc_wrapper(10);\n    }\n\n#ifdef __clang__\n    #pragma nounroll\n#endif\n    for (std::size_t i = count / 2; i < count; ++i) {\n        ptrs[i] = malloc_wrapper(10);\n    }\n\n    auto get_samples = []() {\n        auto stats0 = seastar::memory::sampled_memory_profile();\n        auto stats1 = seastar::memory::sampled_memory_profile();\n\n        // two back-to-back copies of the sample should have the same value\n        BOOST_CHECK_EQUAL(stats0, stats1);\n\n        // check that we get the same value from the raw array iterface\n        std::vector<seastar::memory::allocation_site> stats2(stats0.size());\n        auto sz2 = seastar::memory::sampled_memory_profile(stats2.data(), stats2.size());\n        BOOST_CHECK_EQUAL(stats0.size(), sz2);\n        BOOST_CHECK_EQUAL(stats0, stats2);\n\n        // check with +1 size, we expect to still only get size elements\n        std::vector<seastar::memory::allocation_site> stats3(stats0.size() + 1);\n        auto sz3 = seastar::memory::sampled_memory_profile(stats3.data(), stats3.size());\n        BOOST_CHECK_EQUAL(stats0.size(), sz3);\n        stats3.resize(sz3);\n        BOOST_CHECK_EQUAL(stats0, stats3);\n\n        return stats0;\n    };\n\n    // NB: the test framework allocates\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = get_samples();\n        BOOST_REQUIRE_EQUAL(stats.size(), 2);\n        BOOST_REQUIRE_EQUAL(stats[0].size, stats[0].count * 100);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(100);\n\n    for (auto ptr : ptrs) {\n        free((void*)ptr);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = get_samples();\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    return seastar::make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_sampled_profile_collection_large)\n{\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    std::size_t count = 100;\n    std::vector<volatile char*> ptrs(count);\n\n    seastar::memory::set_heap_profiling_sampling_rate(1000000);\n\n#ifdef __clang__\n    #pragma nounroll\n#endif\n    for (std::size_t i = 0; i < count / 2; ++i) {\n        ptrs[i] = malloc_wrapper(100000);\n    }\n\n#ifdef __clang__\n    #pragma nounroll\n#endif\n    for (std::size_t i = count / 2; i < count; ++i) {\n        ptrs[i] = malloc_wrapper(100000);\n    }\n\n    // NB: the test framework allocate\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 2);\n        BOOST_REQUIRE_EQUAL(stats[0].size, stats[0].count * 1000000);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(1000000);\n\n    for (auto ptr : ptrs) {\n        free((void*)ptr);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        // NOTE this is because right now the tracking structure doesn't delete call sites ever\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    return seastar::make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_sampled_profile_collection_max_sites)\n{\n    std::size_t count = 1010;\n    std::vector<volatile char*> ptrs(count);\n\n    seastar::memory::set_heap_profiling_sampling_rate(100);\n\n    #pragma GCC unroll 1010\n    for (std::size_t i = 0; i < count; ++i) {\n        volatile char* ptr = static_cast<char*>(malloc(1000));\n        *ptr = 'c'; // to prevent compiler from considering this a dead allocation and optimizing it out\n        ptrs[i] = ptr;\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 1000);\n    }\n\n    for (auto ptr : ptrs) {\n        free((void*)ptr);\n    }\n\n    return seastar::make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_change_sample_rate)\n{\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    std::size_t sample_rate = 100;\n    std::size_t count = 10000;\n    std::vector<volatile char*> ptrs(count);\n\n    seastar::memory::set_heap_profiling_sampling_rate(sample_rate);\n\n#ifdef __clang__\n    #pragma nounroll\n#endif\n    for (std::size_t i = 0; i < count; ++i) {\n        ptrs[i] = malloc_wrapper(10);\n    }\n\n    // NB: the test framework allocates\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    size_t last_alloc_size = 0;\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 1);\n        last_alloc_size = stats[0].size;\n        BOOST_REQUIRE_EQUAL(stats[0].size, stats[0].count * sample_rate);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(sample_rate);\n\n    size_t free_iter = 0;\n    // free some of the allocations to check size changes\n    for (size_t i = 0; i < count / 4; ++i, ++free_iter) {\n        free((void*)ptrs[free_iter]);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 1);\n        BOOST_REQUIRE_EQUAL(stats[0].size, stats[0].count * sample_rate);\n        BOOST_REQUIRE_NE(stats[0].size, last_alloc_size);\n        BOOST_REQUIRE_GT(stats[0].size, 0);\n        last_alloc_size = stats[0].size;\n    }\n\n    // now increase the sampling rate with outstanding allocations from the old rate\n    seastar::memory::set_heap_profiling_sampling_rate(sample_rate * 100);\n\n    for (size_t i = 0; i < count / 4; ++i, ++free_iter) {\n        free((void*)ptrs[free_iter]);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 1);\n        BOOST_REQUIRE_LT(stats[0].size, last_alloc_size); // should not have underflowed\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(sample_rate);\n\n    // free the rest\n    for (size_t i = 0; i < count / 2; ++i, ++free_iter) {\n        free((void*)ptrs[free_iter]);\n    }\n\n    seastar::memory::set_heap_profiling_sampling_rate(0);\n\n    {\n        auto stats = seastar::memory::sampled_memory_profile();\n        BOOST_REQUIRE_EQUAL(stats.size(), 0);\n    }\n\n    return seastar::make_ready_future();\n}\n\n\n#endif // SEASTAR_HEAPPROF\n\n#endif // #ifndef SEASTAR_DEFAULT_ALLOCATOR\n\nSEASTAR_TEST_CASE(test_large_allocation_warning_off_by_one) {\n#ifndef SEASTAR_DEFAULT_ALLOCATOR\n    constexpr size_t large_alloc_threshold = 1024*1024;\n    seastar::memory::scoped_large_allocation_warning_threshold mtg(large_alloc_threshold);\n    BOOST_REQUIRE(seastar::memory::get_large_allocation_warning_threshold() == large_alloc_threshold);\n    auto old_large_allocs_count = memory::stats().large_allocations();\n    volatile auto obj = (char*)malloc(large_alloc_threshold);\n    *obj = 'c'; // to prevent compiler from considering this a dead allocation and optimizing it out\n\n    // Verify large allocation was detected by allocator.\n    BOOST_REQUIRE(memory::stats().large_allocations() == old_large_allocs_count+1);\n\n    free(obj);\n#endif\n    return make_ready_future<>();\n}\n\n#if !(__GLIBC__ == 2 && __GLIBC_MINOR__ >= 43) && !(__GLIBC__ > 2)\n\nextern \"C\" {\n\nvoid free_sized(void* ptr, size_t size);\nvoid free_aligned_sized(void* ptr, size_t alignment, size_t size);\n\n}\n\n#endif\n\nSEASTAR_TEST_CASE(c23_free_sized) {\n    auto p1 = malloc(100);\n    free_sized(p1, 100);\n    void* p2;\n    int r = posix_memalign(&p2, 1024, 4096);\n    BOOST_REQUIRE_EQUAL(r, 0);\n    free_aligned_sized(p2, 1024, 4096);\n    return make_ready_future<>();\n}\n"
  },
  {
    "path": "tests/unit/allocator_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#include <seastar/core/memory.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/util/assert.hh>\n#include <cmath>\n#include <iostream>\n#include <iomanip>\n#include <algorithm>\n#include <memory>\n#include <chrono>\n#include <boost/program_options.hpp>\n\nusing namespace seastar;\n\nstruct allocation {\n    size_t n;\n    std::unique_ptr<char[]> data;\n    char poison;\n    allocation(size_t n, char poison) : n(n), data(new char[n]), poison(poison) {\n        std::fill_n(data.get(), n, poison);\n    }\n    ~allocation() {\n        verify();\n    }\n    allocation(allocation&& x) noexcept = default;\n    void verify() {\n        if (data) {\n            SEASTAR_ASSERT(std::find_if(data.get(), data.get() + n, [this] (char c) {\n                return c != poison;\n            }) == data.get() + n);\n        }\n    }\n    allocation& operator=(allocation&& x) {\n        verify();\n        if (this != &x) {\n            data = std::move(x.data);\n            n = x.n;\n            poison = x.poison;\n        }\n        return *this;\n    }\n};\n\ntemplate <size_t N>\nstruct alignas(N) cpp17_allocation final {\n    char v;\n};\n\nstruct test17 {\n    struct handle {\n        const test17* d;\n        void* p;\n        handle(const test17* d, void* p) : d(d), p(p) {}\n        handle(const handle&) = delete;\n        handle(handle&& x) noexcept : d(std::exchange(x.d, nullptr)), p(std::exchange(x.p, nullptr)) {}\n        handle& operator=(const handle&) = delete;\n        handle& operator=(handle&& x) noexcept {\n            std::swap(d, x.d);\n            std::swap(p, x.p);\n            return *this;\n        }\n        ~handle() {\n            if (d) {\n                d->free(p);\n            }\n        }\n    };\n    virtual ~test17() {}\n    virtual handle alloc() const = 0;\n    virtual void free(void* ptr) const = 0;\n};\n\ntemplate <size_t N>\nstruct test17_concrete : test17 {\n    using value_type = cpp17_allocation<N>;\n    static_assert(sizeof(value_type) == N, \"language does not guarantee size >= align\");\n    virtual handle alloc() const override {\n        auto ptr = new value_type();\n        SEASTAR_ASSERT((reinterpret_cast<uintptr_t>(ptr) & (N - 1)) == 0);\n        return handle{this, ptr};\n    }\n    virtual void free(void* ptr) const override {\n        delete static_cast<value_type*>(ptr);\n    }\n};\n\nvoid test_cpp17_aligned_allocator() {\n    std::vector<std::unique_ptr<test17>> tv;\n    tv.push_back(std::make_unique<test17_concrete<1>>());\n    tv.push_back(std::make_unique<test17_concrete<2>>());\n    tv.push_back(std::make_unique<test17_concrete<4>>());\n    tv.push_back(std::make_unique<test17_concrete<8>>());\n    tv.push_back(std::make_unique<test17_concrete<16>>());\n    tv.push_back(std::make_unique<test17_concrete<64>>());\n    tv.push_back(std::make_unique<test17_concrete<128>>());\n    tv.push_back(std::make_unique<test17_concrete<2048>>());\n    tv.push_back(std::make_unique<test17_concrete<4096>>());\n    tv.push_back(std::make_unique<test17_concrete<4096*16>>());\n    tv.push_back(std::make_unique<test17_concrete<4096*256>>());\n\n    std::default_random_engine random_engine(testing::local_random_engine());\n    std::uniform_int_distribution<> type_dist(0, 1);\n    std::uniform_int_distribution<size_t> size_dist(0, tv.size() - 1);\n    std::uniform_real_distribution<> which_dist(0, 1);\n\n    std::vector<test17::handle> allocs;\n    for (unsigned i = 0; i < 10000; ++i) {\n        auto type = type_dist(random_engine);\n        switch (type) {\n        case 0: {\n            size_t sz_idx = size_dist(random_engine);\n            allocs.push_back(tv[sz_idx]->alloc());\n            break;\n        }\n        case 1:\n            if (!allocs.empty()) {\n                size_t idx = which_dist(random_engine) * allocs.size();\n                std::swap(allocs[idx], allocs.back());\n                allocs.pop_back();\n            }\n            break;\n        }\n    }\n}\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n    bpo::options_description opts(\"Allowed options\");\n    opts.add_options()\n            (\"help\", \"produce this help message\")\n            (\"iterations\", bpo::value<unsigned>(), \"run s specified number of iterations\")\n            (\"time\", bpo::value<float>()->default_value(5.0), \"run for a specified amount of time, in seconds\")\n            (\"random-seed\", boost::program_options::value<unsigned>(), \"Random number generator seed\");\n            ;\n    bpo::variables_map vm;\n    bpo::store(bpo::parse_command_line(ac, av, opts), vm);\n    bpo::notify(vm);\n    test_cpp17_aligned_allocator();\n    auto seed = vm.count(\"random-seed\") ? vm[\"random-seed\"].as<unsigned>() : std::random_device{}();\n    std::default_random_engine random_engine(seed);\n    std::exponential_distribution<> distr(0.2);\n    std::uniform_int_distribution<> type(0, 1);\n    std::uniform_int_distribution<int> poison(-128, 127);\n    std::uniform_real_distribution<> which(0, 1);\n    std::vector<allocation> allocations;\n    auto iteration = [&] {\n        auto typ = type(random_engine);\n        switch (typ) {\n        case 0: {\n            size_t n = std::min<double>(std::exp(distr(random_engine)), 1 << 25);\n            try {\n                allocations.emplace_back(n, poison(random_engine));\n            } catch (std::bad_alloc&) {\n\n            }\n            break;\n        }\n        case 1: {\n            if (allocations.empty()) {\n                break;\n            }\n            size_t i = which(random_engine) * allocations.size();\n            allocations[i] = std::move(allocations.back());\n            allocations.pop_back();\n            break;\n        }\n        }\n    };\n    if (vm.count(\"help\")) {\n        std::cout << opts << \"\\n\";\n        return 1;\n    }\n    std::cout << \"random-seed=\" << seed << \"\\n\";\n    if (vm.count(\"iterations\")) {\n        auto iterations = vm[\"iterations\"].as<unsigned>();\n        for (unsigned i = 0; i < iterations; ++i) {\n            iteration();\n        }\n    } else {\n        auto time = vm[\"time\"].as<float>();\n        using clock = steady_clock_type;\n        auto end = clock::now() + std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::seconds(1) * time);\n        while (clock::now() < end) {\n            for (unsigned i = 0; i < 1000; ++i) {\n                iteration();\n            }\n        }\n    }\n    return 0;\n}\n"
  },
  {
    "path": "tests/unit/api.json",
    "content": "{\n  \"apiVersion\": \"0.0.1\",\n  \"swaggerVersion\": \"1.2\",\n  \"basePath\": \"{{Protocol}}://{{Host}}\",\n  \"resourcePath\": \"/hello\",\n  \"produces\": [\n    \"application/json\"\n  ],\n  \"apis\": [\n    {\n      \"path\": \"/hello/world/{var1}/{var2}\",\n      \"operations\": [\n        {\n          \"method\": \"GET\",\n          \"summary\": \"Returns the number of seconds since the system was booted\",\n          \"type\": \"long\",\n          \"nickname\": \"hello_world\",\n          \"produces\": [\n            \"application/json\"\n          ],\n          \"parameters\": [\n            {\n              \"name\": \"var2\",\n              \"description\": \"Full path of file or directory\",\n              \"required\": true,\n              \"allowMultiple\": true,\n              \"type\": \"string\",\n              \"paramType\": \"path\"\n            },\n            {\n              \"name\": \"var1\",\n              \"description\": \"Full path of file or directory\",\n              \"required\": true,\n              \"allowMultiple\": false,\n              \"type\": \"string\",\n              \"paramType\": \"path\"\n            },\n            {\n              \"name\": \"query_enum\",\n              \"description\": \"The operation to perform\",\n              \"required\": true,\n              \"allowMultiple\": false,\n              \"type\": \"string\",\n              \"paramType\": \"query\",\n              \"enum\": [\n                \"VAL1\",\n                \"VAL2\",\n                \"VAL3\"\n              ]\n            },\n            {\n              \"name\": \"use_streaming\",\n              \"description\": \"Whether to return the response as a stream_object\",\n              \"required\": true,\n              \"allowMultiple\": false,\n              \"type\": \"boolean\",\n              \"paramType\": \"query\"\n            }\n          ]\n        }\n      ]\n    }\n  ],\n  \"models\": {\n    \"my_object\": {\n      \"id\": \"my_object\",\n      \"description\": \"Demonstrate an object\",\n      \"properties\": {\n        \"var1\": {\n          \"type\": \"string\",\n          \"description\": \"The first parameter in the path\"\n        },\n        \"var2\": {\n          \"type\": \"string\",\n          \"description\": \"The second parameter in the path\"\n        },\n        \"enum_var\": {\n          \"type\": \"string\",\n          \"description\": \"Demonstrate an enum returned, note this is not the same enum type of the request\",\n          \"enum\": [\n            \"VAL1\",\n            \"VAL2\",\n            \"VAL3\"\n          ]\n        },\n        \"array_var\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"int\"\n          }\n        },\n        \"chunked_array_var\": {\n          \"type\": \"chunked_array\",\n          \"items\": {\n            \"type\": \"int\"\n          }\n        },\n        \"integer_var\": {\n          \"type\": \"integer\"\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "tests/unit/app-template_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2023 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE app_template\n\n#include <string>\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/sleep.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\n// #2148 - always run this.\nBOOST_AUTO_TEST_CASE(app_standard_memory_allocator) {\n    // by default, use conservative settings instead of maxing out the performance\n    // for testing app_template and underlying reactor's handling of different\n    // settings\n    app_template::seastar_options opts;\n    opts.smp_opts.thread_affinity.set_value(false);\n    opts.smp_opts.mbind.set_value(false);\n    opts.smp_opts.smp.set_value(1);\n    opts.smp_opts.lock_memory.set_value(false);\n    opts.smp_opts.memory_allocator = memory_allocator::standard;\n    opts.log_opts.default_log_level.set_value(log_level::error);\n    app_template app{std::move(opts)};\n    // app.run() takes `char**` not `char* const *`, so appease it\n    std::string prog_name{\"prog\"};\n    char* args[] = {prog_name.data()};\n    int expected_status = 42;\n    int actual_status = app.run(\n        std::size(args), std::data(args),\n        [expected_status] {\n            // #2148 - add a small sleep to ensure the reactor does\n            // some of its background stuff, pollers etc for example.\n            // We only need to ensure we get put on the waiting task queue\n            // to provoke the problem, thus a short (probably even to long here)\n            // sleep will do.\n            return seastar::sleep(2s).then([expected_status] {\n                return make_ready_future<int>(expected_status);\n            });\n        });\n    BOOST_CHECK_EQUAL(actual_status, expected_status);\n}\n\nBOOST_AUTO_TEST_CASE(return_0_for_func_returning_void) {\n    app_template app;\n    std::string prog_name{\"prog\"};\n    char* args[] = {prog_name.data()};\n    int status = app.run(std::size(args), std::data(args),\n                         [] { return make_ready_future(); });\n    BOOST_CHECK_EQUAL(status, 0);\n}\n\nBOOST_AUTO_TEST_CASE(return_status_for_func_returning_int) {\n    app_template app;\n    std::string prog_name{\"prog\"};\n    char* args[] = {prog_name.data()};\n    int expected_status = 42;\n    int actual_status = app.run(\n        std::size(args), std::data(args),\n         [expected_status] {\n             return make_ready_future<int>(expected_status);\n         });\n    BOOST_CHECK_EQUAL(actual_status, expected_status);\n}\n"
  },
  {
    "path": "tests/unit/cert.cfg.in",
    "content": "[ req ]\ndefault_bits = @CERT_WIDTH@\ndefault_keyfile = @CERT_PRIVKEY@\ndefault_md = sha256\ndistinguished_name = req_distinguished_name\nreq_extensions = v3_req\nprompt = no\n[ req_distinguished_name ]\nC = @CERT_COUNTRY@\nST = @CERT_STATE@\nL = @CERT_LOCALITY@\nO = @CERT_ORG@\nOU = @CERT_UNIT@\nCN= @CERT_COMMON@\nemailAddress = @CERT_EMAIL@\n[v3_ca]\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid:always,issuer:always\nbasicConstraints = CA:true\n[v3_req]\n# Extensions to add to a certificate request\nbasicConstraints = CA:FALSE\nkeyUsage = nonRepudiation, digitalSignature, keyEncipherment\n\n[req_ext]\nsubjectAltName=email:@CERT_ALT_EMAIL_1@,email:@CERT_ALT_EMAIL_2@,IP:@CERT_ALT_IP_1@,DNS:@CERT_ALT_DNS@\n"
  },
  {
    "path": "tests/unit/checked_ptr_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/checked_ptr.hh>\n#include <seastar/core/weak_ptr.hh>\n\nusing namespace seastar;\n\nstatic_assert(std::is_nothrow_default_constructible_v<checked_ptr<int*>>);\nstatic_assert(std::is_nothrow_move_constructible_v<checked_ptr<int*>>);\nstatic_assert(std::is_nothrow_copy_constructible_v<checked_ptr<int*>>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<checked_ptr<weak_ptr<int>>>);\nstatic_assert(std::is_nothrow_move_constructible_v<checked_ptr<weak_ptr<int>>>);\n\ntemplate <typename T>\nclass may_throw_on_null_ptr : public seastar::weak_ptr<T> {\npublic:\n    may_throw_on_null_ptr(std::nullptr_t) {}\n};\n\nstatic_assert(!std::is_nothrow_default_constructible_v<may_throw_on_null_ptr<int>>);\nstatic_assert(!std::is_nothrow_default_constructible_v<checked_ptr<may_throw_on_null_ptr<int>>>);\nstatic_assert(std::is_nothrow_move_constructible_v<checked_ptr<may_throw_on_null_ptr<int>>>);\nstatic_assert(std::is_nothrow_copy_constructible_v<checked_ptr<may_throw_on_null_ptr<int>>>);\n\nstruct my_st : public weakly_referencable<my_st> {\n        my_st(int a_) : a(a_) {}\n        int a;\n};\n\nvoid const_ref_check_naked(const seastar::checked_ptr<my_st*>& cp) {\n    BOOST_REQUIRE(bool(cp));\n    BOOST_REQUIRE((*cp).a == 3);\n    BOOST_REQUIRE(cp->a == 3);\n    BOOST_REQUIRE(cp.get()->a == 3);\n}\n\nvoid const_ref_check_smart(const seastar::checked_ptr<::weak_ptr<my_st>>& cp) {\n    BOOST_REQUIRE(bool(cp));\n    BOOST_REQUIRE((*cp).a == 3);\n    BOOST_REQUIRE(cp->a == 3);\n    BOOST_REQUIRE(cp.get()->a == 3);\n}\n\nBOOST_AUTO_TEST_CASE(test_checked_ptr_is_empty_when_default_initialized) {\n    seastar::checked_ptr<int*> cp;\n    BOOST_REQUIRE(!bool(cp));\n}\n\nBOOST_AUTO_TEST_CASE(test_checked_ptr_is_empty_when_nullptr_initialized_nakes_ptr) {\n    seastar::checked_ptr<int*> cp = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n}\n\nBOOST_AUTO_TEST_CASE(test_checked_ptr_is_empty_when_nullptr_initialized_smart_ptr) {\n    seastar::checked_ptr<::weak_ptr<my_st>> cp = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n}\n\nBOOST_AUTO_TEST_CASE(test_checked_ptr_is_initialized_after_assignment_naked_ptr) {\n    seastar::checked_ptr<my_st*> cp = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n    my_st i(3);\n    my_st k(3);\n    cp = &i;\n    seastar::checked_ptr<my_st*> cp1(&i);\n    seastar::checked_ptr<my_st*> cp2(&k);\n    BOOST_REQUIRE(bool(cp));\n    BOOST_REQUIRE(cp == cp1);\n    BOOST_REQUIRE(cp != cp2);\n    BOOST_REQUIRE((*cp).a == 3);\n    BOOST_REQUIRE(cp->a == 3);\n    BOOST_REQUIRE(cp.get()->a == 3);\n\n    const_ref_check_naked(cp);\n\n    cp = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n}\n\nBOOST_AUTO_TEST_CASE(test_checked_ptr_is_initialized_after_assignment_smart_ptr) {\n    seastar::checked_ptr<::weak_ptr<my_st>> cp = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n    std::unique_ptr<my_st> i = std::make_unique<my_st>(3);\n    cp = i->weak_from_this();\n    seastar::checked_ptr<::weak_ptr<my_st>> cp1(i->weak_from_this());\n    seastar::checked_ptr<::weak_ptr<my_st>> cp2;\n    BOOST_REQUIRE(bool(cp));\n    BOOST_REQUIRE(cp == cp1);\n    BOOST_REQUIRE(cp != cp2);\n    BOOST_REQUIRE((*cp).a == 3);\n    BOOST_REQUIRE(cp->a == 3);\n    BOOST_REQUIRE(cp.get()->a == 3);\n\n    const_ref_check_smart(cp);\n\n    i = nullptr;\n    BOOST_REQUIRE(!bool(cp));\n    BOOST_REQUIRE(!bool(cp1));\n    BOOST_REQUIRE(!bool(cp2));\n}\n\n"
  },
  {
    "path": "tests/unit/chunk_parsers_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <seastar/core/ragel.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/http/chunk_parsers.hh>\n#include <seastar/http/internal/content_source.hh>\n#include <seastar/testing/test_case.hh>\n#include <tuple>\n#include <utility>\n#include <vector>\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(test_size_and_extensions_parsing) {\n    struct test_set {\n        sstring msg;\n        bool parsable;\n        sstring size = \"\";\n        std::vector<std::pair<sstring, sstring>> extensions;\n\n        temporary_buffer<char> buf() {\n            return temporary_buffer<char>(msg.c_str(), msg.size());\n        }\n    };\n\n    std::vector<test_set> tests = {\n        { \"14;name=value\\r\\n\", true, \"14\", { {\"name\", \"value\"} } },\n        { \"abcdef;name=value;name2=\\\"value2\\\"\\r\\n\", true, \"abcdef\" },\n        { \"1efg;name=value\\r\\n\", false },\n        { \"aa;tchars.^_`|123=t1!#$%&'*+-.~\\r\\n\", true, \"aa\", { {\"tchars.^_`|123\", \"t1!#$%&'*+-.~\"} } },\n        { \"1;quoted=\\\"hello world\\\";quoted-pair=\\\"\\\\a\\\\b\\\\cd\\\\\\\\ef\\\"\\r\\n\", true, \"1\", { {\"quoted\", \"hello world\"}, {\"quoted-pair\", \"abcd\\\\ef\"} } },\n        { \"2;bad-quoted-pair=\\\"abc\\\\\\\"\\r\\n\", false },\n        { \"3;quoted-pair-outside-quoted-string=\\\\q\\\\p\\r\\n\", false },\n        { \"4;whitespace-outside-quoted-string=quoted string\\r\\n\", false },\n        { \"5;quotation-mark-inside-quoted-string=\\\"quoted\\\"mark\\\"\\r\\n\", false },\n        { \"6; bad=space\\r\\n\", false },\n        { \"7;sole-name\\r\\n\", true, \"7\", { { \"sole-name\", \"\"} } },\n        { \"8;empty_value=\\\"\\\"\\r\\n\", true, \"8\", { { \"empty_value\", \"\"} } },\n        { \"0\\r\\n\", true, \"0\" }\n    };\n\n    http_chunk_size_and_ext_parser parser;\n    for (auto& tset : tests) {\n        parser.init();\n        BOOST_REQUIRE(parser(tset.buf()).get().has_value());\n        BOOST_REQUIRE_NE(parser.failed(), tset.parsable);\n        if (tset.parsable) {\n            BOOST_REQUIRE_EQUAL(parser.get_size(), std::move(tset.size));\n            auto exts = parser.get_parsed_extensions();\n            for (auto& ext : tset.extensions) {\n                BOOST_REQUIRE_EQUAL(exts[ext.first], ext.second);\n            }\n        }\n    }\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_trailer_headers_parsing) {\n    struct test_set {\n        sstring msg;\n        bool parsable;\n        sstring header_name = \"\";\n        sstring header_value = \"\";\n\n        temporary_buffer<char> buf() {\n            return temporary_buffer<char>(msg.c_str(), msg.size());\n        }\n    };\n\n    std::vector<test_set> tests = {\n        // the headers follow the same rules as in the request parser\n        { \"Host: test\\r\\n\\r\\n\", true, \"Host\", \"test\" },\n        { \"Header: Field\\r\\n\\r\\n\", true, \"Header\", \"Field\" },\n        { \"Header: \\r\\n\\r\\n\", true, \"Header\", \"\" },\n        { \"Header:  f  i e l d  \\r\\n\\r\\n\", true, \"Header\", \"f  i e l d\" },\n        { \"Header: fiel\\r\\n    d\\r\\n\\r\\n\", true, \"Header\", \"fiel d\" },\n        { \"tchars.^_`|123: printable!@#%^&*()obs_text\\x80\\x81\\xff\\r\\n\\r\\n\", true,\n            \"tchars.^_`|123\", \"printable!@#%^&*()obs_text\\x80\\x81\\xff\" },\n        { \"Header: Field\\r\\nHeader: Field2\\r\\n\\r\\n\", true, \"Header\", \"Field,Field2\" },\n        { \"Header : Field\\r\\n\\r\\n\", false },\n        { \"Header Field\\r\\n\\r\\n\", false },\n        { \"Header@: Field\\r\\n\\r\\n\", false },\n        { \"Header: fiel\\r\\nd \\r\\n\\r\\n\", false },\n        { \"\\r\\n\", true }\n    };\n\n    http_chunk_trailer_parser parser;\n    for (auto& tset : tests) {\n        parser.init();\n        BOOST_REQUIRE(parser(tset.buf()).get().has_value());\n        BOOST_REQUIRE_NE(parser.failed(), tset.parsable);\n        if (tset.parsable) {\n            auto heads = parser.get_parsed_headers();\n            BOOST_REQUIRE_EQUAL(heads[std::move(tset.header_name)], std::move(tset.header_value));\n        }\n    }\n    return make_ready_future<>();\n}\n"
  },
  {
    "path": "tests/unit/chunked_fifo_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB Ltd.\n */\n\n\n#include <boost/test/tools/old/interface.hpp>\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/tools/context.hpp>\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/chunked_fifo.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <stdlib.h>\n#include <chrono>\n#include <deque>\n#include <iterator>\n#include <ranges>\n#if __has_include(<version>)\n#include <version>\n#endif\n\nusing namespace seastar;\n\nstatic_assert(std::weakly_incrementable<chunked_fifo<int>::iterator>);\nstatic_assert(std::weakly_incrementable<chunked_fifo<int>::const_iterator>);\nstatic_assert(std::sentinel_for<chunked_fifo<int>::iterator, chunked_fifo<int>::iterator>);\nstatic_assert(std::sentinel_for<chunked_fifo<int>::const_iterator, chunked_fifo<int>::const_iterator>);\n\nstatic_assert(std::ranges::range<chunked_fifo<const int>>);\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_small) {\n    // Check all the methods of chunked_fifo but with a trivial type (int) and\n    // only a few elements - and in particular a single chunk is enough.\n    chunked_fifo<int> fifo;\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n    fifo.push_back(3);\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n    fifo.push_back(17);\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 17);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n    // The previously allocated chunk should have been freed, and now\n    // a new one will need to be allocated:\n    fifo.push_back(57);\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    BOOST_REQUIRE_EQUAL(fifo.front(), 57);\n    // check miscelleneous methods (at least they shouldn't crash)\n    fifo.clear();\n    fifo.shrink_to_fit();\n    fifo.reserve(1);\n    fifo.reserve(100);\n    fifo.reserve(1280);\n    fifo.shrink_to_fit();\n    fifo.reserve(1280);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_fullchunk) {\n    // Grow a chunked_fifo to exactly fill a chunk, and see what happens when\n    // we cross that chunk.\n    constexpr size_t N = 128;\n    chunked_fifo<int, N> fifo;\n    for (int i = 0; i < static_cast<int>(N); i++) {\n        fifo.push_back(i);\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), N);\n    fifo.push_back(N);\n    BOOST_REQUIRE_EQUAL(fifo.size(), N+1);\n    for (int i = 0 ; i < static_cast<int>(N+1); i++) {\n        BOOST_REQUIRE_EQUAL(fifo.front(), i);\n        BOOST_REQUIRE_EQUAL(fifo.size(), N+1-i);\n        fifo.pop_front();\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n}\n\nstruct trackable_totals {\n    size_t cons_called = 0, dtor_called = 0;\n    size_t mcons_called = 0, ccons_called = 0;\n};\n\nstruct trackable {\n    trackable(trackable_totals& tracker) : _tracker{tracker} {\n        _tracker.cons_called++;\n    }\n\n    trackable(const trackable& rhs) : _tracker{rhs._tracker} {\n        _tracker.ccons_called++;\n    }\n\n    trackable(trackable&& rhs) : _tracker{rhs._tracker} {\n        _tracker.mcons_called++;\n    }\n\n    void operator=(const trackable&) = delete;\n    void operator=(trackable&&) = delete;\n\n    ~trackable() {\n        _tracker.dtor_called++;\n    }\n\n    trackable_totals& _tracker;\n};\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_pop_n) {\n    trackable_totals ctor_calls;\n    constexpr size_t N = 4;\n    chunked_fifo<trackable, N / 2> fifo;\n\n    auto fill_and_reset = [&](size_t size) {\n        fifo.clear();\n        ctor_calls = {};\n        for (size_t i = 0; i < size; i++) {\n            // we add 2, remove 1 in order to stress the case where chunk::begin and end\n            // are outside the range [0, items_per_chunk], i.e., where proper use of mask()\n            // is required\n            fifo.emplace_back(ctor_calls);\n            fifo.emplace_back(ctor_calls);\n            fifo.pop_front_n(1);\n        }\n    };\n\n    for (size_t size : std::views::iota((size_t)0, 2 * N) ) {\n        for (size_t pop_count : std::views::iota((size_t)0, size + 1) ) {\n            BOOST_TEST_CONTEXT(\"size: \" << size << \", pop_count: \" << pop_count) {\n                fill_and_reset(size);\n\n                // note that we add 2 and delete 1 element for every element added in\n                // fill_and_reset so that affects the numbers below\n\n                BOOST_REQUIRE_EQUAL(fifo.size(), size);\n                BOOST_REQUIRE_EQUAL(ctor_calls.cons_called, size * 2);\n                BOOST_REQUIRE_EQUAL(ctor_calls.dtor_called, size);\n\n                fifo.pop_front_n(pop_count);\n\n                BOOST_REQUIRE_EQUAL(fifo.size(), size - pop_count);\n                BOOST_REQUIRE_EQUAL(ctor_calls.cons_called, size * 2);\n                BOOST_REQUIRE_EQUAL(ctor_calls.dtor_called, size + pop_count);\n\n                fifo.emplace_back(ctor_calls);\n\n                BOOST_REQUIRE_EQUAL(fifo.size(), size - pop_count + 1);\n                BOOST_REQUIRE_EQUAL(ctor_calls.cons_called, size * 2 + 1);\n                BOOST_REQUIRE_EQUAL(ctor_calls.dtor_called, size + pop_count);\n            }\n        }\n    }\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_big) {\n    // Grow a chunked_fifo to many elements, and see things are working as\n    // expected\n    chunked_fifo<int> fifo;\n    constexpr size_t N = 100'000;\n    for (int i=0; i < static_cast<int>(N); i++) {\n        fifo.push_back(i);\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), N);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    for (int i = 0 ; i < static_cast<int>(N); i++) {\n        BOOST_REQUIRE_EQUAL(fifo.front(), i);\n        BOOST_REQUIRE_EQUAL(fifo.size(), N-i);\n        fifo.pop_front();\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_constructor) {\n    // Check that chunked_fifo appropriately calls the type's constructor\n    // and destructor, and doesn't need anything else.\n    struct typ {\n        int val;\n        unsigned* constructed;\n        unsigned* destructed;\n        typ(int val, unsigned* constructed, unsigned* destructed)\n            : val(val), constructed(constructed), destructed(destructed) {\n                ++*constructed;\n        }\n        ~typ() { ++*destructed; }\n    };\n    chunked_fifo<typ> fifo;\n    unsigned constructed = 0, destructed = 0;\n    constexpr unsigned N = 1000;\n    for (unsigned i = 0; i < N; i++) {\n        fifo.emplace_back(i, &constructed, &destructed);\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), N);\n    BOOST_REQUIRE_EQUAL(constructed, N);\n    BOOST_REQUIRE_EQUAL(destructed, 0u);\n    for (unsigned i = 0 ; i < N; i++) {\n        BOOST_REQUIRE_EQUAL(fifo.front().val, static_cast<int>(i));\n        BOOST_REQUIRE_EQUAL(fifo.size(), N-i);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(destructed, i+1);\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n    // Check that destructing a fifo also destructs the objects it still\n    // contains\n    constructed = destructed = 0;\n    {\n        chunked_fifo<typ> fifo;\n        for (unsigned i = 0; i < N; i++) {\n            fifo.emplace_back(i, &constructed, &destructed);\n            BOOST_REQUIRE_EQUAL(fifo.front().val, 0);\n            BOOST_REQUIRE_EQUAL(fifo.size(), i+1);\n            BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n            BOOST_REQUIRE_EQUAL(constructed, i+1);\n            BOOST_REQUIRE_EQUAL(destructed, 0u);\n        }\n    }\n    BOOST_REQUIRE_EQUAL(constructed, N);\n    BOOST_REQUIRE_EQUAL(destructed, N);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_copy_move_test) {\n    constexpr size_t N = 4;\n    using ftype = chunked_fifo<trackable, N / 2>;\n\n    trackable_totals calls;\n    ftype fifo1, fifo2;\n\n    auto fill = [&](size_t size, ftype& fifo) {\n        fifo.clear();\n        calls = {};\n        for (size_t i = 0; i < size; i++) {\n            fifo.emplace_back(calls);\n        }\n    };\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n\n    fill(0, fifo1);\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n\n    fill(5, fifo1);\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n\n    {\n        auto fifox{fifo1}; // copy ctor\n\n        BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n        BOOST_REQUIRE_EQUAL(calls.ccons_called, 5);\n        BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n        BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n    }\n\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 5);\n    fifo1.clear();\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 10);\n\n    fill(5, fifo1);\n\n    {\n        // move ctor, no element ctors are called at all\n        auto fifox{std::move(fifo1)};\n\n        BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n        BOOST_REQUIRE_EQUAL(calls.ccons_called, 0);\n        BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n        BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n    }\n\n    fill(5, fifo1);\n\n    fifo2 = fifo1;\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n\n    fifo1.clear();\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 5);\n\n    fifo2 = fifo1;\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 10);\n\n    fill(5, fifo1);\n\n    fifo2 = std::move(fifo1);\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 5);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 0);\n\n    fill(1, fifo1);\n    fill(2, fifo2);\n    calls = {};\n\n    fifo1 = fifo2;\n\n    BOOST_REQUIRE_EQUAL(calls.cons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.ccons_called, 2);\n    BOOST_REQUIRE_EQUAL(calls.mcons_called, 0);\n    BOOST_REQUIRE_EQUAL(calls.dtor_called, 1);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_copy_ctor) {\n\n    using ftype = chunked_fifo<int, 1>;\n\n    ftype f1;\n    f1.push_back(1);\n\n    ftype f2;\n    f2.push_back(1);\n    f2.push_back(2);\n\n    {\n        ftype f0;\n        BOOST_REQUIRE(f0 == ftype{});\n    }\n\n    {\n        ftype f0{f1};\n        BOOST_CHECK(f0 == f1);\n    }\n\n    {\n        ftype f0{f2};\n        BOOST_CHECK(f0 == f2);\n    }\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_construct_fail) {\n    // Check that if we fail to construct the item pushed, the queue remains\n    // empty.\n    class my_exception {};\n    struct typ {\n        typ() {\n            throw my_exception();\n        }\n    };\n    chunked_fifo<typ> fifo;\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n    try {\n        fifo.emplace_back();\n    } catch(my_exception) {\n        // expected, ignore\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_construct_fail2) {\n    // A slightly more elaborate test, with a chunk size of 2\n    // items, and the third addition failing, so the question is\n    // not whether empty() is wrong immediately, but whether after\n    // we pop the two items, it will become true or we'll be left\n    // with an empty chunk.\n    class my_exception {};\n    struct typ {\n        typ(bool fail) {\n            if (fail) {\n                throw my_exception();\n            }\n        }\n    };\n    chunked_fifo<typ, 2> fifo;\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n    fifo.emplace_back(false);\n    fifo.emplace_back(false);\n    try {\n        fifo.emplace_back(true);\n    } catch(my_exception) {\n        // expected, ignore\n    }\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), false);\n    fifo.pop_front();\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE_EQUAL(fifo.empty(), true);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_equals_op) {\n    chunked_fifo<int> empty, vec123;\n    vec123.push_back(1);\n    vec123.push_back(2);\n    vec123.push_back(3);\n    auto empty2 = vec123;\n    empty2.clear();\n\n    BOOST_CHECK(empty == empty);\n    BOOST_CHECK(empty == empty2);\n    BOOST_CHECK(empty != vec123);\n    BOOST_CHECK(vec123 == vec123);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_copy) {\n    chunked_fifo<int> empty, vec123;\n    vec123.push_back(1);\n    vec123.push_back(2);\n    vec123.push_back(3);\n\n    auto empty_copy = empty;\n    BOOST_CHECK(empty_copy.empty());\n\n    auto vec123_copy = vec123;\n    BOOST_CHECK_EQUAL(vec123.size(), 3);\n    BOOST_CHECK(vec123_copy == vec123);\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_assignment) {\n    chunked_fifo<int> empty, vec123, vec;\n    vec123.push_back(1);\n    vec123.push_back(2);\n    vec123.push_back(3);\n\n    BOOST_CHECK(vec.empty());\n    vec = empty;\n    BOOST_CHECK(vec.empty());\n    vec = vec123;\n    BOOST_CHECK_EQUAL(vec.size(), 3);\n    BOOST_CHECK(vec == vec123);\n    vec = {};\n    BOOST_CHECK(vec.empty());\n\n    // move assignment\n    vec = std::move(vec123);\n    BOOST_CHECK_EQUAL(vec.size(), 3);\n    BOOST_CHECK(vec123.empty());\n}\n\n// Enable the following to run some benchmarks on different queue options\n#if 0\n// Unfortunately, C++ lacks the trivial feature of converting a type's name,\n// in compile time, to a string (akin to the C preprocessor's \"#\" feature).\n// Here is a neat trick to replace it - use typeinfo<T>::name() or\n// type_name<T>() to get a constant string name of the type.\n#include <cxxabi.h>\ntemplate <typename T>\nclass typeinfo {\nprivate:\n    static const char *_name;\npublic:\n    static const char *name() {\n        int status;\n        if (!_name)\n            _name = abi::__cxa_demangle(typeid(T).name(), 0, 0, &status);\n        return _name;\n    }\n};\ntemplate<typename T> const char *typeinfo<T>::_name = nullptr;\ntemplate<typename T> const char *type_name() {\n    return typeinfo<T>::name();\n}\n\n\ntemplate<typename FIFO_TYPE> void\nbenchmark_random_push_pop() {\n    // A test involving a random sequence of pushes and pops. Because the\n    // random walk is bounded the 0 end (the queue cannot be popped after\n    // being empty), the queue's expected length at the end of the test is\n    // not zero.\n    // The test uses the same random sequence each time so can be used for\n    // benchmarking different queue implementations on the same sequence.\n    constexpr int N = 1'000'000'000;\n    FIFO_TYPE fifo;\n    unsigned int seed = 0;\n    int entropy = 0;\n    auto start = std::chrono::high_resolution_clock::now();\n    for (int i = 0; i < N; i++) {\n        if (!entropy) {\n            entropy = rand_r(&seed);\n        }\n        if (entropy & 1) {\n            fifo.push_back(i);\n        } else {\n            if (!fifo.empty()) {\n                fifo.pop_front();\n            }\n        }\n        entropy >>= 1;\n    }\n    auto end = std::chrono::high_resolution_clock::now();\n    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();\n    std::cerr << type_name<FIFO_TYPE>() << \", \" << N << \" random push-and-pop \" << fifo.size() << \" \" << ms << \"ms.\\n\";\n}\n\ntemplate<typename FIFO_TYPE> void\nbenchmark_push_pop() {\n    // A benchmark involving repeated push and then pop to a queue, which\n    // will have 0 or 1 items at all times.\n    constexpr int N = 1'000'000'000;\n    FIFO_TYPE fifo;\n    auto start = std::chrono::high_resolution_clock::now();\n    for (int i = 0; i < N; i++) {\n        fifo.push_back(1);\n        fifo.pop_front();\n    }\n    auto end = std::chrono::high_resolution_clock::now();\n    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();\n    std::cerr << type_name<FIFO_TYPE>() << \", \" << N << \" push-and-pop \" << ms << \"ms.\\n\";\n}\n\ntemplate<typename FIFO_TYPE> void\nbenchmark_push_pop_k() {\n    // A benchmark involving repeated pushes of a few items and then popping\n    // to a queue, which will have just one chunk (or 0) at all times.\n    constexpr int N = 1'000'000'000;\n    constexpr int K = 100;\n    FIFO_TYPE fifo;\n    auto start = std::chrono::high_resolution_clock::now();\n    for (int i = 0; i < N/K; i++) {\n        for(int j = 0; j < K; j++) {\n            fifo.push_back(j);\n        }\n        for(int j = 0; j < K; j++) {\n            fifo.pop_front();\n        }\n    }\n    auto end = std::chrono::high_resolution_clock::now();\n    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();\n    std::cerr << type_name<FIFO_TYPE>() << \", \" << N << \" push-and-pop-\" << K << \" \" << ms << \"ms.\\n\";\n}\n\ntemplate<typename FIFO_TYPE> void\nbenchmark_pushes_pops() {\n    // A benchmark of pushing a lot of items, and then popping all of them\n    constexpr int N = 100'000'000;\n    FIFO_TYPE fifo;\n    auto start = std::chrono::high_resolution_clock::now();\n    for (int i = 0; i < N; i++) {\n        fifo.push_back(1);\n    }\n    for (int i = 0; i < N; i++) {\n        fifo.pop_front();\n    }\n    auto end = std::chrono::high_resolution_clock::now();\n    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();\n    std::cerr << type_name<FIFO_TYPE>() << \", \" << N << \" push-all-then-pop-all \" << ms << \"ms.\\n\";\n}\n\ntemplate<typename FIFO_TYPE> void\nbenchmark_all() {\n    std::cerr << \"\\n  --- \" << type_name<FIFO_TYPE>() << \": \\n\";\n    benchmark_random_push_pop<FIFO_TYPE>();\n    benchmark_push_pop<FIFO_TYPE>();\n    benchmark_push_pop_k<FIFO_TYPE>();\n    benchmark_pushes_pops<FIFO_TYPE>();\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_benchmark) {\n    benchmark_all<chunked_fifo<int>>();\n    benchmark_all<circular_buffer<int>>();\n    benchmark_all<std::deque<int>>();\n    benchmark_all<std::list<int>>();\n}\n#endif\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_iterator) {\n    constexpr auto items_per_chunk = 8;\n    auto fifo = chunked_fifo<int, items_per_chunk>{};\n    auto reference = std::deque<int>{};\n\n    BOOST_REQUIRE(fifo.begin() == fifo.end());\n\n    for (int i = 0; i < items_per_chunk * 4; ++i) {\n        fifo.push_back(i);\n        reference.push_back(i);\n        BOOST_REQUIRE(std::equal(fifo.begin(), fifo.end(), reference.begin(), reference.end()));\n    }\n\n    for (int i = 0; i < items_per_chunk * 2; ++i) {\n        fifo.pop_front();\n        reference.pop_front();\n        BOOST_REQUIRE(std::equal(fifo.begin(), fifo.end(), reference.begin(), reference.end()));\n    }\n}\n\nBOOST_AUTO_TEST_CASE(chunked_fifo_const_iterator) {\n    constexpr auto items_per_chunk = 8;\n    auto fifo = chunked_fifo<int, items_per_chunk>{};\n    auto reference = std::deque<int>{};\n\n    BOOST_REQUIRE(fifo.cbegin() == fifo.cend());\n\n    for (int i = 0; i < items_per_chunk * 4; ++i) {\n        fifo.push_back(i);\n        reference.push_back(i);\n        BOOST_REQUIRE(std::equal(fifo.cbegin(), fifo.cend(), reference.cbegin(), reference.cend()));\n    }\n\n    for (int i = 0; i < items_per_chunk * 2; ++i) {\n        fifo.pop_front();\n        reference.pop_front();\n        BOOST_REQUIRE(std::equal(fifo.cbegin(), fifo.cend(), reference.cbegin(), reference.cend()));\n    }\n}\n"
  },
  {
    "path": "tests/unit/circular_buffer_fixed_capacity_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <deque>\n#include <random>\n#include <seastar/core/circular_buffer_fixed_capacity.hh>\n\n#include <boost/range/algorithm/sort.hpp>\n#include <boost/range/algorithm/equal.hpp>\n#include <boost/range/algorithm/reverse.hpp>\n\nusing namespace seastar;\n\nusing cb16_t = circular_buffer_fixed_capacity<int, 16>;\n\nstruct int_with_stats {\n    int val;\n    unsigned *num_deleted;\n    unsigned *num_moved;\n    operator int() const { return val; }\n    int_with_stats(int val, unsigned* num_deleted, unsigned* num_moved)\n        : val(val), num_deleted(num_deleted), num_moved(num_moved) {}\n\n    ~int_with_stats() { ++(*num_deleted); }\n    int_with_stats(const int_with_stats&) = delete;\n    int_with_stats(int_with_stats&& o) noexcept : val(o.val), num_deleted(o.num_deleted), num_moved(o.num_moved) {\n        ++(*num_moved);\n    }\n    int_with_stats& operator=(int_with_stats&& o) noexcept {\n        this->~int_with_stats();\n        new (this) int_with_stats(std::move(o));\n        return *this;\n    }\n};\n\nBOOST_AUTO_TEST_CASE(test_edge_cases) {\n    unsigned num_deleted = 0;\n    unsigned num_moved = 0;\n    auto get_val = [&num_deleted, &num_moved] (int val) {\n        return int_with_stats{val, &num_deleted, &num_moved};\n    };\n\n    {\n        circular_buffer_fixed_capacity<int_with_stats, 16> cb;\n        BOOST_REQUIRE(cb.begin() == cb.end());\n        cb.push_front(get_val(3));  // underflows indexes\n        BOOST_REQUIRE_EQUAL(cb[0], 3);\n        BOOST_REQUIRE(cb.begin() < cb.end());\n        cb.push_back(get_val(4));\n        BOOST_REQUIRE_EQUAL(cb.size(), 2u);\n        BOOST_REQUIRE_EQUAL(cb[0], 3);\n        BOOST_REQUIRE_EQUAL(cb[1], 4);\n        cb.pop_back();\n        BOOST_REQUIRE_EQUAL(cb.back(), 3);\n        cb.push_front(get_val(1));\n        cb.pop_back();\n        BOOST_REQUIRE_EQUAL(cb.back(), 1);\n\n        BOOST_REQUIRE_EQUAL(num_deleted, 5);\n        BOOST_REQUIRE_EQUAL(num_moved, 3);\n\n        cb.push_front(get_val(0));\n        cb.push_back(get_val(2));\n        BOOST_REQUIRE_EQUAL(cb.size(), 3);\n        BOOST_REQUIRE_EQUAL(cb[0], 0);\n        BOOST_REQUIRE_EQUAL(cb[1], 1);\n        BOOST_REQUIRE_EQUAL(cb[2], 2);\n        BOOST_REQUIRE_EQUAL(num_deleted, 7);\n        BOOST_REQUIRE_EQUAL(num_moved, 5);\n\n        circular_buffer_fixed_capacity<int_with_stats, 16> cb2 = std::move(cb);\n        BOOST_REQUIRE_EQUAL(cb2.size(), 3);\n        BOOST_REQUIRE_EQUAL(cb2[0], 0);\n        BOOST_REQUIRE_EQUAL(cb2[1], 1);\n        BOOST_REQUIRE_EQUAL(cb2[2], 2);\n        BOOST_REQUIRE_EQUAL(num_deleted, 7);\n        BOOST_REQUIRE_EQUAL(num_moved, 8);\n    }\n    BOOST_REQUIRE_EQUAL(num_deleted, 13);\n    BOOST_REQUIRE_EQUAL(num_moved, 8);\n}\n\nusing deque = std::deque<int>;\n\nBOOST_AUTO_TEST_CASE(test_random_walk) {\n    auto rand = std::default_random_engine();\n    auto op_gen = std::uniform_int_distribution<unsigned>(0, 7);\n    deque d;\n    cb16_t c;\n    for (auto i = 0; i != 1000000; ++i) {\n        auto op = op_gen(rand);\n        switch (op) {\n        case 0:\n            if (d.size() < 16) {\n                auto n = rand();\n                c.push_back(n);\n                d.push_back(n);\n            }\n            break;\n        case 1:\n            if (d.size() < 16) {\n                auto n = rand();\n                c.push_front(n);\n                d.push_front(n);\n            }\n            break;\n        case 2:\n            if (!d.empty()) {\n                auto n = d.back();\n                auto m = c.back();\n                BOOST_REQUIRE_EQUAL(n, m);\n                c.pop_back();\n                d.pop_back();\n            }\n            break;\n        case 3:\n            if (!d.empty()) {\n                auto n = d.front();\n                auto m = c.front();\n                BOOST_REQUIRE_EQUAL(n, m);\n                c.pop_front();\n                d.pop_front();\n            }\n            break;\n        case 4:\n            boost::sort(c);\n            boost::sort(d);\n            break;\n        case 5:\n            if (!d.empty()) {\n                auto u = std::uniform_int_distribution<size_t>(0, d.size() - 1);\n                auto idx = u(rand);\n                auto m = c[idx];\n                auto n = c[idx];\n                BOOST_REQUIRE_EQUAL(m, n);\n            }\n            break;\n        case 6:\n            c.clear();\n            d.clear();\n            break;\n        case 7:\n            boost::reverse(c);\n            boost::reverse(d);\n            break;\n        default:\n            abort();\n        }\n        BOOST_REQUIRE_EQUAL(c.size(), d.size());\n        BOOST_REQUIRE(boost::equal(c, d));\n    }\n}\n"
  },
  {
    "path": "tests/unit/circular_buffer_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <stdlib.h>\n#include <chrono>\n#include <deque>\n#include <random>\n#include <ranges>\n#if __has_include(<version>)\n#include <version>\n#endif\n\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/util/assert.hh>\n\nusing namespace seastar;\n\nstatic_assert(std::ranges::range<circular_buffer<int>>);\n\nBOOST_AUTO_TEST_CASE(test_erasing) {\n    circular_buffer<int> buf;\n\n    buf.push_back(3);\n    buf.erase(buf.begin(), buf.end());\n\n    BOOST_REQUIRE(buf.size() == 0);\n    BOOST_REQUIRE(buf.empty());\n\n    buf.push_back(1);\n    buf.push_back(2);\n    buf.push_back(3);\n    buf.push_back(4);\n    buf.push_back(5);\n\n    buf.erase(std::remove_if(buf.begin(), buf.end(), [] (int v) { return (v & 1) == 0; }), buf.end());\n\n    BOOST_REQUIRE(buf.size() == 3);\n    BOOST_REQUIRE(!buf.empty());\n    {\n        auto i = buf.begin();\n        BOOST_REQUIRE_EQUAL(*i++, 1);\n        BOOST_REQUIRE_EQUAL(*i++, 3);\n        BOOST_REQUIRE_EQUAL(*i++, 5);\n        BOOST_REQUIRE(i == buf.end());\n    }\n}\n\nBOOST_AUTO_TEST_CASE(test_erasing_at_beginning_or_end_does_not_invalidate_iterators) {\n    // This guarantee comes from std::deque, which circular_buffer is supposed to mimic.\n\n    circular_buffer<int> buf;\n\n    buf.push_back(1);\n    buf.push_back(2);\n    buf.push_back(3);\n    buf.push_back(4);\n    buf.push_back(5);\n\n    int* ptr_to_3 = &buf[2];\n    auto iterator_to_3 = buf.begin() + 2;\n    SEASTAR_ASSERT(*ptr_to_3 == 3);\n    SEASTAR_ASSERT(*iterator_to_3 == 3);\n\n    buf.erase(buf.begin(), buf.begin() + 2);\n\n    BOOST_REQUIRE(*ptr_to_3 == 3);\n    BOOST_REQUIRE(*iterator_to_3 == 3);\n\n    buf.erase(buf.begin() + 1, buf.end());\n\n    BOOST_REQUIRE(*ptr_to_3 == 3);\n    BOOST_REQUIRE(*iterator_to_3 == 3);\n\n    BOOST_REQUIRE(buf.size() == 1);\n}\n\nBOOST_AUTO_TEST_CASE(test_erasing_in_the_middle) {\n    circular_buffer<int> buf;\n\n    for (int i = 0; i < 10; ++i) {\n        buf.push_back(i);\n    }\n\n    auto i = buf.erase(buf.begin() + 3, buf.begin() + 6);\n    BOOST_REQUIRE_EQUAL(*i, 6);\n\n    i = buf.begin();\n    BOOST_REQUIRE_EQUAL(*i++, 0);\n    BOOST_REQUIRE_EQUAL(*i++, 1);\n    BOOST_REQUIRE_EQUAL(*i++, 2);\n    BOOST_REQUIRE_EQUAL(*i++, 6);\n    BOOST_REQUIRE_EQUAL(*i++, 7);\n    BOOST_REQUIRE_EQUAL(*i++, 8);\n    BOOST_REQUIRE_EQUAL(*i++, 9);\n    BOOST_REQUIRE(i == buf.end());\n}\n\nBOOST_AUTO_TEST_CASE(test_underflow_index_iterator_comparison) {\n    circular_buffer<int> buf;\n\n    const auto seed = std::random_device()();\n    std::cout << \"seed=\" << seed << std::endl;\n    auto rnd_engine = std::mt19937(seed);\n    std::uniform_int_distribution<unsigned> count_dist(0, 20);\n    std::uniform_int_distribution<unsigned> bool_dist(false, true);\n\n    auto push_back = [&buf] (unsigned n) {\n        for (unsigned i = 0; i < n; ++i) {\n            buf.push_back(i);\n        }\n    };\n    auto push_front = [&buf] (unsigned n) {\n        for (unsigned i = 0; i < n; ++i) {\n            buf.push_front(i);\n        }\n    };\n\n    for (unsigned i = 0; i < 16; ++i) {\n        const auto push_back_count = count_dist(rnd_engine);\n        const auto push_front_count = count_dist(rnd_engine);\n        std::cout << \"round[\" << i << \"]: \" << buf.size() << \" front: \" << push_front_count << \" back: \" << push_back_count << std::endl;\n        if (bool_dist(rnd_engine)) {\n            push_back(push_back_count);\n            push_front(std::max(20 - push_back_count, push_front_count));\n        } else {\n            push_front(push_front_count);\n            push_back(std::max(20 - push_front_count, push_back_count));\n        }\n\n        if (buf.empty()) {\n            continue;\n        }\n\n        for (auto it1 = buf.begin(); it1 != buf.end(); ++it1) {\n            bool bypass = false;\n            for (auto it2 = buf.end(); it2 != buf.begin(); --it2) {\n                auto itl = it1;\n                auto ith = it2;\n                if (bypass) {\n                    std::swap(itl, ith);\n                }\n                if (itl == ith) {\n                    bypass = true;\n                } else {\n                    BOOST_REQUIRE(itl < ith);\n                    BOOST_REQUIRE(ith > itl);\n                    BOOST_REQUIRE(!(ith < itl));\n                    BOOST_REQUIRE(!(itl > ith));\n                }\n\n                BOOST_REQUIRE(itl <= ith);\n                BOOST_REQUIRE(itl <= itl);\n                BOOST_REQUIRE(ith <= ith);\n                BOOST_REQUIRE(ith >= itl);\n                BOOST_REQUIRE(itl >= itl);\n                BOOST_REQUIRE(ith >= ith);\n            }\n        }\n\n        const auto erase_count = count_dist(rnd_engine);\n        const auto offset = count_dist(rnd_engine);\n        std::cout << \"round[\" << i << \"]: \" << erase_count << \" @ \" << offset << std::endl;\n        buf.erase(buf.begin() + offset, buf.begin() + std::min(size_t(offset + erase_count), buf.size()));\n    }\n}\n"
  },
  {
    "path": "tests/unit/closeable_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#include <ranges>\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/gate.hh>\n#include <seastar/util/closeable.hh>\n#include <seastar/core/loop.hh>\n\n#include \"expected_exception.hh\"\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(deferred_close_test) {\n  return do_with(gate(), 0, 42, [] (gate& g, int& count, int& expected) {\n    return async([&] {\n        auto close_gate = deferred_close(g);\n\n        for (auto i = 0; i < expected; i++) {\n            (void)with_gate(g, [&count] {\n                ++count;\n            });\n        }\n    }).then([&] {\n        // destroying close_gate should invoke g.close()\n        // and wait for all background continuations to complete\n        BOOST_REQUIRE(g.is_closed());\n        BOOST_REQUIRE_EQUAL(count, expected);\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(move_deferred_close_test) {\n  return do_with(gate(), [] (gate& g) {\n    return async([&] {\n        auto close_gate = make_shared(deferred_close(g));\n        // g.close() should not be called when deferred_close is moved away\n        BOOST_REQUIRE(!g.is_closed());\n    }).then([&] {\n        // Before this test is exercised, gate::close() would run into a\n        // assert failure when leaving previous continuation, if gate::close()\n        // is called twice, so this test only verifies the behavior with the\n        // release build.\n        BOOST_REQUIRE(g.is_closed());\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(close_now_test) {\n  return do_with(gate(), 0, 42, [] (gate& g, int& count, int& expected) {\n    return async([&] {\n        auto close_gate = deferred_close(g);\n\n        for (auto i = 0; i < expected; i++) {\n            (void)with_gate(g, [&count] {\n                ++count;\n            });\n        }\n\n        close_gate.close_now();\n        BOOST_REQUIRE(g.is_closed());\n        BOOST_REQUIRE_EQUAL(count, expected);\n        // gate must not be double-closed.\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(cancel_deferred_close_test) {\n    gate g;\n    {\n        auto close_gate = deferred_close(g);\n        close_gate.cancel();\n    }\n    g.check(); // should not throw\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(with_closeable_test) {\n    return do_with(0, 42, [] (int& count, int& expected) {\n        return with_closeable(gate(), [&] (gate& g) {\n            for (auto i = 0; i < expected; i++) {\n                (void)with_gate(g, [&count] {\n                    ++count;\n                });\n            }\n            return 17;\n        }).then([&] (int res) {\n            // res should be returned by the function called\n            // by with_closeable.\n            BOOST_REQUIRE_EQUAL(res, 17);\n            // closing the gate should wait for\n            // all background continuations to complete\n            BOOST_REQUIRE_EQUAL(count, expected);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(with_closeable_exception_test) {\n    return do_with(0, 42, [] (int& count, int& expected) {\n        return with_closeable(gate(), [&] (gate& g) {\n            for (auto i = 0; i < expected; i++) {\n                (void)with_gate(g, [&count] {\n                    ++count;\n                });\n            }\n            throw expected_exception();\n        }).handle_exception_type([&] (const expected_exception&) {\n            // closing the gate should also happen when func throws,\n            // waiting for all background continuations to complete\n            BOOST_REQUIRE_EQUAL(count, expected);\n        });\n    });\n}\n\nnamespace {\n\nclass count_stops {\n    int _count = -1;\n    int* _ptr = nullptr;\npublic:\n    count_stops(int* ptr = nullptr) noexcept\n        : _ptr(ptr ? ptr : &_count)\n    {\n        *_ptr = 0;\n    }\n\n    count_stops(count_stops&& o) noexcept {\n        std::exchange(_count, o._count);\n        if (o._ptr == &o._count) {\n            _ptr = &_count;\n        } else {\n            std::exchange(_ptr, o._ptr);\n        }\n    }\n\n    future<> stop() noexcept {\n        ++*_ptr;\n        return make_ready_future<>();\n    }\n\n    int stopped() const noexcept {\n        return *_ptr;\n    }\n};\n\n} // anonymous namespace\n\nSEASTAR_TEST_CASE(cancel_deferred_stop_test) {\n    count_stops cs;\n    {\n        auto stop = deferred_stop(cs);\n        stop.cancel();\n    }\n    BOOST_REQUIRE_EQUAL(cs.stopped(), 0);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(deferred_stop_test) {\n  return do_with(count_stops(), [] (count_stops& cs) {\n    return async([&] {\n        auto stop_counting = deferred_stop(cs);\n    }).then([&] {\n        // cs.stop() should be called when stop_counting is destroyed\n        BOOST_REQUIRE_EQUAL(cs.stopped(), 1);\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(move_deferred_stop_test) {\n  return do_with(count_stops(), [] (count_stops& cs) {\n    return async([&] {\n        auto stop = make_shared(deferred_stop(cs));\n    }).then([&] {\n        // cs.stop() should be called once and only once\n        // when stop is destroyed\n        BOOST_REQUIRE_EQUAL(cs.stopped(), 1);\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(stop_now_test) {\n  return do_with(count_stops(), [] (count_stops& cs) {\n    return async([&] {\n        auto stop_counting = deferred_stop(cs);\n\n        stop_counting.stop_now();\n        // cs.stop() should not be called again\n        // when stop_counting is destroyed\n        BOOST_REQUIRE_EQUAL(cs.stopped(), 1);\n    }).then([&] {\n        // cs.stop() should be called exactly once\n        BOOST_REQUIRE_EQUAL(cs.stopped(), 1);\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(with_stoppable_test) {\n    return do_with(0, [] (int& stopped) {\n        return with_stoppable(count_stops(&stopped), [] (count_stops& cs) {\n            return 17;\n        }).then([&] (int res) {\n            // res should be returned by the function called\n            // by with_closeable.\n            BOOST_REQUIRE_EQUAL(res, 17);\n            // cs.stop() should be called before count_stops is destroyed\n            BOOST_REQUIRE_EQUAL(stopped, 1);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(with_stoppable_exception_test) {\n    return do_with(0, [] (int& stopped) {\n        return with_stoppable(count_stops(&stopped), [] (count_stops& cs) {\n            throw expected_exception();\n        }).handle_exception_type([&] (const expected_exception&) {\n            // cs.stop() should be called before count_stops is destroyed\n            // also when func throws\n            BOOST_REQUIRE_EQUAL(stopped, 1);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(move_open_gate_test) {\n    gate g1;\n    g1.enter();\n    // move an open gate\n    gate g2 = std::move(g1);\n    // the state in g1 should be moved into g2\n    BOOST_CHECK_EQUAL(g1.get_count(), 0);\n    BOOST_REQUIRE_EQUAL(g2.get_count(), 1);\n    g2.leave();\n    g2.close().get();\n    BOOST_CHECK(!g1.is_closed());\n    BOOST_CHECK(g2.is_closed());\n}\n\nSEASTAR_THREAD_TEST_CASE(move_closing_gate_test) {\n    gate g1;\n    g1.enter();\n    auto fut = g1.close();\n    // move a closing gate\n    gate g2 = std::move(g1);\n    BOOST_CHECK_EQUAL(g1.get_count(), 0);\n    BOOST_REQUIRE_EQUAL(g2.get_count(), 1);\n    g2.leave();\n    fut.get();\n    BOOST_CHECK(!g1.is_closed());\n    BOOST_CHECK(g2.is_closed());\n}\n\nSEASTAR_THREAD_TEST_CASE(move_closed_gate_test) {\n    gate g1;\n    g1.close().get();\n    // move a closed gate\n    gate g2 = std::move(g1);\n    BOOST_CHECK_EQUAL(g1.get_count(), 0);\n    BOOST_CHECK_EQUAL(g2.get_count(), 0);\n    BOOST_CHECK(!g1.is_closed());\n    BOOST_CHECK(g2.is_closed());\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_basic_test) {\n    gate g;\n    auto gh = g.hold();\n    auto fut = g.close();\n    BOOST_CHECK(!fut.available());\n    gh.release();\n    fut.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_closed_test) {\n    gate g;\n    g.close().get();\n    BOOST_REQUIRE_THROW(g.hold(), gate_closed_exception);\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_move_test) {\n    gate g;\n    auto gh0 = g.hold();\n    auto fut = g.close();\n    BOOST_CHECK(!fut.available());\n    auto gh1 = std::move(gh0);\n    BOOST_CHECK(!fut.available());\n    gh1.release();\n    fut.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_copy_test) {\n    gate g;\n    auto gh0 = g.hold();\n    auto gh1 = gh0;\n    auto fut = g.close();\n    BOOST_CHECK(!fut.available());\n    gh0.release();\n    BOOST_CHECK(!fut.available());\n    gh1.release();\n    fut.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_copy_and_move_test) {\n    gate g0;\n    auto gh00 = g0.hold();\n    auto gh01 = gh00;\n    auto fut0 = g0.close();\n    BOOST_CHECK(!fut0.available());\n    gate g1;\n    auto gh1 = g1.hold();\n    auto fut1 = g1.close();\n    BOOST_CHECK(!fut1.available());\n    gh01.release();\n    BOOST_CHECK(!fut0.available());\n    BOOST_CHECK(!fut1.available());\n    gh00 = std::move(gh1);\n    fut0.get();\n    BOOST_CHECK(!fut1.available());\n    gh00.release();\n    fut1.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_copy_after_close_test) {\n    gate g;\n    auto gh0 = g.hold();\n    auto fut = g.close();\n    BOOST_CHECK(g.is_closed());\n    gate::holder gh1 = gh0;\n    BOOST_CHECK(!fut.available());\n    gh0.release();\n    BOOST_CHECK(!fut.available());\n    gh1.release();\n    fut.get();\n}\n\nSEASTAR_TEST_CASE(gate_holder_parallel_copy_test) {\n    constexpr int expected = 42;\n    return do_with(0, [expected] (int& count) {\n        return with_closeable(gate(), [&] (gate& g) {\n            auto gh = g.hold();\n            // Copying the gate::holder in the lambda below should keep it open\n            // until all instances complete\n            (void)parallel_for_each(std::views::iota(0, expected), [&count, gh = gh] (int) {\n                count++;\n                return make_ready_future<>();\n            });\n            return 17;\n        }).then([&, expected] (int res) {\n            // res should be returned by the function called\n            // by with_closeable.\n            BOOST_REQUIRE_EQUAL(res, 17);\n            // closing the gate should wait for\n            // all background continuations to complete\n            BOOST_REQUIRE_EQUAL(count, expected);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(gate_holder_try_close_test) {\n    gate g;\n    auto gh0 = g.try_hold();\n    BOOST_CHECK(gh0.has_value());\n    auto fut = g.close();\n    BOOST_CHECK(g.is_closed());\n    auto failed_gh = g.try_hold();\n    BOOST_CHECK(!failed_gh.has_value());\n    auto gh1 = std::move(gh0);\n    BOOST_CHECK(!fut.available());\n    gh0.reset();\n    BOOST_CHECK(!fut.available());\n    gh1.reset();\n    fut.get();\n}\n"
  },
  {
    "path": "tests/unit/condition_variable_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/shared_mutex.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/when_any.hh>\n#include <seastar/core/with_timeout.hh>\n#include <boost/range/irange.hpp>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\nusing steady_clock = std::chrono::steady_clock;\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_signal_consume) {\n    condition_variable cv;\n\n    cv.signal();\n    auto f = cv.wait();\n\n    BOOST_REQUIRE_EQUAL(f.available(), true);\n    f.get();\n\n    auto f2 = cv.wait();\n\n    BOOST_REQUIRE_EQUAL(f2.available(), false);\n\n    cv.signal();\n\n    with_timeout(steady_clock::now() + 10ms, std::move(f2)).get();\n\n    std::vector<future<>> waiters;\n    waiters.emplace_back(cv.wait());\n    waiters.emplace_back(cv.wait());\n    waiters.emplace_back(cv.wait());\n\n    BOOST_REQUIRE_EQUAL(std::count_if(waiters.begin(), waiters.end(), std::mem_fn(&future<>::available)), 0u);\n\n    cv.signal();\n\n    BOOST_REQUIRE_EQUAL(std::count_if(waiters.begin(), waiters.end(), std::mem_fn(&future<>::available)), 1u);\n    // FIFO\n    BOOST_REQUIRE_EQUAL(waiters.front().available(), true);\n\n    cv.broadcast();\n\n    BOOST_REQUIRE_EQUAL(std::count_if(waiters.begin(), waiters.end(), std::mem_fn(&future<>::available)), 3u);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_pred) {\n    condition_variable cv;\n    bool ready = false;\n\n    try {\n        cv.wait(100ms, [&] { return ready; }).get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n    // should not affect outcome.\n    cv.signal();\n\n    try {\n        cv.wait(100ms, [&] { return ready; }).get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n\n}\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_signal_break) {\n    condition_variable cv;\n\n    std::vector<future<>> waiters;\n    waiters.emplace_back(cv.wait());\n    waiters.emplace_back(cv.wait());\n    waiters.emplace_back(cv.wait());\n\n    BOOST_REQUIRE_EQUAL(std::count_if(waiters.begin(), waiters.end(), std::mem_fn(&future<>::available)), 0u);\n\n    cv.broken();\n\n    for (auto& f : waiters) {\n        try {\n            f.get();\n        } catch (broken_condition_variable&) {\n            // ok\n            continue;\n        }\n        BOOST_FAIL(\"should not reach\");\n    }\n\n    try {\n        auto f = cv.wait();\n        f.get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (broken_condition_variable&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_timeout) {\n    condition_variable cv;\n\n    auto f = cv.wait(100ms);\n    BOOST_REQUIRE_EQUAL(f.available(), false);\n\n    sleep(200ms).get();\n    BOOST_REQUIRE_EQUAL(f.available(), true);\n\n    try {\n        f.get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_pred_wait) {\n    condition_variable cv;\n\n    bool ready = false;\n\n    timer<> t;\n    t.set_callback([&] { ready = true; cv.signal(); });\n    t.arm(100ms);\n\n    cv.wait([&] { return ready; }).get();\n\n    ready = false;\n\n    try {\n        cv.wait(10ms, [&] { return ready; }).get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (timed_out_error&) {\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n\n    ready = true;\n    cv.signal();\n\n    cv.wait(10ms, [&] { return ready; }).get();\n\n    for (int i = 0; i < 2; ++i) {\n        ready = false;\n        t.set_callback([&] { cv.broadcast();});\n        t.arm_periodic(10ms);\n\n        try {\n            cv.wait(300ms, [&] { return ready; }).get();\n            BOOST_FAIL(\"should not reach\");\n        } catch (timed_out_error&) {\n            BOOST_FAIL(\"should not reach\");\n        } catch (condition_variable_timed_out&) {\n            // ok\n        } catch (...) {\n            BOOST_FAIL(\"should not reach\");\n        }\n        t.cancel();\n        cv.signal();\n    }\n\n    ready = true;\n    cv.signal();\n\n    cv.wait([&] { return ready; }).get();\n    // signal state should remain on\n    cv.wait().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_has_waiter) {\n    condition_variable cv;\n\n    BOOST_REQUIRE_EQUAL(cv.has_waiters(), false);\n\n    auto f = cv.wait();\n    BOOST_REQUIRE_EQUAL(cv.has_waiters(), true);\n\n    cv.signal();\n    f.get();\n    BOOST_REQUIRE_EQUAL(cv.has_waiters(), false);\n}\n\nSEASTAR_TEST_CASE(test_condition_variable_signal_consume_coroutine) {\n    condition_variable cv;\n\n    cv.signal();\n    co_await with_timeout(steady_clock::now() + 10ms, [&]() -> future<> {\n        co_await cv.when();\n    }());\n\n    try {\n        co_await with_timeout(steady_clock::now() + 10ms, [&]() -> future<> {\n            co_await cv.when();\n        }());\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        BOOST_FAIL(\"should not reach\");\n    } catch (timed_out_error&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n\n    try {\n        co_await with_timeout(steady_clock::now() + 10s, [&]() -> future<> {\n            co_await cv.when(100ms);\n        }());\n        BOOST_FAIL(\"should not reach\");\n    } catch (timed_out_error&) {\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n\n}\n\nSEASTAR_TEST_CASE(test_condition_variable_pred_when) {\n    condition_variable cv;\n\n    bool ready = false;\n\n    timer<> t;\n    t.set_callback([&] { ready = true; cv.signal(); });\n    t.arm(100ms);\n\n    co_await cv.when([&] { return ready; });\n\n    ready = false;\n\n    try {\n        co_await cv.when(10ms, [&] { return ready; });\n        BOOST_FAIL(\"should not reach\");\n    } catch (timed_out_error&) {\n        BOOST_FAIL(\"should not reach\");\n    } catch (condition_variable_timed_out&) {\n        // ok\n    } catch (...) {\n        BOOST_FAIL(\"should not reach\");\n    }\n\n    ready = true;\n    cv.signal();\n\n    co_await cv.when(10ms, [&] { return ready; });\n\n    for (int i = 0; i < 2; ++i) {\n        ready = false;\n        t.set_callback([&] { cv.broadcast();});\n        t.arm_periodic(10ms);\n\n        try {\n            co_await cv.when(300ms, [&] { return ready; });\n            BOOST_FAIL(\"should not reach\");\n        } catch (timed_out_error&) {\n            BOOST_FAIL(\"should not reach\");\n        } catch (condition_variable_timed_out&) {\n            // ok\n        } catch (...) {\n            BOOST_FAIL(\"should not reach\");\n        }\n        t.cancel();\n        cv.signal();\n    }\n\n    ready = true;\n    cv.signal();\n\n    co_await cv.when([&] { return ready; });\n    // signal state should remain on\n    co_await cv.when();\n}\n\n\n\nSEASTAR_TEST_CASE(test_condition_variable_when_signal) {\n    condition_variable cv;\n\n    bool ready = false;\n\n    timer<> t;\n    t.set_callback([&] { cv.signal(); ready = true; });\n    t.arm(100ms);\n\n    co_await cv.when();\n    // ensure we did not resume before timer ran fully\n    BOOST_REQUIRE_EQUAL(ready, true);\n}\n\nSEASTAR_TEST_CASE(test_condition_variable_when_timeout) {\n    condition_variable cv;\n\n    bool ready = false;\n\n    // create \"background\" fiber\n    auto f = [&]() -> future<> {\n        try {\n            co_await cv.when(100ms, [&] { return ready; });\n        } catch (timed_out_error&) {\n            BOOST_FAIL(\"should not reach\");\n        } catch (condition_variable_timed_out&) {\n            BOOST_FAIL(\"should not reach\");\n        } catch (...) {\n            BOOST_FAIL(\"should not reach\");\n        }\n    }();\n\n    // ensure we wake up waiter before timeuot\n    ready = true;\n    cv.signal();\n\n    // now busy-spin until the timer should be expired\n    while (cv.has_waiters()) {\n    }\n\n    // he should not have run yet...\n    BOOST_REQUIRE_EQUAL(f.available(), false);\n    // now, if the code is broken, the timer will run once we switch out,\n    // and cause the wait to time out, even though it did not. -> assert\n\n    co_await std::move(f);\n}\n\n// Check that exception from a predicate is propogated to the waiter\nSEASTAR_THREAD_TEST_CASE(test_condition_variable_wait_predicate_throws) {\n    condition_variable cv;\n\n    auto f = cv.wait([i = 1] () mutable {\n                     if (i == 0) {\n                        throw std::runtime_error(\"Predicate error\");\n                     }\n                     i--;\n                     return false;\n             });\n    cv.signal();\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n}\n\n// Check that exception from a predicate is propogated to the waiter\nSEASTAR_TEST_CASE(test_condition_variable_when_predicate_throws) {\n    condition_variable cv;\n\n    (void)sleep(100ms).then([&] { cv.signal(); });\n\n    BOOST_REQUIRE_THROW(co_await\n              cv.when([i = 1] () mutable {\n                     if (i == 0) {\n                        throw std::runtime_error(\"Predicate error\");\n                     }\n                     i--;\n                     return false;\n             }), std::runtime_error);\n}\n"
  },
  {
    "path": "tests/unit/conf-example.yaml",
    "content": "metrics:\n- name: hist1\n  type: histogram\n  values: [1000,2000,3000]\n  labels:\n    private: \"1\"\n- name: gag1\n  type: gauge\n  values: [5]\n  labels:\n    private: \"1\"\n- name: count1\n  type: counter\n  labels:\n    private: \"1\"\n  values: [7]\n- name: counter_1\n  type: counter\n  labels:\n    private: \"2\"\n  values: [1]\n- name: counter_1\n  type: counter\n  labels:\n    private: \"3\"\n  values: [2]\nmetric_family_config:\n- name: test_group_counter_1\n  aggregate_labels:\n    - private\n"
  },
  {
    "path": "tests/unit/connect_test.cc",
    "content": "#include <seastar/core/reactor.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/net/ip.hh>\n\nusing namespace seastar;\nusing namespace net;\n\nSEASTAR_TEST_CASE(test_connection_attempt_is_shutdown) {\n    ipv4_addr server_addr(\"127.0.0.1\");\n    auto unconn = make_socket();\n    auto f = unconn\n        .connect(make_ipv4_address(server_addr))\n        .then_wrapped([] (auto&& f) {\n            try {\n                f.get();\n                BOOST_REQUIRE(false);\n            } catch (...) {}\n        });\n    unconn.shutdown();\n    return f.finally([unconn = std::move(unconn)] {});\n}\n\nSEASTAR_TEST_CASE(test_unconnected_socket_shutsdown_established_connection) {\n    // Use a random port to reduce chance of conflict.\n    // TODO: retry a few times on failure.\n    std::default_random_engine& rnd = testing::local_random_engine;\n    auto distr = std::uniform_int_distribution<uint16_t>(12000, 65000);\n    auto sa = make_ipv4_address({\"127.0.0.1\", distr(rnd)});\n    return do_with(engine().net().listen(sa, listen_options()), [sa] (auto& listener) {\n        auto f = listener.accept();\n        auto unconn = make_socket();\n        auto connf = unconn.connect(sa);\n        return connf.then([unconn = std::move(unconn)] (auto&& conn) mutable {\n            unconn.shutdown();\n            return do_with(std::move(conn), [] (auto& conn) {\n                return do_with(conn.output(1), [] (auto& out) {\n                    return out.write(\"ping\").then_wrapped([] (auto&& f) {\n                        try {\n                            f.get();\n                            BOOST_REQUIRE(false);\n                        } catch (...) {}\n                    });\n                });\n            });\n        }).finally([f = std::move(f)] () mutable {\n            return std::move(f);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_accept_after_abort) {\n    std::default_random_engine& rnd = testing::local_random_engine;\n    auto distr = std::uniform_int_distribution<uint16_t>(12000, 65000);\n    auto sa = make_ipv4_address({\"127.0.0.1\", distr(rnd)});\n    return do_with(seastar::server_socket(engine().net().listen(sa, listen_options())), [] (auto& listener) {\n        using ftype = future<accept_result>;\n        promise<ftype> p;\n        future<ftype> done = p.get_future();\n        auto f = listener.accept().then_wrapped([&listener, p = std::move(p)] (auto f) mutable {\n            f.ignore_ready_future();\n            p.set_value(listener.accept());\n        });\n        listener.abort_accept();\n        return done.then([] (ftype f) {\n          return f.then_wrapped([] (ftype f) {\n            BOOST_REQUIRE(f.failed());\n            if (f.available()) {\n                f.ignore_ready_future();\n            }\n          });\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/content_source_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <seastar/core/sstring.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/http/internal/content_source.hh>\n#include <seastar/testing/test_case.hh>\n#include <tuple>\n\nusing namespace seastar;\n\nclass buf_source_impl : public data_source_impl {\n    temporary_buffer<char> _tmp;\npublic:\n    buf_source_impl(sstring str) : _tmp(str.c_str(), str.size()) {};\n    virtual future<temporary_buffer<char>> get() override {\n        if (_tmp.empty()) {\n            return make_ready_future<temporary_buffer<char>>();\n        }\n        return make_ready_future<temporary_buffer<char>>(std::move(_tmp));\n    }\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        _tmp.trim_front(std::min(_tmp.size(), n));\n        return make_ready_future<temporary_buffer<char>>();\n    }\n};\n\nSEASTAR_TEST_CASE(test_incomplete_content) {\n    return seastar::async([] {\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"asdfghjkl;\"))));\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::content_length_source_impl>(inp, 20)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"asdfghjkl;\", 10) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n        BOOST_REQUIRE(inp.eof());\n      }\n\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"4\\r\\n132\"))));\n        std::unordered_map<sstring, sstring> tmp, tmp2;\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::chunked_source_impl>(inp, tmp, tmp2)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"132\", 3) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n      }\n    });\n}\n\nSEASTAR_TEST_CASE(test_complete_content) {\n    return seastar::async([] {\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"asdfghjkl;1234567890\"))));\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::content_length_source_impl>(inp, 20)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"asdfghjkl;1234567890\", 20) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n      }\n\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"4\\r\\n1324\\r\\n0\\r\\n\\r\\n\"))));\n        std::unordered_map<sstring, sstring> tmp, tmp2;\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::chunked_source_impl>(inp, tmp, tmp2)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"1324\", 4) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n      }\n    });\n}\n\nSEASTAR_TEST_CASE(test_more_than_requests_content) {\n    return seastar::async([] {\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"asdfghjkl;1234567890xyz\"))));\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::content_length_source_impl>(inp, 20)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"asdfghjkl;1234567890\", 20) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n        auto content3 = inp.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"xyz\", 3) == content3);\n      }\n\n      {\n        auto inp = input_stream<char>(data_source(std::make_unique<buf_source_impl>(sstring(\"4\\r\\n1324\\r\\n0\\r\\n\\r\\nxyz\"))));\n        std::unordered_map<sstring, sstring> tmp, tmp2;\n        auto content_strm = input_stream<char>(data_source(std::make_unique<httpd::internal::chunked_source_impl>(inp, tmp, tmp2)));\n\n        auto content1 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"1324\", 4) == content1);\n        auto content2 = content_strm.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == content2);\n        BOOST_REQUIRE(content_strm.eof());\n        auto content3 = inp.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>(\"xyz\", 3) == content3);\n      }\n    });\n}\n\nclass single_bytes_source_impl : public data_source_impl {\n    temporary_buffer<char> _tmp;\npublic:\n    single_bytes_source_impl(temporary_buffer<char> tmp)\n        : _tmp(std::move(tmp)) {\n    }\n    virtual future<temporary_buffer<char>> get() override {\n        if (_tmp.empty()) {\n            return make_ready_future<temporary_buffer<char>>();\n        }\n        auto byte = _tmp.share(0, 1);\n        _tmp.trim_front(1);\n        return make_ready_future<temporary_buffer<char>>(std::move(byte));\n    }\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        _tmp.trim_front(std::min(_tmp.size(), n));\n        return make_ready_future<temporary_buffer<char>>();\n    }\n};\n\nSEASTAR_TEST_CASE(test_single_bytes_source) {\n    return seastar::async([] {\n        sstring input_str = \"test input\";\n        auto ds = data_source(std::make_unique<single_bytes_source_impl>(temporary_buffer<char>(input_str.c_str(), input_str.size())));\n        for (auto& ch : input_str) {\n            temporary_buffer<char> one_letter_buf(1);\n            *one_letter_buf.get_write() = ch;\n            auto get_buf = ds.get().get();\n            BOOST_REQUIRE(one_letter_buf == get_buf);\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_fragmented_chunks) {\n    // Test if a message that cannot be parsed as a http request is being replied with a 400 Bad Request response\n    return seastar::async([] {\n        sstring request_string = \"a;chunk=ext\\r\\n1234567890\\r\\n0\\r\\ntrailer: part\\r\\n\\r\\n\";\n        auto inp = input_stream<char>(data_source(std::make_unique<single_bytes_source_impl>(temporary_buffer<char>(request_string.c_str(), request_string.size()))));\n        std::unordered_map<sstring, sstring> chunk_extensions;\n        std::unordered_map<sstring, sstring> trailing_headers;\n        auto content_stream = input_stream<char>(data_source(std::make_unique<httpd::internal::chunked_source_impl>(inp, chunk_extensions, trailing_headers)));\n        for (auto& ch : sstring(\"1234567890\")) {\n            temporary_buffer<char> one_letter_buf(1);\n            *one_letter_buf.get_write() = ch;\n            auto read_buf = content_stream.read().get();\n            BOOST_REQUIRE(one_letter_buf == read_buf);\n        }\n        auto read_buf = content_stream.read().get();\n        BOOST_REQUIRE(temporary_buffer<char>() == read_buf);\n        BOOST_REQUIRE(chunk_extensions[sstring(\"chunk\")] == sstring(\"ext\"));\n        BOOST_REQUIRE(trailing_headers[sstring(\"trailer\")] == sstring(\"part\"));\n    });\n}\n"
  },
  {
    "path": "tests/unit/coroutines_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 ScyllaDB Ltd.\n */\n\n#include <exception>\n#include <numeric>\n#include <ranges>\n#include <any>\n\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/coroutine/all.hh>\n#include <seastar/coroutine/maybe_yield.hh>\n#include <seastar/coroutine/switch_to.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/exception.hh>\n#include <seastar/coroutine/try_future.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/later.hh>\n\nusing seastar::broken_promise;\nusing seastar::circular_buffer;\nusing seastar::create_scheduling_group;\nusing seastar::current_scheduling_group;\nusing seastar::default_scheduling_group;\nusing seastar::future;\nusing seastar::make_exception_future;\nusing seastar::make_ready_future;\nusing seastar::need_preempt;\nusing seastar::promise;\nusing seastar::scheduling_group;\nusing seastar::semaphore;\nusing seastar::semaphore_timed_out;\nusing seastar::sleep;\nusing seastar::yield;\n\nnamespace coroutine = seastar::coroutine;\nnamespace testing = seastar::testing;\n\nusing namespace std::chrono_literals;\n\nnamespace {\n\nfuture<int> old_fashioned_continuations() {\n    return yield().then([] {\n        return 42;\n    });\n}\n\nfuture<int> simple_coroutine() {\n    co_await yield();\n    co_return 53;\n}\n\nfuture<int> ready_coroutine() {\n    co_return 64;\n}\n\nfuture<std::tuple<int, double>> tuple_coroutine() {\n    co_return std::tuple(1, 2.);\n}\n\nfuture<int> failing_coroutine() {\n    co_await yield();\n    throw 42;\n}\n\n[[gnu::noinline]] int throw_exception(int x) {\n    throw x;\n}\n\nfuture<int> failing_coroutine2() noexcept {\n    co_await yield();\n    co_return throw_exception(17);\n}\n\n}\n\nSEASTAR_TEST_CASE(test_simple_coroutines) {\n    BOOST_REQUIRE_EQUAL(co_await old_fashioned_continuations(), 42);\n    BOOST_REQUIRE_EQUAL(co_await simple_coroutine(), 53);\n    BOOST_REQUIRE_EQUAL(ready_coroutine().get(), 64);\n    BOOST_REQUIRE(co_await tuple_coroutine() == std::tuple(1, 2.));\n    BOOST_REQUIRE_EXCEPTION((void)co_await failing_coroutine(), int, [] (auto v) { return v == 42; });\n    BOOST_CHECK_EQUAL(co_await failing_coroutine().then_wrapped([] (future<int> f) -> future<int> {\n        BOOST_REQUIRE(f.failed());\n        try {\n            std::rethrow_exception(f.get_exception());\n        } catch (int v) {\n           co_return v;\n        }\n    }), 42);\n    BOOST_REQUIRE_EXCEPTION((void)co_await failing_coroutine2(), int, [] (auto v) { return v == 17; });\n    BOOST_CHECK_EQUAL(co_await failing_coroutine2().then_wrapped([] (future<int> f) -> future<int> {\n        BOOST_REQUIRE(f.failed());\n        try {\n            std::rethrow_exception(f.get_exception());\n        } catch (int v) {\n           co_return v;\n        }\n    }), 17);\n}\n\nSEASTAR_TEST_CASE(test_abandond_coroutine) {\n    std::optional<future<int>> f;\n    {\n        auto p1 = promise<>();\n        auto p2 = promise<>();\n        auto p3 = promise<>();\n        f = p1.get_future().then([&] () -> future<int> {\n            p2.set_value();\n            BOOST_CHECK_THROW(co_await p3.get_future(), broken_promise);\n            co_return 1;\n        });\n        p1.set_value();\n        co_await p2.get_future();\n    }\n    BOOST_CHECK_EQUAL(co_await std::move(*f), 1);\n}\n\nSEASTAR_TEST_CASE(test_scheduling_group) {\n    auto other_sg = co_await create_scheduling_group(\"the other group\", 10.f);\n    std::exception_ptr ex;\n\n    try {\n        auto p1 = promise<>();\n        auto p2 = promise<>();\n\n        auto p1b = promise<>();\n        auto p2b = promise<>();\n        auto f1 = p1b.get_future();\n        auto f2 = p2b.get_future();\n\n        BOOST_REQUIRE(current_scheduling_group() == default_scheduling_group());\n        auto f_ret = with_scheduling_group(other_sg,\n                [other_sg_cap = other_sg] (future<> f1, future<> f2, promise<> p1, promise<> p2) -> future<int> {\n            // Make a copy in the coroutine before the lambda is destroyed.\n            auto other_sg = other_sg_cap;\n            BOOST_REQUIRE(current_scheduling_group() == other_sg);\n            BOOST_REQUIRE(other_sg == other_sg);\n            p1.set_value();\n            co_await std::move(f1);\n            BOOST_REQUIRE(current_scheduling_group() == other_sg);\n            p2.set_value();\n            co_await std::move(f2);\n            BOOST_REQUIRE(current_scheduling_group() == other_sg);\n            co_return 42;\n        }, p1.get_future(), p2.get_future(), std::move(p1b), std::move(p2b));\n\n        co_await std::move(f1);\n        BOOST_REQUIRE(current_scheduling_group() == default_scheduling_group());\n        p1.set_value();\n        co_await std::move(f2);\n        BOOST_REQUIRE(current_scheduling_group() == default_scheduling_group());\n        p2.set_value();\n        BOOST_REQUIRE_EQUAL(co_await std::move(f_ret), 42);\n        BOOST_REQUIRE(current_scheduling_group() == default_scheduling_group());\n    } catch (...) {\n        ex = std::current_exception();\n    }\n    co_await destroy_scheduling_group(other_sg);\n    if (ex) {\n        std::rethrow_exception(std::move(ex));\n    }\n}\n\nfuture<scheduling_group> switch_to_with_context(scheduling_group& sg) {\n    scheduling_group new_sg = co_await coroutine::switch_to(sg);\n    BOOST_REQUIRE(current_scheduling_group() == sg);\n    co_return new_sg;\n}\n\nSEASTAR_TEST_CASE(test_switch_to) {\n    auto other_sg0 = co_await create_scheduling_group(\"other group 0\", 10.f);\n    auto other_sg1 = co_await create_scheduling_group(\"other group 1\", 10.f);\n    auto other_sg2 = co_await create_scheduling_group(\"other group 2\", 10.f);\n    std::exception_ptr ex;\n\n    try {\n        auto base_sg = current_scheduling_group();\n\n        auto prev_sg = co_await coroutine::switch_to(other_sg0);\n        BOOST_REQUIRE(current_scheduling_group() == other_sg0);\n        BOOST_REQUIRE(prev_sg == base_sg);\n\n        auto same_sg = co_await coroutine::switch_to(other_sg0);\n        BOOST_REQUIRE(current_scheduling_group() == other_sg0);\n        BOOST_REQUIRE(same_sg == other_sg0);\n\n        auto nested_sg = co_await coroutine::switch_to(other_sg1);\n        BOOST_REQUIRE(current_scheduling_group() == other_sg1);\n        BOOST_REQUIRE(nested_sg == other_sg0);\n\n        co_await switch_to_with_context(other_sg2);\n\n        co_await coroutine::switch_to(base_sg);\n        BOOST_REQUIRE(current_scheduling_group() == base_sg);\n    } catch (...) {\n        ex = std::current_exception();\n    }\n\n    co_await destroy_scheduling_group(other_sg1);\n    co_await destroy_scheduling_group(other_sg0);\n    if (ex) {\n        std::rethrow_exception(std::move(ex));\n    }\n}\n\nfuture<> check_thread_inherits_sg_from_coroutine_frame(scheduling_group expected_sg) {\n    return seastar::async([expected_sg] {\n        BOOST_REQUIRE(current_scheduling_group() == expected_sg);\n    });\n}\n\nfuture<> check_coroutine_inherits_sg_from_another_one(scheduling_group expected_sg) {\n    co_await yield();\n    BOOST_REQUIRE(current_scheduling_group() == expected_sg);\n}\n\nfuture<> switch_to_sg_and_perform_inheriting_checks(scheduling_group base_sg, scheduling_group new_sg) {\n    BOOST_REQUIRE(current_scheduling_group() == base_sg);\n    co_await coroutine::switch_to(new_sg);\n    BOOST_REQUIRE(current_scheduling_group() == new_sg);\n\n    co_await check_thread_inherits_sg_from_coroutine_frame(new_sg);\n    co_await check_coroutine_inherits_sg_from_another_one(new_sg);\n\n    // don't restore previous sg on purpose, expecting it will be restored once coroutine goes out of scope\n}\n\nSEASTAR_TEST_CASE(test_switch_to_sg_restoration_and_inheriting) {\n    auto new_sg = co_await create_scheduling_group(\"other group 0\", 10.f);\n    std::exception_ptr ex;\n\n    try {\n        auto base_sg = current_scheduling_group();\n\n        co_await switch_to_sg_and_perform_inheriting_checks(base_sg, new_sg);\n        // seastar automatically restores base_sg once it goes out of coroutine frame\n        BOOST_REQUIRE(current_scheduling_group() == base_sg);\n\n        co_await seastar::async([base_sg, new_sg] {\n            switch_to_sg_and_perform_inheriting_checks(base_sg, new_sg).get();\n            BOOST_REQUIRE(current_scheduling_group() == base_sg);\n        });\n\n        co_await switch_to_sg_and_perform_inheriting_checks(base_sg, new_sg).finally([base_sg] {\n            BOOST_REQUIRE(current_scheduling_group() == base_sg);\n        });\n    } catch (...) {\n        ex = std::current_exception();\n    }\n\n    co_await destroy_scheduling_group(new_sg);\n    if (ex) {\n        std::rethrow_exception(std::move(ex));\n    }\n}\n\nSEASTAR_TEST_CASE(test_preemption) {\n    bool x = false;\n    unsigned preempted = 0;\n    auto f = yield().then([&x] {\n            x = true;\n        });\n\n    // try to preempt 1000 times. 1 should be enough if not for\n    // task queue shaffling in debug mode which may cause co-routine\n    // continuation to run first.\n    while(preempted < 1000 && !x) {\n        preempted += need_preempt();\n        co_await make_ready_future<>();\n    }\n    auto save_x = x;\n    // wait for yield() to complete\n    co_await std::move(f);\n    BOOST_REQUIRE(save_x);\n    co_return;\n}\n\nSEASTAR_TEST_CASE(test_no_preemption) {\n    bool x = false;\n    unsigned preempted = 0;\n    auto f = yield().then([&x] {\n            x = true;\n        });\n\n    // preemption should not happen, we explicitly asked for continuing if possible\n    while(preempted < 1000 && !x) {\n        preempted += need_preempt();\n        co_await coroutine::without_preemption_check(make_ready_future<>());\n    }\n    auto save_x = x;\n    // wait for yield() to complete\n    co_await std::move(f);\n    BOOST_REQUIRE(!save_x);\n    co_return;\n}\n\nSEASTAR_TEST_CASE(test_all_simple) {\n    auto [a, b] = co_await coroutine::all(\n        [] { return make_ready_future<int>(1); },\n        [] { return make_ready_future<int>(2); }\n    );\n    BOOST_REQUIRE_EQUAL(a, 1);\n    BOOST_REQUIRE_EQUAL(b, 2);\n}\n\nSEASTAR_TEST_CASE(test_all_permutations) {\n    std::vector<std::chrono::milliseconds> delays = { 0ms, 0ms, 2ms, 2ms, 4ms, 6ms };\n    auto make_delayed_future_returning_nr = [&] (int nr) {\n        return [=] {\n            auto delay = delays[nr];\n            return delay == 0ms ? make_ready_future<int>(nr) : sleep(delay).then([nr] { return make_ready_future<int>(nr); });\n        };\n    };\n    do {\n        auto [a, b, c, d, e, f] = co_await coroutine::all(\n            make_delayed_future_returning_nr(0),\n            make_delayed_future_returning_nr(1),\n            make_delayed_future_returning_nr(2),\n            make_delayed_future_returning_nr(3),\n            make_delayed_future_returning_nr(4),\n            make_delayed_future_returning_nr(5)\n        );\n        BOOST_REQUIRE_EQUAL(a, 0);\n        BOOST_REQUIRE_EQUAL(b, 1);\n        BOOST_REQUIRE_EQUAL(c, 2);\n        BOOST_REQUIRE_EQUAL(d, 3);\n        BOOST_REQUIRE_EQUAL(e, 4);\n        BOOST_REQUIRE_EQUAL(f, 5);\n    } while (std::ranges::next_permutation(delays).found);\n}\n\nSEASTAR_TEST_CASE(test_all_ready_exceptions) {\n    try {\n        co_await coroutine::all(\n            [] () -> future<> { throw 1; },\n            [] () -> future<> { throw 2; }\n        );\n    } catch (int e) {\n        BOOST_REQUIRE(e == 1 || e == 2);\n    }\n}\n\nSEASTAR_TEST_CASE(test_all_nonready_exceptions) {\n    try {\n        co_await coroutine::all(\n            [] () -> future<> {\n                co_await sleep(1ms);\n                throw 1;\n            },\n            [] () -> future<> {\n                co_await sleep(1ms);\n                throw 2;\n            }\n        );\n    } catch (int e) {\n        BOOST_REQUIRE(e == 1 || e == 2);\n    }\n}\n\nSEASTAR_TEST_CASE(test_all_heterogeneous_types) {\n    auto [a, b] = co_await coroutine::all(\n        [] () -> future<int> {\n            co_await sleep(1ms);\n            co_return 1;\n        },\n        [] () -> future<> {\n            co_await sleep(1ms);\n        },\n        [] () -> future<long> {\n            co_await sleep(1ms);\n            co_return 2L;\n        }\n    );\n    BOOST_REQUIRE_EQUAL(a, 1);\n    BOOST_REQUIRE_EQUAL(b, 2L);\n}\n\nSEASTAR_TEST_CASE(test_all_noncopyable_types) {\n    auto [a] = co_await coroutine::all(\n        [] () -> future<std::unique_ptr<int>> {\n            co_return std::make_unique<int>(6);\n        }\n    );\n    BOOST_REQUIRE_EQUAL(*a, 6);\n}\n\nSEASTAR_TEST_CASE(test_all_throw_in_input_func) {\n    int nr_completed = 0;\n    bool exception_seen = false;\n    try {\n        co_await coroutine::all(\n            [&] () -> future<int> {\n                co_await sleep(1ms);\n                ++nr_completed;\n                co_return 7;\n            },\n            [&] () -> future<int> {\n                throw 9;\n            },\n            [&] () -> future<int> {\n                co_await sleep(1ms);\n                ++nr_completed;\n                co_return 7;\n            }\n        );\n    } catch (int n) {\n        BOOST_REQUIRE_EQUAL(n, 9);\n        exception_seen = true;\n    }\n    BOOST_REQUIRE_EQUAL(nr_completed, 2);\n    BOOST_REQUIRE(exception_seen);\n}\n\nstruct counter_ref {\nprivate:\n    int& _counter;\n\npublic:\n    explicit counter_ref(int& cnt)\n            : _counter(cnt) {\n        ++_counter;\n    }\n\n    ~counter_ref() {\n        --_counter;\n    }\n};\n\ntemplate<typename Ex>\nstatic future<> check_coroutine_throws(auto fun) {\n    // The counter keeps track of the number of alive \"counter_ref\" objects.\n    // If it is not zero then it means that some destructors weren't run\n    // while quitting the coroutine.\n    int counter = 0;\n    BOOST_REQUIRE_THROW(co_await fun(counter), Ex);\n    BOOST_REQUIRE_EQUAL(counter, 0);\n    co_await fun(counter).then_wrapped([&counter] (auto f) {\n        BOOST_REQUIRE(f.failed());\n        BOOST_REQUIRE_THROW(std::rethrow_exception(f.get_exception()), Ex);\n        BOOST_REQUIRE_EQUAL(counter, 0);\n    });\n}\n\nSEASTAR_TEST_CASE(test_coroutine_exception) {\n    co_await check_coroutine_throws<std::runtime_error>([] (int& counter) -> future<int> {\n        counter_ref ref{counter};\n        co_return coroutine::exception(std::make_exception_ptr(std::runtime_error(\"threw\")));\n    });\n    co_await check_coroutine_throws<std::runtime_error>([] (int& counter) -> future<int> {\n        counter_ref ref{counter};\n        co_await coroutine::exception(std::make_exception_ptr(std::runtime_error(\"threw\")));\n        co_return 42;\n    });\n    co_await check_coroutine_throws<std::logic_error>([] (int& counter) -> future<> {\n        counter_ref ref{counter};\n        co_await coroutine::return_exception(std::logic_error(\"threw\"));\n        co_return;\n    });\n    co_await check_coroutine_throws<int>([] (int& counter) -> future<> {\n        counter_ref ref{counter};\n        co_await coroutine::return_exception(42);\n        co_return;\n    });\n}\n\nSEASTAR_TEST_CASE(test_coroutine_return_exception_ptr) {\n    co_await check_coroutine_throws<std::runtime_error>([] (int& counter) -> future<> {\n        co_await coroutine::return_exception(std::runtime_error(\"threw\"));\n    });\n    co_await check_coroutine_throws<std::runtime_error>([] (int& counter) -> future<> {\n        auto ex = std::make_exception_ptr(std::runtime_error(\"threw\"));\n        co_await coroutine::return_exception_ptr(std::move(ex));\n    });\n    co_await check_coroutine_throws<std::runtime_error>([] (int& counter) -> future<> {\n        auto ex = std::make_exception_ptr(std::runtime_error(\"threw\"));\n        co_await coroutine::return_exception_ptr(ex);\n    });\n    co_await check_coroutine_throws<int>([] (int& counter) -> future<> {\n        co_await coroutine::return_exception_ptr(std::make_exception_ptr(3));\n    });\n}\n\nSEASTAR_TEST_CASE(test_maybe_yield) {\n    int var = 0;\n    bool done = false;\n    auto spinner = [&] () -> future<> {\n        // increment a variable continuously, but yield so an observer can see it.\n        while (!done) {\n            ++var;\n            co_await coroutine::maybe_yield();\n        }\n    };\n    auto spinner_fut = spinner();\n    int snapshot = var;\n    for (int nr_changes = 0; nr_changes < 10; ++nr_changes) {\n        // Try to observe the value changing in time, yield to\n        // allow the spinner to advance it.\n        while (snapshot == var) {\n            co_await coroutine::maybe_yield();\n        }\n        snapshot = var;\n    }\n    done = true;\n    co_await std::move(spinner_fut);\n    BOOST_REQUIRE(true); // the test will hang if it doesn't work.\n}\n\n#ifndef __clang__\n\n#include \"tl-generator.hh\"\ntl::generator<int> simple_generator(int max)\n{\n    for (int i = 0; i < max; ++i) {\n        co_yield i;\n    }\n}\n\nSEASTAR_TEST_CASE(generator)\n{\n    // test ability of seastar::parallel_for_each to deal with move-only views\n    int accum = 0;\n    co_await seastar::parallel_for_each(simple_generator(10), [&](int i) {\n        accum += i;\n        return seastar::make_ready_future<>();\n    });\n    BOOST_REQUIRE_EQUAL(accum, 45);\n\n    // test ability of seastar::max_concurrent_for_each to deal with move-only views\n    accum = 0;\n    co_await seastar::max_concurrent_for_each(simple_generator(10), 10, [&](int i) {\n        accum += i;\n        return seastar::make_ready_future<>();\n    });\n    BOOST_REQUIRE_EQUAL(accum, 45);\n}\n\n#endif\n\nSEASTAR_TEST_CASE(test_parallel_for_each_empty) {\n    std::vector<int> values;\n    int count = 0;\n\n    co_await coroutine::parallel_for_each(values, [&] (int x) {\n        ++count;\n    });\n    BOOST_REQUIRE_EQUAL(count, 0); // the test will hang if it doesn't work.\n}\n\nSEASTAR_TEST_CASE(test_parallel_for_each_exception) {\n    std::array<int, 5> values = { 10, 2, 1, 4, 8 };\n    int count = 0;\n    auto& eng = testing::local_random_engine;\n    auto dist = std::uniform_int_distribution<unsigned>();\n    int throw_at = dist(eng) % values.size();\n\n    BOOST_TEST_MESSAGE(fmt::format(\"Will throw at value #{}/{}\", throw_at, values.size()));\n\n    auto f0 = coroutine::parallel_for_each(values, [&] (int x) {\n        if (count++ == throw_at) {\n            BOOST_TEST_MESSAGE(\"throw\");\n            throw std::runtime_error(\"test\");\n        }\n    });\n    // An exception thrown by the functor must be propagated\n    BOOST_REQUIRE_THROW(co_await std::move(f0), std::runtime_error);\n    // Functor must be called on all values, even if there's an exception\n    BOOST_REQUIRE_EQUAL(count, values.size());\n\n    count = 0;\n    throw_at = dist(eng) % values.size();\n    BOOST_TEST_MESSAGE(fmt::format(\"Will throw at value #{}/{}\", throw_at, values.size()));\n\n    auto f1 = coroutine::parallel_for_each(values, [&] (int x) -> future<> {\n        co_await sleep(std::chrono::milliseconds(x));\n        if (count++ == throw_at) {\n            throw std::runtime_error(\"test\");\n        }\n    });\n    BOOST_REQUIRE_THROW(co_await std::move(f1), std::runtime_error);\n    BOOST_REQUIRE_EQUAL(count, values.size());\n}\n\nSEASTAR_TEST_CASE(test_parallel_for_each) {\n    std::vector<int> values = { 3, 1, 4 };\n    int sum_of_squares = 0;\n\n    int expected = std::accumulate(values.begin(), values.end(), 0, [] (int sum, int x) {\n        return sum + x * x;\n    });\n\n    // Test all-ready futures\n    co_await coroutine::parallel_for_each(values, [&sum_of_squares] (int x) {\n        sum_of_squares += x * x;\n    });\n    BOOST_REQUIRE_EQUAL(sum_of_squares, expected);\n\n    // Test non-ready futures\n    sum_of_squares = 0;\n    co_await coroutine::parallel_for_each(values, [&sum_of_squares] (int x) -> future<> {\n        if (x > 1) {\n            co_await sleep(std::chrono::milliseconds(x));\n        }\n        sum_of_squares += x * x;\n    });\n    BOOST_REQUIRE_EQUAL(sum_of_squares, expected);\n\n    // Test legacy subrange\n    sum_of_squares = 0;\n    co_await coroutine::parallel_for_each(values.begin(), values.end() - 1, [&sum_of_squares] (int x) -> future<> {\n        if (x > 1) {\n            co_await sleep(std::chrono::milliseconds(x));\n        }\n        sum_of_squares += x * x;\n    });\n    BOOST_REQUIRE_EQUAL(sum_of_squares, 10);\n\n    // clang 13.0.1 doesn't support subrange\n    // so provide also a Iterator/Sentinel based constructor.\n    // See https://github.com/llvm/llvm-project/issues/46091\n#ifndef __clang__\n    // Test std::ranges::subrange\n    sum_of_squares = 0;\n    co_await coroutine::parallel_for_each(std::ranges::subrange(values.begin(), values.end() - 1), [&sum_of_squares] (int x) -> future<> {\n        if (x > 1) {\n            co_await sleep(std::chrono::milliseconds(x));\n        }\n        sum_of_squares += x * x;\n    });\n    BOOST_REQUIRE_EQUAL(sum_of_squares, 10);\n#endif\n}\n\nSEASTAR_TEST_CASE(test_void_as_future) {\n    auto f = co_await coroutine::as_future(make_ready_future<>());\n    BOOST_REQUIRE(f.available());\n\n    f = co_await coroutine::as_future(make_exception_future<>(std::runtime_error(\"exception\")));\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n\n    semaphore sem(0);\n    (void)sleep(1ms).then([&] { sem.signal(); });\n    f = co_await coroutine::as_future(sem.wait());\n    BOOST_REQUIRE(f.available());\n\n    f = co_await coroutine::as_future(sem.wait(duration_cast<semaphore::duration>(1ms)));\n    BOOST_REQUIRE_THROW(f.get(), semaphore_timed_out);\n}\n\nSEASTAR_TEST_CASE(test_void_as_future_without_preemption_check) {\n    auto f = co_await coroutine::as_future_without_preemption_check(make_ready_future<>());\n    BOOST_REQUIRE(f.available());\n\n    f = co_await coroutine::as_future_without_preemption_check(make_exception_future<>(std::runtime_error(\"exception\")));\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n\n    semaphore sem(0);\n    (void)sleep(1ms).then([&] { sem.signal(); });\n    f = co_await coroutine::as_future_without_preemption_check(sem.wait());\n    BOOST_REQUIRE(f.available());\n\n    f = co_await coroutine::as_future_without_preemption_check(sem.wait(duration_cast<semaphore::duration>(1ms)));\n    BOOST_REQUIRE_THROW(f.get(), semaphore_timed_out);\n}\n\nSEASTAR_TEST_CASE(test_non_void_as_future) {\n    auto f = co_await coroutine::as_future(make_ready_future<int>(42));\n    BOOST_REQUIRE_EQUAL(f.get(), 42);\n\n    f = co_await coroutine::as_future(make_exception_future<int>(std::runtime_error(\"exception\")));\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n\n    auto p = promise<int>();\n    (void)sleep(1ms).then([&] { p.set_value(314); });\n    f = co_await coroutine::as_future(p.get_future());\n    BOOST_REQUIRE_EQUAL(f.get(), 314);\n\n    auto gen_exception = [] () -> future<int> {\n        co_await sleep(1ms);\n        co_return coroutine::exception(std::make_exception_ptr(std::runtime_error(\"exception\")));\n    };\n    f = co_await coroutine::as_future(gen_exception());\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n}\n\nSEASTAR_TEST_CASE(test_non_void_as_future_without_preemption_check) {\n    auto f = co_await coroutine::as_future_without_preemption_check(make_ready_future<int>(42));\n    BOOST_REQUIRE_EQUAL(f.get(), 42);\n\n    f = co_await coroutine::as_future_without_preemption_check(make_exception_future<int>(std::runtime_error(\"exception\")));\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n\n    auto p = promise<int>();\n    (void)sleep(1ms).then([&] { p.set_value(314); });\n    f = co_await coroutine::as_future_without_preemption_check(p.get_future());\n    BOOST_REQUIRE_EQUAL(f.get(), 314);\n\n    auto gen_exception = [] () -> future<int> {\n        co_await sleep(1ms);\n        co_return coroutine::exception(std::make_exception_ptr(std::runtime_error(\"exception\")));\n    };\n    f = co_await coroutine::as_future_without_preemption_check(gen_exception());\n    BOOST_REQUIRE_THROW(f.get(), std::runtime_error);\n}\n\nSEASTAR_TEST_CASE(test_as_future_preemption) {\n    bool stop = false;\n\n    auto get_ready_future = [&] {\n        return stop ? make_exception_future<>(std::runtime_error(\"exception\")) : make_ready_future<>();\n    };\n\n    auto wait_for_stop = [&] () -> future<bool> {\n        for (;;) {\n            auto f = co_await coroutine::as_future(get_ready_future());\n            if (f.failed()) {\n                co_return coroutine::exception(f.get_exception());\n            }\n        }\n    };\n\n    auto f0 = wait_for_stop();\n\n    auto set_stop = [&] () -> future<> {\n        for (;;) {\n            stop = true;\n            if (f0.available()) {\n                co_return;\n            }\n            co_await coroutine::maybe_yield();\n        }\n    };\n\n    co_await set_stop();\n\n    BOOST_REQUIRE_THROW(f0.get(), std::runtime_error);\n}\n\nSEASTAR_TEST_CASE(test_lambda_coroutine_in_continuation) {\n    auto dist = std::uniform_real_distribution<>(0.0, 1.0);\n    auto rand_eng = std::default_random_engine(std::random_device()());\n    double n = dist(rand_eng);\n    auto sin1 = std::sin(n); // avoid optimizer tricks\n    auto boo = std::array<char, 1025>(); // bias coroutine size towards 1024 size class\n    auto sin2 = co_await yield().then(coroutine::lambda([n, boo] () -> future<double> {\n        // Expect coroutine capture to be freed after co_await without coroutine::lambda\n        co_await yield();\n        // Try to overwrite recently-release coroutine frame by allocating in similar size-class\n        std::vector<char*> garbage;\n        for (size_t sz = 1024; sz < 2048; ++sz) {\n            for (int ctr = 0; ctr < 100; ++ctr) {\n                auto p = static_cast<char*>(malloc(sz));\n                std::memset(p, 0, sz);\n                garbage.push_back(p);\n            }\n        }\n        for (auto p : garbage) {\n            std::free(p);\n        }\n        (void)boo;\n        co_return std::sin(n);\n    }));\n    BOOST_REQUIRE_EQUAL(sin1, sin2);\n}\n\n#ifdef __cpp_explicit_this_parameter\n\nSEASTAR_TEST_CASE(test_lambda_value_capture_coroutine) {\n    // Note: crashes without \"this auto\"\n    auto f1 = [p = std::make_unique<int>(7)] (this auto) -> future<int> {\n        co_await yield();\n        co_return *p + 2;\n    }();\n    // f1 is not ready at this point (due to the yield). Verify that the capture\n    // group was copied to the coroutine frame.\n    auto n = co_await std::move(f1);\n    BOOST_REQUIRE_EQUAL(n, 9);\n}\n\nSEASTAR_TEST_CASE(test_lambda_value_capture_continuation) {\n    // Note: crashes without \"this auto\"\n    return yield().then([p = std::make_unique<int>(7)] (this auto) -> future<int> {\n        co_await yield();\n        co_return *p + 2;\n    }).then([] (int n) {\n        BOOST_REQUIRE_EQUAL(n, 9);\n    });\n}\n\n\n#endif\n\nclass test_exception : std::exception { };\n\nfuture<> return_ex_void() {\n    fmt::print(\"return_ex_void()\\n\");\n    return make_exception_future<>(test_exception{});\n}\n\nfuture<> return_void() {\n    fmt::print(\"return_void()\\n\");\n    co_await sleep(1ms);\n}\n\nfuture<int> return_ex_int() {\n    fmt::print(\"return_ex_int()\\n\");\n    return make_exception_future<int>(test_exception{});\n}\n\nfuture<int> return_int() {\n    fmt::print(\"return_int()\\n\");\n    co_await sleep(1ms);\n    co_return 128;\n}\n\ntemplate <typename T>\nstruct result_wrapper {\n    T value;\n    explicit result_wrapper(T v) : value(v) {}\n};\n\ntemplate <>\nstruct result_wrapper<seastar::internal::monostate> {\n};\n\ntemplate <bool CheckPreempt, std::invocable<> F>\n// Use result_wrapper to create a mismatch between the return type of\n// the coroutine and that of the underlying function, to ensure that\n// try_future handles this case correctly.\nfuture<result_wrapper<typename std::invoke_result_t<F>::value_type>>\ndo_run_try_future_test(F underlying_func, int& ctor_dtor_counter, bool& run_past) {\n    auto func = [&] () -> future<result_wrapper<typename std::invoke_result_t<F>::value_type>> {\n        counter_ref c1{ctor_dtor_counter};\n        counter_ref c2{ctor_dtor_counter};\n\n        BOOST_REQUIRE_GT(ctor_dtor_counter, 0);\n\n        using return_future_type = std::invoke_result_t<F>;\n        using return_type = typename return_future_type::value_type;\n        constexpr bool is_void = std::is_same_v<return_future_type, future<>>;\n\n        std::any ret;\n\n        try {\n            if constexpr (is_void) {\n                if constexpr (CheckPreempt) {\n                    co_await seastar::coroutine::try_future(underlying_func());\n                } else {\n                    co_await seastar::coroutine::try_future_without_preemption_check(underlying_func());\n                }\n            } else {\n                if constexpr (CheckPreempt) {\n                    ret = co_await seastar::coroutine::try_future(underlying_func());\n                } else {\n                    ret = co_await seastar::coroutine::try_future_without_preemption_check(underlying_func());\n                }\n            }\n            run_past = true;\n        } catch (...) {\n            BOOST_FAIL(fmt::format(\"Exception should be handled in try_future, bug caught: {}\", std::current_exception()));\n        }\n\n        if constexpr (is_void) {\n            co_return result_wrapper<seastar::internal::monostate>{};\n        } else {\n            co_return result_wrapper(std::any_cast<return_type>(ret));\n        }\n    };\n\n    return seastar::do_with(func, [] (auto& func) {\n        const auto cxx_exception_before = seastar::engine().cxx_exceptions();\n        return func().then_wrapped([cxx_exception_before] (auto&& fut) {\n            const auto cxx_exception_after = seastar::engine().cxx_exceptions();\n            BOOST_REQUIRE_EQUAL(cxx_exception_before, cxx_exception_after);\n            return std::move(fut);\n        });\n    });\n}\n\ntemplate <bool CheckPreempt, std::invocable<> F>\nfuture<> run_try_future_test(F underlying_func, std::optional<std::any> expected_value, std::source_location sl = std::source_location::current()) {\n    fmt::print(\"running test case at {}:{}\\n\", sl.file_name(), sl.line());\n\n    int ctor_dtor_counter{0};\n    bool run_past{false};\n\n    const bool throws = !expected_value.has_value();\n\n    using return_future_type = std::invoke_result_t<F>;\n    using return_type = typename return_future_type::value_type;\n    constexpr bool is_void = std::is_same_v<return_future_type, future<>>;\n\n    try {\n        if constexpr (is_void) {\n            co_await do_run_try_future_test<CheckPreempt>(std::move(underlying_func), ctor_dtor_counter, run_past);\n            BOOST_REQUIRE(expected_value);\n        } else {\n            auto res = co_await do_run_try_future_test<CheckPreempt>(std::move(underlying_func), ctor_dtor_counter, run_past);\n            BOOST_REQUIRE(expected_value);\n            BOOST_REQUIRE_EQUAL(res.value, std::any_cast<return_type>(*expected_value));\n        }\n    } catch (test_exception&) {\n        BOOST_REQUIRE(throws);\n    } catch (...) {\n        BOOST_FAIL(fmt::format(\"Unexpected exception {}\", std::current_exception()));\n    }\n\n    BOOST_REQUIRE_EQUAL(run_past, !throws);\n    BOOST_REQUIRE_EQUAL(ctor_dtor_counter, 0);\n}\n\nSEASTAR_TEST_CASE(test_try_future) {\n    co_await run_try_future_test<true>(return_void, std::any{});\n    co_await run_try_future_test<false>(return_void, std::any{});\n    co_await run_try_future_test<true>(return_ex_void, std::nullopt);\n    co_await run_try_future_test<false>(return_ex_void, std::nullopt);\n\n    co_await run_try_future_test<true>(return_int, 128);\n    co_await run_try_future_test<false>(return_int, 128);\n    co_await run_try_future_test<true>(return_ex_int, std::nullopt);\n    co_await run_try_future_test<false>(return_ex_int, std::nullopt);\n}\n"
  },
  {
    "path": "tests/unit/defer_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/util/defer.hh>\n\nusing namespace seastar;\n\nBOOST_AUTO_TEST_CASE(test_defer_does_not_run_when_canceled) {\n    bool ran = false;\n    {\n        auto d = defer([&] () noexcept {\n            ran = true;\n        });\n        d.cancel();\n    }\n    BOOST_REQUIRE(!ran);\n}\n\nBOOST_AUTO_TEST_CASE(test_defer_runs) {\n    bool ran = false;\n    {\n        auto d = defer([&] () noexcept {\n            ran = true;\n        });\n    }\n    BOOST_REQUIRE(ran);\n}\n\nBOOST_AUTO_TEST_CASE(test_defer_runs_once_when_moved) {\n    int ran = 0;\n    {\n        auto d = defer([&] () noexcept {\n            ++ran;\n        });\n        {\n            auto d2 = std::move(d);\n        }\n        BOOST_REQUIRE_EQUAL(1, ran);\n    }\n    BOOST_REQUIRE_EQUAL(1, ran);\n}\n\nBOOST_AUTO_TEST_CASE(test_defer_does_not_run_when_moved_after_cancelled) {\n    int ran = 0;\n    {\n        auto d = defer([&] () noexcept {\n            ++ran;\n        });\n        d.cancel();\n        {\n            auto d2 = std::move(d);\n        }\n    }\n    BOOST_REQUIRE_EQUAL(0, ran);\n}\n"
  },
  {
    "path": "tests/unit/deleter_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Lightbits Labs Ltd. - All Rights Reserved\n*/\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/deleter.hh>\n\nusing namespace seastar;\n\nstruct TestObject {\n      TestObject() : has_ref(true){}\n      TestObject(TestObject&& other) {\n          has_ref = true;\n          other.has_ref = false;\n      }\n      ~TestObject() {\n          if (has_ref) {\n              ++deletions_called;\n          }\n      }\n      static int deletions_called;\n      int has_ref;\n};\nint TestObject::deletions_called = 0;\n\nBOOST_AUTO_TEST_CASE(test_deleter_append_does_not_free_shared_object) {\n    {\n        deleter tested;\n        {\n            auto obj1 = TestObject();\n            deleter del1 = make_object_deleter(std::move(obj1));\n            auto obj2 = TestObject();\n            deleter del2 = make_object_deleter(std::move(obj2));\n            del1.append(std::move(del2));\n            tested = del1.share();\n            auto obj3 = TestObject();\n            deleter del3 = make_object_deleter(std::move(obj3));\n            del1.append(std::move(del3));\n        }\n        // since deleter tested still holds references to first two objects, last objec should be deleted\n        BOOST_REQUIRE(TestObject::deletions_called == 1);\n    }\n    BOOST_REQUIRE(TestObject::deletions_called == 3);\n}\n\nBOOST_AUTO_TEST_CASE(test_deleter_append_same_shared_object_twice) {\n    TestObject::deletions_called = 0;\n    {\n        deleter tested;\n        {\n            deleter del1 = make_object_deleter(TestObject());\n            auto del2 = del1.share();\n\n            tested.append(std::move(del1));\n            tested.append(std::move(del2));\n        }\n        BOOST_REQUIRE(TestObject::deletions_called == 0);\n    }\n    BOOST_REQUIRE(TestObject::deletions_called == 1);\n}\n"
  },
  {
    "path": "tests/unit/directory_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/test_runner.hh>\n\n#include <seastar/util/std-compat.hh>\n#ifdef SEASTAR_COROUTINES_ENABLED\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/generator.hh>\n#endif\n#include <seastar/core/reactor.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/file.hh>\n#include <seastar/util/assert.hh>\n\nusing namespace seastar;\n\nconst char* de_type_desc(directory_entry_type t)\n{\n    switch (t) {\n    case directory_entry_type::unknown:\n        return \"unknown\";\n    case directory_entry_type::block_device:\n        return \"block_device\";\n    case directory_entry_type::char_device:\n        return \"char_device\";\n    case directory_entry_type::directory:\n        return \"directory\";\n    case directory_entry_type::fifo:\n        return \"fifo\";\n    case directory_entry_type::link:\n        return \"link\";\n    case directory_entry_type::regular:\n        return \"regular\";\n    case directory_entry_type::socket:\n        return \"socket\";\n    }\n    SEASTAR_ASSERT(0 && \"should not get here\");\n    return nullptr;\n}\n\nSEASTAR_TEST_CASE(test_lister) {\n    class lister {\n        file _f;\n        subscription<directory_entry> _listing;\n    public:\n        lister(file f)\n                : _f(std::move(f))\n                , _listing(_f.list_directory([this] (directory_entry de) { return report(de); })) {\n        }\n        future<> done() { return _listing.done(); }\n    private:\n        future<> report(directory_entry de) {\n            stat_data sd = co_await file_stat(de.name, follow_symlink::no);\n            if (de.type) {\n                SEASTAR_ASSERT(*de.type == sd.type);\n            } else {\n                SEASTAR_ASSERT(sd.type == directory_entry_type::unknown);\n            }\n            fmt::print(\"{} (type={})\\n\", de.name, de_type_desc(sd.type));\n        }\n    };\n    fmt::print(\"--- Regular lister test ---\\n\");\n    file f = co_await open_directory(\".\");\n    lister l(std::move(f));\n    co_await l.done();\n}\n\nfuture<> lister_generator_test(file f) {\n    auto lister = f.experimental_list_directory();\n\n    while (auto de = co_await lister()) {\n        auto& entry = *de;\n        auto sd = co_await file_stat(entry.name, follow_symlink::no);\n        if (entry.type) {\n            SEASTAR_ASSERT(entry.type == sd.type);\n        } else {\n            SEASTAR_ASSERT(sd.type == directory_entry_type::unknown);\n        }\n        fmt::print(\"{} (type={})\\n\", entry.name, de_type_desc(sd.type));\n    }\n    co_await f.close();\n}\n\nclass test_file_impl : public file_impl {\n    file _lower;\npublic:\n    test_file_impl(file&& f) : _lower(std::move(f)) {}\n\n    virtual future<> flush() override { return get_file_impl(_lower)->flush(); }\n    virtual future<struct stat> stat() override { return get_file_impl(_lower)->stat(); }\n    virtual future<struct stat> statat(std::string_view name, int flags) override { return get_file_impl(_lower)->statat(name, flags); }\n    virtual future<> truncate(uint64_t length) override { return get_file_impl(_lower)->truncate(length); }\n    virtual future<> discard(uint64_t offset, uint64_t length) override { return get_file_impl(_lower)->discard(offset, length); }\n    virtual future<> allocate(uint64_t position, uint64_t length) override { return get_file_impl(_lower)->allocate(position, length); }\n    virtual future<uint64_t> size() override { return get_file_impl(_lower)->size(); }\n    virtual future<> close() override { return _lower.close(); }\n    virtual subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)> next) override { return get_file_impl(_lower)->list_directory(std::move(next)); }\n    // ! no override for generator list_directory, so that fallback is used\n\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* i) override { return get_file_impl(_lower)->write_dma(pos, buffer, len, i); }\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* i) override { return get_file_impl(_lower)->write_dma(pos, std::move(iov), i); }\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* i) override { return get_file_impl(_lower)->read_dma(pos, buffer, len, i); }\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* i) override { return get_file_impl(_lower)->read_dma(pos, std::move(iov), i); }\n    virtual future<temporary_buffer<uint8_t>> dma_read_bulk(uint64_t offset, size_t range_size, io_intent* i) override { return get_file_impl(_lower)->dma_read_bulk(offset, range_size, i); }\n};\n\nSEASTAR_TEST_CASE(test_generator_lister) {\n    auto f = co_await engine().open_directory(\".\");\n    co_await lister_generator_test(std::move(f));\n}\n\nSEASTAR_TEST_CASE(test_generator_early_abort) {\n    auto f = co_await engine().open_directory(\".\");\n    {\n        auto lister = f.experimental_list_directory();\n        [[maybe_unused]] auto de = co_await lister();\n    } // destroys the lister object before it reports EOF\n    co_await f.close();\n}\n\nSEASTAR_TEST_CASE(test_generator_fallback) {\n    auto lf = co_await engine().open_directory(\".\");\n    auto tf = ::seastar::make_shared<test_file_impl>(std::move(lf));\n    auto f2 = file(std::move(tf));\n    co_await lister_generator_test(std::move(f2));\n}\n\nSEASTAR_TEST_CASE(test_generator_fallback_early_abort) {\n    auto lf = co_await engine().open_directory(\".\");\n    auto tf = ::seastar::make_shared<test_file_impl>(std::move(lf));\n    auto f = file(std::move(tf));\n    {\n        auto lister = f.experimental_list_directory();\n        [[maybe_unused]] auto de = co_await lister();\n    } // destroys the lister object before it reports EOF\n    co_await f.close();\n}\n\nSEASTAR_TEST_CASE(test_generator_with_statat) {\n    auto f = co_await engine().open_directory(\".\");\n    auto lister = f.experimental_list_directory();\n\n    while (auto de = co_await lister()) {\n        auto& entry = *de;\n        auto sd1 = co_await f.statat(entry.name);\n        auto sd2 = co_await file_stat(f, entry.name, follow_symlink::no);\n        BOOST_REQUIRE_EQUAL(sd1.st_ino, sd2.inode_number);\n\n        auto sd3 = co_await file_stat(entry.name, follow_symlink::no);\n        BOOST_REQUIRE_EQUAL(sd1.st_ino, sd3.inode_number);\n    }\n    co_await f.close();\n}\n"
  },
  {
    "path": "tests/unit/distributed_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/closeable.hh>\n#include <seastar/util/later.hh>\n#include <mutex>\n#include <ranges>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nstruct async_service : public seastar::async_sharded_service<async_service> {\n    thread_local static bool deleted;\n    ~async_service() {\n        deleted = true;\n    }\n    void run() {\n        auto ref = shared_from_this();\n        // Wait a while and check.\n        (void)sleep(std::chrono::milliseconds(100 + 100 * this_shard_id())).then([this, ref] {\n           check();\n        });\n    }\n    virtual void check() {\n        SEASTAR_ASSERT(!deleted);\n    }\n    future<> stop() { return make_ready_future<>(); }\n};\n\nthread_local bool async_service::deleted = false;\n\nstruct X {\n    sstring echo(sstring arg) {\n        return arg;\n    }\n    int cpu_id_squared() const {\n        auto id = this_shard_id();\n        return id * id;\n    }\n    future<> stop() { return make_ready_future<>(); }\n};\n\ntemplate <typename T, typename Func>\nfuture<> do_with_distributed(Func&& func) {\n    auto x = make_shared<sharded<T>>();\n    return func(*x).finally([x] {\n        return x->stop();\n    }).finally([x]{});\n}\n\nSEASTAR_TEST_CASE(test_that_each_core_gets_the_arguments) {\n    return do_with_distributed<X>([] (auto& x) {\n        return x.start().then([&x] {\n            return x.map_reduce([] (sstring msg){\n                if (msg != \"hello\") {\n                    throw std::runtime_error(\"wrong message\");\n                }\n            }, &X::echo, sstring(\"hello\"));\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_functor_version) {\n    return do_with_distributed<X>([] (auto& x) {\n        return x.start().then([&x] {\n            return x.map_reduce([] (sstring msg){\n                if (msg != \"hello\") {\n                    throw std::runtime_error(\"wrong message\");\n                }\n            }, [] (X& x) { return x.echo(\"hello\"); });\n        });\n    });\n}\n\nstruct Y {\n    sstring s;\n    Y(sstring s) : s(std::move(s)) {}\n    future<> stop() { return make_ready_future<>(); }\n};\n\nSEASTAR_TEST_CASE(test_constructor_argument_is_passed_to_each_core) {\n    return do_with_distributed<Y>([] (auto& y) {\n        return y.start(sstring(\"hello\")).then([&y] {\n            return y.invoke_on_all([] (Y& y) {\n                if (y.s != \"hello\") {\n                    throw std::runtime_error(format(\"expected message mismatch, is \\\"%s\\\"\", y.s));\n                }\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce) {\n    return do_with_distributed<X>([] (sharded<X>& x) {\n        return x.start().then([&x] {\n            return x.map_reduce0(std::mem_fn(&X::cpu_id_squared),\n                                 0,\n                                 std::plus<int>()).then([] (int result) {\n                int n = smp::count - 1;\n                if (result != (n * (n + 1) * (2*n + 1)) / 6) {\n                    throw std::runtime_error(\"map_reduce failed\");\n                }\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(const X& x) {\n            return yield().then([this, &x] {\n                BOOST_REQUIRE(!destroyed);\n                return x.cpu_id_squared();\n            });\n        }\n    };\n    struct reduce {\n        long& res;\n        bool destroyed = false;\n        reduce(long& result)\n            : res{result} {}\n        reduce(const reduce&) = default;\n        ~reduce() {\n            destroyed = true;\n        }\n        auto operator()(int x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                res += x;\n            });\n        }\n    };\n    return do_with_distributed<X>([] (sharded<X>& x) {\n        return x.start().then([&x] {\n            return do_with(0L, [&x] (auto& result) {\n                return x.map_reduce(reduce{result}, map{}).then([&result] {\n                    long n = smp::count - 1;\n                    long expected = (n * (n + 1) * (2*n + 1)) / 6;\n                    BOOST_REQUIRE_EQUAL(result, expected);\n                });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce0_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(const X& x) const {\n            return yield().then([this, &x] {\n                BOOST_REQUIRE(!destroyed);\n                return x.cpu_id_squared();\n            });\n        }\n    };\n    struct reduce {\n        bool destroyed = false;\n        reduce() = default;\n        reduce(const reduce&) = default;\n        ~reduce() {\n            destroyed = true;\n        }\n        auto operator()(long res, int x) {\n            BOOST_REQUIRE(!destroyed);\n            return res + x;\n        }\n    };\n    return do_with_distributed<X>([] (sharded<X>& x) {\n        return x.start().then([&x] {\n            return x.map_reduce0(map{}, 0L, reduce{}).then([] (long result) {\n                long n = smp::count - 1;\n                long expected = (n * (n + 1) * (2*n + 1)) / 6;\n                BOOST_REQUIRE_EQUAL(result, expected);\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(const X& x) const {\n            return yield().then([this, &x] {\n                BOOST_REQUIRE(!destroyed);\n                return x.cpu_id_squared();\n            });\n        }\n    };\n    return do_with_distributed<X>([] (sharded<X>& x) {\n        return x.start().then([&x] {\n            return x.map(map{}).then([] (std::vector<int> result) {\n                BOOST_REQUIRE_EQUAL(result.size(), smp::count);\n                for (size_t i = 0; i < (size_t)smp::count; i++) {\n                    BOOST_REQUIRE_EQUAL(result[i], i * i);\n                }\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_async) {\n    return do_with_distributed<async_service>([] (sharded<async_service>& x) {\n        return x.start().then([&x] {\n            return x.invoke_on_all(&async_service::run);\n        });\n    }).then([] {\n        return sleep(std::chrono::milliseconds(100 * (smp::count + 1)));\n    });\n}\n\nSEASTAR_TEST_CASE(test_invoke_on_others) {\n    return seastar::async([] {\n        struct my_service {\n            int counter = 0;\n            void up() { ++counter; }\n            future<> stop() { return make_ready_future<>(); }\n        };\n        for (unsigned c = 0; c < smp::count; ++c) {\n            smp::submit_to(c, [c] {\n                return seastar::async([c] {\n                    sharded<my_service> s;\n                    s.start().get();\n                    s.invoke_on_others([](auto& s) { s.up(); }).get();\n                    if (s.local().counter != 0) {\n                        throw std::runtime_error(\"local modified\");\n                    }\n                    s.invoke_on_all([c](auto& remote) {\n                        if (this_shard_id() != c) {\n                            if (remote.counter != 1) {\n                                throw std::runtime_error(\"remote not modified\");\n                            }\n                        }\n                    }).get();\n                    s.stop().get();\n                });\n            }).get();\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_smp_invoke_on_others) {\n    return seastar::async([] {\n        std::vector<std::vector<int>> calls;\n        calls.reserve(smp::count);\n        for (unsigned i = 0; i < smp::count; i++) {\n            auto& sv = calls.emplace_back();\n            sv.reserve(smp::count);\n        }\n\n        smp::invoke_on_all([&calls] {\n            return smp::invoke_on_others([&calls, from = this_shard_id()] {\n                calls[this_shard_id()].emplace_back(from);\n            });\n        }).get();\n\n        for (unsigned i = 0; i < smp::count; i++) {\n            BOOST_REQUIRE_EQUAL(calls[i].size(), smp::count - 1);\n            for (unsigned f = 0; f < smp::count; f++) {\n                auto r = std::find(calls[i].begin(), calls[i].end(), f);\n                BOOST_REQUIRE_EQUAL(r == calls[i].end(), i == f);\n            }\n        }\n    });\n}\n\nstruct remote_worker {\n    unsigned current = 0;\n    unsigned max_concurrent_observed = 0;\n    unsigned expected_max;\n    semaphore sem{0};\n    remote_worker(unsigned expected_max) : expected_max(expected_max) {\n    }\n    future<> do_work() {\n        ++current;\n        max_concurrent_observed = std::max(current, max_concurrent_observed);\n        if (max_concurrent_observed >= expected_max && sem.current() == 0) {\n            sem.signal(semaphore::max_counter());\n        }\n        return sem.wait().then([this] {\n            // Sleep a bit to check if the concurrency goes over the max\n            return sleep(100ms).then([this] {\n                max_concurrent_observed = std::max(current, max_concurrent_observed);\n                --current;\n            });\n        });\n    }\n    future<> do_remote_work(shard_id t, smp_service_group ssg) {\n        return smp::submit_to(t,  ssg, [this] {\n            return do_work();\n        });\n    }\n};\n\nSEASTAR_TEST_CASE(test_smp_service_groups) {\n    return async([] {\n        smp_service_group_config ssgc1;\n        ssgc1.max_nonlocal_requests = 1;\n        auto ssg1 = create_smp_service_group(ssgc1).get();\n        smp_service_group_config ssgc2;\n        ssgc2.max_nonlocal_requests = 1000;\n        auto ssg2 = create_smp_service_group(ssgc2).get();\n        shard_id other_shard = smp::count - 1;\n        remote_worker rm1(1);\n        remote_worker rm2(1000);\n        auto bunch1 = parallel_for_each(std::views::iota(0, 20), [&] (int ignore) { return rm1.do_remote_work(other_shard, ssg1); });\n        auto bunch2 = parallel_for_each(std::views::iota(0, 2000), [&] (int ignore) { return rm2.do_remote_work(other_shard, ssg2); });\n        bunch1.get();\n        bunch2.get();\n        if (smp::count > 1) {\n            SEASTAR_ASSERT(rm1.max_concurrent_observed == 1);\n            SEASTAR_ASSERT(rm2.max_concurrent_observed == 1000);\n        }\n        destroy_smp_service_group(ssg1).get();\n        destroy_smp_service_group(ssg2).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_smp_service_groups_re_construction) {\n    // During development of the feature, we saw a bug where the vector\n    // holding the groups did not expand correctly. This test triggers the\n    // bug.\n    return async([] {\n        auto ssg1 = create_smp_service_group({}).get();\n        auto ssg2 = create_smp_service_group({}).get();\n        destroy_smp_service_group(ssg1).get();\n        auto ssg3 = create_smp_service_group({}).get();\n        destroy_smp_service_group(ssg2).get();\n        destroy_smp_service_group(ssg3).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_smp_timeout) {\n    return async([] {\n        smp_service_group_config ssgc1;\n        ssgc1.max_nonlocal_requests = 1;\n        auto ssg1 = create_smp_service_group(ssgc1).get();\n\n        auto _ = defer([ssg1] () noexcept {\n            destroy_smp_service_group(ssg1).get();\n        });\n\n        const shard_id other_shard = smp::count - 1;\n\n        // Ugly but beats using sleeps.\n        std::mutex mut;\n        std::unique_lock<std::mutex> lk(mut);\n\n        // Submitted to the remote shard.\n        auto fut1 = smp::submit_to(other_shard, ssg1, [&mut] {\n            std::cout << \"Running request no. 1\" << std::endl;\n            std::unique_lock<std::mutex> lk(mut);\n            std::cout << \"Request no. 1 done\" << std::endl;\n        });\n        // Consume the only unit from the semaphore.\n        auto fut2 = smp::submit_to(other_shard, ssg1, [] {\n            std::cout << \"Running request no. 2 - done\" << std::endl;\n        });\n\n        auto fut_timedout = smp::submit_to(other_shard, smp_submit_to_options(ssg1, smp_timeout_clock::now() + 10ms), [] {\n            std::cout << \"Running timed-out request - done\" << std::endl;\n        });\n\n        {\n            auto notify = defer([lk = std::move(lk)] () noexcept { });\n\n            try {\n                fut_timedout.get();\n                throw std::runtime_error(\"smp::submit_to() didn't timeout as expected\");\n            } catch (semaphore_timed_out& e) {\n                std::cout << \"Expected timeout received: \" << e.what() << std::endl;\n            } catch (...) {\n                std::throw_with_nested(std::runtime_error(\"smp::submit_to() failed with unexpected exception\"));\n            }\n        }\n\n        fut1.get();\n        fut2.get();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_sharded_parameter) {\n    struct dependency {\n        unsigned val = this_shard_id() * 7;\n    };\n    struct some_service {\n        bool ok = false;\n        some_service(unsigned non_shard_dependent, unsigned shard_dependent, dependency& dep, unsigned shard_dependent_2) {\n            ok =\n                    non_shard_dependent == 43\n                    && shard_dependent == this_shard_id() * 3\n                    && dep.val == this_shard_id() * 7\n                    && shard_dependent_2 == -dep.val;\n        }\n    };\n    sharded<dependency> s_dep;\n    s_dep.start().get();\n    auto undo1 = deferred_stop(s_dep);\n\n    sharded<some_service> s_service;\n    s_service.start(\n            43, // should be copied verbatim\n            sharded_parameter([] { return this_shard_id() * 3; }),\n            std::ref(s_dep),\n            sharded_parameter([] (dependency& d) { return -d.val; }, std::ref(s_dep))\n            ).get();\n    auto undo2 = deferred_stop(s_service);\n\n    auto all_ok = s_service.map_reduce0(std::mem_fn(&some_service::ok), true, std::multiplies<>()).get();\n    BOOST_REQUIRE(all_ok);\n}\n"
  },
  {
    "path": "tests/unit/dns_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n#include <vector>\n#include <algorithm>\n\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/net/inet_address.hh>\n\nusing namespace seastar;\nusing namespace seastar::net;\n\nstatic const sstring seastar_name = \"seastar.io\";\n\nstatic future<> test_resolve(dns_resolver::options opts) {\n    auto d = dns_resolver(std::move(opts));\n\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\n    for (auto hostname : {\"seastar.io\", \"scylladb.com\", \"kernel.org\", \"www.google.com\"}) {\n        hostent e = co_await d.get_host_by_name(hostname, inet_address::family::INET);\n        BOOST_REQUIRE_EQUAL(e.addr_list.size(), e.addr_entries.size());\n        for (auto i = 0ul; i < e.addr_entries.size(); ++i) {\n            BOOST_REQUIRE_EQUAL(e.addr_entries[i].addr, e.addr_list[i]);\n            BOOST_REQUIRE(e.addr_entries[i].ttl.count() != 0);\n        }\n\n        hostent a;\n        try {\n            a = co_await d.get_host_by_addr(e.addr_entries.front().addr);\n        } catch (const std::system_error& e) {\n            if (e.code().category() != dns::error_category()) {\n                throw;\n            }\n            continue;\n        }\n        hostent e2 = co_await d.get_host_by_name(a.names.front(), inet_address::family::INET);\n        BOOST_REQUIRE(std::count(e2.addr_list.begin(), e2.addr_list.end(), e.addr_list.front()));\n        BOOST_REQUIRE(std::count_if(e2.addr_entries.begin(), e2.addr_entries.end(), [&e](const auto& item){return e.addr_entries.front().addr == item.addr;}));\n        BOOST_REQUIRE(!e2.addr_entries.empty());\n        BOOST_REQUIRE(e2.addr_entries[0].ttl.count() != 0);\n        co_await d.close();\n        co_return;\n    }\n#pragma GCC diagnostic pop\n    BOOST_FAIL(\"No more hosts to try\");\n}\n\nstatic future<> test_bad_name(dns_resolver::options opts) {\n    auto d = ::make_lw_shared<dns_resolver>(std::move(opts));\n    return d->get_host_by_name(\"apa.ninja.gnu\", inet_address::family::INET).then_wrapped([d](future<hostent> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should not succeed\");\n        } catch (...) {\n            // ok.\n        }\n    }).finally([d]{\n        return d->close();\n    });\n}\n\nusing enable_if_with_networking = boost::unit_test::enable_if<SEASTAR_TESTING_WITH_NETWORKING>;\n\nSEASTAR_TEST_CASE(test_resolve_numeric,\n                  *enable_if_with_networking()) {\n    auto d = ::make_lw_shared<dns_resolver>(engine().net(), dns_resolver::options());\n    return d->get_host_by_name(\"127.0.0.1\").then_wrapped([d](future<hostent> f) {\n        auto ent = f.get();\n        BOOST_REQUIRE_EQUAL(ent.addr_entries.size(), 1);\n        BOOST_REQUIRE_EQUAL(ent.addr_entries[0].ttl.count(), std::numeric_limits<signed int>::max());\n    }).finally([d]{\n        return d->close();\n    });\n}\n\nSEASTAR_TEST_CASE(test_resolve_udp,\n                  *enable_if_with_networking()) {\n    return test_resolve(dns_resolver::options());\n}\n\nSEASTAR_TEST_CASE(test_bad_name_udp,\n                  *enable_if_with_networking()) {\n    return test_bad_name(dns_resolver::options());\n}\n\nSEASTAR_TEST_CASE(test_timeout_udp,\n                  *enable_if_with_networking()) {\n    dns_resolver::options opts;\n    opts.servers = std::vector<inet_address>({ inet_address(\"1.2.3.4\") }); // not a server\n    opts.udp_port = 29953; // not a dns port\n    opts.timeout = std::chrono::milliseconds(500);\n\n    auto d = ::make_lw_shared<dns_resolver>(engine().net(), opts);\n    return d->get_host_by_name(seastar_name, inet_address::family::INET).then_wrapped([d](future<hostent> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should not succeed\");\n        } catch (...) {\n            // ok.\n        }\n    }).finally([d]{\n        return d->close();\n    });\n}\n\n// NOTE: cannot really test timeout in TCP mode, because seastar sockets do not support\n// connect with timeout -> cannot complete connect future in dns::do_connect in reasonable\n// time.\n\n// But we can test for connection refused working as expected.\nSEASTAR_TEST_CASE(test_connection_refused_tcp) {\n    dns_resolver::options opts;\n    opts.servers = std::vector<inet_address>({ inet_address(\"127.0.0.1\") });\n    opts.use_tcp_query = true;\n    opts.tcp_port = 29953; // not a dns port\n\n    auto d = ::make_lw_shared<dns_resolver>(engine().net(), opts);\n    return d->get_host_by_name(seastar_name, inet_address::family::INET).then_wrapped([d](future<hostent> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should not succeed\");\n        } catch (...) {\n            // ok.\n        }\n    }).finally([d]{\n        return d->close();\n    });\n}\n\nSEASTAR_TEST_CASE(test_resolve_tcp,\n                  *enable_if_with_networking()) {\n    dns_resolver::options opts;\n    opts.use_tcp_query = true;\n    return test_resolve(opts);\n}\n\nSEASTAR_TEST_CASE(test_bad_name_tcp,\n                  *enable_if_with_networking()) {\n    dns_resolver::options opts;\n    opts.use_tcp_query = true;\n    return test_bad_name(opts);\n}\n\nstatic const sstring imaps_service = \"imaps\";\nstatic const sstring gmail_domain = \"gmail.com\";\n\nstatic future<> test_srv() {\n    auto d = ::make_lw_shared<dns_resolver>();\n    return d->get_srv_records(dns_resolver::srv_proto::tcp,\n                              imaps_service,\n                              gmail_domain).then([d](dns_resolver::srv_records records) {\n        BOOST_REQUIRE(!records.empty());\n        for (auto& record : records) {\n            // record.target should end with \"gmail.com\"\n            BOOST_REQUIRE_GT(record.target.size(), gmail_domain.size());\n            BOOST_REQUIRE_EQUAL(record.target.compare(record.target.size() - gmail_domain.size(),\n                                                      gmail_domain.size(),\n                                                      gmail_domain),\n                                0);\n        }\n    }).finally([d]{\n        return d->close();\n    });\n}\n\nSEASTAR_TEST_CASE(test_srv_tcp,\n                  *enable_if_with_networking()) {\n    return test_srv();\n}\n\n\nSEASTAR_TEST_CASE(test_parallel_resolve_name,\n                  *enable_if_with_networking()) {\n    dns_resolver::options opts;\n    opts.use_tcp_query = true;\n\n    auto d = ::make_lw_shared<dns_resolver>(std::move(opts));\n    return when_all(\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\")\n    ).finally([d](auto&&...) {}).discard_result();\n}\n\nSEASTAR_TEST_CASE(test_parallel_resolve_name_udp,\n                  *enable_if_with_networking()) {\n    dns_resolver::options opts;\n\n    auto d = ::make_lw_shared<dns_resolver>(std::move(opts));\n    return when_all(\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\"),\n        d->resolve_name(\"www.google.com\")\n    ).finally([d](auto&...) {}).discard_result();\n}\n"
  },
  {
    "path": "tests/unit/epoll_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2026 Redpanda Data\n */\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/net/api.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <sys/socket.h>\n\nusing namespace seastar;\n\n// Tests the issue described and fixed by\n// https://github.com/scylladb/seastar/pull/1945 /\n// 25e2487da55fb37d39ecd72e68c8e9d87f784c62 and to a degree also\n// https://github.com/scylladb/seastar/pull/3204\n//\n// Sockets which errored out via EPOLLERR or EPOLLHUP would cause busy spinning\n// in the reactor until eventually closed.\nSEASTAR_THREAD_TEST_CASE(epoll_busy_spin_on_socket_error_test) {\n    auto start_busy_time = engine().total_busy_time();\n\n    listen_options lo;\n    lo.reuse_address = true;\n    server_socket ss = seastar::listen(ipv4_addr(0), lo);\n\n    auto server = seastar::async([&] {\n        accept_result acc = ss.accept().get();\n        auto in = acc.connection.input();\n\n        auto buf = in.read().get();\n\n        // Now sleep, client will have sent RST in the meantime\n        // Reactor will busy spin during this time.\n        seastar::sleep(std::chrono::seconds(3)).get();\n\n        in.close().get();\n    });\n\n    auto client = seastar::async([&] {\n        connected_socket socket = connect(ss.local_address()).get();\n        auto out = socket.output();\n\n        out.write(\"hello world\").get();\n        out.flush().get();\n\n        // 0 linger forces RST on close\n        linger linger_opt{};\n        linger_opt.l_onoff = 1;\n        linger_opt.l_linger = 0;\n        socket.set_sockopt(SOL_SOCKET, SO_LINGER, &linger_opt, sizeof(linger_opt));\n\n        out.close().get();\n    });\n\n    when_all(std::move(server), std::move(client)).get();\n\n    auto end_busy_time = engine().total_busy_time();\n    auto busy_duration = end_busy_time - start_busy_time;\n    auto busy_ms = std::chrono::duration_cast<std::chrono::milliseconds>(busy_duration).count();\n\n    // Expectation is that we would busy spin for 3 seconds, we check that we\n    // are below 1.5 seconds to avoid any kind of flakiness.\n    BOOST_REQUIRE_LT(busy_ms, 1500);\n}\n"
  },
  {
    "path": "tests/unit/exception_logging_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#include <exception>\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/util/log.hh>\n#include <seastar/util/backtrace.hh>\n#include <ostream>\n#include <regex>\n\n\nusing namespace seastar;\n\n// a class which is not derived from std::exception\n// to play the part of the unknown object in the logging\n// function.\nclass unknown_obj {\n    sstring _message;\npublic:\n    unknown_obj(std::string message) : _message(message) {}\n};\n\n// This functions generates an exception chain nesting_level+1 deep\n// for each nesting level it throws one of two types of objects, an\n// unknown (non std::exception) object or a runtime error which is\n// derived from std::exception, it chooses the type of thrown object\n// according to the bit in the `nesting_level` place in the\n// `tests_instance` paramter or in other words according to:\n// bool(test_instance & (1<<nesting_level))\nvoid exception_generator(uint32_t test_instance, int nesting_level) {\n    try {\n        if (nesting_level > 0) {\n            exception_generator(test_instance>>1, nesting_level-1);\n        }\n    } catch(...) {\n        auto msg = fmt::format(\"Exception Level {}\", nesting_level);\n        if(test_instance&1) {\n            // Throw a non std::exception derived type\n            std::throw_with_nested(unknown_obj(msg));\n        } else {\n            std::throw_with_nested(std::runtime_error(msg));\n        }\n    }\n    if (nesting_level == 0) {\n        if (test_instance & 1) {\n            throw unknown_obj(fmt::format(\"Exception Level {}\", nesting_level));\n        } else {\n            throw std::runtime_error(fmt::format(\"Exception Level {}\", nesting_level));\n        }\n    }\n}\n\n// This function generates the expected logging output string of an exception\n// thrown by the  exception generator function with a specific output.\nstd::string exception_generator_str(uint32_t test_instance,int nesting_level) {\n    std::ostringstream ret;\n    const std::string runtime_err_str = \"std::runtime_error\";\n    const std::string unknown_obj_str = \"unknown_obj\";\n    const std::string nested_exception_with_unknown_obj_str = \"std::_Nested_exception<unknown_obj>\";\n    const std::string nested_exception_with_runtime_err_str = \"std::_Nested_exception<std::runtime_error>\";\n\n    for(; nesting_level > 0; nesting_level--) {\n        if (test_instance & 1) {\n            fmt::print(ret, \"{}\", nested_exception_with_unknown_obj_str);\n        } else {\n            fmt::print(ret, \"{} (Exception Level {})\", nested_exception_with_runtime_err_str,\n                       nesting_level);\n        }\n        ret << \": \";\n        test_instance >>= 1;\n    }\n\n\n    if (test_instance & 1) {\n        fmt::print(ret, \"{}\", unknown_obj_str);\n    } else {\n        fmt::print(ret, \"{} (Exception Level {})\", runtime_err_str, nesting_level);\n    }\n    return ret.str();\n}\n\n// Test all variations of nested exceptions of some\n// depth\nBOOST_AUTO_TEST_CASE(nested_exception_logging1) {\n\n    constexpr int levels_to_test = 3;\n\n    for(int level = 0; level < levels_to_test; level++) {\n        for(int inst = (1 << (level + 1)) - 1; inst >= 0; inst--) {\n            std::ostringstream log_msg;\n            try {\n                exception_generator(inst, level);\n            } catch(...) {\n                log_msg << std::current_exception();\n            }\n            BOOST_REQUIRE_EQUAL(log_msg.str(), exception_generator_str(inst, level));\n        }\n    }\n}\n\n// Test logging of nested exception not mixed in with anything\nBOOST_AUTO_TEST_CASE(nested_exception_logging2) {\n    std::ostringstream log_msg;\n    try {\n        throw std::nested_exception();\n    } catch(...) {\n        log_msg << std::current_exception();\n    }\n\n    BOOST_REQUIRE_EQUAL(log_msg.str(), std::string(\"std::nested_exception: <no exception>\"));\n}\n\nclass very_important_exception : public std::exception {\n    const char* my_name = \"very important information\";\n\npublic:\n    const char* what() const noexcept {\n        return my_name;\n    }\n};\n\n// Test logging of nested exception that have std::system_error mixed with other exceptions\n// so that std::system_error is in the middle of the exception chain.\nBOOST_AUTO_TEST_CASE(nested_exception_logging3) {\n    std::ostringstream log_msg;\n\n    try {\n        throw very_important_exception();\n        } catch (...) {\n        try {\n            std::throw_with_nested(std::system_error(1, std::generic_category(), \"my error\"));\n        } catch (...) {\n            try {\n                std::throw_with_nested(unknown_obj(\"This is an unknown object\"));\n            } catch (...) {\n                log_msg << std::current_exception();\n            }\n        }\n    }\n\n    std::string expected_string(\"std::_Nested_exception<unknown_obj>: std::_Nested_exception<std::system_error> (error generic:1, my error: Operation not permitted): very_important_exception (very important information)\");\n\n    BOOST_REQUIRE_EQUAL(log_msg.str(), expected_string);\n}\n\nBOOST_AUTO_TEST_CASE(unknown_object_thrown_test) {\n    std::ostringstream log_msg;\n    try {\n        throw unknown_obj(\"This is an unknown object\");\n    } catch(...) {\n        log_msg << std::current_exception();\n    }\n\n    BOOST_REQUIRE_EQUAL(log_msg.str(), std::string(\"unknown_obj\"));\n\n}\n\nBOOST_AUTO_TEST_CASE(format_error_test) {\n    static seastar::logger l(\"format_error_test\");\n\n    std::ostringstream log_msg;\n    l.set_ostream(log_msg);\n\n    const char* fmt = \"bad format string: {}\";\n#ifdef SEASTAR_LOGGER_COMPILE_TIME_FMT\n    // {fmt} v8.0 and up comes with compile-time format string checking, so\n    // malformed format_string passed to `logger.error(format_string, args)`\n    // can be identified at compile time. but a runtime variable passed to\n    // `logger.error(msg)` cannot be considered as a format string anymore\n    // when compiled with {fmt} v8.0 and up. so we have to test with a runtime\n    // format string here\n    l.error(fmt::runtime(fmt));\n#else\n    l.error(fmt);\n#endif\n\n    BOOST_TEST_MESSAGE(log_msg.str());\n    BOOST_REQUIRE_NE(log_msg.str().find(__builtin_FILE()), std::string::npos);\n    BOOST_REQUIRE_NE(log_msg.str().find(__builtin_FUNCTION()), std::string::npos);\n    BOOST_REQUIRE_NE(log_msg.str().find(fmt), std::string::npos);\n}\n\nBOOST_AUTO_TEST_CASE(throw_with_backtrace_exception_logging) {\n    std::ostringstream log_msg;\n    try {\n        throw_with_backtrace<std::runtime_error>(\"throw_with_backtrace_exception_logging\");\n    } catch(...) {\n        log_msg << std::current_exception();\n    }\n\n#ifndef SEASTAR_BACKTRACE_UNIMPLEMENTED\n    auto regex_str = \"backtraced<std::runtime_error> \\\\(throw_with_backtrace_exception_logging Backtrace:(\\\\s+(\\\\S+\\\\+)?0x[0-9a-f]+)+\\\\)\";\n    std::regex expected_msg_re(regex_str, std::regex_constants::ECMAScript | std::regex_constants::icase);\n    BOOST_REQUIRE(std::regex_search(log_msg.str(), expected_msg_re));\n#endif\n}\n\nBOOST_AUTO_TEST_CASE(throw_with_backtrace_nested_exception_logging) {\n    std::ostringstream log_msg;\n    try {\n        throw_with_backtrace<std::runtime_error>(\"outer\");\n    } catch(...) {\n        try {\n            std::throw_with_nested(unknown_obj(\"This is an unknown object\"));\n        } catch (...) {\n            log_msg << std::current_exception();\n        }\n    }\n\n#ifndef SEASTAR_BACKTRACE_UNIMPLEMENTED\n    auto regex_str = \"std::_Nested_exception<unknown_obj>.*backtraced<std::runtime_error> \\\\(outer Backtrace:(\\\\s+(\\\\S+\\\\+)?0x[0-9a-f]+)+\\\\)\";\n    std::regex expected_msg_re(regex_str, std::regex_constants::ECMAScript | std::regex_constants::icase);\n    BOOST_REQUIRE(std::regex_search(log_msg.str(), expected_msg_re));\n#endif\n}\n\nBOOST_AUTO_TEST_CASE(throw_with_backtrace_seastar_nested_exception_logging) {\n    std::ostringstream log_msg;\n    try {\n        throw unknown_obj(\"This is an unknown object\");\n    } catch (...) {\n        auto outer = std::current_exception();\n        try {\n            throw_with_backtrace<std::runtime_error>(\"inner\");\n        } catch (...) {\n            auto inner = std::current_exception();\n            try {\n                throw seastar::nested_exception(std::move(inner), std::move(outer));\n            } catch (...) {\n                log_msg << std::current_exception();\n            }\n        }\n    }\n\n#ifndef SEASTAR_BACKTRACE_UNIMPLEMENTED\n    auto regex_str = \"seastar::nested_exception:.*backtraced<std::runtime_error> \\\\(inner Backtrace:(\\\\s+(\\\\S+\\\\+)?0x[0-9a-f]+)+\\\\)\"\n            \" \\\\(while cleaning up after unknown_obj\\\\)\";\n    std::regex expected_msg_re(regex_str, std::regex_constants::ECMAScript | std::regex_constants::icase);\n    BOOST_REQUIRE(std::regex_search(log_msg.str(), expected_msg_re));\n#endif\n}\n\nBOOST_AUTO_TEST_CASE(double_throw_with_backtrace_seastar_nested_exception_logging) {\n    std::ostringstream log_msg;\n    try {\n        throw_with_backtrace<std::runtime_error>(\"outer\");\n    } catch (...) {\n        auto outer = std::current_exception();\n        try {\n            throw_with_backtrace<std::runtime_error>(\"inner\");\n        } catch (...) {\n            auto inner = std::current_exception();\n            try {\n                throw seastar::nested_exception(std::move(inner), std::move(outer));\n            } catch (...) {\n                log_msg << std::current_exception();\n            }\n        }\n    }\n\n#ifndef SEASTAR_BACKTRACE_UNIMPLEMENTED\n    auto regex_str = \"seastar::nested_exception:.*backtraced<std::runtime_error> \\\\(inner Backtrace:(\\\\s+(\\\\S+\\\\+)?0x[0-9a-f]+)+\\\\)\"\n            \" \\\\(while cleaning up after .*backtraced<std::runtime_error> \\\\(outer Backtrace:(\\\\s+(\\\\S+\\\\+)?0x[0-9a-f]+)+\\\\)\\\\)\";\n    std::regex expected_msg_re(regex_str, std::regex_constants::ECMAScript | std::regex_constants::icase);\n    BOOST_REQUIRE(std::regex_search(log_msg.str(), expected_msg_re));\n#endif\n}\n"
  },
  {
    "path": "tests/unit/execution_stage_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#include <algorithm>\n#include <vector>\n#include <chrono>\n\n#include <seastar/core/thread.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/defer.hh>\n\nusing namespace std::chrono_literals;\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(test_create_stage_from_lvalue_function_object) {\n    return seastar::async([] {\n        auto dont_move = [obj = make_shared<int>(53)] { return *obj; };\n        auto stage = seastar::make_execution_stage(\"test\", dont_move);\n        BOOST_REQUIRE_EQUAL(stage().get(), 53);\n        BOOST_REQUIRE_EQUAL(dont_move(), 53);\n    });\n}\n\nSEASTAR_TEST_CASE(test_create_stage_from_rvalue_function_object) {\n    return seastar::async([] {\n        auto dont_copy = [obj = std::make_unique<int>(42)] { return *obj; };\n        auto stage = seastar::make_execution_stage(\"test\", std::move(dont_copy));\n        BOOST_REQUIRE_EQUAL(stage().get(), 42);\n    });\n}\n\nint func() {\n    return 64;\n}\n\nSEASTAR_TEST_CASE(test_create_stage_from_function) {\n    return seastar::async([] {\n        auto stage = seastar::make_execution_stage(\"test\", func);\n        BOOST_REQUIRE_EQUAL(stage().get(), 64);\n    });\n}\n\ntemplate<typename Function, typename Verify>\nvoid test_simple_execution_stage(Function&& func, Verify&& verify) {\n    auto stage = seastar::make_execution_stage(\"test\", std::forward<Function>(func));\n\n    std::vector<int> vs;\n    std::default_random_engine& gen = testing::local_random_engine;\n    std::uniform_int_distribution<> dist(0, 100'000);\n    std::generate_n(std::back_inserter(vs), 1'000, [&] { return dist(gen); });\n\n    std::vector<future<int>> fs;\n    for (auto v : vs) {\n        fs.emplace_back(stage(v));\n    }\n\n    for (auto i = 0u; i < fs.size(); i++) {\n        verify(vs[i], std::move(fs[i]));\n    }\n}\n\nSEASTAR_TEST_CASE(test_simple_stage_returning_int) {\n    return seastar::async([] {\n        test_simple_execution_stage([] (int x) {\n            if (x % 2) {\n                return x * 2;\n            } else {\n                throw x;\n            }\n        }, [] (int original, future<int> result) {\n            if (original % 2) {\n                BOOST_REQUIRE_EQUAL(original * 2, result.get());\n            } else {\n                BOOST_REQUIRE_EXCEPTION(result.get(), int, [&] (int v) { return original == v; });\n            }\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_simple_stage_returning_future_int) {\n    return seastar::async([] {\n        test_simple_execution_stage([] (int x) {\n            if (x % 2) {\n                return make_ready_future<int>(x * 2);\n            } else {\n                return make_exception_future<int>(x);\n            }\n        }, [] (int original, future<int> result) {\n            if (original % 2) {\n                BOOST_REQUIRE_EQUAL(original * 2, result.get());\n            } else {\n                BOOST_REQUIRE_EXCEPTION(result.get(), int, [&] (int v) { return original == v; });\n            }\n        });\n    });\n}\n\ntemplate<typename T>\nvoid test_execution_stage_avoids_copy() {\n    auto stage = seastar::make_execution_stage(\"test\", [] (T obj) {\n        return std::move(obj);\n    });\n\n    auto f = stage(T());\n    T obj = f.get();\n    (void)obj;\n}\n\nSEASTAR_TEST_CASE(test_stage_moves_when_cannot_copy) {\n    return seastar::async([] {\n        struct noncopyable_but_movable {\n            noncopyable_but_movable() = default;\n            noncopyable_but_movable(const noncopyable_but_movable&) = delete;\n            noncopyable_but_movable(noncopyable_but_movable&&) = default;\n        };\n\n        test_execution_stage_avoids_copy<noncopyable_but_movable>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_stage_prefers_move_to_copy) {\n    return seastar::async([] {\n        struct copyable_and_movable {\n            copyable_and_movable() = default;\n            copyable_and_movable(const copyable_and_movable&) {\n                BOOST_FAIL(\"should not copy\");\n            }\n            copyable_and_movable(copyable_and_movable&&) = default;\n        };\n\n        test_execution_stage_avoids_copy<copyable_and_movable>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_rref_decays_to_value) {\n    return seastar::async([] {\n        auto stage = seastar::make_execution_stage(\"test\", [] (std::vector<int>&& vec) {\n            return vec.size();\n        });\n\n        std::vector<int> tmp;\n        std::vector<future<size_t>> fs;\n        for (auto i = 0; i < 100; i++) {\n            tmp.resize(i);\n            fs.emplace_back(stage(std::move(tmp)));\n            tmp = std::vector<int>();\n        }\n\n        for (size_t i = 0; i < 100; i++) {\n            BOOST_REQUIRE_EQUAL(fs[i].get(), i);\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_lref_does_not_decay) {\n    return seastar::async([] {\n        auto stage = seastar::make_execution_stage(\"test\", [] (int& v) {\n            v++;\n        });\n\n        int value = 0;\n        std::vector<future<>> fs;\n        for (auto i = 0; i < 100; i++) {\n            //fs.emplace_back(stage(value)); // should fail to compile\n            fs.emplace_back(stage(seastar::ref(value)));\n        }\n\n        for (auto&& f : fs) {\n            f.get();\n        }\n        BOOST_REQUIRE_EQUAL(value, 100);\n    });\n}\n\nSEASTAR_TEST_CASE(test_explicit_reference_wrapper_is_not_unwrapped) {\n    return seastar::async([] {\n        auto stage = seastar::make_execution_stage(\"test\", [] (seastar::reference_wrapper<int> v) {\n            v.get()++;\n        });\n\n        int value = 0;\n        std::vector<future<>> fs;\n        for (auto i = 0; i < 100; i++) {\n            //fs.emplace_back(stage(value)); // should fail to compile\n            fs.emplace_back(stage(seastar::ref(value)));\n        }\n\n        for (auto&& f : fs) {\n            f.get();\n        }\n        BOOST_REQUIRE_EQUAL(value, 100);\n    });\n}\n\nSEASTAR_TEST_CASE(test_function_is_class_member) {\n    return seastar::async([] {\n        struct foo {\n            int value = -1;\n            int member(int x) {\n                return std::exchange(value, x);\n            }\n        };\n\n        auto stage = seastar::make_execution_stage(\"test\", &foo::member);\n\n        foo object;\n        std::vector<future<int>> fs;\n        for (auto i = 0; i < 100; i++) {\n            fs.emplace_back(stage(&object, i));\n        }\n\n        for (auto i = 0; i < 100; i++) {\n            BOOST_REQUIRE_EQUAL(fs[i].get(), i - 1);\n        }\n        BOOST_REQUIRE_EQUAL(object.value, 99);\n    });\n}\n\nSEASTAR_TEST_CASE(test_function_is_const_class_member) {\n    return seastar::async([] {\n        struct foo {\n            int value = 999;\n            int member() const {\n                return value;\n            }\n        };\n        auto stage = seastar::make_execution_stage(\"test\", &foo::member);\n\n        const foo object;\n        BOOST_REQUIRE_EQUAL(stage(&object).get(), 999);\n    });\n}\n\nSEASTAR_TEST_CASE(test_stage_stats) {\n    return seastar::async([] {\n        auto stage = seastar::make_execution_stage(\"test\", [] { });\n\n        BOOST_REQUIRE_EQUAL(stage.get_stats().function_calls_enqueued, 0u);\n        BOOST_REQUIRE_EQUAL(stage.get_stats().function_calls_executed, 0u);\n\n        auto fs = std::vector<future<>>();\n        static constexpr auto call_count = 53u;\n        for (auto i = 0u; i < call_count; i++) {\n            fs.emplace_back(stage());\n        }\n\n        BOOST_REQUIRE_EQUAL(stage.get_stats().function_calls_enqueued, call_count);\n\n        for (auto i = 0u; i < call_count; i++) {\n            fs[i].get();\n            BOOST_REQUIRE_GE(stage.get_stats().tasks_scheduled, 1u);\n            BOOST_REQUIRE_GE(stage.get_stats().function_calls_executed, i);\n        }\n        BOOST_REQUIRE_EQUAL(stage.get_stats().function_calls_executed, call_count);\n    });\n}\n\nSEASTAR_TEST_CASE(test_unique_stage_names_are_enforced) {\n    return seastar::async([] {\n        {\n            auto stage = seastar::make_execution_stage(\"test\", [] {});\n            BOOST_REQUIRE_THROW(seastar::make_execution_stage(\"test\", [] {}), std::invalid_argument);\n            stage().get();\n        }\n\n        auto stage = seastar::make_execution_stage(\"test\", [] {});\n        stage().get();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_inheriting_concrete_execution_stage) {\n    auto sg1 = seastar::create_scheduling_group(\"sg1\", 300).get();\n    auto ksg1 = seastar::defer([&] () noexcept { seastar::destroy_scheduling_group(sg1).get(); });\n    auto sg2 = seastar::create_scheduling_group(\"sg2\", 100).get();\n    auto ksg2 = seastar::defer([&] () noexcept { seastar::destroy_scheduling_group(sg2).get(); });\n    auto check_sg = [] (seastar::scheduling_group sg) {\n        BOOST_REQUIRE(seastar::current_scheduling_group() == sg);\n    };\n    auto es = seastar::inheriting_concrete_execution_stage<void, seastar::scheduling_group>(\"stage\", check_sg);\n    auto make_attr = [] (scheduling_group sg) {\n        seastar::thread_attributes a;\n        a.sched_group = sg;\n        return a;\n    };\n    bool done = false;\n    auto make_test_thread = [&] (scheduling_group sg) {\n        return seastar::thread(make_attr(sg), [&, sg] {\n            while (!done) {\n                es(sg).get(); // will check if executed with same sg\n            };\n        });\n    };\n    auto th1 = make_test_thread(sg1);\n    auto th2 = make_test_thread(sg2);\n    seastar::sleep(10ms).get();\n    done = true;\n    th1.join().get();\n    th2.join().get();\n}\n\nstruct a_struct {};\n\nSEASTAR_THREAD_TEST_CASE(test_inheriting_concrete_execution_stage_reference_parameters) {\n    // mostly a compile test, but take the opportunity to test that passing\n    // by reference preserves the address\n    auto check_ref = [] (a_struct& ref, a_struct* ptr) {\n        BOOST_REQUIRE_EQUAL(&ref, ptr);\n    };\n    auto es = seastar::inheriting_concrete_execution_stage<void, a_struct&, a_struct*>(\"stage\", check_ref);\n    a_struct obj;\n    es(seastar::ref(obj), &obj).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_execution_stage_rename) {\n    auto do_nothing = []{};\n    auto stage = seastar::make_execution_stage(\"test\", do_nothing);\n    BOOST_REQUIRE_EQUAL(stage.name(), \"test\");\n\n    // Default behaviour - should not change the name\n    stage.update_name_and_metric_group();\n    BOOST_REQUIRE_EQUAL(stage.name(), \"test\");\n}\n"
  },
  {
    "path": "tests/unit/expected_exception.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2025-present ScyllaDB\n */\n\n#pragma once\n\n#include <exception>\n\nnamespace seastar {\n\nclass expected_exception : public std::runtime_error {\npublic:\n    expected_exception() : std::runtime_error(\"expected\") {}\n};\n\n} // namespace seastar\n"
  },
  {
    "path": "tests/unit/expiring_fifo_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/core/expiring_fifo.hh>\n#include <seastar/util/later.hh>\n#include <boost/range/irange.hpp>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_no_expiry_operations) {\n    expiring_fifo<int> fifo;\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n\n    fifo.push_back(1);\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    fifo.push_back(2);\n    fifo.push_back(3);\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 3u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 2);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(!fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n    BOOST_REQUIRE(bool(fifo));\n    BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n\n    fifo.pop_front();\n\n    BOOST_REQUIRE(fifo.empty());\n    BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    BOOST_REQUIRE(!bool(fifo));\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_expiry_operations) {\n    return seastar::async([] {\n        std::vector<int> expired;\n        struct my_expiry {\n            std::vector<int>& e;\n            void operator()(int& v) { e.push_back(v); }\n        };\n\n        expiring_fifo<int, my_expiry, manual_clock> fifo(my_expiry{expired});\n\n        fifo.push_back(1, manual_clock::now() + 1s);\n\n        BOOST_REQUIRE(!fifo.empty());\n        BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n        BOOST_REQUIRE(bool(fifo));\n        BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE(fifo.empty());\n        BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n        BOOST_REQUIRE(!bool(fifo));\n        BOOST_REQUIRE_EQUAL(expired.size(), 1u);\n        BOOST_REQUIRE_EQUAL(expired[0], 1);\n\n        expired.clear();\n\n        fifo.push_back(1);\n        fifo.push_back(2, manual_clock::now() + 1s);\n        fifo.push_back(3);\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE(!fifo.empty());\n        BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n        BOOST_REQUIRE(bool(fifo));\n        BOOST_REQUIRE_EQUAL(expired.size(), 1u);\n        BOOST_REQUIRE_EQUAL(expired[0], 2);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n        expired.clear();\n\n        fifo.push_back(1, manual_clock::now() + 1s);\n        fifo.push_back(2, manual_clock::now() + 1s);\n        fifo.push_back(3);\n        fifo.push_back(4, manual_clock::now() + 2s);\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE(!fifo.empty());\n        BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n        BOOST_REQUIRE(bool(fifo));\n        BOOST_REQUIRE_EQUAL(expired.size(), 2u);\n        std::sort(expired.begin(), expired.end());\n        BOOST_REQUIRE_EQUAL(expired[0], 1);\n        BOOST_REQUIRE_EQUAL(expired[1], 2);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 3);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 4);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n        expired.clear();\n\n        fifo.push_back(1);\n        fifo.push_back(2, manual_clock::now() + 1s);\n        fifo.push_back(3, manual_clock::now() + 1s);\n        fifo.push_back(4, manual_clock::now() + 1s);\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE(!fifo.empty());\n        BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n        BOOST_REQUIRE(bool(fifo));\n        BOOST_REQUIRE_EQUAL(expired.size(), 3u);\n        std::sort(expired.begin(), expired.end());\n        BOOST_REQUIRE_EQUAL(expired[0], 2);\n        BOOST_REQUIRE_EQUAL(expired[1], 3);\n        BOOST_REQUIRE_EQUAL(expired[2], 4);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n\n        expired.clear();\n\n        fifo.push_back(1);\n        fifo.push_back(2, manual_clock::now() + 1s);\n        fifo.push_back(3, manual_clock::now() + 1s);\n        fifo.push_back(4, manual_clock::now() + 1s);\n        fifo.push_back(5);\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE_EQUAL(fifo.size(), 2u);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 1);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 1u);\n        BOOST_REQUIRE_EQUAL(fifo.front(), 5);\n        fifo.pop_front();\n        BOOST_REQUIRE_EQUAL(fifo.size(), 0u);\n    });\n}\n"
  },
  {
    "path": "tests/unit/fair_queue_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/fair_queue.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/print.hh>\n#include <boost/range/irange.hpp>\n#include <chrono>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nstruct request {\n    fair_queue_entry fqent;\n    std::function<void(request& req)> handle;\n    unsigned index;\n\n    template <typename Func>\n    request(fair_queue_entry::capacity_t cap, unsigned index, Func&& h)\n        : fqent(cap)\n        , handle(std::move(h))\n        , index(index)\n    {}\n\n    void submit() {\n        handle(*this);\n        delete this;\n    }\n};\n\nconstexpr unsigned test_weight_scale = 1000;\n\nclass test_env {\n    fair_queue _fq;\n    std::vector<int> _results;\n    std::vector<std::vector<std::exception_ptr>> _exceptions;\n    fair_queue::class_id _nr_classes = 0;\n    unsigned _nr_groups = 0;\n    std::vector<request> _inflight;\n\n    static fair_queue::config fq_config() {\n        fair_queue::config cfg;\n        cfg.forgiving_factor = 50 * test_weight_scale;\n        return cfg;\n    }\n\n    void drain() {\n        do {} while (tick() != 0);\n    }\npublic:\n    test_env()\n        : _fq(fq_config())\n    {\n    }\n\n    // As long as there is a request sitting in the queue, tick() will process\n    // at least one request. The only situation in which tick() will return nothing\n    // is if no requests were sent to the fair_queue (obviously).\n    //\n    // Because of this property, one useful use of tick() is to implement a drain()\n    // method (see above) in which all requests currently sent to the queue are drained\n    // before the queue is destroyed.\n    unsigned tick(unsigned n = 1) {\n        unsigned dispatched = 0;\n        unsigned processed = 0;\n        while (dispatched < n) {\n            auto* req = _fq.top();\n            if (req == nullptr) {\n                break;\n            }\n\n            dispatched++;\n            _fq.pop_front();\n            boost::intrusive::get_parent_from_member(req, &request::fqent)->submit();\n        }\n\n        for (unsigned i = 0; i < n; ++i) {\n            std::vector<request> curr;\n            curr.swap(_inflight);\n\n            for (auto& req : curr) {\n                processed++;\n                _results[req.index]++;\n                _fq.notify_request_finished(req.fqent.capacity());\n            }\n        }\n        return processed;\n    }\n\n    ~test_env() {\n        drain();\n        for (fair_queue::class_id id = 0; id < _nr_classes; id++) {\n            _fq.unregister_priority_class(id);\n        }\n    }\n\n    unsigned register_priority_group(uint32_t shares) {\n        _fq.ensure_priority_group(_nr_groups, shares);\n        return _nr_groups++;\n    }\n\n    size_t register_priority_class(uint32_t shares, std::optional<unsigned> group = {}) {\n        _results.push_back(0);\n        _exceptions.push_back(std::vector<std::exception_ptr>());\n        _fq.register_priority_class(_nr_classes, shares, group);\n        return _nr_classes++;\n    }\n\n    void do_op(fair_queue::class_id id, unsigned weight) {\n        unsigned index = id;\n        auto cap = fair_queue_entry::capacity_t(test_weight_scale * weight);\n        auto req = std::make_unique<request>(cap, index, [this, index] (request& req) mutable noexcept {\n            try {\n                _inflight.push_back(std::move(req));\n            } catch (...) {\n                auto eptr = std::current_exception();\n                _exceptions[index].push_back(eptr);\n                _fq.notify_request_finished(req.fqent.capacity());\n            }\n        });\n\n        _fq.queue(id, req->fqent);\n        req.release();\n    }\n\n    void update_shares(fair_queue::class_id id, uint32_t shares) {\n        _fq.update_shares_for_class(id, shares);\n    }\n\n    void reset_results(unsigned index) {\n        _results[index] = 0;\n    }\n\n    // Verify if the ratios are what we expect. Because we can't be sure about\n    // precise timing issues, we can always be off by some percentage. In simpler\n    // tests we really expect it to very low, but in more complex tests, with share\n    // changes, for instance, they can accumulate\n    //\n    // The ratios argument is the ratios towards the first class\n    void verify(sstring name, std::vector<unsigned> ratios, unsigned expected_error = 1) {\n        SEASTAR_ASSERT(ratios.size() == _results.size());\n        auto str = name + \":\";\n        for (auto i = 0ul; i < _results.size(); ++i) {\n            str += format(\" r[{:d}] = {:d}\", i, _results[i]);\n        }\n        std::cout << str << std::endl;\n        for (auto i = 0ul; i < ratios.size(); ++i) {\n            int min_expected = ratios[i] * (_results[0] - expected_error);\n            int max_expected = ratios[i] * (_results[0] + expected_error);\n            BOOST_CHECK_GE(_results[i], min_expected);\n            BOOST_CHECK_LE(_results[i], max_expected);\n            BOOST_CHECK_EQUAL(_exceptions[i].size(), 0);\n        }\n    }\n\n    void verify_f(sstring name, std::vector<float> ratios, float error) {\n        SEASTAR_ASSERT(ratios.size() == _results.size());\n        auto str = name + \":\";\n        std::vector<float> adjusted_results;\n        adjusted_results.reserve(ratios.size());\n        for (auto i = 0ul; i < _results.size(); ++i) {\n            str += format(\" r[{:d}] = {:d}\", i, _results[i]);\n            adjusted_results.push_back(_results[i] / ratios[i]);\n        }\n        std::cout << str << std::endl;\n        float average_result = 0.0;\n        for (auto ar : adjusted_results) {\n            average_result += ar;\n        }\n        average_result /= adjusted_results.size();\n        for (auto ar : adjusted_results) {\n            auto dev = std::abs(ar - average_result) / average_result;\n            BOOST_CHECK_LE(dev, error);\n        }\n    }\n\n    void verify_x(sstring name, std::vector<int> values) {\n        SEASTAR_ASSERT(values.size() == _results.size());\n        auto str = name + \":\";\n        for (auto i = 0ul; i < _results.size(); ++i) {\n            str += format(\" r[{:d}] = {:d}\", i, _results[i]);\n        }\n        std::cout << str <<std::endl;\n        for (unsigned i = 0; i < _results.size(); i++) {\n            BOOST_CHECK_EQUAL(_results[i], values[i]);\n        }\n    }\n};\n\n// Equal ratios. Expected equal results.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_equal_2classes) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n\n    yield().get();\n    // allow half the requests in\n    env.tick(10);\n    env.verify(\"equal_2classes\", {1, 1});\n    env.tick(90);\n    env.verify(\"equal_2classes_more\", {1, 1});\n}\n\n// Equal results, spread among 4 classes.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_equal_4classes) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n    auto c = env.register_priority_class(10);\n    auto d = env.register_priority_class(10);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n        env.do_op(c, 1);\n        env.do_op(d, 1);\n    }\n    yield().get();\n    // allow half the requests in\n    env.tick(200);\n    env.verify(\"equal_4classes\", {1, 1, 1, 1});\n}\n\n// Class2 twice as powerful. Expected class2 to have 2 x more requests.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_different_shares) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(20);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n    yield().get();\n    // allow half the requests in\n    env.tick(10);\n    env.verify(\"different_shares\", {1, 2});\n    env.tick(90);\n    env.verify(\"different_shares_more\", {1, 2});\n}\n\n// Classes equally powerful. But Class1 issues twice as expensive requests. Expected Class2 to have 2 x more requests.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_different_weights) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 2);\n        env.do_op(b, 1);\n    }\n    yield().get();\n    // allow half the requests in\n    env.tick(10);\n    env.verify(\"different_weights\", {1, 2});\n    env.tick(90);\n    env.verify(\"different_weights_more\", {1, 2});\n}\n\n// Class2 pushes many requests over. Right after, don't expect Class2 to be able to push anything else.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_dominant_queue) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(b, 1);\n    }\n    yield().get();\n\n    // consume all requests\n    env.tick(100);\n    // zero statistics.\n    env.reset_results(b);\n    for (int i = 0; i < 20; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n    // allow half the requests in\n    env.tick(20);\n    env.verify(\"dominant_queue\", {1, 0});\n}\n\n// Class2 pushes many requests at first. Right after, don't expect Class1 to be able to do the same\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_forgiving_queue) {\n    test_env env;\n\n    // The fair_queue preemption logic allows one class to gain exclusive\n    // queue access for at most tau duration. Test queue configures the\n    // request rate to be 1/us and tau to be 50us, so after (re-)activation\n    // a queue can overrun its peer by at most 50 requests.\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 1);\n    }\n    yield().get();\n\n    // consume all requests\n    env.tick(100);\n    env.reset_results(a);\n\n    for (int i = 0; i < 100; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n    yield().get();\n\n    // allow half the requests in\n    env.tick(100);\n    // 50 requests should be passed from b, other 100 should be shared 1:1\n    env.verify(\"forgiving_queue\", {1, 3}, 2);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_forgiving_group) {\n    test_env env;\n\n    auto g = env.register_priority_group(10);\n    auto a = env.register_priority_class(10, g);\n    auto b = env.register_priority_class(10, g);\n    auto c = env.register_priority_class(10);\n\n    auto step = [&env] (std::string name, unsigned pc, unsigned nr, std::vector<int> results) {\n        for (unsigned i = 0; i < nr; i++) {\n            env.do_op(pc, 1);\n        }\n        env.tick(100);\n        env.verify_x(\"forgiving_group_\" + name, std::move(results));\n    };\n\n    auto drain = [&] () {\n        env.tick(300);\n        env.reset_results(a);\n        env.reset_results(b);\n        env.reset_results(c);\n    };\n\n    // Test one -- activate in top-nesetd-nested order\n    {\n        // Step-1 -- let the top-level clas work on its own and dispatch 100\n        // requests. At this point those 100 requests will be dispatched.\n        step(\"1_top\", c, 300, {0, 0, 100});\n\n        // Step-2 -- activate a sub-class and its class group, then dispatch\n        // 100 more requests. Queue fogrives 50 requests to the group upon\n        // activation, then dispatches remaining 50 in 1:1 proportion between\n        // two active classes.\n        step(\"1_nest_1\", a, 200, {75, 0, 125});\n\n        // Step-3 -- activate another sub-class, then dispatch 100 more requests.\n        // This time queue forgives 50 requests to the sub-class, but not to the\n        // group, because it's still active since previous step. After it, the\n        // top-level class and a subgroup are dispatched in 1:1 ratio, but since\n        // the newly activated sub-class has 50 forgiven requests, only _this_\n        // class will be dispatched from in this group, the other sub-class would\n        // remain idle during this 100-requests dispatch.\n        step(\"1_nest_2\", b, 100, {75, 50, 175});\n\n        // Finally, continue dispatching. At this point no more \"forgiven\" requests\n        // are left and all three classes are dispatched from according to their\n        // shares. Given it's {1:1}:2 ratios, the addition should be {25, 25, 50}\n        step(\"1_cont\", 0, 0, {100, 75, 225});\n\n        drain();\n    }\n\n    // Test two -- activate in nested-outer-nested order\n    {\n        // Step-1 -- netsted class works on its own and dispatches\n        // all the requests it wants.\n        step(\"2_nest_1\", a, 300, {100, 0, 0});\n\n        // Step-2 -- top-level class activates and gets 50 fogiven requests.\n        // 50 remaining are shared between a and c in 1:1 proportion, just like\n        // it was in the top-nest-nest sequence above, no difference here\n        step(\"2_top\", c, 200, {125, 0, 75});\n\n        // Step-3 -- another nested class activates and gets 50 forgiven\n        // requests. 100 requests are shared between c and {a+b} in 1:1\n        // fraction, so c gets 50 and a+b get another 50 which all go to\n        // b class for its \"forgiven\" credit\n        step(\"2_nest_2\", b, 100, {125, 50, 125});\n\n        // Remaining requests are shared as in test-1 above: {25, 25, 50}\n        step(\"2_cont\", 0, 0, {150, 75, 175});\n\n        drain();\n    }\n\n    // Test two -- activate in nested-nested-outer order\n    {\n        // First two steps don't differ from the previous runs -- first\n        // class gets 100 requests, then the 2nd activates and gets 50\n        // forgiven and remaining 50 are split evenly.\n        step(\"3_nest_1\", a, 300, {100, 0, 0});\n        step(\"3_nest_2\", b, 200, {125, 75, 0});\n\n        // Step-3 -- top level class activates and get 50 forgiven\n        // requests. After it remaining 50 are divided between it and\n        // the a+b group in 1:1 fraction, so class c gets 25 more.\n        // Classes in group shared the remaining 25 requests in 1:1 as\n        // well, so one class get 12 requests and another one gets 13.\n        step(\"3_top\", c, 100, {137, 88, 75});\n\n        // The remaining requests are as well shared in {25, 25, 50}\n        // proportion just as before (give c more requests so that it\n        // can dispatch its goal).\n        step(\"3_cont\", c, 50, {162, 113, 125});\n\n        drain();\n    }\n}\n\n// Classes push requests and then update swap their shares. In the end, should have executed\n// the same number of requests.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_update_shares) {\n    test_env env;\n\n    auto a = env.register_priority_class(20);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 500; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n\n    yield().get();\n    // allow 25% of the requests in\n    env.tick(250);\n    env.update_shares(a, 10);\n    env.update_shares(b, 20);\n\n    yield().get();\n    // allow 25% of the requests in\n    env.tick(250);\n    env.verify(\"update_shares\", {1, 1}, 2);\n}\n\n// Classes run for a longer period of time. Balance must be kept over many timer\n// periods.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_longer_run) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(10);\n\n    for (int i = 0; i < 20000; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n    // In total allow half the requests in, but do it over a\n    // long period of time, ticking slowly\n    for (int i = 0; i < 1000; ++i) {\n        sleep(1ms).get();\n        env.tick(2);\n    }\n    env.verify(\"longer_run\", {1, 1}, 2);\n}\n\n// Classes run for a longer period of time. Proportional balance must be kept over many timer\n// periods, despite unequal shares..\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_longer_run_different_shares) {\n    test_env env;\n\n    auto a = env.register_priority_class(10);\n    auto b = env.register_priority_class(20);\n\n    for (int i = 0; i < 20000; ++i) {\n        env.do_op(a, 1);\n        env.do_op(b, 1);\n    }\n\n    // In total allow half the requests in, but do it over a\n    // long period of time, ticking slowly\n    for (int i = 0; i < 1000; ++i) {\n        sleep(1ms).get();\n        env.tick(3);\n    }\n    env.verify(\"longer_run_different_shares\", {1, 2}, 2);\n}\n\n// Classes run with random shares and random requests weights. Proportional operations expected.\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_random_run) {\n    test_env env;\n\n    std::default_random_engine& generator = testing::local_random_engine;\n    std::uniform_int_distribution<uint32_t> shares(1, 5);\n    std::uniform_int_distribution<uint32_t> weights(1, 5);\n\n    struct test_class {\n        unsigned shares;\n        unsigned weight;\n        float expected;\n        size_t cls;\n    };\n\n    auto add_class = [&] (std::optional<unsigned> group) {\n        auto s = shares(generator);\n        auto w = weights(generator);\n        std::cout << format(\"Add class with {} shares and {} request weight, in group = {}\", s, w, group.has_value()) << std::endl;\n        return test_class {\n            .shares = s,\n            .weight = w,\n            .expected = float(s)/float(w),\n            .cls = env.register_priority_class(s, group),\n        };\n    };\n\n    unsigned group_shares = 0;\n    std::optional<unsigned> group;\n    std::uniform_int_distribution<unsigned> in_group(0, 1);\n    if (in_group(generator)) {\n        group_shares = shares(generator);\n        group = env.register_priority_group(group_shares);\n        std::cout << format(\"Add group with {} shares\", group_shares) << std::endl;\n    }\n\n    auto a = add_class({});\n    auto b = add_class(group);\n    auto c = add_class(group);\n\n    if (group) {\n        b.expected = float(group_shares) * (float(b.shares) / float(b.shares + c.shares)) / float(b.weight);\n        c.expected = float(group_shares) * (float(c.shares) / float(b.shares + c.shares)) / float(c.weight);\n    }\n\n    unsigned reqs = 3000;\n\n    // Enough requests for the maximum run (half per queue, + leeway)\n    for (uint32_t i = 0; i < reqs; ++i) {\n        env.do_op(a.cls, a.weight);\n        env.do_op(b.cls, b.weight);\n        env.do_op(c.cls, c.weight);\n    }\n\n    yield().get();\n    // In total allow one-third of the requests in\n    env.tick(reqs);\n\n    env.verify_f(format(\"random_run ({:d} requests)\", reqs), {a.expected, b.expected, c.expected}, 0.05);\n}\n\n// Exhaustive test for waking-up a tree of classes with requests\nSEASTAR_THREAD_TEST_CASE(test_fair_queue_nested_wakeups) {\n    test_env env;\n\n    std::vector<unsigned> pcs;\n    unsigned g0 = env.register_priority_group(1);\n    unsigned g1 = env.register_priority_group(1);\n    pcs.push_back(env.register_priority_class(1, {}));\n    pcs.push_back(env.register_priority_class(1, {}));\n    pcs.push_back(env.register_priority_class(1, g0));\n    pcs.push_back(env.register_priority_class(1, g0));\n    pcs.push_back(env.register_priority_class(1, g0));\n    pcs.push_back(env.register_priority_class(1, g1));\n    pcs.push_back(env.register_priority_class(1, g1));\n\n    for (unsigned nr_reqs = 1; nr_reqs < 5; nr_reqs++) {\n        std::vector<unsigned> targets(nr_reqs, 0);\n\n        auto next = [&] {\n            for (unsigned i = 0; i < targets.size(); i++) {\n                if (targets[i] < pcs.size() - 1) {\n                    targets[i]++;\n                    return true;\n                }\n\n                targets[i] = 0;\n            }\n\n            return false;\n        };\n\n        do {\n            for (unsigned i = 0; i < targets.size(); i++) {\n                env.do_op(pcs[targets[i]], 1);\n            }\n\n            auto res = env.tick(targets.size());\n            BOOST_REQUIRE_EQUAL(res, targets.size());\n            res = env.tick(1);\n            BOOST_REQUIRE_EQUAL(res, 0);\n        } while (next());\n    }\n}\n"
  },
  {
    "path": "tests/unit/file_io_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014-2015 Cloudius Systems, Ltd.\n */\n\n#include <filesystem>\n\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/condition-variable.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/layered_file.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/stall_sampler.hh>\n#include <seastar/core/aligned_buffer.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/tmp_file.hh>\n#include <seastar/util/alloc_failure_injector.hh>\n#include <seastar/util/closeable.hh>\n#include <seastar/util/internal/magic.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n#include <seastar/util/later.hh>\n\n#include <boost/range/adaptor/transformed.hpp>\n#include <iostream>\n#include <sys/statfs.h>\n#include <fcntl.h>\n\nusing namespace seastar;\nnamespace fs = std::filesystem;\n\nconstexpr open_flags default_create_open_flags = open_flags::rw | open_flags::create;\n\nSEASTAR_TEST_CASE(open_flags_test) {\n    open_flags flags = default_create_open_flags | open_flags::exclusive;\n    BOOST_REQUIRE(std::underlying_type_t<open_flags>(flags) ==\n                  (std::underlying_type_t<open_flags>(open_flags::rw) |\n                   std::underlying_type_t<open_flags>(open_flags::create) |\n                   std::underlying_type_t<open_flags>(open_flags::exclusive)));\n\n    open_flags mask = open_flags::create  | open_flags::exclusive;\n    BOOST_REQUIRE((flags & mask) == mask);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(access_flags_test) {\n    access_flags flags = access_flags::read | access_flags::write  | access_flags::execute;\n    BOOST_REQUIRE(std::underlying_type_t<open_flags>(flags) ==\n                  (std::underlying_type_t<open_flags>(access_flags::read) |\n                   std::underlying_type_t<open_flags>(access_flags::write) |\n                   std::underlying_type_t<open_flags>(access_flags::execute)));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(file_exists_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, default_create_open_flags).get();\n        f.close().get();\n        auto exists = file_exists(filename).get();\n        BOOST_REQUIRE(exists);\n        remove_file(filename).get();\n        exists = file_exists(filename).get();\n        BOOST_REQUIRE(!exists);\n    });\n}\n\nSEASTAR_TEST_CASE(handle_bad_alloc_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, default_create_open_flags).get();\n        f.close().get();\n        bool exists = false;\n        memory::with_allocation_failures([&] {\n            exists = file_exists(filename).get();\n        });\n        BOOST_REQUIRE(exists);\n    });\n}\n\nSEASTAR_TEST_CASE(file_access_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, default_create_open_flags).get();\n        f.close().get();\n        auto is_accessible = file_accessible(filename, access_flags::read | access_flags::write).get();\n        BOOST_REQUIRE(is_accessible);\n    });\n}\n\n// Test that a file handle can be transferred to another shard, and\n// that a file obtained from the handle is the same one, as it was\n// when the file was opened.\nSEASTAR_TEST_CASE(file_ro_dup_test) {\n    if (seastar::smp::count < 2) {\n        fmt::print(\"This test needs at least 2 shards to run\\n\");\n        return make_ready_future<>();\n    }\n    return tmp_dir::do_with([] (tmp_dir& t) -> future<> {\n        struct file_open_info {\n            seastar::file_handle handle;\n            struct stat st;\n        };\n        auto fi = co_await smp::submit_to(0, [&t] () -> future<file_open_info> {\n            sstring filename = (t.get_path() / \"testfile.tmp\").native();\n            auto f = co_await open_file_dma(filename, open_flags::ro | open_flags::create);\n            auto st = co_await f.stat();\n            auto fh = f.dup();\n            co_await f.close();\n            co_return file_open_info{ std::move(fh), st };\n        });\n        co_await smp::submit_to(1, [fi = std::move(fi)] () mutable -> future<> {\n            auto f = std::move(fi.handle).to_file();\n            auto st = co_await f.stat();\n            BOOST_REQUIRE_EQUAL(st.st_dev, fi.st.st_dev);\n            BOOST_REQUIRE_EQUAL(st.st_ino, fi.st.st_ino);\n        });\n    });\n}\n\n// Test that non-read-only files do not generate file-handle, but throw\nSEASTAR_TEST_CASE(file_rw_dup_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n        BOOST_REQUIRE_THROW(f.dup(), std::runtime_error);\n    });\n}\n\nstruct file_test {\n    file_test(file&& f) : f(std::move(f)) {}\n    file f;\n    semaphore sem = { 0 };\n    semaphore par = { 1000 };\n};\n\nSEASTAR_TEST_CASE(test1) {\n    // Note: this tests generates a file \"testfile.tmp\" with size 4096 * max (= 40 MB).\n  return tmp_dir::do_with([] (tmp_dir& t) {\n    static constexpr auto max = 10000;\n    sstring filename = (t.get_path() / \"testfile.tmp\").native();\n    return open_file_dma(filename, default_create_open_flags).then([filename] (file f) {\n        auto ft = new file_test{std::move(f)};\n        for (size_t i = 0; i < max; ++i) {\n            // Don't wait for future, use semaphore to signal when done instead.\n            (void)ft->par.wait().then([ft, i] {\n                auto wbuf = allocate_aligned_buffer<unsigned char>(4096, 4096);\n                std::fill(wbuf.get(), wbuf.get() + 4096, i);\n                auto wb = wbuf.get();\n                (void)ft->f.dma_write(i * 4096, wb, 4096).then(\n                        [ft, i, wbuf = std::move(wbuf)] (size_t ret) mutable {\n                    BOOST_REQUIRE(ret == 4096);\n                    auto rbuf = allocate_aligned_buffer<unsigned char>(4096, 4096);\n                    auto rb = rbuf.get();\n                    (void)ft->f.dma_read(i * 4096, rb, 4096).then(\n                            [ft, rbuf = std::move(rbuf), wbuf = std::move(wbuf)] (size_t ret) mutable {\n                        BOOST_REQUIRE(ret == 4096);\n                        BOOST_REQUIRE(std::equal(rbuf.get(), rbuf.get() + 4096, wbuf.get()));\n                        ft->sem.signal(1);\n                        ft->par.signal();\n                    });\n                });\n            });\n        }\n        return ft->sem.wait(max).then([ft] () mutable {\n            return ft->f.flush();\n        }).then([ft] {\n            return ft->f.close();\n        }).then([ft] () mutable {\n            delete ft;\n        });\n    });\n  });\n}\n\nSEASTAR_TEST_CASE(parallel_write_fsync) {\n    return internal::report_reactor_stalls([] {\n        return tmp_dir::do_with_thread([] (tmp_dir& t) {\n            // Plan: open a file and write to it like crazy. In parallel fsync() it all the time.\n            auto fname = (t.get_path() / \"testfile.tmp\").native();\n            auto sz = uint64_t(32*1024*1024);\n            auto buffer_size = 32768;\n            auto write_concurrency = 16;\n            auto fsync_every = 1024*1024;\n            auto max_write_ahead_of_fsync = 4*1024*1024; // ensures writes don't complete too quickly\n            auto written = uint64_t(0);\n            auto fsynced_at = uint64_t(0);\n\n            file f = open_file_dma(fname, default_create_open_flags | open_flags::truncate).get();\n            auto close_f = deferred_close(f);\n            // Avoid filesystem problems with size-extending operations\n            f.truncate(sz).get();\n\n            auto fsync_semaphore = semaphore(0);\n            auto may_write_condvar = condition_variable();\n            auto fsync_thread = thread([&] {\n                auto fsynced = uint64_t(0);\n                while (fsynced < sz) {\n                    fsync_semaphore.wait(fsync_every).get();\n                    fsynced_at = written;\n                    // Signal the condition variable now so that writes proceed\n                    // in parallel with the fsync\n                    may_write_condvar.broadcast();\n                    f.flush().get();\n                    fsynced += fsync_every;\n                }\n            });\n\n            auto write_semaphore = semaphore(write_concurrency);\n            while (written < sz) {\n                write_semaphore.wait().get();\n                may_write_condvar.wait([&] {\n                    return written <= fsynced_at + max_write_ahead_of_fsync;\n                }).get();\n                auto buf = temporary_buffer<char>::aligned(f.memory_dma_alignment(), buffer_size);\n                memset(buf.get_write(), 0, buf.size());\n                // Write asynchronously, signal when done.\n                (void)f.dma_write(written, buf.get(), buf.size()).then([&fsync_semaphore, &write_semaphore, buf = std::move(buf)] (size_t w) {\n                    fsync_semaphore.signal(buf.size());\n                    write_semaphore.signal();\n                });\n                written += buffer_size;\n            }\n            write_semaphore.wait(write_concurrency).get();\n\n            fsync_thread.join().get();\n            close_f.close_now();\n            remove_file(fname).get();\n        });\n    }).then([] (internal::stall_report sr) {\n        std::cout << \"parallel_write_fsync: \" << sr << \"\\n\";\n    });\n}\n\nSEASTAR_TEST_CASE(test_iov_max) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    static constexpr size_t buffer_size = 4096;\n    static constexpr size_t buffer_count = IOV_MAX * 2 + 1;\n\n    std::vector<temporary_buffer<char>> original_buffers;\n    std::vector<iovec> iovecs;\n    for (auto i = 0u; i < buffer_count; i++) {\n        original_buffers.emplace_back(temporary_buffer<char>::aligned(buffer_size, buffer_size));\n        std::fill_n(original_buffers.back().get_write(), buffer_size, char(i));\n        iovecs.emplace_back(iovec { original_buffers.back().get_write(), buffer_size });\n    }\n\n    auto filename = (t.get_path() / \"testfile.tmp\").native();\n    auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n    auto close_f = deferred_close(f);\n    size_t left = buffer_size * buffer_count;\n    size_t position = 0;\n    while (left) {\n        auto written = f.dma_write(position, iovecs).get();\n        iovecs.erase(iovecs.begin(), iovecs.begin() + written / buffer_size);\n        SEASTAR_ASSERT(written % buffer_size == 0);\n        position += written;\n        left -= written;\n    }\n\n    BOOST_CHECK(iovecs.empty());\n\n    std::vector<temporary_buffer<char>> read_buffers;\n    for (auto i = 0u; i < buffer_count; i++) {\n        read_buffers.emplace_back(temporary_buffer<char>::aligned(buffer_size, buffer_size));\n        std::fill_n(read_buffers.back().get_write(), buffer_size, char(0));\n        iovecs.emplace_back(iovec { read_buffers.back().get_write(), buffer_size });\n    }\n\n    left = buffer_size * buffer_count;\n    position = 0;\n    while (left) {\n        auto read = f.dma_read(position, iovecs).get();\n        iovecs.erase(iovecs.begin(), iovecs.begin() + read / buffer_size);\n        SEASTAR_ASSERT(read % buffer_size == 0);\n        position += read;\n        left -= read;\n    }\n\n    for (auto i = 0u; i < buffer_count; i++) {\n        BOOST_CHECK(std::equal(original_buffers[i].get(), original_buffers[i].get() + original_buffers[i].size(),\n                               read_buffers[i].get(), read_buffers[i].get() + read_buffers[i].size()));\n    }\n  });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_sanitize_iovecs) {\n    auto buf = temporary_buffer<char>::aligned(4096, 4096);\n\n    auto iovec_equal = [] (const iovec& a, const iovec& b) {\n        return a.iov_base == b.iov_base && a.iov_len == b.iov_len;\n    };\n\n    { // Single fragment, sanitize is noop\n        auto original_iovecs = std::vector<iovec> { { buf.get_write(), buf.size() } };\n        auto actual_iovecs = original_iovecs;\n        auto actual_length = internal::sanitize_iovecs(actual_iovecs, 4096);\n        BOOST_CHECK_EQUAL(actual_length, 4096);\n        BOOST_CHECK_EQUAL(actual_iovecs.size(), 1);\n        BOOST_CHECK(iovec_equal(original_iovecs.back(), actual_iovecs.back()));\n    }\n\n    { // one 1024 buffer and IOV_MAX+6 buffers of 512; 4096 byte disk alignment, sanitize needs to drop buffers\n        auto original_iovecs = std::vector<iovec>{};\n        for (auto i = 0u; i < IOV_MAX + 7; i++) {\n            original_iovecs.emplace_back(iovec { buf.get_write(), i == 0 ? 1024u : 512u });\n        }\n        auto actual_iovecs = original_iovecs;\n        auto actual_length = internal::sanitize_iovecs(actual_iovecs, 4096);\n        BOOST_CHECK_EQUAL(actual_length, 512 * IOV_MAX);\n        BOOST_CHECK_EQUAL(actual_iovecs.size(), IOV_MAX - 1);\n\n        original_iovecs.resize(IOV_MAX - 1);\n        BOOST_CHECK(std::equal(original_iovecs.begin(), original_iovecs.end(),\n                               actual_iovecs.begin(), actual_iovecs.end(), iovec_equal));\n    }\n\n    { // IOV_MAX-1 buffers of 512, one 1024 buffer, and 6 512 buffers; 4096 byte disk alignment, sanitize needs to drop and trim buffers\n        auto original_iovecs = std::vector<iovec>{};\n        for (auto i = 0u; i < IOV_MAX + 7; i++) {\n            original_iovecs.emplace_back(iovec { buf.get_write(), i == (IOV_MAX - 1) ? 1024u : 512u });\n        }\n        auto actual_iovecs = original_iovecs;\n        auto actual_length = internal::sanitize_iovecs(actual_iovecs, 4096);\n        BOOST_CHECK_EQUAL(actual_length, 512 * IOV_MAX);\n        BOOST_CHECK_EQUAL(actual_iovecs.size(), IOV_MAX);\n\n        original_iovecs.resize(IOV_MAX);\n        original_iovecs.back().iov_len = 512;\n        BOOST_CHECK(std::equal(original_iovecs.begin(), original_iovecs.end(),\n                               actual_iovecs.begin(), actual_iovecs.end(), iovec_equal));\n    }\n\n    { // IOV_MAX+8 buffers of 512; 4096 byte disk alignment, sanitize needs to drop buffers\n        auto original_iovecs = std::vector<iovec>{};\n        for (auto i = 0u; i < IOV_MAX + 8; i++) {\n            original_iovecs.emplace_back(iovec { buf.get_write(), 512 });\n        }\n        auto actual_iovecs = original_iovecs;\n        auto actual_length = internal::sanitize_iovecs(actual_iovecs, 4096);\n        BOOST_CHECK_EQUAL(actual_length, 512 * IOV_MAX);\n        BOOST_CHECK_EQUAL(actual_iovecs.size(), IOV_MAX);\n\n        original_iovecs.resize(IOV_MAX);\n        BOOST_CHECK(std::equal(original_iovecs.begin(), original_iovecs.end(),\n                               actual_iovecs.begin(), actual_iovecs.end(), iovec_equal));\n    }\n}\n\nSEASTAR_TEST_CASE(test_chmod) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    auto oflags = open_flags::rw | open_flags::create;\n    sstring filename = (t.get_path() / \"testfile.tmp\").native();\n    if (file_exists(filename).get()) {\n        remove_file(filename).get();\n    }\n\n    auto orig_umask = umask(0);\n\n    // test default_file_permissions\n    auto f = open_file_dma(filename, oflags).get();\n    f.close().get();\n    auto sd = file_stat(filename).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_file_permissions));\n\n    // test chmod with new_permissions\n    auto new_permissions = file_permissions::user_read | file_permissions::group_read | file_permissions::others_read;\n    BOOST_REQUIRE(new_permissions != file_permissions::default_file_permissions);\n    BOOST_REQUIRE(file_exists(filename).get());\n    chmod(filename, new_permissions).get();\n    sd = file_stat(filename).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(new_permissions));\n    remove_file(filename).get();\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_open_file_dma_permissions) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    auto oflags = open_flags::rw | open_flags::create;\n    sstring filename = (t.get_path() / \"testfile.tmp\").native();\n    if (file_exists(filename).get()) {\n        remove_file(filename).get();\n    }\n\n    auto orig_umask = umask(0);\n\n    // test default_file_permissions\n    auto f = open_file_dma(filename, oflags).get();\n    f.close().get();\n    auto sd = file_stat(filename).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_file_permissions));\n    remove_file(filename).get();\n\n    // test options.create_permissions\n    auto options = file_open_options();\n    options.create_permissions = file_permissions::user_read | file_permissions::group_read | file_permissions::others_read;\n    BOOST_REQUIRE(options.create_permissions != file_permissions::default_file_permissions);\n    f = open_file_dma(filename, oflags, options).get();\n    f.close().get();\n    sd = file_stat(filename).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(options.create_permissions));\n    remove_file(filename).get();\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_make_directory_permissions) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    sstring dirname = (t.get_path() / \"testdir.tmp\").native();\n    auto orig_umask = umask(0);\n\n    // test default_dir_permissions with make_directory\n    make_directory(dirname).get();\n    auto sd = file_stat(dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    remove_file(dirname).get();\n\n    // test make_directory\n    auto create_permissions = file_permissions::user_read | file_permissions::group_read | file_permissions::others_read;\n    BOOST_REQUIRE(create_permissions != file_permissions::default_dir_permissions);\n    make_directory(dirname, create_permissions).get();\n    sd = file_stat(dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(create_permissions));\n    remove_file(dirname).get();\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_touch_directory_permissions) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    sstring dirname = (t.get_path() / \"testdir.tmp\").native();\n    auto orig_umask = umask(0);\n\n    // test default_dir_permissions with touch_directory\n    touch_directory(dirname).get();\n    auto sd = file_stat(dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    remove_file(dirname).get();\n\n    // test touch_directory, dir creation\n    auto create_permissions = file_permissions::user_read | file_permissions::group_read | file_permissions::others_read;\n    BOOST_REQUIRE(create_permissions != file_permissions::default_dir_permissions);\n    BOOST_REQUIRE(!file_exists(dirname).get());\n    touch_directory(dirname, create_permissions).get();\n    sd = file_stat(dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(create_permissions));\n\n    // test touch_directory of existing dir, dir mode need not change\n    BOOST_REQUIRE(file_exists(dirname).get());\n    touch_directory(dirname, file_permissions::default_dir_permissions).get();\n    sd = file_stat(dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(create_permissions));\n    remove_file(dirname).get();\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_recursive_touch_directory_permissions) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    sstring base_dirname = (t.get_path() / \"testbasedir.tmp\").native();\n    sstring dirpath = base_dirname + \"/\" + \"testsubdir.tmp\";\n    if (file_exists(dirpath).get()) {\n        remove_file(dirpath).get();\n    }\n    if (file_exists(base_dirname).get()) {\n        remove_file(base_dirname).get();\n    }\n\n    auto orig_umask = umask(0);\n\n    // test default_dir_permissions with recursive_touch_directory\n    recursive_touch_directory(dirpath).get();\n    auto sd = file_stat(base_dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    sd = file_stat(dirpath).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    remove_file(dirpath).get();\n\n    // test recursive_touch_directory, dir creation\n    auto create_permissions = file_permissions::user_read | file_permissions::group_read | file_permissions::others_read;\n    BOOST_REQUIRE(create_permissions != file_permissions::default_dir_permissions);\n    BOOST_REQUIRE(file_exists(base_dirname).get());\n    BOOST_REQUIRE(!file_exists(dirpath).get());\n    recursive_touch_directory(dirpath, create_permissions).get();\n    sd = file_stat(base_dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    sd = file_stat(dirpath).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(create_permissions));\n\n    // test recursive_touch_directory of existing dir, dir mode need not change\n    BOOST_REQUIRE(file_exists(dirpath).get());\n    recursive_touch_directory(dirpath, file_permissions::default_dir_permissions).get();\n    sd = file_stat(base_dirname).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_dir_permissions));\n    sd = file_stat(dirpath).get();\n    BOOST_CHECK_EQUAL(sd.mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(create_permissions));\n    remove_file(dirpath).get();\n    remove_file(base_dirname).get();\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_file_stat_method) {\n  return tmp_dir::do_with_thread([] (tmp_dir& t) {\n    auto oflags = open_flags::rw | open_flags::create;\n    sstring filename = (t.get_path() / \"testfile.tmp\").native();\n\n    auto orig_umask = umask(0);\n\n    auto f = open_file_dma(filename, oflags).get();\n    auto close_f = deferred_close(f);\n    auto st = f.stat().get();\n    BOOST_CHECK_EQUAL(st.st_mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_file_permissions));\n\n    umask(orig_umask);\n  });\n}\n\nSEASTAR_TEST_CASE(test_dir_statat_method) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        auto path = t.get_path();\n        const char* filename = \"testfile.tmp\";\n\n        // Create file in tmp dir\n        auto orig_umask = umask(0);\n\n        auto f = open_file_dma((path / filename).native(), oflags).get();\n        auto close_f = deferred_close(f);\n        size_t buffer_size = 4096;\n        auto buf = temporary_buffer<char>::aligned(f.memory_dma_alignment(), buffer_size);\n        memset(buf.get_write(), 0, buf.size());\n        auto written = f.dma_write(0, buf.get(), buf.size()).get();\n        BOOST_REQUIRE_EQUAL(written, buffer_size);\n        close_f.close_now();\n\n        // Open dir and get the file stat using statat\n        auto dir = open_directory(path.native()).get();\n        auto close_dir = deferred_close(dir);\n\n        auto file_st = dir.statat(filename).get();\n        BOOST_REQUIRE_EQUAL(file_st.st_size, buffer_size);\n\n        const char* linkname = \"test_link.tmp\";\n        auto rc = ::symlink((path / filename).c_str(), (path / linkname).c_str());\n        BOOST_REQUIRE_EQUAL(rc, 0);\n\n        auto link_st = dir.statat(linkname, AT_SYMLINK_NOFOLLOW).get();\n        BOOST_REQUIRE_NE(link_st.st_ino, file_st.st_ino);\n\n        // Default flags are expected to follow symlinks\n        auto st = dir.statat(linkname).get();\n        BOOST_REQUIRE_EQUAL(st.st_ino, file_st.st_ino);\n\n        // Test non-existing file resolving into exception\n        BOOST_REQUIRE_EXCEPTION(dir.statat(\"no-such-file\").get(), std::system_error, [] (auto& ex) {\n            return ex.code() == std::error_code(ENOENT, std::system_category());\n        });\n\n        umask(orig_umask);\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_write_lifetime_method) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n\n        auto f1 = open_file_dma(filename, oflags).get();\n        auto close_f1 = deferred_close(f1);\n        auto f2 = open_file_dma(filename, oflags).get();\n        auto close_f2 = deferred_close(f2);\n\n        // Write life time hint values\n        std::vector<uint64_t> hint_set = {RWF_WRITE_LIFE_NOT_SET,\n                                                RWH_WRITE_LIFE_NONE,\n                                                RWH_WRITE_LIFE_SHORT,\n                                                RWH_WRITE_LIFE_MEDIUM,\n                                                RWH_WRITE_LIFE_LONG,\n                                                RWH_WRITE_LIFE_EXTREME};\n\n        for (auto i = 0ul; i < hint_set.size(); ++i) {\n            auto hint = hint_set[i];\n\n            // Set and verify the lifetime hint of the inode\n            f1.set_inode_lifetime_hint(hint).get();\n            auto o_hint1 = f1.get_inode_lifetime_hint().get();\n            BOOST_CHECK_EQUAL(hint, o_hint1);\n        }\n\n        // Perform invalid ops\n        uint64_t hint = RWH_WRITE_LIFE_EXTREME + 1;\n        BOOST_REQUIRE_THROW(f1.set_inode_lifetime_hint(hint).get(), std::system_error);\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_fcntl) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n\n        auto f = open_file_dma(filename, oflags).get();\n        auto close_f = deferred_close(f);\n\n        // Set and verify a lease value\n        auto lease = F_WRLCK;\n        BOOST_REQUIRE(!f.fcntl(F_SETLEASE, lease).get());\n        auto o_lease = f.fcntl(F_GETLEASE).get();\n        BOOST_CHECK_EQUAL(lease, o_lease);\n\n        // Use _short version and test the same\n        o_lease = f.fcntl_short(F_GETLEASE).get();\n        BOOST_CHECK_EQUAL(lease, o_lease);\n\n        // Perform invalid ops\n        BOOST_REQUIRE_THROW(f.fcntl(F_SETLEASE, (uintptr_t)~0ul).get(), std::system_error);\n        BOOST_REQUIRE_THROW(f.fcntl_short(F_SETLEASE, (uintptr_t)~0ul).get(), std::system_error);\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_ioctl) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        uint64_t block_size = 0;\n\n        auto f = open_file_dma(filename, oflags).get();\n        auto close_f = deferred_close(f);\n\n        // Issueing an FS ioctl which is applicable on regular files\n        // and can be executed as normal user\n        try {\n            BOOST_REQUIRE(!f.ioctl(FIGETBSZ, &block_size).get());\n            BOOST_REQUIRE(block_size != 0);\n\n            // Use _short version and test the same\n            BOOST_REQUIRE(!f.ioctl_short(FIGETBSZ, &block_size).get());\n            BOOST_REQUIRE(block_size != 0);\n        } catch (std::system_error& e) {\n            // anon_bdev filesystems do not support FIGETBSZ, and return EINVAL\n            BOOST_REQUIRE_EQUAL(e.code().value(), EINVAL);\n        }\n\n        // Perform invalid ops\n        BOOST_REQUIRE_THROW(f.ioctl(FIGETBSZ, 0ul).get(), std::system_error);\n        BOOST_REQUIRE_THROW(f.ioctl_short(FIGETBSZ, 0ul).get(), std::system_error);\n    });\n}\n\nclass test_layered_file : public layered_file_impl {\npublic:\n    explicit test_layered_file(file f) : layered_file_impl(std::move(f)) {}\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent*) override {\n        abort();\n    }\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent*) override {\n        abort();\n    }\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent*) override {\n        abort();\n    }\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent*) override {\n        abort();\n    }\n    virtual future<> flush(void) override {\n        abort();\n    }\n    virtual future<struct stat> stat(void) override {\n        abort();\n    }\n    virtual future<struct stat> statat(std::string_view, int) override {\n        abort();\n    }\n    virtual future<> truncate(uint64_t length) override {\n        abort();\n    }\n    virtual future<> discard(uint64_t offset, uint64_t length) override {\n        abort();\n    }\n    virtual future<> allocate(uint64_t position, uint64_t length) override {\n        abort();\n    }\n    virtual future<uint64_t> size(void) override {\n        abort();\n    }\n    virtual future<> close() override {\n        abort();\n    }\n    virtual std::unique_ptr<file_handle_impl> dup() override {\n        abort();\n    }\n    virtual subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)> next) override {\n        abort();\n    }\n    virtual future<temporary_buffer<uint8_t>> dma_read_bulk(uint64_t offset, size_t range_size, io_intent*) override {\n        abort();\n    }\n};\n\nSEASTAR_TEST_CASE(test_underlying_file) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, oflags).get();\n        auto close_f = deferred_close(f);\n        auto lf = file(make_shared<test_layered_file>(f));\n        BOOST_CHECK_EQUAL(f.memory_dma_alignment(), lf.memory_dma_alignment());\n        BOOST_CHECK_EQUAL(f.disk_read_dma_alignment(), lf.disk_read_dma_alignment());\n        BOOST_CHECK_EQUAL(f.disk_write_dma_alignment(), lf.disk_write_dma_alignment());\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_stat_method_with_file) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create | open_flags::truncate;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        file ref;\n\n        auto orig_umask = umask(0);\n\n        auto st = with_file(open_file_dma(filename, oflags), [&ref] (file& f) {\n            // make a copy of f to verify f is auto-closed when `with_file` returns.\n            ref = f;\n            return f.stat();\n        }).get();\n        BOOST_CHECK_EQUAL(st.st_mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_file_permissions));\n\n        // verify that the file was auto-closed\n        BOOST_REQUIRE_THROW(ref.stat().get(), std::system_error);\n\n        umask(orig_umask);\n    });\n}\n\nSEASTAR_TEST_CASE(test_open_error_with_file) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto open_file = [&t] (bool do_open) {\n            auto oflags = open_flags::ro;\n            sstring filename = (t.get_path() / \"testfile.tmp\").native();\n            if (do_open) {\n                return open_file_dma(filename, oflags);\n            } else {\n                throw std::runtime_error(\"expected exception\");\n            }\n        };\n        bool got_exception = false;\n\n        BOOST_REQUIRE_NO_THROW(with_file(open_file(true), [] (file& f) {\n                BOOST_REQUIRE(false);\n            }).handle_exception_type([&got_exception] (const std::system_error& e) {\n                got_exception = true;\n                BOOST_REQUIRE(e.code().value() == ENOENT);\n            }).get());\n        BOOST_REQUIRE(got_exception);\n\n        got_exception = false;\n        BOOST_REQUIRE_THROW(with_file(open_file(false), [] (file& f) {\n                BOOST_REQUIRE(false);\n            }).handle_exception_type([&got_exception] (const std::runtime_error& e) {\n                got_exception = true;\n            }).get(), std::runtime_error);\n        BOOST_REQUIRE(!got_exception);\n    });\n}\n\nSEASTAR_TEST_CASE(test_with_file_close_on_failure) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create | open_flags::truncate;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n\n        auto orig_umask = umask(0);\n\n        // error-free case\n        auto ref = with_file_close_on_failure(open_file_dma(filename, oflags), [] (file& f) {\n            return f;\n        }).get();\n        auto st = ref.stat().get();\n        ref.close().get();\n        BOOST_CHECK_EQUAL(st.st_mode & static_cast<mode_t>(file_permissions::all_permissions), static_cast<mode_t>(file_permissions::default_file_permissions));\n\n        // close-on-error case\n        BOOST_REQUIRE_THROW(with_file_close_on_failure(open_file_dma(filename, oflags), [&ref] (file& f) {\n            ref = f;\n            throw std::runtime_error(\"expected exception\");\n        }).get(), std::runtime_error);\n\n        // verify that file was auto-closed on error\n        BOOST_REQUIRE_THROW(ref.stat().get(), std::system_error);\n\n        umask(orig_umask);\n    });\n}\n\nSEASTAR_TEST_CASE(test_nowait_flag_correctness) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto oflags = open_flags::rw | open_flags::create;\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto is_tmpfs = [&] (sstring filename) {\n            struct ::statfs buf;\n            int fd = ::open(filename.c_str(), static_cast<int>(open_flags::ro));\n            SEASTAR_ASSERT(fd != -1);\n            auto r = ::fstatfs(fd, &buf);\n            if (r == -1) {\n                return false;\n            }\n            return buf.f_type == internal::fs_magic::tmpfs;\n        };\n\n        if (!seastar::reactor::test::linux_aio_nowait()) {\n            BOOST_TEST_WARN(0, \"Skipping this test because RWF_NOWAIT is not supported by the system\");\n            return;\n        }\n\n        auto f = open_file_dma(filename, oflags).get();\n        auto close_f = deferred_close(f);\n\n        if (is_tmpfs(filename)) {\n            BOOST_TEST_WARN(0, \"Skipping this test because TMPFS was detected, and RWF_NOWAIT is only supported by disk-based FSes\");\n            return;\n        }\n\n        for (auto i = 0; i < 10; i++) {\n            auto wbuf = allocate_aligned_buffer<unsigned char>(4096, 4096);\n            std::fill(wbuf.get(), wbuf.get() + 4096, i);\n            auto wb = wbuf.get();\n            f.dma_write(i * 4096, wb, 4096).get();\n            f.flush().get();\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_destruct_just_constructed_append_challenged_file) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto oflags = open_flags::rw | open_flags::create;\n        auto f = open_file_dma(filename, oflags).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_destruct_just_constructed_append_challenged_file_with_sloppy_size) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto oflags = open_flags::rw | open_flags::create;\n        file_open_options opt;\n        opt.sloppy_size = true;\n        auto f = open_file_dma(filename, oflags, opt).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_destruct_append_challenged_file_after_write) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto buf = allocate_aligned_buffer<unsigned char>(4096, 4096);\n        std::fill(buf.get(), buf.get() + 4096, 0);\n\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n        f.dma_write(0, buf.get(), 4096).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_destruct_append_challenged_file_after_read) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto buf = allocate_aligned_buffer<unsigned char>(4096, 4096);\n        std::fill(buf.get(), buf.get() + 4096, 0);\n\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n        f.dma_write(0, buf.get(), 4096).get();\n        f.flush().get();\n        f.close().get();\n\n        f = open_file_dma(filename, open_flags::rw).get();\n        f.dma_read(0, buf.get(), 4096).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_dma_iovec) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        static constexpr size_t alignment = 4096;\n        auto wbuf = allocate_aligned_buffer<char>(alignment, alignment);\n        size_t size = 1234;\n        std::fill_n(wbuf.get(), alignment, char(0));\n        std::fill_n(wbuf.get(), size, char(42));\n        std::vector<iovec> iovecs;\n\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n        iovecs.push_back(iovec{ wbuf.get(), alignment });\n        auto count = f.dma_write(0, iovecs).get();\n        BOOST_REQUIRE_EQUAL(count, alignment);\n        f.truncate(size).get();\n        f.close().get();\n\n        auto rbuf = allocate_aligned_buffer<char>(alignment, alignment);\n\n        // this tests the posix_file_impl\n        f = open_file_dma(filename, open_flags::ro).get();\n        std::fill_n(rbuf.get(), alignment, char(0));\n        iovecs.clear();\n        iovecs.push_back(iovec{ rbuf.get(), alignment });\n        count = f.dma_read(0, iovecs).get();\n        BOOST_REQUIRE_EQUAL(count, size);\n\n        BOOST_REQUIRE(std::equal(wbuf.get(), wbuf.get() + alignment, rbuf.get(), rbuf.get() + alignment));\n\n        // this tests the append_challenged_posix_file_impl\n        f = open_file_dma(filename, open_flags::rw).get();\n        std::fill_n(rbuf.get(), alignment, char(0));\n        iovecs.clear();\n        iovecs.push_back(iovec{ rbuf.get(), alignment });\n        count = f.dma_read(0, iovecs).get();\n        BOOST_REQUIRE_EQUAL(count, size);\n\n        BOOST_REQUIRE(std::equal(wbuf.get(), wbuf.get() + alignment, rbuf.get(), rbuf.get() + alignment));\n    });\n}\n\nSEASTAR_TEST_CASE(test_intent) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n        auto buf = allocate_aligned_buffer<unsigned char>(1024, 1024);\n        std::fill(buf.get(), buf.get() + 1024, 'a');\n        f.dma_write(0, buf.get(), 1024).get();\n        std::fill(buf.get(), buf.get() + 1024, 'b');\n        io_intent intent;\n        auto f1 = f.dma_write(0, buf.get(), 512);\n        auto f2 = f.dma_write(512, buf.get(), 512, &intent);\n        intent.cancel();\n\n        bool cancelled = false;\n        f1.get();\n        try {\n            f2.get();\n        } catch (cancelled_error& ex) {\n            cancelled = true;\n        }\n        auto rbuf = allocate_aligned_buffer<unsigned char>(1024, 1024);\n        f.dma_read(0, rbuf.get(), 1024).get();\n        BOOST_REQUIRE(rbuf.get()[0] == 'b');\n        if (cancelled) {\n            BOOST_REQUIRE(rbuf.get()[512] == 'a');\n        } else {\n            // The file::dma_write doesn't preemt, but if it\n            // suddenly will, the 2nd write will pass before\n            // the intent would be cancelled\n            BOOST_TEST_WARN(0, \"Write won the race with cancellation\");\n            BOOST_REQUIRE(rbuf.get()[512] == 'b');\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(parallel_overwrite) {\n    // Avoid /tmp for tmp_dir, since it can be tmpfs\n    return tmp_dir::do_with(\"XXXXXXXX.tmp\", [] (tmp_dir& t) {\n        return async([&] {\n            // Check that overwrites at disk_overwrite_dma_alignment() do not cause stalls. First,\n            // create a file.\n            auto fname = (t.get_path() / \"testfile.tmp\").native();\n            auto sz = uint64_t(1*1024*1024);\n            auto buffer_size = 128*1024;\n\n            file f = open_file_dma(fname, open_flags::rw | open_flags::create | open_flags::truncate).get();\n            // Avoid filesystem problems with size-extending operations\n            f.truncate(sz).get();\n            auto buf = allocate_aligned_buffer<unsigned char>(buffer_size, f.memory_dma_alignment());\n            for (uint64_t offset = 0; offset < sz; offset += buffer_size) {\n                f.dma_write(offset, buf.get(), buffer_size).get();\n            }\n\n            auto random_engine = testing::local_random_engine;\n            auto dist = std::uniform_int_distribution(uint64_t(0), sz-1);\n            auto offsets  = std::vector<uint64_t>();\n            std::generate_n(std::back_insert_iterator(offsets), 5000, [&] { return align_down(dist(random_engine), f.disk_overwrite_dma_alignment()); });\n            auto stall_report = internal::report_reactor_stalls([&] {\n                return max_concurrent_for_each(offsets, 10, [&] (uint64_t offset) {\n                    return f.dma_write(offset, buf.get(), f.disk_overwrite_dma_alignment()).discard_result();\n                });\n            }).get();\n            std::cout << \"parallel_overwrite: \" << stall_report << \" (overwrite dma alignment \" << f.disk_overwrite_dma_alignment() << \")\\n\";\n\n            f.close().get();\n            remove_file(fname).get();\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_oversized_io_works) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        sstring filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::rw | open_flags::create).get();\n\n        size_t max_write = f.disk_write_max_length();\n        size_t max_read = f.disk_read_max_length();\n        size_t buf_size = std::max(max_write, max_read) + 4096;\n\n        auto buf = allocate_aligned_buffer<unsigned char>(buf_size, 4096);\n        std::fill(buf.get(), buf.get() + buf_size, 'a');\n\n        f.dma_write(0, buf.get(), buf_size).get();\n        f.flush().get();\n        f.close().get();\n\n        std::fill(buf.get(), buf.get() + buf_size, 'b');\n        f = open_file_dma(filename, open_flags::rw).get();\n        f.dma_read(0, buf.get(), buf_size).get();\n\n        BOOST_REQUIRE((size_t)std::count_if(buf.get(), buf.get() + buf_size, [](auto x) { return x == 'a'; }) == buf_size);\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_system_space) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        const auto& name = t.get_path().native();\n        auto st = engine().statvfs(name).get();\n        auto si = file_system_space(name).get();\n\n        BOOST_REQUIRE_EQUAL(st.f_blocks * st.f_frsize, si.capacity);\n        BOOST_REQUIRE_LE(si.free, si.capacity);\n        BOOST_REQUIRE_LE(si.available, si.capacity);\n        BOOST_REQUIRE_LE(si.available, si.free);\n    });\n}\n\n#include \"../src/core/file-impl.hh\"\n\nnamespace seastar::testing {\nclass append_challenged_posix_file_test {\n    const unsigned _max_appends;\n    const bool _fsync_is_exclusive;\n    file_desc _fd;\n    shared_ptr<append_challenged_posix_file_impl> _file;\n    static constexpr size_t _block_size = 1024;\n    uint64_t _pos = 0;\n    unsigned _appends = 0;\n    unsigned _writes = 0;\n    unsigned _flushes = 0;\n    using opcode = append_challenged_posix_file_impl::opcode;\n\n    struct req {\n        unsigned& counter;\n        uint64_t pos;\n        promise<> complete;\n        req(unsigned& c, uint64_t p = 0) noexcept : counter(c), pos(p), complete() {\n            counter++;\n        }\n    };\n\n    std::deque<req> _queue;\n\npublic:\n    append_challenged_posix_file_test(tmp_dir& t, unsigned append_concurrency, bool fsync_is_exclusive)\n        : _max_appends(append_concurrency)\n        , _fsync_is_exclusive(fsync_is_exclusive)\n        , _fd(file_desc::open((t.get_path() / \"testfile.tmp\").native(), O_RDWR | O_CREAT | O_TRUNC, 0600))\n        , _file(seastar::testing::make_append_challenged_posix_file(_fd, _max_appends, _fsync_is_exclusive, {}))\n    {\n    }\n\n    future<> flush() {\n        return _file->enqueue(opcode::flush, 0, 0, [this] {\n            if (_fsync_is_exclusive) {\n                BOOST_CHECK_EQUAL(_writes, 0);\n            }\n            _queue.emplace_back(_flushes);\n            return _queue.back().complete.get_future();\n        });\n    }\n\n    future<> write_one() {\n        auto pos = _pos;\n        _pos += _block_size;\n        return _file->enqueue(opcode::write, pos, _block_size, [this, pos] {\n            uint64_t fsize = _fd.size();\n            if (pos + _block_size > fsize) {\n                _appends++;\n                BOOST_CHECK_LE(_appends, _max_appends);\n            }\n            if (_fsync_is_exclusive) {\n                BOOST_CHECK_EQUAL(_flushes, 0);\n            }\n            _queue.emplace_back(_writes, pos);\n            return _queue.back().complete.get_future();\n        });\n    }\n\n    bool complete_one() {\n        if (!_queue.empty()) {\n            auto r = std::move(_queue.front());\n            _queue.pop_front();\n            r.counter--;\n            r.complete.set_value();\n            return true;\n        }\n\n        return false;\n    }\n\n    future<> close() {\n        BOOST_CHECK_EQUAL(_appends, _max_appends);\n        return _file->close();\n    }\n};\n}\n\nstatic void test_append_challenged_posix_file_concurrency(tmp_dir& t, unsigned c) {\n    seastar::testing::append_challenged_posix_file_test test(t, c, false);\n\n    std::deque<future<>> futs;\n    for (unsigned i = 0; i < 64; i++) {\n        futs.push_back(test.write_one());\n    }\n\n    while (!futs.empty()) {\n        while (!test.complete_one()) {\n            yield().get();\n        }\n        futs.front().get();\n        futs.pop_front();\n    }\n    test.close().get();\n}\n\nstatic void test_append_challenged_posix_file_flush(tmp_dir& t) {\n    seastar::testing::append_challenged_posix_file_test test(t, 1, true);\n\n    std::deque<future<>> futs;\n    for (unsigned i = 0; i < 4; i++) {\n        futs.push_back(test.write_one());\n    }\n    futs.push_back(test.flush());\n    for (unsigned i = 0; i < 4; i++) {\n        futs.push_back(test.write_one());\n    }\n\n    while (!futs.empty()) {\n        while (!test.complete_one()) {\n            yield().get();\n        }\n        futs.front().get();\n        futs.pop_front();\n    }\n    test.close().get();\n}\n\nSEASTAR_TEST_CASE(test_append_challenged_posix_file_impl) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        test_append_challenged_posix_file_concurrency(t, 0);\n        test_append_challenged_posix_file_concurrency(t, 1);\n        test_append_challenged_posix_file_flush(t);\n    });\n}\n\n// Helper class for test_posix_file_impl.\n// Copies data from external buffer in it read() operation, but\n// doesn't provide more than a single block at one, to force the\n// do_dma_read_bulk() issue several reads\nclass test_posix_file_impl : public posix_file_impl {\n    const temporary_buffer<char>& _data;\npublic:\n    static constexpr size_t block_size = 512;\n    virtual future<size_t> write_dma(uint64_t pos, const void* buffer, size_t len, io_intent* i) noexcept override { abort(); }\n    virtual future<size_t> write_dma(uint64_t pos, std::vector<iovec> iov, io_intent* i) noexcept override { abort(); }\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent* i) noexcept override { abort(); }\n    virtual std::unique_ptr<seastar::file_handle_impl> dup() override { abort(); }\n\n    virtual future<size_t> read_dma(uint64_t pos, void* buffer, size_t len, io_intent* i) noexcept override {\n        fmt::print(\"  IO {}:{}\\n\", pos, len);\n\n        if (pos >= _data.size()) {\n            return make_ready_future<size_t>(0);\n        }\n\n        BOOST_REQUIRE(!(pos & (block_size - 1)));\n        BOOST_REQUIRE(!(len & (block_size - 1)));\n        BOOST_REQUIRE(!((uintptr_t)(buffer) & (block_size - 1)));\n\n        size_t avail = std::min(_data.size() - pos, size_t(block_size));\n        size_t rlen = std::min(len, avail);\n        std::memcpy(buffer, _data.get() + pos, rlen);\n        return make_ready_future<size_t>(rlen);\n    }\n\n    test_posix_file_impl(const temporary_buffer<char>& d)\n        : posix_file_impl(0, {}, nullptr, 0, block_size, block_size, block_size, block_size, true, true, true)\n        , _data(d)\n    {}\n};\n\n// Test that posix file bulk reading code works\n// Prepares a lengthy buffer and a file reading from it, then\n// runs a test with various (aligned and (!) not) offsets and\n// lengths, checking that the returned buffer for sanity and\n// to contain the bytes that it claims to contain\nSEASTAR_THREAD_TEST_CASE(test_posix_file_dma_read_bulk) {\n    static constexpr size_t data_size = 5 * test_posix_file_impl::block_size + 123;\n    temporary_buffer<char> data(data_size);\n    auto random_engine = testing::local_random_engine;\n    auto dist = std::uniform_int_distribution<char>();\n    std::ranges::generate(data.get_write(), data.end(), [&] { return dist(random_engine); });\n    auto f = std::make_unique<test_posix_file_impl>(data);\n\n    auto read_and_validate = [&] (size_t offset, size_t length) {\n        fmt::print(\"Check {}:{}\\n\", offset, length);\n        auto buf = f->dma_read_bulk(offset, length, nullptr).get();\n        fmt::print(\" read returned {} bytes\\n\", buf.size());\n        if (offset >= data.size()) {\n            // read past EOF --> empty buffer\n            BOOST_REQUIRE_EQUAL(buf.size(), 0);\n        } else {\n            size_t avail = std::min(data.size() - offset, length);\n            // must have read at least what's available\n            BOOST_REQUIRE_GE(buf.size(), avail);\n            // no read past EOF\n            BOOST_REQUIRE_LE(offset + buf.size(), data.size());\n            BOOST_REQUIRE(std::memcmp(buf.get(), data.get() + offset, buf.size()) == 0);\n        }\n    };\n\n    for (size_t offset : {size_t(0), test_posix_file_impl::block_size, 3 * test_posix_file_impl::block_size, data_size - test_posix_file_impl::block_size, data_size}) {\n        for (size_t len : { test_posix_file_impl::block_size, test_posix_file_impl::block_size * 3 }) {\n            for (ssize_t mis_off : { 0, -41, 87 }) {\n                for (ssize_t mis_len : { 0, -63, 25 }) {\n                    if (ssize_t(offset) + mis_off < 0) {\n                        continue;\n                    }\n\n                    read_and_validate(offset + mis_off, len + mis_len);\n                }\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "tests/unit/file_utils_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB\n */\n\n#include <stdlib.h>\n#include <random>\n\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n\n#include <seastar/core/file.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/tmp_file.hh>\n#include <seastar/util/file.hh>\n\n#include \"expected_exception.hh\"\n\nusing namespace seastar;\nnamespace fs = std::filesystem;\n\nSEASTAR_TEST_CASE(test_make_tmp_file) {\n    return make_tmp_file().then([] (tmp_file tf) {\n        return async([tf = std::move(tf)] () mutable {\n            const sstring tmp_path = tf.get_path().native();\n            BOOST_REQUIRE(file_exists(tmp_path).get());\n            tf.close().get();\n            tf.remove().get();\n            BOOST_REQUIRE(!file_exists(tmp_path).get());\n        });\n    });\n}\n\nstatic temporary_buffer<char> get_init_buffer(file& f) {\n    auto buf = temporary_buffer<char>::aligned(f.memory_dma_alignment(), f.memory_dma_alignment());\n    memset(buf.get_write(), 0, buf.size());\n    return buf;\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tmp_file) {\n    size_t expected = ~0;\n    size_t actual = 0;\n\n    tmp_file::do_with([&] (tmp_file& tf) mutable {\n        auto& f = tf.get_file();\n        auto buf = get_init_buffer(f);\n        return do_with(std::move(buf), [&] (auto& buf) mutable {\n            expected = buf.size();\n            return f.dma_write(0, buf.get(), buf.size()).then([&] (size_t written) {\n                actual = written;\n                return make_ready_future<>();\n            });\n        });\n    }).get();\n    BOOST_REQUIRE_EQUAL(expected , actual);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_non_existing_TMPDIR) {\n    BOOST_REQUIRE_EXCEPTION(tmp_file::do_with(\"/tmp/non-existing-TMPDIR\", [] (tmp_file& tf) {}).get(),\n            std::system_error, testing::exception_predicate::message_contains(\"No such file or directory\"));\n}\n\nstatic future<> touch_file(const sstring& filename, open_flags oflags = open_flags::rw | open_flags::create) noexcept {\n    return open_file_dma(filename, oflags).then([] (file f) {\n        return f.close().finally([f] {});\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_recursive_remove_directory) {\n    struct test_dir {\n        test_dir *parent;\n        sstring name;\n        std::list<sstring> sub_files = {};\n        std::list<test_dir> sub_dirs = {};\n\n        test_dir(test_dir* parent, sstring name)\n            : parent(parent)\n            , name(std::move(name))\n        { }\n\n        fs::path path() const {\n            if (!parent) {\n                return fs::path(name.c_str());\n            }\n            return parent->path() / name.c_str();\n        }\n\n        void fill_random_file(std::uniform_int_distribution<unsigned>& dist, std::default_random_engine& eng) {\n            sub_files.emplace_back(format(\"file-{}\", dist(eng)));\n        }\n\n        test_dir& fill_random_dir(std::uniform_int_distribution<unsigned>& dist, std::default_random_engine& eng) {\n            sub_dirs.emplace_back(this, format(\"dir-{}\", dist(eng)));\n            return sub_dirs.back();\n        }\n\n        void random_fill(int level, int levels, std::uniform_int_distribution<unsigned>& dist, std::default_random_engine& eng) {\n            int num_files = dist(eng) % 10;\n            int num_dirs = (level < levels - 1) ? (1 + dist(eng) % 3) : 0;\n\n            for (int i = 0; i < num_files; i++) {\n                fill_random_file(dist, eng);\n            }\n\n            if (num_dirs) {\n                level++;\n                for (int i = 0; i < num_dirs; i++) {\n                    fill_random_dir(dist, eng).random_fill(level, levels, dist, eng);\n                }\n            }\n        }\n\n        future<> populate() {\n            return touch_directory(path().native()).then([this] {\n                return parallel_for_each(sub_files, [this] (auto& name) {\n                    return touch_file((path() / name.c_str()).native());\n                }).then([this] {\n                    return parallel_for_each(sub_dirs, [] (auto& sub_dir) {\n                        return sub_dir.populate();\n                    });\n                });\n            });\n        }\n    };\n\n    auto& eng = testing::local_random_engine;\n    auto dist = std::uniform_int_distribution<unsigned>();\n    int levels = 1 + dist(eng) % 3;\n    test_dir root = { nullptr, default_tmpdir().native() };\n    test_dir base = { &root, format(\"base-{}\", dist(eng)) };\n    base.random_fill(0, levels, dist, eng);\n    base.populate().get();\n    recursive_remove_directory(base.path()).get();\n    BOOST_REQUIRE(!file_exists(base.path().native()).get());\n}\n\nSEASTAR_TEST_CASE(test_make_tmp_dir) {\n    return make_tmp_dir().then([] (tmp_dir td) {\n        return async([td = std::move(td)] () mutable {\n            const sstring tmp_path = td.get_path().native();\n            BOOST_REQUIRE(file_exists(tmp_path).get());\n            td.remove().get();\n            BOOST_REQUIRE(!file_exists(tmp_path).get());\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tmp_dir) {\n    size_t expected;\n    size_t actual;\n    tmp_dir::do_with([&] (tmp_dir& td) {\n        return tmp_file::do_with(td.get_path(), [&] (tmp_file& tf) {\n            auto& f = tf.get_file();\n            auto buf = get_init_buffer(f);\n            return do_with(std::move(buf), [&] (auto& buf) mutable {\n                expected = buf.size();\n                return f.dma_write(0, buf.get(), buf.size()).then([&] (size_t written) {\n                    actual = written;\n                    return make_ready_future<>();\n                });\n            });\n        });\n    }).get();\n    BOOST_REQUIRE_EQUAL(expected , actual);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tmp_dir_with_path) {\n    size_t expected;\n    size_t actual;\n    tmp_dir::do_with(\".\", [&] (tmp_dir& td) {\n        return tmp_file::do_with(td.get_path(), [&] (tmp_file& tf) {\n            auto& f = tf.get_file();\n            auto buf = get_init_buffer(f);\n            return do_with(std::move(buf), [&] (auto& buf) mutable {\n                expected = buf.size();\n                return tf.get_file().dma_write(0, buf.get(), buf.size()).then([&] (size_t written) {\n                    actual = written;\n                    return make_ready_future<>();\n                });\n            });\n        });\n    }).get();\n    BOOST_REQUIRE_EQUAL(expected , actual);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tmp_dir_with_non_existing_path) {\n    BOOST_REQUIRE_EXCEPTION(tmp_dir::do_with(\"/tmp/this_name_should_not_exist\", [] (tmp_dir&) {}).get(),\n            std::system_error, testing::exception_predicate::message_contains(\"No such file or directory\"));\n}\n\nSEASTAR_TEST_CASE(tmp_dir_with_thread_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& td) {\n        tmp_file tf = make_tmp_file(td.get_path()).get();\n        auto& f = tf.get_file();\n        auto buf = get_init_buffer(f);\n        auto expected = buf.size();\n        auto actual = f.dma_write(0, buf.get(), buf.size()).get();\n        BOOST_REQUIRE_EQUAL(expected, actual);\n        tf.close().get();\n        tf.remove().get();\n    });\n}\n\nSEASTAR_TEST_CASE(tmp_dir_with_leftovers_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& td) {\n        fs::path path = td.get_path() / \"testfile.tmp\";\n        touch_file(path.native()).get();\n        BOOST_REQUIRE(file_exists(path.native()).get());\n    });\n}\n\nSEASTAR_TEST_CASE(tmp_dir_do_with_fail_func_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& outer) {\n        BOOST_REQUIRE_THROW(tmp_dir::do_with([] (tmp_dir& inner) mutable {\n            return make_exception_future<>(expected_exception());\n        }).get(), expected_exception);\n    });\n}\n\nSEASTAR_TEST_CASE(tmp_dir_do_with_fail_remove_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& outer) {\n        auto saved_default_tmpdir = default_tmpdir();\n        sstring outer_path = outer.get_path().native();\n        sstring inner_path;\n        sstring inner_path_renamed;\n        set_default_tmpdir(outer_path.c_str());\n        BOOST_REQUIRE_THROW(tmp_dir::do_with([&] (tmp_dir& inner) mutable {\n            inner_path = inner.get_path().native();\n            inner_path_renamed = inner_path + \".renamed\";\n            return rename_file(inner_path, inner_path_renamed);\n        }).get(), std::system_error);\n        BOOST_REQUIRE(!file_exists(inner_path).get());\n        BOOST_REQUIRE(file_exists(inner_path_renamed).get());\n        set_default_tmpdir(saved_default_tmpdir.c_str());\n    });\n}\n\nSEASTAR_TEST_CASE(tmp_dir_do_with_thread_fail_func_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& outer) {\n        BOOST_REQUIRE_THROW(tmp_dir::do_with_thread([] (tmp_dir& inner) mutable {\n            throw expected_exception();\n        }).get(), expected_exception);\n    });\n}\n\nSEASTAR_TEST_CASE(tmp_dir_do_with_thread_fail_remove_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& outer) {\n        auto saved_default_tmpdir = default_tmpdir();\n        sstring outer_path = outer.get_path().native();\n        sstring inner_path;\n        sstring inner_path_renamed;\n        set_default_tmpdir(outer_path.c_str());\n        BOOST_REQUIRE_THROW(tmp_dir::do_with_thread([&] (tmp_dir& inner) mutable {\n            inner_path = inner.get_path().native();\n            inner_path_renamed = inner_path + \".renamed\";\n            return rename_file(inner_path, inner_path_renamed);\n        }).get(), std::system_error);\n        BOOST_REQUIRE(!file_exists(inner_path).get());\n        BOOST_REQUIRE(file_exists(inner_path_renamed).get());\n        set_default_tmpdir(saved_default_tmpdir.c_str());\n    });\n}\n\nSEASTAR_TEST_CASE(test_read_entire_file_contiguous) {\n    return tmp_file::do_with([] (tmp_file& tf) {\n        return async([&tf] {\n            file& f = tf.get_file();\n            auto& eng = testing::local_random_engine;\n            auto dist = std::uniform_int_distribution<unsigned>();\n            size_t size = f.memory_dma_alignment() * (1 + dist(eng) % 1000);\n            auto wbuf = temporary_buffer<char>::aligned(f.memory_dma_alignment(), size);\n            for (size_t i = 0; i < size; i++) {\n                static char chars[] = \"abcdefghijklmnopqrstuvwxyz0123456789\";\n                wbuf.get_write()[i] = chars[dist(eng) % sizeof(chars)];\n            }\n\n            BOOST_REQUIRE_EQUAL(f.dma_write(0, wbuf.begin(), wbuf.size()).get(), wbuf.size());\n            f.flush().get();\n\n            sstring res = util::read_entire_file_contiguous(tf.get_path()).get();\n            BOOST_REQUIRE_EQUAL(res, std::string_view(wbuf.begin(), wbuf.size()));\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/foreign_ptr_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <iostream>\n\nusing namespace seastar;\n\nnamespace seastar {\n\nextern logger seastar_logger;\n\n}\n\nSEASTAR_TEST_CASE(make_foreign_ptr_from_lw_shared_ptr) {\n    auto p = make_foreign(make_lw_shared<sstring>(\"foo\"));\n    BOOST_REQUIRE(p->size() == 3);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(make_foreign_ptr_from_shared_ptr) {\n    auto p = make_foreign(make_shared<sstring>(\"foo\"));\n    BOOST_REQUIRE(p->size() == 3);\n    return make_ready_future<>();\n}\n\n\nSEASTAR_TEST_CASE(foreign_ptr_copy_test) {\n    return seastar::async([] {\n        auto ptr = make_foreign(make_shared<sstring>(\"foo\"));\n        BOOST_REQUIRE(ptr->size() == 3);\n        auto ptr2 = ptr.copy().get();\n        BOOST_REQUIRE(ptr2->size() == 3);\n    });\n}\n\nSEASTAR_TEST_CASE(foreign_ptr_get_test) {\n    auto p = make_foreign(std::make_unique<sstring>(\"foo\"));\n    BOOST_REQUIRE_EQUAL(p.get(), &*p);\n    return make_ready_future<>();\n};\n\nSEASTAR_TEST_CASE(foreign_ptr_release_test) {\n    auto p = make_foreign(std::make_unique<sstring>(\"foo\"));\n    auto raw_ptr = p.get();\n    BOOST_REQUIRE(bool(p));\n    BOOST_REQUIRE(p->size() == 3);\n    auto released_p = p.release();\n    BOOST_REQUIRE(!bool(p));\n    BOOST_REQUIRE(released_p->size() == 3);\n    BOOST_REQUIRE_EQUAL(raw_ptr, released_p.get());\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(foreign_ptr_reset_test) {\n    auto fp = make_foreign(std::make_unique<sstring>(\"foo\"));\n    BOOST_REQUIRE(bool(fp));\n    BOOST_REQUIRE(fp->size() == 3);\n\n    fp.reset(std::make_unique<sstring>(\"foobar\"));\n    BOOST_REQUIRE(bool(fp));\n    BOOST_REQUIRE(fp->size() == 6);\n\n    fp.reset();\n    BOOST_REQUIRE(!bool(fp));\n    return make_ready_future<>();\n}\n\nclass dummy {\n    unsigned _cpu;\npublic:\n    dummy() : _cpu(this_shard_id()) { }\n    ~dummy() { BOOST_REQUIRE_EQUAL(_cpu, this_shard_id()); }\n};\n\nSEASTAR_TEST_CASE(foreign_ptr_cpu_test) {\n    if (smp::count == 1) {\n        std::cerr << \"Skipping multi-cpu foreign_ptr tests. Run with --smp=2 to test multi-cpu delete and reset.\";\n        return make_ready_future<>();\n    }\n\n    using namespace std::chrono_literals;\n\n    return seastar::async([] {\n        auto p = smp::submit_to(1, [] {\n            return make_foreign(std::make_unique<dummy>());\n        }).get();\n\n        p.reset(std::make_unique<dummy>());\n    }).then([] {\n        // Let ~foreign_ptr() take its course. RIP dummy.\n        return seastar::sleep(100ms);\n    });\n}\n\nSEASTAR_TEST_CASE(foreign_ptr_move_assignment_test) {\n    if (smp::count == 1) {\n        std::cerr << \"Skipping multi-cpu foreign_ptr tests. Run with --smp=2 to test multi-cpu delete and reset.\";\n        return make_ready_future<>();\n    }\n\n    using namespace std::chrono_literals;\n\n    return seastar::async([] {\n        auto p = smp::submit_to(1, [] {\n            return make_foreign(std::make_unique<dummy>());\n        }).get();\n\n        p = foreign_ptr<std::unique_ptr<dummy>>();\n    }).then([] {\n        // Let ~foreign_ptr() take its course. RIP dummy.\n        return seastar::sleep(100ms);\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(foreign_ptr_destroy_test) {\n    if (smp::count == 1) {\n        std::cerr << \"Skipping multi-cpu foreign_ptr tests. Run with --smp=2 to test multi-cpu delete and reset.\";\n        return;\n    }\n\n    using namespace std::chrono_literals;\n\n    std::vector<promise<bool>> done;\n    done.resize(smp::count);\n\n    struct deferred {\n        std::vector<promise<bool>>& done;\n        deferred(std::vector<promise<bool>>& done_)\n            : done(done_)\n        {}\n        ~deferred() {\n            seastar_logger.info(\"~deferred\");\n            internal::run_in_background([&done = done, shard = this_shard_id()] {\n                return smp::submit_to(0, [&done, shard] {\n                    done[shard].set_value(true);\n                    done[shard ^ 1].set_value(false);\n                });\n            });\n        }\n    };\n\n    auto val = smp::submit_to(1, [&] () mutable {\n        return make_foreign(std::make_unique<deferred>(done));\n    }).get();\n\n    val.destroy().get();\n\n    BOOST_REQUIRE_EQUAL(done[1].get_future().get(), true);\n    BOOST_REQUIRE_EQUAL(done[0].get_future().get(), false);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_foreign_ptr_use_count) {\n    shard_id shard = (this_shard_id() + 1) % smp::count;\n    auto p0 = smp::submit_to(shard, [] {\n        return make_foreign(make_lw_shared<sstring>(\"foo\"));\n    }).get();\n    smp::submit_to(shard, [&] {\n        BOOST_REQUIRE_EQUAL(p0.unwrap_on_owner_shard().use_count(), 1);\n    }).get();\n    auto p1 = p0.copy().get();\n    smp::submit_to(shard, [&] {\n        BOOST_REQUIRE_EQUAL(p0.unwrap_on_owner_shard().use_count(), 2);\n    }).get();\n    smp::submit_to(shard, [&] {\n        auto ptr = p0.release();\n        BOOST_REQUIRE_EQUAL(p0.unwrap_on_owner_shard().use_count(), 0);\n        BOOST_REQUIRE_EQUAL(p1.unwrap_on_owner_shard().use_count(), 2);\n        ptr = {};\n        BOOST_REQUIRE_EQUAL(p1.unwrap_on_owner_shard().use_count(), 1);\n    }).get();\n    p1.reset();\n    smp::submit_to(shard, [&] {\n        BOOST_REQUIRE_EQUAL(p0.unwrap_on_owner_shard().use_count(), 0);\n    }).get();\n}\n"
  },
  {
    "path": "tests/unit/fsnotifier_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 ScyllaDB Ltd.\n */\n\n#include <random>\n#include <algorithm>\n\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/core/fsnotify.hh>\n\n#include \"tmpdir.hh\"\n\nnamespace fs = std::filesystem;\nusing namespace seastar;\nusing experimental::fsnotifier;\n\nstatic bool find_event(const std::vector<fsnotifier::event>& events, const fsnotifier::watch& w, fsnotifier::flags mask, std::optional<sstring> path = {}) {\n    auto i = std::find_if(events.begin(), events.end(), [&](const fsnotifier::event& e) {\n        return (e.mask & mask) != fsnotifier::flags{}\n            && e.id == w\n            && (!path || *path == e.name)\n            ;\n    });\n    return i != events.end();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_notify_modify_close_delete) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n    auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n    auto w = fsn.create_watch(p.native(), fsnotifier::flags::delete_self\n        | fsnotifier::flags::modify\n        | fsnotifier::flags::close\n    ).get();\n\n    auto os = make_file_output_stream(f).get();\n    os.write(\"kossa\").get();\n    os.flush().get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::modify));\n    }\n\n    os.close().get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::close_write));\n    }\n\n    remove_file(p.native()).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::delete_self));\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::ignored));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_notify_overwrite) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n\n    auto write_file = [](fs::path& p, sstring content) {\n        auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n        auto os = make_file_output_stream(f).get();\n        os.write(content).get();\n        os.flush().get();\n        os.close().get();\n    };\n\n    write_file(p, \"kossa\");\n\n    auto w = fsn.create_watch(p.native(), fsnotifier::flags::delete_self\n        | fsnotifier::flags::modify\n        | fsnotifier::flags::close\n    ).get();\n\n    write_file(p, \"kossabello\");\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::modify));\n    }\n\n    write_file(p, \"kossaruffalobill\");\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::modify));\n    }\n\n    auto p2 = tmp.path() / \"tmp.apa\";\n    write_file(p2, \"le apa\");\n\n    auto w2 = fsn.create_watch(tmp.path().native(), fsnotifier::flags::move_to).get();\n\n    rename_file(p2.native(), p.native()).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::delete_self));\n        BOOST_REQUIRE(find_event(events, w2, fsnotifier::flags::move_to, p.filename().native()));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_notify_create_delete_child) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n    auto w = fsn.create_watch(tmp.path().native(), fsnotifier::flags::create_child\n        | fsnotifier::flags::delete_child\n    ).get();\n\n    auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::create_child));\n    }\n\n    f.close().get();\n    remove_file(p.native()).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::delete_child));\n        BOOST_REQUIRE(!find_event(events, w, fsnotifier::flags::ignored));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_notify_open) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n    auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n    f.close().get();\n\n    auto w = fsn.create_watch(p.native(), fsnotifier::flags::open).get();\n\n    auto f2 = open_file_dma(p.native(), open_flags::ro).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::open));\n    }\n\n    f2.close().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_notify_move) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n    auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n\n    f.close().get();\n\n    auto w = fsn.create_watch(tmp.path().native(), fsnotifier::flags::move).get();\n    auto p2 = tmp.path() / \"kossa.mu\";\n\n    rename_file(p.native(), p2.native()).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::move_from, p.filename().native()));\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::move_to, p2.filename().native()));\n    }\n\n    tmpdir tmp2;\n    auto p3 = tmp2.path() / \"ninja.mission\";\n    auto w2 = fsn.create_watch(tmp2.path().native(), fsnotifier::flags::move).get();\n\n    rename_file(p2.native(), p3.native()).get();\n\n    {\n        auto events = fsn.wait().get();\n        BOOST_REQUIRE(find_event(events, w, fsnotifier::flags::move_from, p2.filename().native()));\n        BOOST_REQUIRE(find_event(events, w2, fsnotifier::flags::move_to, p3.filename().native()));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shutdown_notifier) {\n    tmpdir tmp;\n    fsnotifier fsn;\n\n    auto p = tmp.path() / \"kossa.dat\";\n    auto f = open_file_dma(p.native(), open_flags::create|open_flags::rw).get();\n\n    f.close().get();\n\n    auto w = fsn.create_watch(tmp.path().native(), fsnotifier::flags::delete_child).get();\n    auto fut = fsn.wait();\n\n    fsn.shutdown();\n\n    auto events = fut.get();\n    BOOST_REQUIRE(events.empty());\n}\n"
  },
  {
    "path": "tests/unit/fstream_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <algorithm>\n#include <iostream>\n#include <numeric>\n#include <seastar/core/fstream.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/print.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/tmp_file.hh>\n#include <boost/range/adaptor/transformed.hpp>\n#include <boost/algorithm/cxx11/any_of.hpp>\n#include \"mock_file.hh\"\n#include <boost/range/irange.hpp>\n#include <seastar/util/closeable.hh>\n#include <seastar/util/alloc_failure_injector.hh>\n\nusing namespace seastar;\nnamespace fs = std::filesystem;\n\nstruct writer {\n    output_stream<char> out;\n    static future<shared_ptr<writer>> make(file f) {\n        return make_file_output_stream(std::move(f)).then([] (output_stream<char>&& os) {\n            return make_shared<writer>(writer{std::move(os)});\n        });\n    }\n};\n\nstruct reader {\n    input_stream<char> in;\n    reader(file f) : in(make_file_input_stream(std::move(f))) {}\n    reader(file f, file_input_stream_options options) : in(make_file_input_stream(std::move(f), std::move(options))) {}\n};\n\nSEASTAR_TEST_CASE(test_fstream) {\n    return tmp_dir::do_with([] (tmp_dir& t) {\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        return open_file_dma(filename,\n                open_flags::rw | open_flags::create | open_flags::truncate).then([filename] (file f) {\n            return writer::make(std::move(f)).then([filename] (shared_ptr<writer> w) {\n                auto buf = static_cast<char*>(::malloc(4096));\n                memset(buf, 0, 4096);\n                buf[0] = '[';\n                buf[1] = 'A';\n                buf[4095] = ']';\n                return w->out.write(buf, 4096).then([buf, w] {\n                    ::free(buf);\n                    return make_ready_future<>();\n                }).then([w] {\n                    auto buf = static_cast<char*>(::malloc(8192));\n                    memset(buf, 0, 8192);\n                    buf[0] = '[';\n                    buf[1] = 'B';\n                    buf[8191] = ']';\n                    return w->out.write(buf, 8192).then([buf, w] {\n                        ::free(buf);\n                        return w->out.close().then([w] {});\n                    });\n                }).then([filename] {\n                    return open_file_dma(filename, open_flags::ro);\n                }).then([] (file f) {\n                    /*  file content after running the above:\n                     * 00000000  5b 41 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |[A..............|\n                     * 00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|\n                     * *\n                     * 00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 5d  |...............]|\n                     * 00001000  5b 42 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |[B..............|\n                     * 00001010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|\n                     * *\n                     * 00002ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 5d  |...............]|\n                     * 00003000\n                     */\n                    auto r = make_shared<reader>(std::move(f));\n                    return r->in.read_exactly(4096 + 8192).then([r] (temporary_buffer<char> buf) {\n                        auto p = buf.get();\n                        BOOST_REQUIRE(p[0] == '[' && p[1] == 'A' && p[4095] == ']');\n                        BOOST_REQUIRE(p[4096] == '[' && p[4096 + 1] == 'B' && p[4096 + 8191] == ']');\n                        return make_ready_future<>();\n                    }).then([r] {\n                        return r->in.close();\n                    }).finally([r] {});\n                });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_consume_skip_bytes) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename,\n                               open_flags::rw | open_flags::create | open_flags::truncate).get();\n        auto w = writer::make(std::move(f)).get();\n        auto write_block = [w] (char c, size_t size) {\n            std::vector<char> vec(size, c);\n            w->out.write(&vec.front(), vec.size()).get();\n        };\n        write_block('a', 8192);\n        write_block('b', 8192);\n        w->out.close().get();\n        /*  file content after running the above:\n         * 00000000  61 61 61 61 61 61 61 61  61 61 61 61 61 61 61 61  |aaaaaaaaaaaaaaaa|\n         * *\n         * 00002000  62 62 62 62 62 62 62 62  62 62 62 62 62 62 62 62  |bbbbbbbbbbbbbbbb|\n         * *\n         * 00004000\n         */\n        f = open_file_dma(filename, open_flags::ro).get();\n        auto r = make_lw_shared<reader>(std::move(f), file_input_stream_options{512});\n        auto close_r_in = deferred_close(r->in);\n        struct consumer {\n            uint64_t _count = 0;\n            using consumption_result_type = typename input_stream<char>::consumption_result_type;\n            using stop_consuming_type = typename consumption_result_type::stop_consuming_type;\n            using tmp_buf = stop_consuming_type::tmp_buf;\n\n            /*\n             * Consumer reads the file as follows:\n             *  - first 8000 bytes are read in 512-byte chunks and checked\n             *  - next 2000 bytes are skipped (jumping over both read buffer size and DMA block)\n             *  - the remaining 6384 bytes are read and checked\n             */\n            future<consumption_result_type> operator()(tmp_buf buf) {\n                if (_count < 8000) {\n                    auto delta = std::min(buf.size(), 8000 - _count);\n                    for (auto c : buf.share(0, delta)) {\n                        BOOST_REQUIRE_EQUAL(c, 'a');\n                    }\n                    buf.trim_front(delta);\n                    _count += delta;\n\n                    if (_count == 8000) {\n                        return make_ready_future<consumption_result_type>(skip_bytes{2000 - buf.size()});\n                    } else {\n                        SEASTAR_ASSERT(buf.empty());\n                        return make_ready_future<consumption_result_type>(continue_consuming{});\n                    }\n                    return make_ready_future<consumption_result_type>(continue_consuming{});\n                } else {\n                    for (auto c : buf) {\n                        BOOST_REQUIRE_EQUAL(c, 'b');\n                    }\n                    _count += buf.size();\n                    if (_count < 14384) {\n                        return make_ready_future<consumption_result_type>(continue_consuming{});\n                    } else if (_count > 14384) {\n                        BOOST_FAIL(\"Read more than expected\");\n                    }\n                    return make_ready_future<consumption_result_type>(stop_consuming_type({}));\n                }\n            }\n        };\n        r->in.consume(consumer{}).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_fstream_unaligned) {\n  return tmp_dir::do_with([] (tmp_dir& t) {\n    auto filename = (t.get_path() / \"testfile.tmp\").native();\n    return open_file_dma(filename,\n            open_flags::rw | open_flags::create | open_flags::truncate).then([filename] (file f) {\n        return writer::make(std::move(f)).then([filename] (shared_ptr<writer> w) {\n            auto buf = static_cast<char*>(::malloc(40));\n            memset(buf, 0, 40);\n            buf[0] = '[';\n            buf[1] = 'A';\n            buf[39] = ']';\n            return w->out.write(buf, 40).then([buf, w] {\n                ::free(buf);\n                return w->out.close().then([w] {});\n            }).then([filename] {\n                return open_file_dma(filename, open_flags::ro);\n            }).then([] (file f) {\n                return do_with(std::move(f), [] (file& f) {\n                    return f.size().then([] (size_t size) {\n                        // assert that file was indeed truncated to the amount of bytes written.\n                        BOOST_REQUIRE(size == 40);\n                        return make_ready_future<>();\n                    });\n                });\n            }).then([filename] {\n                return open_file_dma(filename, open_flags::ro);\n            }).then([] (file f) {\n                auto r = make_shared<reader>(std::move(f));\n                return r->in.read_exactly(40).then([r] (temporary_buffer<char> buf) {\n                    auto p = buf.get();\n                    BOOST_REQUIRE(p[0] == '[' && p[1] == 'A' && p[39] == ']');\n                    return make_ready_future<>();\n                }).then([r] {\n                    return r->in.close();\n                }).finally([r] {});\n            });\n        });\n    });\n  });\n}\n\nfuture<> test_consume_until_end(uint64_t size) {\n  return tmp_dir::do_with([size] (tmp_dir& t) {\n    auto filename = (t.get_path() / \"testfile.tmp\").native();\n    return open_file_dma(filename,\n            open_flags::rw | open_flags::create | open_flags::truncate).then([size] (file f) {\n          return make_file_output_stream(f).then([size] (output_stream<char>&& os) {\n            return do_with(std::move(os), [size] (output_stream<char>& out) {\n                std::vector<char> buf(size);\n                std::iota(buf.begin(), buf.end(), 0);\n                return out.write(buf.data(), buf.size()).then([&out] {\n                   return out.flush();\n                });\n          });\n            }).then([f] {\n                return f.size();\n            }).then([size, f] (size_t real_size) {\n                BOOST_REQUIRE_EQUAL(size, real_size);\n            }).then([size, f] {\n                auto consumer = [offset = uint64_t(0), size] (temporary_buffer<char> buf) mutable -> future<input_stream<char>::unconsumed_remainder> {\n                    if (!buf) {\n                        return make_ready_future<input_stream<char>::unconsumed_remainder>(temporary_buffer<char>());\n                    }\n                    BOOST_REQUIRE(offset + buf.size() <= size);\n                    std::vector<char> expected(buf.size());\n                    std::iota(expected.begin(), expected.end(), offset);\n                    offset += buf.size();\n                    BOOST_REQUIRE(std::equal(buf.begin(), buf.end(), expected.begin()));\n                    return make_ready_future<input_stream<char>::unconsumed_remainder>(std::nullopt);\n                };\n                return do_with(make_file_input_stream(f), std::move(consumer), [] (input_stream<char>& in, auto& consumer) {\n                    return in.consume(consumer).then([&in] {\n                        return in.close();\n                    });\n                });\n            }).finally([f] () mutable {\n                return f.close();\n            });\n    });\n  });\n}\n\n\nSEASTAR_TEST_CASE(test_consume_aligned_file) {\n    return test_consume_until_end(4096);\n}\n\nSEASTAR_TEST_CASE(test_consume_empty_file) {\n    return test_consume_until_end(0);\n}\n\nSEASTAR_TEST_CASE(test_consume_unaligned_file) {\n    return test_consume_until_end(1);\n}\n\nSEASTAR_TEST_CASE(test_consume_unaligned_file_large) {\n    return test_consume_until_end((1 << 20) + 1);\n}\n\nSEASTAR_TEST_CASE(test_input_stream_esp_around_eof) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto flen = uint64_t(5341);\n        auto rdist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n        auto reng = testing::local_random_engine;\n        auto data = boost::copy_range<std::vector<uint8_t>>(\n                boost::irange<uint64_t>(0, flen)\n                | boost::adaptors::transformed([&] (int x) { return rdist(reng); }));\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename,\n                open_flags::rw | open_flags::create | open_flags::truncate).get();\n        auto close_f = deferred_close(f);\n        auto out = make_file_output_stream(f).get();\n        out.write(reinterpret_cast<const char*>(data.data()), data.size()).get();\n        out.flush().get();\n        //out.close().get();  // FIXME: closes underlying stream:?!\n        struct range { uint64_t start; uint64_t end; };\n        auto ranges = std::vector<range>{{\n            range{0, flen},\n            range{0, flen * 2},\n            range{0, flen + 1},\n            range{0, flen - 1},\n            range{0, 1},\n            range{1, 2},\n            range{flen - 1, flen},\n            range{flen - 1, flen + 1},\n            range{flen, flen + 1},\n            range{flen + 1, flen + 2},\n            range{1023, flen-1},\n            range{1023, flen},\n            range{1023, flen + 2},\n            range{8193, 8194},\n            range{1023, 1025},\n            range{1023, 1024},\n            range{1024, 1025},\n            range{1023, 4097},\n        }};\n        auto opt = file_input_stream_options();\n        opt.buffer_size = 512;\n        for (auto&& r : ranges) {\n            auto start = r.start;\n            auto end = r.end;\n            auto len = end - start;\n            auto in = make_file_input_stream(f, start, len, opt);\n            std::vector<uint8_t> readback;\n            auto more = true;\n            while (more) {\n                auto rdata = in.read().get();\n                for (size_t i = 0; i < rdata.size(); ++i) {\n                    readback.push_back(rdata.get()[i]);\n                }\n                more = !rdata.empty();\n            }\n            //in.close().get();\n            auto xlen = std::min(end, flen) - std::min(flen, start);\n            if (xlen != readback.size()) {\n                BOOST_FAIL(format(\"Expected {:d} bytes but got {:d}, start={:d}, end={:d}\", xlen, readback.size(), start, end));\n            }\n            BOOST_REQUIRE(std::equal(readback.begin(), readback.end(), data.begin() + std::min(start, flen)));\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(without_api_prefix) {\n    return tmp_dir::do_with_thread([](tmp_dir& t) {\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename,\n                open_flags::rw | open_flags::create | open_flags::truncate).get();\n        output_stream<char> out = make_file_output_stream(f).get();\n        out.close().get();\n    });\n}\n\nSEASTAR_TEST_CASE(file_handle_test) {\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        auto filename = (t.get_path() / \"testfile.tmp\").native();\n        auto f = open_file_dma(filename, open_flags::create | open_flags::truncate | open_flags::rw).get();\n        auto close_f = deferred_close(f);\n        auto buf = static_cast<char*>(aligned_alloc(4096, 4096));\n        auto del = defer([&] () noexcept { ::free(buf); });\n        for (unsigned i = 0; i < 4096; ++i) {\n            buf[i] = i;\n        }\n        f.dma_write(0, buf, 4096).get();\n        auto bad = std::vector<unsigned>(smp::count); // std::vector<bool> is special and unsuitable because it uses bitfields\n        close_f.close_now();\n        f = open_file_dma(filename, open_flags::ro).get();\n        close_f = deferred_close(f);\n        smp::invoke_on_all([fh = f.dup(), &bad] {\n            return seastar::async([fh, &bad] {\n                auto f = fh.to_file();\n                auto buf = static_cast<char*>(aligned_alloc(4096, 4096));\n                auto del = defer([&] () noexcept { ::free(buf); });\n                f.dma_read(0, buf, 4096).get();\n                for (unsigned i = 0; i < 4096; ++i) {\n                    bad[this_shard_id()] |= buf[i] != char(i);\n                }\n            });\n        }).get();\n        BOOST_REQUIRE(!boost::algorithm::any_of_equal(bad, 1u));\n    });\n}\n\nSEASTAR_TEST_CASE(test_fstream_slow_start) {\n    return seastar::async([] {\n        static constexpr size_t file_size = 128 * 1024 * 1024;\n        static constexpr size_t buffer_size = 260 * 1024;\n        static constexpr size_t read_ahead = 1;\n\n        auto mock_file = make_shared<mock_read_only_file>(file_size);\n\n        auto history = make_lw_shared<file_input_stream_history>();\n\n        file_input_stream_options options{};\n        options.buffer_size = buffer_size;\n        options.read_ahead = read_ahead;\n        options.dynamic_adjustments = history;\n\n        static constexpr size_t requests_at_slow_start = 2; // 1 request + 1 read-ahead\n        static constexpr size_t requests_at_full_speed = read_ahead + 1; // 1 request + read_ahead\n\n        std::optional<size_t> initial_read_size;\n\n        auto read_whole_file_with_slow_start = [&] (auto fstr) {\n            uint64_t total_read = 0;\n            size_t previous_buffer_length = 0;\n\n            // We don't want to assume too much about fstream internals, but with\n            // no history we should start with a buffer sizes somewhere in\n            // (0, buffer_size) range.\n            mock_file->set_read_size_verifier([&] (size_t length) {\n                BOOST_CHECK_LE(length, initial_read_size.value_or(buffer_size - 1));\n                BOOST_CHECK_GE(length, initial_read_size.value_or(1));\n                previous_buffer_length = length;\n                if (!initial_read_size) {\n                    initial_read_size = length;\n                }\n            });\n\n            // Slow start phase\n            while (true) {\n                // We should leave slow start before reading the whole file.\n                BOOST_CHECK_LT(total_read, file_size);\n\n                mock_file->set_allowed_read_requests(requests_at_slow_start);\n                auto buf = fstr.read().get();\n                BOOST_CHECK_GT(buf.size(), 0u);\n\n                mock_file->set_read_size_verifier([&] (size_t length) {\n                    // There is no reason to reduce buffer size.\n                    BOOST_CHECK_LE(length, std::min(previous_buffer_length * 2, buffer_size));\n                    BOOST_CHECK_GE(length, previous_buffer_length);\n                    previous_buffer_length = length;\n                });\n\n                BOOST_TEST_MESSAGE(format(\"Size {:d}\", buf.size()));\n                total_read += buf.size();\n                if (buf.size() == buffer_size) {\n                    BOOST_TEST_MESSAGE(\"Leaving slow start phase.\");\n                    break;\n                }\n            }\n\n            // Reading at full speed now\n            mock_file->set_expected_read_size(buffer_size);\n            while (total_read != file_size) {\n                mock_file->set_allowed_read_requests(requests_at_full_speed);\n                auto buf = fstr.read().get();\n                total_read += buf.size();\n            }\n\n            mock_file->set_allowed_read_requests(requests_at_full_speed);\n            auto buf = fstr.read().get();\n            BOOST_CHECK_EQUAL(buf.size(), 0u);\n            SEASTAR_ASSERT(buf.size() == 0);\n        };\n\n        auto read_while_file_at_full_speed = [&] (auto fstr) {\n            uint64_t total_read = 0;\n\n            mock_file->set_expected_read_size(buffer_size);\n            while (total_read != file_size) {\n                mock_file->set_allowed_read_requests(requests_at_full_speed);\n                auto buf = fstr.read().get();\n                total_read += buf.size();\n            }\n\n            mock_file->set_allowed_read_requests(requests_at_full_speed);\n            auto buf = fstr.read().get();\n            BOOST_CHECK_EQUAL(buf.size(), 0u);\n        };\n\n        auto read_and_skip_a_lot = [&] (auto fstr) {\n            uint64_t total_read = 0;\n            size_t previous_buffer_size = buffer_size;\n\n            mock_file->set_allowed_read_requests(std::numeric_limits<size_t>::max());\n            mock_file->set_read_size_verifier([&] (size_t length) {\n                // There is no reason to reduce buffer size.\n                BOOST_CHECK_LE(length, previous_buffer_size);\n                BOOST_CHECK_GE(length, initial_read_size.value_or(1));\n                previous_buffer_size = length;\n            });\n            while (total_read != file_size) {\n                auto buf = fstr.read().get();\n                total_read += buf.size();\n\n                buf = fstr.read().get();\n                total_read += buf.size();\n\n                auto skip_by = std::min(file_size - total_read, buffer_size * 2);\n                fstr.skip(skip_by).get();\n                total_read += skip_by;\n            }\n\n            // We should be back at slow start at this stage.\n            BOOST_CHECK_LT(previous_buffer_size, buffer_size);\n            if (initial_read_size) {\n                BOOST_CHECK_EQUAL(previous_buffer_size, *initial_read_size);\n            }\n\n            mock_file->set_allowed_read_requests(requests_at_full_speed);\n            auto buf = fstr.read().get();\n            BOOST_CHECK_EQUAL(buf.size(), 0u);\n\n        };\n\n        auto make_fstream = [&] {\n            struct fstream_wrapper {\n                input_stream<char> s;\n                explicit fstream_wrapper(input_stream<char>&& s) : s(std::move(s)) {}\n                fstream_wrapper(fstream_wrapper&&) = default;\n                fstream_wrapper& operator=(fstream_wrapper&&) = default;\n                future<temporary_buffer<char>> read() {\n                    return s.read();\n                }\n                future<> skip(uint64_t n) {\n                    return s.skip(n);\n                }\n                ~fstream_wrapper() {\n                    s.close().get();\n                }\n            };\n            return fstream_wrapper(make_file_input_stream(file(mock_file), 0, file_size, options));\n        };\n\n        BOOST_TEST_MESSAGE(\"Reading file, no history, expectiong a slow start\");\n        read_whole_file_with_slow_start(make_fstream());\n        BOOST_TEST_MESSAGE(\"Reading file again, everything good so far, read at full speed\");\n        read_while_file_at_full_speed(make_fstream());\n        BOOST_TEST_MESSAGE(\"Reading and skipping a lot\");\n        read_and_skip_a_lot(make_fstream());\n        BOOST_TEST_MESSAGE(\"Reading file, bad history, we are back at slow start...\");\n        read_whole_file_with_slow_start(make_fstream());\n        BOOST_TEST_MESSAGE(\"Reading file yet again, should've recovered by now\");\n        read_while_file_at_full_speed(make_fstream());\n    });\n}\n\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n\nSEASTAR_TEST_CASE(test_close_error) {\n    using namespace seastar::memory;\n\n    return tmp_dir::do_with_thread([] (tmp_dir& t) {\n        bool done = false;\n        for (size_t i = 0; !done; i++) {\n            bool got_close_error = false;\n            sstring filename = (t.get_path() / format(\"testfile-{}.tmp\", i).c_str()).native();\n            file f = open_file_dma(filename, open_flags::rw | open_flags::create | open_flags::truncate).get();\n            auto opts = file_output_stream_options{};\n            opts.write_behind = 16;\n            std::unique_ptr<output_stream<char>> out = std::make_unique<output_stream<char>>(make_file_output_stream(std::move(f), opts).get());\n            size_t size = 4096;\n            std::vector<char> buf(size);\n            std::iota(buf.begin(), buf.end(), 0);\n            size_t file_length = 1 * 1024 * 1024;\n            auto fut = make_ready_future<>();\n            for (size_t len = 0; len < file_length; len += size) {\n                fut = fut.finally([&] { return out->write(buf.data(), size); });\n            }\n            fut.get();\n            try {\n                local_failure_injector().fail_after(i);\n                out->close().get();\n                done = true;\n                local_failure_injector().cancel();\n            } catch (const std::bad_alloc&) {\n                got_close_error = true;\n            }\n            BOOST_REQUIRE(got_close_error || done);\n            out.reset();\n            remove_file(filename).get();\n        }\n    });\n}\n\n#endif // SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n"
  },
  {
    "path": "tests/unit/futures_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <boost/test/tools/old/interface.hpp>\n#include <cstddef>\n#include <forward_list>\n#include <iterator>\n#include <ranges>\n#include <stdexcept>\n#include <type_traits>\n#include <vector>\n#include <seastar/testing/test_case.hh>\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/stream.hh>\n#include <seastar/util/backtrace.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/shared_future.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/when_any.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/later.hh>\n#include <boost/iterator/counting_iterator.hpp>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <boost/range/iterator_range.hpp>\n#include <boost/range/irange.hpp>\n\n#include <seastar/core/internal/api-level.hh>\n#include <unistd.h>\n\n#include \"expected_exception.hh\"\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nstatic_assert(std::is_nothrow_default_constructible_v<gate>,\n    \"seastar::gate constructor must not throw\");\nstatic_assert(std::is_nothrow_move_constructible_v<gate>,\n    \"seastar::gate move constructor must not throw\");\n\nstatic_assert(std::is_nothrow_default_constructible_v<shared_future<>>);\nstatic_assert(std::is_nothrow_copy_constructible_v<shared_future<>>);\nstatic_assert(std::is_nothrow_move_constructible_v<shared_future<>>);\n\nstatic_assert(std::is_nothrow_move_constructible_v<shared_promise<>>);\n\n#if defined(__clang__) || (defined(__GNUC__) && __GNUC__ >= 13)\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wself-move\"\n#endif\nSEASTAR_TEST_CASE(test_self_move) {\n    future_state<std::tuple<std::unique_ptr<int>>> s1;\n    s1.set(std::make_unique<int>(42));\n    s1 = std::move(s1); // no crash, but the value of s1 is not defined.\n\n    future_state<std::unique_ptr<int>> s2;\n    s2.set(std::make_unique<int>(42));\n    std::swap(s2, s2);\n    BOOST_REQUIRE_EQUAL(*std::move(s2).get(), 42);\n\n    promise<std::unique_ptr<int>> p1;\n    p1.set_value(std::make_unique<int>(42));\n    p1 = std::move(p1); // no crash, but the value of p1 is not defined.\n\n    promise<std::unique_ptr<int>> p2;\n    p2.set_value(std::make_unique<int>(42));\n    std::swap(p2, p2);\n    BOOST_REQUIRE_EQUAL(*p2.get_future().get(), 42);\n\n    auto  f1 = make_ready_future<std::unique_ptr<int>>(std::make_unique<int>(42));\n    f1 = std::move(f1); // no crash, but the value of f1 is not defined.\n\n    auto f2 = make_ready_future<std::unique_ptr<int>>(std::make_unique<int>(42));\n    std::swap(f2, f2);\n    BOOST_REQUIRE_EQUAL(*f2.get(), 42);\n\n    return make_ready_future<>();\n}\n#if defined(__clang__) || (defined(__GNUC__) && __GNUC__ >= 13)\n#pragma GCC diagnostic pop\n#endif\n\nstatic subscription<int> get_empty_subscription(std::function<future<> (int)> func) {\n    stream<int> s;\n    auto ret = s.listen(func);\n    s.close();\n    return ret;\n}\n\nstruct int_container {\n    int_container() = default;\n    int_container(int_container&&) noexcept = default;\n    // this template can be matched by an initializer like `{foo}`, which was used in\n    // uninitialized_wrapper_base::uninitialized_set() to perform placement new.\n    template <typename T>\n    int_container(const std::vector<T>&) {\n        static_assert(std::is_constructible_v<int, T>);\n    }\n};\n\nSEASTAR_TEST_CASE(test_future_value_constructible_from_range) {\n    // verify that the type a future's value is constructible from a range\n    using vector_type = std::vector<int_container>;\n    std::ignore = seastar::make_ready_future<vector_type>(vector_type{});\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_stream) {\n    auto sub = get_empty_subscription([](int x) {\n        return make_ready_future<>();\n    });\n    return sub.done();\n}\n\nSEASTAR_TEST_CASE(test_stream_drop_sub) {\n    auto s = make_lw_shared<stream<int>>();\n    const int expected = 42;\n    std::optional<future<>> ret;\n    {\n        auto sub = s->listen([expected](int actual) {\n            BOOST_REQUIRE_EQUAL(expected, actual);\n            return make_ready_future<>();\n        });\n        ret = sub.done();\n        // It is ok to drop the subscription when we only want the competition future.\n    }\n    return s->produce(expected).then([ret = std::move(*ret), s] () mutable {\n        s->close();\n        return std::move(ret);\n    });\n}\n\nSEASTAR_TEST_CASE(test_reference) {\n    int a = 42;\n    future<int&> orig = make_ready_future<int&>(a);\n    future<int&> fut = std::move(orig);\n    int& r = fut.get();\n    r = 43;\n    BOOST_REQUIRE_EQUAL(a, 43);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_set_future_state_with_tuple) {\n    future_state<std::tuple<int>> s1;\n    promise<int> p1;\n    const std::tuple<int> v1(42);\n    s1.set(v1);\n    p1.set_value(v1);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_set_value_make_exception_in_copy) {\n    struct throw_in_copy {\n        throw_in_copy() noexcept = default;\n        throw_in_copy(throw_in_copy&& x) noexcept {\n        }\n        throw_in_copy(const throw_in_copy& x) {\n            throw 42;\n        }\n    };\n    promise<throw_in_copy> p1;\n    throw_in_copy v;\n    p1.set_value(v);\n    BOOST_REQUIRE_THROW(p1.get_future().get(), int);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_set_exception_in_constructor) {\n    struct throw_in_constructor {\n        throw_in_constructor() {\n            throw 42;\n        }\n    };\n    future<throw_in_constructor> f = make_ready_future<throw_in_constructor>();\n    BOOST_REQUIRE(f.failed());\n    BOOST_REQUIRE_THROW(f.get(), int);\n}\n\nSEASTAR_TEST_CASE(test_finally_is_called_on_success_and_failure) {\n    auto finally1 = make_shared<bool>();\n    auto finally2 = make_shared<bool>();\n\n    return make_ready_future().then([] {\n    }).finally([=] {\n        *finally1 = true;\n    }).then([] {\n        throw std::runtime_error(\"\");\n    }).finally([=] {\n        *finally2 = true;\n    }).then_wrapped([=] (auto&& f) {\n        BOOST_REQUIRE(*finally1);\n        BOOST_REQUIRE(*finally2);\n\n        // Should be failed.\n        try {\n            f.get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n    });\n}\n\nSEASTAR_TEST_CASE(test_get_on_promise) {\n    auto p = promise<uint32_t>();\n    p.set_value(10);\n    BOOST_REQUIRE_EQUAL(10u, p.get_future().get());\n    return make_ready_future();\n}\n\n// An exception class with a controlled what() overload\nclass test_exception : public std::exception {\n    sstring _what;\npublic:\n    explicit test_exception(sstring what) : _what(std::move(what)) {}\n    virtual const char* what() const noexcept override {\n        return _what.c_str();\n    }\n};\n\nSEASTAR_TEST_CASE(test_get_on_exceptional_promise) {\n    auto p = promise<>();\n    p.set_exception(test_exception(\"test\"));\n    BOOST_REQUIRE_THROW(p.get_future().get(), test_exception);\n    return make_ready_future();\n}\n\nstatic void check_finally_exception(std::exception_ptr ex) {\n  BOOST_REQUIRE_EQUAL(fmt::format(\"{}\", ex),\n        \"seastar::nested_exception: test_exception (bar) (while cleaning up after test_exception (foo))\");\n  try {\n      // convert to the concrete type nested_exception\n      std::rethrow_exception(ex);\n  } catch (seastar::nested_exception& ex) {\n    try {\n        std::rethrow_exception(ex.inner);\n    } catch (test_exception& inner) {\n        BOOST_REQUIRE_EQUAL(inner.what(), \"bar\");\n    }\n    try {\n        ex.rethrow_nested();\n    } catch (test_exception& outer) {\n        BOOST_REQUIRE_EQUAL(outer.what(), \"foo\");\n    }\n  }\n}\n\nSEASTAR_TEST_CASE(test_finally_exception) {\n    return make_ready_future<>().then([] {\n        throw test_exception(\"foo\");\n    }).finally([] {\n        throw test_exception(\"bar\");\n    }).handle_exception(check_finally_exception);\n}\n\nSEASTAR_TEST_CASE(test_finally_exceptional_future) {\n    return make_ready_future<>().then([] {\n        throw test_exception(\"foo\");\n    }).finally([] {\n       return make_exception_future<>(test_exception(\"bar\"));\n    }).handle_exception(check_finally_exception);\n}\n\nSEASTAR_TEST_CASE(test_finally_waits_for_inner) {\n    auto finally = make_shared<bool>();\n    auto p = make_shared<promise<>>();\n\n    auto f = make_ready_future().then([] {\n    }).finally([=] {\n        return p->get_future().then([=] {\n            *finally = true;\n        });\n    }).then([=] {\n        BOOST_REQUIRE(*finally);\n    });\n    BOOST_REQUIRE(!*finally);\n    p->set_value();\n    return f;\n}\n\nSEASTAR_TEST_CASE(test_finally_is_called_on_success_and_failure__not_ready_to_armed) {\n    auto finally1 = make_shared<bool>();\n    auto finally2 = make_shared<bool>();\n\n    promise<> p;\n    auto f = p.get_future().finally([=] {\n        *finally1 = true;\n    }).then([] {\n        throw std::runtime_error(\"\");\n    }).finally([=] {\n        *finally2 = true;\n    }).then_wrapped([=] (auto &&f) {\n        BOOST_REQUIRE(*finally1);\n        BOOST_REQUIRE(*finally2);\n        try {\n            f.get();\n        } catch (...) {} // silence exceptional future ignored messages\n    });\n\n    p.set_value();\n    return f;\n}\n\nSEASTAR_TEST_CASE(test_exception_from_finally_fails_the_target) {\n    promise<> pr;\n    auto f = pr.get_future().finally([=] {\n        throw std::runtime_error(\"\");\n    }).then([] {\n        BOOST_REQUIRE(false);\n    }).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n        } catch (...) {} // silence exceptional future ignored messages\n    });\n\n    pr.set_value();\n    return f;\n}\n\nSEASTAR_TEST_CASE(test_exception_from_finally_fails_the_target_on_already_resolved) {\n    return make_ready_future().finally([=] {\n        throw std::runtime_error(\"\");\n    }).then([] {\n        BOOST_REQUIRE(false);\n    }).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n        } catch (...) {} // silence exceptional future ignored messages\n    });\n}\n\nSEASTAR_TEST_CASE(test_exception_thrown_from_then_wrapped_causes_future_to_fail) {\n    return make_ready_future().then_wrapped([] (auto&& f) {\n        throw std::runtime_error(\"\");\n    }).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n    });\n}\n\nSEASTAR_TEST_CASE(test_exception_thrown_from_then_wrapped_causes_future_to_fail__async_case) {\n    promise<> p;\n\n    auto f = p.get_future().then_wrapped([] (auto&& f) {\n        throw std::runtime_error(\"\");\n    }).then_wrapped([] (auto&& f) {\n        try {\n            f.get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n    });\n\n    p.set_value();\n\n    return f;\n}\n\nSEASTAR_TEST_CASE(test_failing_intermediate_promise_should_fail_the_master_future) {\n    promise<> p1;\n    promise<> p2;\n\n    auto f = p1.get_future().then([f = p2.get_future()] () mutable {\n        return std::move(f);\n    }).then([] {\n        BOOST_REQUIRE(false);\n    });\n\n    p1.set_value();\n    p2.set_exception(std::runtime_error(\"boom\"));\n\n    return std::move(f).then_wrapped([](auto&& f) {\n        try {\n            f.get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n    });\n}\n\nSEASTAR_TEST_CASE(test_future_forwarding__not_ready_to_unarmed) {\n    promise<> p1;\n    promise<> p2;\n\n    auto f1 = p1.get_future();\n    auto f2 = p2.get_future();\n\n    f1.forward_to(std::move(p2));\n\n    BOOST_REQUIRE(!f2.available());\n\n    auto called = f2.then([] {});\n\n    p1.set_value();\n    return called;\n}\n\nSEASTAR_TEST_CASE(test_future_forwarding__not_ready_to_armed) {\n    promise<> p1;\n    promise<> p2;\n\n    auto f1 = p1.get_future();\n    auto f2 = p2.get_future();\n\n    auto called = f2.then([] {});\n\n    f1.forward_to(std::move(p2));\n\n    BOOST_REQUIRE(!f2.available());\n\n    p1.set_value();\n\n    return called;\n}\n\nSEASTAR_TEST_CASE(test_future_forwarding__ready_to_unarmed) {\n    promise<> p2;\n\n    auto f1 = make_ready_future<>();\n    auto f2 = p2.get_future();\n\n    std::move(f1).forward_to(std::move(p2));\n    BOOST_REQUIRE(f2.available());\n\n    return std::move(f2).then_wrapped([] (future<> f) {\n        BOOST_REQUIRE(!f.failed());\n    });\n}\n\nSEASTAR_TEST_CASE(test_future_forwarding__ready_to_armed) {\n    promise<> p2;\n\n    auto f1 = make_ready_future<>();\n    auto f2 = p2.get_future();\n\n    auto called = std::move(f2).then([] {});\n\n    BOOST_REQUIRE(f1.available());\n\n    f1.forward_to(std::move(p2));\n    return called;\n}\n\nstatic void forward_dead_unarmed_promise_with_dead_future_to(promise<>& p) {\n    promise<> p2;\n    p.get_future().forward_to(std::move(p2));\n}\n\nSEASTAR_TEST_CASE(test_future_forwarding__ready_to_unarmed_soon_to_be_dead) {\n    promise<> p1;\n    forward_dead_unarmed_promise_with_dead_future_to(p1);\n    make_ready_future<>().forward_to(std::move(p1));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_exception_can_be_thrown_from_do_until_body) {\n    return do_until([] { return false; }, [] {\n        throw expected_exception();\n        return now();\n    }).then_wrapped([] (auto&& f) {\n       try {\n           f.get();\n           BOOST_FAIL(\"should have failed\");\n       } catch (const expected_exception& e) {\n           // expected\n       }\n    });\n}\n\nSEASTAR_TEST_CASE(test_exception_can_be_thrown_from_do_until_condition) {\n    return do_until([] { throw expected_exception(); return false; }, [] {\n        return now();\n    }).then_wrapped([] (auto&& f) {\n       try {\n           f.get();\n           BOOST_FAIL(\"should have failed\");\n       } catch (const expected_exception& e) {\n           // expected\n       }\n    });\n}\n\nSEASTAR_TEST_CASE(test_bare_value_can_be_returned_from_callback) {\n    return now().then([] {\n        return 3;\n    }).then([] (int x) {\n        BOOST_REQUIRE(x == 3);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_all_iterator_range) {\n    std::vector<future<size_t>> futures;\n    for (size_t i = 0; i != 1000000; ++i) {\n        // Use a mix of available and unavailable futures to exercise\n        // both paths in when_all().\n        auto fut = (i % 2) == 0 ? make_ready_future<>() : yield();\n        futures.push_back(fut.then([i] { return i; }));\n    }\n    // Verify the above statement is correct\n    BOOST_REQUIRE(!std::all_of(futures.begin(), futures.end(),\n            [] (auto& f) { return f.available(); }));\n    auto p = make_shared(std::move(futures));\n    return when_all(p->begin(), p->end()).then([p] (std::vector<future<size_t>> ret) {\n        BOOST_REQUIRE(std::all_of(ret.begin(), ret.end(), [] (auto& f) { return f.available(); }));\n        BOOST_REQUIRE(std::all_of(ret.begin(), ret.end(), [&ret] (auto& f) { return f.get() == size_t(&f - ret.data()); }));\n    });\n}\n\ntemplate<typename Container>\nvoid test_iterator_range_estimate() {\n    Container container{1,2,3};\n\n    BOOST_REQUIRE_EQUAL(internal::iterator_range_estimate_vector_capacity(\n        container.begin(), container.end()), 3);\n}\n\nBOOST_AUTO_TEST_CASE(test_iterator_range_estimate_vector_capacity) {\n    test_iterator_range_estimate<std::vector<int>>();\n    test_iterator_range_estimate<std::list<int>>();\n    test_iterator_range_estimate<std::forward_list<int>>();\n    {\n        int n = 42;\n        auto seq = std::views::iota(0, n);\n        BOOST_REQUIRE_EQUAL(internal::iterator_range_estimate_vector_capacity(\n            seq.begin(), seq.end()), n);\n    }\n    {\n        // for ranges that generate elements on-the-fly, advancing an iterator\n        // might actually consume or transform the underlying sequence, in this\n        // case, the function under test returns 0.\n        auto seq = std::views::iota(1);\n        BOOST_REQUIRE_EQUAL(internal::iterator_range_estimate_vector_capacity(\n            seq.begin(), seq.end()), 0);\n    }\n}\n\n// helper function for when_any tests\ntemplate<typename Container>\nfuture<> when_all_but_one_succeed(Container& futures, size_t leave_out)\n{\n    auto sz = futures.size();\n    SEASTAR_ASSERT(sz >= 1);\n    SEASTAR_ASSERT(leave_out < sz);\n    std::vector<future<size_t>> all_but_one_tmp;\n    all_but_one_tmp.reserve(sz - 1);\n    for (size_t i = 0 ; i < sz; i++){\n        if (i == leave_out) { continue; }\n        all_but_one_tmp.push_back(std::move(futures[i]));\n    }\n    auto all_but_one = make_shared(std::move(all_but_one_tmp));\n    return when_all_succeed(all_but_one->begin(), all_but_one->end()).then([all_but_one] (auto&& _) {\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_iterator_range_i) {\n    std::vector<future<size_t>> futures;\n    for (size_t i = 0; i != 100; ++i) {\n        auto fut = yield();\n        futures.push_back(fut.then([i] { return i; }));\n    }\n\n    // Verify the above statement is correct\n    BOOST_REQUIRE(std::all_of(futures.begin(), futures.end(), [](auto &f) { return !f.available(); }));\n\n    auto p = make_shared(std::move(futures));\n    return seastar::when_any(p->begin(), p->end()).then([p](auto &&ret_obj) {\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].available());\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].get() == ret_obj.index);\n        return when_all_but_one_succeed(ret_obj.futures, ret_obj.index);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_iterator_range_ii) {\n    std::vector<future<size_t>> futures;\n    for (size_t i = 0; i != 100; ++i) {\n        if (i == 42) {\n            auto fut = seastar::make_ready_future<>();\n            futures.push_back(fut.then([i] { return i; }));\n        } else {\n            auto fut = seastar::sleep(100ms);\n            futures.push_back(fut.then([i] { return i; }));\n        }\n    }\n    auto p = make_shared(std::move(futures));\n    return seastar::when_any(p->begin(), p->end()).then([p](auto &&ret_obj) {\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].available());\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].get() == ret_obj.index);\n        BOOST_REQUIRE(ret_obj.index == 42);\n        return when_all_but_one_succeed(ret_obj.futures, ret_obj.index);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_iterator_range_iii) {\n    std::vector<future<size_t>> futures;\n    for (size_t i = 0; i != 100; ++i) {\n        if (i == 42) {\n            auto fut = seastar::sleep(5ms);\n            futures.push_back(fut.then([i] { return i; }));\n        } else {\n            auto fut = seastar::sleep(100ms);\n            futures.push_back(fut.then([i] { return i; }));\n        }\n    }\n    auto p = make_shared(std::move(futures));\n    return seastar::when_any(p->begin(), p->end()).then([p](auto &&ret_obj) {\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].available());\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].get() == ret_obj.index);\n        BOOST_REQUIRE(ret_obj.index == 42);\n        return when_all_but_one_succeed(ret_obj.futures, ret_obj.index);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_iterator_range_iv) {\n    std::vector<future<size_t>> futures;\n    for (size_t i = 0; i != 100; ++i) {\n        if (i == 42) {\n            auto fut = yield().then([] { return seastar::make_exception_future(std::runtime_error(\"test\")); } );\n            futures.push_back(fut.then([i] { return i; }));\n        } else {\n            auto fut = seastar::sleep(100ms);\n            futures.push_back(fut.then([i] { return i; }));\n        }\n    }\n    auto p = make_shared(std::move(futures));\n    return seastar::when_any(p->begin(), p->end()).then([p](auto &&ret_obj) {\n        BOOST_REQUIRE(ret_obj.futures[ret_obj.index].available());\n        BOOST_REQUIRE_THROW(ret_obj.futures[ret_obj.index].get(), std::runtime_error);\n        return when_all_but_one_succeed(ret_obj.futures, ret_obj.index);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_variadic_i)\n{\n    auto f_int = yield().then([] { return make_ready_future<int>(42); });\n    auto f_string = sleep(100ms).then([] { return make_ready_future<sstring>(\"hello\"); });\n    auto f_l33tspeak = sleep(100ms).then([] {\n        return make_ready_future<std::tuple<char, int, int, char, char, int, char>>(\n            std::make_tuple('s', 3, 4, 's', 't', 4, 'r'));\n    });\n    return when_any(std::move(f_int), std::move(f_string), std::move(f_l33tspeak)).then([](auto&& wa_result) {\n        BOOST_REQUIRE(wa_result.index == 0);\n        auto [one, two, three] = std::move(wa_result.futures);\n        BOOST_REQUIRE(one.get() == 42);\n        return when_all_succeed(std::move(two), std::move(three)).then([](auto _) { return seastar::make_ready_future<>(); });\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_any_variadic_ii)\n{\n    struct foo {\n        int bar = 86;\n    };\n\n    auto f_int = sleep(100ms).then([] { return make_ready_future<int>(42); });\n    auto f_foo = sleep(75ms).then([] { return make_ready_future<foo>(); });\n    auto f_string = sleep(1ms).then([] { return make_ready_future<sstring>(\"hello\"); });\n    auto f_l33tspeak = sleep(50ms).then([] {\n        return make_ready_future<std::tuple<char, int, int, char, char, int, char>>(\n            std::make_tuple('s', 3, 4, 's', 't', 4, 'r'));\n    });\n    return when_any(std::move(f_int), std::move(f_foo), std::move(f_string), std::move(f_l33tspeak))\n        .then([](auto&& wa_result) {\n            BOOST_REQUIRE(wa_result.index == 2);\n            auto [one, two, three, four] = std::move(wa_result.futures);\n            BOOST_REQUIRE(three.get() == \"hello\");\n            return when_any(std::move(one), std::move(two), std::move(four)).then([](auto wa_nextresult) {\n                auto [one, two, four] = std::move(wa_nextresult.futures);\n                BOOST_REQUIRE(wa_nextresult.index == 2);\n                BOOST_REQUIRE(four.get() == std::make_tuple('s', 3, 4, 's', 't', 4, 'r'));\n                return when_any(std::move(one), std::move(two)).then([](auto wa_result) {\n                    auto [one, two] = std::move(wa_result.futures);\n                    BOOST_REQUIRE(wa_result.index == 1);\n                    BOOST_REQUIRE(two.get().bar == foo{}.bar);\n                    return one.then([](int x) { BOOST_REQUIRE(x == 42); });\n                });\n            });\n        });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce) {\n    auto square = [] (long x) { return make_ready_future<long>(x*x); };\n    long n = 1000;\n    return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n            square, long(0), std::plus<long>()).then([n] (auto result) {\n        auto m = n - 1; // counting does not include upper bound\n        BOOST_REQUIRE_EQUAL(result, (m * (m + 1) * (2*m + 1)) / 6);\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce_simple) {\n    return do_with(0L, [] (auto& res) {\n        long n = 10;\n        return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n                [] (long x) { return x; },\n                [&res] (long x) { res += x; }).then([n, &res] {\n            long expected = (n * (n - 1)) / 2;\n            BOOST_REQUIRE_EQUAL(res, expected);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce_tuple) {\n    return do_with(0L, 0L, [] (auto& res0, auto& res1) {\n        long n = 10;\n        return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n                [] (long x) { return std::tuple<long, long>(x, -x); },\n                [&res0, &res1] (std::tuple<long, long> t) { res0 += std::get<0>(t); res1 += std::get<1>(t); }).then([n, &res0, &res1] {\n            long expected = (n * (n - 1)) / 2;\n            BOOST_REQUIRE_EQUAL(res0, expected);\n            BOOST_REQUIRE_EQUAL(res1, -expected);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(long x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                return x;\n            });\n        }\n    };\n    struct reduce {\n        long& res;\n        bool destroyed = false;\n        reduce(long& result)\n            : res{result} {}\n        reduce(const reduce&) = default;\n        ~reduce() {\n            destroyed = true;\n        }\n        auto operator()(long x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                res += x;\n            });\n        }\n    };\n    return do_with(0L, [] (auto& res) {\n        long n = 10;\n        return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n                map{}, reduce{res}).then([n, &res] {\n            long expected = (n * (n - 1)) / 2;\n            BOOST_REQUIRE_EQUAL(res, expected);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce0_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(long x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                return x;\n            });\n        }\n    };\n    struct reduce {\n        bool destroyed = false;\n        reduce() = default;\n        reduce(const reduce&) = default;\n        ~reduce() {\n            destroyed = true;\n        }\n        auto operator()(long res, long x) {\n            BOOST_REQUIRE(!destroyed);\n            return res + x;\n        }\n    };\n    long n = 10;\n    return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n            map{}, 0L, reduce{}).then([n] (long res) {\n        long expected = (n * (n - 1)) / 2;\n        BOOST_REQUIRE_EQUAL(res, expected);\n    });\n}\n\nSEASTAR_TEST_CASE(test_map_reduce1_lifetime) {\n    struct map {\n        bool destroyed = false;\n        map() = default;\n        map(const map&) = default;\n        ~map() {\n            destroyed = true;\n        }\n        auto operator()(long x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                return x;\n            });\n        }\n    };\n    struct reduce {\n        long res = 0;\n        bool destroyed = false;\n        reduce() = default;\n        reduce(const reduce&) = default;\n        ~reduce() {\n            BOOST_TEST_MESSAGE(\"~reduce()\");\n            destroyed = true;\n        }\n        auto operator()(long x) {\n            return yield().then([this, x] {\n                BOOST_REQUIRE(!destroyed);\n                res += x;\n                return make_ready_future<>();\n            });\n        }\n        auto get() {\n            return sleep(std::chrono::milliseconds(10)).then([this] {\n                BOOST_REQUIRE(!destroyed);\n                return res;\n            });\n        }\n    };\n    long n = 10;\n    return map_reduce(boost::make_counting_iterator<long>(0), boost::make_counting_iterator<long>(n),\n                      map{}, reduce{}).then([n] (long res) {\n        long expected = (n * (n - 1)) / 2;\n        BOOST_REQUIRE_EQUAL(res, expected);\n    });\n}\n\nSEASTAR_TEST_CASE(map_reduce_with_throwing_mapper) {\n    try {\n        auto vec = std::vector<int>{1, 2, 3, 4, 5, 6, 7};\n        auto ret = co_await map_reduce(\n            vec,\n            // Mapper: identity function, but throws\n            [] (int x) -> future<int> {\n                if (x == 5) {\n                    throw std::runtime_error(\"test\");\n                }\n                co_return x;\n            },\n            // Initial value (and accumulator): move-only type\n            std::make_unique<int>(0),\n            // Reducer: test that it does not act on a moved-from value\n            [] (std::unique_ptr<int> acc, int x) -> std::unique_ptr<int> {\n                BOOST_REQUIRE(bool(acc));\n                *acc += x;\n                return acc;\n            }\n        );\n        BOOST_FAIL(\"should have thrown\");\n    } catch (...) {\n        // Exception is expected and uninteresting\n    }\n}\n\nSEASTAR_TEST_CASE(map_reduce_with_throwing_reducer) {\n    try {\n        auto vec = std::vector<int>{1, 2, 3, 4, 5, 6, 7};\n        auto ret = co_await map_reduce(\n            vec,\n            // Mapper: square function\n            [] (int x) -> future<int> {\n                co_return x * x;\n            },\n            // Initial value (and accumulator): move-only type\n            std::make_unique<int>(0),\n            // Reducer: simple sum, but randomly throws\n            [] (std::unique_ptr<int> acc, int x) -> std::unique_ptr<int> {\n                BOOST_REQUIRE(bool(acc));\n                if (*acc > 14) {\n                    throw std::runtime_error(\"accumulator overflow, launch missiles\");\n                }\n                *acc += x;\n                return acc;\n            }\n        );\n        BOOST_FAIL(\"should have thrown\");\n    } catch (...) {\n        // Exception is expected and uninteresting\n    }\n}\n\n// This test doesn't actually test anything - it just waits for the future\n// returned by sleep to complete. However, a bug we had in sleep() caused\n// this test to fail the sanitizer in the debug build, so this is a useful\n// regression test.\nSEASTAR_TEST_CASE(test_sleep) {\n    return sleep(std::chrono::milliseconds(100));\n}\n\nSEASTAR_TEST_CASE(test_do_with_1) {\n    return do_with(1, [] (int& one) {\n       BOOST_REQUIRE_EQUAL(one, 1);\n       return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_2) {\n    return do_with(1, 2L, [] (int& one, long two) {\n        BOOST_REQUIRE_EQUAL(one, 1);\n        BOOST_REQUIRE_EQUAL(two, 2);\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_3) {\n    return do_with(1, 2L, 3, [] (int& one, long two, int three) {\n        BOOST_REQUIRE_EQUAL(one, 1);\n        BOOST_REQUIRE_EQUAL(two, 2);\n        BOOST_REQUIRE_EQUAL(three, 3);\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_4) {\n    return do_with(1, 2L, 3, 4, [] (int& one, long two, int three, int four) {\n        BOOST_REQUIRE_EQUAL(one, 1);\n        BOOST_REQUIRE_EQUAL(two, 2);\n        BOOST_REQUIRE_EQUAL(three, 3);\n        BOOST_REQUIRE_EQUAL(four, 4);\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_5) {\n    using func = noncopyable_function<void()>;\n    return do_with(func([] {}), [] (func&) {\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_6) {\n    const int x = 42;\n    return do_with(int(42), x, [](int&, int&) {\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_with_7) {\n    const int x = 42;\n    return do_with(x, [](int&) {\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_while_stopping_immediately) {\n    return do_with(int(0), [] (int& count) {\n        return repeat([&count] {\n            ++count;\n            return stop_iteration::yes;\n        }).then([&count] {\n            BOOST_REQUIRE(count == 1);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_while_stopping_after_two_iterations) {\n    return do_with(int(0), [] (int& count) {\n        return repeat([&count] {\n            ++count;\n            return count == 2 ? stop_iteration::yes : stop_iteration::no;\n        }).then([&count] {\n            BOOST_REQUIRE(count == 2);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_while_failing_in_the_first_step) {\n    return repeat([] {\n        throw expected_exception();\n        return stop_iteration::no;\n    }).then_wrapped([](auto&& f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should not happen\");\n        } catch (const expected_exception&) {\n            // expected\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_do_while_failing_in_the_second_step) {\n    return do_with(int(0), [] (int& count) {\n        return repeat([&count] {\n            ++count;\n            if (count > 1) {\n                throw expected_exception();\n            }\n            return yield().then([] { return stop_iteration::no; });\n        }).then_wrapped([&count](auto&& f) {\n            try {\n                f.get();\n                BOOST_FAIL(\"should not happen\");\n            } catch (const expected_exception&) {\n                BOOST_REQUIRE(count == 2);\n            }\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_parallel_for_each) {\n    return async([] {\n        // empty\n        parallel_for_each(std::vector<int>(), [] (int) -> future<> {\n            BOOST_FAIL(\"should not reach\");\n            abort();\n        }).get();\n\n        // immediate result\n        auto range = boost::copy_range<std::vector<int>>(boost::irange(1, 6));\n        auto sum = 0;\n        parallel_for_each(range, [&sum] (int v) {\n            sum += v;\n            return make_ready_future<>();\n        }).get();\n        BOOST_REQUIRE_EQUAL(sum, 15);\n\n        // all suspend\n        sum = 0;\n        parallel_for_each(range, [&sum] (int v) {\n            return yield().then([&sum, v] {\n                sum += v;\n            });\n        }).get();\n        BOOST_REQUIRE_EQUAL(sum, 15);\n\n        // throws immediately\n        BOOST_CHECK_EXCEPTION(parallel_for_each(range, [] (int) -> future<> {\n            throw 5;\n        }).get(), int, [] (int v) { return v == 5; });\n\n        // throws after suspension\n        BOOST_CHECK_EXCEPTION(parallel_for_each(range, [] (int) {\n            return yield().then([] {\n                throw 5;\n            });\n        }).get(), int, [] (int v) { return v == 5; });\n    });\n}\n\nSEASTAR_TEST_CASE(test_parallel_for_each_early_failure) {\n    return do_with(0, [] (int& counter) {\n        return parallel_for_each(std::views::iota(0, 11000), [&counter] (int i) {\n            using namespace std::chrono_literals;\n            // force scheduling\n            return sleep((i % 31 + 1) * 1ms).then([&counter, i] {\n                ++counter;\n                if (i % 1777 == 1337) {\n                    return make_exception_future<>(i);\n                }\n                return make_ready_future<>();\n            });\n        }).then_wrapped([&counter] (future<> f) {\n            BOOST_REQUIRE_EQUAL(counter, 11000);\n            BOOST_REQUIRE(f.failed());\n            try {\n                f.get();\n                BOOST_FAIL(\"wanted an exception\");\n            } catch (int i) {\n                BOOST_REQUIRE(i % 1777 == 1337);\n            } catch (...) {\n                BOOST_FAIL(\"bad exception type\");\n            }\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_parallel_for_each_waits_for_all_fibers_even_if_one_of_them_failed) {\n    auto can_exit = make_lw_shared<bool>(false);\n    return parallel_for_each(std::views::iota(0, 2), [can_exit] (int i) {\n        return yield().then([i, can_exit] {\n            if (i == 1) {\n                throw expected_exception();\n            } else {\n                using namespace std::chrono_literals;\n                return sleep(300ms).then([can_exit] {\n                    *can_exit = true;\n                });\n            }\n        });\n    }).then_wrapped([can_exit] (auto&& f) {\n        try {\n            f.get();\n        } catch (...) {\n            // expected\n        }\n        BOOST_REQUIRE(*can_exit);\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_parallel_for_each_broken_promise) {\n    auto fut = [] {\n        std::vector<promise<>> v(2);\n        return parallel_for_each(v, [] (promise<>& p) {\n            return p.get_future();\n        });\n    }();\n    BOOST_CHECK_THROW(fut.get(), broken_promise);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_repeat_broken_promise) {\n    auto get_fut = [] {\n        promise<stop_iteration> pr;\n        return pr.get_future();\n    };\n\n    future<> r = repeat([fut = get_fut()] () mutable {\n        return std::move(fut);\n    });\n\n    BOOST_CHECK_THROW(r.get(), broken_promise);\n}\n\n#ifndef SEASTAR_SHUFFLE_TASK_QUEUE\nSEASTAR_TEST_CASE(test_high_priority_task_runs_in_the_middle_of_loops) {\n    auto counter = make_lw_shared<int>(0);\n    auto flag = make_lw_shared<bool>(false);\n    return repeat([counter, flag] {\n        if (*counter == 1) {\n            BOOST_REQUIRE(*flag);\n            return stop_iteration::yes;\n        }\n        engine().add_high_priority_task(make_task([flag] {\n            *flag = true;\n        }));\n        ++(*counter);\n        return stop_iteration::no;\n    });\n}\n#endif\n\nSEASTAR_TEST_CASE(futurize_invoke_val_exception) {\n    return futurize_invoke([] (int arg) { throw expected_exception(); return arg; }, 1).then_wrapped([] (future<int> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should have thrown\");\n        } catch (expected_exception& e) {}\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_val_ok) {\n    return futurize_invoke([] (int arg) { return arg * 2; }, 2).then_wrapped([] (future<int> f) {\n        try {\n            auto x = f.get();\n            BOOST_REQUIRE_EQUAL(x, 4);\n        } catch (expected_exception& e) {\n            BOOST_FAIL(\"should not have thrown\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_val_future_exception) {\n    return futurize_invoke([] (int a) {\n        return sleep(std::chrono::milliseconds(100)).then([] {\n            throw expected_exception();\n            return make_ready_future<int>(0);\n        });\n    }, 0).then_wrapped([] (future<int> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should have thrown\");\n        } catch (expected_exception& e) { }\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_val_future_ok) {\n    return futurize_invoke([] (int a) {\n        return sleep(std::chrono::milliseconds(100)).then([a] {\n            return make_ready_future<int>(a * 100);\n        });\n    }, 2).then_wrapped([] (future<int> f) {\n        try {\n            auto x = f.get();\n            BOOST_REQUIRE_EQUAL(x, 200);\n        } catch (expected_exception& e) {\n            BOOST_FAIL(\"should not have thrown\");\n        }\n    });\n}\nSEASTAR_TEST_CASE(futurize_invoke_void_exception) {\n    return futurize_invoke([] (auto arg) { throw expected_exception(); }, 0).then_wrapped([] (future<> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should have thrown\");\n        } catch (expected_exception& e) {}\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_void_ok) {\n    return futurize_invoke([] (auto arg) { }, 0).then_wrapped([] (future<> f) {\n        try {\n            f.get();\n        } catch (expected_exception& e) {\n            BOOST_FAIL(\"should not have thrown\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_void_future_exception) {\n    return futurize_invoke([] (auto a) {\n        return sleep(std::chrono::milliseconds(100)).then([] {\n            throw expected_exception();\n        });\n    }, 0).then_wrapped([] (future<> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"should have thrown\");\n        } catch (expected_exception& e) { }\n    });\n}\n\nSEASTAR_TEST_CASE(futurize_invoke_void_future_ok) {\n    auto a = make_lw_shared<int>(1);\n    return futurize_invoke([] (int& a) {\n        return sleep(std::chrono::milliseconds(100)).then([&a] {\n            a *= 100;\n        });\n    }, *a).then_wrapped([a] (future<> f) {\n        try {\n            f.get();\n            BOOST_REQUIRE_EQUAL(*a, 100);\n        } catch (expected_exception& e) {\n            BOOST_FAIL(\"should not have thrown\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_unused_shared_future_is_not_a_broken_future) {\n    promise<> p;\n    shared_future<> s(p.get_future());\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_shared_future_propagates_value_to_all) {\n    return seastar::async([] {\n        promise<shared_ptr<int>> p; // shared_ptr<> to check it deals with emptyable types\n        shared_future<shared_ptr<int>> f(p.get_future());\n\n        auto f1 = f.get_future();\n        auto f2 = f.get_future();\n\n        p.set_value(make_shared<int>(1));\n        BOOST_REQUIRE(*f1.get() == 1);\n        BOOST_REQUIRE(*f2.get() == 1);\n    });\n}\n\ntemplate<typename... T>\nvoid check_fails_with_expected(future<T...> f) {\n    try {\n        f.get();\n        BOOST_FAIL(\"Should have failed\");\n    } catch (expected_exception&) {\n        // expected\n    }\n}\n\nSEASTAR_TEST_CASE(test_shared_future_propagates_value_to_copies) {\n    return seastar::async([] {\n        promise<int> p;\n        auto sf1 = shared_future<int>(p.get_future());\n        auto sf2 = sf1;\n\n        auto f1 = sf1.get_future();\n        auto f2 = sf2.get_future();\n\n        p.set_value(1);\n\n        BOOST_REQUIRE(f1.get() == 1);\n        BOOST_REQUIRE(f2.get() == 1);\n    });\n}\n\nSEASTAR_TEST_CASE(test_obtaining_future_from_shared_future_after_it_is_resolved) {\n    promise<int> p1;\n    promise<int> p2;\n    auto sf1 = shared_future<int>(p1.get_future());\n    auto sf2 = shared_future<int>(p2.get_future());\n    p1.set_value(1);\n    p2.set_exception(expected_exception());\n    return sf2.get_future().then_wrapped([f1 = sf1.get_future()] (auto&& f) mutable {\n        check_fails_with_expected(std::move(f));\n        return std::move(f1);\n    }).then_wrapped([] (auto&& f) {\n        BOOST_REQUIRE(f.get() == 1);\n    });\n}\n\nSEASTAR_TEST_CASE(test_valueless_shared_future) {\n    return seastar::async([] {\n        promise<> p;\n        shared_future<> f(p.get_future());\n\n        auto f1 = f.get_future();\n        auto f2 = f.get_future();\n\n        p.set_value();\n\n        f1.get();\n        f2.get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_shared_future_propagates_errors_to_all) {\n    promise<int> p;\n    shared_future<int> f(p.get_future());\n\n    auto f1 = f.get_future();\n    auto f2 = f.get_future();\n\n    p.set_exception(expected_exception());\n\n    return f1.then_wrapped([f2 = std::move(f2)] (auto&& f) mutable {\n        check_fails_with_expected(std::move(f));\n        return std::move(f2);\n    }).then_wrapped([] (auto&& f) mutable {\n        check_fails_with_expected(std::move(f));\n    });\n}\n\nSEASTAR_TEST_CASE(test_ignored_future_warning) {\n    // This doesn't warn:\n    promise<> p;\n    p.set_exception(expected_exception());\n    future<> f = p.get_future();\n    f.ignore_ready_future();\n\n    // And by analogy, neither should this\n    shared_promise<> p2;\n    p2.set_exception(expected_exception());\n    future<> f2 = p2.get_shared_future();\n    f2.ignore_ready_future();\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_futurize_from_tuple) {\n    std::tuple<int> v1 = std::make_tuple(3);\n    std::tuple<> v2 = {};\n    future<int> fut1 = futurize<int>::from_tuple(v1);\n    future<> fut2 = futurize<void>::from_tuple(v2);\n    BOOST_REQUIRE(fut1.get() == std::get<0>(v1));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_repeat_until_value) {\n    return do_with(int(), [] (int& counter) {\n        return repeat_until_value([&counter] () -> future<std::optional<int>> {\n            if (counter == 10000) {\n                return make_ready_future<std::optional<int>>(counter);\n            } else {\n                ++counter;\n                return make_ready_future<std::optional<int>>(std::nullopt);\n            }\n        }).then([&counter] (int result) {\n            BOOST_REQUIRE(counter == 10000);\n            BOOST_REQUIRE(result == counter);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_repeat_until_value_implicit_future) {\n    // Same as above, but returning std::optional<int> instead of future<std::optional<int>>\n    return do_with(int(), [] (int& counter) {\n        return repeat_until_value([&counter] {\n            if (counter == 10000) {\n                return std::optional<int>(counter);\n            } else {\n                ++counter;\n                return std::optional<int>(std::nullopt);\n            }\n        }).then([&counter] (int result) {\n            BOOST_REQUIRE(counter == 10000);\n            BOOST_REQUIRE(result == counter);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_repeat_until_value_exception) {\n    return repeat_until_value([] {\n        throw expected_exception();\n        return std::optional<int>(43);\n    }).then_wrapped([] (future<int> f) {\n        check_fails_with_expected(std::move(f));\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_allx) {\n    return when_all(yield(), yield(), make_ready_future()).discard_result();\n}\n\n// A noncopyable and nonmovable struct\nstruct non_copy_non_move {\n    non_copy_non_move() = default;\n    non_copy_non_move(non_copy_non_move&&) = delete;\n    non_copy_non_move(const non_copy_non_move&) = delete;\n};\n\nSEASTAR_TEST_CASE(test_when_all_functions) {\n    auto f = [x = non_copy_non_move()] {\n        (void)x;\n        return make_ready_future<int>(42);\n    };\n    return when_all(f, [] {\n        throw 42;\n        return make_ready_future<>();\n    }, yield()).then([] (std::tuple<future<int>, future<>, future<>> res) {\n        BOOST_REQUIRE_EQUAL(std::get<0>(res).get(), 42);\n\n        BOOST_REQUIRE(std::get<1>(res).available());\n        BOOST_REQUIRE(std::get<1>(res).failed());\n        std::get<1>(res).ignore_ready_future();\n\n        BOOST_REQUIRE(std::get<2>(res).available());\n        BOOST_REQUIRE(!std::get<2>(res).failed());\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_all_succeed_functions) {\n    auto f = [x = non_copy_non_move()] {\n        (void)x;\n        return make_ready_future<int>(42);\n    };\n    return when_all_succeed(f, [] {\n        throw 42;\n        return make_ready_future<>();\n    }, yield()).then_wrapped([] (future<std::tuple<int>> res) {\n        BOOST_REQUIRE(res.available());\n        BOOST_REQUIRE(res.failed());\n        res.ignore_ready_future();\n        return make_ready_future<>();\n    });\n}\n\ntemplate<typename E, typename... T>\nstatic void check_failed_with(future<T...>&& f) {\n    BOOST_REQUIRE(f.failed());\n    try {\n        f.get();\n        BOOST_FAIL(\"exception expected\");\n    } catch (const E& e) {\n        // expected\n    } catch (...) {\n        BOOST_FAIL(format(\"wrong exception: {}\", std::current_exception()));\n    }\n}\n\ntemplate<typename... T>\nstatic void check_timed_out(future<T...>&& f) {\n    check_failed_with<timed_out_error>(std::move(f));\n}\n\nSEASTAR_TEST_CASE(test_with_timeout_when_it_times_out) {\n    return seastar::async([] {\n        promise<> pr;\n        auto f = with_timeout(manual_clock::now() + 2s, pr.get_future());\n\n        BOOST_REQUIRE(!f.available());\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        BOOST_REQUIRE(!f.available());\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        check_timed_out(std::move(f));\n\n        pr.set_value();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_future_get_future_after_timeout) {\n    // This used to crash because shared_future checked if the list of\n    // pending futures was empty to decide if it had already called\n    // then_wrapped. If all pending futures timed out, it would call\n    // it again.\n    promise<> pr;\n    shared_future<with_clock<manual_clock>> sfut(pr.get_future());\n    future<> fut1 = sfut.get_future(manual_clock::now() + 1s);\n\n    manual_clock::advance(1s);\n\n    check_timed_out(std::move(fut1));\n\n    future<> fut2 = sfut.get_future(manual_clock::now() + 1s);\n    manual_clock::advance(1s);\n    check_timed_out(std::move(fut2));\n\n    future<> fut3 = sfut.get_future(manual_clock::now() + 1s);\n    pr.set_value();\n    fut3.get();\n}\n\nSEASTAR_TEST_CASE(test_custom_exception_factory_in_with_timeout) {\n    return seastar::async([] {\n        class custom_error : public std::exception {\n        public:\n            virtual const char* what() const noexcept {\n                return \"timedout\";\n            }\n        };\n        struct my_exception_factory {\n            static auto timeout() {\n                return custom_error();\n            }\n        };\n        promise<> pr;\n        auto f = with_timeout<my_exception_factory>(manual_clock::now() + 1s, pr.get_future());\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        check_failed_with<custom_error>(std::move(f));\n    });\n}\n\nSEASTAR_TEST_CASE(test_with_timeout_when_it_does_not_time_out) {\n    return seastar::async([] {\n        {\n            promise<int> pr;\n            auto f = with_timeout(manual_clock::now() + 1s, pr.get_future());\n\n            pr.set_value(42);\n\n            BOOST_REQUIRE_EQUAL(f.get(), 42);\n        }\n\n        // Check that timer was indeed cancelled\n        manual_clock::advance(1s);\n        yield().get();\n    });\n}\n\ntemplate<typename... T>\nstatic void check_aborted(future<T...>&& f) {\n    check_failed_with<abort_requested_exception>(std::move(f));\n}\n\nSEASTAR_TEST_CASE(test_shared_future_with_timeout) {\n    return seastar::async([] {\n        shared_promise<with_clock<manual_clock>, int> pr;\n        auto f1 = pr.get_shared_future(manual_clock::now() + 1s);\n        auto f2 = pr.get_shared_future(manual_clock::now() + 2s);\n        auto f3 = pr.get_shared_future();\n\n        BOOST_REQUIRE(!f1.available());\n        BOOST_REQUIRE(!f2.available());\n        BOOST_REQUIRE(!f3.available());\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        check_timed_out(std::move(f1));\n        BOOST_REQUIRE(!f2.available());\n        BOOST_REQUIRE(!f3.available());\n\n        manual_clock::advance(1s);\n        yield().get();\n\n        check_timed_out(std::move(f2));\n        BOOST_REQUIRE(!f3.available());\n\n        pr.set_value(42);\n\n        BOOST_REQUIRE_EQUAL(42, f3.get());\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_future_with_abort) {\n    abort_source as;\n    abort_source as2;\n    shared_promise<with_clock<manual_clock>, int> pr;\n    auto f1 = pr.get_shared_future(as);\n    auto f2 = pr.get_shared_future(as2);\n    auto f3 = pr.get_shared_future();\n\n    BOOST_REQUIRE(!f1.available());\n    BOOST_REQUIRE(!f2.available());\n    BOOST_REQUIRE(!f3.available());\n\n    as.request_abort();\n\n    check_aborted(std::move(f1));\n    BOOST_REQUIRE(!f2.available());\n    BOOST_REQUIRE(!f3.available());\n\n    as2.request_abort();\n\n    check_aborted(std::move(f2));\n    BOOST_REQUIRE(!f3.available());\n\n    pr.set_value(42);\n\n    BOOST_REQUIRE_EQUAL(42, f3.get());\n\n    auto f4 = pr.get_shared_future(as);\n    BOOST_REQUIRE(f4.available());\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_promise_with_outstanding_future_is_immediately_available) {\n    shared_promise<> pr1;\n    auto f1 = pr1.get_shared_future();\n    pr1.set_value();\n    BOOST_REQUIRE(pr1.available());\n    BOOST_REQUIRE_NO_THROW(f1.get());\n\n    shared_promise<> pr2;\n    auto f2 = pr2.get_shared_future();\n    pr2.set_exception(std::runtime_error(\"oops\"));\n    BOOST_REQUIRE(pr2.available());\n    BOOST_REQUIRE_THROW(f2.get(), std::runtime_error);\n}\n\nnamespace seastar {\nclass shared_future_tester {\npublic:\n    template <typename... T>\n    static bool has_scheduled_task(const shared_future<T...>& f) noexcept {\n        return f._state->has_scheduled_task();\n    }\n};\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_future_task_scheduled_only_if_there_are_waiting_futures) {\n    {\n        // Case 1: promise is eventually satisfied, get_future is not called\n        promise<> pr1;\n        shared_future<> f1(pr1.get_future());\n        BOOST_REQUIRE(!shared_future_tester::has_scheduled_task(f1));\n\n        pr1.set_value();\n        // get_future was not called, so no task should have been scheduled\n        BOOST_REQUIRE(!shared_future_tester::has_scheduled_task(f1));\n    }\n\n    {\n        // Case 2: promise is eventually satisfied, get_future was called\n        promise<> pr2;\n        shared_future<> f2(pr2.get_future());\n        auto f2f = f2.get_future();\n\n        // get_future was called, so the task is scheduled to happen after promise is resolved\n        BOOST_REQUIRE(shared_future_tester::has_scheduled_task(f2));\n\n        pr2.set_value();\n        f2f.get();\n        // f2f is resolved by shared future's task, so it must have run\n        BOOST_REQUIRE(!shared_future_tester::has_scheduled_task(f2));\n    }\n\n    {\n        // Case 3: shared future is ready from the start\n        shared_future<> f3(make_ready_future<>());\n        BOOST_REQUIRE(!shared_future_tester::has_scheduled_task(f3));\n\n        auto f3f = f3.get_future();\n        // Calling get_future on a ready shared future does not schedule the task\n        BOOST_REQUIRE(!shared_future_tester::has_scheduled_task(f3));\n        BOOST_REQUIRE(f3f.available());\n    }\n}\n\nSEASTAR_TEST_CASE(test_when_all_succeed_tuples) {\n    return seastar::when_all_succeed(\n        make_ready_future<>(),\n        make_ready_future<sstring>(\"hello world\"),\n        make_ready_future<int>(42),\n        make_ready_future<>(),\n        make_ready_future<std::tuple<int, sstring>>(std::tuple(84, \"hi\")),\n        make_ready_future<bool>(true)\n    ).then_unpack([] (sstring msg, int v, std::tuple<int, sstring> t, bool b) {\n        BOOST_REQUIRE_EQUAL(msg, \"hello world\");\n        BOOST_REQUIRE_EQUAL(v, 42);\n        BOOST_REQUIRE_EQUAL(std::get<0>(t), 84);\n        BOOST_REQUIRE_EQUAL(std::get<1>(t), \"hi\");\n        BOOST_REQUIRE_EQUAL(b, true);\n\n        return seastar::when_all_succeed(\n                make_exception_future<>(42),\n                make_ready_future<sstring>(\"hello world\"),\n                make_exception_future<int>(43),\n                make_ready_future<>()\n        ).then_unpack([] (sstring, int) {\n            BOOST_FAIL(\"shouldn't reach\");\n            return false;\n        }).handle_exception([] (auto excp) {\n            try {\n                std::rethrow_exception(excp);\n            } catch (int v) {\n                BOOST_REQUIRE(v == 42 || v == 43);\n                return true;\n            } catch (...) { }\n            return false;\n        }).then([] (auto ret) {\n            BOOST_REQUIRE(ret);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_all_succeed_vector_overload) {\n    std::vector<future<int>> vecs_noexcept;\n    vecs_noexcept.reserve(10);\n    for(int i = 0; i < 10; i++) {\n        vecs_noexcept.emplace_back(i % 2 == 0 ? make_ready_future<int>(42) : yield().then([] { return 42; }));\n    }\n\n    std::vector<future<int>> vecs_except;\n    vecs_except.reserve(10);\n    for(int i = 0; i < 10; i++) {\n        vecs_except.emplace_back(i % 2 == 0 ? make_ready_future<int>(42) : make_exception_future<int>(43));\n    }\n\n    return seastar::when_all_succeed(std::move(vecs_noexcept))\n    .then([vecs_except = std::move(vecs_except)] (std::vector<int> vals) mutable {\n        bool all = std::all_of(vals.cbegin(), vals.cend(), [](int val) { return val == 42; });\n        BOOST_REQUIRE(all);\n        return seastar::when_all_succeed(std::move(vecs_except));\n    }).then_wrapped([] (auto vals_fut) {\n        auto vals = vals_fut.get();\n        BOOST_FAIL(\"shouldn't reach\");\n        return false;\n    })\n    .handle_exception([] (auto excp) {\n        try {\n            std::rethrow_exception(excp);\n        } catch (int v) {\n            BOOST_REQUIRE(v == 43);\n            return true;\n        } catch (...) { }\n        return false;\n    }).then([] (auto ret) {\n        BOOST_REQUIRE(ret);\n    });\n}\n\nSEASTAR_TEST_CASE(test_when_all_succeed_vector) {\n    std::vector<future<>> vecs;\n    vecs.emplace_back(make_ready_future<>());\n    vecs.emplace_back(make_ready_future<>());\n    vecs.emplace_back(make_ready_future<>());\n    vecs.emplace_back(make_ready_future<>());\n    return seastar::when_all_succeed(vecs.begin(), vecs.end()).then([] {\n        std::vector<future<>> vecs;\n        vecs.emplace_back(make_ready_future<>());\n        vecs.emplace_back(make_ready_future<>());\n        vecs.emplace_back(make_exception_future<>(42));\n        vecs.emplace_back(make_exception_future<>(43));\n        return seastar::when_all_succeed(vecs.begin(), vecs.end());\n    }).then([] {\n        BOOST_FAIL(\"shouldn't reach\");\n        return false;\n    }).handle_exception([] (auto excp) {\n        try {\n            std::rethrow_exception(excp);\n        } catch (int v) {\n            BOOST_REQUIRE(v == 42 || v == 43);\n            return true;\n        } catch (...) { }\n        return false;\n    }).then([] (auto ret) {\n        BOOST_REQUIRE(ret);\n\n        std::vector<future<int>> vecs;\n        vecs.emplace_back(make_ready_future<int>(1));\n        vecs.emplace_back(make_ready_future<int>(2));\n        vecs.emplace_back(make_ready_future<int>(3));\n        return seastar::when_all_succeed(vecs.begin(), vecs.end());\n    }).then([] (std::vector<int> vals) {\n        BOOST_REQUIRE_EQUAL(vals.size(), 3u);\n        BOOST_REQUIRE_EQUAL(vals[0], 1);\n        BOOST_REQUIRE_EQUAL(vals[1], 2);\n        BOOST_REQUIRE_EQUAL(vals[2], 3);\n\n        std::vector<future<int>> vecs;\n        vecs.emplace_back(make_ready_future<int>(1));\n        vecs.emplace_back(make_ready_future<int>(2));\n        vecs.emplace_back(make_exception_future<int>(42));\n        vecs.emplace_back(make_exception_future<int>(43));\n        return seastar::when_all_succeed(vecs.begin(), vecs.end());\n    }).then([] (std::vector<int>) {\n        BOOST_FAIL(\"shouldn't reach\");\n        return false;\n    }).handle_exception([] (auto excp) {\n        try {\n            std::rethrow_exception(excp);\n        } catch (int v) {\n            BOOST_REQUIRE(v == 42 || v == 43);\n            return true;\n        } catch (...) { }\n        return false;\n    }).then([] (auto ret) {\n        BOOST_REQUIRE(ret);\n    });\n}\n\nSEASTAR_TEST_CASE(test_futurize_mutable) {\n    int count = 0;\n    return seastar::repeat([count]() mutable {\n        ++count;\n        if (count == 3) {\n            return seastar::stop_iteration::yes;\n        }\n        return seastar::stop_iteration::no;\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_broken_promises) {\n    std::optional<future<>> f;\n    std::optional<future<>> f2;\n    { // Broken after attaching a continuation\n        auto p = promise<>();\n        f = p.get_future();\n        f2 = f->then_wrapped([&] (future<> f3) {\n            BOOST_CHECK(f3.failed());\n            BOOST_CHECK_THROW(f3.get(), broken_promise);\n            f = { };\n        });\n    }\n    f2->get();\n    BOOST_CHECK(!f);\n\n    { // Broken before attaching a continuation\n        auto p = promise<>();\n        f = p.get_future();\n    }\n    f->then_wrapped([&] (future<> f3) {\n        BOOST_CHECK(f3.failed());\n        BOOST_CHECK_THROW(f3.get(), broken_promise);\n        f = { };\n    }).get();\n    BOOST_CHECK(!f);\n\n    { // Broken before suspending a thread\n        auto p = promise<>();\n        f = p.get_future();\n    }\n    BOOST_CHECK_THROW(f->get(), broken_promise);\n}\n\nSEASTAR_TEST_CASE(test_warn_on_broken_promise_with_no_future) {\n    // Example code where we expect a \"Exceptional future ignored\"\n    // warning.\n    promise<> p;\n    // Intentionally destroy the future\n    (void)p.get_future();\n\n    reactor::test::with_allow_abandoned_failed_futures(1, [&] {\n        p.set_exception(std::runtime_error(\"foo\"));\n    });\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_destroy_promise_after_state_take_value) {\n    future<> f = make_ready_future<>();\n    auto p = std::make_unique<seastar::promise<>>();\n    f = p->get_future();\n    p->set_value();\n    auto g = f.then([] {});\n    p.reset();\n    return g;\n}\n\nSEASTAR_THREAD_TEST_CASE(test_exception_future_with_backtrace) {\n    int counter = 0;\n    auto inner = [&] (bool return_exception) mutable {\n        if (!return_exception) {\n            return make_ready_future<int>(++counter);\n        } else {\n            return make_exception_future_with_backtrace<int>(expected_exception());\n        }\n    };\n    auto outer = [&] (bool return_exception) {\n        return inner(return_exception).then([] (int i) {\n            return make_ready_future<int>(-i);\n        });\n    };\n\n    BOOST_REQUIRE_EQUAL(outer(false).get(), -1);\n    BOOST_REQUIRE_EQUAL(counter, 1);\n\n    BOOST_CHECK_THROW(outer(true).get(), expected_exception);\n    BOOST_REQUIRE_EQUAL(counter, 1);\n\n    // Example code where we expect a \"Exceptional future ignored\"\n    // warning.\n    (void)outer(true).then_wrapped([](future<int> fut) {\n        reactor::test::with_allow_abandoned_failed_futures(1, [fut = std::move(fut)]() mutable {\n            auto foo = std::move(fut);\n        });\n    });\n}\n\nclass throw_on_move {\n    int _i;\npublic:\n    throw_on_move(int i = 0) noexcept {\n        _i = i;\n    }\n    throw_on_move(const throw_on_move&) = delete;\n    throw_on_move(throw_on_move&&) {\n        _i = -1;\n        throw expected_exception();\n    }\n\n    int value() const {\n        return _i;\n    }\n};\n\nSEASTAR_TEST_CASE(test_async_throw_on_move) {\n    return async([] (throw_on_move t) {\n        BOOST_CHECK(false);\n    }, throw_on_move()).handle_exception_type([] (const expected_exception&) {\n        return make_ready_future<>();\n    });\n}\n\nfuture<> func4() {\n    return yield().then([] {\n        seastar_logger.info(\"backtrace: {}\", current_backtrace());\n    });\n}\n\nvoid func3() {\n    seastar::async([] {\n        func4().get();\n    }).get();\n}\n\nfuture<> func2() {\n    return seastar::async([] {\n        func3();\n    });\n}\n\nfuture<> func1() {\n    return yield().then([] {\n        return func2();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_backtracing) {\n    func1().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_then_unpack) {\n    make_ready_future<std::tuple<>>().then_unpack([] () {\n        BOOST_REQUIRE(true);\n    }).get();\n    make_ready_future<std::tuple<int>>(std::tuple<int>(1)).then_unpack([] (int x) {\n        BOOST_REQUIRE(x == 1);\n    }).get();\n    make_ready_future<std::tuple<int, long>>(std::tuple<int, long>(1, 2)).then_unpack([] (int x, long y) {\n        BOOST_REQUIRE(x == 1 && y == 2);\n    }).get();\n    make_ready_future<std::tuple<std::unique_ptr<int>>>(std::tuple(std::make_unique<int>(42))).then_unpack([] (std::unique_ptr<int> p1) {\n        BOOST_REQUIRE(*p1 == 42);\n    }).get();\n}\n\nfuture<> test_then_function_f() {\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_then_function) {\n    return make_ready_future<>().then(test_then_function_f);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_with_gate) {\n    gate g;\n    int counter = 0;\n    int gate_closed_errors = 0;\n    int other_errors = 0;\n\n    // test normal operation when gate is opened\n    BOOST_CHECK_NO_THROW(with_gate(g, [&] { counter++; }).get());\n    BOOST_REQUIRE_EQUAL(counter, 1);\n\n    // test that an exception returned by the calling func\n    // is propagated to with_gate future\n    counter = gate_closed_errors = other_errors = 0;\n    BOOST_CHECK_NO_THROW(with_gate(g, [&] {\n            counter++;\n            return make_exception_future<>(expected_exception());\n        }).handle_exception_type([&] (gate_closed_exception& e) {\n            gate_closed_errors++;\n        }).handle_exception([&] (std::exception_ptr) {\n            other_errors++;\n        }).get());\n    BOOST_REQUIRE(counter);\n    BOOST_REQUIRE(!gate_closed_errors);\n    BOOST_REQUIRE(other_errors);\n\n    g.close().get();\n\n    // test that with_gate.get() throws when the gate is closed\n    counter = gate_closed_errors = other_errors = 0;\n    BOOST_CHECK_THROW(with_gate(g, [&] { counter++; }).get(), gate_closed_exception);\n    BOOST_REQUIRE(!counter);\n\n    // test that with_gate throws when the gate is closed\n    counter = gate_closed_errors = other_errors = 0;\n    BOOST_CHECK_THROW(with_gate(g, [&] {\n            counter++;\n        }).then_wrapped([&] (future<> f) {\n            auto eptr = f.get_exception();\n            try {\n                std::rethrow_exception(eptr);\n            } catch (gate_closed_exception& e) {\n                gate_closed_errors++;\n            } catch (...) {\n                other_errors++;\n            }\n        }).get(), gate_closed_exception);\n    BOOST_REQUIRE(!counter);\n    BOOST_REQUIRE(!gate_closed_errors);\n    BOOST_REQUIRE(!other_errors);\n\n    // test that try_with_gate returns gate_closed_exception when the gate is closed\n    counter = gate_closed_errors = other_errors = 0;\n    try_with_gate(g, [&] { counter++; }).handle_exception_type([&] (gate_closed_exception& e) {\n        gate_closed_errors++;\n    }).handle_exception([&] (std::exception_ptr) {\n        other_errors++;\n    }).get();\n    BOOST_REQUIRE(!counter);\n    BOOST_REQUIRE(gate_closed_errors);\n    BOOST_REQUIRE(!other_errors);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_max_concurrent_for_each) {\n    BOOST_TEST_MESSAGE(\"empty range\");\n    max_concurrent_for_each(std::vector<int>(), 3, [] (int) {\n        BOOST_FAIL(\"should not reach\");\n        return make_exception_future<>(std::bad_function_call());\n    }).get();\n\n    auto range = boost::copy_range<std::vector<int>>(boost::irange(1, 8));\n\n    BOOST_TEST_MESSAGE(\"iterator\");\n    auto sum = 0;\n    max_concurrent_for_each(range.begin(), range.end(), 3,  [&sum] (int v) {\n        sum += v;\n        return make_ready_future<>();\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"const iterator\");\n    sum = 0;\n    max_concurrent_for_each(range.cbegin(), range.cend(), 3,  [&sum] (int v) {\n        sum += v;\n        return make_ready_future<>();\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"reverse iterator\");\n    sum = 0;\n    max_concurrent_for_each(range.rbegin(), range.rend(), 3,  [&sum] (int v) {\n        sum += v;\n        return make_ready_future<>();\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"immediate result\");\n    sum = 0;\n    max_concurrent_for_each(range, 3,  [&sum] (int v) {\n        sum += v;\n        return make_ready_future<>();\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"suspend\");\n    sum = 0;\n    max_concurrent_for_each(range, 3, [&sum] (int v) {\n        return yield().then([&sum, v] {\n            sum += v;\n        });\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"throw immediately\");\n    sum = 0;\n    BOOST_CHECK_EXCEPTION(max_concurrent_for_each(range, 3, [&sum] (int v) {\n        sum += v;\n        if (v == 1) {\n            throw 5;\n        }\n        return make_ready_future<>();\n    }).get(), int, [] (int v) { return v == 5; });\n    BOOST_REQUIRE_EQUAL(sum, 28);\n\n    BOOST_TEST_MESSAGE(\"throw after suspension\");\n    sum = 0;\n    BOOST_CHECK_EXCEPTION(max_concurrent_for_each(range, 3, [&sum] (int v) {\n        return yield().then([&sum, v] {\n            sum += v;\n            if (v == 2) {\n                throw 5;\n            }\n        });\n    }).get(), int, [] (int v) { return v == 5; });\n\n    BOOST_TEST_MESSAGE(\"concurrency higher than vector length\");\n    sum = 0;\n    max_concurrent_for_each(range, range.size() + 3,  [&sum] (int v) {\n        sum += v;\n        return make_ready_future<>();\n    }).get();\n    BOOST_REQUIRE_EQUAL(sum, 28);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_for_each_set) {\n    std::bitset<32> s;\n    s.set(4);\n    s.set(0);\n\n    auto range = bitsets::for_each_set(s);\n    unsigned res = 0;\n    do_for_each(range, [&res] (auto i) {\n        res |= 1 << i;\n    }).get();\n    BOOST_REQUIRE_EQUAL(res, 17);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_yield) {\n    bool flag = false;\n    auto one = yield().then([&] {\n        flag = true;\n    });\n    BOOST_REQUIRE_EQUAL(flag, false);\n    one.get();\n    BOOST_REQUIRE_EQUAL(flag, true);\n\n#ifndef SEASTAR_DEBUG\n    // same thing, with now(), but for non-DEBUG only, otherwise .then() doesn't\n    // use the ready-future fast-path and always schedules a task\n    flag = false;\n    auto two = now().then([&] {\n        flag = true;\n    });\n    // now() does not yield\n    BOOST_REQUIRE_EQUAL(flag, true);\n#endif\n}\n\n// The seastar::make_exception_future() function has two distinct cases - it\n// can create an exceptional future from an existing std::exception_ptr, or\n// from an any object which will be wrapped in an std::exception_ptr using\n// std::make_exception_ptr. We want to test here these two cases, as well\n// what happens when the given parameter is almost a std::exception_ptr,\n// just with different qualifiers, like && or const (see issue #1010).\nSEASTAR_TEST_CASE(test_make_exception_future) {\n    // When make_exception_future() is given most types - like int and\n    // std::runtime_error - a copy of the given value get stored in the\n    // future (internally, it is wrapped using std::make_exception_ptr):\n    future<> f1 = make_exception_future<>(3);\n    BOOST_REQUIRE(f1.failed());\n    BOOST_REQUIRE_THROW(f1.get(), int);\n    future<> f2 = make_exception_future<>(std::runtime_error(\"hello\"));\n    BOOST_REQUIRE(f2.failed());\n    BOOST_REQUIRE_THROW(f2.get(), std::runtime_error);\n    // However, if make_exception_future() is given an std::exception_ptr\n    // it behaves differently - the exception stored in the future will be\n    // the one held in the given exception_ptr - not the exception_ptr object\n    // itself.\n    std::exception_ptr e3 = std::make_exception_ptr(3);\n    future<> f3 = make_exception_future<>(e3);\n    BOOST_REQUIRE(f3.failed());\n    BOOST_REQUIRE_THROW(f3.get(), int); // expecting int, not std::exception_ptr\n    // If make_exception_future() is given an std::exception_ptr by rvalue,\n    // it should also work correctly:\n    // An unnamed rvalue:\n    future<> f4 = make_exception_future<>(std::make_exception_ptr(3));\n    BOOST_REQUIRE(f4.failed());\n    BOOST_REQUIRE_THROW(f4.get(), int); // expecting int, not std::exception_ptr\n    // A rvalue reference (a move):\n    std::exception_ptr e5 = std::make_exception_ptr(3);\n    future<> f5 = make_exception_future<>(std::move(e5)); // note std::move()\n    BOOST_REQUIRE(f5.failed());\n    BOOST_REQUIRE_THROW(f5.get(), int); // expecting int, not std::exception_ptr\n    // A rvalue reference to a *const* exception_ptr:\n    // Reproduces issue #1010 - a const exception_ptr sounds odd, but can\n    // happen accidentally when capturing an exception_ptr in a non-mutable\n    // lambda.\n    // Note that C++ is fine with std::move() being used on a const object,\n    // it will simply fall back to a copy instead of a move. And a copy does\n    // work (without std::move(), it works).\n    const std::exception_ptr e6 = std::make_exception_ptr(3); // note const!\n    future<> f6 = make_exception_future<>(std::move(e6)); // note std::move()\n    BOOST_REQUIRE(f6.failed());\n    BOOST_REQUIRE_THROW(f6.get(), int); // expecting int, not std::exception_ptr\n\n    return make_ready_future<>();\n}\n\n// Reproduce use-after-free similar to #1514\nSEASTAR_TEST_CASE(test_run_in_background) {\n    engine().run_in_background([] {\n        return sleep(1ms).then([] {\n            return smp::invoke_on_all([] {\n                return sleep(1ms);\n            });\n        });\n    });\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_manual_clock_advance) {\n    bool expired = false;\n    auto t = timer<manual_clock>([&] {\n        expired = true;\n    });\n    t.arm(2ms);\n    manual_clock::advance(1ms);\n    BOOST_REQUIRE(!expired);\n    manual_clock::advance(1ms);\n    BOOST_REQUIRE(expired);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_ready_future_across_shards) {\n    if (smp::count == 1) {\n        seastar_logger.info(\"test_ready_future_across_shards requires at least 2 shards\");\n        return;\n    }\n\n    auto other_shard = (this_shard_id() + 1) % smp::count;\n    auto f1 = make_ready_future<int>(42);\n    smp::submit_to(other_shard, [f1 = std::move(f1)] () mutable {\n        BOOST_REQUIRE_EQUAL(f1.get(), 42);\n    }).get();\n}\n\nstruct canary_stats {\n    int copies = 0;\n    int moves = 0;\n    int copy_assignments = 0;\n    int move_assignments = 0;\n    // calls to operator()()\n    int lvalue_calls = 0;\n    // calls to operator()() &&\n    int rvalue_calls = 0;\n};\n\nstruct canary {\n    enum class state {\n        alive = 100,\n        moved_from = 101,\n        destroyed = 102\n    };\n\n    std::shared_ptr<canary_stats> _stats;\n    state _state = state::alive;\n\n    explicit canary(std::shared_ptr<canary_stats> stats) : _stats(std::move(stats)) {\n        seastar_logger.trace(\"canary: {} constructor\", fmt::ptr(this));\n    }\n\n    canary(const canary& other) : _stats(other._stats), _state(state::alive) {\n        ++_stats->copies;\n        seastar_logger.trace(\"canary: {} copy constructor (total copies: {})\", fmt::ptr(this), _stats->copies);\n    }\n\n    canary(canary&& other) noexcept : _stats(other._stats), _state(state::alive) {\n        ++_stats->moves;\n        other._state = state::moved_from;\n        seastar_logger.trace(\"canary: {} move constructor (total moves: {})\", fmt::ptr(this), _stats->moves);\n    }\n\n    canary& operator=(const canary& other) {\n        if (this != &other) {\n            _stats = other._stats;\n            _state = state::alive;\n            ++_stats->copy_assignments;\n        }\n        seastar_logger.trace(\"canary: {} copy assignment operator (total copy assignments: {})\", fmt::ptr(this), _stats->copy_assignments);\n        return *this;\n    }\n\n    canary& operator=(canary&& other) noexcept {\n        if (this != &other) {\n            _stats = std::move(other._stats);\n            _state = state::alive;\n            ++_stats->move_assignments;\n            other._state = state::moved_from;\n        }\n        seastar_logger.trace(\"canary: {} move assignment operator (total move assignments: {})\", fmt::ptr(this), _stats->move_assignments);\n        return *this;\n    }\n\n    ~canary() {\n        _state = state::destroyed;\n        seastar_logger.trace(\"canary: {} destructor\", fmt::ptr(this));\n    }\n\n    void assert_valid() const {\n        BOOST_REQUIRE_MESSAGE(_state == state::alive, \"canary not alive: \" << static_cast<int>(_state));\n    }\n\n    bool is_moved_from() const {\n        return _state == state::moved_from;\n    }\n\n    const canary_stats& stats() const { return *_stats; }\n};\n\nstruct nocopy {\n    nocopy() = default;\n    nocopy(const nocopy&) = delete;\n    nocopy(nocopy&&) = default;\n};\n\n// return if_erased if TYPE_ERASE_MORE is defined, else if_not_erased\n// useful to tests where the internal behavior differs based on type erasure\nconstexpr auto if_erased(auto if_erased, auto if_not_erased) {\n#ifdef SEASTAR_TYPE_ERASE_MORE\n    return if_erased;\n#else\n    return if_not_erased;\n#endif\n}\n\n// returns 1 if SEASTAR_DEBUG or 0 otherwise\n// useful since in DEBUG mode we take different code paths\n// and so we may need to adjust expected values\nconstexpr int is_debug() {\n#ifdef SEASTAR_DEBUG\n    return 1;\n#else\n    return 0;\n#endif\n}\n\ntemplate<bool EnableCopy, bool ThenByValue, class Ret>\nstruct canary_callable_base {\n    canary c;\n\n    explicit canary_callable_base(std::shared_ptr<canary_stats> stats) : c{stats} {}\n\n    canary_callable_base(const canary_callable_base&) requires EnableCopy = default;\n    canary_callable_base& operator=(const canary_callable_base&) requires EnableCopy = default;\n    canary_callable_base(const canary_callable_base&) requires (!EnableCopy) = delete;\n    canary_callable_base& operator=(const canary_callable_base&) requires (!EnableCopy) = delete;\n    canary_callable_base(canary_callable_base&&) = default;\n    canary_callable_base& operator=(canary_callable_base&&) = default;\n\n    auto make() {\n        if constexpr (std::is_same_v<Ret, future<>>) {\n            return make_ready_future();\n        }\n    }\n\n    Ret operator()() & {\n        c.assert_valid();\n        c._stats->lvalue_calls++;\n        return make();\n    }\n\n    Ret operator()() && {\n        c.assert_valid();\n        c._stats->rvalue_calls++;\n        return make();\n    }\n\n    // below are called for then_wrapped\n    Ret operator()(future<> f) & requires ThenByValue {\n        BOOST_CHECK(!f.failed());\n        c.assert_valid();\n        c._stats->lvalue_calls++;\n        return make();\n    }\n\n    Ret operator()(future<> f) && requires ThenByValue {\n        BOOST_CHECK(!f.failed());\n        c.assert_valid();\n        c._stats->rvalue_calls++;\n        return make();\n    }\n\n    Ret operator()(future<>&& f) & requires (!ThenByValue) {\n        BOOST_CHECK(!f.failed());\n        c.assert_valid();\n        c._stats->lvalue_calls++;\n        return make();\n    }\n\n    Ret operator()(future<>&& f) && requires (!ThenByValue) {\n        BOOST_CHECK(!f.failed());\n        c.assert_valid();\n        c._stats->rvalue_calls++;\n        return make();\n    }\n\n};\n\n\ntemplate <class Traits>\nvoid test_lifetimes_and_movement() {\n    // test that calling the future \"entry points\", such as then(), finally(), etc\n    // do the expected number of copies and moves, and that the expected operator()\n    // overload is called (lvalue vs rvalue-qualified), and that objects are not used\n    // after destruction and so on.\n\n    using canary_callable = canary_callable_base<true, Traits::then_by_value, typename Traits::return_type>;\n    using canary_callable_move_only = canary_callable_base<false, Traits::then_by_value, typename Traits::return_type>;\n\n    auto extra_moves = Traits::extra_moves;\n\n    auto apply = [] (future<>&& f, auto&& func) {\n        return Traits::apply_continuation(std::move(f), std::forward<decltype(func)>(func));\n    };\n\n    auto apply_unready = [&] (auto&& func) {\n        return apply(yield(), std::forward<decltype(func)>(func)).get();\n    };\n\n    auto apply_ready = [&] (auto&& func) {\n        return apply(make_ready_future(), std::forward<decltype(func)>(func)).get();\n    };\n\n    // Test 1: Passing move-only rvalue lambda to then()\n    BOOST_TEST_CHECKPOINT(\"Test 1: Passing move-only rvalue lambda to then()\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        apply_unready(canary_callable_move_only{stats});\n        // Rvalue lambda gets moved into then() implementation\n        BOOST_CHECK_EQUAL(stats->copies, 0);\n        BOOST_CHECK_EQUAL(stats->moves, if_erased(4, 1) + extra_moves);\n        BOOST_CHECK_EQUAL(stats->copy_assignments, 0);\n        BOOST_CHECK_EQUAL(stats->move_assignments, 0);\n\n        BOOST_CHECK_EQUAL(stats->lvalue_calls, 0);\n        BOOST_CHECK_EQUAL(stats->rvalue_calls, 1);\n    }\n\n    // Test 2: Passing lvalue lambda to then()\n    BOOST_TEST_CHECKPOINT(\"Test 2: Passing lvalue lambda to then()\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        canary_callable callable{stats};\n        apply_unready(callable);\n        // Lvalue lambda gets copied once, then moved through then() chain\n        BOOST_CHECK_EQUAL(stats->copies, 1);\n        BOOST_CHECK_EQUAL(stats->moves, if_erased(4, 1) + extra_moves);\n        BOOST_CHECK_EQUAL(stats->copy_assignments, 0);\n        BOOST_CHECK_EQUAL(stats->move_assignments, 0);\n\n        BOOST_CHECK_EQUAL(stats->lvalue_calls, 0);\n        BOOST_CHECK_EQUAL(stats->rvalue_calls, 1);\n\n        BOOST_CHECK(!callable.c.is_moved_from());\n    }\n\n    // Test 3: Passing std::move(lvalue lambda) to then()\n    BOOST_TEST_CHECKPOINT(\"Test 3: Passing std::move(lvalue lambda) to then()\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        canary_callable_move_only callable{stats};\n        apply_unready(std::move(callable));\n        // std::move(lvalue lambda) behaves like rvalue lambda - only moves\n        BOOST_CHECK_EQUAL(stats->copies, 0);\n        BOOST_CHECK_EQUAL(stats->moves, if_erased(4, 1) + extra_moves);\n        BOOST_CHECK_EQUAL(stats->copy_assignments, 0);\n        BOOST_CHECK_EQUAL(stats->move_assignments, 0);\n\n        BOOST_CHECK_EQUAL(stats->lvalue_calls, 0);\n        BOOST_CHECK_EQUAL(stats->rvalue_calls, 1);\n\n        BOOST_CHECK(callable.c.is_moved_from());\n    }\n\n    // for ready futures at the start of the chain, SEASTAR_DEBUG\n    // incurs an extra move since we disable the \"sync exec\" path\n    // for ready futures in debug\n\n    // Test 4: ready future with rvalue lambda\n    BOOST_TEST_CHECKPOINT(\"Test 4: ready future with rvalue lambda\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        apply_ready(canary_callable_move_only{stats});\n        // Ready future: lambda moved fewer times since future is already ready\n        BOOST_CHECK_EQUAL(stats->copies, 0);\n        BOOST_CHECK_EQUAL(stats->moves, if_erased(3, 0) + is_debug() + extra_moves);\n        BOOST_CHECK_EQUAL(stats->copy_assignments, 0);\n        BOOST_CHECK_EQUAL(stats->move_assignments, 0);\n\n        BOOST_CHECK_EQUAL(stats->lvalue_calls, 0);\n        BOOST_CHECK_EQUAL(stats->rvalue_calls, 1);\n    }\n\n    // Test 5: ready future with lvalue lambda\n    BOOST_TEST_CHECKPOINT(\"Test 5: ready future with lvalue lambda\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        canary_callable callable{stats};\n        apply_ready(callable);\n        // Ready future: lvalue lambda copied once, then moved fewer times\n        BOOST_CHECK_EQUAL(stats->copies, 1);\n        BOOST_CHECK_EQUAL(stats->moves, if_erased(3, 0) + is_debug() + extra_moves);\n        BOOST_CHECK_EQUAL(stats->copy_assignments, 0);\n        BOOST_CHECK_EQUAL(stats->move_assignments, 0);\n\n        BOOST_CHECK_EQUAL(stats->lvalue_calls, 0);\n        BOOST_CHECK_EQUAL(stats->rvalue_calls, 1);\n\n        BOOST_CHECK(!callable.c.is_moved_from());\n    }\n\n    // Test 6: tests that a local lvalue passed into then()-alike\n    // has its lifetime extended until after the continuation runs.\n    // See https://github.com/scylladb/seastar/issues/3157\n    BOOST_TEST_CHECKPOINT(\"Test 6: local lvalue lifetime extension\");\n    {\n        auto stats = std::make_shared<canary_stats>();\n        auto local_lvalue = [=]() {\n            canary_callable local{stats};\n            // local is destroyed after the return, but the continuation\n            // runs after that, so it must be copied/moved to avoid UB\n            return apply(yield(), local);\n        };\n\n        local_lvalue().get();\n    }\n}\n\nstruct base_traits {\n    static constexpr bool then_by_value = true;\n    static constexpr int extra_moves = 0;\n    using return_type = future<>;\n};\n\nstruct then_traits : base_traits {\n    static auto apply_continuation(future<>&& f, auto&& cont) {\n        return f.then(std::forward<decltype(cont)>(cont));\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_lifetimes_and_copies_then) {\n    test_lifetimes_and_movement<then_traits>();\n}\n\nstruct then_wrapped_traits : base_traits {\n    static auto apply_continuation(future<>&& f, auto&& cont) {\n        return f.then_wrapped(std::forward<decltype(cont)>(cont));\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_lifetimes_and_copies_then_wrapped) {\n    test_lifetimes_and_movement<then_wrapped_traits>();\n}\n\nstruct then_wrapped_ref_traits : then_wrapped_traits {\n    static constexpr bool then_by_value = false;\n};\n\nSEASTAR_THREAD_TEST_CASE(test_lifetimes_and_copies_then_wrapped_ref) {\n    // when then_wrapped is called with func taking `future&&` a different\n    // code path is used so we test this separately\n    test_lifetimes_and_movement<then_wrapped_ref_traits>();\n}\n\nstruct finally_traits : base_traits {\n    static constexpr int extra_moves = 1; // finally does an extra move into the finally_body\n    static auto apply_continuation(future<>&& f, auto&& cont) {\n        return f.finally(std::forward<decltype(cont)>(cont));\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_lifetimes_and_copies_finally) {\n    test_lifetimes_and_movement<finally_traits>();\n}\n\nstruct finally_void_traits : finally_traits {\n    using return_type = void;\n    static auto apply_continuation(future<>&& f, auto&& cont) {\n        return f.finally(std::forward<decltype(cont)>(cont));\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_lifetimes_and_copies_finally_void) {\n    // when finally does not return a future, a different code path\n    // is taken, so test it\n    test_lifetimes_and_movement<finally_void_traits>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_foreign_promise_set_value) {\n    if (smp::count == 1) {\n        seastar_logger.info(\"test_foreign_promise_set_value requires at least 2 shards\");\n        return;\n    }\n\n    promise<int> pr;\n    auto other_shard = (this_shard_id() + 1) % smp::count;\n\n    auto getter = pr.get_future();\n\n    auto setter = smp::submit_to(other_shard, [&] {\n        pr.set_value(this_shard_id());\n    });\n\n    setter.get();\n\n    BOOST_REQUIRE_EQUAL(getter.get(), other_shard);\n}\n\n#if __cpp_lib_three_way_comparison\nSEASTAR_TEST_CASE(test_bool_class_spaceship_operator) {\n    using test_bool_class = bool_class<struct test_bool_class_tag>;\n    auto a = test_bool_class::yes;\n    auto b = test_bool_class::no;\n\n    BOOST_REQUIRE_EQUAL(a, a);\n    BOOST_REQUIRE(a <=> a == 0);\n    BOOST_REQUIRE_EQUAL(a, test_bool_class(true));\n    BOOST_REQUIRE(a <=> test_bool_class(true) == 0);\n    BOOST_REQUIRE_EQUAL(b, b);\n    BOOST_REQUIRE(b <=> b == 0);\n    BOOST_REQUIRE_EQUAL(b, test_bool_class(false));\n    BOOST_REQUIRE(b <=> test_bool_class(false) == 0);\n    BOOST_REQUIRE_GT(a, b);\n    BOOST_REQUIRE(a <=> b > 0);\n    BOOST_REQUIRE_LT(b, a);\n    BOOST_REQUIRE(b <=> a < 0);\n\n    return make_ready_future();\n}\n#endif\n\nvoid compile_tests() {\n    // never executed, only to check that certain constructs compile\n\n    auto f = make_ready_future();\n\n    // check that it compiles with rvalue handlers\n    f = std::move(f).handle_exception([](auto e){});\n    f = std::move(f).handle_exception_type([](const std::runtime_error& e){});\n\n    // and lvalue handlers\n    auto lvalue_exn_handler = [](std::exception_ptr e){};\n    f = std::move(f).handle_exception(lvalue_exn_handler);\n\n    auto lvalue_type_handler = [](const std::runtime_error& e){};\n    f = std::move(f).handle_exception_type(lvalue_type_handler);\n\n    // static asserts for function_traits\n\n    // Test plain function pointer\n    int plain_func(double, char);\n    static_assert(std::is_same_v<function_traits<decltype(plain_func)>::return_type, int>);\n    static_assert(function_traits<decltype(plain_func)>::arity == 2);\n    static_assert(std::is_same_v<function_traits<decltype(plain_func)>::arg<0>::type, double>);\n    static_assert(std::is_same_v<function_traits<decltype(plain_func)>::arg<1>::type, char>);\n    static_assert(std::is_same_v<function_traits<decltype(plain_func)>::args_as_tuple, std::tuple<double, char>>);\n\n    // Test function pointer type\n    using func_ptr = void(*)(int, float, long);\n    static_assert(std::is_same_v<function_traits<func_ptr>::return_type, void>);\n    static_assert(function_traits<func_ptr>::arity == 3);\n    static_assert(std::is_same_v<function_traits<func_ptr>::arg<0>::type, int>);\n    static_assert(std::is_same_v<function_traits<func_ptr>::arg<1>::type, float>);\n    static_assert(std::is_same_v<function_traits<func_ptr>::arg<2>::type, long>);\n\n    // Test noexcept function pointer\n    using noexcept_func_ptr = bool(*)(std::string) noexcept;\n    static_assert(std::is_same_v<function_traits<noexcept_func_ptr>::return_type, bool>);\n    static_assert(function_traits<noexcept_func_ptr>::arity == 1);\n    static_assert(std::is_same_v<function_traits<noexcept_func_ptr>::arg<0>::type, std::string>);\n\n    // Test lambda (non-capturing)\n    auto lambda = [](int x, double y) -> float { return x + y; };\n    static_assert(std::is_same_v<function_traits<decltype(lambda)>::return_type, float>);\n    static_assert(function_traits<decltype(lambda)>::arity == 2);\n    static_assert(std::is_same_v<function_traits<decltype(lambda)>::arg<0>::type, int>);\n    static_assert(std::is_same_v<function_traits<decltype(lambda)>::arg<1>::type, double>);\n\n    // Test mutable lambda\n    auto mutable_lambda = [](char c) mutable -> int { return c; };\n    static_assert(std::is_same_v<function_traits<decltype(mutable_lambda)>::return_type, int>);\n    static_assert(function_traits<decltype(mutable_lambda)>::arity == 1);\n    static_assert(std::is_same_v<function_traits<decltype(mutable_lambda)>::arg<0>::type, char>);\n\n    // Test lambda reference\n    auto lambda_ref_test = [](short s, long l) -> double { return s + l; };\n    using lambda_ref_type = decltype(lambda_ref_test)&;\n    static_assert(std::is_same_v<function_traits<lambda_ref_type>::return_type, double>);\n    static_assert(function_traits<lambda_ref_type>::arity == 2);\n    static_assert(std::is_same_v<function_traits<lambda_ref_type>::arg<0>::type, short>);\n    static_assert(std::is_same_v<function_traits<lambda_ref_type>::arg<1>::type, long>);\n\n    // Test const lambda reference\n    const auto const_lambda = [](int x) -> int { return x * 2; };\n    static_assert(std::is_same_v<function_traits<decltype(const_lambda)>::return_type, int>);\n    static_assert(function_traits<decltype(const_lambda)>::arity == 1);\n\n    // Test std::reference_wrapper of lambda\n    auto wrapped_lambda = [](std::string s, size_t n) -> bool { return s.size() > n; };\n    auto ref_wrapper = std::ref(wrapped_lambda);\n    static_assert(std::is_same_v<function_traits<decltype(ref_wrapper)>::return_type, bool>);\n    static_assert(function_traits<decltype(ref_wrapper)>::arity == 2);\n    static_assert(std::is_same_v<function_traits<decltype(ref_wrapper)>::arg<0>::type, std::string>);\n    static_assert(std::is_same_v<function_traits<decltype(ref_wrapper)>::arg<1>::type, size_t>);\n\n    // Test member function pointer\n    struct test_struct {\n        int member_func(double d, char c) { return d + c; }\n        void const_member_func(float f) const {}\n        bool noexcept_member_func(int x) noexcept { return x > 0; }\n        void const_noexcept_member_func(long l) const noexcept {}\n    };\n\n    using member_func_type = decltype(&test_struct::member_func);\n    static_assert(std::is_same_v<function_traits<member_func_type>::return_type, int>);\n    static_assert(function_traits<member_func_type>::arity == 2);\n    static_assert(std::is_same_v<function_traits<member_func_type>::arg<0>::type, double>);\n    static_assert(std::is_same_v<function_traits<member_func_type>::arg<1>::type, char>);\n\n    // Test const member function pointer\n    using const_member_func_type = decltype(&test_struct::const_member_func);\n    static_assert(std::is_same_v<function_traits<const_member_func_type>::return_type, void>);\n    static_assert(function_traits<const_member_func_type>::arity == 1);\n    static_assert(std::is_same_v<function_traits<const_member_func_type>::arg<0>::type, float>);\n\n    // Test noexcept member function pointer\n    using noexcept_member_func_type = decltype(&test_struct::noexcept_member_func);\n    static_assert(std::is_same_v<function_traits<noexcept_member_func_type>::return_type, bool>);\n    static_assert(function_traits<noexcept_member_func_type>::arity == 1);\n    static_assert(std::is_same_v<function_traits<noexcept_member_func_type>::arg<0>::type, int>);\n\n    // Test const noexcept member function pointer\n    using const_noexcept_member_func_type = decltype(&test_struct::const_noexcept_member_func);\n    static_assert(std::is_same_v<function_traits<const_noexcept_member_func_type>::return_type, void>);\n    static_assert(function_traits<const_noexcept_member_func_type>::arity == 1);\n    static_assert(std::is_same_v<function_traits<const_noexcept_member_func_type>::arg<0>::type, long>);\n\n    // Test function with no arguments\n    auto no_args_lambda = []() -> int { return 42; };\n    static_assert(std::is_same_v<function_traits<decltype(no_args_lambda)>::return_type, int>);\n    static_assert(function_traits<decltype(no_args_lambda)>::arity == 0);\n    static_assert(std::is_same_v<function_traits<decltype(no_args_lambda)>::args_as_tuple, std::tuple<>>);\n\n    // Test function with many arguments\n    auto many_args = [](int, float, double, char, short, long, bool, std::string) -> void {};\n    static_assert(std::is_same_v<function_traits<decltype(many_args)>::return_type, void>);\n    static_assert(function_traits<decltype(many_args)>::arity == 8);\n    static_assert(std::is_same_v<function_traits<decltype(many_args)>::arg<3>::type, char>);\n    static_assert(std::is_same_v<function_traits<decltype(many_args)>::arg<7>::type, std::string>);\n}\n"
  },
  {
    "path": "tests/unit/gate_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2021 ScyllaDB\n */\n\n#include <exception>\n#include <functional>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/format.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/util/log.hh>\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\nusing namespace seastar;\n\nstatic_assert(std::is_nothrow_default_constructible_v<gate>);\nstatic_assert(std::is_nothrow_move_constructible_v<gate>);\nstatic_assert(std::is_nothrow_move_assignable_v<gate>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<named_gate>);\nstatic_assert(std::is_nothrow_move_constructible_v<named_gate>);\nstatic_assert(std::is_nothrow_move_assignable_v<named_gate>);\n\ntemplate <typename Func>\nstatic future<> check_gate_closed_exception(Func func) {\n    try {\n        co_await futurize_invoke(func);\n        BOOST_FAIL(\"func was expected to throw gate_closed_exception\");\n    } catch (const gate_closed_exception& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), \"gate closed\");\n    } catch (...) {\n        BOOST_FAIL(format(\"unexpected exception: {}\", std::current_exception()));\n    }\n}\n\nSEASTAR_TEST_CASE(basic_gate_test) {\n    gate g;\n\n    BOOST_REQUIRE_EQUAL(g.get_count(), 0);\n    BOOST_REQUIRE(!g.is_closed());\n    BOOST_REQUIRE_NO_THROW(g.check());\n    BOOST_REQUIRE_EQUAL(g.get_count(), 0);\n    BOOST_REQUIRE_NO_THROW(g.enter());\n    BOOST_REQUIRE_EQUAL(g.get_count(), 1);\n    auto gh0 = g.try_hold();\n    BOOST_REQUIRE(gh0.has_value());\n    BOOST_REQUIRE_EQUAL(g.get_count(), 2);\n    auto gh1 = g.hold();\n    BOOST_REQUIRE_EQUAL(g.get_count(), 3);\n    BOOST_REQUIRE(!g.is_closed());\n    auto f = g.close();\n    BOOST_REQUIRE(!f.available());\n    g.leave();\n    BOOST_REQUIRE_EQUAL(g.get_count(), 2);\n    gh0->release();\n    BOOST_REQUIRE_EQUAL(g.get_count(), 1);\n    gh1.release();\n    BOOST_REQUIRE_EQUAL(g.get_count(), 0);\n    BOOST_REQUIRE_NO_THROW(co_await std::move(f));\n    BOOST_REQUIRE(g.is_closed());\n}\n\nSEASTAR_TEST_CASE(gate_closed_test) {\n    gate g;\n\n    BOOST_REQUIRE(!g.is_closed());\n    BOOST_REQUIRE_NO_THROW(g.check());\n    BOOST_REQUIRE_NO_THROW(g.enter());\n    auto gh0 = g.try_hold();\n    BOOST_REQUIRE(gh0.has_value());\n    auto gh1 = g.hold();\n    BOOST_REQUIRE(!g.is_closed());\n    auto f = g.close();\n    BOOST_REQUIRE(!f.available());\n    g.leave();\n    gh0->release();\n    gh1.release();\n    BOOST_REQUIRE_NO_THROW(co_await std::move(f));\n    BOOST_REQUIRE(g.is_closed());\n\n    BOOST_REQUIRE(!g.try_hold().has_value());\n\n    co_await check_gate_closed_exception([&] { g.check(); });\n    co_await check_gate_closed_exception([&] { g.enter(); });\n    co_await check_gate_closed_exception([&] { g.hold(); });\n    co_await check_gate_closed_exception([&] () -> future<> { co_await with_gate(g, [] { return make_ready_future(); }); });\n    co_await check_gate_closed_exception([&] () -> future<> { co_await try_with_gate(g, [] { return make_ready_future(); }); });\n}\n\ntemplate <typename Func>\nstatic future<> check_named_gate_closed_exception(Func func, sstring name) {\n    try {\n        co_await futurize_invoke(func);\n        BOOST_FAIL(\"func was expected to throw gate_closed_exception\");\n    } catch (const gate_closed_exception& e) {\n        BOOST_REQUIRE_EQUAL(e.what(), fmt::format(\"{} gate closed\", name));\n    } catch (...) {\n        BOOST_FAIL(format(\"unexpected exception: {}\", std::current_exception()));\n    }\n}\n\nSEASTAR_TEST_CASE(named_gate_closed_test) {\n    auto test_named_gate = [] (named_gate& g, const sstring name) -> future<> {\n        if (!g.is_closed()) {\n            BOOST_REQUIRE_NO_THROW(g.check());\n            BOOST_REQUIRE_NO_THROW(g.enter());\n            auto gh0 = g.try_hold();\n            BOOST_REQUIRE(gh0.has_value());\n            auto gh1 = g.hold();\n            BOOST_REQUIRE(!g.is_closed());\n            auto f = g.close();\n            BOOST_REQUIRE(!f.available());\n            g.leave();\n            gh0->release();\n            gh1.release();\n            BOOST_REQUIRE_NO_THROW(co_await std::move(f));\n            BOOST_REQUIRE(g.is_closed());\n        }\n\n        BOOST_REQUIRE(!g.try_hold().has_value());\n\n        co_await check_named_gate_closed_exception([&] { g.check(); }, name);\n        co_await check_named_gate_closed_exception([&] { g.enter(); }, name);\n        co_await check_named_gate_closed_exception([&] { g.hold(); }, name);\n        co_await check_named_gate_closed_exception([&] () -> future<> { co_await with_gate(g, [] { return make_ready_future(); }); }, name);\n        co_await check_named_gate_closed_exception([&] () -> future<> { co_await try_with_gate(g, [] { return make_ready_future(); }); }, name);\n    };\n\n    BOOST_TEST_MESSAGE(\"test_named_gate(default_constructed)\");\n    named_gate default_constructed;\n    co_await test_named_gate(default_constructed, \"named\");\n\n    sstring foo = \"foo\";\n    BOOST_TEST_MESSAGE(\"test_named_gate(name_constructed)\");\n    named_gate name_constructed(foo);\n    co_await test_named_gate(name_constructed, foo);\n    BOOST_TEST_MESSAGE(\"test_named_gate(move_constructed)\");\n    auto move_constructed = std::move(name_constructed);\n    co_await test_named_gate(move_constructed, foo);\n    BOOST_TEST_MESSAGE(\"test_named_gate(move_assigned)\");\n    sstring bar = \"bar\";\n    named_gate move_assigned(foo);\n    move_assigned = named_gate(bar);\n    co_await test_named_gate(move_assigned, bar);\n}\n\nSEASTAR_TEST_CASE(unique_named_gate_test) {\n    auto g = std::make_unique<named_gate>(\"test gate\");\n    auto gh = g->hold();\n    gh.release();\n    g.reset();\n    return make_ready_future();\n}\n"
  },
  {
    "path": "tests/unit/generator_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2022 Kefu Chai ( tchaikov@gmail.com )\n */\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/as_future.hh>\n#include <seastar/coroutine/generator.hh>\n#include <seastar/coroutine/exception.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/bool_class.hh>\n#include <seastar/util/later.hh>\n#include <string>\n#include <string_view>\n#if __cplusplus >= 202302L && defined(__cpp_lib_generator)\n#include <generator>\ntemplate <typename T>\nusing sync_generator = std::generator<const T&>;\n#else\n#include \"tl-generator.hh\"\ntemplate <typename T>\nusing sync_generator = tl::generator<T>;\n#endif\n\nusing namespace seastar;\n\nusing do_suspend = bool_class<struct do_suspend_tag>;\n\nsync_generator<int>\nsync_fibonacci_sequence(unsigned count) {\n    auto a = 0, b = 1;\n    for (unsigned i = 0; i < count; ++i) {\n        if (std::numeric_limits<decltype(a)>::max() - a < b) {\n            throw std::out_of_range(\n                fmt::format(\"fibonacci[{}] is greater than the largest value of int\", i));\n        }\n        int next = std::exchange(a, std::exchange(b, a + b));\n        // tl::generator::yield_value() only accepts arguments of reference type,\n        // instead of a cv-qualified value.\n        co_yield next;\n    }\n}\n\ncoroutine::experimental::generator<int>\nasync_fibonacci_sequence(unsigned count, do_suspend suspend) {\n    auto a = 0, b = 1;\n    for (unsigned i = 0; i < count; ++i) {\n        if (std::numeric_limits<decltype(a)>::max() - a < b) {\n            throw std::out_of_range(\n                fmt::format(\"fibonacci[{}] is greater than the largest value of int\", i));\n        }\n        if (suspend) {\n            co_await yield();\n        }\n        co_yield std::exchange(a, std::exchange(b, a + b));\n    }\n}\n\nseastar::future<>\nverify_fib_drained(coroutine::experimental::generator<int> actual_fibs, unsigned count) {\n    auto expected_fibs = sync_fibonacci_sequence(count);\n    auto expected_fib = std::begin(expected_fibs);\n\n    while (auto actual_fib = co_await actual_fibs()) {\n        BOOST_REQUIRE(expected_fib != std::end(expected_fibs));\n        BOOST_REQUIRE_EQUAL(*actual_fib, *expected_fib);\n        ++expected_fib;\n    }\n    BOOST_REQUIRE(expected_fib == std::end(expected_fibs));\n}\n\nSEASTAR_TEST_CASE(test_generator_drained_with_suspend) {\n    constexpr int count = 4;\n    return verify_fib_drained(async_fibonacci_sequence(count, do_suspend::yes),\n                              count);\n}\n\nSEASTAR_TEST_CASE(test_generator_drained_without_suspend) {\n    constexpr int count = 4;\n    return verify_fib_drained(async_fibonacci_sequence(count, do_suspend::no),\n                              count);\n}\n\nseastar::future<> test_generator_not_drained(do_suspend suspend) {\n    auto fib = async_fibonacci_sequence(42, suspend);\n    auto actual_fib = co_await fib();\n    BOOST_REQUIRE(actual_fib.has_value());\n    BOOST_REQUIRE_EQUAL(*actual_fib, 0);\n}\n\nSEASTAR_TEST_CASE(test_generator_not_drained_with_suspend) {\n    return test_generator_not_drained(do_suspend::yes);\n}\n\nSEASTAR_TEST_CASE(test_generator_not_drained_without_suspend) {\n    return test_generator_not_drained(do_suspend::no);\n}\n\ncoroutine::experimental::generator<std::string_view, std::string>\ngenerate_value_and_ref(std::vector<std::string_view> strings) {\n    co_yield \"[\";\n    std::string s;\n    for (auto sv : strings) {\n        s = sv;\n        s.push_back(',');\n        co_yield s;\n    }\n    co_yield \"]\";\n}\n\nSEASTAR_TEST_CASE(test_generator_value_reference) {\n    using namespace std::literals;\n    std::vector<std::string_view> expected_quoted = {\"[\"sv, \"foo,\"sv, \"bar,\"sv, \"]\"sv};\n    auto actual_quoted = generate_value_and_ref({\"foo\"sv, \"bar\"sv});\n    size_t idx = 0;\n    while (auto actual = co_await actual_quoted()) {\n        BOOST_REQUIRE(idx < expected_quoted.size());\n        // generator<std::string_view, std::string> returns std::string_view as a value\n        BOOST_REQUIRE_EQUAL(*actual, expected_quoted[idx]);\n        ++idx;\n    }\n    BOOST_REQUIRE_EQUAL(idx, expected_quoted.size());\n}\n\ncoroutine::experimental::generator<std::string>\ngenerate_yield_rvalue_reference(const std::vector<std::string> strings) {\n    for (auto& s: strings) {\n        co_yield s;\n    }\n}\n\nSEASTAR_TEST_CASE(test_generator_rvalue_reference) {\n    std::vector<std::string> expected_strings = {\"hello\", \"world\"};\n    auto actual_strings = generate_yield_rvalue_reference(expected_strings);\n    size_t idx = 0;\n    while (auto actual = co_await actual_strings()) {\n        BOOST_REQUIRE(idx < expected_strings.size());\n        // generator<std::string> returns std::string&& wrapped in reference_wrapper\n        BOOST_REQUIRE_EQUAL(actual->get(), expected_strings[idx]);\n        ++idx;\n    }\n    BOOST_REQUIRE_EQUAL(idx, expected_strings.size());\n}\n\nSEASTAR_TEST_CASE(test_generator_move_ctor) {\n    constexpr int count = 4;\n    auto actual_fibs = async_fibonacci_sequence(count, do_suspend::no);\n    return verify_fib_drained(std::move(actual_fibs), count);\n}\n\nSEASTAR_TEST_CASE(test_generator_swap) {\n    int count_a = 4;\n    int count_b = 42;\n    auto fibs_a = async_fibonacci_sequence(count_a, do_suspend::no);\n    auto fibs_b = async_fibonacci_sequence(count_b, do_suspend::no);\n    std::swap(fibs_a, fibs_b);\n    std::swap(count_a, count_b);\n    co_await verify_fib_drained(std::move(fibs_a), count_a);\n    co_await verify_fib_drained(std::move(fibs_b), count_b);\n}\n\nstruct counter_t {\n    int n;\n    int* count;\n    counter_t(counter_t&& other) noexcept\n        : n{std::exchange(other.n, -1)},\n          count{std::exchange(other.count, nullptr)}\n    {}\n    counter_t(int n, int* count) noexcept\n        : n{n}, count{count} {\n        ++(*count);\n    }\n    ~counter_t() noexcept {\n        if (count) {\n            --(*count);\n        }\n    }\n};\n\nstd::ostream& operator<<(std::ostream& os, const counter_t& c) {\n    return os << c.n;\n}\n\ncoroutine::experimental::generator<counter_t>\nfiddle(int n, int* total) {\n    int i = 0;\n    while (true) {\n        if (i++ == n) {\n            throw std::invalid_argument(\"Eureka from generator!\");\n        }\n        co_yield counter_t{i, total};\n    }\n}\n\nSEASTAR_TEST_CASE(test_generator_throws_from_generator) {\n    int total = 0;\n    auto count_to = [total=&total](unsigned n) -> seastar::future<> {\n        auto count = fiddle(n, total);\n        for (unsigned i = 0; i < 2 * n; i++) {\n            co_await count();\n        }\n    };\n    auto f = co_await coroutine::as_future(count_to(42));\n    BOOST_REQUIRE(f.failed());\n    BOOST_REQUIRE_THROW(std::rethrow_exception(f.get_exception()), std::invalid_argument);\n    BOOST_REQUIRE_EQUAL(total, 0);\n}\n\nSEASTAR_TEST_CASE(test_generator_throws_from_consumer) {\n    int total = 0;\n    auto count_to = [total=&total](unsigned n) -> seastar::future<> {\n        auto count = fiddle(n, total);\n        for (unsigned i = 0; i < n; i++) {\n            if (i == n / 2) {\n                throw std::invalid_argument(\"Eureka from consumer!\");\n            }\n            co_await count();\n        }\n    };\n    auto f = co_await coroutine::as_future(count_to(42));\n    BOOST_REQUIRE(f.failed());\n    BOOST_REQUIRE_THROW(std::rethrow_exception(f.get_exception()), std::invalid_argument);\n    BOOST_REQUIRE_EQUAL(total, 0);\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_empty_sequence) {\n    using int_gen = coroutine::experimental::generator<int, int, std::vector<int>>;\n    auto seq = std::invoke([]() -> int_gen {\n        co_return;\n    });\n    while (auto val = co_await seq()) {\n        BOOST_FAIL(\"found element in an empty sequence\");\n    }\n}\n\ncoroutine::experimental::generator<int, int, std::vector<int>>\nasync_fibonacci_sequence_batch(unsigned count, unsigned batch_size, do_suspend suspend) {\n    auto a = 0, b = 1;\n    std::vector<int> batch;\n    for (unsigned i = 0; i < count; ++i) {\n        if (std::numeric_limits<decltype(a)>::max() - a < b) {\n            throw std::out_of_range(\n                fmt::format(\"fibonacci[{}] is greater than the largest value of int\", i));\n        }\n        if (suspend) {\n            co_await yield();\n        }\n        int next = std::exchange(a, std::exchange(b, a + b));\n        batch.push_back(next);\n        if (batch.size() == batch_size) {\n            co_yield std::exchange(batch, {});\n        }\n    }\n    if (!batch.empty()) {\n        co_yield std::move(batch);\n    }\n}\n\nseastar::future<>\nverify_fib_drained(coroutine::experimental::generator<int, int, std::vector<int>> actual_fibs, unsigned count) {\n    auto expected_fibs = sync_fibonacci_sequence(count);\n    auto expected_fib = std::begin(expected_fibs);\n\n    while (auto actual_fib = co_await actual_fibs()) {\n        BOOST_REQUIRE(expected_fib != std::end(expected_fibs));\n        BOOST_REQUIRE_EQUAL(*actual_fib, *expected_fib);\n        ++expected_fib;\n    }\n    BOOST_REQUIRE(expected_fib == std::end(expected_fibs));\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_drained_with_suspend) {\n    constexpr unsigned count = 4;\n    constexpr unsigned batch_size = 2;\n    return verify_fib_drained(async_fibonacci_sequence_batch(count, batch_size, do_suspend::yes),\n                              count);\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_drained_without_suspend) {\n    constexpr int count = 4;\n    constexpr int batch_size = 2;\n    return verify_fib_drained(async_fibonacci_sequence_batch(count, batch_size, do_suspend::no),\n                              count);\n}\n\nseastar::future<> test_batch_generator_not_drained(do_suspend suspend) {\n    auto fib = async_fibonacci_sequence_batch(42, 12, suspend);\n    auto actual_fib = co_await fib();\n    BOOST_REQUIRE(actual_fib.has_value());\n    BOOST_REQUIRE_EQUAL(*actual_fib, 0);\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_not_drained_with_suspend) {\n    return test_batch_generator_not_drained(do_suspend::yes);\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_not_drained_without_suspend) {\n    return test_batch_generator_not_drained(do_suspend::no);\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_move_away) {\n    struct move_only {\n        int value;\n        move_only(int value)\n            : value{value}\n        {}\n        move_only(const move_only&) = delete;\n        move_only& operator=(const move_only&) = delete;\n        move_only(move_only&&) noexcept = default;\n        move_only& operator=(move_only&&) noexcept = default;\n    };\n\n    using batch_type = std::vector<move_only>;\n    // Changed from move_only&& to move_only& because buffer elements are lvalues\n    using move_only_gen = coroutine::experimental::generator<move_only&, move_only, batch_type>;\n\n    constexpr int count = 4;\n    constexpr unsigned batch_size = 2;\n    auto numbers = std::invoke([]() -> move_only_gen {\n        batch_type batch;\n        for (int i = 0; i < count; i++) {\n            batch.push_back(i);\n            if (batch.size() == batch_size) {\n                co_yield std::exchange(batch, {});\n            }\n        }\n        if (!batch.empty()) {\n            co_yield std::move(batch);\n        }\n    });\n\n    int expected_n = 0;\n    while (auto n = co_await numbers()) {\n        BOOST_REQUIRE_EQUAL(n->get().value, expected_n++);\n    }\n}\n\nSEASTAR_TEST_CASE(test_batch_generator_convertible) {\n    struct convertible {\n        const std::string value;\n        convertible(std::string&& value)\n            : value{std::move(value)}\n        {}\n        explicit operator int() const {\n            return std::stoi(value);\n        }\n    };\n\n    using batch_type = std::vector<convertible>;\n    using move_only_gen = coroutine::experimental::generator<int, convertible, batch_type>;\n\n    constexpr int count = 4;\n    constexpr unsigned batch_size = 2;\n    auto numbers = std::invoke([]() -> move_only_gen {\n        batch_type batch;\n        for (int i = 0; i < count; i++) {\n            batch.push_back(fmt::to_string(i));\n            if (batch.size() == batch_size) {\n                co_yield std::exchange(batch, {});\n            }\n        }\n        if (!batch.empty()) {\n            co_yield std::move(batch);\n        }\n    });\n\n    int expected_n = 0;\n    while (auto n = co_await numbers()) {\n        BOOST_REQUIRE_EQUAL(*n, expected_n++);\n    }\n}\n\n// ADL test helper (must be at namespace scope for ADL to work)\nnamespace test_adl_ns {\n    struct adl_container {\n        using value_type = int;\n        std::vector<int> data;\n        size_t max_elements = 5;\n\n        size_t size() const { return data.size(); }\n        size_t capacity() const { return data.capacity(); }\n        void push_back(int v) { data.push_back(v); }\n        void clear() { data.clear(); }\n        int& operator[](size_t idx) { return data[idx]; }\n    };\n\n    // ADL free function - priority 2 (found via ADL)\n    bool can_push_more(const adl_container& c) {\n        return c.data.size() < c.max_elements;\n    }\n}\n\n// Test can_push_more() CPO with all three customization points\nSEASTAR_TEST_CASE(test_can_push_more_cpo) {\n    using namespace coroutine::experimental::internal::buffered;\n\n    // Test 1: Default implementation (size < capacity)\n    {\n        std::vector<int> vec;\n        vec.reserve(10);\n        BOOST_REQUIRE(can_push_more(vec));  // size=0, capacity=10\n        vec.push_back(1);\n        vec.push_back(2);\n        BOOST_REQUIRE(can_push_more(vec));  // size=2, capacity=10\n        vec.resize(10);\n        BOOST_REQUIRE(!can_push_more(vec));  // size=10, capacity=10\n    }\n\n    // Test 2: Member function customization\n    {\n        struct member_func_container {\n            using value_type [[maybe_unused]] = int;\n            std::vector<int> data;\n            size_t memory_limit = 100;\n            size_t memory_used = 0;\n\n            size_t size() const { return data.size(); }\n            size_t capacity() const { return data.capacity(); }\n            void push_back(int v) {\n                data.push_back(v);\n                memory_used += sizeof(int);\n            }\n            void clear() {\n                data.clear();\n                memory_used = 0;\n            }\n            int& operator[](size_t idx) { return data[idx]; }\n\n            // Member function customization - priority 1\n            bool can_push_more() const {\n                return memory_used < memory_limit;\n            }\n        };\n\n        member_func_container cont;\n        cont.data.reserve(1000);  // Large capacity\n        BOOST_REQUIRE(can_push_more(cont));  // memory_used=0 < memory_limit=100\n\n        // Fill with data\n        for (size_t i = 0; i < 20; ++i) {\n            cont.push_back(i);\n        }\n        BOOST_REQUIRE(cont.memory_used == 20 * sizeof(int));  // 80 bytes\n        BOOST_REQUIRE(can_push_more(cont));  // 80 < 100\n\n        cont.push_back(42);\n        cont.push_back(43);\n        cont.push_back(44);\n        BOOST_REQUIRE(cont.memory_used == 23 * sizeof(int));  // 92 bytes\n        BOOST_REQUIRE(can_push_more(cont));  // 92 < 100\n\n        cont.push_back(45);\n        cont.push_back(46);\n        BOOST_REQUIRE(cont.memory_used == 25 * sizeof(int));  // 100 bytes\n        BOOST_REQUIRE(!can_push_more(cont));  // 100 >= 100\n    }\n\n    // Test 3: ADL free function customization\n    {\n        test_adl_ns::adl_container cont;\n        cont.data.reserve(100);  // Large capacity\n        BOOST_REQUIRE(can_push_more(cont));  // size=0 < max_elements=5\n\n        cont.push_back(1);\n        cont.push_back(2);\n        cont.push_back(3);\n        BOOST_REQUIRE(can_push_more(cont));  // size=3 < max_elements=5\n\n        cont.push_back(4);\n        BOOST_REQUIRE(can_push_more(cont));  // size=4 < max_elements=5\n\n        cont.push_back(5);\n        BOOST_REQUIRE(!can_push_more(cont));  // size=5 >= max_elements=5\n    }\n\n    co_return;\n}\n"
  },
  {
    "path": "tests/unit/httpd_test.cc",
    "content": "/*\n * Copyright 2015 Cloudius Systems\n */\n\n#include <seastar/http/function_handlers.hh>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/handlers.hh>\n#include <seastar/http/matcher.hh>\n#include <seastar/http/matchrules.hh>\n#include <seastar/http/reply.hh>\n#include <seastar/http/routes.hh>\n#include <seastar/http/exception.hh>\n#include <seastar/http/transformers.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/future.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/units.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include \"loopback_socket.hh\"\n#include \"memory-data-sink.hh\"\n#include <boost/algorithm/string.hpp>\n#include <seastar/core/thread.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/http/json_path.hh>\n#include <seastar/http/response_parser.hh>\n#include <sstream>\n#include <seastar/core/shared_future.hh>\n#include <seastar/http/client.hh>\n#include <seastar/http/url.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/short_streams.hh>\n#include <seastar/util/closeable.hh>\n#include <seastar/net/tls.hh>\n\nusing namespace seastar;\nusing namespace httpd;\n\nclass handl : public httpd::handler_base {\npublic:\n    virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n        rep->done(\"html\");\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n};\n\nclass loopback_http_factory : public http::experimental::connection_factory {\n    loopback_socket_impl lsi;\npublic:\n    explicit loopback_http_factory(loopback_connection_factory& f) : lsi(f) {}\n    virtual future<connected_socket> make(abort_source* as) override {\n        return lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr()));\n    }\n};\n\n// NOTE: Remove this once `query_parameters` is removed\n#pragma GCC diagnostic push\n#pragma GCC diagnostic ignored \"-Wdeprecated-declarations\"\nstd::unordered_map<sstring, sstring>& deprecated_query_parameters(request& r) noexcept {\n    return r.query_parameters;\n}\n#pragma GCC diagnostic pop\n\nSEASTAR_TEST_CASE(test_reply)\n{\n    http::reply r;\n    r.set_content_type(\"txt\");\n    BOOST_REQUIRE_EQUAL(r._headers[\"Content-Type\"], sstring(\"text/plain\"));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_str_matcher)\n{\n\n    str_matcher m(\"/hello\");\n    parameters param;\n    BOOST_REQUIRE_EQUAL(m.match(\"/abc/hello\", 4, param), 10u);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_param_matcher)\n{\n\n    param_matcher m(\"param\");\n    parameters param;\n    BOOST_REQUIRE_EQUAL(m.match(\"/abc/hello\", 4, param), 10u);\n    BOOST_REQUIRE_EQUAL(param.path(\"param\"), \"/hello\");\n    BOOST_REQUIRE_EQUAL(param.get_decoded_param(\"param\"), \"hello\");\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_match_rule)\n{\n\n    parameters param;\n    handl* h = new handl();\n    match_rule mr(h);\n    mr.add_str(\"/hello\").add_param(\"param\");\n    httpd::handler_base* res = mr.get(\"/hello/val1\", param);\n    BOOST_REQUIRE_EQUAL(res, h);\n    BOOST_REQUIRE_EQUAL(param.get_decoded_param(\"param\"), \"val1\");\n    res = mr.get(\"/hell/val1\", param);\n    httpd::handler_base* nl = nullptr;\n    BOOST_REQUIRE_EQUAL(res, nl);\n\n    // Test that URL decoding happens correctly on the path part of\n    // the URL. Reproduces issue #725.\n    res = mr.get(\"/hello/hello%21hi\", param);\n    BOOST_REQUIRE_EQUAL(param.get_decoded_param(\"param\"), \"hello!hi\");\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_match_rule_order)\n{\n    parameters param;\n    routes route;\n\n    handl* h1 = new handl();\n    route.add(operation_type::GET, url(\"/hello\"), h1);\n\n    handl* h2 = new handl();\n    route.add(operation_type::GET, url(\"/hello\"), h2);\n\n    auto rh = route.get_handler(GET, \"/hello\", param);\n    BOOST_REQUIRE_EQUAL(rh, h1);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_put_drop_rule)\n{\n    routes rts;\n    auto h = std::make_unique<handl>();\n    parameters params;\n\n    {\n        auto reg = handler_registration(rts, *h, \"/hello\", operation_type::GET);\n        auto res = rts.get_handler(operation_type::GET, \"/hello\", params);\n        BOOST_REQUIRE_EQUAL(res, h.get());\n    }\n\n    auto res = rts.get_handler(operation_type::GET, \"/hello\", params);\n    httpd::handler_base* nl = nullptr;\n    BOOST_REQUIRE_EQUAL(res, nl);\n    return make_ready_future<>();\n}\n\n// Putting a duplicated exact rule would result\n// in a memory leak due to the fact that rules are implemented\n// as raw pointers. In order to prevent such leaks,\n// an exception is thrown if somebody tries to put\n// a duplicated rule without removing the old one first.\n// The interface demands that the callee allocates the handle,\n// so it should also expect the callee to free it before\n// overwriting.\nSEASTAR_TEST_CASE(test_duplicated_exact_rule)\n{\n    parameters param;\n    routes route;\n\n    handl* h1 = new handl;\n    route.put(operation_type::GET, \"/hello\", h1);\n\n    handl* h2 = new handl;\n    BOOST_REQUIRE_THROW(route.put(operation_type::GET, \"/hello\", h2), std::runtime_error);\n\n    delete route.drop(operation_type::GET, \"/hello\");\n    route.put(operation_type::GET, \"/hello\", h2);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_add_del_cookie)\n{\n    routes rts;\n    handl* h = new handl();\n    match_rule mr(h);\n    mr.add_str(\"/hello\");\n    parameters params;\n\n    {\n        auto reg = rule_registration(rts, mr, operation_type::GET);\n        auto res = rts.get_handler(operation_type::GET, \"/hello\", params);\n        BOOST_REQUIRE_EQUAL(res, h);\n    }\n\n    auto res = rts.get_handler(operation_type::GET, \"/hello\", params);\n    httpd::handler_base* nl = nullptr;\n    BOOST_REQUIRE_EQUAL(res, nl);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_formatter)\n{\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(true), \"true\");\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(false), \"false\");\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(1), \"1\");\n    const char* txt = \"efg\";\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(txt), \"\\\"efg\\\"\");\n    sstring str = \"abc\";\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(str), \"\\\"abc\\\"\");\n    float f = 1;\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(f), \"1\");\n    f = 1.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(f), std::out_of_range);\n    f = -1.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(f), std::out_of_range);\n    f = 0.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(f), std::invalid_argument);\n    double d = -1;\n    BOOST_REQUIRE_EQUAL(json::formatter::to_json(d), \"-1\");\n    d = 1.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(d), std::out_of_range);\n    d = -1.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(d), std::out_of_range);\n    d = 0.0/0.0;\n    BOOST_CHECK_THROW(json::formatter::to_json(d), std::invalid_argument);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_decode_url) {\n    http::request req;\n    req._url = \"/a?q=%23%24%23\";\n    sstring url = req.parse_query_param();\n    const auto& query_parameters = deprecated_query_parameters(req);\n    BOOST_REQUIRE_EQUAL(url, \"/a\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"q\"), \"#$#\");\n    BOOST_REQUIRE_EQUAL(query_parameters.at(\"q\"), \"#$#\");\n    req._url = \"/a?a=%23%24%23&b=%22%26%22\";\n    req.parse_query_param();\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"a\"), \"#$#\");\n    BOOST_REQUIRE_EQUAL(query_parameters.at(\"a\"), \"#$#\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"b\"), \"\\\"&\\\"\");\n    BOOST_REQUIRE_EQUAL(query_parameters.at(\"b\"), \"\\\"&\\\"\");\n    req._url = \"/a?b=%22%26%22&a=%21&b=%23%24%23&a=%23%24%23\";\n    req.parse_query_param();\n    const auto& b = req.get_query_param_array(\"b\");\n    auto expected_b = std::vector<sstring>{\"\\\"&\\\"\", \"#$#\"};\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(b.begin(), b.end(), expected_b.begin(), expected_b.end());\n    const auto& a = req.get_query_param_array(\"a\");\n    auto expected_a = std::vector<sstring>{\"!\", \"#$#\"};\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(a.begin(), a.end(), expected_a.begin(), expected_a.end());\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_decode_path) {\n    http::request req;\n    req.param = httpd::parameters();\n    req.param.set(\"param1\", \"/a+b\");\n    req.param.set(\"param2\", \"/same%2Ba%2Bb\");\n    req.param.set(\"param3\", \"/another_param\");\n    req.param.set(\"param4\", \"/yet%20another\");\n    req.param.set(\"invalid_param\", \"/%2\");\n\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"param1\"), \"a+b\");\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"param2\"), \"same+a+b\");\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"param3\"), \"another_param\");\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"param4\"), \"yet another\");\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"invalid_param\"), \"\");\n    BOOST_REQUIRE_EQUAL(req.get_path_param(\"missing_param\"), \"\");\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_routes) {\n    handl* h1 = new handl();\n    handl* h2 = new handl();\n    routes route;\n    route.add(operation_type::GET, url(\"/api\").remainder(\"path\"), h1);\n    route.add(operation_type::GET, url(\"/\"), h2);\n    std::unique_ptr<http::request> req = std::make_unique<http::request>();\n    std::unique_ptr<http::reply> rep = std::make_unique<http::reply>();\n\n    auto f1 =\n            route.handle(\"/api\", std::move(req), std::move(rep)).then(\n                    [] (std::unique_ptr<http::reply> rep) {\n                        BOOST_REQUIRE_EQUAL((int )rep->_status, (int )http::reply::status_type::ok);\n                    });\n    req.reset(new http::request);\n    rep.reset(new http::reply);\n\n    auto f2 =\n            route.handle(\"/\", std::move(req), std::move(rep)).then(\n                    [] (std::unique_ptr<http::reply> rep) {\n                        BOOST_REQUIRE_EQUAL((int )rep->_status, (int )http::reply::status_type::ok);\n                    });\n    req.reset(new http::request);\n    rep.reset(new http::reply);\n    auto f3 =\n            route.handle(\"/api/abc\", std::move(req), std::move(rep)).then(\n                    [] (std::unique_ptr<http::reply> rep) {\n                    });\n    req.reset(new http::request);\n    rep.reset(new http::reply);\n    auto f4 =\n            route.handle(\"/ap\", std::move(req), std::move(rep)).then(\n                    [] (std::unique_ptr<http::reply> rep) {\n                        BOOST_REQUIRE_EQUAL((int )rep->_status,\n                                            (int )http::reply::status_type::not_found);\n                    });\n    return when_all(std::move(f1), std::move(f2), std::move(f3), std::move(f4))\n            .then([] (std::tuple<future<>, future<>, future<>, future<>> fs) {\n        std::get<0>(fs).get();\n        std::get<1>(fs).get();\n        std::get<2>(fs).get();\n        std::get<3>(fs).get();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_text_route) {\n    routes route;\n    route.add(operation_type::GET, url(\"/hello\"), new function_handler([](const_req req) {\n        return \"hello, you\";\n    }, \"txt\"));\n\n    auto reply = route.handle(\"/hello\", std::make_unique<http::request>(),\n            std::make_unique<http::reply>()).get();\n\n    BOOST_CHECK_EQUAL((int )reply->_status, (int )http::reply::status_type::ok);\n    BOOST_CHECK_EQUAL(reply->_headers[\"Content-Type\"], \"text/plain\");\n    BOOST_CHECK_EQUAL(reply->_content, \"hello, you\");\n}\n\nSEASTAR_TEST_CASE(test_json_path) {\n    shared_ptr<bool> res1 = make_shared<bool>(false);\n    shared_ptr<bool> res2 = make_shared<bool>(false);\n    shared_ptr<bool> res3 = make_shared<bool>(false);\n    shared_ptr<bool> res4 = make_shared<bool>(false);\n    shared_ptr<routes> route = make_shared<routes>();\n    path_description path1(\"/my/path\",GET,\"path1\",\n        {{\"param1\", path_description::url_component_type::PARAM}\n        ,{\"/text\", path_description::url_component_type::FIXED_STRING}},{});\n    path_description path2(\"/my/path\",GET,\"path2\",\n            {{\"param1\", path_description::url_component_type::PARAM}\n            ,{\"param2\", path_description::url_component_type::PARAM}},{});\n    path_description path3(\"/my/path\",GET,\"path3\",\n            {{\"param1\", path_description::url_component_type::PARAM}\n            ,{\"param2\", path_description::url_component_type::PARAM_UNTIL_END_OF_PATH}},{});\n    path_description path4(\"/double/encoded\",GET,\"path4\",\n            {{\"param1\", path_description::url_component_type::PARAM_UNTIL_END_OF_PATH}},{});\n\n    path1.set(*route, [res1] (const_req req) {\n        (*res1) = true;\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param1\"), \"value1\");\n        return \"\";\n    });\n\n    path2.set(*route, [res2] (const_req req) {\n        (*res2) = true;\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param1\"), \"value4+value4 value4\");\n\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param2\"), \"text4+text4\");\n        return \"\";\n    });\n\n    path3.set(*route, [res3] (const_req req) {\n        (*res3) = true;\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param1\"), \"value3\");\n\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param2\"), \"text2/text3\");\n        return \"\";\n    });\n\n    path4.set(*route, [res4] (const_req req) {\n        (*res4) = true;\n        BOOST_REQUIRE_EQUAL(req.get_path_param(\"param1\"), \"example%20\");\n        return \"\";\n    });\n\n    auto check_handler = [route](auto raw_url, auto res) {\n        http::request req;\n        req._url = raw_url;\n        sstring url = req.parse_query_param();\n        return route->handle(url, std::make_unique<http::request>(), std::make_unique<http::reply>()).then([res, route] (auto f) {\n            BOOST_REQUIRE_EQUAL(*res, true);\n        });\n    };\n\n    auto f1 = check_handler(\"/my/path/value1/text\", res1);\n    auto f2 = check_handler(\"/my/path/value4+value4%20value4/text4%2Btext4\", res2);\n    auto f3 = check_handler(\"/my/path/value3/text2/text3\", res3);\n    auto f4 = check_handler(\"/double/encoded/example%2520\", res4);\n\n    return when_all(std::move(f1), std::move(f2), std::move(f3), std::move(f4))\n                .then([] (auto fs) {\n            std::get<0>(fs).get();\n            std::get<1>(fs).get();\n            std::get<2>(fs).get();\n            std::get<3>(fs).get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_match_rule_order_with_param) {\n    parameters param;\n    routes route;\n\n    handl* h1 = new handl();\n    handl* h2 = new handl();\n    handl* h3 = new handl();\n    handl* h4 = new handl();\n\n    route.add(operation_type::GET, url(\"/draw\"), h1);\n    route.add(operation_type::GET, url(\"/draw\").remainder(\"path\"), h2);\n    route.add(operation_type::GET, url(\"/ward\").remainder(\"path\"), h3);\n    route.add(operation_type::GET, url(\"/ward\"), h4);\n\n    BOOST_REQUIRE_EQUAL(route.get_handler(GET, \"/draw\", param), h1);\n    BOOST_REQUIRE_EQUAL(route.get_handler(GET, \"/ward\", param), h3); // ATTN: it's NOT h4\n\n    return make_ready_future<>();\n}\n\nfuture<> test_transformer_stream(std::stringstream& ss, content_replace& cr, std::vector<sstring>&& buffer_parts) {\n    std::unique_ptr<seastar::http::request> req = std::make_unique<seastar::http::request>();\n    ss.str(\"\");\n    req->_headers[\"Host\"] = \"localhost\";\n    output_stream_options opts;\n    opts.trim_to_size = true;\n    return do_with(output_stream<char>(cr.transform(std::move(req), \"json\", output_stream<char>(testing::memory_data_sink(ss), 32000, opts))),\n            std::vector<sstring>(std::move(buffer_parts)), [] (output_stream<char>& os, std::vector<sstring>& parts) {\n        return do_for_each(parts, [&os](auto& p) {\n            return os.write(p);\n        }).then([&os] {\n            return os.close();\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_transformer) {\n    return do_with(std::stringstream(), content_replace(\"json\"), [] (std::stringstream& ss, content_replace& cr) {\n        output_stream_options opts;\n        opts.trim_to_size = true;\n        return do_with(output_stream<char>(cr.transform(std::make_unique<seastar::http::request>(), \"html\", output_stream<char>(testing::memory_data_sink(ss), 32000, opts))),\n                [] (output_stream<char>& os) {\n            return os.write(sstring(\"hello-{{Protocol}}-xyz-{{Host}}\")).then([&os] {\n                return os.close();\n            });\n        }).then([&ss, &cr] () {\n            BOOST_REQUIRE_EQUAL(ss.str(), \"hello-{{Protocol}}-xyz-{{Host}}\");\n            return test_transformer_stream(ss, cr, {\"hell\", \"o-{\", \"{Pro\", \"tocol}}-xyz-{{Ho\", \"st}}{{Pr\"}).then([&ss, &cr] {\n                BOOST_REQUIRE_EQUAL(ss.str(), \"hello-http-xyz-localhost{{Pr\");\n                return test_transformer_stream(ss, cr, {\"hell\", \"o-{{\", \"Pro\", \"tocol}}{{Protocol}}-{{Protoxyz-{{Ho\", \"st}}{{Pr\"}).then([&ss, &cr] {\n                    BOOST_REQUIRE_EQUAL(ss.str(), \"hello-httphttp-{{Protoxyz-localhost{{Pr\");\n                    return test_transformer_stream(ss, cr, {\"hell\", \"o-{{Pro\", \"t{{Protocol}}ocol}}\", \"{{Host}}\"}).then([&ss] {\n                        BOOST_REQUIRE_EQUAL(ss.str(), \"hello-{{Prothttpocol}}localhost\");\n                    });\n                });\n            });\n        });\n    });\n}\n\nstruct http_consumer {\n    std::map<sstring, std::string> _headers;\n    std::string _body;\n    uint32_t _remain = 0;\n    std::string _current;\n    std::optional<char> last;\n    uint32_t _size = 0;\n    bool _concat = true;\n\n    enum class status_type {\n        READING_HEADERS,\n        CHUNK_SIZE,\n        CHUNK_BODY,\n        CHUNK_END,\n        READING_BODY_BY_SIZE,\n        DONE\n    };\n    status_type status = status_type::READING_HEADERS;\n\n    bool read(const temporary_buffer<char>& b) {\n        for (auto c : b) {\n            if (last && *last =='\\r' && c == '\\n') {\n                if (_current == \"\") {\n                    if (status == status_type::READING_HEADERS || (status == status_type::CHUNK_BODY && _remain == 0)) {\n                        if (status == status_type::READING_HEADERS && _headers.find(\"Content-Length\") != _headers.end()) {\n                            _remain = stoi(_headers[\"Content-Length\"], nullptr, 16);\n                            if (_remain == 0) {\n                                status = status_type::DONE;\n                                break;\n                            }\n                            status = status_type::READING_BODY_BY_SIZE;\n                        } else {\n                            status = status_type::CHUNK_SIZE;\n                        }\n                    } else if (status == status_type::CHUNK_END) {\n                        status = status_type::DONE;\n                        break;\n                    }\n                } else {\n                    switch (status) {\n                    case status_type::READING_HEADERS: add_header(_current);\n                    break;\n                    case status_type::CHUNK_SIZE: set_chunk(_current);\n                    break;\n                    default:\n                        break;\n                    }\n                    _current = \"\";\n                }\n                last = std::nullopt;\n            } else {\n                if (last) {\n                    if (status == status_type::CHUNK_BODY || status == status_type::READING_BODY_BY_SIZE) {\n                        if (_concat) {\n                            _body = _body + *last;\n                        }\n                        _size++;\n                        _remain--;\n                        if (_remain <= 1 && status == status_type::READING_BODY_BY_SIZE) {\n                            if (_concat) {\n                                _body = _body + c;\n                            }\n                            _size++;\n                            status = status_type::DONE;\n                            break;\n                        }\n                    } else {\n                        _current = _current + *last;\n                    }\n\n                }\n                last = c;\n            }\n        }\n        return status == status_type::DONE;\n    }\n\n    void set_chunk(const std::string& s) {\n        _remain = stoi(s, nullptr, 16);\n        if (_remain == 0) {\n            status = status_type::CHUNK_END;\n        } else {\n            status = status_type::CHUNK_BODY;\n        }\n    }\n\n    void add_header(const std::string& s) {\n        std::vector<std::string> strs;\n        boost::split(strs, s, boost::is_any_of(\":\"));\n        if (strs.size() > 1) {\n            _headers[strs[0]] = strs[1];\n        }\n    }\n};\n\nclass test_client_server {\npublic:\n    static future<> write_request(output_stream<char>& output) {\n        return output.write(sstring(\"GET /test HTTP/1.1\\r\\nHost: myhost.org\\r\\n\\r\\n\")).then([&output]{\n                return output.flush();\n        });\n    }\n\n    static future<> run_test(json::json_return_type::body_writer_type&& write_func, std::function<bool(size_t, http_consumer&)> reader) {\n        return do_with(loopback_connection_factory(1), foreign_ptr<shared_ptr<http_server>>(make_shared<http_server>(\"test\")),\n                [reader, &write_func] (loopback_connection_factory& lcf, auto& server) {\n            return do_with(loopback_socket_impl(lcf), [&server, &lcf, reader, &write_func](loopback_socket_impl& lsi) {\n                httpd::http_server_tester::listeners(*server).emplace_back(lcf.get_server_socket());\n\n                auto client = seastar::async([&lsi, reader] {\n                    connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n                    input_stream<char> input(c_socket.input());\n                    output_stream<char> output(c_socket.output());\n                    bool more = true;\n                    size_t count = 0;\n                    while (more) {\n                        http_consumer htp;\n                        htp._concat = false;\n\n                        write_request(output).get();\n                        repeat([&input, &htp] {\n                            return input.read().then([&htp](const temporary_buffer<char>& b) mutable {\n                                return (b.size() == 0 || htp.read(b)) ? make_ready_future<stop_iteration>(stop_iteration::yes) :\n                                        make_ready_future<stop_iteration>(stop_iteration::no);\n                            });\n                        }).get();\n                        std::cout << htp._body << std::endl;\n                        more = reader(count, htp);\n                        count++;\n                    }\n                    if (input.eof()) {\n                        input.close().get();\n                    }\n                });\n\n                auto server_setup = seastar::async([&server, &write_func] {\n                    class test_handler : public handler_base {\n                        size_t count = 0;\n                        http_server& _server;\n                        json::json_return_type::body_writer_type _write_func;\n                        promise<> _all_message_sent;\n                    public:\n                        test_handler(http_server& server, json::json_return_type::body_writer_type&& write_func) : _server(server), _write_func(std::move(write_func)) {\n                        }\n                        future<std::unique_ptr<http::reply>> handle(const sstring& path,\n                                std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n                            rep->write_body(\"json\", std::move(_write_func));\n                            count++;\n                            _all_message_sent.set_value();\n                            return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                        }\n                        future<> wait_for_message() {\n                            return _all_message_sent.get_future();\n                        }\n                    };\n                    auto handler = new test_handler(*server, std::move(write_func));\n                    server->_routes.put(GET, \"/test\", handler);\n                    when_all(server->do_accepts(0), handler->wait_for_message()).get();\n                });\n                return when_all(std::move(client), std::move(server_setup));\n            }).discard_result().then_wrapped([&server] (auto f) {\n                f.ignore_ready_future();\n                return server->stop();\n            });\n        });\n    }\n    static future<> run(std::vector<std::tuple<bool, size_t>> tests) {\n        return do_with(loopback_connection_factory(1), foreign_ptr<shared_ptr<http_server>>(make_shared<http_server>(\"test\")),\n                [tests] (loopback_connection_factory& lcf, auto& server) {\n            return do_with(loopback_socket_impl(lcf), [&server, &lcf, tests](loopback_socket_impl& lsi) {\n                httpd::http_server_tester::listeners(*server).emplace_back(lcf.get_server_socket());\n\n                auto client = seastar::async([&lsi, tests] {\n                    connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n                    input_stream<char> input(c_socket.input());\n                    output_stream<char> output(c_socket.output());\n                    bool more = true;\n                    size_t count = 0;\n                    while (more) {\n                        http_consumer htp;\n                        write_request(output).get();\n                        repeat([&input, &htp] {\n                            return input.read().then([&htp](const temporary_buffer<char>& b) mutable {\n                                return (b.size() == 0 || htp.read(b)) ? make_ready_future<stop_iteration>(stop_iteration::yes) :\n                                        make_ready_future<stop_iteration>(stop_iteration::no);\n                            });\n                        }).get();\n                        if (std::get<bool>(tests[count])) {\n                            BOOST_REQUIRE_EQUAL(htp._body.length(), std::get<size_t>(tests[count]));\n                        } else {\n                            BOOST_REQUIRE_EQUAL(input.eof(), true);\n                            more = false;\n                        }\n                        count++;\n                        if (count == tests.size()) {\n                            more = false;\n                        }\n                    }\n                    if (input.eof()) {\n                        input.close().get();\n                    }\n                });\n\n                auto server_setup = seastar::async([&server, tests] {\n                    class test_handler : public handler_base {\n                        size_t count = 0;\n                        http_server& _server;\n                        std::vector<std::tuple<bool, size_t>> _tests;\n                        promise<> _all_message_sent;\n                    public:\n                        test_handler(http_server& server, const std::vector<std::tuple<bool, size_t>>& tests) : _server(server), _tests(tests) {\n                        }\n                        future<std::unique_ptr<http::reply>> handle(const sstring& path,\n                                std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n                            rep->write_body(\"txt\", make_writer(std::get<size_t>(_tests[count]), std::get<bool>(_tests[count])));\n                            count++;\n                            if (count == _tests.size()) {\n                                _all_message_sent.set_value();\n                            }\n                            return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n                        }\n                        future<> wait_for_message() {\n                            return _all_message_sent.get_future();\n                        }\n                    };\n                    auto handler = new test_handler(*server, tests);\n                    server->_routes.put(GET, \"/test\", handler);\n                    when_all(server->do_accepts(0), handler->wait_for_message()).get();\n                });\n                return when_all(std::move(client), std::move(server_setup));\n            }).discard_result().then_wrapped([&server] (auto f) {\n                f.ignore_ready_future();\n                return server->stop();\n            });\n        });\n    }\n\n    static noncopyable_function<future<>(output_stream<char>&& o_stream)> make_writer(size_t len, bool success) {\n        return [len, success] (output_stream<char>&& o_stream) mutable {\n            return do_with(output_stream<char>(std::move(o_stream)), uint32_t(len/10), [success](output_stream<char>& str, uint32_t& remain) {\n                if (remain == 0) {\n                    if (success) {\n                        return str.close();\n                    } else {\n                        throw std::runtime_error(\"Throwing exception before writing\");\n                    }\n                }\n                return repeat([&str, &remain] () mutable {\n                    return str.write(\"1234567890\").then([&remain]() mutable {\n                        remain--;\n                        return (remain == 0)? make_ready_future<stop_iteration>(stop_iteration::yes) : make_ready_future<stop_iteration>(stop_iteration::no);\n                    });\n                }).then([&str, success] {\n                    if (!success) {\n                        return str.flush();\n                    }\n                    return make_ready_future<>();\n                }).then([&str, success] {\n                    if (success) {\n                        return str.close();\n                    } else {\n                        throw std::runtime_error(\"Throwing exception after writing\");\n                    }\n                });\n            });\n        };\n    }\n};\n\nSEASTAR_TEST_CASE(test_message_with_error_non_empty_body) {\n    std::vector<std::tuple<bool, size_t>> tests = {\n        std::make_tuple(true, 100),\n        std::make_tuple(false, 10000)};\n    return test_client_server::run(tests);\n}\n\nSEASTAR_TEST_CASE(test_simple_chunked) {\n    std::vector<std::tuple<bool, size_t>> tests = {\n        std::make_tuple(true, 100000),\n        std::make_tuple(true, 100)};\n    return test_client_server::run(tests);\n}\n\nSEASTAR_TEST_CASE(test_http_client_server_full) {\n    std::vector<std::tuple<bool, size_t>> tests = {\n        std::make_tuple(true, 100),\n        std::make_tuple(true, 10000),\n        std::make_tuple(true, 100),\n        std::make_tuple(true, 0),\n        std::make_tuple(true, 5000),\n        std::make_tuple(true, 10000),\n        std::make_tuple(true, 9000),\n        std::make_tuple(true, 10000)};\n    return test_client_server::run(tests);\n}\n\n/*\n * return string in the given size\n * The string size takes the quotes into consideration.\n */\nstd::string get_value(int size) {\n    std::stringstream res;\n    for (auto i = 0; i < size - 2; i++) {\n        res << \"a\";\n    }\n    return res.str();\n}\n\n/*\n * A helper object that map to a big json string\n * in the format of:\n * {\"valu\":\"aaa....aa\",\"valu\":\"aaa....aa\",\"valu\":\"aaa....aa\"...}\n *\n * The object can have an arbitrary size in multiplication of 10000 bytes\n *  */\nstruct extra_big_object : public json::json_base {\n    json::json_element<sstring>* value;\n    extra_big_object(size_t size) {\n        value = new json::json_element<sstring>;\n        // size = brackets + (\"name\" + \":\" + get_value()) * n + \",\" * (n-1)\n        // size = 2 + (valu + 4 + get_value) * n - 1\n        value->_name = \"valu\";\n        *value = get_value(9992);\n        for (size_t i = 0; i < size/10000; i++) {\n            _elements.emplace_back(value);\n        }\n    }\n\n    virtual ~extra_big_object() {\n        delete value;\n    }\n\n    extra_big_object(const extra_big_object& o) {\n        value = new json::json_element<sstring>;\n        value->_name = o.value->_name;\n        *value = (*o.value)();\n        for (size_t i = 0; i < o._elements.size(); i++) {\n            _elements.emplace_back(value);\n        }\n    }\n};\n\nSEASTAR_TEST_CASE(json_stream) {\n    std::vector<extra_big_object> vec;\n    size_t num_objects = 1000;\n    // each object is 1000001 bytes plus a comma for all but the last and plus\n    // two for the [] backets\n    size_t total_size = num_objects * 1000002 + 1;\n    for (size_t i = 0; i < num_objects; i++) {\n        vec.emplace_back(1000000);\n    }\n    return test_client_server::run_test(json::stream_object(vec), [total_size](size_t s, http_consumer& h) {\n        BOOST_REQUIRE_EQUAL(h._size, total_size);\n        return false;\n    });\n}\n\n// See issue https://github.com/scylladb/seastar/issues/1701\nSEASTAR_TEST_CASE(dont_abort) {\n    return test_client_server::run_test(\n            [](seastar::output_stream<char>&& stream_) -> future<> {\n        auto stream = std::move(stream_);\n        co_await stream.write(seastar::temporary_buffer<char>{3});\n        co_await stream.close();\n    }\n    , [](size_t s, http_consumer& h) {\n        BOOST_REQUIRE_EQUAL(h._size, 3);\n        return false;\n    });\n\n}\n\n\nclass json_test_handler : public handler_base {\n    http::body_writer_type _write_func;\npublic:\n    json_test_handler(http::body_writer_type&& write_func) : _write_func(std::move(write_func)) {\n    }\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        rep->write_body(\"json\", http::body_writer_type(std::ref(_write_func)));\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n};\n\nSEASTAR_TEST_CASE(content_length_limit) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        server.set_content_length_limit(11);\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        future<> client = seastar::async([&lsi] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            output_stream<char> output(c_socket.output());\n\n            output.write(sstring(\"GET /test HTTP/1.1\\r\\nHost: test\\r\\n\\r\\n\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"200 OK\"), std::string::npos);\n\n            output.write(sstring(\"GET /test HTTP/1.1\\r\\nHost: test\\r\\nContent-Length: 11\\r\\n\\r\\nxxxxxxxxxxx\")).get();\n            output.flush().get();\n            resp = input.read().get();\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"200 OK\"), std::string::npos);\n\n            output.write(sstring(\"GET /test HTTP/1.1\\r\\nHost: test\\r\\nContent-Length: 17\\r\\n\\r\\nxxxxxxxxxxxxxxxx\")).get();\n            output.flush().get();\n            resp = input.read().get();\n            BOOST_REQUIRE_EQUAL(std::string(resp.get(), resp.size()).find(\"200 OK\"), std::string::npos);\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"413 Payload Too Large\"), std::string::npos);\n\n            input.close().get();\n            output.close().get();\n        });\n\n        auto handler = new json_test_handler(json::stream_object(\"hello\"));\n        server._routes.put(GET, \"/test\", handler);\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_unexpected_reply_status) {\n    return seastar::async([] {\n        class handl : public httpd::handler_base {\n        public:\n            virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n                    std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                fmt::print(\"Returning exception from server\\n\");\n                return make_exception_future<std::unique_ptr<http::reply>>(httpd::server_error_exception(\"error\"));\n            }\n        };\n\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n            {\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                BOOST_REQUIRE_THROW(cln.make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                    BOOST_REQUIRE(false); // should throw before handling response\n                    return make_ready_future<>();\n                }, http::reply::status_type::ok).get(), httpd::unexpected_status_error);\n            }\n            {\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                std::optional<http::reply::status_type> status;\n                cln.make_request(std::move(req), [&status] (const http::reply& rep, input_stream<char>&& in) {\n                    status = rep._status;\n                    return make_ready_future<>();\n                }).get();\n                BOOST_REQUIRE(status.has_value());\n                BOOST_REQUIRE_EQUAL(status.value(), http::reply::status_type::internal_server_error);\n            }\n\n            cln.close().get();\n        });\n\n        server._routes.put(GET, \"/test\", new handl());\n        server.do_accepts(0).get();\n        client.get();\n        server.stop().get();\n    });\n}\n\nstatic void read_simple_http_request(input_stream<char>& in) {\n    sstring req;\n    while (true) {\n        auto r = in.read().get();\n        req += sstring(r.get(), r.size());\n        if (req.ends_with(\"\\r\\n\\r\\n\")) {\n            break;\n        }\n    }\n}\n\nSEASTAR_TEST_CASE(test_client_response_eof) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                out.write(\"HTT\").get(); // write incomplete response\n                out.flush().get();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            BOOST_REQUIRE_EXCEPTION(cln.make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok).get(), std::system_error, [] (auto& ex) {\n                return ex.code().value() == ECONNABORTED;\n            });\n\n            cln.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_head_empty_body) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                out.write(format(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\nContent-Length: {}\\r\\n\\r\\n\", 128)).get();\n                out.flush().get();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n            auto req = http::request::make(\"HEAD\", \"test\", \"/test\");\n            cln.make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                return seastar::async([&rep, in = std::move(in)] () mutable {\n                    BOOST_REQUIRE_EQUAL(rep._status, http::reply::status_type::ok);\n                    BOOST_REQUIRE_EQUAL(rep.content_length, 128);\n                    auto buf = in.read().get();\n                    BOOST_REQUIRE(buf.empty());\n                    in.close().get();\n                });\n            }).get();\n\n            cln.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_retry_nested) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                    read_simple_http_request(in);\n                    output_stream<char> out = sk.output();\n                    sstring r200(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\n\\r\\n\");\n                    out.write(r200).get(); // now write complete response\n                    out.flush().get();\n                    out.close().get();\n            });\n        }).then([&ss] {\n            return ss.accept().then([] (accept_result ar) {\n                return seastar::async([sk = std::move(ar.connection)] () mutable {\n                    input_stream<char> in = sk.input();\n                    read_simple_http_request(in);\n                    output_stream<char> out = sk.output();\n                    sstring r200(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\n\\r\\n\");\n                    out.write(r200).get(); // now write complete response\n                    out.flush().get();\n                    out.close().get();\n                });\n            });\n        });\n\n        future<> client_ex = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 2, http::experimental::client::retry_requests::yes);\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            size_t count = 0;\n            BOOST_REQUIRE_EXCEPTION(cln.make_request(\n                                           std::move(req),\n                                           [&count](const http::reply&, input_stream<char>&&) {\n                                               ++count;\n                                               try {\n                                                   try {\n                                                       try {\n                                                           throw std::system_error(EPIPE, std::system_category());\n                                                       } catch (...) {\n                                                           std::throw_with_nested(std::runtime_error(\"Some exception\"));\n                                                       }\n                                                   } catch (...) {\n                                                       std::throw_with_nested(std::system_error(ENOBUFS, std::system_category()));\n                                                   }\n                                               } catch (...) {\n                                                   return make_exception_future(std::current_exception());\n                                               }\n                                           },\n                                           http::reply::status_type::ok)\n                                        .get(),\n                                    std::system_error,\n                                    [](auto& ex) { return ex.code().value() == ENOBUFS; });\n\n            cln.close().get();\n            BOOST_REQUIRE_EQUAL(count, 2);\n        });\n\n        when_all(std::move(client_ex), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_response_parse_error) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                out.write(\"HTTTT\").get(); // write invalid line\n                out.flush().get();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            BOOST_REQUIRE_EXCEPTION(cln.make_request(std::move(req), [] (const http::reply& rep, input_stream<char>&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok).get(), httpd::response_parsing_exception, [] (auto& ex) {\n                return sstring(ex.what()).contains(\"Invalid http server response. Reason: Parsing error at offset 3: encountered \\\"TT\\\".\");\n            });\n\n            cln.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_abort_new_conn) {\n    class delayed_factory : public http::experimental::connection_factory {\n    public:\n        virtual future<connected_socket> make(abort_source* as) override {\n            SEASTAR_ASSERT(as != nullptr);\n            return sleep_abortable(std::chrono::seconds(1), *as).then([] {\n                return make_exception_future<connected_socket>(std::runtime_error(\"Shouldn't happen\"));\n            });\n        }\n    };\n\n    return seastar::async([] {\n        auto cln = http::experimental::client(std::make_unique<delayed_factory>());\n        abort_source as;\n        auto f = cln.make_request(http::request::make(\"GET\", \"test\", \"/test\"), [] (const auto& rep, auto&& in) {\n            return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n        }, http::reply::status_type::ok, &as);\n\n        as.request_abort();\n        BOOST_REQUIRE_THROW(f.get(), abort_requested_exception);\n        cln.close().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_abort_cached_conn) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        promise<> server_paused;\n        promise<> server_resume;\n        future<> server = ss.accept().then([&] (accept_result ar) {\n            return seastar::async([&server_paused, &server_resume, sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                server_paused.set_value();\n                server_resume.get_future().get();\n                output_stream<char> out = sk.output();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 1 /* max connections */);\n            // this request gets handled by server and ...\n            auto f1 = cln.make_request(http::request::make(\"GET\", \"test\", \"/test\"), [] (const auto& rep, auto&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok);\n            server_paused.get_future().get();\n            // ... this should hang waiting for cached connection\n            abort_source as;\n            auto f2 = cln.make_request(http::request::make(\"GET\", \"test\", \"/test\"), [] (const auto& rep, auto&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok, &as);\n\n            as.request_abort();\n            BOOST_REQUIRE_THROW(f2.get(), abort_requested_exception);\n            server_resume.set_value();\n            cln.close().get();\n            try {\n                f1.get();\n            } catch (...) {\n            }\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_abort_send_request) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([&] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 1 /* max connections */);\n            abort_source as;\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            promise<> client_paused;\n            promise<> client_resume;\n            req.write_body(\"txt\", [&] (output_stream<char>&& out) {\n                return seastar::async([&client_paused, &client_resume, out = std::move(out)] () mutable {\n                    auto cl = deferred_close(out);\n                    client_paused.set_value();\n                    client_resume.get_future().get();\n                    out.write(\"foo\").get();\n                    out.flush().get();\n                });\n            });\n            auto f = cln.make_request(std::move(req), [] (const auto& rep, auto&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok, &as);\n            client_paused.get_future().get();\n            as.request_abort();\n            client_resume.set_value();\n            BOOST_REQUIRE_THROW(f.get(), abort_requested_exception);\n            cln.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_abort_recv_response) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        promise<> server_paused;\n        promise<> server_resume;\n        future<> server = ss.accept().then([&] (accept_result ar) {\n            return seastar::async([&server_paused, &server_resume, sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                server_paused.set_value();\n                server_resume.get_future().get();\n                output_stream<char> out = sk.output();\n                out.close().get();\n            });\n        });\n\n        future<> client = seastar::async([&] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 1 /* max connections */);\n            abort_source as;\n            auto f = cln.make_request(http::request::make(\"GET\", \"test\", \"/test\"), [] (const auto& rep, auto&& in) {\n                return make_exception_future<>(std::runtime_error(\"Shouldn't happen\"));\n            }, http::reply::status_type::ok, &as);\n            server_paused.get_future().get();\n            as.request_abort();\n            BOOST_REQUIRE_THROW(f.get(), abort_requested_exception);\n            server_resume.set_value();\n            cln.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_retry_request) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        auto ss = lcf.get_server_socket();\n        future<> server = ss.accept().then([] (accept_result ar) {\n            return seastar::async([sk = std::move(ar.connection)] () mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                out.write(\"HTTTT\").get(); // write incomplete response\n                out.flush().get();\n                out.close().get();\n            });\n        }).then([&ss] {\n            return ss.accept().then([] (accept_result ar) {\n                return seastar::async([sk = std::move(ar.connection)] () mutable {\n                    input_stream<char> in = sk.input();\n                    read_simple_http_request(in);\n                    output_stream<char> out = sk.output();\n                    sstring r200(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\n\\r\\n\");\n                    out.write(r200).get(); // now write complete response\n                    out.flush().get();\n                    out.close().get();\n                });\n            });\n        });\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 2, http::experimental::client::retry_requests::yes);\n            auto req = http::request::make(\"GET\", \"test\", \"/test\");\n            bool got_response = false;\n            cln.make_request(std::move(req), [&] (const http::reply& rep, input_stream<char>&& in) {\n                got_response = true;\n                return make_ready_future<>();\n            }, http::reply::status_type::ok).get();\n            cln.close().get();\n            BOOST_REQUIRE(got_response);\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_100_continue) {\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        server.set_content_length_limit(11);\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n        future<> client = seastar::async([&lsi] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            output_stream<char> output(c_socket.output());\n\n            for (auto version : {sstring(\"1.0\"), sstring(\"1.1\")}) {\n                for (auto content : {sstring(\"\"), sstring(\"xxxxxxxxxxx\")}) {\n                    for (auto expect : {sstring(\"\"), sstring(\"Expect: 100-continue\\r\\n\"), sstring(\"Expect: 100-cOnTInUE\\r\\n\")}) {\n                        bool need_cont = (version == \"1.1\" && expect.length());\n                        bool body_sent = false;\n                        auto content_len = content.empty() ? sstring(\"\") : (sstring(\"Content-Length: \") + to_sstring(content.length()) + sstring(\"\\r\\n\"));\n                        sstring req = sstring(\"GET /test HTTP/\") + version + sstring(\"\\r\\nHost: test\\r\\nConnection: Keep-Alive\\r\\n\") + content_len + expect + sstring(\"\\r\\n\");\n                        sstring resp;\n\n                        output.write(req).get();\n                        output.flush().get();\n\n                        if (!need_cont) {\n                            output.write(content).get();\n                            output.flush().get();\n                            body_sent = true;\n                        }\n\n                        while (true) {\n                            auto r = input.read().get();\n                            BOOST_REQUIRE(!r.empty());\n                            resp += sstring(r.get(), r.size());\n\n                            if (need_cont && !body_sent) {\n                                if (resp.find(\"100 Continue\") != std::string::npos) {\n                                    output.write(content).get();\n                                    output.flush().get();\n                                    body_sent = true;\n                                }\n                            }\n\n                            if (resp.ends_with(\"\\\"hello\\\"\\r\\n0\\r\\n\\r\\n\")) {\n                                break;\n                            }\n                        }\n\n                        if (need_cont) {\n                            BOOST_REQUIRE(body_sent);\n                        }\n                        BOOST_REQUIRE_NE(resp.find(\"200 OK\"), std::string::npos);\n                    }\n                }\n            }\n            output.write(sstring(\"GET /test HTTP/1.1\\r\\nHost: test\\r\\nContent-Length: 17\\r\\nExpect: 100-continue\\r\\n\\r\\n\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            BOOST_REQUIRE_EQUAL(std::string(resp.get(), resp.size()).find(\"100 Continue\"), std::string::npos);\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"413 Payload Too Large\"), std::string::npos);\n\n            input.close().get();\n            output.close().get();\n        });\n\n        auto handler = new json_test_handler(json::stream_object(\"hello\"));\n        server._routes.put(GET, \"/test\", handler);\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\n\nSEASTAR_TEST_CASE(test_unparsable_request) {\n    // Test if a message that cannot be parsed as a http request is being replied with a 400 Bad Request response\n    return seastar::async([] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n        future<> client = seastar::async([&lsi] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            output_stream<char> output(c_socket.output());\n\n            output.write(sstring(\"GET /test HTTP/1.1\\r\\nhello\\r\\nContent-Length: 17\\r\\nExpect: 100-continue\\r\\n\\r\\n\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"400 Bad Request\"), std::string::npos);\n            BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(\"Can't parse the request\"), std::string::npos);\n\n            input.close().get();\n            output.close().get();\n        });\n\n        auto handler = new json_test_handler(json::stream_object(\"hello\"));\n        server._routes.put(GET, \"/test\", handler);\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\nstruct echo_handler : public handler_base {\n    bool chunked_reply;\n    echo_handler(bool chunked_reply_) : handler_base(), chunked_reply(chunked_reply_) {}\n\n    future<std::unique_ptr<http::reply>> do_handle(std::unique_ptr<http::request>& req, std::unique_ptr<http::reply>& rep, sstring& content) {\n        for (auto it : req->chunk_extensions) {\n            content += it.first;\n            if (it.second != \"\") {\n                content += to_sstring(\"=\") + it.second;\n            }\n        }\n        for (auto it : req->trailing_headers) {\n            content += it.first;\n            if (it.second != \"\") {\n                content += to_sstring(\": \") + it.second;\n            }\n        }\n        if (!chunked_reply) {\n            rep->write_body(\"txt\", content);\n        } else {\n            rep->write_body(\"txt\", [ c = content ] (output_stream<char>& out) -> future<> {\n                co_await out.write(std::move(c));\n            });\n        }\n        return make_ready_future<std::unique_ptr<http::reply>>(std::move(rep));\n    }\n};\n\n/*\n * A request handler that responds with the same body that was used in the request using the requests content_stream\n *  */\nstruct echo_stream_handler : public echo_handler {\n    echo_stream_handler(bool chunked_reply = false) : echo_handler(chunked_reply) {}\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        return do_with(std::move(req), std::move(rep), sstring(), [this] (std::unique_ptr<http::request>& req, std::unique_ptr<http::reply>& rep, sstring& rep_content) {\n            return do_until([&req] { return req->content_stream->eof(); }, [&req, &rep_content] {\n                return req->content_stream->read().then([&rep_content] (temporary_buffer<char> tmp) {\n                    rep_content += to_sstring(std::move(tmp));\n                });\n            }).then([&req, &rep, &rep_content, this] {\n                return this->do_handle(req, rep, rep_content);\n            });\n        });\n    }\n};\n\n/*\n * Same handler as above, but without using streams\n * String content is deprecated, but keep testing it until removed\n *  */\nstruct echo_string_handler : public echo_handler {\n    echo_string_handler(bool chunked_reply = false) : echo_handler(chunked_reply) {}\n    future<std::unique_ptr<http::reply>> handle(const sstring& path,\n            std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) override {\n        return this->do_handle(req, rep, http::internal::deprecated_content(*req));\n    }\n};\n\n/*\n * Checks if the server responds to the request equivalent to the concatenation of all req_parts with a reply containing\n * the resp_parts strings, assuming that the content streaming is set to stream and the /test route is handled by handl\n * */\nfuture<> check_http_reply (std::vector<sstring>&& req_parts, std::vector<std::string>&& resp_parts, bool stream, handler_base* handl) {\n    return seastar::async([req_parts = std::move(req_parts), resp_parts = std::move(resp_parts), stream, handl] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        server.set_content_streaming(stream);\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n        future<> client = seastar::async([req_parts = std::move(req_parts), resp_parts = std::move(resp_parts), &lsi] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            output_stream<char> output(c_socket.output());\n\n            for (auto& str : req_parts) {\n                output.write(std::move(str)).get();\n                output.flush().get();\n            }\n            auto resp = input.read().get();\n            for (auto& str : resp_parts) {\n                BOOST_REQUIRE_NE(std::string(resp.get(), resp.size()).find(std::move(str)), std::string::npos);\n            }\n\n            input.close().get();\n            output.close().get();\n        });\n\n        server._routes.put(GET, \"/test\", handl);\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n};\n\nfuture<> head_handler_no_body(bool chunked) {\n    return seastar::async([chunked] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        future<> client = seastar::async([&lsi, chunked] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            output_stream<char> output(c_socket.output());\n\n            output.write(sstring(\"HEAD /test HTTP/1.1\\r\\nHost: test\\r\\nContent-Length: 1\\r\\n\\r\\nA\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            auto resp_s = std::string(resp.get(), resp.size());\n            fmt::print(\"resp:[{}]\", resp_s);\n            BOOST_REQUIRE_NE(resp_s.find(\"200 OK\"), std::string::npos);\n\n            // RFC7231 section 4.3.2\n            // The HEAD method is identical to GET except that the server MUST NOT\n            // send a message body in the response (i.e., the response terminates at\n            // the end of the header section).\n            //\n            // The server SHOULD send the same header fields in response to a HEAD\n            // request as it would have sent if the request had been a GET, except\n            // that the payload header fields MAY be omitted\n\n            // Seastar HTTPD doesn't omit header fields ..\n            if (chunked) {\n                BOOST_REQUIRE_NE(resp_s.find(\"Transfer-Encoding: chunked\\r\\n\"), std::string::npos);\n            } else {\n                BOOST_REQUIRE_NE(resp_s.find(\"Content-Length: 1\\r\\n\"), std::string::npos);\n            }\n\n            // ... but does omit the body itself\n            BOOST_REQUIRE_EQUAL(resp_s.find(\"\\r\\n\\r\\n\"), resp_s.size() - 4);\n\n            input.close().get();\n            output.close().get();\n        });\n\n        server._routes.put(HEAD, \"/test\", new echo_string_handler(chunked));\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\nSEASTAR_TEST_CASE(head_handler_no_body_content_length) {\n    return head_handler_no_body(false);\n}\n\nSEASTAR_TEST_CASE(head_handler_no_body_chunked) {\n    return head_handler_no_body(true);\n}\n\nstatic future<> test_basic_content(bool streamed, bool chunked_reply) {\n    return seastar::async([streamed, chunked_reply] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        if (streamed) {\n            server.set_content_streaming(true);\n        }\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n        future<> client = seastar::async([&lcf, chunked_reply] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n\n            {\n                fmt::print(\"Simple request test\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    if (chunked_reply) {\n                        // need to drop empty chunk\n                        return seastar::async([in = std::move(in)] () mutable {\n                            util::skip_entire_stream(in).get();\n                        });\n                    }\n                    return make_ready_future<>();\n                }, http::reply::status_type::ok).get();\n                // in fact, this case is to make sure that _next_ cases won't collect\n                // garbage from the client connection\n            }\n\n            {\n                fmt::print(\"Request with body test\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", sstring(\"12345 78901\\t34521345\"));\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    BOOST_REQUIRE_EQUAL(resp._status, http::reply::status_type::ok);\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, 20);\n                    }\n                    return seastar::async([in = std::move(in)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, sstring(\"12345 78901\\t34521345\"));\n                    });\n                }).get();\n            }\n\n            {\n                fmt::print(\"Request with content-length body\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", 12, [] (output_stream<char>& out) -> future<> {\n                    co_await out.write(sstring(\"1234567890\"));\n                    co_await out.write(sstring(\"AB\"));\n                });\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, 12);\n                    }\n                    return seastar::async([in = std::move(in)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, sstring(\"1234567890AB\"));\n                    });\n                }, http::reply::status_type::ok).get();\n            }\n\n            {\n                const size_t size = 128*1024;\n                fmt::print(\"Request with {}-kbytes content-length body\\n\", size >> 10);\n                temporary_buffer<char> jumbo(size);\n                temporary_buffer<char> jumbo_copy(size);\n                for (size_t i = 0; i < size; i++) {\n                    jumbo.get_write()[i] = 'a' + i % ('z' - 'a');\n                    jumbo_copy.get_write()[i] = jumbo[i];\n                }\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", size, [jumbo = std::move(jumbo)] (output_stream<char>& out) -> future<> {\n                    co_await out.write(jumbo.get(), jumbo.size());\n                });\n                cln.make_request(std::move(req), [chunked_reply, size, jumbo_copy = std::move(jumbo_copy)] (const http::reply& resp, input_stream<char>&& in) mutable {\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, size);\n                    }\n                    return seastar::async([in = std::move(in), jumbo_copy = std::move(jumbo_copy)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, to_sstring(std::move(jumbo_copy)));\n                    });\n                }, http::reply::status_type::ok).get();\n            }\n\n            {\n                fmt::print(\"Request whose content-length body is written via a temporary_buffer\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", 5, [] (output_stream<char>& out) -> future<> {\n                    const char* msg = \"hello\";\n                    co_await out.write(seastar::temporary_buffer<char>{msg, 5});\n                });\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    BOOST_REQUIRE_EQUAL(resp._status, http::reply::status_type::ok);\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, 5);\n                    }\n                    return seastar::async([in = std::move(in)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, sstring(\"hello\"));\n                    });\n                }).get();\n            }\n\n            {\n                fmt::print(\"Request with chunked body\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", [] (output_stream<char>& out) -> future<> {\n                    co_await out.write(sstring(\"req\"));\n                    co_await out.write(sstring(\"1234\\r\\n7890\"));\n                });\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    BOOST_REQUIRE_EQUAL(resp._status, http::reply::status_type::ok);\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, 13);\n                    }\n                    return seastar::async([in = std::move(in)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, sstring(\"req1234\\r\\n7890\"));\n                    });\n                }).get();\n            }\n\n            {\n                fmt::print(\"Request with expect-continue\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", sstring(\"foobar\"));\n                req.set_expects_continue();\n                cln.make_request(std::move(req), [&] (const http::reply& resp, input_stream<char>&& in) {\n                    BOOST_REQUIRE_EQUAL(resp._status, http::reply::status_type::ok);\n                    if (!chunked_reply) {\n                        BOOST_REQUIRE_EQUAL(resp.content_length, 6);\n                    }\n                    return seastar::async([in = std::move(in)] () mutable {\n                        sstring body = util::read_entire_stream_contiguous(in).get();\n                        BOOST_REQUIRE_EQUAL(body, sstring(\"foobar\"));\n                    });\n                }).get();\n            }\n\n            {\n                fmt::print(\"Request with incomplete content-length body\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", 12, [] (output_stream<char>& out) -> future<> {\n                    co_await out.write(sstring(\"1234567890A\"));\n                });\n                BOOST_REQUIRE_THROW(cln.make_request(std::move(req), [] (const auto& resp, auto&& in) {\n                    BOOST_REQUIRE(false); // should throw before handling response\n                    return make_ready_future<>();\n                }).get(), std::runtime_error);\n            }\n\n            {\n                bool callback_completed = false;\n                fmt::print(\"Request with too large content-length body\\n\");\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                req.write_body(\"txt\", 12, [&callback_completed] (output_stream<char>&& out) {\n                    return seastar::async([out = std::move(out), &callback_completed] () mutable {\n                        out.write(sstring(\"1234567890ABC\")).get();\n                        out.flush().get();\n                        out.close().get();\n                        callback_completed = true;\n                    });\n                });\n                BOOST_REQUIRE_NE(callback_completed, true); // should throw early\n                BOOST_REQUIRE_THROW(cln.make_request(std::move(req), [] (const auto& resp, auto&& in) {\n                    BOOST_REQUIRE(false); // should throw before handling response\n                    return make_ready_future<>();\n                }).get(), std::runtime_error);\n            }\n\n            cln.close().get();\n        });\n\n        handler_base* handler;\n        if (streamed) {\n            handler = new echo_stream_handler(chunked_reply);\n        } else {\n            handler = new echo_string_handler(chunked_reply);\n        }\n        server._routes.put(GET, \"/test\", handler);\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_string_content) {\n    return test_basic_content(false, false);\n}\n\nSEASTAR_TEST_CASE(test_string_content_chunked) {\n    return test_basic_content(false, true);\n}\n\nSEASTAR_TEST_CASE(test_stream_content) {\n    return test_basic_content(true, false);\n}\n\nSEASTAR_TEST_CASE(test_stream_content_chunked) {\n    return test_basic_content(true, true);\n}\n\nSEASTAR_TEST_CASE(test_not_implemented_encoding) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: gzip, chunked\\r\\n\\r\\n\",\n        \"a\\r\\n1234567890\\r\\n\",\n        \"a\\r\\n1234521345\\r\\n\",\n        \"0\\r\\n\\r\\n\"\n    }, {\"501 Not Implemented\", \"Encodings other than \\\"chunked\\\" are not implemented (received encoding: \\\"gzip, chunked\\\")\"}, false, new echo_string_handler());\n}\n\nSEASTAR_TEST_CASE(test_full_chunk_format) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\",\n        \"a;abc-def;hello=world;aaaa\\r\\n1234567890\\r\\n\",\n        \"a;a0-!#$%&'*+.^_`|~=\\\"quoted string obstext\\x80\\x81\\xff quoted_pair: \\\\a\\\"\\r\\n1234521345\\r\\n\",\n        \"0\\r\\na:b\\r\\n~|`_^.+*'&%$#!-0a:  ~!@#$%^&*()_+\\x80\\x81\\xff\\r\\n  obs fold  \\r\\n\\r\\n\"\n    }, {\"12345678901234521345\", \"abc-def\", \"hello=world\", \"aaaa\", \"a0-!#$%&'*+.^_`|~=quoted string obstext\\x80\\x81\\xff quoted_pair: a\",\n        \"a: b\", \"~|`_^.+*'&%$#!-0a: ~!@#$%^&*()_+\\x80\\x81\\xff obs fold\"\n    }, false, new echo_string_handler());\n}\n\nSEASTAR_TEST_CASE(test_chunk_extension_parser_fail) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\",\n        \"7; \\r\\nnoparse\\r\\n\",\n        \"0\\r\\n\\r\\n\"\n    }, {\"400 Bad Request\", \"Can't parse chunk size and extensions\"}, false, new echo_string_handler());\n}\n\nSEASTAR_TEST_CASE(test_trailer_part_parser_fail) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\",\n        \"8\\r\\nparsable\\r\\n\",\n        \"0\\r\\ngood:header\\r\\nbad=header\\r\\n\\r\\n\"\n    }, {\"400 Bad Request\", \"Can't parse chunked request trailer\"}, false, new echo_string_handler());\n}\n\nSEASTAR_TEST_CASE(test_too_long_chunk) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\",\n        \"a\\r\\n1234567890\\r\\n\",\n        \"a\\r\\n1234521345X\\r\\n\",\n        \"0\\r\\n\\r\\n\"\n    }, {\"400 Bad Request\", \"The actual chunk length exceeds the specified length\"}, true, new echo_stream_handler());\n}\n\nSEASTAR_TEST_CASE(test_bad_chunk_length) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n\",\n        \"a\\r\\n1234567890\\r\\n\",\n        \"aX\\r\\n1234521345\\r\\n\",\n        \"0\\r\\n\\r\\n\"\n    }, {\"400 Bad Request\", \"Can't parse chunk size and extensions\"}, true, new echo_stream_handler());\n}\n\nSEASTAR_TEST_CASE(test_close_response) {\n    return check_http_reply({\n        \"GET /test HTTP/1.1\\r\\nHost: test\\r\\nConnection: close\\r\\n\\r\\n\"\n    }, {\"200 OK\", \"Connection: close\"}, true, new echo_stream_handler()).then([] {\n        return check_http_reply({\n            \"/test\\r\\n\\r\\n\"\n        }, {\"400 Bad Request\", \"Connection: close\", \"Can't parse the request\"}, false, new echo_string_handler());\n    });\n}\n\nSEASTAR_TEST_CASE(case_insensitive_header) {\n    std::unique_ptr<seastar::http::request> req = std::make_unique<seastar::http::request>();\n    req->_headers[\"conTEnt-LengtH\"] = \"17\";\n    BOOST_REQUIRE_EQUAL(req->get_header(\"content-length\"), \"17\");\n    BOOST_REQUIRE_EQUAL(req->get_header(\"Content-Length\"), \"17\");\n    BOOST_REQUIRE_EQUAL(req->get_header(\"cOnTeNT-lEnGTh\"), \"17\");\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(broken_reply) {\n    http_response_parser parser;\n    parser.init();\n    char r101[] = \"Lorem ipsum dolor sit amet, consectetur adipiscing elit.\";\n\n    parser.parse(r101, r101 + sizeof(r101), r101 + sizeof(r101));\n    BOOST_REQUIRE_EQUAL(parser.failed(), true);\n    BOOST_REQUIRE_EQUAL(parser.error_message().starts_with(\"Parsing error at offset 0: encountered \\\"Lorem ipsum dolor sit amet, cons\\\"\"), true);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(case_insensitive_header_reply) {\n    http_response_parser parser;\n    parser.init();\n    char r101[] = \"HTTP/1.1 200 OK\\r\\ncontent-length: 17\\r\\ncontent-type: application/json\\r\\n\\r\\n{\\\"Hello\\\":\\\"World\\\"}\";\n\n    parser.parse(r101, r101 + sizeof(r101), r101 + sizeof(r101));\n    auto response = parser.get_parsed_response();\n    std::unique_ptr<seastar::http::reply> rep = std::make_unique<seastar::http::reply>(std::move(*response));\n    BOOST_REQUIRE_EQUAL(rep->get_header(\"content-length\"), \"17\");\n    BOOST_REQUIRE_EQUAL(rep->get_header(\"Content-Length\"), \"17\");\n    BOOST_REQUIRE_EQUAL(rep->get_header(\"cOnTeNT-lEnGTh\"), \"17\");\n\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(multiple_connections) {\n    loopback_connection_factory lcf = loopback_connection_factory::with_pending_capacity(smp::count + 1, 1);\n    http_server server(\"test\");\n    httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n    socket_address addr{ipv4_addr()};\n\n    std::vector<connected_socket> socks;\n    // Make sure one shard has two connections pending.\n    for (unsigned i = 0; i <= smp::count; ++i) {\n        socks.push_back(loopback_socket_impl(lcf).connect(addr, addr).get());\n    }\n\n    server.do_accepts(0).get();\n    server.stop().get();\n    lcf.destroy_all_shards().get();\n}\n\nSEASTAR_TEST_CASE(http_parse_response_status) {\n    http_response_parser parser;\n    parser.init();\n    char r101[] = \"HTTP/1.1 101 Switching Protocols\\r\\n\\r\\n\";\n    char r200[] = \"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\nhello\\r\\n\";\n\n    parser.parse(r101, r101 + sizeof(r101), r101 + sizeof(r101));\n    auto response = parser.get_parsed_response();\n    BOOST_REQUIRE_EQUAL(response->_status, http::reply::status_type::switching_protocols);\n\n    parser.init();\n    parser.parse(r200, r200 + sizeof(r200), r200 + sizeof(r200));\n    response = parser.get_parsed_response();\n    BOOST_REQUIRE_EQUAL(response->_status, http::reply::status_type::ok);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(http_parse_response_small_json) {\n    http_response_parser parser;\n    parser.init();\n    char r101[] = \"HTTP/1.1 200 OK\\r\\ndate: Thu, 25 May 2023 16:13:59 GMT\\r\\nserver: uvicorn\\r\\ncontent-length: 17\\r\\ncontent-type: application/json\\r\\n\\r\\n{\\\"Hello\\\":\\\"World\\\"}\";\n\n    parser.parse(r101, r101 + sizeof(r101), r101 + sizeof(r101));\n    auto resp = parser.get_parsed_response();\n    BOOST_REQUIRE_EQUAL((int)resp->_status, 200);\n    BOOST_REQUIRE_EQUAL(resp->get_header(\"Content-Length\"), \"17\");\n    BOOST_REQUIRE_EQUAL(resp->get_header(\"Content-Type\"), \"application/json\");\n    BOOST_REQUIRE_EQUAL(resp->get_header(\"server\"), \"uvicorn\");\n    BOOST_REQUIRE_EQUAL(resp->get_header(\"date\"), \"Thu, 25 May 2023 16:13:59 GMT\");\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_shared_future) {\n// This test is only valid until API level 8\n// where json_return_type became non-copyable.\n#if SEASTAR_API_LEVEL < 8\n    shared_promise<json::json_return_type> p;\n    auto fut = p.get_shared_future();\n\n    (void)yield().then([p = std::move(p)] () mutable {\n        p.set_value(json::json_void());\n    });\n\n    return std::move(fut).discard_result();\n#else\n    fmt::print(\"test_shared_future is invalid since API Level 8\\n\");\n    return make_ready_future<>();\n#endif\n}\n\nSEASTAR_TEST_CASE(test_url_encode_decode) {\n    sstring encoded, decoded;\n    bool ok;\n\n    sstring all_valid = \"~abcdefghijklmnopqrstuvwhyz-ABCDEFGHIJKLMNOPQRSTUVWXYZ_0123456789.\";\n    encoded = http::internal::url_encode(all_valid);\n    ok = http::internal::url_decode(encoded, decoded);\n    BOOST_REQUIRE_EQUAL(ok, true);\n    BOOST_REQUIRE_EQUAL(decoded, all_valid);\n    BOOST_REQUIRE_EQUAL(all_valid, encoded);\n\n    sstring some_invalid = \"a?/!@#$%^&*()[]=.\\\\ \\tZ\";\n    encoded = http::internal::url_encode(some_invalid);\n    ok = http::internal::url_decode(encoded, decoded);\n    BOOST_REQUIRE_EQUAL(ok, true);\n    BOOST_REQUIRE_EQUAL(decoded, some_invalid);\n    for (size_t i = 0; i < encoded.length(); i++) {\n        if (encoded[i] != '%' && encoded[i] != '+') {\n            auto f = std::find(std::begin(all_valid), std::end(all_valid), encoded[i]);\n            BOOST_REQUIRE_NE(f, std::end(all_valid));\n        }\n    }\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_url_param_encode_decode) {\n    http::request to_send;\n    auto& to_send_query_parameters = deprecated_query_parameters(to_send);\n    to_send._url = \"/foo/bar\";\n    to_send_query_parameters[\"a\"] = \"a+a*a\";\n    to_send_query_parameters[\"b\"] = \"b/b\\%b\";\n    to_send_query_parameters[\"c\"] = \"\";\n\n    http::request to_recv;\n    to_recv._url = to_send.format_url();\n    sstring url = to_recv.parse_query_param();\n\n    BOOST_REQUIRE_EQUAL(url, to_send._url);\n    const auto& to_recv_query_parameters = deprecated_query_parameters(to_recv);\n    BOOST_REQUIRE_EQUAL(to_recv_query_parameters.size(), to_send_query_parameters.size());\n    for (const auto& p : to_send_query_parameters) {\n        auto it = to_recv_query_parameters.find(p.first);\n        BOOST_REQUIRE(it != to_recv_query_parameters.end());\n        BOOST_REQUIRE_EQUAL(it->second, p.second);\n    }\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_url_param_encode_decode_multiple) {\n    http::request to_send;\n    to_send._url = \"/foo/bar\";\n\n    to_send.set_query_param(\"a\", {\"a+a*a\", \"a/a\\%a\", \"\", \"\", \"a a\", \"lasta\"})\n        .set_query_param(\"b\", {\"b/b\\%b\", \"\", \"lastb\"});\n\n    http::request to_recv;\n    to_recv._url = to_send.format_url();\n    sstring url = to_recv.parse_query_param();\n    BOOST_REQUIRE_EQUAL(url, to_send._url);\n    const auto& to_send_a = to_send.get_query_param_array(\"a\");\n    const auto& to_recv_a = to_recv.get_query_param_array(\"a\");\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(to_send_a.begin(), to_send_a.end(), to_recv_a.begin(), to_recv_a.end());\n    const auto& to_send_b = to_send.get_query_param_array(\"b\");\n    const auto& to_recv_b = to_recv.get_query_param_array(\"b\");\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(to_send_b.begin(), to_send_b.end(), to_recv_b.begin(), to_recv_b.end());\n\n    BOOST_REQUIRE_EQUAL(to_recv.get_query_param(\"a\"), to_send.get_query_param(\"a\"));\n    BOOST_REQUIRE_EQUAL(to_recv.get_query_param(\"b\"), to_send.get_query_param(\"b\"));\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_url_param_empty) {\n    sstring test_url = \"/foo/bar?key=v&key=&key&key2&key\";\n    http::request req;\n    req._url = test_url;\n    req.parse_query_param();\n    const auto& query_params = deprecated_query_parameters(req);\n    BOOST_REQUIRE_EQUAL(query_params.size(), 2);\n    BOOST_REQUIRE(query_params.at(\"key\").empty());\n    BOOST_REQUIRE(query_params.at(\"key2\").empty());\n\n    std::vector<sstring> expected_key = {\"v\", \"\", \"\", \"\"};\n    const auto& actual_key = req.get_query_param_array(\"key\");\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(actual_key.begin(), actual_key.end(), expected_key.begin(), expected_key.end());\n\n    std::vector<sstring> expected_key2 = {\"\"};\n    const auto& actual_key2 = req.get_query_param_array(\"key2\");\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(actual_key2.begin(), actual_key2.end(), expected_key2.begin(), expected_key2.end());\n\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"key\"), \"\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"key2\"), \"\");\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_url_params_get_set) {\n    http::request req;\n    req._url = \"/foo/bar\";\n    http::request::query_parameters_type params = {\n        {\"a\", {\"a+a*a\", \"a/a%a\", \"a a\", \"lasta\"}},\n        {\"b\", {\"b/b%b\", \"lastb\"}}\n    };\n\n    req.set_query_params(params);\n\n    const auto& req_params = req.get_query_params();\n    for (const auto&[key, values] : params) {\n        auto it = req_params.find(key);\n        BOOST_REQUIRE(it != req_params.end());\n        BOOST_REQUIRE_EQUAL_COLLECTIONS(it->second.begin(), it->second.end(), values.begin(), values.end());\n    }\n\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"a\"), \"lasta\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"b\"), \"lastb\");\n\n    req.set_query_param(\"a\", \"new_a\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"a\"), \"new_a\");\n\n    req.set_query_param(\"c\", \"c/c\\%c\");\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"c\"), \"c/c%c\");\n\n    std::vector<sstring> d_params = {\"d/d%d\", \"lastd\"};\n    req.set_query_param(\"d\", d_params);\n    BOOST_REQUIRE_EQUAL(req.get_query_param(\"d\"), \"lastd\");\n    const auto& d_values = req.get_query_param_array(\"d\");\n    BOOST_REQUIRE_EQUAL_COLLECTIONS(d_values.begin(), d_values.end(), d_params.begin(), d_params.end());\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_unexpected_exception_format) {\n    try {\n        throw httpd::unexpected_status_error(http::reply::status_type::see_other);\n    } catch (const std::exception& ex) {\n        BOOST_REQUIRE_EQUAL(sstring(ex.what()), format(\"{}\", http::reply::status_type::see_other));\n    }\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_redirect_exception) {\n    return seastar::async([] {\n        class perm_handle : public httpd::handler_base {\n        public:\n            virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n                    std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                return make_exception_future<std::unique_ptr<http::reply>>(httpd::redirect_exception(\"/perm_loc\"));\n            }\n        };\n\n        class temp_handle : public httpd::handler_base {\n        public:\n            virtual future<std::unique_ptr<http::reply> > handle(const sstring& path,\n                    std::unique_ptr<http::request> req, std::unique_ptr<http::reply> rep) {\n                return make_exception_future<std::unique_ptr<http::reply>>(httpd::redirect_exception(\"/temp_loc\",\n                        http::reply::status_type::moved_temporarily));\n            }\n        };\n\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        future<> client = seastar::async([&lcf] {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf));\n\n            auto test = [&](sstring path, sstring expected_dest, http::reply::status_type expected_status) {\n                auto req = http::request::make(\"GET\", \"test\", path);\n                std::optional<http::reply::status_type> status;\n                sstring location;\n                cln.make_request(std::move(req), [&] (const http::reply& rep, input_stream<char>&& in) {\n                    status = rep._status;\n                    location = rep.get_header(\"Location\");\n                    return make_ready_future<>();\n                }).get();\n                BOOST_REQUIRE(status.has_value());\n                BOOST_REQUIRE_EQUAL(status.value(), expected_status);\n                BOOST_REQUIRE_EQUAL(location, expected_dest);\n            };\n\n            test(\"/perm\", \"/perm_loc\", http::reply::status_type::moved_permanently);\n            test(\"/temp\", \"/temp_loc\", http::reply::status_type::moved_temporarily);\n\n            cln.close().get();\n        });\n\n        server._routes.put(GET, \"/perm\", new perm_handle());\n        server._routes.put(GET, \"/temp\", new temp_handle());\n        server.do_accepts(0).get();\n        client.get();\n        server.stop().get();\n    });\n}\n\nBOOST_AUTO_TEST_CASE(test_path_decode_unchanged) {\n    auto unchanged_chars = seastar::sstring{\n      \"~abcdefghijklmnopqrstuvwhyz-ABCDEFGHIJKLMNOPQRSTUVWXYZ_0123456789.+\"};\n    auto result = seastar::sstring{};\n    auto success = http::internal::path_decode(unchanged_chars, result);\n\n    BOOST_REQUIRE(success);\n    BOOST_REQUIRE_EQUAL(result, unchanged_chars);\n}\n\nBOOST_AUTO_TEST_CASE(test_path_decode_changed) {\n    auto changed_chars = seastar::sstring{\"%20\"};\n    auto result = seastar::sstring{};\n    auto success = http::internal::path_decode(changed_chars, result);\n\n    BOOST_REQUIRE(success);\n\n    auto expected_chars = seastar::sstring{\" \"};\n    BOOST_REQUIRE_EQUAL(result, expected_chars);\n}\n\nnamespace seastar::http {\nstd::ostream& boost_test_print_type(std::ostream& os, reply::status_class sc) {\n    constexpr std::string_view status_strings[] {\n        \"1xx: Informational\",\n        \"2xx: Success\",\n        \"3xx: Redirection\",\n        \"4xx: Client Error\",\n        \"5xx: Server Error\",\n        \"Unclassified\"\n    };\n    auto status = static_cast<std::underlying_type_t<reply::status_class>>(sc) - 1u;\n    if (status < std::size(status_strings)) {\n        return os << status_strings[status];\n    }\n    return os << \"Unclassified\";\n}\n} // namespace seastar::http\n\nBOOST_AUTO_TEST_CASE(test_http_status_classification) {\n    size_t informational = 0;\n    size_t success = 0;\n    size_t redirection = 0;\n    size_t client_error = 0;\n    size_t server_error = 0;\n    size_t unclassified = 0;\n    for (auto i = -100; i < 700; ++i) {\n        auto classification = http::reply::classify_status(static_cast<http::reply::status_type>(i));\n        if (i >= 100 && i < 200) {\n            ++informational;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::informational);\n        } else if (i >= 200 && i < 300) {\n            ++success;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::success);\n        } else if (i >= 300 && i < 400) {\n            ++redirection;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::redirection);\n        } else if (i >= 400 && i < 500) {\n            ++client_error;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::client_error);\n        } else if (i >= 500 && i < 600) {\n            ++server_error;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::server_error);\n        } else {\n            ++unclassified;\n            BOOST_REQUIRE_EQUAL(classification, http::reply::status_class::unclassified);\n        }\n    }\n    BOOST_REQUIRE_EQUAL(informational, 100);\n    BOOST_REQUIRE_EQUAL(success, 100);\n    BOOST_REQUIRE_EQUAL(redirection, 100);\n    BOOST_REQUIRE_EQUAL(client_error, 100);\n    BOOST_REQUIRE_EQUAL(server_error, 100);\n    BOOST_REQUIRE_EQUAL(unclassified, 300);\n}\n\n// #2661. Check that trying a http connection with a wire error\n// can be handled.\nSEASTAR_THREAD_TEST_CASE(test_http_with_broken_wire) {\n    tls::credentials_builder b;\n    auto creds = b.build_certificate_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto server = seastar::listen(::make_ipv4_address( {0x7f000001, 0}), opts);\n    auto addr = server.local_address();\n\n    http::experimental::client c(addr, creds);\n    http::request req;\n    req._version = \"1.1\";\n    req.write_body(\"html\", std::string(5134, 'a'));\n\n    auto sa = server.accept();\n    auto f = c.make_request(std::move(req), [](const http::reply&, input_stream<char>&& body) -> future<> {\n        // don't care\n        co_return;\n    });\n\n    auto s = sa.get();\n\n    s.connection.input().close().get();\n    s.connection.output().close().get();\n\n    BOOST_CHECK_THROW(f.get(), std::system_error);\n\n    c.close().get();\n}\n\nfuture<> test_client_close_connection(bool chunked) {\n    return async([chunked] {\n        loopback_connection_factory lcf(1);\n        auto make_test_request = [&lcf, chunked]() {\n            auto cln = http::experimental::client(std::make_unique<loopback_http_factory>(lcf), 1, http::experimental::client::retry_requests::no);\n            size_t content_length = 0;\n            for (auto _ [[maybe_unused]] : {1, 2}) {\n                auto req = http::request::make(\"GET\", \"test\", \"/test\");\n                auto make_request = cln.make_request(\n                    std::move(req),\n                    [&content_length, chunked](const http::reply& resp, input_stream<char>&& in) {\n                        content_length = chunked ? 128_KiB : resp.content_length;\n                        return async([&content_length, in = std::move(in)]() mutable {\n                            // just read some bytes and abandon\n                            auto buff = in.read().get();\n                            BOOST_REQUIRE(buff.size() < content_length);\n                            in.close().get();\n                        });\n                    },\n                    http::reply::status_type::ok);\n                BOOST_REQUIRE_NO_THROW(make_request.get());\n            }\n            cln.close().get();\n        };\n\n        size_t response_size = 0;\n        auto make_response = [&response_size, chunked](accept_result ar) {\n            return async([response_size, sk = std::move(ar.connection), chunked]() mutable {\n                input_stream<char> in = sk.input();\n                read_simple_http_request(in);\n                output_stream<char> out = sk.output();\n                size_t responses = 0;\n                // In the case the leftover data on the socket is smaller than 128KiB we are going to drain it and leave the connection alive, so here we\n                // have to loop two times to fulfill two request from the client. On the other hand if the leftover data is larger than 128KiB we are going\n                // to close the connection, so we have to loop only once to fulfill one request from the client and make another `accept`\n                while (true) {\n                    if (responses == 2) {\n                        break;\n                    }\n                    ++responses;\n                    try {\n                        if (!chunked) {\n                            out.write(format(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\nContent-Length: {}\\r\\n\\r\\n\", response_size)).get();\n                            out.flush().get();\n                        } else {\n                            out.write(format(\"HTTP/1.1 200 OK\\r\\nHost: localhost\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n{:x}\\r\\n\", response_size)).get();\n                            out.flush().get();\n                        }\n                        out.write(sstring(response_size / 2, 'a')).get();\n                        out.flush().get();\n\n                        out.write(sstring(response_size / 2, 'a')).get();\n                        out.flush().get();\n\n                        if (chunked) {\n                            out.write(format(\"\\r\\n0\\r\\n\\r\\n\")).get();\n                            out.flush().get();\n                        }\n                    } catch (...) {\n                        break;\n                    }\n                }\n                out.close().handle_exception_type([](std::system_error& ex){\n                    if (ex.code().value() == EPIPE) {\n                        return make_ready_future<>();\n                    } else {\n                        return make_exception_future<>(ex);\n                    }\n                }).get();\n            });\n        };\n\n        for (auto size : {128_KiB, 260_KiB}) {\n            response_size = size;\n            auto ss = lcf.get_server_socket();\n            auto server = ss.accept().then(make_response);\n            if (size > 128_KiB || chunked) {\n                // In this case the client is going to reset the connection so we have to `accept` again\n                server = server.then([&ss, &make_response] { return ss.accept().then(make_response); });\n            }\n            auto client = async([&make_test_request] { make_test_request(); });\n\n            when_all(std::move(server), std::move(client)).discard_result().get();\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_client_close_connection_content_length) {\n    return test_client_close_connection(false);\n}\n\nSEASTAR_TEST_CASE(test_client_close_connection_chunked) {\n    return test_client_close_connection(true);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_content_length_data_sink) {\n    auto do_check = [] (size_t len, sstring value, bool zero_copy) {\n        size_t written = 32;\n        size_t expected = 0;\n        std::stringstream ss;\n        sstring expected_ss;\n        output_stream<char> data = output_stream<char>(testing::memory_data_sink(ss));\n        output_stream<char> out = http::internal::make_http_content_length_output_stream(data, len, written);\n        BOOST_CHECK_EQUAL(written, 0);\n\n        unsigned values = 0;\n        while (true) {\n            expected += value.size();\n            if (zero_copy) {\n                out.write(temporary_buffer<char>(value.c_str(), value.size())).get();\n            } else {\n                out.write(value).get();\n            }\n            auto f = out.flush();\n            if (expected > len) {\n                BOOST_CHECK_EXCEPTION(f.get(), std::runtime_error, [] (const auto& e) { return sstring(e.what()).starts_with(\"body content length overflow\"); });\n                BOOST_CHECK_EQUAL(written, expected - value.size());\n                break;\n            }\n\n            f.get();\n            BOOST_CHECK_EQUAL(written, expected);\n            data.flush().get();\n            expected_ss += value;\n            values++;\n        }\n\n        BOOST_CHECK_EQUAL(values, len / value.size());\n        BOOST_CHECK_EQUAL(ss.str(), expected_ss);\n    };\n\n    do_check(2, \"1\", false);\n    do_check(2, \"12\", false);\n    do_check(2, \"123\", false);\n    do_check(2, \"1\", true);\n    do_check(2, \"12\", true);\n    do_check(2, \"123\", true);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reply_cookies) {\n    auto reply = std::make_unique<http::reply>();\n    reply->set_cookie(\"cookie1\", \"1\");\n    reply->set_cookie(\"cookie2\", \"2\");\n    reply->add_header(\"Content-Encoding\", \"gzip\");\n\n    std::stringstream ss;\n    auto os = output_stream<char>(data_sink(std::make_unique<testing::memory_data_sink_impl>(ss)));\n    auto close_os = deferred_close(os);\n\n    reply->write_reply_headers(os).get();\n\n    os.flush().get();\n\n    auto headers_str = ss.str();\n\n    auto e = std::remove(headers_str.begin(), headers_str.end(), '\\r');\n    sstring headers_str_no_cr(headers_str.begin(), e);\n\n    std::vector<sstring> lines;\n    boost::split(lines, headers_str_no_cr, boost::is_any_of(\"\\n\"));\n    BOOST_REQUIRE_EQUAL(lines.size(), 4);\n    lines.pop_back(); // last line is empty\n    std::sort(lines.begin(), lines.end());\n\n    // Check that both headers and cookies are present\n    BOOST_REQUIRE_EQUAL(lines[0], \"Content-Encoding: gzip\");\n    BOOST_REQUIRE_EQUAL(lines[1], \"Set-Cookie: cookie1=1\");\n    BOOST_REQUIRE_EQUAL(lines[2], \"Set-Cookie: cookie2=2\");\n}\n"
  },
  {
    "path": "tests/unit/https-server.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n\n#\n# Copyright (C) 2023 Kefu Chai ( tchaikov@gmail.com )\n#\n\n\nimport argparse\nimport socket\nimport ssl\nfrom http.server import HTTPServer as _HTTPServer\nfrom http.server import SimpleHTTPRequestHandler\n\n\nclass HTTPSServer(_HTTPServer):\n    def __init__(self, addr, port, context):\n        super().__init__((addr, port), SimpleHTTPRequestHandler)\n        self.context = context\n\n    def get_request(self):\n        sock, addr = self.socket.accept()\n        ssl_conn = self.context.wrap_socket(sock, server_side=True)\n        return ssl_conn, addr\n\n    def get_listen_port(self):\n        if self.socket.family == socket.AF_INET:\n            addr, port = self.socket.getsockname()\n            return port\n        elif self.socket.family == socket.AF_INET6:\n            address, port, flowinfo, scope_id = self.socket.getsockname()\n            return port\n        else:\n            raise Exception(f\"unknown family: {self.socket.family}\")\n\n\nif __name__ == \"__main__\":\n    parser = argparse.ArgumentParser(description=\"httpd for testing TLS\")\n    parser.add_argument('--server', action='store',\n                        help='server address in <host>:<port> format',\n                        default='localhost:11311')\n    parser.add_argument('--cert', action='store',\n                        help='path to the certificate')\n    parser.add_argument('--key', action='store',\n                        help='path to the private key')\n    args = parser.parse_args()\n    host, port = args.server.split(':')\n\n    context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)\n    context.load_cert_chain(certfile=args.cert, keyfile=args.key)\n    with HTTPSServer(host, int(port), context) as server:\n        # print out the listening port when ready to serve\n        print(server.get_listen_port(), flush=True)\n        server.serve_forever()\n"
  },
  {
    "path": "tests/unit/io_queue_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n\n#include <chrono>\n#include <ratio>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/file.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/io_intent.hh>\n#include <seastar/core/disk_params.hh>\n#include <seastar/core/internal/io_request.hh>\n#include <seastar/core/internal/io_sink.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/integrated-length.hh>\n\nusing namespace seastar;\n\nstruct fake_file {\n    std::unordered_map<uint64_t, int> data;\n\n    static internal::io_request make_write_req(size_t idx, int* buf) {\n        return internal::io_request::make_write(0, idx, buf, 1, false);\n    }\n\n    static internal::io_request make_writev_req(size_t idx, int* buf, size_t nr, size_t buf_len, std::vector<::iovec>& vecs) {\n        vecs.reserve(nr);\n        for (unsigned i = 0; i < nr; i++) {\n            vecs.push_back({ &buf[i], buf_len });\n        }\n        return internal::io_request::make_writev(0, idx, vecs, false);\n    }\n\n    void execute_write_req(const internal::io_request& rq, io_completion* desc) {\n        const auto& op = rq.as<internal::io_request::operation::write>();\n        data[op.pos] = *(reinterpret_cast<int*>(op.addr));\n        desc->complete_with(op.size);\n    }\n\n    void execute_writev_req(const internal::io_request& rq, io_completion* desc) {\n        size_t len = 0;\n        const auto& op = rq.as<internal::io_request::operation::writev>();\n        for (unsigned i = 0; i < op.iov_len; i++) {\n            data[op.pos + i] = *(reinterpret_cast<int*>(op.iovec[i].iov_base));\n            len += op.iovec[i].iov_len;\n        }\n        desc->complete_with(len);\n    }\n};\n\nstruct io_queue_for_tests {\n    io_group_ptr group;\n    internal::io_sink sink;\n    io_queue queue;\n    timer<> kicker;\n\n    io_queue_for_tests(const io_queue::config& cfg = io_queue::config{0})\n        : group(std::make_shared<io_group>(cfg, 1))\n        , sink()\n        , queue(group, sink)\n        , kicker([this] { kick(); })\n    {\n        kicker.arm_periodic(std::chrono::microseconds(500));\n    }\n\n    void kick() {\n        for (auto&& fg : group->_fgs) {\n            fg.replenish_capacity(std::chrono::steady_clock::now());\n        }\n    }\n\n    future<size_t> queue_request(internal::priority_class pc, internal::io_direction_and_length dnl, internal::io_request req, io_intent* intent, iovec_keeper iovs) noexcept {\n        return queue.queue_request(pc, dnl, std::move(req), intent, std::move(iovs));\n    }\n\n    size_t max_request_length(int dnl_idx) const noexcept {\n        return group->max_request_length(dnl_idx);\n    }\n\n    constexpr size_t request_length_limit() const noexcept {\n        return io_group::request_length_limit;\n    }\n\n    void find_or_create_class(internal::priority_class pc) {\n        queue.find_or_create_class(pc);\n    }\n\n    fair_queue& get_fair_queue() {\n        return queue._streams[0].fq;\n    }\n\n    bool is_class_registered(internal::priority_class pc) const noexcept {\n        return queue._priority_classes.size() > pc.id() && (queue._priority_classes[pc.id()] != nullptr);\n    }\n};\n\ninternal::priority_class get_default_pc() {\n    return internal::priority_class(current_scheduling_group());\n}\n\nSEASTAR_THREAD_TEST_CASE(test_basic_flow) {\n    io_queue_for_tests tio;\n    fake_file file;\n\n    auto val = std::make_unique<int>(42);\n    auto f = tio.queue_request(get_default_pc(), internal::io_direction_and_length(internal::io_direction_and_length::write_idx, 0), file.make_write_req(0, val.get()), nullptr, {})\n    .then([&file] (size_t len) {\n        BOOST_REQUIRE(file.data[0] == 42);\n    });\n\n    seastar::sleep(std::chrono::milliseconds(500)).get();\n    tio.queue.poll_io_queue();\n    tio.sink.drain([&file] (const internal::io_request& rq, io_completion* desc) -> bool {\n        file.execute_write_req(rq, desc);\n        return true;\n    });\n\n    f.get();\n}\n\nenum class part_flaw { none, partial, error };\n\nstatic void do_test_large_request_flow(part_flaw flaw) {\n    io_queue_for_tests tio;\n    fake_file file;\n    int values[3] = { 13, 42, 73 };\n\n    auto limits = tio.queue.get_request_limits();\n\n    std::vector<::iovec> vecs;\n    auto f = tio.queue_request(get_default_pc(), internal::io_direction_and_length(internal::io_direction_and_length::write_idx, limits.max_write * 3),\n                    file.make_writev_req(0, values, 3, limits.max_write, vecs), nullptr, std::move(vecs))\n    .then([&file, &values, &limits, flaw] (size_t len) {\n        size_t expected = limits.max_write;\n\n        BOOST_REQUIRE_EQUAL(file.data[0 * limits.max_write], values[0]);\n\n        if (flaw == part_flaw::none) {\n            BOOST_REQUIRE_EQUAL(file.data[1 * limits.max_write], values[1]);\n            BOOST_REQUIRE_EQUAL(file.data[2 * limits.max_write], values[2]);\n            expected += 2 * limits.max_write;\n        }\n\n        if (flaw == part_flaw::partial) {\n            BOOST_REQUIRE_EQUAL(file.data[1 * limits.max_write], values[1]);\n            expected += limits.max_write / 2;\n        }\n\n        BOOST_REQUIRE_EQUAL(len, expected);\n    });\n\n    for (int i = 0; i < 3; i++) {\n        seastar::sleep(std::chrono::milliseconds(500)).get();\n        tio.queue.poll_io_queue();\n        tio.sink.drain([&file, i, flaw] (const internal::io_request& rq, io_completion* desc) -> bool {\n            if (i == 1) {\n                if (flaw == part_flaw::partial) {\n                    const auto& op = rq.as<internal::io_request::operation::writev>();\n                    op.iovec[0].iov_len /= 2;\n                }\n                if (flaw == part_flaw::error) {\n                    desc->complete_with(-EIO);\n                    return true;\n                }\n            }\n            file.execute_writev_req(rq, desc);\n            return true;\n        });\n    }\n\n    f.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_large_request_flow) {\n    do_test_large_request_flow(part_flaw::none);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_large_request_flow_partial) {\n    do_test_large_request_flow(part_flaw::partial);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_large_request_flow_error) {\n    do_test_large_request_flow(part_flaw::error);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_intent_safe_ref) {\n    auto get_cancelled = [] (internal::intent_reference& iref) -> bool {\n        try {\n            iref.retrieve();\n            return false;\n        } catch(seastar::cancelled_error& err) {\n            return true;\n        }\n    };\n\n    io_intent intent, intent_x;\n\n    internal::intent_reference ref_orig(&intent);\n    BOOST_REQUIRE(ref_orig.retrieve() == &intent);\n\n    // Test move armed\n    internal::intent_reference ref_armed(std::move(ref_orig));\n    BOOST_REQUIRE(ref_orig.retrieve() == nullptr);\n    BOOST_REQUIRE(ref_armed.retrieve() == &intent);\n\n    internal::intent_reference ref_armed_2(&intent_x);\n    ref_armed_2 = std::move(ref_armed);\n    BOOST_REQUIRE(ref_armed.retrieve() == nullptr);\n    BOOST_REQUIRE(ref_armed_2.retrieve() == &intent);\n\n    intent.cancel();\n    BOOST_REQUIRE(get_cancelled(ref_armed_2));\n\n    // Test move cancelled\n    internal::intent_reference ref_cancelled(std::move(ref_armed_2));\n    BOOST_REQUIRE(ref_armed_2.retrieve() == nullptr);\n    BOOST_REQUIRE(get_cancelled(ref_cancelled));\n\n    internal::intent_reference ref_cancelled_2(&intent_x);\n    ref_cancelled_2 = std::move(ref_cancelled);\n    BOOST_REQUIRE(ref_cancelled.retrieve() == nullptr);\n    BOOST_REQUIRE(get_cancelled(ref_cancelled_2));\n\n    // Test move empty\n    internal::intent_reference ref_empty(std::move(ref_orig));\n    BOOST_REQUIRE(ref_empty.retrieve() == nullptr);\n\n    internal::intent_reference ref_empty_2(&intent_x);\n    ref_empty_2 = std::move(ref_empty);\n    BOOST_REQUIRE(ref_empty_2.retrieve() == nullptr);\n}\n\nstatic constexpr int nr_requests = 24;\n\nSEASTAR_THREAD_TEST_CASE(test_io_cancellation) {\n    fake_file file;\n\n    auto sg0 = create_scheduling_group(\"a\", 100).get();\n    auto sg1 = create_scheduling_group(\"b\", 100).get();\n    auto stop = defer([&] () noexcept {\n        destroy_scheduling_group(sg0).get();\n        destroy_scheduling_group(sg1).get();\n    });\n\n    io_queue_for_tests tio;\n    auto pc0 = internal::priority_class(sg0);\n    auto pc1 = internal::priority_class(sg1);\n\n    size_t idx = 0;\n    int val = 100;\n\n    io_intent live, dead;\n\n    std::vector<future<>> finished;\n    std::vector<future<>> cancelled;\n\n    auto queue_legacy_request = [&] (io_queue_for_tests& q, internal::priority_class pc) {\n        auto buf = std::make_unique<int>(val);\n        auto f = q.queue_request(pc, internal::io_direction_and_length(internal::io_direction_and_length::write_idx, 0), file.make_write_req(idx, buf.get()), nullptr, {})\n            .then([&file, idx, val, buf = std::move(buf)] (size_t len) {\n                BOOST_REQUIRE(file.data[idx] == val);\n                return make_ready_future<>();\n            });\n        finished.push_back(std::move(f));\n        idx++;\n        val++;\n    };\n\n    auto queue_live_request = [&] (io_queue_for_tests& q, internal::priority_class pc) {\n        auto buf = std::make_unique<int>(val);\n        auto f = q.queue_request(pc, internal::io_direction_and_length(internal::io_direction_and_length::write_idx, 0), file.make_write_req(idx, buf.get()), &live, {})\n            .then([&file, idx, val, buf = std::move(buf)] (size_t len) {\n                BOOST_REQUIRE(file.data[idx] == val);\n                return make_ready_future<>();\n            });\n        finished.push_back(std::move(f));\n        idx++;\n        val++;\n    };\n\n    auto queue_dead_request = [&] (io_queue_for_tests& q, internal::priority_class pc) {\n        auto buf = std::make_unique<int>(val);\n        auto f = q.queue_request(pc, internal::io_direction_and_length(internal::io_direction_and_length::write_idx, 0), file.make_write_req(idx, buf.get()), &dead, {})\n            .then_wrapped([buf = std::move(buf)] (auto&& f) {\n                try {\n                    f.get();\n                    BOOST_REQUIRE(false);\n                } catch(...) {}\n                return make_ready_future<>();\n            })\n            .then([&file, idx] () {\n                BOOST_REQUIRE(file.data[idx] == 0);\n            });\n        cancelled.push_back(std::move(f));\n        idx++;\n        val++;\n    };\n\n    auto seed = std::random_device{}();\n    std::default_random_engine reng(seed);\n    std::uniform_int_distribution<> dice(0, 5);\n\n    for (int i = 0; i < nr_requests; i++) {\n        int pc = dice(reng) % 2;\n        if (dice(reng) < 3) {\n            fmt::print(\"queue live req to pc {}\\n\", pc);\n            queue_live_request(tio, pc == 0 ? pc0 : pc1);\n        } else if (dice(reng) < 5) {\n            fmt::print(\"queue dead req to pc {}\\n\", pc);\n            queue_dead_request(tio, pc == 0 ? pc0 : pc1);\n        } else {\n            fmt::print(\"queue legacy req to pc {}\\n\", pc);\n            queue_legacy_request(tio, pc == 0 ? pc0 : pc1);\n        }\n    }\n\n    dead.cancel();\n\n    // cancelled requests must resolve right at once\n\n    when_all_succeed(cancelled.begin(), cancelled.end()).get();\n\n    seastar::sleep(std::chrono::milliseconds(500)).get();\n    tio.queue.poll_io_queue();\n    tio.sink.drain([&file] (const internal::io_request& rq, io_completion* desc) -> bool {\n        file.execute_write_req(rq, desc);\n        return true;\n    });\n\n    when_all_succeed(finished.begin(), finished.end()).get();\n}\n\nSEASTAR_TEST_CASE(test_request_buffer_split) {\n    auto ensure = [] (const std::vector<internal::io_request::part>& parts, const internal::io_request& req, int idx, uint64_t pos, size_t size, uintptr_t mem) {\n        BOOST_REQUIRE(parts[idx].req.opcode() == req.opcode());\n        const auto& op = req.as<internal::io_request::operation::read>();\n        const auto& sub_op = parts[idx].req.as<internal::io_request::operation::read>();\n        BOOST_REQUIRE_EQUAL(sub_op.fd, op.fd);\n        BOOST_REQUIRE_EQUAL(sub_op.pos, pos);\n        BOOST_REQUIRE_EQUAL(sub_op.size, size);\n        BOOST_REQUIRE_EQUAL(sub_op.addr, reinterpret_cast<void*>(mem));\n        BOOST_REQUIRE_EQUAL(sub_op.nowait_works, op.nowait_works);\n        BOOST_REQUIRE_EQUAL(parts[idx].iovecs.size(), 0);\n        BOOST_REQUIRE_EQUAL(parts[idx].size, sub_op.size);\n    };\n\n    // No split\n    {\n        internal::io_request req = internal::io_request::make_read(5, 13, reinterpret_cast<void*>(0x420), 17, true);\n        auto parts = req.split(21);\n        BOOST_REQUIRE_EQUAL(parts.size(), 1);\n        ensure(parts, req, 0, 13, 17, 0x420);\n    }\n\n    // Without tail\n    {\n        internal::io_request req = internal::io_request::make_read(7, 24, reinterpret_cast<void*>(0x4321), 24, true);\n        auto parts = req.split(12);\n        BOOST_REQUIRE_EQUAL(parts.size(), 2);\n        ensure(parts, req, 0, 24,      12, 0x4321);\n        ensure(parts, req, 1, 24 + 12, 12, 0x4321 + 12);\n    }\n\n    // With tail\n    {\n        internal::io_request req = internal::io_request::make_read(9, 42, reinterpret_cast<void*>(0x1234), 33, true);\n        auto parts = req.split(13);\n        BOOST_REQUIRE_EQUAL(parts.size(), 3);\n        ensure(parts, req, 0, 42,      13, 0x1234);\n        ensure(parts, req, 1, 42 + 13, 13, 0x1234 + 13);\n        ensure(parts, req, 2, 42 + 26,  7, 0x1234 + 26);\n    }\n\n    return make_ready_future<>();\n}\n\nstatic void show_request(const internal::io_request& req, void* buf_off, std::string pfx = \"\") {\n    if (!seastar_logger.is_enabled(log_level::trace)) {\n        return;\n    }\n\n    const auto& op = req.as<internal::io_request::operation::readv>();\n    seastar_logger.trace(\"{}{} iovecs on req:\", pfx, op.iov_len);\n    for (unsigned i = 0; i < op.iov_len; i++) {\n        seastar_logger.trace(\"{}  base={} len={}\", pfx, reinterpret_cast<uintptr_t>(op.iovec[i].iov_base) - reinterpret_cast<uintptr_t>(buf_off), op.iovec[i].iov_len);\n    }\n}\n\nstatic void show_request_parts(const std::vector<internal::io_request::part>& parts, void* buf_off) {\n    if (!seastar_logger.is_enabled(log_level::trace)) {\n        return;\n    }\n\n    seastar_logger.trace(\"{} parts\", parts.size());\n    for (const auto& p : parts) {\n        seastar_logger.trace(\"  size={} iovecs={}\", p.size, p.iovecs.size());\n        seastar_logger.trace(\"  {} iovecs on part:\", p.iovecs.size());\n        for (const auto& iov : p.iovecs) {\n            seastar_logger.trace(\"    base={} len={}\", reinterpret_cast<uintptr_t>(iov.iov_base) - reinterpret_cast<uintptr_t>(buf_off), iov.iov_len);\n        }\n        show_request(p.req, buf_off, \"  \");\n    }\n}\n\nSEASTAR_TEST_CASE(test_request_iovec_split) {\n    char large_buffer[1025];\n\n    auto clear_buffer = [&large_buffer] {\n        memset(large_buffer, 0, sizeof(large_buffer));\n    };\n\n    auto bump_buffer = [] (const std::vector<::iovec>& vecs) {\n        for (auto&& v : vecs) {\n            for (unsigned i = 0; i < v.iov_len; i++) {\n                (reinterpret_cast<char*>(v.iov_base))[i]++;\n            }\n        }\n    };\n\n    auto check_buffer = [&large_buffer] (size_t len, char value) {\n        SEASTAR_ASSERT(len < sizeof(large_buffer));\n        bool fill_match = true;\n        bool train_match = true;\n        for (unsigned i = 0; i < sizeof(large_buffer); i++) {\n            if (i < len) {\n                if (large_buffer[i] != value) {\n                    fill_match = false;\n                }\n            } else {\n                if (large_buffer[i] != '\\0') {\n                    train_match = false;\n                }\n            }\n        }\n        BOOST_REQUIRE_EQUAL(fill_match, true);\n        BOOST_REQUIRE_EQUAL(train_match, true);\n    };\n\n    auto ensure = [] (const std::vector<internal::io_request::part>& parts, const internal::io_request& req, int idx, uint64_t pos) {\n        BOOST_REQUIRE(parts[idx].req.opcode() == req.opcode());\n        const auto& op = req.as<internal::io_request::operation::writev>();\n        const auto& sub_op = parts[idx].req.as<internal::io_request::operation::writev>();\n        BOOST_REQUIRE_EQUAL(sub_op.fd, op.fd);\n        BOOST_REQUIRE_EQUAL(sub_op.pos, pos);\n        BOOST_REQUIRE_EQUAL(sub_op.iov_len, parts[idx].iovecs.size());\n        BOOST_REQUIRE_EQUAL(sub_op.nowait_works, op.nowait_works);\n        BOOST_REQUIRE_EQUAL(parts[idx].size, internal::iovec_len(parts[idx].iovecs));\n\n        for (unsigned iov = 0; iov < parts[idx].iovecs.size(); iov++) {\n            BOOST_REQUIRE_EQUAL(sub_op.iovec[iov].iov_base, parts[idx].iovecs[iov].iov_base);\n            BOOST_REQUIRE_EQUAL(sub_op.iovec[iov].iov_len, parts[idx].iovecs[iov].iov_len);\n        }\n    };\n\n    std::default_random_engine& reng = testing::local_random_engine;\n    auto dice = std::uniform_int_distribution<uint16_t>(1, 31);\n    auto stop = std::chrono::steady_clock::now() + std::chrono::seconds(4);\n    uint64_t iter = 0;\n    unsigned no_splits = 0;\n    unsigned no_tails = 0;\n\n    do {\n        seastar_logger.debug(\"===== iter {} =====\", iter++);\n        std::vector<::iovec> vecs;\n        unsigned nr_vecs = dice(reng) % 13 + 1;\n        seastar_logger.debug(\"Generate {} iovecs\", nr_vecs);\n        size_t total = 0;\n        for (unsigned i = 0; i < nr_vecs; i++) {\n            ::iovec iov;\n            iov.iov_base = reinterpret_cast<void*>(large_buffer + total);\n            iov.iov_len = dice(reng);\n            SEASTAR_ASSERT(iov.iov_len != 0);\n            total += iov.iov_len;\n            vecs.push_back(std::move(iov));\n        }\n\n        SEASTAR_ASSERT(total > 0);\n        clear_buffer();\n        bump_buffer(vecs);\n        check_buffer(total, 1);\n\n        size_t file_off = dice(reng);\n        internal::io_request req = internal::io_request::make_readv(5, file_off, vecs, true);\n\n        show_request(req, large_buffer);\n\n        size_t max_len = dice(reng) * 3;\n        unsigned nr_parts = (total + max_len - 1) / max_len;\n        seastar_logger.debug(\"Split {} into {}-bytes ({} parts)\", total, max_len, nr_parts);\n        auto parts = req.split(max_len);\n        show_request_parts(parts, large_buffer);\n        BOOST_REQUIRE_EQUAL(parts.size(), nr_parts);\n\n        size_t parts_total = 0;\n        for (unsigned p = 0; p < nr_parts; p++) {\n            ensure(parts, req, p, file_off + parts_total);\n            if (p < nr_parts - 1) {\n                BOOST_REQUIRE_EQUAL(parts[p].size, max_len);\n            }\n            parts_total += parts[p].size;\n            bump_buffer(parts[p].iovecs);\n        }\n        BOOST_REQUIRE_EQUAL(parts_total, total);\n        check_buffer(total, 2);\n\n        if (parts.size() == 1) {\n            no_splits++;\n        }\n        if (parts.back().size == max_len) {\n            no_tails++;\n        }\n    } while (std::chrono::steady_clock::now() < stop || iter < 32 || no_splits < 16 || no_tails < 16);\n\n    seastar_logger.info(\"{} iters ({} no-splits, {} no-tails)\", iter, no_splits, no_tails);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tb_params) {\n    internal::disk_config_params disk_config(1);\n    internal::disk_params d;\n    size_t iops_req_size = 512;\n    size_t bw_req_size = 128 << 10; // 128k\n\n    // Test multipl datapoints starting at 1GB/s for read/write bandwidth\n    // and 15k req/s for read/write IOPS\n    for (uint64_t i = 1; i < 65; ++i) {\n        d.read_bytes_rate = i << 30;  // iGB/s\n        d.write_bytes_rate = i << 30;\n        d.read_req_rate = i * 15000;\n        d.write_req_rate = i * 15000;\n\n        auto io_config = disk_config.generate_config(d, 0, 1);\n        io_config.mountpoint = std::to_string(i);\n\n        io_queue_for_tests tio(io_config);\n\n        auto cost_read_512 = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::read_idx, iops_req_size));\n        auto cost_write_512 = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::write_idx, iops_req_size));\n\n        auto cost_read_128k = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::read_idx, bw_req_size));\n        auto cost_write_128k = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::write_idx, bw_req_size));\n        seastar_logger.info(\"{} read req/s, {} write req/s, {} read bytes/s, {} write bytes/s\",\n                            d.read_req_rate, d.write_req_rate, d.read_bytes_rate, d.write_bytes_rate);\n\n        seastar_logger.info(\"Read 512 cost: {}, Write 512 cost: {}\", cost_read_512, cost_write_512);\n        seastar_logger.info(\"Read 128k cost: {}, Write 128k cost: {}\", cost_read_128k, cost_write_128k);\n\n        const auto& fg = internal::get_throttler(tio.queue, internal::io_direction_and_length::write_idx);\n        const auto& tb = fg.token_bucket();\n        double rate = tb.rate();\n\n        auto fg_rate = std::chrono::duration<double, io_throttler::rate_resolution>(std::chrono::seconds(1)).count();\n        double iops_read = rate / cost_read_512 * fg_rate;\n        double iops_write = rate / cost_write_512 * fg_rate;\n        double bandwidth_read = rate / cost_read_128k * 131072 * fg_rate;\n        double bandwidth_write = rate / cost_write_128k * 131072 * fg_rate;\n        seastar_logger.info(\"IOPS read: {}, IOPS write: {}, Bandwidth read: {}, Bandwidth write: {}\",\n                            iops_read, iops_write, bandwidth_read, bandwidth_write);\n\n        float error_margin = 0.05f; // 5% error margin\n        BOOST_CHECK((d.read_req_rate - iops_read) / d.read_req_rate < error_margin);\n        BOOST_CHECK((d.write_req_rate - iops_write) / d.write_req_rate < error_margin);\n        BOOST_CHECK((d.read_bytes_rate - bandwidth_read) / d.read_bytes_rate < error_margin);\n        BOOST_CHECK((d.write_bytes_rate - bandwidth_write) / d.write_bytes_rate < error_margin);\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_unconfigured_io_queue) {\n    io_queue_for_tests tio;\n\n    BOOST_CHECK_EQUAL(tio.max_request_length(internal::io_direction_and_length::read_idx), tio.request_length_limit());\n    BOOST_CHECK_EQUAL(tio.max_request_length(internal::io_direction_and_length::write_idx), tio.request_length_limit());\n\n    for (uint64_t reqsize = 512; reqsize < 128 << 10; reqsize <<= 1) {\n        auto cost_read = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::read_idx, reqsize));\n        auto cost_write = tio.queue.request_capacity(internal::io_direction_and_length(internal::io_direction_and_length::write_idx, reqsize));\n\n        BOOST_CHECK_EQUAL(cost_read, 0);\n        BOOST_CHECK_EQUAL(cost_write, 0);\n    }\n}\n\nnamespace seastar::testing {\nclass fair_queue_test {\n    fair_queue& _fq;\npublic:\n    fair_queue_test(fair_queue& fq) noexcept : _fq(fq) {}\n\n    unsigned nr_children_for(std::optional<unsigned> index) {\n        return index.has_value() ? _fq._priority_groups[*index]->_nr_children : _fq._root._nr_children;\n    }\n\n    bool is_root_group(unsigned index) {\n        return _fq._priority_groups[index]->_parent == &_fq._root;\n    }\n\n    std::optional<unsigned> get_parent_index(internal::priority_class pc) {\n        auto& pe = reinterpret_cast<fair_queue::priority_entry&>(*_fq._priority_classes[pc.id()]);\n        if (pe._parent == &_fq._root) {\n            return {};\n        }\n\n        unsigned i = 0;\n        for (auto& pg : _fq._priority_groups) {\n            if (pe._parent == pg.get()) {\n                break;\n            }\n            i++;\n        }\n        return i;\n    }\n};\n}\n\nSEASTAR_THREAD_TEST_CASE(test_nested_priority_classes_basic_linkage) {\n    io_queue_for_tests tio;\n\n    auto ssg1 = create_scheduling_supergroup(100).get();\n    auto ssg2 = create_scheduling_supergroup(200).get();\n    auto sg0 = create_scheduling_group(\"a\", 300).get();\n    auto sg1 = create_scheduling_group(\"b\", \"b\", 400, ssg1).get();\n    auto sg2 = create_scheduling_group(\"c\", \"c\", 500, ssg2).get();\n    auto stop = defer([&] () noexcept {\n        destroy_scheduling_group(sg0).get();\n        destroy_scheduling_group(sg1).get();\n        destroy_scheduling_group(sg2).get();\n    });\n\n    tio.find_or_create_class(internal::priority_class(sg0));\n    tio.find_or_create_class(internal::priority_class(sg1));\n    tio.find_or_create_class(internal::priority_class(sg2));\n\n    seastar::testing::fair_queue_test fq(tio.get_fair_queue());\n\n    BOOST_CHECK_EQUAL(fq.nr_children_for({}), 3);\n    BOOST_CHECK_EQUAL(fq.nr_children_for(0), 1);\n    BOOST_CHECK_EQUAL(fq.nr_children_for(1), 1);\n\n    BOOST_CHECK(fq.is_root_group(0));\n    BOOST_CHECK(fq.is_root_group(1));\n\n    BOOST_CHECK(!fq.get_parent_index(internal::priority_class(sg0)));\n    BOOST_CHECK(fq.get_parent_index(internal::priority_class(sg1)) == 0);\n    BOOST_CHECK(fq.get_parent_index(internal::priority_class(sg2)) == 1);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_destroy_priority_class_with_requests) {\n    io_queue_for_tests tio;\n\n    auto sg = create_scheduling_group(\"a\", 100).get();\n    auto pc = internal::priority_class(sg);\n\n    auto fx = tio.queue_request(pc,\n        internal::io_direction_and_length(internal::io_direction_and_length::read_idx, 0),\n        internal::io_request::make_write(0, 0, nullptr, 1, false),\n        nullptr, {});\n\n    // Push this request through to make io-queue instantiate the priority class\n    tio.queue.poll_io_queue();\n    tio.sink.drain([] (const internal::io_request& rq, io_completion* desc) -> bool {\n        desc->complete_with(1);\n        return true;\n    });\n    BOOST_REQUIRE_EQUAL(fx.get(), 1);\n\n    BOOST_REQUIRE(tio.is_class_registered(pc));\n    tio.queue.destroy_priority_class(internal::priority_class(sg));\n    destroy_scheduling_group(sg).get();\n    BOOST_REQUIRE(!tio.is_class_registered(pc));\n}\n\nSEASTAR_THREAD_TEST_CASE(test_gauge_integrator_test) {\n    util::integrated_length<unsigned short, std::chrono::steady_clock> gi;\n    auto seed = std::random_device{}();\n    std::default_random_engine reng(seed);\n    std::uniform_int_distribution<> qlen(0, 250);\n    std::uniform_int_distribution<> dur(100, 1000);\n    auto now = std::chrono::steady_clock::now();\n    auto prev_value = gi.integral();\n\n    auto accumulate = [&] (std::function<unsigned short()> v) {\n        std::chrono::microseconds total_duration(0);\n        for (int i = 0; i < 13425; i++) {\n            auto d = std::chrono::microseconds(dur(reng));\n            total_duration += d;\n            now += d;\n            gi = v();\n            gi.checkpoint(now);\n        }\n        return std::chrono::duration_cast<std::chrono::seconds>(total_duration);\n    };\n\n    // step one -- check static value\n    for (int m = 0; m < 10; m++) {\n        auto value = qlen(reng);\n        auto seconds = accumulate([value] { return value; });\n        auto iv = gi.integral();\n        auto delta = iv - prev_value;\n        prev_value = iv;\n        fmt::print(\"value={:<6d} integral={:<10d} duration={:<4d} result={:<6d}\\n\",\n            value, delta, seconds.count(), delta / seconds.count()\n        );\n        BOOST_REQUIRE_GE(delta / seconds.count(), (unsigned short)(value * 0.9));\n        BOOST_REQUIRE_LE(delta / seconds.count(), (unsigned short)(value * 1.9));\n    }\n\n    // step two -- check disperse values\n    for (int m = 0; m < 16; m++) {\n        auto min_v = std::numeric_limits<unsigned short>::max();\n        auto max_v = std::numeric_limits<unsigned short>::min();\n        auto seconds = accumulate([&] {\n                auto v = qlen(reng);\n                min_v = std::min<unsigned short>(v, min_v);\n                max_v = std::max<unsigned short>(v, max_v);\n                return v;\n        });\n        auto iv = gi.integral();\n        auto delta = iv - prev_value;\n        prev_value = iv;\n        fmt::print(\"value=[{:d},{:d}] integral={:<10d} duration={:<4d} result={:<6d}\\n\",\n            min_v, max_v, delta, seconds.count(), delta / seconds.count()\n        );\n        BOOST_REQUIRE_GE(delta / seconds.count(), min_v);\n        BOOST_REQUIRE_LE(delta / seconds.count(), max_v);\n    }\n    fmt::print(\"done\\n\");\n}\n\nstatic future<size_t> run_and_check_bandwidth(io_queue_for_tests& tio, internal::priority_class pc, size_t bandwidth_goal, unsigned parallelizm = 1, size_t req_size = 128*1024) {\n    fmt::print(\"Run {} workload\\n\", pc.id());\n    bool keep_going = true;\n    uint64_t nr_requests = 0;\n\n    auto submitter = parallel_for_each(std::views::iota(0u, parallelizm), [&] (auto i) {\n        return do_until([&keep_going] { return !keep_going; }, [&] {\n            return tio.queue_request(pc,\n                internal::io_direction_and_length(internal::io_direction_and_length::read_idx, req_size),\n                internal::io_request::make_write(0, 0, nullptr, req_size, false),\n                nullptr, {}).then([&nr_requests] (auto size) {\n                    nr_requests++;\n                    return make_ready_future<>();\n                });\n        });\n    });\n\n\n    auto start = std::chrono::steady_clock::now();\n    auto stop = start + std::chrono::seconds(60);\n    size_t real_bandwidth = 0;\n\n    while (true) {\n        co_await seastar::sleep(std::chrono::seconds(1));\n        auto now = std::chrono::steady_clock::now();\n        real_bandwidth = (nr_requests * req_size) / std::chrono::duration_cast<std::chrono::seconds>(now - start).count();\n        fmt::print(\"Measured for {} {} MB/s, goal {} MB/s\\n\", pc.id(), real_bandwidth >> 20, bandwidth_goal >> 20);\n        if (real_bandwidth >= bandwidth_goal || now >= stop) {\n            break;\n        }\n    }\n\n    keep_going = false;\n    co_await std::move(submitter);\n    co_return real_bandwidth;\n}\n\nstruct background_drain {\n    io_queue_for_tests& tio;\n    bool keep_going;\n    future<> done;\n\n    future<> start_draining() {\n        return async([this] {\n            while (keep_going) {\n                tio.queue.poll_io_queue();\n                tio.sink.drain([] (const internal::io_request& rq, io_completion* desc) -> bool {\n                    const auto& op = rq.as<internal::io_request::operation::write>();\n                    desc->complete_with(op.size);\n                    return true;\n                });\n                maybe_yield().get();\n            }\n        });\n    }\n\n    background_drain(io_queue_for_tests& t)\n        : tio(t)\n        , keep_going(true)\n        , done(start_draining())\n    { }\n\n    future<> stop() {\n        fmt::print(\"Stop drainer\\n\");\n        keep_going = false;\n        return std::move(done);\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_class_bandwidth_throttler) {\n    io_queue_for_tests tio;\n\n    const size_t bandwidth = 100*1024*1024;\n\n    auto sg = create_scheduling_group(\"a\", 100).get();\n    auto pc = internal::priority_class(sg);\n    tio.queue.update_bandwidth_for_class(pc, bandwidth).get();\n\n    background_drain drain(tio);\n\n    auto bw = run_and_check_bandwidth(tio, pc, bandwidth * 0.9).get();\n    BOOST_REQUIRE_LE(bw, bandwidth * 1.15);\n\n    drain.stop().get();\n    destroy_scheduling_group(sg).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_class_group_bandwidth_throttler) {\n    io_queue_for_tests tio;\n\n    const size_t burst = 10*1024*1024;\n    const size_t bandwidth = 100*1024*1024;\n\n    auto ssg = create_scheduling_supergroup(100).get();\n    auto sg = create_scheduling_group(\"a\", \"a\", 100, ssg).get();\n    auto pc = internal::priority_class(sg);\n    tio.queue.update_bandwidth_for_class_group(ssg.index(), bandwidth).get();\n\n    background_drain drain(tio);\n\n    auto bw = run_and_check_bandwidth(tio, pc, bandwidth * 0.9).get();\n    BOOST_REQUIRE_LE(bw, bandwidth + burst);\n\n    drain.stop().get();\n    destroy_scheduling_group(sg).get();\n    destroy_scheduling_supergroup(ssg).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_2_class_group_bandwidth_throttler) {\n    io_queue_for_tests tio;\n\n    const size_t burst = 10*1024*1024;\n    const size_t bandwidth = 70*1024*1024;\n    const size_t group_bandwidth = 100*1024*1024;\n\n    auto ssg = create_scheduling_supergroup(100).get();\n    auto sg0 = create_scheduling_group(\"a\", \"a\", 100, ssg).get();\n    auto pc0 = internal::priority_class(sg0);\n    auto sg1 = create_scheduling_group(\"b\", \"b\", 100, ssg).get();\n    auto pc1 = internal::priority_class(sg1);\n\n    tio.queue.update_bandwidth_for_class(pc0, bandwidth).get();\n    tio.queue.update_bandwidth_for_class(pc1, bandwidth).get();\n    tio.queue.update_bandwidth_for_class_group(ssg.index(), group_bandwidth).get();\n\n    background_drain drain(tio);\n\n    // Set goal to be 40% of the maximum, as both classes will hit\n    // the group limit and won't reach their personal limits\n    auto f0 = run_and_check_bandwidth(tio, pc0, bandwidth * 0.4);\n    auto f1 = run_and_check_bandwidth(tio, pc1, bandwidth * 0.4);\n\n    auto bw0 = f0.get();\n    auto bw1 = f1.get();\n\n    // None of the classes must exceed its personal bandwidth\n    BOOST_REQUIRE_LE(bw0, bandwidth + burst);\n    BOOST_REQUIRE_LE(bw1, bandwidth + burst);\n    // Both classes must not exceed the group bandwidth\n    BOOST_REQUIRE_LE(bw0 + bw1, group_bandwidth + burst);\n\n    drain.stop().get();\n    destroy_scheduling_group(sg1).get();\n    destroy_scheduling_group(sg0).get();\n    destroy_scheduling_supergroup(ssg).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_2_class_group_bandwidth_throttler_1_unlimited) {\n    io_queue_for_tests tio;\n\n    const size_t burst = 10*1024*1024;\n    const size_t bandwidth = 40*1024*1024;\n    const size_t group_bandwidth = 100*1024*1024;\n\n    auto ssg = create_scheduling_supergroup(100).get();\n    auto sg0 = create_scheduling_group(\"a\", \"a\", 100, ssg).get();\n    auto pc0 = internal::priority_class(sg0);\n    auto sg1 = create_scheduling_group(\"b\", \"b\", 100, ssg).get();\n    auto pc1 = internal::priority_class(sg1);\n\n    tio.queue.update_bandwidth_for_class(pc0, bandwidth).get();\n    tio.queue.update_bandwidth_for_class_group(ssg.index(), group_bandwidth).get();\n\n    background_drain drain(tio);\n\n    // Set goal to be 40% of the maximum, as both classes will hit\n    // the group limit and won't reach their personal limits\n    auto f0 = run_and_check_bandwidth(tio, pc0, bandwidth * 0.4);\n    auto f1 = run_and_check_bandwidth(tio, pc1, group_bandwidth * 0.4);\n\n    auto bw0 = f0.get();\n    auto bw1 = f1.get();\n\n    // Limited class must not exceed its personal bandwidth\n    BOOST_REQUIRE_LE(bw0, bandwidth + burst);\n    // Both classes must not exceed the group bandwidth\n    BOOST_REQUIRE_LE(bw0 + bw1, group_bandwidth + burst);\n\n    drain.stop().get();\n    destroy_scheduling_group(sg1).get();\n    destroy_scheduling_group(sg0).get();\n    destroy_scheduling_supergroup(ssg).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_2_class_group_bandwidth_throttler_fair_shares) {\n    const size_t burst = 10*1024*1024;\n    const size_t bandwidth = 100*1024*1024;\n\n    io_queue::config cfg{0};\n    cfg.blocks_count_rate = io_queue::read_request_base_count * ((bandwidth * 10) >> io_queue::block_size_shift);\n    io_queue_for_tests tio(cfg);\n\n    auto ssg = create_scheduling_supergroup(200).get();\n    auto sg0 = create_scheduling_group(\"a\", \"a\", 400, ssg).get();\n    auto pc0 = internal::priority_class(sg0);\n    auto sg1 = create_scheduling_group(\"b\", \"b\", 100, ssg).get();\n    auto pc1 = internal::priority_class(sg1);\n\n    tio.queue.update_bandwidth_for_class_group(ssg.index(), bandwidth).get();\n\n    background_drain drain(tio);\n\n    auto f0 = run_and_check_bandwidth(tio, pc0, bandwidth * 0.8 * 0.95, 4);\n    auto f1 = run_and_check_bandwidth(tio, pc1, bandwidth * 0.2 * 0.95, 4);\n    auto bw0 = f0.get();\n    auto bw1 = f1.get();\n\n    // Check that shares are roughly respected\n    BOOST_REQUIRE_LE(float(bw0) / float(bw1), 4.05);\n    BOOST_REQUIRE_GE(float(bw0) / float(bw1), 3.95);\n    BOOST_REQUIRE_LE(bw0 + bw1, bandwidth + burst);\n\n    drain.stop().get();\n    destroy_scheduling_group(sg1).get();\n    destroy_scheduling_group(sg0).get();\n    destroy_scheduling_supergroup(ssg).get();\n}\n"
  },
  {
    "path": "tests/unit/ipv6_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/log.hh>\n\nusing namespace seastar;\n\nstatic logger iplog(\"ipv6\");\n\nstatic bool check_ipv6_support() {\n    if (!engine().net().supports_ipv6()) {\n        iplog.info(\"No IPV6 support detected. Skipping...\");\n        return false;\n    }\n    return true;\n}\n\nSEASTAR_TEST_CASE(udp_packet_test) {\n    if (!check_ipv6_support()) {\n        return make_ready_future<>();\n    }\n\n    auto sc = make_bound_datagram_channel(ipv6_addr{\"::1\"});\n\n    BOOST_REQUIRE(sc.local_address().addr().is_ipv6());\n\n    auto cc = make_bound_datagram_channel(ipv6_addr{\"::1\"});\n\n    auto f1 = cc.send(sc.local_address(), \"apa\");\n\n    return f1.then([cc = std::move(cc), sc = std::move(sc)]() mutable {\n        auto src = cc.local_address();\n        cc.close();\n        auto f2 = sc.receive();\n\n        return f2.then([sc = std::move(sc), src](auto pkt) mutable {\n            auto a = sc.local_address();\n            sc.close();\n            BOOST_REQUIRE_EQUAL(src, pkt.get_src());\n            auto dst = pkt.get_dst();\n            // Don't always get a dst address.\n            if (dst != socket_address()) {\n                BOOST_REQUIRE_EQUAL(a, pkt.get_dst());\n            }\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(tcp_packet_test) {\n    if (!check_ipv6_support()) {\n        return make_ready_future<>();\n    }\n\n    return async([] {\n        auto sc = server_socket(engine().net().listen(ipv6_addr{\"::1\"}, {}));\n        auto la = sc.local_address();\n\n        BOOST_REQUIRE(la.addr().is_ipv6());\n\n        auto cc = connect(la).get();\n        auto lc = std::move(sc.accept().get().connection);\n\n        auto strm = cc.output();\n        strm.write(\"los lobos\").get();\n        strm.flush().get();\n\n        auto in = lc.input();\n\n        using consumption_result_type = typename input_stream<char>::consumption_result_type;\n        using stop_consuming_type = typename consumption_result_type::stop_consuming_type;\n        using tmp_buf = stop_consuming_type::tmp_buf;\n\n        in.consume([](tmp_buf buf) {\n            return make_ready_future<consumption_result_type>(stop_consuming<char>({}));\n        }).get();\n\n        strm.close().get();\n        in.close().get();\n        sc.abort_accept();\n    });\n}\n\nSEASTAR_TEST_CASE(ipv6_equal_test) {\n    const uint16_t port{8080};\n    const uint16_t port2{8088};\n\n    const std::string str_addr1{\"abcd:fedc:ba98:7654:3210:0123:4567:89ab\"};\n    const std::string str_addr2{\"0123:4567:89ab:cdef:3210:0123:4567:89ab\"};\n    const std::string str_addr3{\"abcd:fedc:ba98:7654:3210:0123:4567:8900\"};\n\n    socket_address sock_addr1(ipv6_addr(str_addr1, port));\n    socket_address sock_addr2(ipv6_addr(str_addr2, port));\n    socket_address sock_addr3(ipv6_addr(str_addr1, port));\n\n    socket_address sock_addr4(ipv6_addr(str_addr3, port));\n    socket_address sock_addr5(ipv6_addr(str_addr1, port2));\n\n    BOOST_CHECK_NE(sock_addr1, sock_addr2);\n    BOOST_CHECK_EQUAL(sock_addr1, sock_addr3);\n    BOOST_CHECK_NE(sock_addr1, sock_addr4);\n    BOOST_CHECK_NE(sock_addr1, sock_addr5);\n\n    return make_ready_future();\n}\n\n"
  },
  {
    "path": "tests/unit/json2code_test.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n#\n# Copyright (C) 2024 Scylladb, Ltd.\n#\n\nimport argparse\nimport json\nimport subprocess\nimport sys\nimport unittest\nimport urllib.request\nimport urllib.parse\nfrom http import HTTPStatus\n\n\nclass TestJson2Code(unittest.TestCase):\n    rest_api_httpd = None\n    server = None\n    port = 15789\n\n    @classmethod\n    def setUpClass(cls):\n        args = [cls.rest_api_httpd, '--port', f'{cls.port}', '--smp=2']\n        cls.server = subprocess.Popen(args,\n                                      stdin=subprocess.PIPE,\n                                      stdout=subprocess.PIPE,\n                                      bufsize=0, text=True)\n        # wait until the server is ready for serve\n        cls.server.stdout.readline()\n\n    @classmethod\n    def tearDownClass(cls):\n        cls.server.terminate()\n\n    def test_path_params(self):\n        var1 = 'bon'\n        var2 = 'jour'\n        query_enum = 'VAL2'\n        response = self._do_query(var1, var2, query_enum)\n        self.assertEqual(response['var1'], f'/{var1}')\n        self.assertEqual(response['var2'], f'/{var2}')\n        self.assertEqual(response['enum_var'], query_enum)\n\n    def test_bad_enum(self):\n        var1 = 'bon'\n        var2 = 'jour'\n        query_enum = 'unknown value'\n        response = self._do_query(var1, var2, query_enum)\n        self.assertEqual(response['var1'], f'/{var1}')\n        self.assertEqual(response['var2'], f'/{var2}')\n        self.assertEqual(response['enum_var'], 'Unknown')\n\n    def test_missing_path_param(self):\n        query_enum = 'VAL2'\n        params = urllib.parse.urlencode({'query_enum': query_enum})\n        url = f'http://localhost:{self.port}/hello/world/?{params}'\n        with self.assertRaises(urllib.error.HTTPError) as e:\n            with urllib.request.urlopen(url):\n                pass\n        ex = e.exception\n        self.assertEqual(ex.code, HTTPStatus.NOT_FOUND)\n        response = json.loads(ex.read().decode('utf-8'))\n        self.assertEqual(response['message'], 'Not found')\n        self.assertEqual(response['code'], HTTPStatus.NOT_FOUND)\n\n    def test_arrays(self):\n        response = self._do_query('x', 'x', 'VAL2')\n        expected = [1, 2, 3]\n\n        self.assertEqual(response['array_var'], expected)\n        self.assertEqual(response['chunked_array_var'], expected)\n\n    def _do_query(self, var1: str, var2: str, query_enum: str):\n        def query(streaming: bool = False):\n            params = urllib.parse.urlencode({\n                'query_enum': query_enum,\n                'use_streaming': str(streaming).lower()\n            })\n            url = f'http://localhost:{self.port}/hello/world/{var1}/{var2}?{params}'\n            with urllib.request.urlopen(url) as f:\n                return json.load(f)\n\n        # always query both the streaming and not-streaming variations and\n        # ensure their output is the same\n        stream_no, stream_yes = query(False), query(True)\n        self.assertEqual(stream_no, stream_yes)\n        return stream_no\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--rest-api-httpd',\n                        required=True,\n                        help='Path of the rest_api_httpd executable')\n    opts, remaining = parser.parse_known_args()\n    remaining.insert(0, sys.argv[0])\n    TestJson2Code.rest_api_httpd = opts.rest_api_httpd\n    unittest.main(argv=remaining)\n"
  },
  {
    "path": "tests/unit/json_formatter_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2016 ScyllaDB.\n */\n#include <vector>\n\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/json/formatter.hh>\n#include <seastar/json/json_elements.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include \"memory-data-sink.hh\"\n\nusing namespace seastar;\nusing namespace json;\n\nstatic_assert(internal::is_string_like<std::string_view>);\nstatic_assert(internal::is_string_like<std::string>);\nstatic_assert(internal::is_string_like<sstring>);\n\nSEASTAR_TEST_CASE(test_simple_values) {\n    BOOST_CHECK_EQUAL(\"3\", formatter::to_json(3));\n    BOOST_CHECK_EQUAL(\"3\", formatter::to_json(3.0));\n    BOOST_CHECK_EQUAL(\"3.5\", formatter::to_json(3.5));\n    BOOST_CHECK_EQUAL(\"true\", formatter::to_json(true));\n    BOOST_CHECK_EQUAL(\"false\", formatter::to_json(false));\n\n    BOOST_CHECK_EQUAL(\"\\\"apa\\\"\", formatter::to_json(\"apa\")); // to_json(const char*)\n    BOOST_CHECK_EQUAL(\"\\\"apa\\\"\", formatter::to_json(sstring(\"apa\"))); // to_json(const sstring&)\n    BOOST_CHECK_EQUAL(\"\\\"apa\\\"\", formatter::to_json(\"apa\", 3)); // to_json(const char*, size_t)\n\n    using namespace std::string_literals;\n    sstring str = \"\\0 COWA\\bU\\nGA [{\\r}]\\x1a\"s,\n            expected = \"\\\"\\\\u0000 COWA\\\\bU\\\\nGA [{\\\\r}]\\\\u001A\\\"\"s;\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(str)); // to_json(const sstring&)\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(str.c_str(), str.size())); // to_json(const char*, size_t)\n\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_collections) {\n    BOOST_CHECK_EQUAL(\"{1:2,3:4}\", formatter::to_json(std::map<int,int>({{1,2},{3,4}})));\n    BOOST_CHECK_EQUAL(\"[1,2,3,4]\", formatter::to_json(std::vector<int>({1,2,3,4})));\n    BOOST_CHECK_EQUAL(\"[{1:2},{3:4}]\", formatter::to_json(std::vector<std::pair<int,int>>({{1,2},{3,4}})));\n    BOOST_CHECK_EQUAL(\"[{1:2},{3:4}]\", formatter::to_json(std::vector<std::map<int,int>>({{{1,2}},{{3,4}}})));\n    BOOST_CHECK_EQUAL(\"[[1,2],[3,4]]\", formatter::to_json(std::vector<std::vector<int>>({{1,2},{3,4}})));\n\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_ranges) {\n    BOOST_CHECK_EQUAL(\"[1,2,3,4]\", formatter::to_json(std::views::iota(1, 5)));\n#ifdef __cpp_lib_ranges_enumerate\n    BOOST_CHECK_EQUAL(\"[{0:5},{1:6},{2:7},{3:8}]\", formatter::to_json(std::views::iota(5, 9) | std::views::enumerate));\n#endif\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_strings) {\n    sstring s = \"hello, world\";\n    const char* expected = \"\\\"hello, world\\\"\";\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(s));\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(std::string(s)));\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(std::string_view(s)));\n    BOOST_CHECK_EQUAL(expected, formatter::to_json(s.c_str()));\n\n    return make_ready_future();\n}\n\nstruct object_json : public json_base {\n    json_element<sstring> subject;\n    json_list<long> values;\n\n    void register_params() {\n      add(&subject, \"subject\");\n      add(&values, \"values\");\n    }\n\n    object_json() { register_params(); }\n\n    object_json(const object_json &e) {\n      register_params();\n      subject = e.subject;\n      values = e.values;\n    }\n};\n\nSEASTAR_TEST_CASE(test_jsonable) {\n    object_json obj;\n    obj.subject = \"foo\";\n    obj.values.push(1);\n    obj.values.push(2);\n    obj.values.push(3);\n\n    BOOST_CHECK_EQUAL(\"{\\\"subject\\\": \\\"foo\\\", \\\"values\\\": [1,2,3]}\", formatter::to_json(obj));\n    return make_ready_future();\n}\n\ntemplate<typename F>\nvoid formatter_check_expected(sstring expected, F f, bool close = true) {\n    std::stringstream ss;\n    auto out = output_stream<char>(testing::memory_data_sink(ss), 8);\n\n    f(out);\n    if (close) {\n        out.close().get();\n    }\n\n    BOOST_CHECK_EQUAL(expected, ss.str());\n}\n\nSEASTAR_THREAD_TEST_CASE(test_stream_range_as_array_simple) {\n    sstring expected = R\"([\"1\", \"2\", \"3\"])\";\n    formatter_check_expected(expected, [] (auto& out) {\n        auto mapper = stream_range_as_array(std::vector<int>{1,2,3}, [] (auto i) {\n            return std::to_string(i);\n        });\n\n        mapper(std::move(out)).get();\n    }, false);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_stream_range_as_array) {\n    sstring expected = R\"([{\"subject\":\"1\",\"values\":[1]}, {\"subject\":\"2\",\"values\":[2]}, {\"subject\":\"3\",\"values\":[3]}])\";\n    formatter_check_expected(expected, [] (auto& out) {\n        auto mapper = stream_range_as_array(std::vector<int>{1,2,3}, [] (auto i) {\n            object_json obj;\n            obj.subject = std::to_string(i);\n            obj.values.push(i);\n            return obj;\n        });\n\n        mapper(std::move(out)).get();\n    }, false);\n}\n\nSEASTAR_THREAD_TEST_CASE(formatter_write) {\n\n    formatter_check_expected(\"3\", [] (auto &out) {\n        json::formatter::write(out, 3).get();\n    });\n    formatter_check_expected(\"false\", [] (auto &out) {\n        json::formatter::write(out, false).get();\n    });\n    formatter_check_expected(\"\\\"foo\\\"\", [] (auto &out) {\n        json::formatter::write(out, \"foo\").get();\n    });\n\n    formatter_check_expected(\"{1:2,3:4}\", [] (auto& out) {\n        json::formatter::write(out, std::map<int, int>({{1, 2}, {3, 4}})).get();\n    });\n    formatter_check_expected(\"{3:4,1:2}\", [] (auto& out) {\n        json::formatter::write(out, std::unordered_map<int, int>({{1, 2}, {3, 4}})).get();\n    });\n    formatter_check_expected(\"[1,2,3,4]\", [] (auto &out) {\n        json::formatter::write(out, std::vector<int>({1, 2, 3, 4})).get();\n    });\n\n    formatter_check_expected(\"[{1:2},{3:4}]\", [] (auto &out) {\n        json::formatter::write(out, std::vector<std::pair<int, int>>({{1, 2}, {3, 4}})).get();\n    });\n    formatter_check_expected(\"[{1:2},{3:4}]\", [] (auto &out) {\n        json::formatter::write(out, std::vector<std::map<int, int>>({{{1, 2}}, {{3, 4}}})).get();\n    });\n    formatter_check_expected(\"[[1,2],[3,4]]\", [] (auto &out) {\n        json::formatter::write(out, std::vector<std::vector<int>>({{1, 2}, {3, 4}})).get();\n    });\n    formatter_check_expected(\"[1,2,3,4]\", [] (auto& out) {\n        json::formatter::write(out, std::views::iota(1, 5)).get();\n    });\n#ifdef __cpp_lib_ranges_enumerate\n    formatter_check_expected(\"[{0:5},{1:6},{2:7},{3:8}]\", [] (auto& out) {\n        json::formatter::write(out, std::views::iota(5, 9) | std::views::enumerate).get();\n    });\n#endif\n}\n"
  },
  {
    "path": "tests/unit/libc_wrapper_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2024 ScyllaDB.\n */\n#include <coroutine>\n#include <iostream>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n\n#include <seastar/testing/test_case.hh>\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(getgrnam_group_name_does_not_exist_test) {\n    // A better approach would be to use a test setup and teardown to create a fake group\n    // with members in it and in the teardown delete the fake group.\n    std::optional<struct group_details> grp = co_await getgrnam(\"roo\");\n    BOOST_REQUIRE(!grp.has_value());\n}\n\nSEASTAR_TEST_CASE(getgrnam_group_name_exists_test) {\n    std::optional<struct group_details> grp = co_await getgrnam(\"root\");\n    BOOST_REQUIRE(grp.has_value());\n    BOOST_REQUIRE_EQUAL(grp.value().group_name.c_str(), \"root\");\n}"
  },
  {
    "path": "tests/unit/linux_perf_event_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#define BOOST_TEST_MODULE linux_perf_event\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/testing/linux_perf_event.hh>\n\nBOOST_AUTO_TEST_CASE(test_always_zero_reads_zero) {\n    auto event = linux_perf_event::always_zero();\n    BOOST_CHECK_EQUAL(event.read(), 0u);\n    BOOST_CHECK_EQUAL(event.read(), 0u);\n}\n\nBOOST_AUTO_TEST_CASE(test_always_zero_enable_disable) {\n    auto event = linux_perf_event::always_zero();\n    // These should be no-ops and not crash\n    event.enable();\n    event.disable();\n    BOOST_CHECK_EQUAL(event.read(), 0u);\n}\n\nBOOST_AUTO_TEST_CASE(test_always_zero_move) {\n    auto event = linux_perf_event::always_zero();\n    auto event2 = std::move(event);\n    BOOST_CHECK_EQUAL(event2.read(), 0u);\n\n    linux_perf_event event3 = linux_perf_event::always_zero();\n    event3 = std::move(event2);\n    BOOST_CHECK_EQUAL(event3.read(), 0u);\n}\n"
  },
  {
    "path": "tests/unit/locking_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <chrono>\n#include <exception>\n#include <ranges>\n#include <stdexcept>\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/coroutine/parallel_for_each.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/rwlock.hh>\n#include <seastar/core/shared_mutex.hh>\n#include <seastar/util/alloc_failure_injector.hh>\n#include <seastar/util/log.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_THREAD_TEST_CASE(test_rwlock) {\n    rwlock l;\n\n    l.for_write().lock().get();\n    BOOST_REQUIRE(!l.try_write_lock());\n    BOOST_REQUIRE(!l.try_read_lock());\n    l.for_write().unlock();\n\n    l.for_read().lock().get();\n    BOOST_REQUIRE(!l.try_write_lock());\n    BOOST_REQUIRE(l.try_read_lock());\n    l.for_read().lock().get();\n    l.for_read().unlock();\n    l.for_read().unlock();\n    l.for_read().unlock();\n\n    BOOST_REQUIRE(l.try_write_lock());\n    l.for_write().unlock();\n}\n\nSEASTAR_TEST_CASE(test_with_lock_mutable) {\n    return do_with(rwlock(), [](rwlock& l) {\n        return with_lock(l.for_read(), [p = std::make_unique<int>(42)] () mutable {});\n    });\n}\n\nSEASTAR_TEST_CASE(test_rwlock_exclusive) {\n    return do_with(rwlock(), unsigned(0), [] (rwlock& l, unsigned& counter) {\n        return parallel_for_each(std::views::iota(0, 10), [&l, &counter] (int idx) {\n            return with_lock(l.for_write(), [&counter] {\n                BOOST_REQUIRE_EQUAL(counter, 0u);\n                ++counter;\n                return sleep(1ms).then([&counter] {\n                    --counter;\n                    BOOST_REQUIRE_EQUAL(counter, 0u);\n                });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_rwlock_shared) {\n    return do_with(rwlock(), unsigned(0), unsigned(0), [] (rwlock& l, unsigned& counter, unsigned& max) {\n        return parallel_for_each(std::views::iota(0, 10), [&l, &counter, &max] (int idx) {\n            return with_lock(l.for_read(), [&counter, &max] {\n                ++counter;\n                max = std::max(max, counter);\n                return sleep(1ms).then([&counter] {\n                    --counter;\n                });\n            });\n        }).finally([&counter, &max] {\n            BOOST_REQUIRE_EQUAL(counter, 0u);\n            BOOST_REQUIRE_NE(max, 0u);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rwlock_failed_func) {\n    rwlock l;\n\n    // verify that the rwlock is unlocked when func fails\n    future<> fut = with_lock(l.for_read(), [] {\n        throw std::runtime_error(\"injected\");\n    });\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    fut = with_lock(l.for_write(), [] {\n        throw std::runtime_error(\"injected\");\n    });\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    BOOST_REQUIRE(l.try_write_lock());\n    l.for_write().unlock();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rwlock_abort) {\n    rwlock l;\n\n    l.write_lock().get();\n\n    {\n        abort_source as;\n        auto f = l.write_lock(as);\n        BOOST_REQUIRE(!f.available());\n\n        (void)sleep(1ms).then([&as] {\n            as.request_abort();\n        });\n\n        BOOST_REQUIRE_THROW(f.get(), semaphore_aborted);\n    }\n\n    {\n        abort_source as;\n        auto f = l.read_lock(as);\n        BOOST_REQUIRE(!f.available());\n\n        (void)sleep(1ms).then([&as] {\n            as.request_abort();\n        });\n\n        BOOST_REQUIRE_THROW(f.get(), semaphore_aborted);\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rwlock_hold_abort) {\n    rwlock l;\n\n    auto wh = l.hold_write_lock().get();\n\n    {\n        abort_source as;\n        auto f = l.hold_write_lock(as);\n        BOOST_REQUIRE(!f.available());\n\n        (void)sleep(1ms).then([&as] {\n            as.request_abort();\n        });\n\n        BOOST_REQUIRE_THROW(f.get(), semaphore_aborted);\n    }\n\n    {\n        abort_source as;\n        auto f = l.hold_read_lock(as);\n        BOOST_REQUIRE(!f.available());\n\n        (void)sleep(1ms).then([&as] {\n            as.request_abort();\n        });\n\n        BOOST_REQUIRE_THROW(f.get(), semaphore_aborted);\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rwlock_hold) {\n    rwlock l;\n\n    auto rl = l.hold_read_lock().get();\n\n    auto opt_rl = l.try_hold_read_lock();\n    BOOST_REQUIRE(opt_rl.has_value());\n    BOOST_REQUIRE(!l.try_hold_write_lock());\n\n    rl.return_all();\n    BOOST_REQUIRE(!l.try_hold_write_lock());\n\n    opt_rl->return_all();\n    auto opt_wl = l.try_hold_write_lock();\n    BOOST_REQUIRE(opt_wl.has_value());\n\n    BOOST_REQUIRE(!l.try_hold_read_lock());\n    BOOST_REQUIRE(!l.try_hold_write_lock());\n}\n\nSEASTAR_THREAD_TEST_CASE(test_failed_with_lock) {\n    struct test_lock {\n        future<> lock() noexcept {\n            return make_exception_future<>(std::runtime_error(\"injected\"));\n        }\n        void unlock() noexcept {\n            BOOST_REQUIRE(false);\n        }\n    };\n\n    test_lock l;\n\n    // if l.lock() fails neither the function nor l.unlock()\n    // should be called.\n    BOOST_REQUIRE_THROW(with_lock(l, [] {\n        BOOST_REQUIRE(false);\n    }).get(), std::runtime_error);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_mutex) {\n    shared_mutex sm;\n\n    sm.lock().get();\n    BOOST_REQUIRE(!sm.try_lock());\n    BOOST_REQUIRE(!sm.try_lock_shared());\n    sm.unlock();\n\n    sm.lock_shared().get();\n    BOOST_REQUIRE(!sm.try_lock());\n    BOOST_REQUIRE(sm.try_lock_shared());\n    sm.lock_shared().get();\n    sm.unlock_shared();\n    sm.unlock_shared();\n    sm.unlock_shared();\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_exclusive) {\n    return do_with(shared_mutex(), unsigned(0), [] (shared_mutex& sm, unsigned& counter) {\n        return parallel_for_each(std::views::iota(0, 10), [&sm, &counter] (int idx) {\n            return with_lock(sm, [&counter] {\n                BOOST_REQUIRE_EQUAL(counter, 0u);\n                ++counter;\n                return sleep(1ms).then([&counter] {\n                    --counter;\n                    BOOST_REQUIRE_EQUAL(counter, 0u);\n                });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_shared) {\n    return do_with(shared_mutex(), unsigned(0), unsigned(0), [] (shared_mutex& sm, unsigned& counter, unsigned& max) {\n        return parallel_for_each(std::views::iota(0, 10), [&sm, &counter, &max] (int idx) {\n            return with_shared(sm, [&counter, &max] {\n                ++counter;\n                max = std::max(max, counter);\n                return sleep(1ms).then([&counter] {\n                    --counter;\n                });\n            });\n        }).finally([&counter, &max] {\n            BOOST_REQUIRE_EQUAL(counter, 0u);\n            BOOST_REQUIRE_NE(max, 0u);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_mutex_failed_func) {\n    shared_mutex sm;\n\n    // verify that the shared_mutex is unlocked when func fails\n    future<> fut = with_shared(sm, [] {\n        throw std::runtime_error(\"injected\");\n    });\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    fut = with_lock(sm, [] {\n        throw std::runtime_error(\"injected\");\n    });\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_mutex_throwing_func) {\n    shared_mutex sm;\n    struct X {\n        int x;\n        X(int x_) noexcept : x(x_) {};\n        X(X&& o) : x(o.x) {\n            throw std::runtime_error(\"X moved\");\n        }\n    };\n\n    // verify that the shared_mutex is unlocked when func move fails\n    future<> fut = with_shared(sm, [x = X(0)] {});\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    fut = with_lock(sm, [x = X(0)] {});\n    BOOST_REQUIRE_THROW(fut.get(), std::runtime_error);\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_shared_mutex_failed_lock) {\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n    shared_mutex sm;\n\n    // if l.lock() fails neither the function nor l.unlock()\n    // should be called.\n    sm.lock().get();\n    seastar::memory::local_failure_injector().fail_after(0);\n    BOOST_REQUIRE_THROW(with_shared(sm, [] {\n        BOOST_REQUIRE(false);\n    }).get(), std::bad_alloc);\n\n    seastar::memory::local_failure_injector().fail_after(0);\n    BOOST_REQUIRE_THROW(with_lock(sm, [] {\n        BOOST_REQUIRE(false);\n    }).get(), std::bad_alloc);\n    sm.unlock();\n\n    seastar::memory::local_failure_injector().cancel();\n#endif // SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\n}\n\nstruct expected_exception : public std::exception {\n    int value;\n    expected_exception(int v) noexcept : value(v) {}\n};\n\nstruct moved_exception : public std::exception {\n    int count;\n    moved_exception(int c) noexcept : count(c) {}\n};\n\nstruct throw_on_move {\n    int value;\n    int delay;\n    int count = 0;\n\n    throw_on_move(int v, int d = 0) noexcept : value(v), delay(d) {}\n    throw_on_move(const throw_on_move& o) = default;\n    throw_on_move(throw_on_move&& o)\n        : value(o.value)\n        , delay(o.delay)\n        , count(o.count + 1)\n    {\n        if (count >= delay) {\n            throw moved_exception(count);\n        }\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_with_shared_typed_return_nothrow_move_func) {\n    shared_mutex sm;\n\n    auto expected = 42;\n    auto res = with_shared(sm, [expected] {\n        return expected;\n    }).get();\n    BOOST_REQUIRE_EQUAL(res, expected);\n\n    try {\n        with_shared(sm, [expected] {\n            if (expected == 42) {\n                throw expected_exception(expected);\n            }\n            return expected;\n        }).get();\n        BOOST_FAIL(\"No exception was thrown\");\n    } catch (const expected_exception& e) {\n        BOOST_REQUIRE_EQUAL(e.value, expected);\n    } catch (const std::exception& e) {\n        BOOST_FAIL(format(\"Unexpected exception type: {}\", e.what()));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_with_shared_typed_return_throwing_move_func) {\n    shared_mutex sm;\n\n    int expected_value = 42;\n    bool done = false;\n    for (int move_delay = 0; !done; move_delay++) {\n        try {\n            auto res = with_shared(sm, [exp = throw_on_move(expected_value, move_delay)] {\n                auto expected = std::move(exp);\n                return expected.value;\n            }).get();\n            BOOST_REQUIRE_EQUAL(res, expected_value);\n            done = true;\n        } catch (const moved_exception& e) {\n        } catch (const std::exception& e) {\n            BOOST_FAIL(format(\"Unexpected exception type: {}\", e.what()));\n        }\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_with_lock_typed_return_nothrow_move_func) {\n    shared_mutex sm;\n\n    auto expected = 42;\n    auto res = with_lock(sm, [expected] {\n        return expected;\n    }).get();\n    BOOST_REQUIRE_EQUAL(res, expected);\n\n    try {\n        with_lock(sm, [expected] {\n            if (expected == 42) {\n                throw expected_exception(expected);\n            }\n            return expected;\n        }).get();\n        BOOST_FAIL(\"No exception was thrown\");\n    } catch (const expected_exception& e) {\n        BOOST_REQUIRE_EQUAL(e.value, expected);\n    } catch (const std::exception& e) {\n        BOOST_FAIL(format(\"Unexpected exception type: {}\", e.what()));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_with_lock_typed_return_throwing_move_func) {\n    shared_mutex sm;\n\n    int expected_value = 42;\n    bool done = false;\n    for (int move_delay = 0; !done; move_delay++) {\n        try {\n            auto res = with_lock(sm, [exp = throw_on_move(expected_value, move_delay)] {\n                auto expected = std::move(exp);\n                return expected.value;\n            }).get();\n            BOOST_REQUIRE_EQUAL(res, expected_value);\n            done = true;\n        } catch (const moved_exception& e) {\n        } catch (const std::exception& e) {\n            BOOST_FAIL(format(\"Unexpected exception type: {}\", e.what()));\n        }\n    }\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_locks) {\n    shared_mutex sm;\n\n    {\n        const auto ulock = co_await get_unique_lock(sm);\n        BOOST_REQUIRE(!sm.try_lock());\n        BOOST_REQUIRE(!sm.try_lock_shared());\n    }\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n\n    {\n        const auto slock1 = co_await get_shared_lock(sm);\n        BOOST_REQUIRE(!sm.try_lock());\n        BOOST_REQUIRE(sm.try_lock_shared());\n\n        const auto slock2 = co_await get_shared_lock(sm);\n\n        // This balances out the call to `try_lock_shared()` above.\n        sm.unlock_shared();\n    }\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_exclusive_locks) {\n    shared_mutex sm{};\n    unsigned counter = 0;\n\n    co_await coroutine::parallel_for_each(std::views::iota(0, 10), coroutine::lambda([&sm, &counter] (auto&&) -> future<> {\n        const auto ulock = co_await get_unique_lock(sm);\n        BOOST_REQUIRE_EQUAL(counter, 0u);\n        ++counter;\n        co_await sleep(1ms);\n        --counter;\n        BOOST_REQUIRE_EQUAL(counter, 0u);\n    }));\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_exception_locks) {\n    shared_mutex sm;\n\n    // Verify that the shared_mutex is unlocked when an exception is thrown.\n    try {\n        const auto slock = co_await get_shared_lock(sm);\n        throw std::runtime_error(\"injected\");\n    } catch (const std::runtime_error&) {\n        BOOST_REQUIRE(sm.try_lock());\n        sm.unlock();\n    } catch (...) {\n        BOOST_FAIL(format(\"Unexpected exception type: {}\", std::current_exception()));\n    }\n\n    try {\n        const auto ulock = co_await get_unique_lock(sm);\n        throw std::runtime_error(\"injected\");\n    } catch (const std::runtime_error&) {\n        BOOST_REQUIRE(sm.try_lock());\n        sm.unlock();\n    } catch (...) {\n        BOOST_FAIL(format(\"Unexpected exception type: {}\", std::current_exception()));\n    }\n\n    BOOST_REQUIRE(sm.try_lock());\n    sm.unlock();\n}\n"
  },
  {
    "path": "tests/unit/log_buf_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2020 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/log.hh>\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(log_buf_realloc) {\n    std::array<char, 128> external_buf;\n\n    const auto external_buf_ptr = reinterpret_cast<uintptr_t>(external_buf.data());\n\n    internal::log_buf b(external_buf.data(), external_buf.size());\n\n    BOOST_REQUIRE_EQUAL(reinterpret_cast<uintptr_t>(b.data()), external_buf_ptr);\n\n    auto it = b.back_insert_begin();\n\n    for (auto i = 0; i < 128; ++i) {\n        *it++ = 'a';\n    }\n\n    *it = 'a'; // should trigger realloc\n\n    BOOST_REQUIRE_NE(reinterpret_cast<uintptr_t>(b.data()), reinterpret_cast<uintptr_t>(external_buf.data()));\n\n    const char* p = b.data();\n    for (auto i = 0; i < 129; ++i) {\n        BOOST_REQUIRE_EQUAL(p[i], 'a');\n    }\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(log_buf_insert_iterator_format_to) {\n    constexpr size_t size = 128;\n    auto external_buf = std::make_unique<char[]>(size);\n    auto external_buf_ptr = external_buf.get();\n    char str[size + 1];\n\n    internal::log_buf b(external_buf_ptr, size);\n\n    BOOST_REQUIRE_EQUAL(reinterpret_cast<uintptr_t>(b.data()), reinterpret_cast<uintptr_t>(external_buf_ptr));\n\n    auto it = b.back_insert_begin();\n\n    memset(str, 'a', size);\n    str[size] = '\\0';\n\n    it = fmt::format_to(it, fmt::runtime(str), size);\n    BOOST_REQUIRE_EQUAL(reinterpret_cast<uintptr_t>(b.data()), reinterpret_cast<uintptr_t>(external_buf_ptr));\n\n    *it++ = '\\n';\n    BOOST_REQUIRE_NE(reinterpret_cast<uintptr_t>(b.data()), reinterpret_cast<uintptr_t>(external_buf_ptr));\n\n    memset(str, 'b', size);\n    it = fmt::format_to(it, fmt::runtime(str), size);\n    *it++ = '\\n';\n\n    const char* p = b.data();\n    size_t pos = 0;\n    for (size_t i = 0; i < size; i++) {\n        BOOST_REQUIRE_EQUAL(p[pos++], 'a');\n    }\n    BOOST_REQUIRE_EQUAL(p[pos++], '\\n');\n    for (size_t i = 0; i < size; i++) {\n        BOOST_REQUIRE_EQUAL(p[pos++], 'b');\n    }\n    BOOST_REQUIRE_EQUAL(p[pos++], '\\n');\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(log_buf_clear) {\n    internal::log_buf buf;\n\n    auto it = buf.back_insert_begin();\n\n    fmt::format_to(it, \"abcd\");\n\n    BOOST_CHECK_EQUAL(buf.view(), \"abcd\");\n    auto cap_before = buf.capacity();\n    buf.clear();\n    BOOST_CHECK_EQUAL(cap_before, buf.capacity());\n    BOOST_CHECK_EQUAL(0, buf.size());\n\n    fmt::format_to(it, \"uuvvwwxxyyzz\");\n\n    BOOST_CHECK_EQUAL(buf.view(), \"uuvvwwxxyyzz\");\n    cap_before = buf.capacity();\n    buf.clear();\n    BOOST_CHECK_EQUAL(cap_before, buf.capacity());\n    BOOST_CHECK_EQUAL(0, buf.size());\n\n    return make_ready_future<>();\n}\n"
  },
  {
    "path": "tests/unit/loopback_socket.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#pragma once\n\n#include <ranges>\n#include <system_error>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/circular_buffer.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/net/stack.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nstruct loopback_error_injector {\n    enum class error { none, one_shot, abort };\n    virtual ~loopback_error_injector() {};\n    virtual error server_rcv_error() { return error::none; }\n    virtual error server_snd_error() { return error::none; }\n    virtual error client_rcv_error() { return error::none; }\n    virtual error client_snd_error() { return error::none; }\n    virtual error connect_error()    { return error::none; }\n    virtual std::chrono::microseconds connect_delay() { return std::chrono::microseconds(0); }\n};\n\nclass loopback_buffer {\npublic:\n    enum class type : uint8_t {\n        CLIENT_TX,\n        SERVER_TX\n    };\nprivate:\n    bool _aborted = false;\n    queue<temporary_buffer<char>> _q{128};\n    loopback_error_injector* _error_injector;\n    type _type;\n    std::optional<promise<>> _shutdown;\npublic:\n    loopback_buffer(loopback_error_injector* error_injection, type t) : _error_injector(error_injection), _type(t) {}\n    future<> push(temporary_buffer<char>&& b) {\n        if (_aborted) {\n            return make_exception_future<>(std::system_error(EPIPE, std::system_category()));\n        }\n        if (_error_injector) {\n            auto error = _type == type::CLIENT_TX ? _error_injector->client_snd_error() : _error_injector->server_snd_error();\n            if (error == loopback_error_injector::error::one_shot) {\n                return make_exception_future<>(std::runtime_error(\"test injected glitch on send\"));\n            }\n            if (error == loopback_error_injector::error::abort) {\n                abort();\n                return make_exception_future<>(std::runtime_error(\"test injected error on send\"));\n            }\n        }\n        return _q.push_eventually(std::move(b));\n    }\n    future<temporary_buffer<char>> pop() {\n        if (_aborted) {\n            return make_exception_future<temporary_buffer<char>>(std::system_error(EPIPE, std::system_category()));\n        }\n        if (_error_injector) {\n            auto error = _type == type::CLIENT_TX ? _error_injector->client_rcv_error() : _error_injector->server_rcv_error();\n            if (error == loopback_error_injector::error::one_shot) {\n                return make_exception_future<temporary_buffer<char>>(std::runtime_error(\"test injected glitch on receive\"));\n            }\n            if (error == loopback_error_injector::error::abort) {\n                abort();\n                return make_exception_future<temporary_buffer<char>>(std::runtime_error(\"test injected error on receive\"));\n            }\n        }\n        return _q.pop_eventually();\n    }\n    void abort() noexcept {\n        shutdown();\n        _aborted = true;\n        _q.abort(std::make_exception_ptr(std::system_error(EPIPE, std::system_category())));\n    }\n    void shutdown() noexcept {\n        // it can be called by both -- reader and writer socket impls\n        if (_shutdown.has_value()) {\n            _shutdown->set_value();\n            _shutdown.reset();\n        }\n    }\n\n    future<> wait_input_shutdown() {\n        SEASTAR_ASSERT(!_shutdown.has_value());\n        _shutdown.emplace();\n        return _shutdown->get_future();\n    }\n};\n\nclass loopback_data_sink_impl : public data_sink_impl {\n    lw_shared_ptr<foreign_ptr<lw_shared_ptr<loopback_buffer>>> _buffer;\n    noncopyable_function<void()> _batch_flush_error;\npublic:\n    explicit loopback_data_sink_impl(lw_shared_ptr<foreign_ptr<lw_shared_ptr<loopback_buffer>>> buffer, noncopyable_function<void()> flush_error)\n            : _buffer(buffer)\n            , _batch_flush_error(std::move(flush_error))\n    {\n    }\nprivate:\n    future<> put(std::vector<temporary_buffer<char>> bufs) {\n        return do_with(std::move(bufs), [this] (std::vector<temporary_buffer<char>>& bufs) {\n            return do_for_each(bufs, [this] (temporary_buffer<char>& buf) {\n                return smp::submit_to(_buffer->get_owner_shard(), [this, b = buf.get(), s = buf.size()] {\n                    return (*_buffer)->push(temporary_buffer<char>(b, s));\n                });\n            });\n        });\n    }\npublic:\n#if SEASTAR_API_LEVEL >= 9\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        std::vector<temporary_buffer<char>> stable_bufs(std::make_move_iterator(bufs.begin()), std::make_move_iterator(bufs.end()));\n        return put(std::move(stable_bufs));\n    }\n#else\n    future<> put(net::packet data) override {\n        return put(data.release());\n    }\n#endif\n    future<> close() override {\n        return smp::submit_to(_buffer->get_owner_shard(), [this] {\n            return (*_buffer)->push({}).handle_exception_type([] (std::system_error& err) {\n                if (err.code().value() != EPIPE) {\n                    throw err;\n                }\n            });\n        });\n    }\n\n    bool can_batch_flushes() const noexcept override { return true; }\n    void on_batch_flush_error() noexcept override { _batch_flush_error(); }\n};\n\nclass loopback_data_source_impl : public data_source_impl {\n    bool _eof = false;\n    lw_shared_ptr<loopback_buffer> _buffer;\npublic:\n    explicit loopback_data_source_impl(lw_shared_ptr<loopback_buffer> buffer)\n            : _buffer(std::move(buffer)) {\n    }\n    future<temporary_buffer<char>> get() override {\n        return _buffer->pop().then_wrapped([this] (future<temporary_buffer<char>>&& b) {\n            _eof = b.failed();\n            if (!_eof) {\n                // future::get() is destructive, so we have to play these games\n                // FIXME: make future::get() non-destructive\n                auto&& tmp = b.get();\n                _eof = tmp.empty();\n                b = make_ready_future<temporary_buffer<char>>(std::move(tmp));\n            }\n            return std::move(b);\n        });\n    }\n    future<> close() override {\n        if (!_eof) {\n            _buffer->abort();\n        } else {\n            _buffer->shutdown();\n        }\n        return make_ready_future<>();\n    }\n};\n\n\nclass loopback_connected_socket_impl : public net::connected_socket_impl {\n    lw_shared_ptr<foreign_ptr<lw_shared_ptr<loopback_buffer>>> _tx;\n    lw_shared_ptr<loopback_buffer> _rx;\npublic:\n    loopback_connected_socket_impl(foreign_ptr<lw_shared_ptr<loopback_buffer>> tx, lw_shared_ptr<loopback_buffer> rx)\n            : _tx(make_lw_shared(std::move(tx))), _rx(std::move(rx)) {\n    }\n    data_source source() override {\n        return data_source(std::make_unique<loopback_data_source_impl>(_rx));\n    }\n    data_sink sink() override {\n        return data_sink(std::make_unique<loopback_data_sink_impl>(_tx, [this] { shutdown_input(); }));\n    }\n    void shutdown_input() override {\n        _rx->abort();\n    }\n    void shutdown_output() override {\n        (void)smp::submit_to(_tx->get_owner_shard(), [tx = _tx] {\n            (*tx)->abort();\n        });\n    }\n    void set_nodelay(bool nodelay) override {\n    }\n    bool get_nodelay() const override {\n        return true;\n    }\n    void set_keepalive(bool keepalive) override {}\n    bool get_keepalive() const override {\n        return false;\n    }\n    void set_keepalive_parameters(const net::keepalive_params&) override {}\n    net::keepalive_params get_keepalive_parameters() const override {\n        return net::tcp_keepalive_params {std::chrono::seconds(0), std::chrono::seconds(0), 0};\n    }\n    void set_sockopt(int level, int optname, const void* data, size_t len) override {\n        throw std::runtime_error(\"Setting custom socket options is not supported for loopback\");\n    }\n    int get_sockopt(int level, int optname, void* data, size_t len) const override {\n        throw std::runtime_error(\"Getting custom socket options is not supported for loopback\");\n    }\n    socket_address local_address() const noexcept override {\n        // dummy\n        return {};\n    }\n    socket_address remote_address() const noexcept override {\n        // dummy\n        return {};\n    }\n    future<> wait_input_shutdown() override {\n        return _rx->wait_input_shutdown();\n    }\n};\n\nclass loopback_server_socket_impl : public net::server_socket_impl {\n    lw_shared_ptr<queue<connected_socket>> _pending;\npublic:\n    explicit loopback_server_socket_impl(lw_shared_ptr<queue<connected_socket>> q)\n            : _pending(std::move(q)) {\n    }\n    future<accept_result> accept() override {\n        return _pending->pop_eventually().then([] (connected_socket&& cs) {\n            return make_ready_future<accept_result>(accept_result{std::move(cs), socket_address()});\n        });\n    }\n    void abort_accept() override {\n        _pending->abort(std::make_exception_ptr(std::system_error(ECONNABORTED, std::system_category())));\n    }\n    socket_address local_address() const override {\n        // CMH dummy\n        return {};\n    }\n};\n\n\nclass loopback_connection_factory {\n    unsigned _shard = 0;\n    unsigned _shards_count;\n    unsigned _pending_capacity = 10;\n    std::vector<lw_shared_ptr<queue<connected_socket>>> _pending;\npublic:\n    explicit loopback_connection_factory(unsigned shards_count = smp::count)\n            : _shards_count(shards_count)\n    {\n        _pending.resize(shards_count);\n    }\n\n    static loopback_connection_factory with_pending_capacity(unsigned pending_capacity, unsigned shards_count = smp::count) {\n        auto lcf = loopback_connection_factory(shards_count);\n        lcf._pending_capacity = pending_capacity;\n        return lcf;\n    }\n\n    server_socket get_server_socket() {\n       SEASTAR_ASSERT(this_shard_id() < _shards_count);\n       if (!_pending[this_shard_id()]) {\n           _pending[this_shard_id()] = make_lw_shared<queue<connected_socket>>(_pending_capacity);\n       }\n       return server_socket(std::make_unique<loopback_server_socket_impl>(_pending[this_shard_id()]));\n    }\n    future<> make_new_server_connection(foreign_ptr<lw_shared_ptr<loopback_buffer>> b1, lw_shared_ptr<loopback_buffer> b2) {\n        SEASTAR_ASSERT(this_shard_id() < _shards_count);\n        if (!_pending[this_shard_id()]) {\n            _pending[this_shard_id()] = make_lw_shared<queue<connected_socket>>(_pending_capacity);\n        }\n        return _pending[this_shard_id()]->push_eventually(connected_socket(std::make_unique<loopback_connected_socket_impl>(std::move(b1), b2)));\n    }\n    connected_socket make_new_client_connection(lw_shared_ptr<loopback_buffer> b1, foreign_ptr<lw_shared_ptr<loopback_buffer>> b2) {\n        return connected_socket(std::make_unique<loopback_connected_socket_impl>(std::move(b2), b1));\n    }\n    unsigned next_shard() {\n        return _shard++ % _shards_count;\n    }\n    void destroy_shard(unsigned shard) {\n        SEASTAR_ASSERT(shard < _shards_count);\n        _pending[shard] = nullptr;\n    }\n    future<> destroy_all_shards() {\n        return parallel_for_each(std::views::iota(0u, _shards_count), [this](shard_id shard) {\n            return smp::submit_to(shard, [this] {\n                destroy_shard(this_shard_id());\n            });\n        });\n    }\n};\n\nclass loopback_socket_impl : public net::socket_impl {\n    loopback_connection_factory& _factory;\n    loopback_error_injector* _error_injector;\n    lw_shared_ptr<loopback_buffer> _b1;\n    foreign_ptr<lw_shared_ptr<loopback_buffer>> _b2;\n    std::optional<promise<connected_socket>> _connect_abort;\npublic:\n    loopback_socket_impl(loopback_connection_factory& factory, loopback_error_injector* error_injector = nullptr)\n            : _factory(factory), _error_injector(error_injector)\n    { }\n    future<connected_socket> connect(socket_address sa, socket_address local, seastar::transport proto = seastar::transport::TCP) override {\n        if (_error_injector) {\n            auto error = _error_injector->connect_error();\n            if (error != loopback_error_injector::error::none) {\n                _connect_abort.emplace();\n                return _connect_abort->get_future();\n            }\n        }\n\n        auto shard = _factory.next_shard();\n        _b1 = make_lw_shared<loopback_buffer>(_error_injector, loopback_buffer::type::SERVER_TX);\n        return smp::submit_to(shard, [this, b1 = make_foreign(_b1)] () mutable {\n            auto b2 = make_lw_shared<loopback_buffer>(_error_injector, loopback_buffer::type::CLIENT_TX);\n            _b2 = make_foreign(b2);\n            return _factory.make_new_server_connection(std::move(b1), b2).then([b2] {\n                return make_foreign(b2);\n            });\n        }).then([this] (foreign_ptr<lw_shared_ptr<loopback_buffer>> b2) {\n            if (_error_injector) {\n                auto delay = _error_injector->connect_delay();\n                if (delay != std::chrono::microseconds(0)) {\n                    return seastar::sleep(delay).then([this, b2 = std::move(b2)] () mutable {\n                        return _factory.make_new_client_connection(_b1, std::move(b2));\n                    });\n                }\n            }\n            return make_ready_future<connected_socket>(_factory.make_new_client_connection(_b1, std::move(b2)));\n        });\n    }\n    virtual void set_reuseaddr(bool reuseaddr) override {}\n    virtual bool get_reuseaddr() const override { return false; };\n\n    void shutdown() override {\n        if (_connect_abort) {\n            _connect_abort->set_exception(std::make_exception_ptr(std::system_error(ECONNABORTED, std::system_category())));\n            _connect_abort = std::nullopt;\n        } else {\n            _b1->abort();\n            (void)smp::submit_to(_b2.get_owner_shard(), [b2 = std::move(_b2)] {\n                b2->abort();\n            });\n        }\n    }\n};\n\n}\n"
  },
  {
    "path": "tests/unit/lowres_clock_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#include <seastar/testing/test_case.hh>\n\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/lowres_clock.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/later.hh>\n\n#include <ctime>\n\n#include <chrono>\n#include <thread>\n\nusing namespace seastar;\n\n//\n// Sanity check the accuracy of the steady low-resolution clock.\n//\nSEASTAR_TEST_CASE(steady_clock_sanity) {\n    return do_with(lowres_clock::now(), [](auto &&t1) {\n        static constexpr auto sleep_duration = std::chrono::milliseconds(100);\n\n        return ::seastar::sleep(sleep_duration).then([&t1] {\n            auto const elapsed = lowres_clock::now() - t1;\n            auto const minimum_elapsed = 0.9 * sleep_duration;\n\n            BOOST_REQUIRE(elapsed >= minimum_elapsed);\n\n            return make_ready_future<>();\n        });\n    });\n}\n\n//\n// Sanity check the accuracy of the steady low-resolution clock in debug mode\n// where preemption is handled differently.\n//\nSEASTAR_TEST_CASE(steady_clock_sanity_preempt) {\n    auto t1 = lowres_clock::now();\n\n    // Sleep duration must be higher than the task quota to ensure that\n    // preemption is requested before we yield.\n    static constexpr auto sleep_duration = std::chrono::milliseconds(100);\n    std::this_thread::sleep_for(sleep_duration);\n\n    // Yield to the reactor to give it a chance to update the\n    // low-resolution clock. Yield just schedules an empty task and\n    // waits for it to be executed. There is nothing special about it.\n    co_await ::seastar::yield();\n\n    auto const elapsed = lowres_clock::now() - t1;\n    auto const minimum_elapsed = 0.9 * sleep_duration;\n\n    BOOST_REQUIRE(elapsed >= minimum_elapsed);\n}\n\n//\n// At the very least, we can verify that the low-resolution system clock is within a second of the\n// high-resolution system clock.\n//\nSEASTAR_TEST_CASE(system_clock_sanity) {\n    static const auto check_matching = [] {\n        auto const system_time = std::chrono::system_clock::now();\n        auto const lowres_time = lowres_system_clock::now();\n\n        auto const t1 = std::chrono::system_clock::to_time_t(system_time);\n        auto const t2 = lowres_system_clock::to_time_t(lowres_time);\n\n        std::tm *lt1 = std::localtime(&t1);\n        std::tm *lt2 = std::localtime(&t2);\n\n        return (lt1->tm_isdst == lt2->tm_isdst) &&\n               (lt1->tm_year == lt2->tm_year) &&\n               (lt1->tm_mon == lt2->tm_mon) &&\n               (lt1->tm_yday == lt2->tm_yday) &&\n               (lt1->tm_mday == lt2->tm_mday) &&\n               (lt1->tm_wday == lt2->tm_wday) &&\n               (lt1->tm_hour == lt2->tm_hour) &&\n               (lt1->tm_min == lt2->tm_min) &&\n               (lt1->tm_sec == lt2->tm_sec);\n    };\n\n    //\n    // Check two out of three samples in order to account for the possibility that the high-resolution clock backing\n    // the low-resoltuion clock was captured in the range of the 990th to 999th millisecond of the second. This would\n    // make the low-resolution clock and the high-resolution clock disagree on the current second.\n    //\n\n    return do_with(0ul, 0ul, [](std::size_t& index, std::size_t& success_count) {\n        return repeat([&index, &success_count] {\n            if (index >= 3) {\n                BOOST_REQUIRE_GE(success_count, 2u);\n                return make_ready_future<stop_iteration>(stop_iteration::yes);\n            }\n\n            return ::seastar::sleep(std::chrono::milliseconds(10)).then([&index, &success_count] {\n                if (check_matching()) {\n                    ++success_count;\n                }\n\n                ++index;\n                return stop_iteration::no;\n            });\n        });\n    });\n}\n\n//\n// Verify that the low-resolution clock updates its reported time point over time.\n//\nSEASTAR_TEST_CASE(system_clock_dynamic) {\n    return do_with(lowres_system_clock::now(), [](auto &&t1) {\n        return seastar::sleep(std::chrono::milliseconds(100)).then([&t1] {\n            auto const t2 = lowres_system_clock::now();\n            BOOST_REQUIRE_NE(t1.time_since_epoch().count(), t2.time_since_epoch().count());\n\n            return make_ready_future<>();\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/memory-data-sink.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2025 ScyllaDB\n */\n\n#pragma once\n#include <seastar/util/memory-data-sink.hh>\n\nnamespace seastar {\n\ninline void append_buffers(std::stringstream& ss, std::span<seastar::temporary_buffer<char>> bufs) {\n    for (auto& buf : bufs) {\n        ss.write(buf.get(), buf.size());\n    }\n}\n\nnamespace testing {\n\nusing memory_data_sink_impl = util::basic_memory_data_sink<std::stringstream>;\n\n/*!\n * \\brief a helper data sink that stores everything it gets in a stringstream\n */\n\nclass memory_data_sink : public data_sink {\npublic:\n    memory_data_sink(std::stringstream& ss)\n        : data_sink(std::make_unique<memory_data_sink_impl>(ss)) {}\n};\n\n} // testing namespace\n} // seastar namespace\n"
  },
  {
    "path": "tests/unit/metrics_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2019 ScyllaDB.\n */\n\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/relabel_config.hh>\n#include <seastar/core/scheduling.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/io_queue.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/internal/estimated_histogram.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <boost/range/irange.hpp>\n#include <ranges>\n\nSEASTAR_TEST_CASE(test_add_group) {\n    using namespace seastar::metrics;\n    // Just has to compile:\n    metric_groups()\n        .add_group(\"g1\", {})\n        .add_group(\"g2\", std::vector<metric_definition>());\n    return seastar::make_ready_future();\n}\n\n/**\n * This function return the different name label values\n *  for the named metric.\n *\n *  @note: If the statistic or label doesn't exist, the test\n *  that calls this function will fail.\n *\n * @param metric_name - the metric name\n * @param label_name - the label name\n * @return a set containing all the different values\n *         of the label.\n */\nstatic std::set<seastar::sstring> get_label_values(seastar::sstring metric_name, seastar::sstring label_name) {\n    namespace smi = seastar::metrics::impl;\n    auto all_metrics = smi::get_values();\n    const auto& all_metadata = *all_metrics->metadata;\n    const auto qp_group = find_if(cbegin(all_metadata), cend(all_metadata),\n        [&metric_name] (const auto& x) { return x.mf.name == metric_name; });\n    BOOST_REQUIRE(qp_group != cend(all_metadata));\n    std::set<seastar::sstring> labels;\n    for (const auto& metric : qp_group->metrics) {\n        const auto found = metric.labels().find(label_name);\n        BOOST_REQUIRE(found != metric.labels().cend());\n        labels.insert(found->second);\n    }\n    return labels;\n}\n\nSEASTAR_THREAD_TEST_CASE(test_renaming_scheuling_groups) {\n    // this seams a little bit out of place but the\n    // renaming functionality is primarily for statistics\n    // otherwise those classes could have just been reused\n    // without renaming them.\n    using namespace seastar;\n\n    static const char* name1 = \"A\";\n    static const char* name2 = \"B\";\n    scheduling_group sg =  create_scheduling_group(\"hello\", 111).get();\n    auto rng = std::views::iota(0, 1000);\n    // repeatedly change the group name back and forth in\n    // decresing time intervals to see if it generate double\n    //registration statistics errors.\n    for (auto&& i : rng) {\n        const char* name = i%2 ? name1 : name2;\n        const char* prev_name = i%2 ? name2 : name1;\n        sleep(std::chrono::microseconds(100000/(i+1))).get();\n        rename_scheduling_group(sg, name).get();\n        std::set<sstring> label_vals = get_label_values(sstring(\"scheduler_shares\"), sstring(\"group\"));\n        // validate that the name that we *renamed to* is in the stats\n        BOOST_REQUIRE(label_vals.find(sstring(name)) != label_vals.end());\n        // validate that the name that we *renamed from* is *not* in the stats\n        BOOST_REQUIRE(label_vals.find(sstring(prev_name)) == label_vals.end());\n    }\n\n    smp::invoke_on_all([sg] () {\n        return do_with(std::uniform_int_distribution<int>(), boost::irange<int>(0, 1000),\n                [sg] (std::uniform_int_distribution<int>& dist, boost::integer_range<int>& rng) {\n            // flip a fair coin and rename to one of two options and rename to that\n            // scheduling group name, do it 1000 in parallel on all shards so there\n            // is a chance of collision.\n            return do_for_each(rng, [sg, &dist] (auto i) {\n                bool odd = dist(seastar::testing::local_random_engine)%2;\n                return rename_scheduling_group(sg, odd ? name1 : name2);\n            });\n        });\n    }).get();\n\n    std::set<sstring> label_vals = get_label_values(sstring(\"scheduler_shares\"), sstring(\"group\"));\n    // validate that only one of the names is eventually in the metrics\n    bool name1_found = label_vals.find(sstring(name1)) != label_vals.end();\n    bool name2_found = label_vals.find(sstring(name2)) != label_vals.end();\n    BOOST_REQUIRE((name1_found && !name2_found) || (name2_found && !name1_found));\n}\n\nSEASTAR_THREAD_TEST_CASE(test_renaming_execution_stage) {\n    using namespace seastar;\n    const sstring sg1_name = \"sg1\";\n    const sstring sg2_name = \"sg2\";\n    const sstring stage_name = \"my_stage\";\n\n    inheriting_concrete_execution_stage<void> stage{stage_name, []{}};\n    scheduling_group sg = create_scheduling_group(sg1_name, 111).get();\n    with_scheduling_group(sg, [&] { return stage(); }).get();\n\n    std::set<sstring> label_vals = get_label_values(sstring(\"execution_stages_tasks_scheduled\"), sstring(\"execution_stage\"));\n    BOOST_REQUIRE(label_vals.find(stage_name + sstring(\".\") + sstring(sg1_name)) != label_vals.end());\n\n    rename_scheduling_group(sg, sg2_name).get();\n\n    label_vals = get_label_values(sstring(\"execution_stages_tasks_scheduled\"), sstring(\"execution_stage\"));\n    BOOST_REQUIRE(label_vals.find(stage_name + sstring(\".\") + sstring(sg1_name)) == label_vals.end());\n    BOOST_REQUIRE(label_vals.find(stage_name + sstring(\".\") + sstring(sg2_name)) != label_vals.end());\n\n    destroy_scheduling_group(sg).get();\n}\n\nint count_by_label(const std::string& label) {\n    seastar::foreign_ptr<seastar::metrics::impl::values_reference> values = seastar::metrics::impl::get_values();\n    int count = 0;\n    for (auto&& md : (*values->metadata)) {\n        for (auto&& mi : md.metrics) {\n            if (label == \"\" || mi.labels().find(label) != mi.labels().end()) {\n                count++;\n            }\n        }\n    }\n    return count;\n}\n\nint count_by_fun(std::function<bool(const seastar::metrics::impl::metric_series_metadata&)> f) {\n    seastar::foreign_ptr<seastar::metrics::impl::values_reference> values = seastar::metrics::impl::get_values();\n    int count = 0;\n    for (auto&& md : (*values->metadata)) {\n        for (auto&& mi : md.metrics) {\n            if (f(mi)) {\n                count++;\n            }\n        }\n    }\n    return count;\n}\n\nSEASTAR_THREAD_TEST_CASE(test_relabel_add_labels) {\n    using namespace seastar::metrics;\n    namespace sm = seastar::metrics;\n    sm::metric_groups app_metrics;\n    app_metrics.add_group(\"test\", {\n        sm::make_gauge(\"gauge_1\", sm::description(\"gague 1\"), [] { return 0; }),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), [] { return 1; })\n    });\n\n    std::vector<sm::relabel_config> rl(1);\n    rl[0].source_labels = {\"__name__\"};\n    rl[0].target_label = \"level\";\n    rl[0].replacement = \"1\";\n    rl[0].expr = \"test_counter_.*\";\n\n    sm::metric_relabeling_result success = sm::set_relabel_configs(rl).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 0);\n    BOOST_CHECK_EQUAL(count_by_label(\"level\"), 1);\n    app_metrics.add_group(\"test\", {\n        sm::make_counter(\"counter_2\", sm::description(\"counter 2\"), [] { return 2; })\n    });\n    BOOST_CHECK_EQUAL(count_by_label(\"level\"), 2);\n    sm::set_relabel_configs({}).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_metrics_family_aggregate) {\n    using namespace seastar::metrics;\n    namespace sm = seastar::metrics;\n    sm::metric_groups app_metrics;\n    sm::label lb(\"lb\");\n    app_metrics.add_group(\"test\", {\n        sm::make_gauge(\"gauge_1\", sm::description(\"gague 1\"), [] { return 1; })(lb(\"1\")),\n        sm::make_gauge(\"gauge_1\", sm::description(\"gague 1\"), [] { return 2; })(lb(\"2\")),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), [] { return 3; })(lb(\"1\")),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), [] { return 4; })(lb(\"2\"))\n    });\n    std::vector<sm::relabel_config> rl(2);\n    rl[0].source_labels = {\"__name__\"};\n    rl[0].action = sm::relabel_config::relabel_action::drop;\n\n    rl[1].source_labels = {\"lb\"};\n    rl[1].action = sm::relabel_config::relabel_action::keep;\n    // Dropping the lev label would cause a conflict, but not crash the system\n    sm::set_relabel_configs(rl).get();\n\n    std::vector<sm::metric_family_config> fc(2);\n    fc[0].name = \"test_gauge_1\";\n    fc[0].aggregate_labels = { \"lb\" };\n    fc[1].regex_name = \"test_gauge1.*\";\n    fc[1].aggregate_labels = { \"ll\", \"aa\" };\n    sm::set_metric_family_configs(fc);\n    seastar::foreign_ptr<seastar::metrics::impl::values_reference> values = seastar::metrics::impl::get_values();\n    int count = 0;\n    for (auto&& md : (*values->metadata)) {\n        if (md.mf.name == \"test_gauge_1\") {\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels.size(), 1);\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels[0], \"lb\");\n        } else {\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels.size(), 0);\n        }\n        count++;\n    }\n    BOOST_CHECK_EQUAL(count, 2);\n    app_metrics.add_group(\"test\", {\n        sm::make_gauge(\"gauge1_1\", sm::description(\"gague 1\"), [] { return 1; })(lb(\"1\")),\n        sm::make_gauge(\"gauge1_1\", sm::description(\"gague 1\"), [] { return 2; })(lb(\"2\")),\n        sm::make_counter(\"counter1_1\", sm::description(\"counter 1\"), [] { return 3; })(lb(\"1\")),\n        sm::make_counter(\"counter1_1\", sm::description(\"counter 1\"), [] { return 4; })(lb(\"2\"))\n    });\n    values = seastar::metrics::impl::get_values();\n    count = 0;\n    for (auto&& md : (*values->metadata)) {\n        if (md.mf.name == \"test_gauge_1\") {\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels.size(), 1);\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels[0], \"lb\");\n        } else if (md.mf.name == \"test_gauge1_1\") {\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels.size(), 2);\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels[0], \"ll\");\n        } else {\n            BOOST_CHECK_EQUAL(md.mf.aggregate_labels.size(), 0);\n        }\n        count++;\n    }\n    BOOST_CHECK_EQUAL(count, 4);\n    std::vector<sm::relabel_config> rl1;\n    sm::set_relabel_configs(rl1).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_relabel_drop_label_prevent_runtime_conflicts) {\n    using namespace seastar::metrics;\n    namespace sm = seastar::metrics;\n    sm::metric_groups app_metrics;\n    app_metrics.add_group(\"test2\", {\n        sm::make_gauge(\"gauge_1\", sm::description(\"gague 1\"), { sm::label_instance(\"g\", \"1\")}, [] { return 0; }),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), [] { return 0; }),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), { sm::label_instance(\"lev\", \"2\")}, [] { return 0; })\n    });\n    BOOST_CHECK_EQUAL(count_by_label(\"lev\"), 1);\n\n    std::vector<sm::relabel_config> rl(1);\n    rl[0].source_labels = {\"lev\"};\n    rl[0].expr = \"2\";\n    rl[0].target_label = \"lev\";\n    rl[0].action = sm::relabel_config::relabel_action::drop_label;\n    // Dropping the lev label would cause a conflict, but not crash the system\n    sm::metric_relabeling_result success = sm::set_relabel_configs(rl).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 1);\n    BOOST_CHECK_EQUAL(count_by_label(\"lev\"), 0);\n    BOOST_CHECK_EQUAL(count_by_label(\"err\"), 1);\n\n    //reseting all the labels to their original state\n    success = sm::set_relabel_configs({}).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 0);\n    BOOST_CHECK_EQUAL(count_by_label(\"lev\"), 1);\n    BOOST_CHECK_EQUAL(count_by_label(\"err\"), 0);\n    sm::set_relabel_configs({}).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_relabel_enable_disable_skip_when_empty) {\n    using namespace seastar::metrics;\n    namespace sm = seastar::metrics;\n    sm::metric_groups app_metrics;\n    app_metrics.add_group(\"test3\", {\n        sm::make_gauge(\"gauge_1\", sm::description(\"gague 1\"), { sm::label_instance(\"lev3\", \"3\")}, [] { return 0; }),\n        sm::make_counter(\"counter_1\", sm::description(\"counter 1\"), { sm::label_instance(\"lev3\", \"3\")}, [] { return 0; }),\n        sm::make_counter(\"counter_2\", sm::description(\"counter 2\"), { sm::label_instance(\"lev3\", \"3\")}, [] { return 0; })\n    });\n    std::vector<sm::relabel_config> rl(2);\n    rl[0].source_labels = {\"__name__\"};\n    rl[0].action = sm::relabel_config::relabel_action::drop;\n\n    rl[1].source_labels = {\"lev3\"};\n    rl[1].expr = \"3\";\n    rl[1].action = sm::relabel_config::relabel_action::keep;\n    // We just disable all metrics besides those mark as lev3\n    sm::metric_relabeling_result success = sm::set_relabel_configs(rl).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 0);\n    BOOST_CHECK_EQUAL(count_by_label(\"\"), 3);\n    BOOST_CHECK_EQUAL(count_by_fun([](const seastar::metrics::impl::metric_series_metadata& mi) {\n        return mi.should_skip_when_empty() == sm::skip_when_empty::yes;\n    }), 0);\n\n    std::vector<sm::relabel_config> rl2(3);\n    rl2[0].source_labels = {\"__name__\"};\n    rl2[0].action = sm::relabel_config::relabel_action::drop;\n\n    rl2[1].source_labels = {\"lev3\"};\n    rl2[1].expr = \"3\";\n    rl2[1].action = sm::relabel_config::relabel_action::keep;\n\n    rl2[2].source_labels = {\"__name__\"};\n    rl2[2].expr = \"test3.*\";\n    rl2[2].action = sm::relabel_config::relabel_action::skip_when_empty;\n\n    success = sm::set_relabel_configs(rl2).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 0);\n    BOOST_CHECK_EQUAL(count_by_label(\"\"), 3);\n    BOOST_CHECK_EQUAL(count_by_fun([](const seastar::metrics::impl::metric_series_metadata& mi) {\n        return mi.should_skip_when_empty() == sm::skip_when_empty::yes;\n    }), 3);\n    // clear the configuration\n    success = sm::set_relabel_configs({}).get();\n    app_metrics.add_group(\"test3\", {\n        sm::make_counter(\"counter_3\", sm::description(\"counter 2\"), { sm::label_instance(\"lev3\", \"3\")}, [] { return 0; })(sm::skip_when_empty::yes)\n    });\n    std::vector<sm::relabel_config> rl3(3);\n    rl3[0].source_labels = {\"__name__\"};\n    rl3[0].action = sm::relabel_config::relabel_action::drop;\n\n    rl3[1].source_labels = {\"lev3\"};\n    rl3[1].expr = \"3\";\n    rl3[1].action = sm::relabel_config::relabel_action::keep;\n\n    rl3[2].source_labels = {\"__name__\"};\n    rl3[2].expr = \"test3.*\";\n    rl3[2].action = sm::relabel_config::relabel_action::report_when_empty;\n\n    success = sm::set_relabel_configs(rl3).get();\n    BOOST_CHECK_EQUAL(success.metrics_relabeled_due_to_collision, 0);\n    BOOST_CHECK_EQUAL(count_by_fun([](const seastar::metrics::impl::metric_series_metadata& mi) {\n        return mi.should_skip_when_empty() == sm::skip_when_empty::yes;\n    }), 0);\n    sm::set_relabel_configs({}).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_estimated_histogram) {\n    using namespace seastar::metrics;\n    using namespace std::chrono_literals;\n    internal::time_estimated_histogram histogram1;\n    internal::time_estimated_histogram histogram2;\n    // The number of linearly-spaced buckets between consecutive powers of 2 in time_estimated_histogram.\n    constexpr int PRECISION = 4;\n\n    // The lower bound of time_estimated_histogram is 512 us.\n    std::chrono::steady_clock::duration min = std::chrono::microseconds(512);\n    std::chrono::steady_clock::duration next = min*2;\n\n    for (size_t i = 0; i < 16; i++) {\n        auto delta = (next - min)/PRECISION;\n        for (size_t j = 0; j< PRECISION; j++) {\n            histogram1.add(min + delta*j);\n        }\n        min = next;\n        next *= 2;\n    }\n    BOOST_CHECK_EQUAL(histogram1.count(), 64);\n    for (size_t i = 0; i < 64; i++) {\n        BOOST_CHECK_EQUAL(histogram1.get(i), 1);\n    }\n    min = std::chrono::microseconds(512);\n    next = min*2;\n    for (size_t i = 0; i < 8; i++) {\n        auto delta = (next - min)/PRECISION;\n        for (size_t j = 0; j< PRECISION; j++) {\n            histogram2.add(min + delta*j);\n        }\n        min = next;\n        next *= 2;\n    }\n    BOOST_CHECK_EQUAL(histogram2.count(), 32);\n    for (size_t i = 0; i < 32; i++) {\n        BOOST_CHECK_EQUAL(histogram2.get(i), 1);\n    }\n    for (size_t i = 33; i < 64; i++) {\n        BOOST_CHECK_EQUAL(histogram2.get(i), 0);\n    }\n    histogram1.merge(histogram2);\n    BOOST_CHECK_EQUAL(histogram1.count(), 96);\n    for (size_t i = 0; i < 32; i++) {\n        BOOST_CHECK_EQUAL(histogram1.get(i), 2);\n    }\n    for (size_t i = 33; i < 64; i++) {\n        BOOST_CHECK_EQUAL(histogram1.get(i), 1);\n    }\n    auto mh = histogram1.to_metrics_histogram();\n    for (size_t i = 0; i < 32; i++) {\n        BOOST_CHECK_EQUAL(mh.buckets[i].count, 2 + i*2);\n    }\n    for (size_t i = 33; i < 64; i++) {\n        BOOST_CHECK_EQUAL(mh.buckets[i].count, 33 + i);\n    }\n}\n"
  },
  {
    "path": "tests/unit/metrics_tester.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2024 ScyllaDB\n */\n\n#include <seastar/core/app-template.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/relabel_config.hh>\n#include <seastar/core/internal/estimated_histogram.hh>\n#include <seastar/util/closeable.hh>\n#include \"../../apps/lib/stop_signal.hh\"\n#include <map>\n#include <vector>\n#include <yaml-cpp/yaml.h>\nusing namespace seastar;\nusing namespace std::chrono_literals;\nnamespace sm = seastar::metrics;\nstruct serializer {};\n\nstruct metric_def {\n    sstring name;\n    sstring type;\n    std::vector<double> values;\n    std::vector<sm::label_instance> labels;\n};\n\nstruct config {\n    std::vector<metric_def> metrics;\n    std::vector<sm::metric_family_config> metric_family_config;\n};\n\nnamespace YAML {\ntemplate<>\nstruct convert<metric_def> {\n    static bool decode(const Node& node, metric_def& cfg) {\n        if (node[\"name\"]) {\n            cfg.name = node[\"name\"].as<std::string>();\n        }\n        if (node[\"type\"]) {\n            cfg.type = node[\"type\"].as<std::string>();\n        }\n        if (node[\"values\"]) {\n            cfg.values = node[\"values\"].as<std::vector<double>>();\n        }\n        if (node[\"labels\"]) {\n            const auto labels = node[\"labels\"].as<std::map<std::string, std::string>>();\n            for (auto& [key, value]: labels) {\n                cfg.labels.emplace_back(key, value);\n            }\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<sm::metric_family_config> {\n    static bool decode(const Node& node, sm::metric_family_config& cfg) {\n        if (node[\"name\"]) {\n            cfg.name = node[\"name\"].as<std::string>();\n        }\n        if (node[\"regex_name\"]) {\n            cfg.regex_name = node[\"regex_name\"].as<std::string>();\n        }\n        if (node[\"aggregate_labels\"]) {\n            cfg.aggregate_labels = node[\"aggregate_labels\"].as<std::vector<std::string>>();\n        }\n        return true;\n    }\n};\n\ntemplate<>\nstruct convert<config> {\n    static bool decode(const Node& node, config& cfg) {\n        if (node[\"metrics\"]) {\n            cfg.metrics = node[\"metrics\"].as<std::vector<metric_def>>();\n        }\n        if (node[\"metric_family_config\"]) {\n            cfg.metric_family_config = node[\"metric_family_config\"].as<std::vector<sm::metric_family_config>>();\n        }\n        return true;\n    }\n};\n\n}\nstd::function<sm::internal::time_estimated_histogram()> make_histogram_fun(const metric_def& c) {\n    sm::internal::time_estimated_histogram histogram;\n    for (const auto& v : c.values) {\n        histogram.add_micro(v);\n    }\n    return [histogram]() {return histogram;};\n}\n\nsm::impl::metric_definition_impl make_metrics_definition(const metric_def& jc) {\n    if (jc.type == \"histogram\") {\n        sm::internal::time_estimated_histogram histogram;\n        for (const auto& v : jc.values) {\n            histogram.add_micro(v);\n        }\n        return sm::make_histogram(jc.name, [histogram]() {return histogram.to_metrics_histogram();},\n                sm::description(jc.name), jc.labels );\n    }\n    if (jc.type == \"gauge\") {\n        return sm::make_gauge(jc.name, [val=jc.values[0]] { return val; },\n                sm::description(jc.name), jc.labels );\n    }\n    return sm::make_counter(jc.name, [val=jc.values[0]] { return val; },\n            sm::description(jc.name), jc.labels );\n}\n\nint main(int ac, char** av) {\n    namespace bpo = boost::program_options;\n\n    app_template app;\n    auto opt_add = app.add_options();\n    opt_add\n        (\"listen\", bpo::value<sstring>()->default_value(\"0.0.0.0\"), \"address to start Prometheus server on\")\n        (\"port\", bpo::value<uint16_t>()->default_value(9180), \"Prometheus port\")\n        (\"conf\", bpo::value<sstring>()->default_value(\"./conf.yaml\"), \"config with jobs and options\")\n    ;\n    httpd::http_server_control prometheus_server;\n\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            sm::metric_groups _metrics;\n            seastar_apps_lib::stop_signal stop_signal;\n            auto& opts = app.configuration();\n            auto& listen = opts[\"listen\"].as<sstring>();\n            auto& port = opts[\"port\"].as<uint16_t>();\n            auto& conf = opts[\"conf\"].as<sstring>();\n\n            YAML::Node doc = YAML::LoadFile(conf);\n            auto cfg = doc.as<config>();\n\n            for (auto&& jc : cfg.metrics) {\n                _metrics.add_group(\"test_group\", {\n                        make_metrics_definition(jc)\n                });\n            }\n            smp::invoke_on_all([] {\n                    std::vector<metrics::relabel_config> rl(2);\n                    rl[0].source_labels = {\"__name__\"};\n                    rl[0].expr = \".*\";\n                    rl[0].action = metrics::relabel_config::relabel_action::drop;\n\n                    rl[1].source_labels = {\"private\"};\n                    rl[1].expr = \".*\";\n                    rl[1].action = metrics::relabel_config::relabel_action::keep;\n                    return metrics::set_relabel_configs(rl).then([](metrics::metric_relabeling_result) {\n                        return;\n                    });\n            }).get();\n\n            if (!cfg.metric_family_config.empty()) {\n                sm::set_metric_family_configs(cfg.metric_family_config);\n            }\n\n            prometheus_server.start(\"prometheus\").get();\n            auto stop_server = deferred_stop(prometheus_server);\n\n            prometheus::config pctx;\n            pctx.allow_protobuf = true;\n            prometheus::start(prometheus_server, pctx).get();\n            prometheus_server.listen(socket_address{listen, port}).handle_exception([] (auto ep) {\n                return make_exception_future<>(ep);\n            }).get();\n\n            fmt::print(\"{}\\n\", port);\n            fflush(stdout);\n\n            stop_signal.wait().get();\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/mkcert.gmk",
    "content": "server  = $(shell hostname)\ndomain  = $(shell dnsdomainname)\nname    = $(server)\n\ncountry = SE\nstate   = Stockholm\nlocality= $(state)\norg     = $(domain)\nunit    = $(domain)\nmail    = mx\ncommon  = $(server).$(domain)\nemail   = postmaster@$(domain)\nckey    = ca$(key).pem\n\npubkey  = $(name).pub\nprvkey  = $(name).key\nwidth   = 4096\n\ncsr     = $(name).csr\ncrt     = $(name).crt\n\nroot    = ca$(name).pem\nrootkey = ca$(name).key\n\nconfig  = $(name).cfg\ndays    = 3650\n\nalg \t= RSA\nalg_opt = -pkeyopt rsa_keygen_bits:$(width)\n\nhosts   =\n\nall     \t: $(crt)\n\nclean\t\t:\n\t\t@rm -f $(crt) $(csr) $(pubkey) $(prvkey)\n\n%.key\t\t:\n\t\t@echo generating $@\n\t\topenssl genpkey -out $@ -algorithm $(alg) $(alg_opt)\n\n%.pub  \t\t: %.key\n\t\t@echo generating $@\n\t\topenssl pkey -in $< -out $@\n\n$(config) \t: $(MAKEFILE_LIST)\n\t\t@echo generating $@\n\t\t@(\t\t\t\t\t\t\t\t\t\t\\\n\t\t\techo [ req ]                                                    ;       \\\n\t\t\techo default_bits = $(width)                                    ;       \\\n\t\t\techo default_keyfile = $(prvkey)                                  ;       \\\n\t\t\techo default_md = sha256 ;\t\\\n\t\t\techo distinguished_name = req_distinguished_name        \t;       \\\n\t\t\techo req_extensions = v3_req                                    ;       \\\n\t\t\techo prompt = no                                                ;       \\\n\t\t\techo [ req_distinguished_name ]         \t\t\t;       \\\n\t\t\techo C = $(country)     \t\t\t\t\t;       \\\n\t\t\techo ST = $(state)      \t\t\t\t\t;       \\\n\t\t\techo L = $(locality)            \t\t\t\t;       \\\n\t\t\techo O = $(org)\t\t\t\t\t\t\t;\t\\\n\t\t\techo OU = $(unit)       \t\t\t\t\t;       \\\n\t\t\techo CN= $(common)      \t\t\t\t\t;       \\\n\t\t\techo emailAddress = $(email)            \t\t\t;       \\\n\t\t\techo [v3_ca]            \t\t\t\t\t;       \\\n\t\t\techo subjectKeyIdentifier=hash          \t\t\t;       \\\n\t\t\techo authorityKeyIdentifier=keyid:always,issuer:always  \t;       \\\n\t\t\techo basicConstraints = CA:true         \t\t\t;       \\\n\t\t\techo [v3_req]           \t\t\t\t\t;       \\\n\t\t\techo \"# Extensions to add to a certificate request\"       \t;       \\\n\t\t\techo basicConstraints = CA:FALSE        \t\t\t;       \\\n\t\t\techo keyUsage = nonRepudiation, digitalSignature, keyEncipherment ;     \\\n\t\t\t$(if $(hosts), echo subjectAltName = @alt_names         \t;)      \\\n\t\t\t$(if $(hosts), echo [alt_names]         \t\t\t;)      \\\n\t\t\t$(if $(hosts), index=1; for host in $(hosts);   \t\t\t\\\n\t\t        \tdo echo DNS.$$index = $$host.$(domain);         \t\t\\\n\t\t\t\tindex=$$(($$index + 1));done    ;)      \t\t\t\\\n\t\t) > $@\n\n%.csr\t\t: %.key $(config)\n\t        @echo generating $@\n\t        openssl req -new -key $< -out $@ -config $(config)\n\n%.crt  \t\t: %.csr $(root) $(rootkey)\n\t\t@echo generating $@\n\t\topenssl x509 -req -in $< -CA $(root) -CAkey $(rootkey) -CAcreateserial \t\t\\\n\t\t\t-out $@ -days $(days)\n\n%.pem\t\t: %.key $(config)\n\t        @echo generating $@\n\t\topenssl req -x509 -new -nodes -key $< -days $(days) -config $(config) \t\t\\\n\t\t\t-out $@\n\n.PRECIOUS\t: %.pem %.key %.pub %.crt %.csr\n\n"
  },
  {
    "path": "tests/unit/mkmtls.gmk",
    "content": "server  = $(shell hostname)\ndomain  = $(shell dnsdomainname)\nname    = $(server)\n\ncountry = SE\nstate   = Stockholm\nlocality= $(state)\norg     = $(domain)\nunit    = $(domain)\nmail    = mx\ncommon  = $(server).$(domain)\nsubj\t= \"/C=$(country)/ST=$(state)/L=$(locality)/O=$(domain)/OU=$(domain)/CN=$(common)\"\nclient1\t= \"/C=$(country)/ST=$(state)/L=$(locality)/O=$(domain)/OU=$(domain)/CN=client1.org\"\nclient2\t= \"/C=$(country)/ST=$(state)/L=$(locality)/O=$(domain)/OU=$(domain)/CN=client2.org\"\nmtls_certs  :\n\t\topenssl ecparam -name prime256v1 -genkey -noout -out mtls_ca.key\n\t\topenssl req -new -x509 -sha256 -key mtls_ca.key -out mtls_ca.crt -subj $(subj)\n\t\topenssl ecparam -name prime256v1 -genkey -noout -out mtls_server.key\n\t\topenssl req -new -sha256 -key mtls_server.key -out mtls_server.csr -subj $(subj)\n\t\topenssl x509 -req -in mtls_server.csr -CA mtls_ca.crt -CAkey mtls_ca.key -CAcreateserial -out mtls_server.crt -days 1000 -sha256\n\n\t\topenssl ecparam -name prime256v1 -genkey -noout -out mtls_client1.key\n\t\topenssl req -new -sha256 -key mtls_client1.key -out mtls_client1.csr  -subj $(client1)\n\t\topenssl x509 -req -in mtls_client1.csr -CA mtls_ca.crt -CAkey mtls_ca.key -CAcreateserial -out mtls_client1.crt -days 1000 -sha256\n\n\t\topenssl ecparam -name prime256v1 -genkey -noout -out mtls_client2.key\n\t\topenssl req -new -sha256 -key mtls_client2.key -out mtls_client2.csr  -subj $(client2)\n\t\topenssl x509 -req -in mtls_client2.csr -CA mtls_ca.crt -CAkey mtls_ca.key -CAcreateserial -out mtls_client2.crt -days 1000 -sha256\n\n"
  },
  {
    "path": "tests/unit/mock_file.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <boost/range/numeric.hpp>\n\n#include <seastar/testing/seastar_test.hh>\n#include <seastar/core/file.hh>\n#include <seastar/util/assert.hh>\n\nnamespace seastar {\n\nclass mock_read_only_file final : public file_impl {\n    bool _closed = false;\n    uint64_t _total_file_size;\n    size_t _allowed_read_requests = 0;\n    std::function<void(size_t)> _verify_length;\nprivate:\n    size_t verify_read(uint64_t position, size_t length) {\n        BOOST_CHECK(!_closed);\n        BOOST_CHECK_LE(position, _total_file_size);\n        BOOST_CHECK_LE(position + length, _total_file_size);\n        if (position + length != _total_file_size) {\n            _verify_length(length);\n        }\n        BOOST_CHECK(_allowed_read_requests);\n        SEASTAR_ASSERT(_allowed_read_requests);\n        _allowed_read_requests--;\n        return length;\n    }\npublic:\n    explicit mock_read_only_file(uint64_t file_size) noexcept\n        : _total_file_size(file_size)\n        , _verify_length([] (auto) { })\n    { }\n\n    void set_read_size_verifier(std::function<void(size_t)> fn) {\n        _verify_length = fn;\n    }\n    void set_expected_read_size(size_t expected) {\n        _verify_length = [expected] (auto length) {\n            BOOST_CHECK_EQUAL(length, expected);\n        };\n    }\n    void set_allowed_read_requests(size_t requests) {\n        _allowed_read_requests = requests;\n    }\n\n    virtual future<size_t> write_dma(uint64_t, const void*, size_t, io_intent*) noexcept override {\n        return make_exception_future<size_t>(std::bad_function_call());\n    }\n    virtual future<size_t> write_dma(uint64_t, std::vector<iovec>, io_intent*) noexcept override {\n        return make_exception_future<size_t>(std::bad_function_call());\n    }\n    virtual future<size_t> read_dma(uint64_t pos, void*, size_t len, io_intent*) noexcept override {\n        return make_ready_future<size_t>(verify_read(pos, len));\n    }\n    virtual future<size_t> read_dma(uint64_t pos, std::vector<iovec> iov, io_intent*) noexcept override {\n        auto length = boost::accumulate(iov | boost::adaptors::transformed([] (auto&& iov) { return iov.iov_len; }),\n                                        size_t(0), std::plus<size_t>());\n        return make_ready_future<size_t>(verify_read(pos, length));\n    }\n    virtual future<> flush() noexcept override {\n        return make_ready_future<>();\n    }\n    virtual future<struct stat> stat() noexcept override {\n        return make_exception_future<struct stat>(std::bad_function_call());\n    }\n    virtual future<struct stat> statat(std::string_view name, int flags) noexcept override {\n        return make_exception_future<struct stat>(std::bad_function_call());\n    }\n    virtual future<> truncate(uint64_t) noexcept override {\n        return make_exception_future<>(std::bad_function_call());\n    }\n    virtual future<> discard(uint64_t offset, uint64_t length) noexcept override {\n        return make_exception_future<>(std::bad_function_call());\n    }\n    virtual future<> allocate(uint64_t position, uint64_t length) noexcept override {\n        return make_exception_future<>(std::bad_function_call());\n    }\n    virtual future<uint64_t> size() noexcept override {\n        return make_ready_future<uint64_t>(_total_file_size);\n    }\n    virtual future<> close() noexcept override {\n        BOOST_CHECK(!_closed);\n        _closed = true;\n        return make_ready_future<>();\n    }\n    virtual subscription<directory_entry> list_directory(std::function<future<> (directory_entry de)>) override {\n        throw std::bad_function_call();\n    }\n    virtual future<temporary_buffer<uint8_t>> dma_read_bulk(uint64_t offset, size_t range_size, io_intent*) noexcept override {\n        auto length = verify_read(offset, range_size);\n        return make_ready_future<temporary_buffer<uint8_t>>(temporary_buffer<uint8_t>(length));\n    }\n};\n\n}\n"
  },
  {
    "path": "tests/unit/net_config_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2017 Marek Waszkiewicz ( marek.waszkiewicz77@gmail.com )\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <seastar/net/config.hh>\n#include <boost/test/unit_test.hpp>\n#include <exception>\n#include <sstream>\n\nusing namespace seastar::net;\n\nBOOST_AUTO_TEST_CASE(test_valid_config_with_pci_address) {\n    std::stringstream ss;\n    ss << \"{eth0: {pci-address: 0000:06:00.0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: \"\n          \"255.255.255.0 } , eth1: {pci-address: 0000:06:00.1, dhcp: true } }\";\n    auto device_configs = parse_config(ss);\n\n    // eth0 tests\n    BOOST_REQUIRE(device_configs.find(\"eth0\") != device_configs.end());\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").hw_cfg.pci_address, \"0000:06:00.0\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.dhcp, false);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.ip, \"192.168.100.10\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.gateway, \"192.168.100.1\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.netmask, \"255.255.255.0\");\n\n    // eth1 tests\n    BOOST_REQUIRE(device_configs.find(\"eth1\") != device_configs.end());\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").hw_cfg.pci_address, \"0000:06:00.1\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.dhcp, true);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.ip, \"\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.gateway, \"\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.netmask, \"\");\n}\n\nBOOST_AUTO_TEST_CASE(test_valid_config_with_port_index) {\n    std::stringstream ss;\n    ss << \"{eth0: {port-index: 0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: \"\n          \"255.255.255.0 } , eth1: {port-index: 1, dhcp: true } }\";\n    auto device_configs = parse_config(ss);\n\n    // eth0 tests\n    BOOST_REQUIRE(device_configs.find(\"eth0\") != device_configs.end());\n    BOOST_REQUIRE_EQUAL(*device_configs.at(\"eth0\").hw_cfg.port_index, 0u);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.dhcp, false);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.ip, \"192.168.100.10\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.gateway, \"192.168.100.1\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.netmask, \"255.255.255.0\");\n\n    // eth1 tests\n    BOOST_REQUIRE(device_configs.find(\"eth1\") != device_configs.end());\n    BOOST_REQUIRE_EQUAL(*device_configs.at(\"eth1\").hw_cfg.port_index, 1u);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.dhcp, true);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.ip, \"\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.gateway, \"\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth1\").ip_cfg.netmask, \"\");\n}\n\nBOOST_AUTO_TEST_CASE(test_valid_config_single_device) {\n    std::stringstream ss;\n    ss << \"eth0: {pci-address: 0000:06:00.0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: \"\n          \"255.255.255.0 }\";\n    auto device_configs = parse_config(ss);\n\n    // eth0 tests\n    BOOST_REQUIRE(device_configs.find(\"eth0\") != device_configs.end());\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").hw_cfg.pci_address, \"0000:06:00.0\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.dhcp, false);\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.ip, \"192.168.100.10\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.gateway, \"192.168.100.1\");\n    BOOST_REQUIRE_EQUAL(device_configs.at(\"eth0\").ip_cfg.netmask, \"255.255.255.0\");\n}\n\nBOOST_AUTO_TEST_CASE(test_unsupported_key) {\n    std::stringstream ss;\n    ss << \"{eth0: { some_not_supported_tag: xxx, pci-address: 0000:06:00.0, ip: 192.168.100.10, \"\n          \"gateway: 192.168.100.1, netmask: 255.255.255.0 } , eth1: {pci-address: 0000:06:00.1, \"\n          \"dhcp: true } }\";\n\n    BOOST_REQUIRE_THROW(parse_config(ss), config_exception);\n}\n\nBOOST_AUTO_TEST_CASE(test_bad_yaml_syntax_if_thrown) {\n    std::stringstream ss;\n    ss << \"some bad: [ yaml syntax }\";\n    BOOST_REQUIRE_THROW(parse_config(ss), std::runtime_error);\n}\n\nBOOST_AUTO_TEST_CASE(test_pci_address_and_port_index_if_thrown) {\n    std::stringstream ss;\n    ss << \"{eth0: {pci-address: 0000:06:00.0, port-index: 0, ip: 192.168.100.10, gateway: \"\n          \"192.168.100.1, netmask: 255.255.255.0 } , eth1: {pci-address: 0000:06:00.1, dhcp: true} \"\n          \"}\";\n    BOOST_REQUIRE_THROW(parse_config(ss), config_exception);\n}\n\nBOOST_AUTO_TEST_CASE(test_dhcp_and_ip_if_thrown) {\n    std::stringstream ss;\n    ss << \"{eth0: {pci-address: 0000:06:00.0, ip: 192.168.100.10, gateway: 192.168.100.1, netmask: \"\n          \"255.255.255.0, dhcp: true } , eth1: {pci-address: 0000:06:00.1, dhcp: true} }\";\n    BOOST_REQUIRE_THROW(parse_config(ss), config_exception);\n}\n\nBOOST_AUTO_TEST_CASE(test_ip_missing_if_thrown) {\n    std::stringstream ss;\n    ss << \"{eth0: {pci-address: 0000:06:00.0, gateway: 192.168.100.1, netmask: 255.255.255.0 } , \"\n          \"eth1: {pci-address: 0000:06:00.1, dhcp: true} }\";\n    BOOST_REQUIRE_THROW(parse_config(ss), config_exception);\n}\n"
  },
  {
    "path": "tests/unit/network_interface_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/ethernet.hh>\n#include <seastar/net/ip.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/log.hh>\n\nusing namespace seastar;\n\nstatic logger niflog(\"network_interface_test\");\n\nstatic_assert(std::is_nothrow_default_constructible_v<net::ethernet_address>);\nstatic_assert(std::is_nothrow_copy_constructible_v<net::ethernet_address>);\nstatic_assert(std::is_nothrow_move_constructible_v<net::ethernet_address>);\n\nSEASTAR_TEST_CASE(list_interfaces) {\n    // just verifying we have something. And can access all the stuff.\n    auto interfaces = engine().net().network_interfaces();\n    BOOST_REQUIRE_GT(interfaces.size(), 0);\n\n    for (auto& nif : interfaces) {\n        niflog.info(\"Iface: {}, index = {}, mtu = {}, loopback = {}, virtual = {}, up = {}\",\n            nif.name(), nif.index(), nif.mtu(), nif.is_loopback(), nif.is_virtual(), nif.is_up()\n        );\n        if (nif.hardware_address().size() >= 6) {\n            niflog.info(\"   HW: {}\", net::ethernet_address(nif.hardware_address().data()));\n        }\n        for (auto& addr : nif.addresses()) {\n            niflog.info(\"   Addr: {}\", addr);\n        }\n    }\n\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(match_ipv6_scope) {\n    auto interfaces = engine().net().network_interfaces();\n\n    for (auto& nif : interfaces) {\n        if (nif.is_loopback()) {\n            continue;\n        }\n        auto i = std::find_if(nif.addresses().begin(), nif.addresses().end(), std::mem_fn(&net::inet_address::is_ipv6));\n        if (i == nif.addresses().end()) {\n            continue;\n        }\n\n        std::ostringstream ss;\n        ss << net::inet_address(i->as_ipv6_address()) << \"%\" << nif.name();\n        auto text = ss.str();\n\n        net::inet_address na(text);\n\n        BOOST_REQUIRE_EQUAL(na.as_ipv6_address(), i->as_ipv6_address());\n        // also verify that the inet_address itself matches\n        BOOST_REQUIRE_EQUAL(na, *i);\n        // and that inet_address _without_ scope matches.\n        BOOST_REQUIRE_EQUAL(net::inet_address(na.as_ipv6_address()), *i);\n        BOOST_REQUIRE_EQUAL(na.scope(), nif.index());\n        // and that they are not ipv4 addresses\n        BOOST_REQUIRE_THROW(i->as_ipv4_address(), std::invalid_argument);\n        BOOST_REQUIRE_THROW(na.as_ipv4_address(), std::invalid_argument);\n\n        niflog.info(\"Org: {}, Parsed: {}, Text: {}\", *i, na, text);\n\n    }\n\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(is_standard_addresses_sanity) {\n    BOOST_REQUIRE_EQUAL(net::inet_address(\"127.0.0.1\").is_loopback(), true);\n    BOOST_REQUIRE_EQUAL(net::inet_address(\"127.0.0.11\").is_loopback(), true);\n    BOOST_REQUIRE_EQUAL(net::inet_address(::in_addr{INADDR_ANY}).is_addr_any(), true);\n    auto addr = net::inet_address(\"1.2.3.4\");\n    BOOST_REQUIRE_EQUAL(addr.is_loopback(), false);\n    BOOST_REQUIRE_EQUAL(addr.is_addr_any(), false);\n\n    BOOST_REQUIRE_EQUAL(net::inet_address(\"::1\").is_loopback(), true);\n    BOOST_REQUIRE_EQUAL(net::inet_address(::in6addr_any).is_addr_any(), true);\n    auto addr6 = net::inet_address(\"acf1:f5e5:5a99:337f:ebe2:c57e:0e27:69c6\");\n    BOOST_REQUIRE_EQUAL(addr6.is_loopback(), false);\n    BOOST_REQUIRE_EQUAL(addr6.is_addr_any(), false);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_inet_address_format) {\n    const std::string tests[] = {\n        // IPv4 addresses\n        \"127.0.0.1\",\n        \"192.168.100.123\",\n        // IPv6 addresses\n        // see also https://datatracker.ietf.org/doc/html/rfc5952\n        // leading zeros are removed\n        \"2001:db8:85a3:8d3:1319:8a2e:370:7348\",\n        // consecutive all-zeros must be compressed\n        \"2001:db8::8a2e:370:7334\",\n        \"::1\",\n        \"100::\",\n    };\n\n    for (auto expected : tests) {\n        net::inet_address addr{expected};\n        BOOST_CHECK_EQUAL(fmt::to_string(addr), expected);\n    }\n\n    // scoped addresses\n    for (auto& nwif: seastar::engine().net().network_interfaces()) {\n        const std::string_view address = \"fe80::1ff:fe23:4567:890a\";\n        const sstring zone_id = fmt::to_string(nwif.index());\n        for (auto zone : {zone_id, nwif.name(), nwif.display_name()}) {\n            net::inet_address addr{fmt::format(\"{}%{}\", address, zone)};\n            // we always use the zone-id to represent the zone\n            auto expected = fmt::format(\"{}%{}\", address, zone_id);\n            BOOST_CHECK_EQUAL(fmt::to_string(addr), expected);\n        }\n        // one of them would suffice\n        break;\n    }\n    return make_ready_future();\n}\n\nSEASTAR_TEST_CASE(test_inet_address_parse_invalid) {\n    const sstring tests[] = {\n        // bad IPv4 addresses\n        \"127.0.0\",\n        \"192.168.100,123\",\n        \"192.168.1.2.3\",\n        // bad IPv6 addresses\n        \"fe80:2030:31:24\",\n        \"fe80:2030:12345\",\n        \"fe80:2030:12345%%\",\n        \":\",\n    };\n\n    for (auto s : tests) {\n        BOOST_CHECK_THROW(net::inet_address{s}, std::invalid_argument);\n    }\n    return make_ready_future();\n}\n"
  },
  {
    "path": "tests/unit/noncopyable_function_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/util/noncopyable_function.hh>\n\nusing namespace seastar;\n\nBOOST_AUTO_TEST_CASE(basic_tests) {\n    struct s {\n        int f1(int x) const { return x + 1; }\n        int f2(int x) { return x + 2; }\n        static int f3(int x) { return x + 3; }\n        int operator()(int x) const { return x + 4; }\n    };\n    s obj, obj2;\n    auto fn1 = noncopyable_function<int (const s*, int)>(&s::f1);\n    auto fn2 = noncopyable_function<int (s*, int)>(&s::f2);\n    auto fn3 = noncopyable_function<int (int)>(&s::f3);\n    auto fn4 = noncopyable_function<int (int)>(std::move(obj2));\n    BOOST_REQUIRE_EQUAL(fn1(&obj, 1), 2);\n    BOOST_REQUIRE_EQUAL(fn2(&obj, 1), 3);\n    BOOST_REQUIRE_EQUAL(fn3(1), 4);\n    BOOST_REQUIRE_EQUAL(fn4(1), 5);\n}\n\ntemplate <size_t Extra>\nstruct payload {\n    static unsigned live;\n    char extra[Extra];\n    std::unique_ptr<int> v;\n    payload(int x) : v(std::make_unique<int>(x)) { ++live; }\n    payload(payload&& x) noexcept : v(std::move(x.v)) { ++live; }\n    void operator=(payload&&) = delete;\n    ~payload() { --live; }\n    int operator()() const { return *v; }\n};\n\ntemplate <size_t Extra>\nunsigned payload<Extra>::live;\n\ntemplate <size_t Extra>\nvoid do_move_tests() {\n    using payload = ::payload<Extra>;\n    auto f1 = noncopyable_function<int ()>(payload(3));\n    BOOST_REQUIRE_EQUAL(payload::live, 1u);\n    BOOST_REQUIRE_EQUAL(f1(), 3);\n    auto f2 = noncopyable_function<int ()>();\n    BOOST_CHECK_THROW(f2(), std::bad_function_call);\n    f2 = std::move(f1);\n    BOOST_CHECK_THROW(f1(), std::bad_function_call);\n    BOOST_REQUIRE_EQUAL(f2(), 3);\n    BOOST_REQUIRE_EQUAL(payload::live, 1u);\n    f2 = {};\n    BOOST_REQUIRE_EQUAL(payload::live, 0u);\n    BOOST_CHECK_THROW(f2(), std::bad_function_call);\n}\n\nBOOST_AUTO_TEST_CASE(small_move_tests) {\n    do_move_tests<1>();\n}\n\nBOOST_AUTO_TEST_CASE(large_move_tests) {\n    do_move_tests<1000>();\n}\n\n"
  },
  {
    "path": "tests/unit/output_stream_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/random.hh>\n#include <vector>\n#include <list>\n#include <deque>\n#include <sstream>\n#include \"memory-data-sink.hh\"\n\nusing namespace seastar;\nusing namespace net;\n\nstruct stream_maker {\n    output_stream_options opts;\n    size_t _size;\n\n    stream_maker size(size_t size) && {\n        _size = size;\n        return std::move(*this);\n    }\n\n    stream_maker trim(bool do_trim) && {\n        opts.trim_to_size = do_trim;\n        return std::move(*this);\n    }\n\n    lw_shared_ptr<output_stream<char>> operator()(data_sink sink) {\n        return make_lw_shared<output_stream<char>>(std::move(sink), _size, opts);\n    }\n};\n\n// A sink that records received chunk sizes and concatenates all data,\n// allowing invariant-based assertions rather than exact chunk matching.\nclass invariant_checker_sink final : public data_sink_impl {\n    std::string& _received_data;\n    std::vector<size_t>& _chunk_sizes;\npublic:\n    invariant_checker_sink(std::string& received_data, std::vector<size_t>& chunk_sizes)\n        : _received_data(received_data)\n        , _chunk_sizes(chunk_sizes)\n    { }\n\n    future<> put(std::span<temporary_buffer<char>> bufs) override {\n        size_t put_size = 0;\n        for (auto&& buf : bufs) {\n            _received_data.append(buf.get(), buf.size());\n            put_size += buf.size();\n        }\n        if (put_size > 0) {\n            _chunk_sizes.push_back(put_size);\n        }\n        return make_ready_future<>();\n    }\n\n    future<> close() override { return make_ready_future<>(); }\n};\n\n// Iterates over all combinations of 1..MAX_CHUNKS chunks, each 1..MAX_CHUNK_SIZE\n// bytes, for both trim modes, and both buffered and zero-copy write paths.\n//\n// STREAM_SIZE=5 is chosen as a small non-power-of-two value.\n// MAX_CHUNK_SIZE=3*STREAM_SIZE ensures the split loop is exercised for at\n// least three full buffer-lengths in a single write.\n// MAX_CHUNKS=4 keeps the combination space tractable (~11k sequences).\nstatic constexpr size_t STREAM_SIZE = 5;\nstatic constexpr size_t MAX_CHUNKS = 4;\nstatic constexpr size_t MAX_CHUNK_SIZE = 3 * STREAM_SIZE;\n\nenum class write_type { buffered, zero_copy };\n\nstatic std::string format_context(const std::vector<size_t>& input_chunk_sizes,\n        size_t stream_size, bool trim_to_size,\n        const std::vector<write_type>& write_types) {\n    std::ostringstream os;\n    os << \"stream_size=\" << stream_size\n       << \" trim_to_size=\" << trim_to_size\n       << \" input_chunks=[\";\n    for (size_t i = 0; i < input_chunk_sizes.size(); i++) {\n        if (i > 0) os << \", \";\n        os << input_chunk_sizes[i];\n    }\n    os << \"] write_types=[\";\n    for (size_t i = 0; i < write_types.size(); i++) {\n        if (i > 0) os << \", \";\n        os << (write_types[i] == write_type::buffered ? \"buffered\" : \"zero_copy\");\n    }\n    os << \"]\";\n    return os.str();\n}\n\n// Checks the output invariants after all writes and close():\n// - data integrity: concatenation of output == concatenation of input\n// - no empty chunks reach the sink\n// - for trim_to_size=true:  all non-last chunks are exactly _size bytes\n// - for trim_to_size=false: all non-last chunks are >= _size bytes\n// - if nothing was written, the sink receives no chunks at all\nstatic void check_invariants(const std::string& expected_data,\n        const std::vector<size_t>& chunk_sizes,\n        const std::string& received_data,\n        size_t stream_size, bool trim_to_size,\n        const std::string& ctx) {\n    BOOST_REQUIRE_MESSAGE(received_data == expected_data,\n            \"data integrity check failed: \" << ctx);\n\n    if (expected_data.empty()) {\n        BOOST_REQUIRE_MESSAGE(chunk_sizes.empty(),\n                \"no chunks expected for empty write: \" << ctx);\n        return;\n    }\n\n    BOOST_REQUIRE_MESSAGE(chunk_sizes.back() > 0,\n            \"sink must never receive an empty chunk: \" << ctx);\n\n    for (size_t i = 0; i + 1 < chunk_sizes.size(); i++) {\n        BOOST_REQUIRE_MESSAGE(chunk_sizes[i] > 0,\n                \"sink must never receive an empty chunk: \" << ctx);\n        if (trim_to_size) {\n            BOOST_REQUIRE_MESSAGE(chunk_sizes[i] == stream_size,\n                    \"with trim_to_size all non-last chunks must be exactly _size bytes: \" << ctx);\n        } else {\n            BOOST_REQUIRE_MESSAGE(chunk_sizes[i] >= stream_size,\n                    \"without trim_to_size all non-last chunks must be >= _size bytes: \" << ctx);\n        }\n    }\n}\n\n// Calls fn(chunk_sizes) for every combination of 1..MAX_CHUNKS chunks\n// each of size 1..MAX_CHUNK_SIZE.\ntemplate <typename Fn>\nstatic void for_each_chunk_combination(Fn fn) {\n    std::vector<size_t> combo;\n    std::function<void()> recurse = [&]() {\n        if (!combo.empty()) {\n            fn(combo);\n        }\n        if (combo.size() < MAX_CHUNKS) {\n            for (size_t sz = 1; sz <= MAX_CHUNK_SIZE; sz++) {\n                combo.push_back(sz);\n                recurse();\n                combo.pop_back();\n            }\n        }\n    };\n    recurse();\n}\n\n// Calls fn(write_types) for every assignment of buffered/zero_copy to\n// n write calls (2^n patterns total; n <= MAX_CHUNKS so at most 16).\n#ifndef SEASTAR_DEBUG\ntemplate <typename Fn>\nstatic void for_each_type_pattern(size_t n, Fn fn) {\n    std::vector<write_type> pattern(n);\n    for (size_t mask = 0; mask < (size_t(1) << n); mask++) {\n        for (size_t i = 0; i < n; i++) {\n            pattern[i] = (mask >> i) & 1 ? write_type::zero_copy : write_type::buffered;\n        }\n        fn(pattern);\n    }\n}\n#else\n// In SEASTAR_DEBUG builds, sample ~10% of patterns to keep sanitizer\n// run times acceptable. The all-buffered (0...0) and all-zero-copy\n// (1...1) patterns are always included.\ntemplate <typename Fn>\nstatic void for_each_type_pattern(size_t n, Fn fn) {\n    const size_t total = size_t(1) << n;\n    size_t sample_size = std::max(size_t(2), size_t(std::round(total * 0.1)));\n\n    std::vector<size_t> masks(total);\n    std::iota(masks.begin(), masks.end(), 0);\n    std::shuffle(masks.begin(), masks.end(), seastar::testing::local_random_engine);\n    // Ensure all-buffered (0) and all-zero-copy (total-1) are always first.\n    std::swap(*std::find(masks.begin(), masks.end(), size_t(0)), masks[0]);\n    std::swap(*std::find(masks.begin(), masks.end(), total - 1), masks[1]);\n\n    std::vector<write_type> pattern(n);\n    for (size_t i = 0; i < sample_size; i++) {\n        for (size_t j = 0; j < n; j++) {\n            pattern[j] = (masks[i] >> j) & 1 ? write_type::zero_copy : write_type::buffered;\n        }\n        fn(pattern);\n    }\n}\n#endif\n\n// Builds a string of `len` bytes filled with a cycling pattern,\n// so that data integrity failures produce readable diffs.\nstatic std::string make_data(size_t len) {\n    std::string s(len, '\\0');\n    for (size_t i = 0; i < len; i++) {\n        s[i] = 'a' + (i % 26);\n    }\n    return s;\n}\n\nSEASTAR_THREAD_TEST_CASE(test_splitting_invariants) {\n    for (bool trim_to_size : {false, true}) {\n        for_each_chunk_combination([&](const std::vector<size_t>& chunk_sizes) {\n            for_each_type_pattern(chunk_sizes.size(), [&](const std::vector<write_type>& write_types) {\n                std::string received_data;\n                std::vector<size_t> out_chunk_sizes;\n                auto mk = stream_maker().trim(trim_to_size).size(STREAM_SIZE);\n                auto out = mk(data_sink(std::make_unique<invariant_checker_sink>(\n                        received_data, out_chunk_sizes)));\n\n                std::string expected_data;\n                for (size_t i = 0; i < chunk_sizes.size(); i++) {\n                    auto data = make_data(chunk_sizes[i]);\n                    expected_data += data;\n                    switch (write_types[i]) {\n                    case write_type::buffered:\n                        out->write(data).get();\n                        break;\n                    case write_type::zero_copy:\n                        out->write(temporary_buffer<char>::copy_of(data)).get();\n                        break;\n                    }\n                }\n                out->close().get();\n                check_invariants(expected_data, out_chunk_sizes, received_data,\n                        STREAM_SIZE, trim_to_size,\n                        format_context(chunk_sizes, STREAM_SIZE, trim_to_size, write_types));\n            });\n        });\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_flush_on_empty_buffer_does_not_push_empty_packet_down_stream) {\n    std::stringstream ss;\n    auto out = output_stream<char>(testing::memory_data_sink(ss), 8);\n\n    out.flush().get();\n    BOOST_REQUIRE(ss.str().empty());\n}\n\nSEASTAR_THREAD_TEST_CASE(test_simple_write) {\n    std::stringstream ss;\n    auto out = output_stream<char>(testing::memory_data_sink(ss), 8);\n\n    auto value1 = sstring(\"te\");\n    out.write(value1).get();\n\n\n    auto value2 = sstring(\"st\");\n    out.write(value2).get();\n\n    auto value3 = sstring(\"abcdefgh1234\");\n    out.write(value3).get();\n\n    out.close().get();\n\n    auto value = value1 + value2 + value3;\n\n    BOOST_REQUIRE_EQUAL(ss.str(), value);\n}\n\nnamespace seastar::testing {\n\nclass output_stream_test {\npublic:\n    static bool has_buffer(const ::output_stream<char>& out) {\n        return out._end;\n    }\n    static bool has_zc(const ::output_stream<char>& out) {\n        return !out._zc_bufs.empty();\n    }\n};\n\n}\n\nSEASTAR_THREAD_TEST_CASE(test_mixed_mode_write) {\n    std::stringstream ss;\n    auto out = output_stream<char>(testing::memory_data_sink(ss), 8);\n\n    // First -- put some data in \"buffered\" mode and check that\n    // stream gains a buffer but not a zc packet\n    out.write(\"te\", 2).get();\n    BOOST_REQUIRE(testing::output_stream_test::has_buffer(out) && !testing::output_stream_test::has_zc(out));\n    // Second -- append some zero-copy buffer and check that the\n    // buffer disappears in favor of a bunch of zc packets (implementation detail, but still)\n    out.write(temporary_buffer<char>(\"st\", 2)).get();\n    BOOST_REQUIRE(!testing::output_stream_test::has_buffer(out) && testing::output_stream_test::has_zc(out));\n\n    // Finally -- all data must go away after flush\n    out.flush().get();\n    BOOST_REQUIRE(!testing::output_stream_test::has_buffer(out) && !testing::output_stream_test::has_zc(out));\n\n    out.close().get();\n\n    BOOST_REQUIRE_EQUAL(ss.str(), \"test\");\n}\n\n// Simple (mainly compilation) test for basic_memory_data_sink implementation over standard collections\ntemplate <template <typename T> class Col>\nvoid do_test_memory_data_sink() {\n    using Collection = Col<temporary_buffer<char>>;\n    Collection col;\n    auto s = data_sink(std::make_unique<util::basic_memory_data_sink<Collection>>(col));\n    for (unsigned i = 0; i < 3; i++) {\n        s.put(temporary_buffer<char>::copy_of(fmt::to_string(i))).get();\n    }\n    BOOST_REQUIRE_EQUAL(col.size(), 3);\n    auto it = col.begin();\n    for (unsigned i = 0; i < 3; i++) {\n        BOOST_REQUIRE_EQUAL(internal::to_sstring<std::string>(*it), fmt::to_string(i));\n        it++;\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_memory_data_sink) {\n    do_test_memory_data_sink<std::vector>();\n    do_test_memory_data_sink<std::list>();\n    do_test_memory_data_sink<std::deque>();\n}\n"
  },
  {
    "path": "tests/unit/packet_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/net/packet.hh>\n#include <array>\n\nusing namespace seastar;\nusing namespace net;\n\nBOOST_AUTO_TEST_CASE(test_many_fragments) {\n    std::vector<char> expected;\n\n    auto append = [&expected] (net::packet p, char c, size_t n) {\n        auto tmp = temporary_buffer<char>(n);\n        std::fill_n(tmp.get_write(), n, c);\n        std::fill_n(std::back_inserter(expected), n, c);\n        return net::packet(std::move(p), std::move(tmp));\n    };\n\n    net::packet p;\n    p = append(std::move(p), 'a', 5);\n    p = append(std::move(p), 'b', 31);\n    p = append(std::move(p), 'c', 65);\n    p = append(std::move(p), 'c', 4096);\n    p = append(std::move(p), 'd', 4096);\n\n    auto verify = [&expected] (const net::packet& p) {\n        BOOST_CHECK_EQUAL(p.len(), expected.size());\n        auto expected_it = expected.begin();\n        for (auto&& frag : p.fragments()) {\n            BOOST_CHECK_LE(frag.size, static_cast<size_t>(expected.end() - expected_it));\n            BOOST_CHECK(std::equal(frag.base, frag.base + frag.size, expected_it));\n            expected_it += frag.size;\n        }\n    };\n\n    auto trim_front = [&expected] (net::packet& p, size_t n) {\n        p.trim_front(n);\n        expected.erase(expected.begin(), expected.begin() + n);\n    };\n\n    verify(p);\n\n    trim_front(p, 1);\n    verify(p);\n\n    trim_front(p, 6);\n    verify(p);\n\n    trim_front(p, 29);\n    verify(p);\n\n    trim_front(p, 1024);\n    verify(p);\n\n    net::packet p2;\n    p2 = append(std::move(p2), 'z', 9);\n    p2 = append(std::move(p2), 'x', 7);\n\n    p.append(std::move(p2));\n    verify(p);\n}\n\nBOOST_AUTO_TEST_CASE(test_headers_are_contiguous) {\n    using tcp_header = std::array<char, 20>;\n    using ip_header = std::array<char, 20>;\n    char data[1000] = {};\n    fragment f{data, sizeof(data)};\n    packet p(f);\n    p.prepend_header<tcp_header>();\n    p.prepend_header<ip_header>();\n    BOOST_REQUIRE_EQUAL(p.nr_frags(), 2u);\n}\n\nBOOST_AUTO_TEST_CASE(test_headers_are_contiguous_even_with_small_fragment) {\n    using tcp_header = std::array<char, 20>;\n    using ip_header = std::array<char, 20>;\n    char data[100] = {};\n    fragment f{data, sizeof(data)};\n    packet p(f);\n    p.prepend_header<tcp_header>();\n    p.prepend_header<ip_header>();\n    BOOST_REQUIRE_EQUAL(p.nr_frags(), 2u);\n}\n\nBOOST_AUTO_TEST_CASE(test_headers_are_contiguous_even_with_many_fragments) {\n    using tcp_header = std::array<char, 20>;\n    using ip_header = std::array<char, 20>;\n    char data[100] = {};\n    fragment f{data, sizeof(data)};\n    packet p(f);\n    for (int i = 0; i < 7; ++i) {\n        p.append(packet(f));\n    }\n    p.prepend_header<tcp_header>();\n    p.prepend_header<ip_header>();\n    BOOST_REQUIRE_EQUAL(p.nr_frags(), 9u);\n}\n\n"
  },
  {
    "path": "tests/unit/pipe_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\"). See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership. You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied. See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2021-present ScyllaDB\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/pipe.hh>\n\nusing namespace seastar;\n\nstatic_assert(!std::is_default_constructible_v<seastar::pipe_reader<int>>);\nstatic_assert(!std::is_default_constructible_v<seastar::pipe_writer<int>>);\n\nstatic_assert(std::is_nothrow_move_constructible_v<seastar::pipe_reader<int>>);\nstatic_assert(std::is_nothrow_move_assignable_v<seastar::pipe_reader<int>>);\n\nstatic_assert(std::is_nothrow_move_constructible_v<seastar::pipe_writer<int>>);\nstatic_assert(std::is_nothrow_move_assignable_v<seastar::pipe_writer<int>>);\n\nSEASTAR_THREAD_TEST_CASE(simple_pipe_test) {\n    seastar::pipe<int> p(1);\n\n    auto f0 = p.reader.read();\n    BOOST_CHECK(!f0.available());\n    p.writer.write(17).get();\n    BOOST_REQUIRE_EQUAL(*f0.get(), 17);\n\n    p.writer.write(42).get();\n    auto f2 = p.reader.read();\n    BOOST_CHECK(f2.available());\n    BOOST_REQUIRE_EQUAL(*f2.get(), 42);\n}\n"
  },
  {
    "path": "tests/unit/program_options_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <seastar/util/program-options.hh>\n\n#include <boost/program_options.hpp>\n#include <boost/test/unit_test.hpp>\n\n#include <initializer_list>\n#include <vector>\n\nnamespace bpo = boost::program_options;\n\nusing namespace seastar;\n\nstatic bpo::variables_map parse(const bpo::options_description& desc, std::initializer_list<const char*> args) {\n    std::vector<const char*> raw_args{\"program_options_test\"};\n    for (const char* arg : args) {\n        raw_args.push_back(arg);\n    }\n\n    bpo::variables_map vars;\n    bpo::store(bpo::parse_command_line(raw_args.size(), raw_args.data(), desc), vars);\n    bpo::notify(vars);\n\n    return vars;\n}\n\nBOOST_AUTO_TEST_CASE(string_map) {\n    bpo::options_description desc;\n    desc.add_options()\n            (\"ages\", bpo::value<program_options::string_map>());\n\n    const auto vars = parse(desc, {\"--ages\", \"joe=15:sally=20\", \"--ages\", \"phil=18:joe=11\"});\n    const auto& ages = vars[\"ages\"].as<program_options::string_map>();\n\n    // `string_map` values can be specified multiple times. The last association takes precedence.\n    BOOST_REQUIRE_EQUAL(ages.at(\"joe\"), \"11\");\n    BOOST_REQUIRE_EQUAL(ages.at(\"phil\"), \"18\");\n    BOOST_REQUIRE_EQUAL(ages.at(\"sally\"), \"20\");\n\n    BOOST_REQUIRE_THROW(parse(desc, {\"--ages\", \"tim:\"}), bpo::invalid_option_value);\n}\n"
  },
  {
    "path": "tests/unit/prometheus_http_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2024 ScyllaDB\n */\n\n#include \"loopback_socket.hh\"\n\n#include <seastar/core/metrics.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/http/common.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/util/closeable.hh>\n\n#include <boost/test/tools/old/interface.hpp>\n\nusing namespace seastar;\nusing namespace httpd;\nusing namespace std::literals;\n\nnamespace {\n\nstruct test_metrics {\n    metrics::metric_groups _metrics;\n\n    void setup_metrics() {\n        auto somelabel = metrics::label(\"somekey\");\n\n        _metrics.add_group(\"aaaa\", {\n            metrics::make_gauge(\"escaped_label_value_test\", [] { return 10; }, metrics::description{\"test that special characters are escaped\"}, {somelabel(R\"(special\"\\nvalue)\")}),\n            metrics::make_gauge(\"int_test\", [] { return 10; }, metrics::description{\"simple minimal test\"}),\n            metrics::make_gauge(\"double_test\", [] { return 1234567654321.0; }, metrics::description{\"test that a long double is printed fully and not in scientific notation\"}),\n            metrics::make_counter(\"counter_test\", [] () -> int64_t { return 1234567654321; }, metrics::description{\"test with a long counter value\"}),\n        });\n    }\n};\n\nstruct test_case {\n    // iff true, add a global label in prometheus config\n    bool use_global_label;\n};\n\nstatic const metrics::label_instance global_label{\"global\", \"foo\"};\n\nfuture<> test_prometheus_metrics_body(test_case tc) {\n\n    test_metrics metrics;\n    metrics.setup_metrics();\n\n    co_await seastar::async([tc] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        prometheus::config ctx;\n        if (tc.use_global_label) {\n            ctx.label = global_label;\n        }\n        add_prometheus_routes(server, ctx).get();\n\n        future<> client = seastar::async([&lsi, tc] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            auto close_input = deferred_close(input);\n            output_stream<char> output(c_socket.output());\n            auto close_output = deferred_close(output);\n\n            output.write(sstring(\"GET /metrics HTTP/1.1\\r\\nHost: test\\r\\n\\r\\n\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            auto resp_str = std::string(resp.get(), resp.size());\n            BOOST_REQUIRE(std::ranges::search(resp_str, \"200 OK\"sv));\n\n            auto global_label_str = tc.use_global_label ? fmt::format(\"{}=\\\"{}\\\",\", global_label.key(), global_label.value()) : std::string{};\n\n            std::string expected0 = R\"(seastar_aaaa_escaped_label_value_test{)\" + global_label_str + R\"(shard=\"0\",somekey=\"special\\\"\\\\nvalue\"} 10.000000)\";\n            std::string expected1 = R\"(seastar_aaaa_int_test{)\" + global_label_str + R\"(shard=\"0\"} 10.000000)\";\n            std::string expected2 = R\"(seastar_aaaa_double_test{)\" + global_label_str + R\"(shard=\"0\"} 1234567654321.000000)\";\n            std::string expected3 = R\"(seastar_aaaa_counter_test{)\" + global_label_str + R\"(shard=\"0\"} 1234567654321)\";\n\n            auto all_expected = {expected0, expected1, expected2, expected3};\n\n            for (auto& expected : all_expected) {\n                BOOST_REQUIRE_MESSAGE(std::ranges::search(resp_str, expected),\n                    fmt::format(\"cannot find: {}\\nResponse: {}\\n\", expected, resp_str));\n            }\n        });\n\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n\n}\n\nSEASTAR_TEST_CASE(test_prometheus_metrics) {\n    return test_prometheus_metrics_body({.use_global_label = false});\n}\n\nSEASTAR_TEST_CASE(test_prometheus_metrics_global_label) {\n    return test_prometheus_metrics_body({.use_global_label = true});\n}\n\n// Test that multiple __name__ query parameters filter to only matching metrics\nSEASTAR_TEST_CASE(test_prometheus_multiple_name_filters) {\n    // Create metrics with distinct names\n    metrics::metric_groups test_metrics;\n    test_metrics.add_group(\"test\", {\n        metrics::make_gauge(\"metric_alpha\", [] { return 1; }, metrics::description{\"alpha metric\"}),\n        metrics::make_gauge(\"metric_beta\", [] { return 2; }, metrics::description{\"beta metric\"}),\n        metrics::make_gauge(\"metric_gamma\", [] { return 3; }, metrics::description{\"gamma metric\"}),\n    });\n\n    co_await seastar::async([] {\n        loopback_connection_factory lcf(1);\n        http_server server(\"test\");\n        loopback_socket_impl lsi(lcf);\n        httpd::http_server_tester::listeners(server).emplace_back(lcf.get_server_socket());\n\n        prometheus::config ctx;\n        add_prometheus_routes(server, ctx).get();\n\n        future<> client = seastar::async([&lsi] {\n            connected_socket c_socket = lsi.connect(socket_address(ipv4_addr()), socket_address(ipv4_addr())).get();\n            input_stream<char> input(c_socket.input());\n            auto close_input = deferred_close(input);\n            output_stream<char> output(c_socket.output());\n            auto close_output = deferred_close(output);\n\n            // Request only metric_alpha and metric_gamma using multiple __name__ parameters\n            output.write(sstring(\"GET /metrics?__name__=test_metric_alpha&__name__=test_metric_gamma HTTP/1.1\\r\\nHost: test\\r\\n\\r\\n\")).get();\n            output.flush().get();\n            auto resp = input.read().get();\n            auto resp_str = std::string(resp.get(), resp.size());\n            BOOST_REQUIRE(std::ranges::search(resp_str, \"200 OK\"sv));\n\n            // Should contain alpha and gamma\n            BOOST_REQUIRE_MESSAGE(std::ranges::search(resp_str, \"seastar_test_metric_alpha\"sv),\n                fmt::format(\"should contain metric_alpha\\nResponse: {}\\n\", resp_str));\n            BOOST_REQUIRE_MESSAGE(std::ranges::search(resp_str, \"seastar_test_metric_gamma\"sv),\n                fmt::format(\"should contain metric_gamma\\nResponse: {}\\n\", resp_str));\n\n            // Should NOT contain beta\n            BOOST_REQUIRE_MESSAGE(!std::ranges::search(resp_str, \"seastar_test_metric_beta\"sv),\n                fmt::format(\"should NOT contain metric_beta\\nResponse: {}\\n\", resp_str));\n        });\n\n        server.do_accepts(0).get();\n\n        client.get();\n        server.stop().get();\n    });\n}\n"
  },
  {
    "path": "tests/unit/prometheus_test.py",
    "content": "#!/usr/bin/env python3\n#\n# This file is open source software, licensed to you under the terms\n# of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n# distributed with this work for additional information regarding copyright\n# ownership.  You may not use this file except in compliance with the License.\n#\n# You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n#\n# Copyright (C) 2024 Scylladb, Ltd.\n#\n\nimport argparse\nimport math\nimport json\nimport re\nimport subprocess\nimport sys\nimport time\nimport unittest\nimport urllib.request\nimport urllib.parse\nimport yaml\n\nfrom typing import Optional\nfrom collections import namedtuple\n\n\nclass Exposition:\n    @classmethod\n    def from_hist(cls,\n                  name: str,\n                  hist: list[tuple[float, int]],\n                  sum_: int,\n                  count: int) -> 'Exposition':\n        # ignore these values, we might need to verify them in future\n        _, _ = sum_, count\n        buckets = (cls.value_to_bucket(le - 1) for le, _ in hist)\n        deltas = []\n        last_n = 0\n        for _, n in hist:\n            delta = n - last_n\n            last_n = n\n            deltas.append(delta)\n        return cls(name, dict(zip(buckets, deltas)), {})\n\n    @staticmethod\n    def value_to_bucket(value):\n        low = 2 ** math.floor(math.log(value, 2))\n        high = 2 * low\n        dif = (high - low) / 4\n        return low + dif * math.floor((value - low) / dif)\n\n    @staticmethod\n    def _values_to_histogram(values):\n        hist = {}\n        for val in values:\n            bucket = Exposition.value_to_bucket(val)\n            if bucket in hist:\n                hist[bucket] += 1\n            else:\n                hist[bucket] = 1\n        return hist\n\n    @classmethod\n    def from_conf(cls,\n                  name: str,\n                  type_: str,\n                  values: list[str],\n                  labels: dict[str, str]) -> 'Exposition':\n        if type_ in ('gauge', 'counter'):\n            assert len(values) == 1\n            return cls(name, float(values[0]), labels)\n        if type_ == 'histogram':\n            hist = cls._values_to_histogram(float(v) for v in values)\n            return cls(name, hist, {})\n        raise NotImplementedError(f'unsupported type: {type_}')\n\n    def __init__(self,\n                 name: str,\n                 value: int | list[tuple[float, int]],\n                 labels: dict[str, str]) -> None:\n        self.name = name\n        self.value = value\n        self.labels = labels\n\n    def __repr__(self):\n        return f\"{self.name=}, {self.value=}, {self.labels=}\"\n\n    def __eq__(self, other):\n        if not isinstance(other, Exposition):\n            return False\n        return self.value == other.value\n\n\nclass Metrics:\n    prefix = 'seastar'\n    group = 'test_group'\n    # parse lines like:\n    # rest_api_scheduler_queue_length{group=\"main\",shard=\"0\"} 0.000000\n    # where:\n    #   - \"rest_api\" is the prometheus prefix\n    #   - \"scheduler\" is the metric group name\n    #   - \"queue_length\" is the name of the metric\n    #   - the kv pairs in \"{}\" are labels\"\n    #   - \"0.000000\" is the value of the metric\n    # this format is compatible with\n    # https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md\n    # NOTE: scylla does not include timestamp in the exported metrics\n    pattern = re.compile(r'''(?P<metric_name>\\w+)   # rest_api_scheduler_queue_length\n                             \\{(?P<labels>[^\\}]*)\\} # {group=\"main\",shard=\"0\"}\n                             \\s+                    # <space>\n                             (?P<value>[^\\s]+)      # 0.000000''', re.X)\n\n    def __init__(self, lines: list[str]) -> None:\n        self.lines: list[str] = lines\n\n    @classmethod\n    def full_name(cls, name: str) -> str:\n        '''return the full name of a metrics\n        '''\n        return f'{cls.group}_{name}'\n\n    @staticmethod\n    def _parse_labels(s: str) -> dict[str, str]:\n        return dict(name_value.split('=', 1) for name_value in s.split(','))\n\n    def get(self,\n            name: Optional[str] = None,\n            labels: Optional[dict[str, str]] = None) -> list[Exposition]:\n        '''Return all expositions matching the given name and labels\n        '''\n        full_name = None\n        if name is not None:\n            full_name = f'{self.prefix}_{self.group}_{name}'\n        metric_type = None\n\n        # for histogram and summary as they are represented with multiple lines\n        hist_name = ''\n        hist_buckets = []\n        hist_sum = 0\n        hist_count = 0\n\n        for line in self.lines:\n            if not line:\n                continue\n            if line.startswith('# HELP'):\n                continue\n            if line.startswith('# TYPE'):\n                _, _, type_metric_name, metric_type = line.split()\n                if hist_buckets:\n                    yield Exposition.from_hist(hist_name,\n                                               hist_buckets,\n                                               hist_sum,\n                                               hist_count)\n                    hist_buckets = []\n                if metric_type in ('histogram', 'summary'):\n                    hist_name = type_metric_name\n                continue\n            matched = self.pattern.match(line)\n            assert matched, f'malformed metric line: {line}'\n\n            value_metric_name = matched.group('metric_name')\n            if full_name and not value_metric_name.startswith(full_name):\n                continue\n\n            metric_labels = self._parse_labels(matched.group('labels'))\n            if labels is not None and metric_labels != labels:\n                continue\n\n            metric_value = float(matched.group('value'))\n            if metric_type == 'histogram':\n                if value_metric_name == f'{type_metric_name}_bucket':\n                    last_value = 0\n                    if hist_buckets:\n                        last_value = hist_buckets[-1][1]\n                    if metric_value - last_value != 0:\n                        le = metric_labels['le'].strip('\"')\n                        hist_buckets.append((float(le), metric_value))\n                elif value_metric_name == f'{type_metric_name}_sum':\n                    hist_sum = metric_value\n                elif value_metric_name == f'{type_metric_name}_count':\n                    hist_count = metric_value\n                else:\n                    raise RuntimeError(f'unknown histogram value: {line}')\n            elif metric_type == 'summary':\n                raise NotImplementedError('unsupported type: summary')\n            else:\n                yield Exposition(type_metric_name,\n                                 metric_value,\n                                 metric_labels)\n        if hist_buckets:\n            yield Exposition.from_hist(hist_name,\n                                       hist_buckets,\n                                       hist_sum,\n                                       hist_count)\n\n    def get_help(self, name: str) -> Optional[str]:\n        full_name = f'{self.prefix}_{self.group}_{name}'\n        header = f'# HELP {full_name}'\n        for line in self.lines:\n            if line.startswith(header):\n                tokens = line.split(maxsplit=3)\n                return tokens[-1]\n        return None\n\n\nclass TestPrometheus(unittest.TestCase):\n    exporter_path = None\n    exporter_process = None\n    exporter_config = None\n    port = 10001\n    prometheus = None\n    prometheus_scrape_interval = 15\n\n    @classmethod\n    def setUpClass(cls) -> None:\n        args = [cls.exporter_path,\n                '--port', f'{cls.port}',\n                '--conf', cls.exporter_config,\n                '--smp=2']\n        cls.exporter_process = subprocess.Popen(args,\n                                                stdout=subprocess.PIPE,\n                                                stderr=subprocess.DEVNULL,\n                                                bufsize=0, text=True)\n        # wait until the server is ready for serve\n        cls.exporter_process.stdout.readline()\n\n    @classmethod\n    def tearDownClass(cls) -> None:\n        cls.exporter_process.terminate()\n\n    @classmethod\n    def _get_metrics(cls,\n                     name: Optional[str] = None,\n                     labels: Optional[dict[str, str]] = None,\n                     with_help: bool = True,\n                     aggregate: bool = True) -> Metrics:\n        query: dict[str, str] = {}\n        if name is not None:\n            query['__name__'] = name\n        if labels is not None:\n            query.update(labels)\n        if not with_help:\n            query['__help__'] = 'false'\n        if not aggregate:\n            query['__aggregate__'] = 'false'\n        params = urllib.parse.urlencode(query)\n        host = 'localhost'\n        url = f'http://{host}:{cls.port}/metrics?{params}'\n        with urllib.request.urlopen(url) as f:\n            body = f.read().decode('utf-8')\n            return Metrics(body.rstrip().split('\\n'))\n\n    def test_filtering_by_label_sans_aggregation(self) -> None:\n        labels = {'private': '1'}\n        metrics = self._get_metrics(labels=labels)\n        actual_values = list(metrics.get())\n        expected_values = []\n        with open(self.exporter_config, encoding='utf-8') as f:\n            config = yaml.safe_load(f)\n        for metric in config['metrics']:\n            name = metric['name']\n            metric_name = f'{Metrics.prefix}_{Metrics.group}_{name}'\n            metric_labels = metric['labels']\n            if metric_labels != labels:\n                continue\n            e = Exposition.from_conf(metric_name,\n                                     metric['type'],\n                                     metric['values'],\n                                     metric_labels)\n            expected_values.append(e)\n        self.assertCountEqual(actual_values, expected_values)\n\n    def test_filtering_by_label_with_aggregation(self) -> None:\n        TestCase = namedtuple('TestCase', ['label', 'regex', 'found'])\n        label = 'private'\n        tests = [\n            TestCase(label=label, regex='dne', found=0),\n            TestCase(label=label, regex='404', found=0),\n            TestCase(label=label, regex='2', found=1),\n            # aggregated\n            TestCase(label=label, regex='2|3', found=1),\n        ]\n        for test in tests:\n            with self.subTest(regex=test.regex, found=test.found):\n                metrics = self._get_metrics(labels={test.label: test.regex})\n                values = list(metrics.get())\n                self.assertEqual(len(values), test.found)\n\n    def test_aggregated(self) -> None:\n        name = 'counter_1'\n        # see also rest_api_httpd.cc::aggregate_by_name\n        TestCase = namedtuple('TestCase', ['aggregate', 'expected_values'])\n        tests = [\n            TestCase(aggregate=False, expected_values=[1, 2]),\n            TestCase(aggregate=True, expected_values=[3])\n        ]\n        for test in tests:\n            with self.subTest(aggregate=test.aggregate,\n                              values=test.expected_values):\n                metrics = self._get_metrics(Metrics.full_name(name), aggregate=test.aggregate)\n                expositions = metrics.get(name)\n                actual_values = sorted(e.value for e in expositions)\n                self.assertEqual(actual_values, test.expected_values)\n\n    def test_help(self) -> None:\n        name = 'counter_1'\n        tests = [True, False]\n        for with_help in tests:\n            with self.subTest(with_help=with_help):\n                metrics = self._get_metrics(Metrics.full_name(name), with_help=with_help)\n                msg = metrics.get_help(name)\n                if with_help:\n                    self.assertIsNotNone(msg)\n                else:\n                    self.assertIsNone(msg)\n\n    @staticmethod\n    def _from_native_histogram(values) -> dict[float, float]:\n        results = {}\n        for v in values:\n            bucket = Exposition.value_to_bucket(float(v[2]) - 1)\n            results[bucket] = float(v[3])\n        return results\n\n    @staticmethod\n    def _query_prometheus(host: str, query: str, type_: str) -> float | dict[float, float]:\n        url = f'http://{host}/api/v1/query?query={query}'\n        headers = {\"Accept\": \"application/json\"}\n        req = urllib.request.Request(url, headers=headers)\n        with urllib.request.urlopen(req) as f:\n            results = json.load(f)[\"data\"][\"result\"][0]\n            if type_ == 'histogram':\n                buckets = results[\"histogram\"][1][\"buckets\"]\n                return TestPrometheus._from_native_histogram(buckets)\n            return float(results[\"value\"][1])\n\n    def test_protobuf(self) -> None:\n        if self.prometheus is None:\n            self.skipTest(\"prometheus is not configured\")\n\n        # Prometheus does not allow us to push metrics to it, neither\n        # can we force it to scrape an exporter, so we have to wait\n        # until prometheus scrapes the server\n        time.sleep(self.prometheus_scrape_interval + 1)\n        with open(self.exporter_config, encoding='utf-8') as f:\n            config = yaml.safe_load(f)\n\n        labels = {'private': '1'}\n        for metric in config['metrics']:\n            name = metric['name']\n            metric_name = f'{Metrics.prefix}_{Metrics.group}_{name}'\n            metric_labels = metric['labels']\n            if metric_labels != labels:\n                continue\n            metric_type = metric['type']\n            metric_value = metric['values']\n            e = Exposition.from_conf(metric_name,\n                                     metric_type,\n                                     metric_value,\n                                     metric_labels)\n            res = self._query_prometheus(self.prometheus,\n                                         metric_name,\n                                         metric_type)\n            self.assertEqual(res, e.value)\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--exporter',\n                        required=True,\n                        help='Path to the exporter executable')\n    parser.add_argument('--config',\n                        required=True,\n                        help='Path to the metrics definition file')\n    parser.add_argument('--prometheus',\n                        help='A Prometheus to connect to')\n    parser.add_argument('--prometheus-scrape-interval',\n                        type=int,\n                        help='Prometheus scrape interval (in seconds)',\n                        default=15)\n    opts, remaining = parser.parse_known_args()\n    remaining.insert(0, sys.argv[0])\n    TestPrometheus.exporter_path = opts.exporter\n    TestPrometheus.exporter_config = opts.config\n    TestPrometheus.prometheus = opts.prometheus\n    TestPrometheus.prometheus_scrape_interval = opts.prometheus_scrape_interval\n    unittest.main(argv=remaining)\n"
  },
  {
    "path": "tests/unit/prometheus_text_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n#include <boost/test/tools/old/interface.hpp>\n#include <cstddef>\n#include <seastar/core/internal/estimated_histogram.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/metrics.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/core/metrics_registration.hh>\n#include <seastar/core/prometheus.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/closeable.hh>\n\n#include \"core/prometheus-impl.hh\"\n#include \"memory-data-sink.hh\"\n\n#include <sstream>\n#include <string_view>\n\n\nusing namespace seastar;\nusing namespace httpd;\nusing namespace std::literals;\nusing namespace seastar::prometheus;\n\nnamespace sm = seastar::metrics;\nnamespace sp = seastar::prometheus;\nnamespace mi = sm::impl;\n\nusing labels_list_type = std::vector<std::string>;\n\nthread_local auto impl_ = sm::impl::get_local_impl();\n\nnamespace {\n[[maybe_unused]] void remove_existing_metrics() {\n    const auto& map = seastar::metrics::impl::get_value_map();\n\n    for (auto& family : map) {\n        auto name = family.first;\n        for (auto& series: family.second) {\n            seastar::metrics::impl::unregister_metric(series.second->get_id());\n        }\n    }\n\n    assert(seastar::metrics::impl::get_value_map().size() == 0);\n}\n\n}\n\nusing data_type = seastar::metrics::impl::data_type;\n\nstatic const sp::details::filter_t always_true = [](auto& mi){ return true; };\n\nenum class aggr_mode {\n    NO_AGGR,\n    AGGR_LABEL_0,\n    AGGR_SHARD_LABEL,\n};\n\nstruct test_config {\n    data_type type;\n    // number of metrics to create\n    size_t count = 1;\n    // number of labels on each metric\n    size_t labels_per_metric = 1;\n    std::optional<sp::details::filter_t> filter;\n    std::optional<sp::details::family_filter_t> family_filter;\n    bool show_help = true;\n    aggr_mode aggregation_mode = aggr_mode::NO_AGGR;\n    bool same_metric_name = false;\n    std::optional<sm::label_instance> extra_label;\n};\n\nstatic constexpr uint64_t histo_min = 1, histo_max = 1000000;\nusing histo_type = sm::internal::approximate_exponential_histogram<histo_min, histo_max, 1>;\n\nauto make_historgam() {\n    auto histogram = std::make_shared<histo_type>();\n    for (double v = histo_min; v < histo_max; v *= 1.3) {\n        histogram->add(v);\n    }\n    return histogram;\n}\n\nstruct prometheus_test_fixture {\n    static constexpr uint64_t histo_min = 1, histo_max = 1000000;\n    using histo_type = sm::internal::approximate_exponential_histogram<histo_min, histo_max, 1>;\n    const int histo_buckets = histo_type{}.find_bucket_index(-1) + 1;\n\n    static constexpr size_t name_length = 10;\n\n    static seastar::future<> run_metrics_test(test_config test_conf, prometheus::config config, std::string_view expected) {\n\n        co_await smp::invoke_on_all([] {\n            remove_existing_metrics();\n        });\n\n        sm::metric_groups test_metrics;\n\n        auto nth_label = [](size_t n) {\n            return sm::label(fmt::format(\"label-{}\", n));\n        };\n\n        std::vector<sm::metric_definition> defs;\n\n        auto desc = \"metric description\";\n\n        for (size_t i = 0; i < test_conf.count; ++i) {\n            auto metric_name = test_conf.same_metric_name ? \"metric\" : fmt::format(\"metric_{}\", i);\n            std::vector<sm::label_instance> labels;\n\n            for (size_t label_idx = 0; label_idx < test_conf.labels_per_metric; ++label_idx) {\n                auto label_value = fmt::format(\"label-{}-{}\", label_idx, i);\n                labels.push_back(nth_label(label_idx)(label_value));\n            }\n\n            if (test_conf.extra_label) {\n                labels.push_back(*test_conf.extra_label);\n            }\n\n            sm::impl::metric_definition_impl impl = [&] {\n                if (test_conf.type == data_type::COUNTER) {\n                    return\n                    sm::make_counter(metric_name, sm::description(desc), labels, [] { return 123; });\n                } else if (test_conf.type == data_type::REAL_COUNTER) {\n                    return\n                    sm::make_counter(metric_name, sm::description(desc), labels, [] { return 123.4; });\n                } else if (test_conf.type == data_type::GAUGE) {\n                    return\n                    sm::make_gauge(metric_name, sm::description(desc), labels, [] { return 123.4; });\n                } else if (test_conf.type == data_type::HISTOGRAM) {\n                    return make_histogram(metric_name, sm::description(metric_name), labels,\n                            [histogram = make_historgam()]() { return histogram->to_metrics_histogram(); });\n                } else if (test_conf.type == data_type::SUMMARY) {\n                    // SUMMARY doesn't support specifying labels\n                    return make_summary(metric_name, sm::description(metric_name),\n                            [histogram = make_historgam()]() { return histogram->to_metrics_histogram(); });\n                }\n                BOOST_FAIL(\"unknown data type\");\n                __builtin_unreachable();\n            }();\n\n            if (test_conf.aggregation_mode == aggr_mode::AGGR_LABEL_0) {\n                impl.aggregate({nth_label(0)});\n            } else if (test_conf.aggregation_mode == aggr_mode::AGGR_SHARD_LABEL) {\n                impl.aggregate({sm::shard_label});\n            }\n\n            defs.emplace_back(impl);\n\n        }\n\n        test_metrics.add_group(fmt::format(\"group-{}\", 1), defs);\n\n        using access = prometheus::details::test_access;\n\n        std::stringstream ss;\n        output_stream<char> out{data_sink{std::make_unique<testing::memory_data_sink_impl>(ss, 10)}};\n        auto filter = test_conf.filter.value_or(always_true);\n        auto family_filter = test_conf.family_filter.value_or([](std::string_view) { return true; });\n        co_await access{}.write_body(config,\n            sp::details::write_body_args{\n                .filter = filter,\n                .family_filter = family_filter,\n                .use_protobuf_format = false,\n                .show_help = test_conf.show_help,\n                .enable_aggregation = test_conf.aggregation_mode != aggr_mode::NO_AGGR\n            },\n            std::move(out));\n\n        BOOST_REQUIRE_MESSAGE(expected == ss.str(),\n            fmt::format(\"actual output doesn't match expected\\nexpected output:\\n{}\\nactual output:\\n{}\",\n            expected, ss.str()));\n    }\n};\n\nSEASTAR_TEST_CASE(test_basic_counter) {\n    test_config cfg{data_type::COUNTER};\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_basic_counter_no_labels) {\n    test_config cfg{data_type::COUNTER};\n    cfg.labels_per_metric = 0;\n    cfg.aggregation_mode = aggr_mode::AGGR_SHARD_LABEL;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_histo_no_labels) {\n    // important case b/c of how we inject the le label, which could\n    // fail in the case there are no other labels\n    test_config cfg{data_type::HISTOGRAM};\n    cfg.labels_per_metric = 0;\n    cfg.aggregation_mode = aggr_mode::AGGR_SHARD_LABEL;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric_0)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 histogram)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_sum{} 6.42072e+06)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_count{} 53)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"2.000000\"} 3)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"4.000000\"} 6)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"8.000000\"} 8)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"16.000000\"} 11)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"32.000000\"} 14)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"64.000000\"} 16)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"128.000000\"} 19)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"256.000000\"} 22)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"512.000000\"} 24)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"1024.000000\"} 27)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"2048.000000\"} 30)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"4096.000000\"} 32)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"8192.000000\"} 35)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"16384.000000\"} 37)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"32768.000000\"} 40)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"65536.000000\"} 43)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"131072.000000\"} 45)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"262144.000000\"} 48)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"1000000.000000\"} 51)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{le=\"+Inf\"} 53)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_basic_gauge) {\n    test_config cfg{data_type::GAUGE};\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 gauge)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123.400000)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_basic_histogram) {\n    test_config cfg{data_type::HISTOGRAM};\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric_0)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 histogram)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_sum{label-0=\"label-0-0\",shard=\"0\"} 6.42072e+06)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_count{label-0=\"label-0-0\",shard=\"0\"} 53)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"2.000000\",shard=\"0\"} 3)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"4.000000\",shard=\"0\"} 6)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"8.000000\",shard=\"0\"} 8)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"16.000000\",shard=\"0\"} 11)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"32.000000\",shard=\"0\"} 14)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"64.000000\",shard=\"0\"} 16)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"128.000000\",shard=\"0\"} 19)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"256.000000\",shard=\"0\"} 22)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"512.000000\",shard=\"0\"} 24)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"1024.000000\",shard=\"0\"} 27)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"2048.000000\",shard=\"0\"} 30)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"4096.000000\",shard=\"0\"} 32)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"8192.000000\",shard=\"0\"} 35)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"16384.000000\",shard=\"0\"} 37)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"32768.000000\",shard=\"0\"} 40)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"65536.000000\",shard=\"0\"} 43)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"131072.000000\",shard=\"0\"} 45)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"262144.000000\",shard=\"0\"} 48)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"1000000.000000\",shard=\"0\"} 51)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_bucket{label-0=\"label-0-0\",le=\"+Inf\",shard=\"0\"} 53)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_basic_summary) {\n    test_config cfg{data_type::SUMMARY};\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric_0)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 summary)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_sum{shard=\"0\"} 6.42072e+06)\" \"\\n\"\n        R\"(seastar_group_1_metric_0_count{shard=\"0\"} 53)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"2.000000\",shard=\"0\"} 3)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"4.000000\",shard=\"0\"} 6)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"8.000000\",shard=\"0\"} 8)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"16.000000\",shard=\"0\"} 11)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"32.000000\",shard=\"0\"} 14)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"64.000000\",shard=\"0\"} 16)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"128.000000\",shard=\"0\"} 19)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"256.000000\",shard=\"0\"} 22)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"512.000000\",shard=\"0\"} 24)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"1024.000000\",shard=\"0\"} 27)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"2048.000000\",shard=\"0\"} 30)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"4096.000000\",shard=\"0\"} 32)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"8192.000000\",shard=\"0\"} 35)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"16384.000000\",shard=\"0\"} 37)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"32768.000000\",shard=\"0\"} 40)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"65536.000000\",shard=\"0\"} 43)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"131072.000000\",shard=\"0\"} 45)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"262144.000000\",shard=\"0\"} 48)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{quantile=\"1000000.000000\",shard=\"0\"} 51)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_counter_with_custom_prefix) {\n    test_config cfg{data_type::COUNTER};\n    prometheus::config prom_cfg;\n    prom_cfg.prefix = \"myapp\";\n    return prometheus_test_fixture::run_metrics_test(cfg, prom_cfg,\n        R\"(# HELP myapp_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE myapp_group_1_metric_0 counter)\" \"\\n\"\n        R\"(myapp_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_counter_with_label) {\n    test_config cfg{data_type::COUNTER};\n    prometheus::config prom_cfg;\n    sm::label extra_label{\"env\"};\n    prom_cfg.label = extra_label(\"production\");\n    return prometheus_test_fixture::run_metrics_test(cfg, prom_cfg,\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{env=\"production\",label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_multiple_counters) {\n    test_config cfg{data_type::COUNTER, 3};\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_metrics_filtering) {\n    // Filter to exclude the first metric (label-0=\"label-0-0\")\n    // This should return only metric_1 and metric_2\n    sp::details::filter_t filter = [](const sm::impl::labels_type& labels) {\n        auto it = labels.find(\"label-0\");\n        return it == labels.end() || it->second.value() != \"label-0-0\";\n    };\n\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.filter = filter;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_counter_without_help) {\n    test_config cfg{data_type::COUNTER};\n    cfg.show_help = false;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_counter_with_aggregation) {\n    // Create 2 counters with the same metric name but different labels\n    // Aggregation combines metrics with the same name, sums their values (123+123=246),\n    // and drops the varying labels (keeping only common labels like \"shard\")\n    test_config cfg{data_type::COUNTER, 2};\n    cfg.aggregation_mode = aggr_mode::AGGR_LABEL_0;\n    cfg.same_metric_name = true;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric counter)\" \"\\n\"\n        R\"(seastar_group_1_metric{shard=\"0\"} 246)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_label_value_escaping) {\n    // Test that label values containing special characters (quotes, backslashes, newlines)\n    // are properly escaped according to Prometheus text format specification\n    test_config cfg{data_type::COUNTER};\n    sm::label label_special{\"special\"};\n    cfg.extra_label = label_special(\"value with \\\"quote\\\", \\\\backslash and \\nline break\");\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\",special=\"value with \\\"quote\\\", \\\\backslash and \\nline break\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_label_starting_with_double_underscore) {\n    // Test that labels starting with __ are filtered out and do not appear in the output\n    test_config cfg{data_type::COUNTER};\n    sm::label internal_label{\"__internal\"};\n    cfg.extra_label = internal_label(\"should_not_appear\");\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_basic_counter) {\n    // Create aggregator that aggregates by \"shard\" label\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    // Create two counter metrics with same name but different shard labels\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"counter1\";\n    labels1[\"shard\"] = \"0\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"counter1\";\n    labels2[\"shard\"] = \"1\";\n\n    // Add counter values\n    mi::metric_value value1(123, mi::data_type::COUNTER);\n    mi::metric_value value2(456, mi::data_type::COUNTER);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    // After aggregation, should have one metric with \"shard\" label removed\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 1);\n\n    // Check that the aggregated value has the correct labels (without \"shard\")\n    auto it = values.begin();\n    // Labels are already filtered (aggregated labels removed)\n    BOOST_REQUIRE_EQUAL(it->second.labels.size(), 1);\n    BOOST_REQUIRE_EQUAL(it->second.labels.at(\"name\").value(), \"counter1\");\n\n    // Check that values were summed: 123 + 456 = 579\n    BOOST_REQUIRE_EQUAL(it->second.m.d(), 579);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_multiple_labels) {\n    // Create aggregator that aggregates by \"shard\" and \"cpu\" labels\n    labels_list_type aggregate_labels = {\"shard\", \"cpu\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    // Create metrics with multiple labels\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"metric1\";\n    labels1[\"type\"] = \"typeA\";\n    labels1[\"shard\"] = \"0\";\n    labels1[\"cpu\"] = \"1\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"metric1\";\n    labels2[\"type\"] = \"typeA\";\n    labels2[\"shard\"] = \"1\";\n    labels2[\"cpu\"] = \"2\";\n\n    mi::metric_value value1(100, mi::data_type::COUNTER);\n    mi::metric_value value2(200, mi::data_type::COUNTER);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    // After aggregation, should have one metric with \"shard\" and \"cpu\" removed\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 1);\n\n    auto it = values.begin();\n    // Labels are already filtered (aggregated labels removed)\n    BOOST_REQUIRE_EQUAL(it->second.labels.size(), 2);\n    BOOST_REQUIRE_EQUAL(it->second.labels.at(\"name\").value(), \"metric1\");\n    BOOST_REQUIRE_EQUAL(it->second.labels.at(\"type\").value(), \"typeA\");\n    BOOST_REQUIRE_EQUAL(it->second.m.d(), 300);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_different_label_values) {\n    // Create aggregator that aggregates by \"shard\"\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    // Create metrics with different non-aggregated label values\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"metric1\";\n    labels1[\"shard\"] = \"0\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"metric2\";\n    labels2[\"shard\"] = \"0\";\n\n    mi::metric_value value1(100, mi::data_type::COUNTER);\n    mi::metric_value value2(200, mi::data_type::COUNTER);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    // Should have TWO separate aggregated metrics (different \"name\" values)\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 2);\n\n    // Verify we have both metrics by iterating\n    int count_metric1 = 0;\n    int count_metric2 = 0;\n    for (auto it = values.begin(); it != values.end(); ++it) {\n        if (it->second.labels.at(\"name\").value() == \"metric1\") {\n            count_metric1++;\n            BOOST_REQUIRE_EQUAL(it->second.m.d(), 100);\n        } else if (it->second.labels.at(\"name\").value() == \"metric2\") {\n            count_metric2++;\n            BOOST_REQUIRE_EQUAL(it->second.m.d(), 200);\n        }\n    }\n    BOOST_REQUIRE_EQUAL(count_metric1, 1);\n    BOOST_REQUIRE_EQUAL(count_metric2, 1);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_empty) {\n    // Create aggregator but don't add any metrics\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    BOOST_REQUIRE(aggregator.empty());\n    BOOST_REQUIRE_EQUAL(aggregator.get_values().size(), 0);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_no_aggregation) {\n    // Create aggregator with empty list (no labels to aggregate)\n    labels_list_type aggregate_labels = {};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"metric1\";\n    labels1[\"shard\"] = \"0\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"metric1\";\n    labels2[\"shard\"] = \"1\";\n\n    mi::metric_value value1(100, mi::data_type::COUNTER);\n    mi::metric_value value2(200, mi::data_type::COUNTER);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    // Should have TWO separate metrics (no aggregation)\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 2);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_real_counters) {\n    // Test with real (floating point) counters\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"counter\";\n    labels1[\"shard\"] = \"0\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"counter\";\n    labels2[\"shard\"] = \"1\";\n\n    mi::metric_value value1(123.5, mi::data_type::REAL_COUNTER);\n    mi::metric_value value2(456.7, mi::data_type::REAL_COUNTER);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 1);\n\n    auto it = values.begin();\n    BOOST_REQUIRE_CLOSE(it->second.m.d(), 580.2, 0.001);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_histograms) {\n    // Test with histograms\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    mi::labels_type labels1;\n    labels1[\"name\"] = \"histogram\";\n    labels1[\"shard\"] = \"0\";\n\n    mi::labels_type labels2;\n    labels2[\"name\"] = \"histogram\";\n    labels2[\"shard\"] = \"1\";\n\n    // Create simple histograms\n    sm::histogram h1;\n    h1.sample_count = 10;\n    h1.sample_sum = 100;\n    h1.buckets.push_back(sm::histogram_bucket{5, 1.0});\n    h1.buckets.push_back(sm::histogram_bucket{10, 2.0});\n\n    sm::histogram h2;\n    h2.sample_count = 20;\n    h2.sample_sum = 300;\n    h2.buckets.push_back(sm::histogram_bucket{8, 1.0});\n    h2.buckets.push_back(sm::histogram_bucket{20, 2.0});\n\n    mi::metric_value value1(h1);\n    mi::metric_value value2(h2);\n\n    aggregator.add(value1, labels1);\n    aggregator.add(value2, labels2);\n\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 1);\n\n    auto it = values.begin();\n    const auto& aggregated_hist = it->second.m.get_histogram();\n\n    // Check aggregated histogram values\n    BOOST_REQUIRE_EQUAL(aggregated_hist.sample_count, 30);\n    BOOST_REQUIRE_EQUAL(aggregated_hist.sample_sum, 400);\n    BOOST_REQUIRE_EQUAL(aggregated_hist.buckets.size(), 2);\n    BOOST_REQUIRE_EQUAL(aggregated_hist.buckets[0].count, 13);\n    BOOST_REQUIRE_EQUAL(aggregated_hist.buckets[1].count, 30);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_metric_aggregate_by_labels_same_metric_added_twice) {\n    // Test adding the same metric configuration twice (should aggregate)\n    labels_list_type aggregate_labels = {\"shard\"};\n    metric_aggregate_by_labels aggregator(aggregate_labels);\n\n    mi::labels_type labels;\n    labels[\"name\"] = \"metric\";\n    labels[\"shard\"] = \"0\";\n\n    mi::metric_value value1(100, mi::data_type::COUNTER);\n    mi::metric_value value2(150, mi::data_type::COUNTER);\n\n    aggregator.add(value1, labels);\n    aggregator.add(value2, labels);\n\n    const auto& values = aggregator.get_values();\n    BOOST_REQUIRE_EQUAL(values.size(), 1);\n\n    auto it = values.begin();\n    BOOST_REQUIRE_EQUAL(it->second.m.d(), 250);\n\n    return make_ready_future<>();\n}\n\n// Tests for family_filter functionality\n// These tests use make_family_filter to exercise the same filtering logic used by the HTTP handler\n\nusing name_filter = sp::details::name_filter;\n\nSEASTAR_TEST_CASE(test_family_filter_exact_match) {\n    // Filter to match only metric_1 by exact name\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"group_1_metric_1\", false}\n    });\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_prefix_match) {\n    // Filter to match metrics starting with \"group_1_metric_\" (prefix match)\n    // This should match all 3 metrics\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"group_1_metric_\", true}\n    });\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_multiple_exact_matches) {\n    // Filter to match metric_0 and metric_2 by exact name (not metric_1)\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"group_1_metric_0\", false},\n        name_filter{\"group_1_metric_2\", false}\n    });\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_combined_exact_and_prefix) {\n    // Filter that matches metric_0 exactly OR any metric starting with \"group_1_metric_2\"\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"group_1_metric_0\", false},\n        name_filter{\"group_1_metric_2\", true}\n    });\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_empty_matches_all) {\n    // Empty filter list matches all metrics\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({});\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_no_match) {\n    // Filter with non-existent metric name - should produce empty output\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"nonexistent_metric\", false}\n    });\n    return prometheus_test_fixture::run_metrics_test(cfg, {}, \"\");\n}\n\nSEASTAR_TEST_CASE(test_family_filter_with_label_filter) {\n    // Combine family filter with label filter\n    // Family filter: only metric_0 and metric_1\n    // Label filter: exclude label-0-0\n    // Result: only metric_1 (metric_0 excluded by label filter)\n    sp::details::filter_t label_filter = [](const sm::impl::labels_type& labels) {\n        auto it = labels.find(\"label-0\");\n        return it == labels.end() || it->second.value() != \"label-0-0\";\n    };\n\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"group_1_metric_0\", false},\n        name_filter{\"group_1_metric_1\", false}\n    });\n    cfg.filter = label_filter;\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_exact_match_with_prefix) {\n    // Filter using the full prefixed name \"seastar_group_1_metric_0\"\n    // with prefix \"seastar\" should strip to \"group_1_metric_0\" and match\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"seastar_group_1_metric_0\", false}\n    }, \"seastar\");\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_prefix_match_with_prefix) {\n    // Filter using \"seastar_group_1_metric_\" as prefix filter\n    // with prefix \"seastar\" should strip to \"group_1_metric_\" and match all group_1_metric_* metrics\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"seastar_group_1_metric_\", true}\n    }, \"seastar\");\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_1 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_1 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_1{label-0=\"label-0-1\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\nSEASTAR_TEST_CASE(test_family_filter_mixed_prefixed_and_unprefixed) {\n    // Filter with both prefixed (\"seastar_group_1_metric_0\") and unprefixed (\"group_1_metric_2\") names\n    // Both should match their respective metrics\n    test_config cfg{data_type::COUNTER, 3};\n    cfg.family_filter = sp::details::make_family_filter({\n        name_filter{\"seastar_group_1_metric_0\", false},\n        name_filter{\"group_1_metric_2\", false}\n    }, \"seastar\");\n    return prometheus_test_fixture::run_metrics_test(cfg, {},\n        R\"(# HELP seastar_group_1_metric_0 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_0 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_0{label-0=\"label-0-0\",shard=\"0\"} 123)\" \"\\n\"\n        R\"(# HELP seastar_group_1_metric_2 metric description)\" \"\\n\"\n        R\"(# TYPE seastar_group_1_metric_2 counter)\" \"\\n\"\n        R\"(seastar_group_1_metric_2{label-0=\"label-0-2\",shard=\"0\"} 123)\" \"\\n\"\n    );\n}\n\n"
  },
  {
    "path": "tests/unit/proxy_protocol_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n// Copyright (C) 2025-present ScyllaDB\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/timer.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/net/api.hh>\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <vector>\n\n// Comprehensive tests for proxy protocol v2 implementation\n\nusing namespace seastar;\n\n// Helper function to create a valid proxy protocol v2 header\nstatic std::vector<char> make_proxy_v2_header(\n    bool valid_signature = true,\n    uint8_t version_cmd = 0x21,  // v2 PROXY\n    uint8_t family_proto = 0x11, // IPv4 TCP\n    socket_address src_addr = ipv4_addr(\"192.168.1.100\", 5000),\n    socket_address dst_addr = ipv4_addr(\"10.0.0.1\", 8080),\n    std::vector<char> tlvs = {}) {\n\n    std::vector<char> header;\n\n    // Signature (12 bytes)\n    if (valid_signature) {\n        const char sig[12] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a};\n        header.insert(header.end(), sig, sig + 12);\n    } else {\n        const char bad_sig[12] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0b};\n        header.insert(header.end(), bad_sig, bad_sig + 12);\n    }\n\n    // Version and command\n    header.push_back(version_cmd);\n\n    // Family and protocol\n    header.push_back(family_proto);\n\n    // Length (will be filled in later)\n    size_t len_pos = header.size();\n    header.push_back(0);\n    header.push_back(0);\n\n    // Address information\n    uint16_t addr_len = 0;\n    if ((version_cmd & 0x0F) == 0x01) { // PROXY command\n        if (family_proto == 0x11) { // IPv4 TCP\n            addr_len = 12;\n            auto src_in = src_addr.as_posix_sockaddr_in();\n            auto dst_in = dst_addr.as_posix_sockaddr_in();\n\n            // Source address\n            header.push_back((src_in.sin_addr.s_addr >> 0) & 0xFF);\n            header.push_back((src_in.sin_addr.s_addr >> 8) & 0xFF);\n            header.push_back((src_in.sin_addr.s_addr >> 16) & 0xFF);\n            header.push_back((src_in.sin_addr.s_addr >> 24) & 0xFF);\n\n            // Destination address\n            header.push_back((dst_in.sin_addr.s_addr >> 0) & 0xFF);\n            header.push_back((dst_in.sin_addr.s_addr >> 8) & 0xFF);\n            header.push_back((dst_in.sin_addr.s_addr >> 16) & 0xFF);\n            header.push_back((dst_in.sin_addr.s_addr >> 24) & 0xFF);\n\n            // Source port\n            uint16_t src_port = ntohs(src_in.sin_port);\n            header.push_back((src_port >> 8) & 0xFF);\n            header.push_back((src_port >> 0) & 0xFF);\n\n            // Destination port\n            uint16_t dst_port = ntohs(dst_in.sin_port);\n            header.push_back((dst_port >> 8) & 0xFF);\n            header.push_back((dst_port >> 0) & 0xFF);\n        } else if (family_proto == 0x21) { // IPv6 TCP\n            addr_len = 36;\n            auto src_in6 = src_addr.as_posix_sockaddr_in6();\n            auto dst_in6 = dst_addr.as_posix_sockaddr_in6();\n\n            // Source address (16 bytes)\n            header.insert(header.end(),\n                reinterpret_cast<const char*>(&src_in6.sin6_addr),\n                reinterpret_cast<const char*>(&src_in6.sin6_addr) + 16);\n\n            // Destination address (16 bytes)\n            header.insert(header.end(),\n                reinterpret_cast<const char*>(&dst_in6.sin6_addr),\n                reinterpret_cast<const char*>(&dst_in6.sin6_addr) + 16);\n\n            // Source port\n            uint16_t src_port = ntohs(src_in6.sin6_port);\n            header.push_back((src_port >> 8) & 0xFF);\n            header.push_back((src_port >> 0) & 0xFF);\n\n            // Destination port\n            uint16_t dst_port = ntohs(dst_in6.sin6_port);\n            header.push_back((dst_port >> 8) & 0xFF);\n            header.push_back((dst_port >> 0) & 0xFF);\n        }\n    } else if ((version_cmd & 0x0F) == 0x00) { // LOCAL command\n        addr_len = 0;  // No address data for LOCAL\n    }\n\n    // Add TLVs\n    header.insert(header.end(), tlvs.begin(), tlvs.end());\n    addr_len += tlvs.size();\n\n    // Fill in length\n    header[len_pos] = (addr_len >> 8) & 0xFF;\n    header[len_pos + 1] = addr_len & 0xFF;\n\n    return header;\n}\n\nstruct proxy_test_result {\n    bool got_connection = false;\n    bool timed_out = false;\n};\n\nstruct server_addresses {\n    socket_address remote;\n    socket_address local;\n};\n\n// Helper to run server with timeout - returns result\nstatic future<proxy_test_result> run_server(server_socket& ss) {\n    return seastar::async([&ss] {\n        using namespace std::chrono_literals;\n\n        proxy_test_result result;\n\n        timer<> abort_timer([&ss, &result] {\n            result.timed_out = true;\n            ss.abort_accept();\n        });\n        abort_timer.arm(timer<>::clock::now() + 100ms);\n\n        try {\n            auto ar = ss.accept().get();\n            result.got_connection = true;\n            ar.connection.shutdown_output();\n        } catch (...) {\n            // Expected - abort_accept throws or connection rejected\n        }\n\n        return result;\n    });\n}\n\n// Helper to run server for positive tests - accepts connection and returns addresses\nstatic future<server_addresses> run_server_accept(server_socket& ss) {\n    return seastar::async([&ss] {\n        auto ar = ss.accept().get();\n        server_addresses addrs;\n        addrs.remote = ar.connection.remote_address();\n        addrs.local = ar.connection.local_address();\n        ar.connection.shutdown_output();\n        return addrs;\n    });\n}\n\n// Helper function for negative tests - expects connection to be rejected\nstatic void test_proxy_header_negative(\n    socket_address listen_addr,\n    std::vector<char> header) {\n\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server(ss);\n\n    auto client = seastar::async([&listen_addr, header = std::move(header)] {\n        try {\n            auto s = connect(listen_addr).get();\n            auto out = s.output();\n            out.write(header.data(), header.size()).get();\n            out.flush().get();\n            out.close().get();\n        } catch (...) {\n            // Expected - server may close connection during write/flush/close\n        }\n    });\n\n    auto [result] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE(!result.got_connection);\n    BOOST_REQUIRE(result.timed_out);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_bad_signature) {\n    // Invalid signature should cause connection to be dropped\n    auto header = make_proxy_v2_header(false);  // bad signature\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12001), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_small_packet_incomplete_header) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12002);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server(ss);\n\n    // Send only 10 bytes (header needs 16)\n    auto client = seastar::async([&listen_addr] {\n        try {\n            auto s = connect(listen_addr).get();\n            auto out = s.output();\n            char partial[10] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49};\n            out.write(partial, sizeof(partial)).get();\n            out.flush().get();\n            out.close().get();\n        } catch (...) {\n            // ignore\n        }\n    });\n\n    auto [result] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE(!result.got_connection);\n    BOOST_REQUIRE(result.timed_out);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_small_packet_incomplete_addresses) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12003);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server(ss);\n\n    // Send header claiming 12 bytes of addresses but only send 6\n    auto client = seastar::async([&listen_addr] {\n        try {\n            auto s = connect(listen_addr).get();\n            auto out = s.output();\n            unsigned char buf[22];\n            const unsigned char sig[12] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a};\n            std::memcpy(buf, sig, 12);\n            buf[12] = 0x21; // v2 PROXY\n            buf[13] = 0x11; // IPv4 TCP\n            buf[14] = 0x00; // length high byte\n            buf[15] = 0x0C; // length low byte (12 bytes)\n            // Only send 6 bytes of address data\n            buf[16] = 192;\n            buf[17] = 168;\n            buf[18] = 1;\n            buf[19] = 100;\n            buf[20] = 10;\n            buf[21] = 0;\n            out.write(reinterpret_cast<char*>(buf), sizeof(buf)).get();\n            out.flush().get();\n            out.close().get();\n        } catch (...) {\n            // ignore\n        }\n    });\n\n    auto [result] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE(!result.got_connection);\n    BOOST_REQUIRE(result.timed_out);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_local_mode) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12004);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server_accept(ss);\n\n    // Send LOCAL command with UNSPEC family\n    auto header = make_proxy_v2_header(true, 0x20, 0x00);  // v2 LOCAL, UNSPEC\n    socket_address client_actual_local;\n    auto client = seastar::async([&listen_addr, &client_actual_local, header = std::move(header)] {\n        auto s = connect(listen_addr).get();\n        client_actual_local = s.local_address();\n        auto out = s.output();\n        out.write(header.data(), header.size()).get();\n        out.flush().get();\n        out.close().get();\n    });\n\n    auto [addrs] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    // With LOCAL mode, addresses should be real connection addresses, not from header\n    // We can't check exact match since the client port is ephemeral, but we can verify\n    // that the addresses are not the fake proxy addresses and the IPs match\n    BOOST_REQUIRE_EQUAL(addrs.local, socket_address(listen_addr));\n    BOOST_REQUIRE_EQUAL(addrs.remote, client_actual_local);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_local_mode_wrong_family) {\n    // LOCAL command must use UNSPEC family (0x00), not other families\n    auto header = make_proxy_v2_header(true, 0x20, 0x11);  // v2 LOCAL, but IPv4 TCP (invalid)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12005), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_ipv4_addresses) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12006);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto expected_remote = ipv4_addr(\"192.168.1.100\", 5000);\n    auto expected_local = ipv4_addr(\"10.0.0.1\", 8080);\n\n    auto server = run_server_accept(ss);\n\n    auto header = make_proxy_v2_header(true, 0x21, 0x11, expected_remote, expected_local);\n    auto client = seastar::async([&listen_addr, header = std::move(header)] {\n        auto s = connect(listen_addr).get();\n        auto out = s.output();\n        out.write(header.data(), header.size()).get();\n        out.flush().get();\n        out.close().get();\n    });\n\n    auto [addrs] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    // Verify addresses match what was sent in the header\n    BOOST_REQUIRE_EQUAL(addrs.remote, socket_address(expected_remote));\n    BOOST_REQUIRE_EQUAL(addrs.local, socket_address(expected_local));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_ipv6_addresses) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv6_addr(\"::1\", 12007);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto expected_remote = ipv6_addr(\"2001:db8::1\", 5000);\n    auto expected_local = ipv6_addr(\"2001:db8::2\", 8080);\n\n    auto server = run_server_accept(ss);\n\n    auto header = make_proxy_v2_header(true, 0x21, 0x21, expected_remote, expected_local);\n    auto client = seastar::async([&listen_addr, header = std::move(header)] {\n        auto s = connect(listen_addr).get();\n        auto out = s.output();\n        out.write(header.data(), header.size()).get();\n        out.flush().get();\n        out.close().get();\n    });\n\n    auto [addrs] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    // Verify addresses match what was sent in the header\n    BOOST_REQUIRE_EQUAL(addrs.remote, socket_address(expected_remote));\n    BOOST_REQUIRE_EQUAL(addrs.local, socket_address(expected_local));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_invalid_version) {\n    // Only version 2 (0x2X) is supported\n    auto header = make_proxy_v2_header(true, 0x11);  // version 1 (invalid for v2 format)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12008), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_invalid_command) {\n    // Only LOCAL (0x0) and PROXY (0x1) commands are valid\n    auto header = make_proxy_v2_header(true, 0x22);  // v2 with invalid command (0x2)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12009), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_unsupported_family) {\n    // Only AF_INET (0x1) and AF_INET6 (0x2) are supported for PROXY command\n    auto header = make_proxy_v2_header(true, 0x21, 0x31);  // v2 PROXY, AF_UNIX (not supported)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12010), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_unsupported_protocol) {\n    // Only STREAM (0x1) is supported\n    auto header = make_proxy_v2_header(true, 0x21, 0x12);  // v2 PROXY, IPv4 DGRAM (not supported)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12011), std::move(header));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_length_too_short_ipv4) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12012);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server(ss);\n\n    // IPv4 requires 12 bytes, send header claiming only 8\n    auto client = seastar::async([&listen_addr] {\n        try {\n            auto s = connect(listen_addr).get();\n            auto out = s.output();\n            unsigned char buf[24];\n            const unsigned char sig[12] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a};\n            std::memcpy(buf, sig, 12);\n            buf[12] = 0x21; // v2 PROXY\n            buf[13] = 0x11; // IPv4 TCP\n            buf[14] = 0x00; // length high byte\n            buf[15] = 0x08; // length low byte (8 bytes, too short)\n            // Fill in 8 bytes of data\n            for (int i = 16; i < 24; ++i) {\n                buf[i] = 0;\n            }\n            out.write(reinterpret_cast<char*>(buf), sizeof(buf)).get();\n            out.flush().get();\n            out.close().get();\n        } catch (...) {\n            // ignore\n        }\n    });\n\n    auto [result] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE(!result.got_connection);\n    BOOST_REQUIRE(result.timed_out);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_length_too_short_ipv6) {\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv6_addr(\"::1\", 12013);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto server = run_server(ss);\n\n    // IPv6 requires 36 bytes, send header claiming only 20\n    auto client = seastar::async([&listen_addr] {\n        try {\n            auto s = connect(listen_addr).get();\n            auto out = s.output();\n            std::vector<unsigned char> buf(36);\n            const unsigned char sig[12] = {0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a};\n            std::memcpy(buf.data(), sig, 12);\n            buf[12] = 0x21; // v2 PROXY\n            buf[13] = 0x21; // IPv6 TCP\n            buf[14] = 0x00; // length high byte\n            buf[15] = 0x14; // length low byte (20 bytes, too short)\n            // Fill in 20 bytes of data\n            out.write(reinterpret_cast<char*>(buf.data()), 36).get();\n            out.flush().get();\n            out.close().get();\n        } catch (...) {\n            // ignore\n        }\n    });\n\n    auto [result] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE(!result.got_connection);\n    BOOST_REQUIRE(result.timed_out);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_with_tlvs) {\n    // TLVs should be skipped/ignored by the implementation\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12014);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto expected_remote = ipv4_addr(\"192.168.1.100\", 5000);\n    auto expected_local = ipv4_addr(\"10.0.0.1\", 8080);\n\n    auto server = run_server_accept(ss);\n\n    // Add some TLVs (type, length_hi, length_lo, value...)\n    std::vector<char> tlvs;\n    tlvs.push_back(0x01); // PP2_TYPE_ALPN\n    tlvs.push_back(0x00); // length high\n    tlvs.push_back(0x08); // length low (8 bytes)\n    tlvs.insert(tlvs.end(), 8, 'x'); // dummy data\n\n    auto header = make_proxy_v2_header(true, 0x21, 0x11, expected_remote, expected_local, tlvs);\n    auto client = seastar::async([&listen_addr, header = std::move(header)] {\n        auto s = connect(listen_addr).get();\n        auto out = s.output();\n        out.write(header.data(), header.size()).get();\n        out.flush().get();\n        out.close().get();\n    });\n\n    auto [addrs] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    // Verify addresses still work correctly despite TLVs\n    BOOST_REQUIRE_EQUAL(addrs.remote, socket_address(expected_remote));\n    BOOST_REQUIRE_EQUAL(addrs.local, socket_address(expected_local));\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_extreme_ports) {\n    // Test with port 0 and port 65535\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.proxy_protocol = true;\n\n    auto listen_addr = ipv4_addr(\"127.0.0.1\", 12015);\n    server_socket ss = seastar::listen(listen_addr, lo);\n\n    auto expected_remote = ipv4_addr(\"192.168.1.100\", 0);      // port 0\n    auto expected_local = ipv4_addr(\"10.0.0.1\", 65535);        // port 65535\n\n    auto server = run_server_accept(ss);\n\n    auto header = make_proxy_v2_header(true, 0x21, 0x11, expected_remote, expected_local);\n    auto client = seastar::async([&listen_addr, header = std::move(header)] {\n        auto s = connect(listen_addr).get();\n        auto out = s.output();\n        out.write(header.data(), header.size()).get();\n        out.flush().get();\n        out.close().get();\n    });\n\n    auto [addrs] = when_all_succeed(std::move(server), std::move(client)).get();\n    ss.abort_accept();\n\n    BOOST_REQUIRE_EQUAL(addrs.remote.port(), 0);\n    BOOST_REQUIRE_EQUAL(addrs.local.port(), 65535);\n}\n\nSEASTAR_THREAD_TEST_CASE(proxy_protocol_v2_unspec_family) {\n    // UNSPEC family (0x0) should only be valid with LOCAL command\n    auto header = make_proxy_v2_header(true, 0x21, 0x00);  // v2 PROXY with UNSPEC (invalid combo)\n    test_proxy_header_negative(ipv4_addr(\"127.0.0.1\", 12016), std::move(header));\n}\n"
  },
  {
    "path": "tests/unit/queue_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2018 ScyllaDB\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/queue.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/future-util.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/alloc_failure_injector.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nstatic seastar::logger testlog(\"testlog\");\n\nSEASTAR_THREAD_TEST_CASE(test_queue_pop_eventually) {\n    queue<int> q(100);\n    int pushed = 0;\n    bool pusher_done = false;\n    int popped = 0;\n    bool popper_done = false;\n    int stop = 0;\n    auto test_duration = 1ms;\n    auto watchdog_duration = 10s;\n    timer stop_timer;\n    stop_timer.set_callback([&] {\n        testlog.debug(\"stop_timer: pushed={} pusher_done={} popped={} popper_done={} full={} empty={} stop={}\",\n                pushed, pusher_done, popped, popper_done,\n                q.full(), q.empty(), stop);\n        if (!stop++) {\n            // First callback is for stopping the test.\n            // Second one is a watchdog. Consider the test hung\n            // if this callback isn't canceled within 10 seconds.\n            // That should be long enough to work in absurdly slow test environments.\n            stop_timer.arm(watchdog_duration);\n        } else {\n            testlog.error(\"test_queue_pop_eventually is hung: pushed={} pusher_done={} popped={} popper_done={} full={} empty={} stop={}\",\n                    pushed, pusher_done, popped, popper_done,\n                    q.full(), q.empty(), stop);\n            abort();\n        }\n    });\n    stop_timer.arm(test_duration);\n    auto start = std::chrono::system_clock::now();\n    auto pusher = repeat([&] {\n        auto&& data = pushed;\n        testlog.trace(\"pusher: full={} empty={} stop={}\", q.full(), q.empty(), stop);\n        return q.push_eventually(std::move(data)).then([&] {\n            pushed++;\n            if (stop && !q.empty()) {\n                testlog.debug(\"pusher done\");\n                pusher_done = true;\n                return stop_iteration::yes;\n            }\n            return stop_iteration::no;\n        });\n    });\n    auto popper = repeat([&] {\n        testlog.trace(\"popper: full={} empty={} stop={}\", q.full(), q.empty(), stop);\n        if (q.empty()) {\n            if (pusher_done) {\n                testlog.debug(\"popper done\");\n                popper_done = true;\n                return make_ready_future<stop_iteration>(true);\n            } else if (stop) {\n                testlog.debug(\"popper: full={} empty={} pusher_done={} stop={}\", q.full(), q.empty(), pusher_done, stop);\n            }\n        }\n        return q.pop_eventually().then([&] (int&&) {\n            popped++;\n            return stop_iteration::no;\n        });\n    });\n    pusher.get();\n    popper.get();\n    auto elapsed = std::chrono::system_clock::now() - start;\n    auto elapsed_us = std::chrono::duration_cast<std::chrono::microseconds>(elapsed).count();\n    stop_timer.cancel();\n    BOOST_REQUIRE(q.empty());\n    BOOST_REQUIRE(pushed);\n    BOOST_REQUIRE(pusher_done);\n    BOOST_REQUIRE(popped == pushed);\n    BOOST_REQUIRE(popper_done);\n    testlog.info(\"Pushed and popped {} elemements in {}us, {:.3f} elements/us\", pushed, elapsed_us, double(pushed) / elapsed_us);\n}\n\n#ifdef SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION\nSEASTAR_THREAD_TEST_CASE(test_queue_push_eventually_exception) {\n    int i = 0;\n    queue<int> q(42);\n    int intercepted = 0;\n\n    memory::with_allocation_failures([&] {\n        BOOST_REQUIRE_NO_THROW(q.push_eventually(i++).handle_exception_type([&] (std::bad_alloc&) {\n            intercepted++;\n        }).get());\n    });\n    BOOST_REQUIRE(intercepted);\n}\n#endif\n\nSEASTAR_TEST_CASE(test_queue_pop_after_abort) {\n    return async([] {\n        queue<int> q(1);\n        bool exception = false;\n        bool timer = false;\n        future<> done = make_ready_future();\n        q.abort(std::make_exception_ptr(std::runtime_error(\"boom\")));\n        done = sleep(1ms).then([&] {\n            timer = true;\n            q.abort(std::make_exception_ptr(std::runtime_error(\"boom\")));\n        });\n        try {\n            q.pop_eventually().get();\n        } catch(...) {\n            exception = !timer;\n        }\n        BOOST_REQUIRE(exception);\n        done.get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_queue_push_abort) {\n    return async([] {\n        queue<int> q(1);\n        bool exception = false;\n        bool timer = false;\n        future<> done = make_ready_future();\n        q.abort(std::make_exception_ptr(std::runtime_error(\"boom\")));\n        done = sleep(1ms).then([&] {\n            timer = true;\n            q.abort(std::make_exception_ptr(std::runtime_error(\"boom\")));\n        });\n        try {\n            q.push_eventually(1).get();\n        } catch(...) {\n            exception = !timer;\n        }\n        BOOST_REQUIRE(exception);\n        done.get();\n    });\n}\n"
  },
  {
    "path": "tests/unit/request_parser_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <seastar/core/ragel.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/http/request.hh>\n#include <seastar/http/request_parser.hh>\n#include <seastar/testing/test_case.hh>\n#include <tuple>\n#include <utility>\n#include <vector>\n\nusing namespace seastar;\n\nSEASTAR_TEST_CASE(test_header_parsing) {\n    struct test_set {\n        sstring msg;\n        bool parsable;\n        sstring header_name = \"\";\n        sstring header_value = \"\";\n\n        temporary_buffer<char> buf() {\n            return temporary_buffer<char>(msg.c_str(), msg.size());\n        }\n    };\n\n    std::vector<test_set> tests = {\n        { \"GET /test HTTP/1.1\\r\\nHost: test\\r\\n\\r\\n\", true, \"Host\", \"test\" },\n        { \"GET /hello HTTP/1.0\\r\\nHeader: Field\\r\\n\\r\\n\", true, \"Header\", \"Field\" },\n        { \"GET /hello HTTP/1.0\\r\\nHeader: \\r\\n\\r\\n\", true, \"Header\", \"\" },\n        { \"GET /hello HTTP/1.0\\r\\nHeader:  f  i e l d  \\r\\n\\r\\n\", true, \"Header\", \"f  i e l d\" },\n        { \"GET /hello HTTP/1.0\\r\\nHeader: fiel\\r\\n    d\\r\\n\\r\\n\", true, \"Header\", \"fiel d\" },\n        { \"GET /hello HTTP/1.0\\r\\ntchars.^_`|123: printable!@#%^&*()obs_text\\x80\\x81\\xff\\r\\n\\r\\n\", true,\n            \"tchars.^_`|123\", \"printable!@#%^&*()obs_text\\x80\\x81\\xff\" },\n        { \"GET /hello HTTP/1.0\\r\\nHeader: Field\\r\\nHeader: Field2\\r\\n\\r\\n\", true, \"Header\", \"Field,Field2\" },\n        { \"GET /hello HTTP/1.0\\r\\n\\r\\n\", true },\n        { \"GET /hello HTTP/1.0\\r\\nHeader : Field\\r\\n\\r\\n\", false },\n        { \"GET /hello HTTP/1.0\\r\\nHeader Field\\r\\n\\r\\n\", false },\n        { \"GET /hello HTTP/1.0\\r\\nHeader@: Field\\r\\n\\r\\n\", false },\n        { \"GET /hello HTTP/1.0\\r\\nHeader: fiel\\r\\nd \\r\\n\\r\\n\", false }\n    };\n\n    http_request_parser parser;\n    for (auto& tset : tests) {\n        parser.init();\n        BOOST_REQUIRE(parser(tset.buf()).get().has_value());\n        BOOST_REQUIRE_NE(parser.failed(), tset.parsable);\n        if (tset.parsable) {\n            auto req = parser.get_parsed_request();\n            BOOST_REQUIRE_EQUAL(req->get_header(std::move(tset.header_name)), std::move(tset.header_value));\n        }\n    }\n    return make_ready_future<>();\n}\n"
  },
  {
    "path": "tests/unit/rest_api_httpd.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2024 Scylladb, Ltd.\n */\n\n#include <memory>\n#include <fmt/core.h>\n\n#include <seastar/http/exception.hh>\n#include <seastar/http/httpd.hh>\n#include <seastar/http/handlers.hh>\n#include <seastar/http/function_handlers.hh>\n#include <seastar/http/file_handler.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/http/api_docs.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/util/defer.hh>\n#include \"../../apps/lib/stop_signal.hh\"\n#include \"api.json.hh\"\n\nnamespace bpo = boost::program_options;\n\nusing namespace seastar;\nusing namespace httpd;\nusing namespace api_json::ns_hello_world;\n\nvoid set_routes(routes& r) {\n    api_json::hello_world.set(r, [] (const_req req) -> json::json_return_type {\n        api_json::my_object obj;\n        obj.var1 = req.param.at(\"var1\");\n        obj.var2 = req.param.at(\"var2\");\n        query_enum v = str2query_enum(req.get_query_param(\"query_enum\"));\n        // This demonstrate enum conversion\n        obj.enum_var = v;\n\n        std::vector<int> vec123{1, 2, 3};\n        obj.array_var = vec123;\n        obj.chunked_array_var = vec123;\n\n        auto use_streaming = req.get_query_param(\"use_streaming\");\n\n        if (use_streaming != \"false\" && use_streaming != \"true\") {\n            throw bad_param_exception(\"param use_streaming must be true or false\");\n        }\n\n        if (use_streaming == \"false\") {\n            return obj;\n        } else {\n            return stream_object(std::move(obj));\n        }\n    });\n}\n\nint main(int ac, char** av) {\n    app_template app;\n\n    app.add_options()(\"port\", bpo::value<uint16_t>()->default_value(10000), \"HTTP Server port\");\n\n    return app.run(ac, av, [&] {\n        return seastar::async([&] {\n            seastar_apps_lib::stop_signal stop_signal;\n            auto&& config = app.configuration();\n            uint16_t port = config[\"port\"].as<uint16_t>();\n            auto server = std::make_unique<http_server_control>();\n            auto rb = make_shared<api_registry_builder>(\"apps/httpd/\");\n            server->start().get();\n\n            auto stop_server = defer([&] () noexcept {\n                std::cout << \"Stopping HTTP server\" << std::endl; // This can throw, but won't.\n                server->stop().get();\n            });\n\n            server->set_routes(set_routes).get();\n            server->set_routes([rb](routes& r){rb->set_api_doc(r);}).get();\n            server->set_routes([rb](routes& r) {rb->register_function(r, \"demo\", \"rest api test\");}).get();\n            server->listen(port).get();\n\n            fmt::print(\"{}\\n\", port);\n            fflush(stdout);\n\n            stop_signal.wait().get();\n            return 0;\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/rpc_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2016 ScyllaDB\n */\n\n#include \"loopback_socket.hh\"\n#include \"seastar/core/condition-variable.hh\"\n#include \"seastar/core/temporary_buffer.hh\"\n#include <seastar/rpc/rpc.hh>\n#include <seastar/rpc/rpc_types.hh>\n#include <seastar/rpc/lz4_compressor.hh>\n#include <seastar/rpc/lz4_fragmented_compressor.hh>\n#include <seastar/rpc/multi_algo_compressor_factory.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/metrics_api.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/closeable.hh>\n#include <seastar/util/noncopyable_function.hh>\n#include <seastar/util/later.hh>\n#include <seastar/core/coroutine.hh>\n#include <seastar/core/reactor.hh>\n\n#include <boost/range/numeric.hpp>\n\n#include <span>\n\nusing namespace seastar;\n\nstruct serializer {\n};\n\ntemplate <typename T, typename Output>\ninline\nvoid write_arithmetic_type(Output& out, T v) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    return out.write(reinterpret_cast<const char*>(&v), sizeof(T));\n}\n\ntemplate <typename T, typename Input>\ninline\nT read_arithmetic_type(Input& in) {\n    static_assert(std::is_arithmetic_v<T>, \"must be arithmetic type\");\n    T v;\n    in.read(reinterpret_cast<char*>(&v), sizeof(T));\n    return v;\n}\n\ntemplate <typename Output>\ninline void write(serializer, Output& output, int32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint32_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, int64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, uint64_t v) { return write_arithmetic_type(output, v); }\ntemplate <typename Output>\ninline void write(serializer, Output& output, double v) { return write_arithmetic_type(output, v); }\ntemplate <typename Input>\ninline int32_t read(serializer, Input& input, rpc::type<int32_t>) { return read_arithmetic_type<int32_t>(input); }\ntemplate <typename Input>\ninline uint32_t read(serializer, Input& input, rpc::type<uint32_t>) { return read_arithmetic_type<uint32_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<uint64_t>) { return read_arithmetic_type<uint64_t>(input); }\ntemplate <typename Input>\ninline uint64_t read(serializer, Input& input, rpc::type<int64_t>) { return read_arithmetic_type<int64_t>(input); }\ntemplate <typename Input>\ninline double read(serializer, Input& input, rpc::type<double>) { return read_arithmetic_type<double>(input); }\n\ntemplate <typename Output>\ninline void write(serializer, Output& out, const sstring& v) {\n    write_arithmetic_type(out, uint32_t(v.size()));\n    out.write(v.c_str(), v.size());\n}\n\ntemplate <typename Input>\ninline sstring read(serializer, Input& in, rpc::type<sstring>) {\n    auto size = read_arithmetic_type<uint32_t>(in);\n    sstring ret = uninitialized_string(size);\n    in.read(ret.data(), size);\n    return ret;\n}\n\nusing test_rpc_proto = rpc::protocol<serializer>;\nusing make_socket_fn = std::function<seastar::socket ()>;\n\nclass rpc_loopback_error_injector : public loopback_error_injector {\npublic:\n    struct config {\n        struct {\n            int limit = 0;\n            error kind = error::none;\n        private:\n            friend class rpc_loopback_error_injector;\n            int _x = 0;\n            error inject() {\n                return _x++ >= limit ? kind : error::none;\n            }\n        } server_rcv = {}, server_snd = {}, client_rcv = {}, client_snd = {};\n        error connect_kind = error::none;\n        std::chrono::microseconds connect_delay = std::chrono::microseconds(0);\n    };\nprivate:\n    config _cfg;\npublic:\n    rpc_loopback_error_injector(config cfg) : _cfg(std::move(cfg)) {}\n\n    error server_rcv_error() override {\n        return _cfg.server_rcv.inject();\n    }\n\n    error server_snd_error() override {\n        return _cfg.server_snd.inject();\n    }\n\n    error client_rcv_error() override {\n        return _cfg.client_rcv.inject();\n    }\n\n    error client_snd_error() override {\n        return _cfg.client_snd.inject();\n    }\n\n    error connect_error() override {\n        return _cfg.connect_kind;\n    }\n\n    std::chrono::microseconds connect_delay() override {\n        return _cfg.connect_delay;\n    }\n};\n\nclass rpc_socket_impl : public ::net::socket_impl {\n    rpc_loopback_error_injector _error_injector;\n    loopback_socket_impl _socket;\npublic:\n    rpc_socket_impl(loopback_connection_factory& factory, std::optional<rpc_loopback_error_injector::config> inject_error)\n            :\n              _error_injector(inject_error.value_or(rpc_loopback_error_injector::config{})),\n              _socket(factory, inject_error ? &_error_injector : nullptr) {\n    }\n    virtual future<connected_socket> connect(socket_address sa, socket_address local, transport proto = transport::TCP) override {\n        return _socket.connect(sa, local, proto);\n    }\n    virtual void set_reuseaddr(bool reuseaddr) override {}\n    virtual bool get_reuseaddr() const override { return false; };\n    virtual void shutdown() override {\n        _socket.shutdown();\n    }\n};\n\nstruct rpc_test_config {\n    rpc::resource_limits resource_limits = {};\n    rpc::server_options server_options = {};\n    std::optional<rpc_loopback_error_injector::config> inject_error;\n};\n\ntemplate<typename MsgType = int>\nclass rpc_test_env {\npublic:\n    struct rpc_test_service {\n        test_rpc_proto _proto;\n        test_rpc_proto::server _server;\n        std::vector<MsgType> _handlers;\n\n        rpc_test_service() = delete;\n        explicit rpc_test_service(const rpc_test_config& cfg, loopback_connection_factory& lcf)\n                : _proto(serializer())\n                , _server(_proto, cfg.server_options, lcf.get_server_socket(), cfg.resource_limits)\n        { }\n\n        test_rpc_proto& proto() {\n            return _proto;\n        }\n\n        test_rpc_proto::server& server() {\n            return _server;\n        }\n\n        future<> stop() {\n            return parallel_for_each(_handlers, [this] (auto t) {\n                return proto().unregister_handler(t);\n            }).finally([this] {\n                return server().stop();\n            });\n        }\n\n        template<typename Func>\n        auto register_handler(MsgType t, scheduling_group sg, Func func) {\n            _handlers.emplace_back(t);\n            return proto().register_handler(t, sg, std::move(func));\n        }\n\n        template<typename Func>\n        auto register_handler(MsgType t, Func func) {\n            return register_handler(t, scheduling_group(), std::forward<Func>(func));\n        }\n\n        future<> unregister_handler(MsgType t) {\n            auto it = std::find(_handlers.begin(), _handlers.end(), t);\n            SEASTAR_ASSERT(it != _handlers.end());\n            _handlers.erase(it);\n            return proto().unregister_handler(t);\n        }\n    };\n\nprivate:\n    rpc_test_config _cfg;\n    loopback_connection_factory _lcf;\n    std::unique_ptr<sharded<rpc_test_service>> _service;\n\npublic:\n    rpc_test_env() = delete;\n    explicit rpc_test_env(rpc_test_config cfg)\n        : _cfg(cfg), _service(std::make_unique<sharded<rpc_test_service>>())\n    {\n    }\n\n    using test_fn = std::function<future<> (rpc_test_env<MsgType>& env)>;\n    static future<> do_with(rpc_test_config cfg, test_fn&& func) {\n        return seastar::do_with(rpc_test_env(cfg), [func] (rpc_test_env<MsgType>& env) {\n            return env.start().then([&env, func] {\n                return func(env);\n            }).finally([&env] {\n                return env.stop();\n            });\n        });\n    }\n\n    using thread_test_fn = std::function<void (rpc_test_env<MsgType>& env)>;\n    static future<> do_with_thread(rpc_test_config cfg, thread_test_fn&& func) {\n        return do_with(std::move(cfg), [func] (rpc_test_env<MsgType>& env) {\n            return seastar::async([&env, func] {\n                func(env);\n            });\n        });\n    }\n\n    using thread_test_fn_with_client = std::function<void (rpc_test_env<MsgType>& env, test_rpc_proto::client& cl)>;\n    static future<> do_with_thread(rpc_test_config cfg, rpc::client_options co, thread_test_fn_with_client&& func) {\n        return do_with(std::move(cfg), [func, co = std::move(co)] (rpc_test_env<MsgType>& env) {\n            return seastar::async([&env, func, co = std::move(co)] {\n                test_rpc_proto::client cl(env.proto(), co, env.make_socket(), ipv4_addr());\n                auto stop = deferred_stop(cl);\n                func(env, cl);\n            });\n        });\n    }\n\n    static future<> do_with_thread(rpc_test_config cfg, thread_test_fn_with_client&& func) {\n        return do_with_thread(std::move(cfg), rpc::client_options(), std::move(func));\n    }\n\n    auto make_socket() {\n        return seastar::socket(std::make_unique<rpc_socket_impl>(_lcf, _cfg.inject_error));\n    };\n\n    test_rpc_proto& proto() {\n        return local_service().proto();\n    }\n\n    test_rpc_proto::server& server() {\n        return local_service().server();\n    }\n\n    template<typename Func>\n    future<> register_handler(MsgType t, scheduling_group sg, Func func) {\n        return _service->invoke_on_all([t, func = std::move(func), sg] (rpc_test_service& s) mutable {\n            s.register_handler(t, sg, std::move(func));\n        });\n    }\n\n    template<typename Func>\n    future<> register_handler(MsgType t, Func func) {\n        return register_handler(t, scheduling_group(), std::move(func));\n    }\n\n    future<> unregister_handler(MsgType t) {\n        return _service->invoke_on_all([t] (rpc_test_service& s) mutable {\n            return s.unregister_handler(t);\n        });\n    }\n\n    future<> invoke_on_all(std::function<future<> (rpc_test_service& s)> func) {\n        return _service->invoke_on_all(std::move(func));\n    }\n\nprivate:\n    rpc_test_service& local_service() {\n        return _service->local();\n\n    }\n\n    future<> start() {\n        return _service->start(std::cref(_cfg), std::ref(_lcf));\n    }\n\n    future<> stop() {\n        return _service->stop().then([this] {\n            return _lcf.destroy_all_shards();\n        });\n    }\n};\n\nstruct cfactory : rpc::compressor::factory {\n    mutable int use_compression = 0;\n    const sstring name;\n    cfactory(sstring name_ = \"LZ4\") : name(std::move(name_)) {}\n    const sstring& supported() const override {\n        return name;\n    }\n    class mylz4 : public rpc::lz4_compressor {\n        sstring _name;\n    public:\n        mylz4(const sstring& n) : _name(n) {}\n        sstring name() const override {\n            return _name;\n        }\n    };\n    std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server) const override {\n        if (feature == name) {\n            use_compression++;\n            return std::make_unique<mylz4>(name);\n        } else {\n            return nullptr;\n        }\n    }\n};\n\nSEASTAR_TEST_CASE(test_rpc_connect) {\n    std::vector<future<>> fs;\n\n    for (auto i = 0; i < 2; i++) {\n        for (auto j = 0; j < 4; j++) {\n          for (bool with_delay : { true, false }) {\n            auto factory = std::make_unique<cfactory>();\n            rpc::server_options so;\n            rpc::client_options co;\n            if (i == 1) {\n                so.compressor_factory = factory.get();\n            }\n            if (j & 1) {\n                co.compressor_factory = factory.get();\n            }\n            co.send_timeout_data = j & 2;\n            co.send_handler_duration = with_delay;\n            rpc_test_config cfg;\n            cfg.server_options = so;\n            auto f = rpc_test_env<>::do_with_thread(cfg, co, [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n                env.register_handler(1, [](int a, int b) {\n                    return make_ready_future<int>(a+b);\n                }).get();\n                auto sum = env.proto().make_client<int (int, int)>(1);\n                auto result = sum(c1, 2, 3).get();\n                BOOST_REQUIRE_EQUAL(result, 2 + 3);\n            }).handle_exception([] (auto ep) {\n                BOOST_FAIL(\"No exception expected\");\n            }).finally([factory = std::move(factory), i, j = j & 1] {\n                if (i == 1 && j == 1) {\n                    BOOST_REQUIRE_EQUAL(factory->use_compression, 2);\n                } else {\n                    BOOST_REQUIRE_EQUAL(factory->use_compression, 0);\n                }\n            });\n            fs.emplace_back(std::move(f));\n          }\n        }\n    }\n    return when_all(fs.begin(), fs.end()).discard_result();\n}\n\nSEASTAR_TEST_CASE(test_rpc_connect_multi_compression_algo) {\n    auto factory1 = std::make_unique<cfactory>();\n    auto factory2 = std::make_unique<cfactory>(\"LZ4NEW\");\n    rpc::server_options so;\n    rpc::client_options co;\n    static rpc::multi_algo_compressor_factory server({factory1.get(), factory2.get()});\n    static rpc::multi_algo_compressor_factory client({factory2.get(), factory1.get()});\n    so.compressor_factory = &server;\n    co.compressor_factory = &client;\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    return rpc_test_env<>::do_with_thread(cfg, co, [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        env.register_handler(1, [](int a, int b) {\n            return make_ready_future<int>(a+b);\n        }).get();\n        auto sum = env.proto().make_client<int (int, int)>(1);\n        auto result = sum(c1, 2, 3).get();\n        BOOST_REQUIRE_EQUAL(result, 2 + 3);\n    }).finally([factory1 = std::move(factory1), factory2 = std::move(factory2)] {\n        BOOST_REQUIRE_EQUAL(factory1->use_compression, 0);\n        BOOST_REQUIRE_EQUAL(factory2->use_compression, 2);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_connect_abort) {\n    rpc_test_config cfg;\n    rpc_loopback_error_injector::config ecfg;\n    ecfg.connect_kind = loopback_error_injector::error::abort;\n    cfg.inject_error = ecfg;\n    return rpc_test_env<>::do_with_thread(cfg, [] (rpc_test_env<>& env) {\n        test_rpc_proto::client c1(env.proto(), {}, env.make_socket(), ipv4_addr());\n        env.register_handler(1, []() { return make_ready_future<>(); }).get();\n        auto f = env.proto().make_client<void ()>(1);\n        auto fut = f(c1);\n        c1.stop().get();\n        try {\n            fut.get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n        try {\n            f(c1).get();\n            BOOST_REQUIRE(false);\n        } catch (...) {}\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_cancel) {\n    using namespace std::chrono_literals;\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        bool rpc_executed = false;\n        int good = 0;\n        promise<> handler_called;\n        future<> f_handler_called = handler_called.get_future();\n        env.register_handler(1, [&rpc_executed, &handler_called] {\n            handler_called.set_value(); rpc_executed = true; return sleep(1ms);\n        }).get();\n        auto call = env.proto().make_client<void ()>(1);\n        promise<> cont;\n        rpc::cancellable cancel;\n        c1.suspend_for_testing(cont);\n        auto f = call(c1, cancel);\n        // cancel send side\n        cancel.cancel();\n        cont.set_value();\n        try {\n            f.get();\n        } catch(rpc::canceled_error&) {\n            good += !rpc_executed;\n        };\n        f = call(c1, cancel);\n        // cancel wait side\n        f_handler_called.then([&cancel] {\n            cancel.cancel();\n        }).get();\n        try {\n            f.get();\n        } catch(rpc::canceled_error&) {\n            good += 10*rpc_executed;\n        };\n        BOOST_REQUIRE_EQUAL(good, 11);\n    });\n}\n\nSEASTAR_TEST_CASE(test_message_to_big) {\n    rpc_test_config cfg;\n    cfg.resource_limits = {0, 1, 100};\n    return rpc_test_env<>::do_with_thread(cfg, [] (rpc_test_env<>& env, test_rpc_proto::client& c) {\n        bool good = true;\n        env.register_handler(1, [&] (sstring payload) mutable {\n            good = false;\n        }).get();\n        auto call = env.proto().make_client<void (sstring)>(1);\n        try {\n            call(c, uninitialized_string(101)).get();\n            good = false;\n        } catch(std::runtime_error& err) {\n        } catch(...) {\n            good = false;\n        }\n        BOOST_REQUIRE_EQUAL(good, true);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_remote_verb_error) {\n    rpc_test_config cfg;\n    return rpc_test_env<>::do_with_thread(cfg, [] (rpc_test_env<>& env) {\n        test_rpc_proto::client c1(env.proto(), {}, env.make_socket(), ipv4_addr());\n        env.register_handler(1, []() { throw std::runtime_error(\"error\"); }).get();\n        auto f = env.proto().make_client<void ()>(1);\n        BOOST_REQUIRE_THROW(f(c1).get(), rpc::remote_verb_error);\n        c1.stop().get();\n    });\n}\n\nstruct stream_test_result {\n    bool client_source_closed = false;\n    bool server_source_closed = false;\n    bool sink_exception = false;\n    bool sink_close_exception = false;\n    bool source_done_exception = false;\n    bool server_done_exception = false;\n    bool client_stop_exception = false;\n    bool server_stop_exception = false;\n    int server_sum = 0;\n    bool exception_while_creating_sink = false;\n};\n\nfuture<stream_test_result> stream_test_func(rpc_test_env<>& env, bool stop_client, bool stop_server, bool expect_connection_error = false) {\n    return seastar::async([&env, stop_client, stop_server, expect_connection_error] {\n        stream_test_result r;\n        test_rpc_proto::client c(env.proto(), {}, env.make_socket(), ipv4_addr());\n        future<> server_done = make_ready_future();\n        env.register_handler(1, [&](int i, rpc::source<int> source) {\n            BOOST_REQUIRE_EQUAL(i, 666);\n            auto sink = source.make_sink<serializer, sstring>();\n            auto sink_loop = seastar::async([sink] () mutable {\n                for (auto i = 0; i < 100; i++) {\n                    sink(\"seastar\").get();\n                    sleep(std::chrono::milliseconds(1)).get();\n                }\n            }).finally([sink] () mutable {\n                return sink.flush();\n            }).finally([sink] () mutable {\n                return sink.close();\n            }).finally([sink] {});\n\n            auto source_loop = seastar::async([source, &r] () mutable {\n                while (!r.server_source_closed) {\n                    auto data = source().get();\n                    if (data) {\n                        r.server_sum += std::get<0>(*data);\n                    } else {\n                        r.server_source_closed = true;\n                        try {\n                          // check that reading after eos does not crash\n                          // and throws correct exception\n                           source().get();\n                        } catch (rpc::stream_closed& ex) {\n                          // expected\n                        } catch (...) {\n                           BOOST_FAIL(\"wrong exception on reading from a stream after eos\");\n                        }\n                    }\n                }\n            });\n            server_done = when_all_succeed(std::move(sink_loop), std::move(source_loop)).discard_result();\n            return sink;\n        }).get();\n        auto call = env.proto().make_client<rpc::source<sstring> (int, rpc::sink<int>)>(1);\n        struct failed_to_create_sync{};\n        auto x = [&] {\n            try {\n                return c.make_stream_sink<serializer, int>(env.make_socket()).get();\n            } catch (...) {\n                c.stop().get();\n                throw failed_to_create_sync{};\n            }\n        };\n        try {\n            auto sink = x();\n            auto source = call(c, 666, sink).get();\n            auto source_done = seastar::async([&] {\n                while (!r.client_source_closed) {\n                    auto data = source().get();\n                    if (data) {\n                        BOOST_REQUIRE_EQUAL(std::get<0>(*data), \"seastar\");\n                    } else {\n                        r.client_source_closed = true;\n                    }\n                }\n            });\n            auto check_exception = [] (auto f) {\n                try {\n                    f.get();\n                } catch (...) {\n                    return true;\n                }\n                return false;\n            };\n            future<> stop_client_future = make_ready_future();\n            future<> stop_server_future = make_ready_future();\n            // With a connection error sink() will eventually fail, but we\n            // cannot guarantee when.\n            int max = expect_connection_error ? std::numeric_limits<int>::max()  : 101;\n            for (int i = 1; i < max; i++) {\n                if (i == 50) {\n                    if (stop_client) {\n                        // stop client while stream is in use\n                        stop_client_future = c.stop();\n                    }\n                    if (stop_server) {\n                        // stop server while stream is in use\n                        stop_server_future = env.server().shutdown();\n                    }\n                }\n                sleep(std::chrono::milliseconds(1)).get();\n                r.sink_exception = check_exception(sink(i));\n                if (r.sink_exception) {\n                    break;\n                }\n            }\n            r.sink_close_exception = check_exception(sink.close());\n            r.source_done_exception = check_exception(std::move(source_done));\n            r.server_done_exception = check_exception(std::move(server_done));\n            r.client_stop_exception = check_exception(!stop_client ? c.stop() : std::move(stop_client_future));\n            r.server_stop_exception = check_exception(!stop_server ? make_ready_future<>() : std::move(stop_server_future));\n            return r;\n        } catch(failed_to_create_sync&) {\n            r.exception_while_creating_sink = true;\n            return r;\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_stream_simple) {\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    return rpc_test_env<>::do_with(cfg, [] (rpc_test_env<>& env) {\n        return stream_test_func(env, false, false).then([] (stream_test_result r) {\n            BOOST_REQUIRE(r.client_source_closed);\n            BOOST_REQUIRE(r.server_source_closed);\n            BOOST_REQUIRE(r.server_sum == 5050);\n            BOOST_REQUIRE(!r.sink_exception);\n            BOOST_REQUIRE(!r.sink_close_exception);\n            BOOST_REQUIRE(!r.source_done_exception);\n            BOOST_REQUIRE(!r.server_done_exception);\n            BOOST_REQUIRE(!r.client_stop_exception);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_stream_stop_client) {\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    return rpc_test_env<>::do_with(cfg, [] (rpc_test_env<>& env) {\n        return stream_test_func(env, true, false).then([] (stream_test_result r) {\n            BOOST_REQUIRE(!r.client_source_closed);\n            BOOST_REQUIRE(!r.server_source_closed);\n            BOOST_REQUIRE(r.sink_exception);\n            BOOST_REQUIRE(r.sink_close_exception);\n            BOOST_REQUIRE(r.source_done_exception);\n            BOOST_REQUIRE(r.server_done_exception);\n            BOOST_REQUIRE(!r.client_stop_exception);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_stream_stop_server) {\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    return rpc_test_env<>::do_with(cfg, [] (rpc_test_env<>& env) {\n        return stream_test_func(env, false, true).then([] (stream_test_result r) {\n            BOOST_REQUIRE(!r.client_source_closed);\n            BOOST_REQUIRE(!r.server_source_closed);\n            BOOST_REQUIRE(r.sink_exception);\n            BOOST_REQUIRE(r.sink_close_exception);\n            BOOST_REQUIRE(r.source_done_exception);\n            BOOST_REQUIRE(r.server_done_exception);\n            BOOST_REQUIRE(!r.client_stop_exception);\n            BOOST_REQUIRE(!r.server_stop_exception);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_stream_connection_error) {\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    rpc_loopback_error_injector::config ecfg;\n    ecfg.server_rcv.limit = 50;\n    ecfg.server_rcv.kind = loopback_error_injector::error::abort;\n    cfg.inject_error = ecfg;\n    return rpc_test_env<>::do_with(cfg, [] (rpc_test_env<>& env) {\n        return stream_test_func(env, false, false, true).then([] (stream_test_result r) {\n            BOOST_REQUIRE(!r.client_source_closed);\n            BOOST_REQUIRE(!r.server_source_closed);\n            BOOST_REQUIRE(r.sink_exception);\n            BOOST_REQUIRE(r.sink_close_exception);\n            BOOST_REQUIRE(r.source_done_exception);\n            BOOST_REQUIRE(r.server_done_exception);\n            BOOST_REQUIRE(!r.client_stop_exception);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_stream_negotiation_error) {\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    rpc_loopback_error_injector::config ecfg;\n    ecfg.server_rcv.limit = 0;\n    ecfg.server_rcv.kind = loopback_error_injector::error::abort;\n    cfg.inject_error = ecfg;\n    return rpc_test_env<>::do_with(cfg, [] (rpc_test_env<>& env) {\n        return stream_test_func(env, false, false, true).then([] (stream_test_result r) {\n            BOOST_REQUIRE(r.exception_while_creating_sink);\n        });\n    });\n}\n\nstatic future<> test_rpc_connection_send_glitch(bool on_client) {\n    struct context {\n        int limit;\n        bool no_failures;\n        bool on_client;\n    };\n\n    return do_with(context{0, false, on_client}, [] (auto& ctx) {\n        return do_until([&ctx] {\n            return ctx.no_failures;\n        }, [&ctx] {\n            fmt::print(\"======== 8< ========\\n\");\n            fmt::print(\"  Checking {} limit\\n\", ctx.limit);\n            rpc_test_config cfg;\n            rpc_loopback_error_injector::config ecfg;\n            if (ctx.on_client) {\n                ecfg.client_snd.limit = ctx.limit;\n                ecfg.client_snd.kind = loopback_error_injector::error::one_shot;\n            } else {\n                ecfg.server_snd.limit = ctx.limit;\n                ecfg.server_snd.kind = loopback_error_injector::error::one_shot;\n            }\n            cfg.inject_error = ecfg;\n            return rpc_test_env<>::do_with_thread(cfg, [&ctx] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n                env.register_handler(1, [] {\n                    fmt::print(\"  reply\\n\");\n                    return seastar::sleep(std::chrono::milliseconds(100)).then([] {\n                        return make_ready_future<unsigned>(13);\n                    });\n                }).get();\n\n                ctx.no_failures = true;\n\n                for (int i = 0; i < 4; i++) {\n                    auto call = env.proto().make_client<unsigned ()>(1);\n                    fmt::print(\"  call {}\\n\", i);\n                    try {\n                        auto id = call(c1).get();\n                        fmt::print(\"    response: {}\\n\", id);\n                    } catch (...) {\n                        fmt::print(\"    responce: ex {}\\n\", std::current_exception());\n                        ctx.no_failures = false;\n                        ctx.limit++;\n                        break;\n                    }\n                    seastar::yield().get();\n                }\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_client_send_glitch) {\n    return test_rpc_connection_send_glitch(true);\n}\n\nSEASTAR_TEST_CASE(test_rpc_server_send_glitch) {\n    return test_rpc_connection_send_glitch(false);\n}\n\nSEASTAR_TEST_CASE(test_rpc_scheduling) {\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        auto sg = create_scheduling_group(\"rpc\", 100).get();\n        env.register_handler(1, sg, [] () {\n            return make_ready_future<unsigned>(internal::scheduling_group_index(current_scheduling_group()));\n        }).get();\n        auto call_sg_id = env.proto().make_client<unsigned ()>(1);\n        auto id = call_sg_id(c1).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg));\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_scheduling_connection_based) {\n    auto sg1 = create_scheduling_group(\"sg1\", 100).get();\n    auto sg1_kill = defer([&] () noexcept { destroy_scheduling_group(sg1).get(); });\n    auto sg2 = create_scheduling_group(\"sg2\", 100).get();\n    auto sg2_kill = defer([&] () noexcept { destroy_scheduling_group(sg2).get(); });\n    rpc::resource_limits limits;\n    limits.isolate_connection = [sg1, sg2] (sstring cookie) {\n        auto sg = current_scheduling_group();\n        if (cookie == \"sg1\") {\n            sg = sg1;\n        } else if (cookie == \"sg2\") {\n            sg = sg2;\n        }\n        rpc::isolation_config cfg;\n        cfg.sched_group = sg;\n        return cfg;\n    };\n    rpc_test_config cfg;\n    cfg.resource_limits = limits;\n    rpc_test_env<>::do_with_thread(cfg, [sg1, sg2] (rpc_test_env<>& env) {\n        rpc::client_options co1;\n        co1.isolation_cookie = \"sg1\";\n        test_rpc_proto::client c1(env.proto(), co1, env.make_socket(), ipv4_addr());\n        rpc::client_options co2;\n        co2.isolation_cookie = \"sg2\";\n        test_rpc_proto::client c2(env.proto(), co2, env.make_socket(), ipv4_addr());\n        env.register_handler(1, [] {\n            return make_ready_future<unsigned>(internal::scheduling_group_index(current_scheduling_group()));\n        }).get();\n        auto call_sg_id = env.proto().make_client<unsigned ()>(1);\n        unsigned id;\n        id = call_sg_id(c1).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg1));\n        id = call_sg_id(c2).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg2));\n        c1.stop().get();\n        c2.stop().get();\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_scheduling_connection_based_compatibility) {\n    auto sg1 = create_scheduling_group(\"sg1\", 100).get();\n    auto sg1_kill = defer([&] () noexcept { destroy_scheduling_group(sg1).get(); });\n    auto sg2 = create_scheduling_group(\"sg2\", 100).get();\n    auto sg2_kill = defer([&] () noexcept { destroy_scheduling_group(sg2).get(); });\n    rpc::resource_limits limits;\n    limits.isolate_connection = [sg1, sg2] (sstring cookie) {\n        auto sg = current_scheduling_group();\n        if (cookie == \"sg1\") {\n            sg = sg1;\n        } else if (cookie == \"sg2\") {\n            sg = sg2;\n        }\n        rpc::isolation_config cfg;\n        cfg.sched_group = sg;\n        return cfg;\n    };\n    rpc_test_config cfg;\n    cfg.resource_limits = limits;\n    rpc_test_env<>::do_with_thread(cfg, [sg1, sg2] (rpc_test_env<>& env) {\n        rpc::client_options co1;\n        co1.isolation_cookie = \"sg1\";\n        test_rpc_proto::client c1(env.proto(), co1, env.make_socket(), ipv4_addr());\n        rpc::client_options co2;\n        co2.isolation_cookie = \"sg2\";\n        test_rpc_proto::client c2(env.proto(), co2, env.make_socket(), ipv4_addr());\n        // An old client, that doesn't have an isolation cookie\n        rpc::client_options co3;\n        test_rpc_proto::client c3(env.proto(), co3, env.make_socket(), ipv4_addr());\n        // A server that uses sg1 if the client is old\n        env.register_handler(1, sg1, [] () {\n            return make_ready_future<unsigned>(internal::scheduling_group_index(current_scheduling_group()));\n        }).get();\n        auto call_sg_id = env.proto().make_client<unsigned ()>(1);\n        unsigned id;\n        id = call_sg_id(c1).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg1));\n        id = call_sg_id(c2).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg2));\n        id = call_sg_id(c3).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg1));\n        c1.stop().get();\n        c2.stop().get();\n        c3.stop().get();\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_scheduling_connection_based_async) {\n    scheduling_group sg1 = default_scheduling_group();\n    scheduling_group sg2 = default_scheduling_group();\n    auto sg1_kill = defer([&] () noexcept {\n        if (sg1 != default_scheduling_group())  {\n            destroy_scheduling_group(sg1).get();\n        }\n    });\n    auto sg2_kill = defer([&] () noexcept {\n        if (sg2 != default_scheduling_group()) {\n            destroy_scheduling_group(sg2).get();\n        }\n    });\n    rpc::resource_limits limits;\n    limits.isolate_connection = [&sg1, &sg2] (sstring cookie) {\n        future<seastar::scheduling_group> get_scheduling_group = make_ready_future<>().then([&sg1, &sg2, cookie] {\n            if (cookie == \"sg1\") {\n                if (sg1 == default_scheduling_group()) {\n                    return create_scheduling_group(\"sg1\", 100).then([&sg1] (seastar::scheduling_group sg) {\n                        sg1 = sg;\n                        return sg;\n                    });\n                } else {\n                    return make_ready_future<seastar::scheduling_group>(sg1);\n                }\n            } else if (cookie == \"sg2\") {\n                if (sg2 == default_scheduling_group()) {\n                    return create_scheduling_group(\"sg2\", 100).then([&sg2] (seastar::scheduling_group sg) {\n                        sg2 = sg;\n                        return sg;\n                    });\n                } else {\n                    return make_ready_future<seastar::scheduling_group>(sg2);\n                }\n            }\n            return make_ready_future<seastar::scheduling_group>(current_scheduling_group());\n        });\n\n        return get_scheduling_group.then([] (scheduling_group sg) {\n            rpc::isolation_config cfg;\n            cfg.sched_group = sg;\n            return cfg;\n        });\n    };\n    rpc_test_config cfg;\n    cfg.resource_limits = limits;\n    rpc_test_env<>::do_with_thread(cfg, [&sg1, &sg2] (rpc_test_env<>& env) {\n        rpc::client_options co1;\n        co1.isolation_cookie = \"sg1\";\n        test_rpc_proto::client c1(env.proto(), co1, env.make_socket(), ipv4_addr());\n        rpc::client_options co2;\n        co2.isolation_cookie = \"sg2\";\n        test_rpc_proto::client c2(env.proto(), co2, env.make_socket(), ipv4_addr());\n        env.register_handler(1, [] {\n            return make_ready_future<unsigned>(internal::scheduling_group_index(current_scheduling_group()));\n        }).get();\n        auto call_sg_id = env.proto().make_client<unsigned ()>(1);\n        unsigned id;\n        id = call_sg_id(c1).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg1));\n        id = call_sg_id(c2).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg2));\n        c1.stop().get();\n        c2.stop().get();\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_scheduling_connection_based_compatibility_async) {\n    scheduling_group sg1 = default_scheduling_group();\n    scheduling_group sg2 = default_scheduling_group();\n    scheduling_group sg3 = create_scheduling_group(\"sg3\", 100).get();\n    auto sg1_kill = defer([&] () noexcept {\n        if (sg1 != default_scheduling_group())  {\n            destroy_scheduling_group(sg1).get();\n        }\n    });\n    auto sg2_kill = defer([&] () noexcept {\n        if (sg2 != default_scheduling_group()) {\n            destroy_scheduling_group(sg2).get();\n        }\n    });\n    auto sg3_kill = defer([&] () noexcept { destroy_scheduling_group(sg3).get(); });\n    rpc::resource_limits limits;\n    limits.isolate_connection = [&sg1, &sg2] (sstring cookie) {\n        future<seastar::scheduling_group> get_scheduling_group = make_ready_future<>().then([&sg1, &sg2, cookie] {\n            if (cookie == \"sg1\") {\n                if (sg1 == default_scheduling_group()) {\n                    return create_scheduling_group(\"sg1\", 100).then([&sg1] (seastar::scheduling_group sg) {\n                        sg1 = sg;\n                        return sg;\n                    });\n                } else {\n                    return make_ready_future<seastar::scheduling_group>(sg1);\n                }\n            } else if (cookie == \"sg2\") {\n                if (sg2 == default_scheduling_group()) {\n                    return create_scheduling_group(\"sg2\", 100).then([&sg2] (seastar::scheduling_group sg) {\n                        sg2 = sg;\n                        return sg;\n                    });\n                } else {\n                    return make_ready_future<seastar::scheduling_group>(sg2);\n                }\n            }\n            return make_ready_future<seastar::scheduling_group>(current_scheduling_group());\n        });\n\n        return get_scheduling_group.then([] (scheduling_group sg) {\n            rpc::isolation_config cfg;\n            cfg.sched_group = sg;\n            return cfg;\n        });\n    };\n    rpc_test_config cfg;\n    cfg.resource_limits = limits;\n    rpc_test_env<>::do_with_thread(cfg, [&sg1, &sg2, &sg3] (rpc_test_env<>& env) {\n        rpc::client_options co1;\n        co1.isolation_cookie = \"sg1\";\n        test_rpc_proto::client c1(env.proto(), co1, env.make_socket(), ipv4_addr());\n        rpc::client_options co2;\n        co2.isolation_cookie = \"sg2\";\n        test_rpc_proto::client c2(env.proto(), co2, env.make_socket(), ipv4_addr());\n        // An old client, that doesn't have an isolation cookie\n        rpc::client_options co3;\n        test_rpc_proto::client c3(env.proto(), co3, env.make_socket(), ipv4_addr());\n        // A server that uses sg3 if the client is old\n        env.register_handler(1, sg3, [] () {\n            return make_ready_future<unsigned>(internal::scheduling_group_index(current_scheduling_group()));\n        }).get();\n        auto call_sg_id = env.proto().make_client<unsigned ()>(1);\n        unsigned id;\n        id = call_sg_id(c1).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg1));\n        id = call_sg_id(c2).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg2));\n        id = call_sg_id(c3).get();\n        BOOST_REQUIRE(id == internal::scheduling_group_index(sg3));\n        c1.stop().get();\n        c2.stop().get();\n        c3.stop().get();\n    }).get();\n}\n\nvoid test_compressor(std::function<std::unique_ptr<seastar::rpc::compressor>()> compressor_factory) {\n    using namespace seastar::rpc;\n\n    auto linearize = [&] (const auto& buffer) {\n        return seastar::visit(buffer.bufs,\n            [] (const temporary_buffer<char>& buf) {\n                return buf.clone();\n            },\n            [&] (const std::vector<temporary_buffer<char>>& bufs) {\n                auto buf = temporary_buffer<char>(buffer.size);\n                auto dst = buf.get_write();\n                for (auto& b : bufs) {\n                    dst = std::copy_n(b.get(), b.size(), dst);\n                }\n                return buf;\n            }\n        );\n    };\n\n    auto split_buffer = [&] (temporary_buffer<char> b, size_t chunk_size) {\n        std::vector<temporary_buffer<char>> bufs;\n        auto src = b.get();\n        auto n = b.size();\n        while (n) {\n            auto this_chunk = std::min(chunk_size, n);\n            bufs.emplace_back(this_chunk);\n            std::copy_n(src, this_chunk, bufs.back().get_write());\n            src += this_chunk;\n            n -= this_chunk;\n        }\n        return bufs;\n    };\n\n    auto clone = [&] (const auto& buffer) {\n        auto c = std::decay_t<decltype(buffer)>();\n        c.size = buffer.size;\n        c.bufs = seastar::visit(buffer.bufs,\n            [] (const temporary_buffer<char>& buf) -> decltype(c.bufs) {\n                return buf.clone();\n            },\n            [] (const std::vector<temporary_buffer<char>>& bufs) -> decltype(c.bufs) {\n                std::vector<temporary_buffer<char>> c;\n                c.reserve(bufs.size());\n                for (auto& b : bufs) {\n                    c.emplace_back(b.clone());\n                }\n                return c;\n            }\n        );\n        return c;\n    };\n\n    auto compressor = compressor_factory();\n\n    std::vector<noncopyable_function<std::tuple<sstring, size_t, snd_buf> ()>> inputs;\n\n    auto add_input = [&] (auto func_returning_tuple) {\n        inputs.emplace_back(std::move(func_returning_tuple));\n    };\n\n    auto& eng = testing::local_random_engine;\n    auto dist = std::uniform_int_distribution<int>(0, std::numeric_limits<char>::max());\n\n    auto snd = snd_buf(1);\n    *snd.front().get_write() = 'a';\n    add_input([snd = std::move(snd)] () mutable { return std::tuple(sstring(\"one byte, no headroom\"), size_t(0), std::move(snd)); });\n\n    snd = snd_buf(1);\n    *snd.front().get_write() = 'a';\n    add_input([snd = std::move(snd)] () mutable { return std::tuple(sstring(\"one byte, 128k of headroom\"), size_t(128 * 1024), std::move(snd)); });\n\n    auto gen_fill = [&](size_t s, sstring msg, std::optional<size_t> split = {}) {\n        auto buf = temporary_buffer<char>(s);\n        std::fill_n(buf.get_write(), s, 'a');\n\n        auto snd = snd_buf();\n        snd.size = s;\n        if (split) {\n            snd.bufs = split_buffer(buf.clone(), *split);\n        } else {\n            snd.bufs = buf.clone();\n        }\n        return std::tuple(msg, size_t(0), std::move(snd));\n    };\n\n\n    add_input(std::bind(gen_fill, 16 * 1024, \"single 16 kB buffer of \\'a\\'\"));\n\n    auto gen_rand = [&](size_t s, sstring msg, std::optional<size_t> split = {}) {\n        auto buf = temporary_buffer<char>(s);\n        std::generate_n(buf.get_write(), s, [&] { return dist(eng); });\n\n        auto snd = snd_buf();\n        snd.size = s;\n        if (split) {\n            snd.bufs = split_buffer(buf.clone(), *split);\n        } else {\n            snd.bufs = buf.clone();\n        }\n        return std::tuple(msg, size_t(0), std::move(snd));\n    };\n\n    add_input(std::bind(gen_rand, 16 * 1024, \"single 16 kB buffer of random\"));\n\n    auto gen_text = [&](size_t s, sstring msg, std::optional<size_t> split = {}) {\n        static const std::string_view text = \"The quick brown fox wants bananas for his long term health but sneaks bacon behind his wife's back. \";\n\n        auto buf = temporary_buffer<char>(s);\n        size_t n = 0;\n        while (n < s) {\n            auto rem = std::min(s - n, text.size());\n            std::copy(text.data(), text.data() + rem, buf.get_write() + n);\n            n += rem;\n        }\n\n        auto snd = snd_buf();\n        snd.size = s;\n        if (split) {\n            snd.bufs = split_buffer(buf.clone(), *split);\n        } else {\n            snd.bufs = buf.clone();\n        }\n        return std::tuple(msg, size_t(0), std::move(snd));\n    };\n\n#ifndef SEASTAR_DEBUG\n    auto buffer_sizes = { 1, 4 };\n#else\n    auto buffer_sizes = { 1 };\n#endif\n\n    for (auto s : buffer_sizes) {\n        for (auto ss : { 32, 64, 128, 48, 56, 246, 511 }) {\n            add_input(std::bind(gen_fill, s * 1024 * 1024, format(\"{} MB buffer of \\'a\\' split into {} kB - {}\", s, ss, ss), ss * 1024 - ss));\n            add_input(std::bind(gen_fill, s * 1024 * 1024, format(\"{} MB buffer of \\'a\\' split into {} kB\", s, ss), ss * 1024));\n            add_input(std::bind(gen_rand, s * 1024 * 1024, format(\"{} MB buffer of random split into {} kB\", s, ss), ss * 1024));\n\n            add_input(std::bind(gen_fill, s * 1024 * 1024 + 1, format(\"{} MB + 1B buffer of \\'a\\' split into {} kB\", s, ss), ss * 1024));\n            add_input(std::bind(gen_rand, s * 1024 * 1024 + 1, format(\"{} MB + 1B buffer of random split into {} kB\", s, ss), ss * 1024));\n        }\n\n        for (auto ss : { 128, 246, 511, 3567, 2*1024, 8*1024 }) {\n            add_input(std::bind(gen_fill, s * 1024 * 1024, format(\"{} MB buffer of \\'a\\' split into {} B\", s, ss), ss));\n            add_input(std::bind(gen_rand, s * 1024 * 1024, format(\"{} MB buffer of random split into {} B\", s, ss), ss));\n            add_input(std::bind(gen_text, s * 1024 * 1024, format(\"{} MB buffer of text split into {} B\", s, ss), ss));\n            add_input(std::bind(gen_fill, s * 1024 * 1024 - ss, format(\"{} MB - {}B buffer of \\'a\\' split into {} B\", s, ss, ss), ss));\n            add_input(std::bind(gen_rand, s * 1024 * 1024 - ss, format(\"{} MB - {}B buffer of random split into {} B\", s, ss, ss), ss));\n            add_input(std::bind(gen_text, s * 1024 * 1024 - ss, format(\"{} MB - {}B buffer of random split into {} B\", s, ss, ss), ss));\n        }\n    }\n\n    for (auto s : { 64*1024 + 5670, 16*1024 + 3421, 32*1024 - 321 }) {\n        add_input(std::bind(gen_fill, s, format(\"{} bytes buffer of \\'a\\'\", s)));\n        add_input(std::bind(gen_rand, s, format(\"{} bytes buffer of random\", s)));\n        add_input(std::bind(gen_text, s, format(\"{} bytes buffer of text\", s)));\n    }\n\n    std::vector<std::tuple<sstring, std::function<rcv_buf(snd_buf)>>> transforms {\n        { \"identity\", [] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = std::move(snd.bufs);\n            return rcv;\n        } },\n        { \"linearized\", [&linearize] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = linearize(snd);\n            return rcv;\n        } },\n        { \"split 1 B\", [&] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = split_buffer(linearize(snd), 1);\n            return rcv;\n        } },\n        { \"split 129 B\", [&] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = split_buffer(linearize(snd), 129);\n            return rcv;\n        } },\n        { \"split 4 kB\", [&] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = split_buffer(linearize(snd), 4096);\n            return rcv;\n        } },\n        { \"split 4 kB - 128\", [&] (snd_buf snd) {\n            rcv_buf rcv;\n            rcv.size = snd.size;\n            rcv.bufs = split_buffer(linearize(snd), 4096 - 128);\n            return rcv;\n        } },\n    };\n\n    auto sanity_check = [&] (const auto& buffer) {\n        auto actual_size = seastar::visit(buffer.bufs,\n            [] (const temporary_buffer<char>& buf) {\n                return buf.size();\n            },\n            [] (const std::vector<temporary_buffer<char>>& bufs) {\n                return boost::accumulate(bufs, size_t(0), [] (size_t sz, const temporary_buffer<char>& buf) {\n                    return sz + buf.size();\n                });\n            }\n        );\n        BOOST_CHECK_EQUAL(actual_size, buffer.size);\n    };\n\n    for (auto& in_func : inputs) {\n        auto in = in_func();\n        BOOST_TEST_MESSAGE(\"Input: \" << std::get<0>(in));\n        auto headroom = std::get<1>(in);\n        auto compressed = compressor->compress(headroom, clone(std::get<2>(in)));\n        sanity_check(compressed);\n\n        // Remove headroom\n        BOOST_CHECK_GE(compressed.size, headroom);\n        compressed.size -= headroom;\n        seastar::visit(compressed.bufs,\n            [&] (temporary_buffer<char>& buf) {\n                BOOST_CHECK_GE(buf.size(), headroom);\n                buf.trim_front(headroom);\n            },\n            [&] (std::vector<temporary_buffer<char>>& bufs) {\n                while (headroom) {\n                    BOOST_CHECK(!bufs.empty());\n                    auto to_remove = std::min(bufs.front().size(), headroom);\n                    bufs.front().trim_front(to_remove);\n                    if (bufs.front().empty() && bufs.size() > 1) {\n                        bufs.erase(bufs.begin());\n                    }\n                    headroom -= to_remove;\n                }\n            }\n        );\n\n        auto in_l = linearize(std::get<2>(in));\n\n        for (auto& t : transforms) {\n            BOOST_TEST_MESSAGE(\"  Transform: \" << std::get<0>(t));\n            auto received = std::get<1>(t)(clone(compressed));\n\n            auto decompressed = compressor->decompress(std::move(received));\n            sanity_check(decompressed);\n\n            BOOST_CHECK_EQUAL(decompressed.size, std::get<2>(in).size);\n\n            auto out_l = linearize(decompressed);\n\n            BOOST_CHECK_EQUAL(in_l.size(), out_l.size());\n            BOOST_CHECK(in_l == out_l);\n        }\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_lz4_compressor) {\n    test_compressor([] { return std::make_unique<rpc::lz4_compressor>(); });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_lz4_fragmented_compressor) {\n    test_compressor([] { return std::make_unique<rpc::lz4_fragmented_compressor>(); });\n}\n\n// Test reproducing issue #671: If timeout is time_point::max(), translating\n// it to relative timeout in the sender and then back in the receiver, when\n// these calculations happen across a millisecond boundary, overflowed the\n// integer and mislead the receiver to think the requested timeout was\n// negative, and cause it drop its response, so the RPC call never completed.\nSEASTAR_TEST_CASE(test_max_absolute_timeout) {\n    // The typical failure of this test is a hang. So we use semaphore to\n    // stop the test either when it succeeds, or after a long enough hang.\n    auto success = make_lw_shared<bool>(false);\n    auto done = make_lw_shared<semaphore>(0);\n    auto abrt = make_lw_shared<abort_source>();\n    (void) seastar::sleep_abortable(std::chrono::seconds(3), *abrt).then([done, success] {\n        done->signal(1);\n    }).handle_exception([] (std::exception_ptr) {});\n    rpc::client_options co;\n    co.send_timeout_data = 1;\n    (void)rpc_test_env<>::do_with_thread(rpc_test_config(), co, [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        env.register_handler(1, [](int a, int b) {\n            return make_ready_future<int>(a+b);\n        }).get();\n        auto sum = env.proto().make_client<int (int, int)>(1);\n        // The bug only reproduces if the calculation done on the sender\n        // and receiver sides, happened across a millisecond boundary.\n        // We can't control when it happens, so we just need to loop many\n        // times, at least many milliseconds, to increase the probability\n        // that we catch the bug. Experimentally, if we loop for 200ms, we\n        // catch the bug in #671 virtually every time.\n        auto until = seastar::lowres_clock::now() + std::chrono::milliseconds(200);\n        while (seastar::lowres_clock::now() <= until) {\n            auto result = sum(c1, rpc::rpc_clock_type::time_point::max(), 2, 3).get();\n            BOOST_REQUIRE_EQUAL(result, 2 + 3);\n        }\n    }).then([success, done, abrt] {\n        *success = true;\n        abrt->request_abort();\n        done->signal();\n    });\n    return done->wait().then([done, success] {\n        BOOST_REQUIRE(*success);\n    });\n}\n\n// Similar to the above test: Test that a relative timeout duration::max()\n// also works, and again doesn't cause the timeout wrapping around to the\n// past and causing dropped responses.\nSEASTAR_TEST_CASE(test_max_relative_timeout) {\n    // The typical failure of this test is a hang. So we use semaphore to\n    // stop the test either when it succeeds, or after a long enough hang.\n    auto success = make_lw_shared<bool>(false);\n    auto done = make_lw_shared<semaphore>(0);\n    auto abrt = make_lw_shared<abort_source>();\n    (void) seastar::sleep_abortable(std::chrono::seconds(3), *abrt).then([done, success] {\n        done->signal(1);\n    }).handle_exception([] (std::exception_ptr) {});\n    rpc::client_options co;\n    co.send_timeout_data = 1;\n    (void)rpc_test_env<>::do_with_thread(rpc_test_config(), co, [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        env.register_handler(1, [](int a, int b) {\n            return make_ready_future<int>(a+b);\n        }).get();\n        auto sum = env.proto().make_client<int (int, int)>(1);\n        // The following call used to always hang, when max()+now()\n        // overflowed and appeared to be a negative timeout.\n        auto result = sum(c1, rpc::rpc_clock_type::duration::max(), 2, 3).get();\n        BOOST_REQUIRE_EQUAL(result, 2 + 3);\n    }).then([success, done, abrt] {\n        *success = true;\n        abrt->request_abort();\n        done->signal();\n    });\n    return done->wait().then([done, success] {\n        BOOST_REQUIRE(*success);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_tuple) {\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        env.register_handler(1, [] () {\n            return make_ready_future<rpc::tuple<int, long>>(rpc::tuple<int, long>(1, 0x7'0000'0000L));\n        }).get();\n        auto f1 = env.proto().make_client<rpc::tuple<int, long> ()>(1);\n        auto result = f1(c1).get();\n        BOOST_REQUIRE_EQUAL(std::get<0>(result), 1);\n        BOOST_REQUIRE_EQUAL(std::get<1>(result), 0x7'0000'0000L);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_nonvariadic_client_variadic_server) {\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        // Server is variadic\n        env.register_handler(1, [] () {\n            return make_ready_future<rpc::tuple<int, long>>(rpc::tuple(1, 0x7'0000'0000L));\n        }).get();\n        // Client is non-variadic\n        auto f1 = env.proto().make_client<future<rpc::tuple<int, long>> ()>(1);\n        auto result = f1(c1).get();\n        BOOST_REQUIRE_EQUAL(std::get<0>(result), 1);\n        BOOST_REQUIRE_EQUAL(std::get<1>(result), 0x7'0000'0000L);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_variadic_client_nonvariadic_server) {\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        // Server is nonvariadic\n        env.register_handler(1, [] () {\n            return make_ready_future<rpc::tuple<int, long>>(rpc::tuple<int, long>(1, 0x7'0000'0000L));\n        }).get();\n        // Client is variadic\n        auto f1 = env.proto().make_client<future<rpc::tuple<int, long>> ()>(1);\n        auto result = f1(c1).get();\n        BOOST_REQUIRE_EQUAL(std::get<0>(result), 1);\n        BOOST_REQUIRE_EQUAL(std::get<1>(result), 0x7'0000'0000L);\n    });\n}\n\nSEASTAR_TEST_CASE(test_handler_registration) {\n    rpc_test_config cfg;\n    rpc_loopback_error_injector::config ecfg;\n    ecfg.connect_kind = loopback_error_injector::error::abort;\n    cfg.inject_error = ecfg;\n    return rpc_test_env<>::do_with_thread(cfg, [] (rpc_test_env<>& env) {\n        auto& proto = env.proto();\n\n        // new proto must be empty\n        BOOST_REQUIRE(!proto.has_handlers());\n\n        // non-existing handler should not be found\n        BOOST_REQUIRE(!proto.has_handler(1));\n\n        // unregistered non-existing handler should return ready future\n        auto fut = proto.unregister_handler(1);\n        BOOST_REQUIRE(fut.available() && !fut.failed());\n        fut.get();\n\n        // existing handler should be found\n        auto handler = [] () { return make_ready_future<>(); };\n        proto.register_handler(1, handler);\n        BOOST_REQUIRE(proto.has_handler(1));\n\n        // cannot register handler if already registered\n        BOOST_REQUIRE_THROW(proto.register_handler(1, handler), std::runtime_error);\n\n        // unregistered handler should not be found\n        proto.unregister_handler(1).get();\n        BOOST_REQUIRE(!proto.has_handler(1));\n\n        // re-registering a handler should succeed\n        proto.register_handler(1, handler);\n        BOOST_REQUIRE(proto.has_handler(1));\n\n        // proto with handlers must not be empty\n        BOOST_REQUIRE(proto.has_handlers());\n    });\n}\n\nSEASTAR_TEST_CASE(test_unregister_handler) {\n    using namespace std::chrono_literals;\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        promise<> handler_called;\n        future<> f_handler_called = handler_called.get_future();\n        bool rpc_executed = false;\n        bool rpc_completed = false;\n\n        auto reset_state = [&f_handler_called, &rpc_executed, &rpc_completed] {\n            if (f_handler_called.available()) {\n                f_handler_called.get();\n            }\n            rpc_executed = false;\n            rpc_completed = false;\n        };\n\n        auto get_handler = [&handler_called, &rpc_executed, &rpc_completed] {\n            return [&handler_called, &rpc_executed, &rpc_completed] {\n                handler_called.set_value();\n                rpc_executed = true;\n                return sleep(1ms).then([&rpc_completed] {\n                    rpc_completed = true;\n                });\n            };\n        };\n\n        // handler should not run if unregistered before being called\n        env.register_handler(1, get_handler()).get();\n        env.unregister_handler(1).get();\n        BOOST_REQUIRE(!f_handler_called.available());\n        BOOST_REQUIRE(!rpc_executed);\n        BOOST_REQUIRE(!rpc_completed);\n\n        // verify normal execution path\n        env.register_handler(1, get_handler()).get();\n        auto call = env.proto().make_client<void ()>(1);\n        call(c1).get();\n        BOOST_REQUIRE(f_handler_called.available());\n        BOOST_REQUIRE(rpc_executed);\n        BOOST_REQUIRE(rpc_completed);\n        reset_state();\n\n        // call should fail after handler is unregistered\n        env.unregister_handler(1).get();\n        try {\n            call(c1).get();\n            BOOST_REQUIRE(false);\n        } catch (rpc::unknown_verb_error&) {\n            // expected\n        } catch (...) {\n            std::cerr << \"call failed in an unexpected way: \" << std::current_exception() << std::endl;\n            BOOST_REQUIRE(false);\n        }\n        BOOST_REQUIRE(!f_handler_called.available());\n        BOOST_REQUIRE(!rpc_executed);\n        BOOST_REQUIRE(!rpc_completed);\n\n        // unregistered is allowed while call is in flight\n        auto delayed_unregister = [&env] {\n            return sleep(500us).then([&env] {\n                return env.unregister_handler(1);\n            });\n        };\n\n        env.register_handler(1, get_handler()).get();\n        try {\n            when_all_succeed(call(c1), delayed_unregister()).get();\n            reset_state();\n        } catch (rpc::unknown_verb_error&) {\n            // expected\n        } catch (...) {\n            std::cerr << \"call failed in an unexpected way: \" << std::current_exception() << std::endl;\n            BOOST_REQUIRE(false);\n        }\n\n        // verify that unregister_handler waits for pending requests to finish\n        {\n            promise<> handler_reached_promise;\n            promise<> handler_go_promise;\n            sstring value_to_return = \"before_unregister\";\n            env.register_handler(1, [&]() -> future<sstring> {\n                handler_reached_promise.set_value();\n                return handler_go_promise.get_future().then([&] { return value_to_return; });\n            }).get();\n            auto f = env.proto().make_client<future<sstring>()>(1);\n            auto response_future = f(c1);\n            handler_reached_promise.get_future().get();\n            auto unregister_future = env.unregister_handler(1).then([&] {\n                value_to_return = \"after_unregister\";\n            });\n            BOOST_REQUIRE(!unregister_future.available());\n            sleep(1ms).get();\n            BOOST_REQUIRE(!unregister_future.available());\n            handler_go_promise.set_value();\n            unregister_future.get();\n            BOOST_REQUIRE_EQUAL(response_future.get(), \"before_unregister\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_loggers) {\n    static seastar::logger log(\"dummy\");\n    log.set_level(log_level::debug);\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env, test_rpc_proto::client& c1) {\n        socket_address dummy_addr;\n        auto& proto = env.proto();\n        auto& logger = proto.get_logger();\n        logger(dummy_addr, \"Hello0\");\n        logger(dummy_addr, log_level::debug, \"Hello1\");\n        proto.set_logger(&log);\n        logger(dummy_addr, \"Hello2\");\n        logger(dummy_addr, log_level::debug, \"Hello3\");\n        proto.set_logger(nullptr);\n        logger(dummy_addr, \"Hello6\");\n        logger(dummy_addr, log_level::debug, \"Hello7\");\n    });\n}\n\nSEASTAR_TEST_CASE(test_connection_id_format) {\n    rpc::connection_id cid = rpc::connection_id::make_id(0x123, 1);\n    std::string res = format(\"{}\", cid);\n    BOOST_REQUIRE_EQUAL(res, \"1230001\");\n    return make_ready_future<>();\n}\n\nstatic_assert(std::is_same_v<decltype(rpc::tuple(1U, 1L)), rpc::tuple<unsigned, long>>, \"rpc::tuple deduction guid not working\");\n\nSEASTAR_TEST_CASE(test_client_info) {\n    return rpc_test_env<>::do_with(rpc_test_config(), [] (rpc_test_env<>& env) {\n        rpc::client_info info{.server{env.server()}, .conn_id{0}};\n        const rpc::client_info& const_info = *const_cast<rpc::client_info*>(&info);\n\n        info.attach_auxiliary(\"key\", 0);\n        BOOST_REQUIRE_EQUAL(const_info.retrieve_auxiliary<int>(\"key\"), 0);\n        info.retrieve_auxiliary<int>(\"key\") = 1;\n        BOOST_REQUIRE_EQUAL(const_info.retrieve_auxiliary<int>(\"key\"), 1);\n\n        BOOST_REQUIRE_EQUAL(info.retrieve_auxiliary_opt<int>(\"key\"), &info.retrieve_auxiliary<int>(\"key\"));\n        BOOST_REQUIRE_EQUAL(const_info.retrieve_auxiliary_opt<int>(\"key\"), &const_info.retrieve_auxiliary<int>(\"key\"));\n\n        BOOST_REQUIRE_EQUAL(info.retrieve_auxiliary_opt<int>(\"missing\"), nullptr);\n        BOOST_REQUIRE_EQUAL(const_info.retrieve_auxiliary_opt<int>(\"missing\"), nullptr);\n\n        return make_ready_future<>();\n    });\n}\n\nvoid send_messages_and_check_timeouts(rpc_test_env<>& env, test_rpc_proto::client& cln) {\n    env.register_handler(1, [](int v) {\n        return seastar::sleep(std::chrono::seconds(v)).then([v] {\n            return make_ready_future<int>(v);\n        });\n    }).get();\n\n    auto call = env.proto().template make_client<int(int)>(1);\n    auto start = std::chrono::steady_clock::now();\n    auto f1 = call(cln, std::chrono::milliseconds(400), 3).finally([start] {\n        auto end = std::chrono::steady_clock::now();\n        BOOST_REQUIRE(end - start < std::chrono::seconds(1));\n    });\n    auto f2 = call(cln, std::chrono::milliseconds(600), 3).finally([start] {\n        auto end = std::chrono::steady_clock::now();\n        BOOST_REQUIRE(end - start < std::chrono::seconds(1));\n    });\n    BOOST_REQUIRE_THROW(f1.get(), seastar::rpc::timeout_error);\n    BOOST_REQUIRE_THROW(f2.get(), seastar::rpc::timeout_error);\n}\n\nSEASTAR_TEST_CASE(test_rpc_send_timeout) {\n    rpc_test_config cfg;\n    return rpc_test_env<>::do_with_thread(cfg, [] (auto& env, auto& cln) {\n        send_messages_and_check_timeouts(env, cln);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_send_timeout_on_connect) {\n    rpc_test_config cfg;\n    rpc_loopback_error_injector::config ecfg;\n    ecfg.connect_delay = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::seconds(5));\n    cfg.inject_error = ecfg;\n    return rpc_test_env<>::do_with_thread(cfg, [] (auto& env, auto& cln) {\n        send_messages_and_check_timeouts(env, cln);\n    });\n}\n\nSEASTAR_TEST_CASE(test_rpc_abort_connection) {\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), [] (rpc_test_env<>& env) {\n        test_rpc_proto::client c1(env.proto(), {}, env.make_socket(), ipv4_addr());\n        int arrived = 0;\n        env.register_handler(1, [&arrived] (rpc::client_info& cinfo, int x) {\n            BOOST_REQUIRE_EQUAL(x, arrived++);\n            if (arrived == 2) {\n                cinfo.server.abort_connection(cinfo.conn_id);\n            }\n            // The third message won't arrive because we abort the connection.\n\n            return 0;\n        }).get();\n        auto f = env.proto().make_client<int (int)>(1);\n        BOOST_REQUIRE_EQUAL(f(c1, 0).get(), 0);\n        BOOST_REQUIRE_THROW(f(c1, 1).get(), rpc::closed_error);\n        BOOST_REQUIRE_THROW(f(c1, 2).get(), rpc::closed_error);\n        BOOST_REQUIRE_EQUAL(arrived, 2);\n        c1.stop().get();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_metric_domains) {\n    auto do_one_echo = [] (rpc_test_env<>& env, test_rpc_proto::client& cln, int nr_calls) {\n        env.register_handler(1, [] (int v) { return make_ready_future<int>(v); }).get();\n        auto call = env.proto().template make_client<int(int)>(1);\n        for (int i = 0; i < nr_calls; i++) {\n            call(cln, i).get();\n        }\n    };\n\n    rpc::client_options client_opt;\n\n    client_opt.metrics_domain = \"dom1\";\n    auto p1 = rpc_test_env<>::do_with_thread(rpc_test_config(), client_opt, [&] (auto& env, auto& cln) { do_one_echo(env, cln, 3); });\n    client_opt.metrics_domain = \"dom2\";\n    auto p2 = rpc_test_env<>::do_with_thread(rpc_test_config(), client_opt, [&] (auto& env, auto& cln) { do_one_echo(env, cln, 2); });\n    auto p3 = rpc_test_env<>::do_with_thread(rpc_test_config(), client_opt, [&] (auto& env, auto& cln) { do_one_echo(env, cln, 5); });\n    when_all(std::move(p1), std::move(p2), std::move(p3)).discard_result().get();\n\n    auto get_metrics = [] (std::string name, std::string domain) -> int {\n        const auto& values = seastar::metrics::impl::get_value_map();\n        const auto& mf = values.find(name);\n        BOOST_REQUIRE(mf != values.end());\n        for (auto&& mi : mf->second) {\n            for (auto&&li : mi.first.labels()) {\n                if (li.first == \"domain\" && li.second.value() == domain) {\n                    return mi.second->get_function()().i();\n                }\n            }\n        }\n        BOOST_FAIL(\"cannot find requested metrics\");\n        return 0;\n    };\n\n    // Negotiation messages also count, so +1 for default domain and +2 for \"dom\" one\n    BOOST_CHECK_EQUAL(get_metrics(\"rpc_client_sent_messages\", \"dom1\"), 4);\n    BOOST_CHECK_EQUAL(get_metrics(\"rpc_client_sent_messages\", \"dom2\"), 9);\n}\n\n// Extract a piece of contiguous data from the front of the buffer (and trim the extracted front away).\ntemplate <typename T>\nrequires std::is_trivially_copyable_v<T>\nT read_from_rcv_buf(rpc::rcv_buf& data) {\n    if (data.size < sizeof(T)) {\n        throw std::runtime_error(\"Truncated compressed RPC frame\");\n    }\n    auto it = std::get_if<temporary_buffer<char>>(&data.bufs);\n    if (!it) {\n        it = std::get<std::vector<temporary_buffer<char>>>(data.bufs).data();\n    }\n    std::array<T, 1> out;\n    auto out_span = std::as_writable_bytes(std::span(out)).subspan(0);\n    while (out_span.size()) {\n        size_t n = std::min<size_t>(out_span.size(), it->size());\n        std::memcpy(static_cast<void*>(out_span.data()), it->get(), n);\n        out_span = out_span.subspan(n);\n        it->trim_front(n);\n        ++it;\n        data.size -= n;\n    }\n    return out[0];\n}\n\n\n// Test the use of empty compressed frames as a method of communication between compressors.\nSEASTAR_THREAD_TEST_CASE(test_compressor_empty_frames) {\n    static const sstring compressor_name = \"TEST\";\n    struct tracker {\n        struct compressor;\n        compressor* _compressor;\n\n        // When `send_metadata(x)` is called, this compressor sends a piece of metadata (x) to its peer,\n        // by prepending it to an empty compressed frame.\n        // It can be read from the peer with `receive_metadata()`.\n        struct compressor : public rpc::compressor {\n            tracker& _tracker;\n\n            std::unique_ptr<rpc::compressor> _delegate;\n\n            using metadata = uint64_t;\n            std::deque<metadata> _send_queue;\n            std::deque<metadata> _recv_queue;\n            semaphore _metadata_received{0};\n\n            std::function<future<>()> _send_empty_frame;\n            condition_variable _needs_progress;\n            future<> _progress_fiber;\n\n            future<> start_progress_fiber() {\n                while (true) {\n                    co_await _needs_progress.when([&] { return !_send_queue.empty(); });\n                    co_await _send_empty_frame();\n                }\n            }\n            future<> close() noexcept override {\n                _needs_progress.broken();\n                return std::move(_progress_fiber).handle_exception([] (const auto&) {});\n            }\n            void send_metadata(metadata x) {\n                _send_queue.push_back(x);\n                _needs_progress.signal();\n            }\n            future<metadata> receive_metadata() {\n                co_await _metadata_received.wait();\n                auto x = _recv_queue.front();\n                _recv_queue.pop_front();\n                co_return x;\n            }\n\n            rpc::snd_buf compress(size_t head_space, rpc::snd_buf data) override {\n                if (!_send_queue.empty()) {\n                    auto x = _delegate->compress(head_space + 1 + sizeof(metadata), data.size ? std::move(data) : rpc::snd_buf(temporary_buffer<char>()));\n                    seastar::write_be<uint8_t>(x.front().get_write()+head_space, 1);\n                    seastar::write_be<uint64_t>(x.front().get_write()+head_space+1, _send_queue.front());\n                    _send_queue.pop_front();\n                    return x;\n                } else {\n                    auto x = _delegate->compress(head_space + 1, data.size ? std::move(data) : rpc::snd_buf(temporary_buffer<char>()));\n                    seastar::write_be<uint8_t>(x.front().get_write()+head_space, 0);\n                    return x;\n                }\n            }\n            rpc::rcv_buf decompress(rpc::rcv_buf data) override {\n                if (net::ntoh(read_from_rcv_buf<uint8_t>(data))) {\n                    _recv_queue.push_back(net::ntoh(read_from_rcv_buf<uint64_t>(data)));\n                    _metadata_received.signal();\n                }\n                return _delegate->decompress(std::move(data));\n            }\n            sstring name() const override {\n                return compressor_name;\n            }\n\n            compressor(tracker& tracker, std::function<future<>()> send_empty_frame)\n                : _tracker(tracker)\n                , _delegate(std::make_unique<rpc::lz4_fragmented_compressor>())\n                , _send_empty_frame(std::move(send_empty_frame))\n                , _progress_fiber(start_progress_fiber())\n            {\n                _tracker._compressor = this;\n            }\n            ~compressor() {\n                _tracker._compressor = nullptr;\n            }\n        };\n\n        struct factory : public rpc::compressor::factory {\n            tracker& _tracker;\n\n            factory(tracker& tracker) : _tracker(tracker) {\n            }\n            const sstring& supported() const override {\n                return compressor_name;\n            }\n            std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server, std::function<future<>()> send_empty_frame) const override {\n                if (feature == supported()) {\n                    return std::make_unique<compressor>(_tracker, std::move(send_empty_frame));\n                }\n                return nullptr;\n            }\n            std::unique_ptr<rpc::compressor> negotiate(sstring feature, bool is_server) const override {\n                abort();\n            }\n        };\n    };\n\n    tracker server_tracker;\n    tracker client_tracker;\n    tracker::factory server_factory{server_tracker};\n    tracker::factory client_factory{client_tracker};\n    rpc::server_options so{.compressor_factory = &server_factory};\n    rpc::client_options co{.compressor_factory = &client_factory};\n    rpc_test_config cfg;\n    cfg.server_options = so;\n\n    rpc_test_env<>::do_with_thread(cfg, co, [&] (rpc_test_env<>& env, test_rpc_proto::client& c) {\n        // Perform an RPC once to initialize the connection and compressors.\n        env.register_handler(1, []() { return 42; }).get();\n        auto proto_client = env.proto().make_client<int()>(1);\n        BOOST_REQUIRE_EQUAL(proto_client(c).get(), 42);\n        BOOST_REQUIRE(client_tracker._compressor);\n        BOOST_REQUIRE(server_tracker._compressor);\n        // Check that both compressors can send metadata to each other.\n        for (int i = 0; i < 3; ++i) {\n            client_tracker._compressor->send_metadata(2*i);\n            BOOST_REQUIRE_EQUAL(server_tracker._compressor->receive_metadata().get(), 2*i);\n            server_tracker._compressor->send_metadata(2*i+1);\n            BOOST_REQUIRE_EQUAL(client_tracker._compressor->receive_metadata().get(), 2*i+1);\n        }\n    }).get();\n}\n\nSEASTAR_TEST_CASE(test_timeout_cancel) {\n    rpc::client_options co;\n    co.send_timeout_data = true;\n    return rpc_test_env<>::do_with_thread(rpc_test_config(), co, [&] (rpc_test_env<>& env, test_rpc_proto::client& cl) {\n        abort_source abort_handler;\n        uint32_t id = 1;\n        int sent = 0;\n        int received = 0;\n        condition_variable cond;\n        env.register_handler(id, [&] (int x) -> future<int> {\n            BOOST_TEST_MESSAGE(format(\"received value={}\", x));\n            received = x;\n            cond.signal();\n            co_await sleep_abortable(std::chrono::seconds(10), abort_handler);\n            co_return x;\n        }).get();\n        auto echo = env.proto().make_client<int (int)>(1);\n        {\n            // Relative timeout\n            auto f = echo(cl, std::chrono::milliseconds(10), ++sent);\n            BOOST_REQUIRE_THROW(f.get(), rpc::timeout_error);\n            while (received != sent) {\n                cond.wait(std::chrono::milliseconds(10)).get();\n            }\n        }\n        {\n            // Absolute timeout\n            auto f = echo(cl, seastar::lowres_clock::now() + std::chrono::milliseconds(10), ++sent);\n            BOOST_REQUIRE_THROW(f.get(), rpc::timeout_error);\n            while (received != sent) {\n                cond.wait(std::chrono::milliseconds(10)).get();\n            }\n        }\n        {\n            // Synchronous cancel before relative timeout\n            rpc::cancellable cancel_rpc;\n            auto f = echo(cl, std::chrono::milliseconds(10), cancel_rpc, ++sent);\n            BOOST_REQUIRE(!f.available());\n            cancel_rpc.cancel();\n            BOOST_REQUIRE_THROW(f.get(), rpc::canceled_error);\n            while (received != sent) {\n                cond.wait(std::chrono::milliseconds(10)).get();\n            }\n        }\n        {\n            // Synchronous cancel before absolute timeout\n            rpc::cancellable cancel_rpc;\n            auto f = echo(cl, seastar::lowres_clock::now() + std::chrono::milliseconds(10), cancel_rpc, ++sent);\n            BOOST_REQUIRE(!f.available());\n            cancel_rpc.cancel();\n            BOOST_REQUIRE_THROW(f.get(), rpc::canceled_error);\n            while (received != sent) {\n                cond.wait(std::chrono::milliseconds(10)).get();\n            }\n        }\n        {\n            // Cancel before timeout while rpc is in flight\n            rpc::cancellable cancel_rpc;\n            auto f = echo(cl, seastar::lowres_clock::now() + std::chrono::milliseconds(10), cancel_rpc, +sent);\n            BOOST_REQUIRE(!f.available());\n            // Wait until the rpc is received, then cancel\n            while (received != sent) {\n                cond.wait(std::chrono::milliseconds(10)).get();\n            }\n            cancel_rpc.cancel();\n            BOOST_REQUIRE_THROW(f.get(), rpc::canceled_error);\n        }\n        abort_handler.request_abort();\n        env.unregister_handler(id).get();\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_rpc_stream_backpressure_across_shards) {\n    static seastar::logger log(\"test\");\n    rpc::server_options so;\n    so.streaming_domain = rpc::streaming_domain_type(1);\n    rpc_test_config cfg;\n    cfg.server_options = so;\n    rpc_test_env<>::do_with_thread(cfg, [] (rpc_test_env<>& env) {\n        auto long_task_queue_state = reactor::test::get_long_task_queue_state();\n        auto restore_long_task_queue_state = deferred_action([&long_task_queue_state] () noexcept {\n            reactor::test::restore_long_task_queue_state(long_task_queue_state).get();\n        });\n        smp::invoke_on_all([&] {\n            reactor::test::set_abort_on_too_long_task_queue(true);\n            reactor::test::set_max_task_backlog(500);\n        }).get();\n\n        constexpr int msg_id = 1;\n        env.register_handler(msg_id, [] (shard_id sending_shard, size_t msgs_to_send, rpc::source<sstring> source) {\n            auto sink = source.make_sink<serializer, sstring>();\n\n            // It is safe to drop the future since the caller awaits for the stream to get closed.\n            (void)seastar::async([sending_shard, msgs_to_send, source, sink] () mutable {\n                auto close_sink = deferred_close(sink);\n                log.info(\"Handler: send {} messages to shard {}: starting\", msgs_to_send, sending_shard);\n                sstring data;\n                data.resize(64, 'x');\n                for (size_t i = 0; i < msgs_to_send; ++i) {\n                    sink(data).get();\n                }\n                sink.flush().get();\n                close_sink.close_now();\n                log.info(\"Handler: send {} messages to shard {}: done\", msgs_to_send, sending_shard);\n                // After closing the sink, any further send should throw stream_closed\n                // Reproducer for https://github.com/scylladb/seastar/issues/3088\n                BOOST_REQUIRE_THROW(sink(data).get(), rpc::stream_closed);\n            });\n\n            return sink;\n        }).get();\n\n        size_t msgs_per_shard = 1000000;\n#ifdef SEASTAR_DEBUG\n        msgs_per_shard = 50000;\n#endif\n        env.invoke_on_all([&] (rpc_test_env<>::rpc_test_service& s) {\n            return async([&] {\n                test_rpc_proto::client cl(env.proto(), {}, env.make_socket(), ipv4_addr());\n                auto stop_cl = deferred_stop(cl);\n                auto sink = cl.make_stream_sink<serializer, sstring>(env.make_socket()).get();\n                auto close_sink = deferred_close(sink);\n                auto call = env.proto().make_client<rpc::source<sstring> (shard_id, size_t, rpc::sink<sstring>)>(msg_id);\n                auto source = call(cl, this_shard_id(), msgs_per_shard, sink).get();\n\n                size_t count = 0;\n                bool end_of_stream = false;\n                try {\n                    // Loop indefinitely, until we get rpc::stream_closed\n                    for (;;) {\n                        if (auto data = source().get()) {\n                            if (count && !(count % 100000)) {\n                                log.debug(\"cl_rep_loop: received {} messages...\", count);\n                            }\n                            count++;\n                            continue;\n                        } else {\n                            if (std::exchange(end_of_stream, true)) {\n                                auto msg = \"cl_rep_loop: received second end-of-stream\";\n                                log.error(\"{}\", msg);\n                                throw std::runtime_error(msg);\n                            }\n                            log.debug(\"cl_rep_loop: got end-of-stream\");\n                            // Wait until we get the `stream_closed` error\n                            // to make sure the sender exited.\n                            // Otherwise we'd need another mechanism to await for it.\n                            continue;\n                        }\n                    }\n                } catch (const rpc::stream_closed&) {\n                    log.debug(\"cl_rep_loop: stream closed\");\n                } catch (...) {\n                    auto msg = format(\"cl_rep_loop: unexpected exception: {}\", std::current_exception());\n                    log.error(\"{}\", msg);\n                    throw std::runtime_error(msg);\n                }\n                log.info(\"cl_rep_loop: received {} messages\", count);\n                if (count != msgs_per_shard) {\n                    auto msg = format(\"cl_rep_loop: expected {}, got {}\", msgs_per_shard, count);\n                    log.error(\"{}\", msg);\n                    throw std::runtime_error(msg);\n                }\n            });\n        }).get();\n    }).get();\n}\n"
  },
  {
    "path": "tests/unit/scheduling_group_nesting_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2025 ScyllaDB Ltd.\n */\n\n#include <fmt/core.h>\n#include <fmt/ranges.h>\n#include <fmt/chrono.h>\n#include <vector>\n#include <chrono>\n\n#include <seastar/core/thread.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/testing/random.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/later.hh>\n\nusing namespace seastar;\n\nstruct layout {\n    std::vector<seastar::scheduling_supergroup> sgroups;\n    std::vector<seastar::scheduling_group> groups;\n\n    static future<layout> create();\n    future<> destroy();\n};\n\nfuture<layout> layout::create() {\n    layout ret;\n\n    fmt::print(\"Creating supergroups\\n\");\n    ret.sgroups.resize(2);\n    ret.sgroups[0] = co_await seastar::create_scheduling_supergroup(100);\n    ret.sgroups[1] = co_await seastar::create_scheduling_supergroup(100);\n\n    fmt::print(\"Creating groups\\n\");\n    ret.groups.resize(9);\n    ret.groups[0] = co_await seastar::create_scheduling_group(\"g1\", 100);\n    ret.groups[1] = co_await seastar::create_scheduling_group(\"g2\", 100);\n    ret.groups[2] = co_await seastar::create_scheduling_group(\"g3\", 100);\n    ret.groups[3] = co_await seastar::create_scheduling_group(\"g11\", \"g11\", 100, ret.sgroups[0]);\n    ret.groups[4] = co_await seastar::create_scheduling_group(\"g21\", \"g21\", 100, ret.sgroups[0]);\n    ret.groups[5] = co_await seastar::create_scheduling_group(\"g31\", \"g31\", 100, ret.sgroups[0]);\n    ret.groups[6] = co_await seastar::create_scheduling_group(\"g12\", \"g12\", 100, ret.sgroups[1]);\n    ret.groups[7] = co_await seastar::create_scheduling_group(\"g22\", \"g22\", 100, ret.sgroups[1]);\n    ret.groups[8] = co_await seastar::create_scheduling_group(\"g32\", \"g32\", 100, ret.sgroups[1]);\n\n    co_return ret;\n}\n\nfuture<> layout::destroy() {\n    for (auto sg : groups) {\n        co_await seastar::destroy_scheduling_group(sg);\n    }\n    for (auto sg : sgroups) {\n        co_await seastar::destroy_scheduling_supergroup(sg);\n    }\n}\n\nSEASTAR_TEST_CASE(test_basic_ops) {\n    auto l = co_await layout::create();\n    BOOST_CHECK_EXCEPTION(co_await seastar::destroy_scheduling_supergroup(l.sgroups[0]), std::runtime_error, [] (const std::runtime_error& e) {\n        return e.what() == sstring(\"Supergroup is still populated, destroy all subgroups first\");\n    });\n    auto ssg = internal::scheduling_supergroup_for(l.groups[0]);\n    BOOST_CHECK(ssg == scheduling_supergroup());\n    ssg = internal::scheduling_supergroup_for(l.groups[6]);\n    BOOST_CHECK(!ssg.is_root());\n    BOOST_CHECK(ssg.index() == 1);\n    BOOST_CHECK(ssg == l.sgroups[1]);\n    co_await l.destroy();\n}\n\n#ifndef SEASTAR_SHUFFLE_TASK_QUEUE\nstatic future<> run_busyloops(std::vector<seastar::scheduling_group> groups, std::vector<float> expected) {\n    std::vector<uint64_t> counts;\n    std::vector<future<>> f;\n    bool keep_going = true;\n\n    auto run_and_count = [&keep_going] (uint64_t& c) -> future<> {\n        do {\n            auto stop = std::chrono::steady_clock::now() + std::chrono::microseconds(10);\n            while (std::chrono::steady_clock::now() < stop) ;\n            c++;\n            co_await yield();\n        } while (keep_going);\n    };\n    counts.reserve(groups.size());\n    f.reserve(groups.size());\n    for (auto& g : groups) {\n        counts.push_back(0);\n        f.push_back(with_scheduling_group(g, [&c = counts.back(), &run_and_count] { return run_and_count(c); }));\n    }\n\n    auto warmup = std::chrono::milliseconds(20);\n    fmt::print(\"Warmup for {}\\n\", warmup);\n    co_await seastar::sleep(warmup);\n    for (auto& c : counts) {\n        c = 0;\n    }\n\n    auto sample_period = std::chrono::milliseconds(100);\n    auto stop_at = std::chrono::steady_clock::now() + std::chrono::minutes(2);\n    bool ok = true;\n\n  do {\n    fmt::print(\"Do counting for {}\\n\", sample_period);\n    co_await seastar::sleep(sample_period);\n\n    uint64_t average_adjusted_count = 0;\n    for (unsigned i = 0; i < groups.size(); i++) {\n        average_adjusted_count += counts[i] / expected[i];\n    }\n    average_adjusted_count /= groups.size();\n\n    ok = true;\n    fmt::print(\"--------8<--------\\n\");\n    for (unsigned i = 0; i < groups.size(); i++) {\n        auto dev = float(std::abs(counts[i] / expected[i] - average_adjusted_count)) / average_adjusted_count;\n        fmt::print(\"{}: count={} expected={:.2f} adjusted={} deviation={:.2f}\\n\", i, counts[i], expected[i], int(float(counts[i]) / expected[i]), dev);\n        ok &= (dev < 0.07);\n    }\n    sample_period = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::duration_cast<std::chrono::duration<double>>(sample_period) * 1.618);\n  } while (!ok && (std::chrono::steady_clock::now() <= stop_at));\n\n    keep_going = false;\n    co_await when_all(f.begin(), f.end());\n    BOOST_REQUIRE(ok);\n}\n\nstatic future<> do_test_fairness(layout& l) {\n    std::default_random_engine random_engine(testing::local_random_engine());\n    std::uniform_int_distribution<unsigned> ng_dist(2, 5);\n    std::uniform_int_distribution<unsigned> shares_dist(1, 5);\n\n    for (int test = 0; test < 8; test++) {\n        fmt::print(\"=== TEST-{} ===\\n\", test);\n        unsigned ngroups = ng_dist(random_engine);\n\n        std::vector<unsigned> indices;\n        indices.reserve(l.groups.size());\n        for (unsigned i = 0; i < l.groups.size(); i++) {\n            indices.push_back(i);\n        }\n        std::shuffle(indices.begin(), indices.end(), random_engine);\n        indices.resize(ngroups);\n\n        fmt::print(\"Running in {} groups: {}\\n\", ngroups, indices);\n\n        std::vector<unsigned> run_shares;\n        run_shares.reserve(ngroups);\n        unsigned sg1_shares = 0, sg1_total_shares = 0;\n        unsigned sg2_shares = 0, sg2_total_shares = 0;\n\n        std::vector<seastar::scheduling_group> run_groups;\n        run_groups.reserve(ngroups);\n\n        unsigned root_total_shares = 0;\n        for (unsigned i = 0; i < ngroups; i++) {\n            unsigned idx = indices[i];\n            auto sg = l.groups[idx];\n            auto shares = shares_dist(random_engine);\n            sg.set_shares(shares * 100);\n            run_groups.push_back(sg);\n            run_shares.push_back(shares);\n\n            if (idx < 3) {\n                root_total_shares += shares;\n            } else if (idx < 6) {\n                if (sg1_shares == 0) {\n                    auto ssg = l.sgroups[0];\n                    auto shares = shares_dist(random_engine);\n                    ssg.set_shares(shares * 100);\n                    sg1_shares = shares;\n                    root_total_shares += shares;\n                }\n                sg1_total_shares += shares;\n            } else {\n                if (sg2_shares == 0) {\n                    auto ssg = l.sgroups[1];\n                    auto shares = shares_dist(random_engine);\n                    ssg.set_shares(shares * 100);\n                    sg2_shares = shares;\n                    root_total_shares += shares;\n                }\n                sg2_total_shares += shares;\n            }\n        }\n\n        fmt::print(\"Shares: {}\\n\", run_shares);\n        if (sg1_shares != 0) {\n            fmt::print(\"  supergroup 1 shares = {}\\n\", sg1_shares);\n        }\n        if (sg2_shares != 0) {\n            fmt::print(\"  supergroup 2 shares = {}\\n\", sg2_shares);\n        }\n\n        std::vector<float> expectations;\n        expectations.resize(ngroups);\n        for (unsigned i = 0; i < ngroups; i++) {\n            unsigned idx = indices[i];\n            if (idx < 3) {\n                expectations[i] = float(run_shares[i]) / root_total_shares;\n            } else if (idx < 6) {\n                float sg_expectation = float(sg1_shares) / root_total_shares;\n                expectations[i] = float(run_shares[i]) / sg1_total_shares * sg_expectation;\n            } else {\n                float sg_expectation = float(sg2_shares) / root_total_shares;\n                expectations[i] = float(run_shares[i]) / sg2_total_shares * sg_expectation;\n            }\n        }\n\n        co_await run_busyloops(run_groups, expectations);\n    }\n}\n#else\nstatic future<> do_test_fairness(layout& l) {\n    fmt::print(\"Skipping CPU fairness test in debug mode\\n\");\n    return make_ready_future<>();\n}\n#endif\n\nSEASTAR_TEST_CASE(test_fairness) {\n    auto l = co_await layout::create();\n    co_await do_test_fairness(l);\n    co_await l.destroy();\n}\n\nSEASTAR_TEST_CASE(test_wakeups) {\n    auto l = co_await layout::create();\n\n    std::default_random_engine random_engine(testing::local_random_engine());\n    std::uniform_int_distribution<unsigned> ng_dist(0, l.groups.size() - 1);\n    std::uniform_int_distribution<unsigned> w_dist(2, 8);\n\n    for (unsigned test = 0; test < 32; test++) {\n        struct semaphore_and_number {\n            semaphore sem;\n            unsigned nr;\n            semaphore_and_number(unsigned n) : sem(0), nr(n) {}\n        };\n\n        std::vector<semaphore_and_number> waiters;\n        waiters.reserve(w_dist(random_engine));\n\n        unsigned total_waiters = 0;\n        semaphore ready(0);\n        semaphore done(0);\n\n        for (unsigned i = 0; i < waiters.capacity(); i++) {\n            unsigned nw = w_dist(random_engine);\n            fmt::print(\"Run with {} waiters\\n\", nw);\n            total_waiters += nw;\n            auto& ws = waiters.emplace_back(nw);\n\n            for (unsigned w = 0; w < nw; w++) {\n                auto g = ng_dist(random_engine);\n                (void)with_scheduling_group(l.groups[g], [&s = ws.sem, &ready, &done, i, w, g] () mutable {\n                    ready.signal();\n                    return s.wait().then([&done, i, w, g] {\n                        fmt::print(\"{}:{} wakeup in {}\\n\", i, w, g);\n                        done.signal();\n                    });\n                });\n            }\n        }\n\n        co_await ready.wait(total_waiters);\n\n        for (unsigned i = 0; i < waiters.size(); i++) {\n            auto g = ng_dist(random_engine);\n            (void)with_scheduling_group(l.groups[g], [&s = waiters[i], &done] () mutable {\n                s.sem.signal(s.nr);\n                done.signal();\n            });\n        }\n\n        co_await done.wait(total_waiters + waiters.size());\n    }\n\n    co_await l.destroy();\n}\n"
  },
  {
    "path": "tests/unit/scheduling_group_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB Ltd.\n */\n\n#include <algorithm>\n#include <vector>\n#include <chrono>\n\n#include <seastar/core/thread.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/testing/test_runner.hh>\n#include <seastar/core/execution_stage.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/scheduling_specific.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/with_scheduling_group.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/defer.hh>\n\nusing namespace std::chrono_literals;\n\nusing namespace seastar;\n\n/**\n *  Test setting primitive and object as a value after all groups are created\n */\nSEASTAR_THREAD_TEST_CASE(sg_specific_values_define_after_sg_create) {\n    using ivec  = std::vector<int>;\n    const int num_scheduling_groups = 4;\n    std::vector<scheduling_group> sgs;\n    for (int i = 0; i < num_scheduling_groups; i++) {\n        sgs.push_back(create_scheduling_group(format(\"sg{}\", i).c_str(), 100).get());\n    }\n\n    const auto destroy_scheduling_groups = defer([&sgs] () noexcept {\n       for (scheduling_group sg : sgs) {\n           destroy_scheduling_group(sg).get();\n       }\n    });\n    scheduling_group_key_config key1_conf = make_scheduling_group_key_config<int>();\n    scheduling_group_key key1 = scheduling_group_key_create(key1_conf).get();\n\n    scheduling_group_key_config key2_conf = make_scheduling_group_key_config<ivec>();\n    scheduling_group_key key2 = scheduling_group_key_create(key2_conf).get();\n\n    smp::invoke_on_all([key1, key2, &sgs] () {\n        int factor = this_shard_id() + 1;\n        for (int i=0; i < num_scheduling_groups; i++) {\n            sgs[i].get_specific<int>(key1) = (i + 1) * factor;\n            sgs[i].get_specific<ivec>(key2).push_back((i + 1) * factor);\n        }\n\n        for (int i=0; i < num_scheduling_groups; i++) {\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<int>(key1) = (i + 1) * factor, (i + 1) * factor);\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<ivec>(key2)[0], (i + 1) * factor);\n        }\n\n    }).get();\n\n    smp::invoke_on_all([key1, key2] () {\n        return reduce_scheduling_group_specific<int>(std::plus<int>(), int(0), key1).then([] (int sum) {\n            int factor = this_shard_id() + 1;\n            int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n            BOOST_REQUIRE_EQUAL(expected_sum, sum);\n        }). then([key2] {\n            auto ivec_to_int = [] (ivec& v) {\n                return v.size() ? v[0] : 0;\n            };\n\n            return map_reduce_scheduling_group_specific<ivec>(ivec_to_int, std::plus<int>(), int(0), key2).then([] (int sum) {\n                int factor = this_shard_id() + 1;\n                int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n                BOOST_REQUIRE_EQUAL(expected_sum, sum);\n            });\n\n        });\n    }).get();\n\n\n}\n\n/**\n *  Test setting primitive and object as a value before all groups are created\n */\nSEASTAR_THREAD_TEST_CASE(sg_specific_values_define_before_sg_create) {\n    using ivec  = std::vector<int>;\n    const int num_scheduling_groups = 4;\n    std::vector<scheduling_group> sgs;\n    const auto destroy_scheduling_groups = defer([&sgs] () noexcept {\n       for (scheduling_group sg : sgs) {\n           destroy_scheduling_group(sg).get();\n       }\n    });\n    scheduling_group_key_config key1_conf = make_scheduling_group_key_config<int>();\n    scheduling_group_key key1 = scheduling_group_key_create(key1_conf).get();\n\n    scheduling_group_key_config key2_conf = make_scheduling_group_key_config<ivec>();\n    scheduling_group_key key2 = scheduling_group_key_create(key2_conf).get();\n\n    for (int i = 0; i < num_scheduling_groups; i++) {\n        sgs.push_back(create_scheduling_group(format(\"sg{}\", i).c_str(), 100).get());\n    }\n\n    smp::invoke_on_all([key1, key2, &sgs] () {\n        int factor = this_shard_id() + 1;\n        for (int i=0; i < num_scheduling_groups; i++) {\n            sgs[i].get_specific<int>(key1) = (i + 1) * factor;\n            sgs[i].get_specific<ivec>(key2).push_back((i + 1) * factor);\n        }\n\n        for (int i=0; i < num_scheduling_groups; i++) {\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<int>(key1) = (i + 1) * factor, (i + 1) * factor);\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<ivec>(key2)[0], (i + 1) * factor);\n        }\n\n    }).get();\n\n    smp::invoke_on_all([key1, key2] () {\n        return reduce_scheduling_group_specific<int>(std::plus<int>(), int(0), key1).then([] (int sum) {\n            int factor = this_shard_id() + 1;\n            int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n            BOOST_REQUIRE_EQUAL(expected_sum, sum);\n        }). then([key2] {\n            auto ivec_to_int = [] (ivec& v) {\n                return v.size() ? v[0] : 0;\n            };\n\n            return map_reduce_scheduling_group_specific<ivec>(ivec_to_int, std::plus<int>(), int(0), key2).then([] (int sum) {\n                int factor = this_shard_id() + 1;\n                int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n                BOOST_REQUIRE_EQUAL(expected_sum, sum);\n            });\n\n        });\n    }).get();\n\n}\n\n/**\n *  Test setting primitive and an object as a value before some groups are created\n *  and after some of the groups are created.\n */\nSEASTAR_THREAD_TEST_CASE(sg_specific_values_define_before_and_after_sg_create) {\n    using ivec  = std::vector<int>;\n    const int num_scheduling_groups = 4;\n    std::vector<scheduling_group> sgs;\n    const auto destroy_scheduling_groups = defer([&sgs] () noexcept {\n       for (scheduling_group sg : sgs) {\n           destroy_scheduling_group(sg).get();\n       }\n    });\n\n    for (int i = 0; i < num_scheduling_groups/2; i++) {\n        sgs.push_back(create_scheduling_group(format(\"sg{}\", i).c_str(), 100).get());\n    }\n    scheduling_group_key_config key1_conf = make_scheduling_group_key_config<int>();\n    scheduling_group_key key1 = scheduling_group_key_create(key1_conf).get();\n\n    scheduling_group_key_config key2_conf = make_scheduling_group_key_config<ivec>();\n    scheduling_group_key key2 = scheduling_group_key_create(key2_conf).get();\n\n    for (int i = num_scheduling_groups/2; i < num_scheduling_groups; i++) {\n        sgs.push_back(create_scheduling_group(format(\"sg{}\", i).c_str(), 100).get());\n    }\n\n    smp::invoke_on_all([key1, key2, &sgs] () {\n        int factor = this_shard_id() + 1;\n        for (int i=0; i < num_scheduling_groups; i++) {\n            sgs[i].get_specific<int>(key1) = (i + 1) * factor;\n            sgs[i].get_specific<ivec>(key2).push_back((i + 1) * factor);\n        }\n\n        for (int i=0; i < num_scheduling_groups; i++) {\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<int>(key1) = (i + 1) * factor, (i + 1) * factor);\n            BOOST_REQUIRE_EQUAL(sgs[i].get_specific<ivec>(key2)[0], (i + 1) * factor);\n        }\n\n    }).get();\n\n    smp::invoke_on_all([key1, key2] () {\n        return reduce_scheduling_group_specific<int>(std::plus<int>(), int(0), key1).then([] (int sum) {\n            int factor = this_shard_id() + 1;\n            int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n            BOOST_REQUIRE_EQUAL(expected_sum, sum);\n        }). then([key2] {\n            auto ivec_to_int = [] (ivec& v) {\n                return v.size() ? v[0] : 0;\n            };\n\n            return map_reduce_scheduling_group_specific<ivec>(ivec_to_int, std::plus<int>(), int(0), key2).then([] (int sum) {\n                int factor = this_shard_id() + 1;\n                int expected_sum = ((1 + num_scheduling_groups)*num_scheduling_groups) * factor /2;\n                BOOST_REQUIRE_EQUAL(expected_sum, sum);\n            });\n\n        });\n    }).get();\n}\n\n/*\n * Test that current scheduling group is inherited by seastar::async()\n */\nSEASTAR_THREAD_TEST_CASE(sg_scheduling_group_inheritance_in_seastar_async_test) {\n    scheduling_group sg = create_scheduling_group(\"sg0\", 100).get();\n    auto cleanup = defer([&] () noexcept { destroy_scheduling_group(sg).get(); });\n    thread_attributes attr = {};\n    attr.sched_group = sg;\n    seastar::async(attr, [attr] {\n        BOOST_REQUIRE_EQUAL(internal::scheduling_group_index(current_scheduling_group()),\n                                internal::scheduling_group_index(*(attr.sched_group)));\n\n        seastar::async([attr] {\n            BOOST_REQUIRE_EQUAL(internal::scheduling_group_index(current_scheduling_group()),\n                                internal::scheduling_group_index(*(attr.sched_group)));\n\n            smp::invoke_on_all([sched_group_idx = internal::scheduling_group_index(*(attr.sched_group))] () {\n                BOOST_REQUIRE_EQUAL(internal::scheduling_group_index(current_scheduling_group()), sched_group_idx);\n            }).get();\n        }).get();\n    }).get();\n}\n\n\nSEASTAR_THREAD_TEST_CASE(yield_preserves_sg) {\n    scheduling_group sg = create_scheduling_group(\"sg\", 100).get();\n    auto cleanup = defer([&] () noexcept { destroy_scheduling_group(sg).get(); });\n    with_scheduling_group(sg, [&] {\n        return yield().then([&] {\n            BOOST_REQUIRE_EQUAL(\n                    internal::scheduling_group_index(current_scheduling_group()),\n                    internal::scheduling_group_index(sg));\n        });\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_count) {\n    class scheduling_group_destroyer {\n        scheduling_group _sg;\n    public:\n        scheduling_group_destroyer(scheduling_group sg) : _sg(sg) {}\n        scheduling_group_destroyer(const scheduling_group_destroyer&) = default;\n        ~scheduling_group_destroyer() {\n            destroy_scheduling_group(_sg).get();\n        }\n    };\n\n    std::vector<scheduling_group_destroyer> scheduling_groups_deferred_cleanup;\n    // The line below is necessary in order to skip support of copy and move construction of scheduling_group_destroyer.\n    scheduling_groups_deferred_cleanup.reserve(max_scheduling_groups());\n    // try to create 3 groups too many.\n    for (auto i = internal::scheduling_group_count(); i < max_scheduling_groups() + 3; i++) {\n        try {\n            BOOST_REQUIRE_LE(internal::scheduling_group_count(), max_scheduling_groups());\n            scheduling_groups_deferred_cleanup.emplace_back(create_scheduling_group(format(\"sg_{}\", i), 10).get());\n        } catch (std::runtime_error& e) {\n            // make sure it is the right exception.\n            BOOST_REQUIRE_EQUAL(e.what(), fmt::format(\"Scheduling group limit exceeded while creating sg_{}\", i));\n            // make sure that the scheduling group count makes sense\n            BOOST_REQUIRE_EQUAL(internal::scheduling_group_count(), max_scheduling_groups());\n            // make sure that we expect this exception at this point\n            BOOST_REQUIRE_GE(i, max_scheduling_groups());\n        }\n    }\n    BOOST_REQUIRE_EQUAL(internal::scheduling_group_count(), max_scheduling_groups());\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_rename_callback) {\n    // Tests that all sg-local values receive the rename() callback when scheduling groups\n    // are renamed.\n\n    struct value {\n        uint64_t _id = 0; // Used to check that the value only receives a rename(), not a reconstruction.\n        std::string _sg_name;\n\n        value(const value&) = delete;\n        value& operator=(const value&) = delete;\n\n        value() {\n            rename();\n        }\n        void rename() {\n            _sg_name = current_scheduling_group().name();\n        }\n    };\n\n    scheduling_group_key_config key_conf = make_scheduling_group_key_config<value>();\n\n    std::vector<scheduling_group_key> keys;\n    for (size_t i = 0; i < 3; ++i) {\n        keys.push_back(scheduling_group_key_create(key_conf).get());\n    }\n\n    std::vector<scheduling_group> sgs;\n    const auto destroy_sgs = defer([&sgs] () noexcept {\n        for (auto sg : sgs) {\n           destroy_scheduling_group(sg).get();\n        }\n    });\n    for (size_t s = 0; s < 3; ++s) {\n        sgs.push_back(create_scheduling_group(fmt::format(\"sg-old-{}\", s), 1000).get());\n    }\n\n    smp::invoke_on_all([&sgs, &keys] () {\n        for (size_t s = 0; s < std::size(sgs); ++s) {\n            for (size_t k = 0; k < std::size(keys); ++k) {\n                BOOST_REQUIRE_EQUAL(sgs[s].get_specific<value>(keys[k])._sg_name, fmt::format(\"sg-old-{}\", s));\n                sgs[s].get_specific<value>(keys[k])._id = 1;\n            }\n        }\n    }).get();\n\n    for (size_t s = 0; s < std::size(sgs); ++s) {\n        rename_scheduling_group(sgs[s], fmt::format(\"sg-new-{}\", s)).get();\n    }\n\n    smp::invoke_on_all([&sgs, &keys] () {\n        for (size_t s = 0; s < std::size(sgs); ++s) {\n            for (size_t k = 0; k < std::size(keys); ++k) {\n                BOOST_REQUIRE_EQUAL(sgs[s].get_specific<value>(keys[k])._sg_name, fmt::format(\"sg-new-{}\", s));\n                // Checks that the value only saw a rename(), not a reconstruction.\n                BOOST_REQUIRE_EQUAL(sgs[s].get_specific<value>(keys[k])._id, 1);\n            }\n        }\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_create_and_key_create_in_parallel) {\n    std::vector<scheduling_group> sgs;\n    scheduling_group_key_config key1_conf = make_scheduling_group_key_config<int>();\n\n    when_all_succeed(\n        create_scheduling_group(format(\"sg1\").c_str(), 100),\n        scheduling_group_key_create(key1_conf),\n        create_scheduling_group(format(\"sg2\").c_str(), 100),\n        scheduling_group_key_create(key1_conf),\n        create_scheduling_group(format(\"sg3\").c_str(), 100),\n        scheduling_group_key_create(key1_conf),\n        create_scheduling_group(format(\"sg4\").c_str(), 100),\n        scheduling_group_key_create(key1_conf)\n    ).then_unpack([&sgs] (\n            scheduling_group sg1, scheduling_group_key k1, scheduling_group sg2, scheduling_group_key k2,\n            scheduling_group sg3, scheduling_group_key k3, scheduling_group sg4, scheduling_group_key k4) {\n        sgs.push_back(sg1);\n        sgs.push_back(sg2);\n        sgs.push_back(sg3);\n        sgs.push_back(sg4);\n    }).get();\n\n    for (scheduling_group sg : sgs) {\n        destroy_scheduling_group(sg).get();\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_rename_recreate_with_the_same_name) {\n    auto do_nothing = [] {};\n    inheriting_concrete_execution_stage<void> stage{\"my_stage\", do_nothing};\n\n    std::vector<scheduling_group> sgs;\n    sgs.push_back(create_scheduling_group(\"foo\", 100).get());\n    with_scheduling_group(sgs[0], [&] { return stage(); }).get();\n\n    rename_scheduling_group(sgs[0], \"bar\").get();\n\n    BOOST_CHECK_NO_THROW({\n        sgs.push_back(create_scheduling_group(\"foo\", 100).get());\n        with_scheduling_group(sgs[1], [&] { return stage(); }).get();\n    });\n\n    for (scheduling_group sg : sgs) {\n        destroy_scheduling_group(sg).get();\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_key_constructor_exception_when_creating_new_key) {\n    scheduling_group_key_config key_conf = make_scheduling_group_key_config<int>();\n    scheduling_group_key_create(key_conf).get();\n\n    struct thrower {\n        thrower() {\n            throw std::runtime_error(\"constructor failed\");\n        }\n        ~thrower() {\n            // Shouldn't get here because the constructor shouldn't succeed\n            SEASTAR_ASSERT(false);\n        }\n    };\n    scheduling_group_key_config thrower_conf = make_scheduling_group_key_config<thrower>();\n    BOOST_REQUIRE_THROW(scheduling_group_key_create(thrower_conf).get(), std::runtime_error);\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_create_with_destroy_tasks) {\n    struct nada{};\n\n    // at_destroy() functionality is deprecated, but test until removed.\n    internal::at_destroy([] {}); // nothing really\n\n    scheduling_group_key_config sg_conf = make_scheduling_group_key_config<nada>();\n    scheduling_group_key_create(sg_conf).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(sg_create_check_unique_constructor_invocation) {\n    static thread_local std::set<unsigned> groups;\n\n    // check we don't run constructor in same sched group more than once.\n    struct check {\n        check() {\n            auto sg = current_scheduling_group();\n            auto id = internal::scheduling_group_index(sg);\n            BOOST_REQUIRE(groups.count(id) == 0);\n            groups.emplace(id);\n        }\n    };\n\n    smp::invoke_on_all([] () {\n        groups.clear();\n    }).get();\n\n    scheduling_group_key_config key1_conf = make_scheduling_group_key_config<check>();\n    scheduling_group_key_create(key1_conf).get();\n\n    // clean out data for ASAN.\n    smp::invoke_on_all([] () {\n        groups = {};\n    }).get();\n}\n"
  },
  {
    "path": "tests/unit/semaphore_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/map_reduce.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/shared_mutex.hh>\n#include <ranges>\n#include <stdexcept>\n\n#include \"expected_exception.hh\"\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_semaphore_consume) {\n    semaphore sem(0);\n    sem.consume(1);\n    BOOST_REQUIRE_EQUAL(sem.current(), 0u);\n    BOOST_REQUIRE_EQUAL(sem.waiters(), 0u);\n\n    BOOST_REQUIRE_EQUAL(sem.try_wait(0), false);\n    auto fut = sem.wait(1);\n    BOOST_REQUIRE_EQUAL(fut.available(), false);\n    BOOST_REQUIRE_EQUAL(sem.waiters(), 1u);\n    sem.signal(2);\n    BOOST_REQUIRE_EQUAL(sem.waiters(), 0u);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_semaphore_1) {\n    return do_with(std::make_pair(semaphore(0), 0), [] (std::pair<semaphore, int>& x) {\n        (void)x.first.wait().then([&x] {\n            x.second++;\n        });\n        x.first.signal();\n        return sleep(10ms).then([&x] {\n            BOOST_REQUIRE_EQUAL(x.second, 1);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_2) {\n    auto sem = std::make_optional<semaphore>(0);\n    int x = 0;\n    auto fut = sem->wait().then([&x] {\n        x++;\n    });\n    sleep(10ms).get();\n    BOOST_REQUIRE_EQUAL(x, 0);\n    sem = std::nullopt;\n    BOOST_CHECK_THROW(fut.get(), broken_promise);\n}\n\nSEASTAR_TEST_CASE(test_semaphore_timeout_1) {\n    return do_with(std::make_pair(semaphore(0), 0), [] (std::pair<semaphore, int>& x) {\n        (void)x.first.wait(100ms).then([&x] {\n            x.second++;\n        });\n        (void)sleep(3ms).then([&x] {\n            x.first.signal();\n        });\n        return sleep(200ms).then([&x] {\n            BOOST_REQUIRE_EQUAL(x.second, 1);\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_timeout_2) {\n    auto sem = semaphore(0);\n    int x = 0;\n    auto fut1 = sem.wait(3ms).then([&x] {\n        x++;\n    });\n    bool signaled = false;\n    auto fut2 = sleep(100ms).then([&sem, &signaled] {\n        signaled = true;\n        sem.signal();\n    });\n    sleep(200ms).get();\n    fut2.get();\n    BOOST_REQUIRE_EQUAL(signaled, true);\n    BOOST_CHECK_THROW(fut1.get(), semaphore_timed_out);\n    BOOST_REQUIRE_EQUAL(x, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_mix_1) {\n    auto sem = semaphore(0);\n    int x = 0;\n    auto fut1 = sem.wait(30ms).then([&x] {\n        x++;\n    });\n    auto fut2 = sem.wait().then([&x] {\n        x += 10;\n    });\n    auto fut3 = sleep(100ms).then([&sem] {\n        sem.signal();\n    });\n    sleep(200ms).get();\n    fut3.get();\n    fut2.get();\n    BOOST_CHECK_THROW(fut1.get(), semaphore_timed_out);\n    BOOST_REQUIRE_EQUAL(x, 10);\n}\n\nSEASTAR_TEST_CASE(test_broken_semaphore) {\n    auto sem = make_lw_shared<semaphore>(0);\n    struct oops {};\n    auto check_result = [sem] (future<> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"expecting exception\");\n        } catch (oops& x) {\n            // ok\n            return make_ready_future<>();\n        } catch (...) {\n            BOOST_FAIL(\"wrong exception seen\");\n        }\n        BOOST_FAIL(\"unreachable\");\n        return make_ready_future<>();\n    };\n    auto ret = sem->wait().then_wrapped(check_result);\n    sem->broken(oops());\n    return sem->wait().then_wrapped(check_result).then([ret = std::move(ret)] () mutable {\n        return std::move(ret);\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_default_broken_semaphore) {\n    struct test_semaphore_exception_factory {\n        static semaphore_timed_out timeout() noexcept { return semaphore_timed_out(); }\n    };\n    auto sem = basic_semaphore<test_semaphore_exception_factory>(0);\n    auto fut = sem.wait();\n    BOOST_REQUIRE(!fut.available());\n    sem.broken();\n    BOOST_REQUIRE_THROW(fut.get(), broken_semaphore);\n    BOOST_REQUIRE_THROW(sem.wait().get(), broken_semaphore);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_non_default_broken_semaphore) {\n    struct test_semaphore_exception_factory {\n        static semaphore_timed_out timeout() noexcept { return semaphore_timed_out(); }\n    };\n    auto sem = basic_semaphore<test_semaphore_exception_factory>(0);\n    auto fut = sem.wait();\n    BOOST_REQUIRE(!fut.available());\n    sem.broken(expected_exception());\n    BOOST_REQUIRE_THROW(fut.get(), expected_exception);\n    BOOST_REQUIRE_THROW(sem.wait().get(), expected_exception);\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_exclusive) {\n    return do_with(shared_mutex(), unsigned(0), [] (shared_mutex& sm, unsigned& counter) {\n        return parallel_for_each(std::views::iota(0, 10), [&sm, &counter] (int idx) {\n            return with_lock(sm, [&counter] {\n                BOOST_REQUIRE_EQUAL(counter, 0u);\n                ++counter;\n                return sleep(10ms).then([&counter] {\n                    --counter;\n                    BOOST_REQUIRE_EQUAL(counter, 0u);\n                });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_shared) {\n    return do_with(shared_mutex(), unsigned(0), [] (shared_mutex& sm, unsigned& counter) {\n        auto running_in_parallel = [&sm, &counter] (int instance) {\n            return with_shared(sm, [&counter] {\n                ++counter;\n                return sleep(10ms).then([&counter] {\n                    bool was_parallel = counter != 0;\n                    --counter;\n                    return was_parallel;\n                });\n            });\n        };\n        return map_reduce(std::views::iota(0, 100), running_in_parallel, false, std::bit_or<bool>()).then([&counter] (bool result) {\n            BOOST_REQUIRE_EQUAL(result, true);\n            BOOST_REQUIRE_EQUAL(counter, 0u);\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_shared_mutex_mixed) {\n    return do_with(shared_mutex(), unsigned(0), [] (shared_mutex& sm, unsigned& counter) {\n        auto running_in_parallel = [&sm, &counter] (int instance) {\n            return with_shared(sm, [&counter] {\n                ++counter;\n                return sleep(10ms).then([&counter] {\n                    bool was_parallel = counter != 0;\n                    --counter;\n                    return was_parallel;\n                });\n            });\n        };\n        auto running_alone = [&sm, &counter] (int instance) {\n            return with_lock(sm, [&counter] {\n                BOOST_REQUIRE_EQUAL(counter, 0u);\n                ++counter;\n                return sleep(10ms).then([&counter] {\n                    --counter;\n                    BOOST_REQUIRE_EQUAL(counter, 0u);\n                    return true;\n                });\n            });\n        };\n        auto run = [running_in_parallel, running_alone] (int instance) {\n            if (instance % 9 == 0) {\n                return running_alone(instance);\n            } else {\n                return running_in_parallel(instance);\n            }\n        };\n        return map_reduce(std::views::iota(0, 100), run, false, std::bit_or<bool>()).then([&counter] (bool result) {\n            BOOST_REQUIRE_EQUAL(result, true);\n            BOOST_REQUIRE_EQUAL(counter, 0u);\n        });\n    });\n}\n\n\nSEASTAR_TEST_CASE(test_with_semaphore) {\n    return do_with(semaphore(1), 0, [] (semaphore& sem, int& counter) {\n        return with_semaphore(sem, 1, [&counter] {\n            ++counter;\n        }).then([&counter, &sem] () {\n            return with_semaphore(sem, 1, [&counter] {\n                ++counter;\n                throw 123;\n            }).then_wrapped([&counter] (auto&& fut) {\n                BOOST_REQUIRE_EQUAL(counter, 2);\n                BOOST_REQUIRE(fut.failed());\n                fut.ignore_ready_future();\n            });\n        });\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_valid_splitting) {\n    auto sm = semaphore(2);\n    auto units = get_units(sm, 2, 1min).get();\n    {\n        BOOST_REQUIRE_EQUAL(units.count(), 2);\n        BOOST_REQUIRE_EQUAL(sm.available_units(), 0);\n        auto split = units.split(1);\n        BOOST_REQUIRE_EQUAL(sm.available_units(), 0);\n    }\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 1);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_invalid_splitting) {\n    auto sm = semaphore(2);\n    auto units = get_units(sm, 2, 1min).get();\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 0);\n    BOOST_REQUIRE_THROW(units.split(10), std::invalid_argument);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_return_when_destroyed) {\n    auto sm = semaphore(3);\n  {\n    auto units = get_units(sm, 3, 1min).get();\n    BOOST_REQUIRE_EQUAL(units.count(), 3);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 0);\n    BOOST_REQUIRE_EQUAL(units.return_units(1), 2);\n    BOOST_REQUIRE_EQUAL(units.count(), 2);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 1);\n  }\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 3);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_return_all) {\n    auto sm = semaphore(3);\n    auto units = get_units(sm, 2, 1min).get();\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 1);\n    BOOST_REQUIRE_THROW(units.return_units(10), std::invalid_argument);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 1);\n    units.return_all();\n    BOOST_REQUIRE_EQUAL(units.count(), 0);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), 3);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_try_get_units) {\n    constexpr size_t initial_units = 1;\n    auto sm = semaphore(initial_units);\n\n    auto opt_units = try_get_units(sm, 1);\n    BOOST_REQUIRE(opt_units);\n\n    auto opt_units2 = try_get_units(sm, 1);\n    BOOST_REQUIRE(!opt_units2);\n\n    opt_units.reset();\n    BOOST_REQUIRE_EQUAL(sm.available_units(), initial_units);\n\n    opt_units = try_get_units(sm, 1);\n    BOOST_REQUIRE(opt_units);\n\n    opt_units->return_all();\n    BOOST_REQUIRE_EQUAL(opt_units->count(), 0);\n    BOOST_REQUIRE_EQUAL(sm.available_units(), initial_units);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_abort) {\n    auto sm = semaphore(3);\n    auto units = get_units(sm, 3, 1min).get();\n    BOOST_REQUIRE_EQUAL(units.count(), 3);\n\n    abort_source as;\n\n    auto f = get_units(sm, 1, as);\n    BOOST_REQUIRE(!f.available());\n\n    (void)sleep(1ms).then([&as] {\n        as.request_abort();\n    });\n\n    BOOST_REQUIRE_THROW(f.get(), semaphore_aborted);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_units_bool_operator) {\n    auto sem = semaphore(2);\n    semaphore_units u0;\n    BOOST_REQUIRE(!bool(u0));\n\n    u0 = get_units(sem, 2).get();\n    BOOST_REQUIRE(bool(u0));\n\n    u0.return_units(1);\n    BOOST_REQUIRE(bool(u0));\n\n    auto n = u0.release();\n    BOOST_REQUIRE(!bool(u0));\n    sem.signal(n);\n\n    u0 = get_units(sem, 2).get();\n    BOOST_REQUIRE(bool(u0));\n    auto u1 = std::move(u0);\n    BOOST_REQUIRE(bool(u1));\n    BOOST_REQUIRE(!bool(u0));\n\n    u1.return_all();\n    BOOST_REQUIRE(!bool(u1));\n\n    u0 = get_units(sem, 2).get();\n    BOOST_REQUIRE(bool(u0));\n    u1 = u0.split(1);\n    BOOST_REQUIRE(bool(u1));\n    BOOST_REQUIRE(bool(u0));\n\n    u0.adopt(std::move(u1));\n    BOOST_REQUIRE(!bool(u1));\n    BOOST_REQUIRE(bool(u0));\n}\n\nSEASTAR_THREAD_TEST_CASE(test_named_semaphore_error) {\n    auto sem = make_lw_shared<named_semaphore>(0, named_semaphore_exception_factory{\"name_of_the_semaphore\"});\n    auto check_result = [sem] (future<> f) {\n        try {\n            f.get();\n            BOOST_FAIL(\"Expecting an exception\");\n        } catch (broken_named_semaphore& ex) {\n            BOOST_REQUIRE_NE(std::string(ex.what()).find(\"name_of_the_semaphore\"), std::string::npos);\n        } catch (...) {\n            BOOST_FAIL(\"Expected an instance of broken_named_semaphore with proper semaphore name\");\n        }\n        return make_ready_future<>();\n    };\n    auto ret = sem->wait().then_wrapped(check_result);\n    sem->broken();\n    sem->wait().then_wrapped(check_result).then([ret = std::move(ret)] () mutable {\n        return std::move(ret);\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_named_semaphore_timeout) {\n    auto sem = make_lw_shared<named_semaphore>(0, named_semaphore_exception_factory{\"name_of_the_semaphore\"});\n\n    auto f = sem->wait(named_semaphore::clock::now() + 1ms, 1);\n    try {\n        f.get();\n        BOOST_FAIL(\"Expecting an exception\");\n    } catch (named_semaphore_timed_out& ex) {\n        BOOST_REQUIRE_NE(std::string(ex.what()).find(\"name_of_the_semaphore\"), std::string::npos);\n    } catch (...) {\n        BOOST_FAIL(\"Expected an instance of named_semaphore_timed_out with proper semaphore name\");\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_abort_after_wait) {\n    auto sem = semaphore(0);\n    abort_source as;\n    int x = 0;\n    auto fut1 = sem.wait(as).then([&x] {\n        x++;\n    });\n    as.request_abort();\n    sem.signal();\n    BOOST_CHECK_THROW(fut1.get(), semaphore_aborted);\n    BOOST_REQUIRE_EQUAL(x, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_abort_with_exception_after_wait) {\n    auto sem = semaphore(0);\n    abort_source as;\n    int x = 0;\n    auto fut1 = sem.wait(as).then([&x] {\n        x++;\n    });\n    as.request_abort_ex(expected_exception());\n    sem.signal();\n    BOOST_CHECK_THROW(fut1.get(), expected_exception);\n    BOOST_REQUIRE_EQUAL(x, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_semaphore_abort_before_wait) {\n    auto sem = semaphore(0);\n    abort_source as;\n    int x = 0;\n    as.request_abort();\n    auto fut1 = sem.wait(as).then([&x] {\n        x++;\n    });\n    sem.signal();\n    BOOST_CHECK_THROW(fut1.get(), semaphore_aborted);\n    BOOST_REQUIRE_EQUAL(x, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reassigned_units_are_returned) {\n    auto sem0 = semaphore(1);\n    auto sem1 = semaphore(1);\n    auto units = get_units(sem0, 1).get();\n    auto wait = sem0.wait(1);\n    BOOST_REQUIRE(!wait.available());\n    units = get_units(sem1, 1).get();\n    timer t([] { abort(); });\n    t.arm(1s);\n    // will hang if units are not returned when reassigned\n    wait.get();\n    t.cancel();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_get_units_after_move) {\n    auto sem = std::make_unique<semaphore>([] { return semaphore(0); }());\n    auto f = get_units(*sem, 1);\n    BOOST_REQUIRE(!f.available());\n    sem->signal();\n    f.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_get_units_abort_after_move) {\n    auto sem = std::make_unique<semaphore>([] { return semaphore(0); }());\n    abort_source as;\n    auto f = get_units(*sem, 1, as);\n    BOOST_REQUIRE(!f.available());\n    as.request_abort();\n\n    BOOST_REQUIRE_THROW(f.get(), abort_requested_exception);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_immediate_abort_after_move) {\n    auto sem = std::make_unique<semaphore>([] { return semaphore(0); }());\n    abort_source as;\n    as.request_abort();\n\n    BOOST_REQUIRE_THROW(get_units(*sem, 1, as).get(), abort_requested_exception);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_get_units_after_move_assign) {\n    auto sem = semaphore(0);\n    sem = [] { return semaphore(0); }();\n    auto f = get_units(sem, 1);\n    BOOST_REQUIRE(!f.available());\n    sem.signal();\n    f.get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_get_units_abort_after_move_assign) {\n    auto sem = semaphore(0);\n    sem = [] { return semaphore(0); }();\n    abort_source as;\n    auto f = get_units(sem, 1, as);\n    BOOST_REQUIRE(!f.available());\n    as.request_abort();\n\n    BOOST_REQUIRE_THROW(f.get(), abort_requested_exception);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_immediate_abort_after_move_assign) {\n    auto sem = semaphore(0);\n    sem = [] { return semaphore(0); }();\n    abort_source as;\n    as.request_abort();\n\n    BOOST_REQUIRE_THROW(get_units(sem, 1, as).get(), abort_requested_exception);\n}\n"
  },
  {
    "path": "tests/unit/sharded_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n *  Copyright (C) 2018 Scylladb, Ltd.\n */\n\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/shard_id.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/util/assert.hh>\n\n#include <ranges>\n\nusing namespace seastar;\n\nnamespace {\nclass invoke_on_during_stop final : public peering_sharded_service<invoke_on_during_stop> {\n    bool flag = false;\n\npublic:\n    future<> stop() {\n        return container().invoke_on(0, [] (invoke_on_during_stop& instance) {\n            instance.flag = true;\n        });\n    }\n\n    ~invoke_on_during_stop() {\n        if (this_shard_id() == 0) {\n            SEASTAR_ASSERT(flag);\n        }\n    }\n};\n}\n\nSEASTAR_THREAD_TEST_CASE(invoke_on_during_stop_test) {\n    sharded<invoke_on_during_stop> s;\n    s.start().get();\n    s.stop().get();\n}\n\nclass peering_counter : public peering_sharded_service<peering_counter> {\npublic:\n    future<int> count() const {\n        return container().map_reduce(adder<int>(), [] (auto& pc) { return 1; });\n    }\n\n    future<int> count_from(int base) const {\n        return container().map_reduce0([] (auto& pc) { return 1; }, base, std::plus<int>());\n    }\n\n    future<int> count_from_const(int base) const {\n        return container().map_reduce0(&peering_counter::get_1_c, base, std::plus<int>());\n    }\n\n    future<int> count_from_mutate(int base) {\n        return container().map_reduce0(&peering_counter::get_1_m, base, std::plus<int>());\n    }\n\n    future<int> count_const() const {\n        return container().map_reduce(adder<int>(), &peering_counter::get_1_c);\n    }\n\n    future<int> count_mutate() {\n        return container().map_reduce(adder<int>(), &peering_counter::get_1_m);\n    }\n\nprivate:\n    future<int> get_1_c() const {\n        return make_ready_future<int>(1);\n    }\n\n    future<int> get_1_m() {\n        return make_ready_future<int>(1);\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_const_map_reduces) {\n    sharded<peering_counter> c;\n    c.start().get();\n\n    BOOST_REQUIRE_EQUAL(c.local().count().get(), smp::count);\n    BOOST_REQUIRE_EQUAL(c.local().count_from(1).get(), smp::count + 1);\n\n    c.stop().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_member_map_reduces) {\n    sharded<peering_counter> c;\n    c.start().get();\n\n    BOOST_REQUIRE_EQUAL(std::as_const(c.local()).count_const().get(), smp::count);\n    BOOST_REQUIRE_EQUAL(c.local().count_mutate().get(), smp::count);\n    BOOST_REQUIRE_EQUAL(std::as_const(c.local()).count_from_const(1).get(), smp::count + 1);\n    BOOST_REQUIRE_EQUAL(c.local().count_from_mutate(1).get(), smp::count + 1);\n    c.stop().get();\n}\n\nclass mydata {\npublic:\n    int x = 1;\n    future<> stop() {\n        return make_ready_future<>();\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(invoke_map_returns_non_future_value) {\n    seastar::sharded<mydata> s;\n    s.start().get();\n    s.map([] (mydata& m) {\n        return m.x;\n    }).then([] (std::vector<int> results) {\n        for (auto& x : results) {\n            SEASTAR_ASSERT(x == 1);\n        }\n    }).get();\n    s.stop().get();\n};\n\nSEASTAR_THREAD_TEST_CASE(invoke_map_returns_future_value) {\n    seastar::sharded<mydata> s;\n    s.start().get();\n    s.map([] (mydata& m) {\n        return make_ready_future<int>(m.x);\n    }).then([] (std::vector<int> results) {\n        for (auto& x : results) {\n            SEASTAR_ASSERT(x == 1);\n        }\n    }).get();\n    s.stop().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(invoke_map_returns_future_value_from_thread) {\n    seastar::sharded<mydata> s;\n    s.start().get();\n    s.map([] (mydata& m) {\n        return seastar::async([&m] {\n            return m.x;\n        });\n    }).then([] (std::vector<int> results) {\n        for (auto& x : results) {\n            SEASTAR_ASSERT(x == 1);\n        }\n    }).get();\n    s.stop().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(failed_sharded_start_doesnt_hang) {\n    class fail_to_start {\n    public:\n        fail_to_start() { throw 0; }\n    };\n\n    seastar::sharded<fail_to_start> s;\n    s.start().then_wrapped([] (auto&& fut) { fut.ignore_ready_future(); }).get();\n}\n\nclass argument {\n    int _x;\npublic:\n    argument() : _x(this_shard_id()) {}\n    int get() const { return _x; }\n};\n\nclass service {\npublic:\n    void fn_local(argument& arg) {\n        BOOST_REQUIRE_EQUAL(arg.get(), this_shard_id());\n    }\n\n    void fn_sharded(sharded<argument>& arg) {\n        BOOST_REQUIRE_EQUAL(arg.local().get(), this_shard_id());\n    }\n\n    void fn_sharded_param(int arg) {\n        BOOST_REQUIRE_EQUAL(arg, this_shard_id());\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(invoke_on_all_sharded_arg) {\n    seastar::sharded<service> srv;\n    srv.start().get();\n    seastar::sharded<argument> arg;\n    arg.start().get();\n\n    srv.invoke_on_all(&service::fn_local, std::ref(arg)).get();\n    srv.invoke_on_all(&service::fn_sharded, std::ref(arg)).get();\n    srv.invoke_on_all(&service::fn_sharded_param, sharded_parameter([&arg] { return arg.local().get(); })).get();\n\n    srv.stop().get();\n    arg.stop().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(invoke_on_modifiers) {\n    class checker {\n    public:\n        future<> fn(int a) {\n            return make_ready_future<>();\n        }\n    };\n\n    seastar::sharded<checker> srv;\n    srv.start().get();\n    int a = 42;\n\n    srv.invoke_on_all([a] (checker& s) { return s.fn(a); }).get();\n    srv.invoke_on_all([a] (checker& s) mutable { return s.fn(a); }).get();\n    srv.invoke_on_others([a] (checker& s) { return s.fn(a); }).get();\n    srv.invoke_on_others([a] (checker& s) mutable { return s.fn(a); }).get();\n\n    srv.stop().get();\n}\n\nclass coordinator_synced_shard_map : public peering_sharded_service<coordinator_synced_shard_map> {\n    std::vector<unsigned> unsigned_per_shard;\n    unsigned coordinator_id;\n\npublic:\n    coordinator_synced_shard_map(unsigned coordinator_id) : unsigned_per_shard(smp::count), coordinator_id(coordinator_id) {}\n\n    future<> sync(unsigned value) {\n        return container().invoke_on(coordinator_id, [shard_id = this_shard_id(), value] (coordinator_synced_shard_map& s) {\n            s.unsigned_per_shard[shard_id] = value;\n        });\n    }\n\n    unsigned get_synced(int shard_id) {\n        SEASTAR_ASSERT(this_shard_id() == coordinator_id);\n        return unsigned_per_shard[shard_id];\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(invoke_on_range_contiguous) {\n    sharded<coordinator_synced_shard_map> s;\n    auto coordinator_id = this_shard_id();\n    s.start(coordinator_id).get();\n\n    auto mid = smp::count / 2;\n    auto half1 = std::views::iota(0u, mid);\n    auto half1_id = 1;\n    auto half2 = std::views::iota(mid, smp::count);\n    auto half2_id = 2;\n\n    auto f1 = s.invoke_on(half1, [half1_id] (coordinator_synced_shard_map& s) { return s.sync(half1_id); });\n    auto f2 = s.invoke_on(half2, [half2_id] (coordinator_synced_shard_map& s) { return s.sync(half2_id); });\n    f1.get();\n    f2.get();\n\n    auto f3 = s.invoke_on(coordinator_id, [mid, half1_id, half2_id] (coordinator_synced_shard_map& s) {\n        for (unsigned i = 0; i < mid; ++i) {\n            BOOST_REQUIRE_EQUAL(half1_id, s.get_synced(i));\n        }\n        for (unsigned i = mid; i < smp::count; ++i) {\n            BOOST_REQUIRE_EQUAL(half2_id, s.get_synced(i));\n        }\n    });\n    f3.get();\n\n    s.stop().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(invoke_on_range_fragmented) {\n    sharded<coordinator_synced_shard_map> s;\n    auto coordinator_id = this_shard_id();\n    s.start(coordinator_id).get();\n\n    // TODO: migrate to C++23 std::views::stride\n    auto even = std::views::iota(0u, smp::count) | std::views::filter([](int i) { return i % 2 == 0; });\n    auto even_id = 1;\n    auto odd = std::views::iota(1u, smp::count) | std::views::filter([](int i) { return i % 2 == 1; });\n    auto odd_id = 2;\n\n    auto f1 = s.invoke_on(even, [even_id] (coordinator_synced_shard_map& s) { return s.sync(even_id); });\n    auto f2 = s.invoke_on(odd, [odd_id] (coordinator_synced_shard_map& s) { return s.sync(odd_id); });\n    f1.get();\n    f2.get();\n\n    auto f3 = s.invoke_on(coordinator_id, [even_id, odd_id] (coordinator_synced_shard_map& s) {\n        for (unsigned i = 0; i < smp::count; i += 2) {\n            BOOST_REQUIRE_EQUAL(even_id, s.get_synced(i));\n        }\n        for (unsigned i = 1; i < smp::count; i += 2) {\n            BOOST_REQUIRE_EQUAL(odd_id, s.get_synced(i));\n        }\n    });\n    f3.get();\n\n    s.stop().get();\n}\n"
  },
  {
    "path": "tests/unit/shared_ptr_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2015 Cloudius Systems\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <set>\n#include <unordered_map>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/shared_ptr.hh>\n\nusing namespace seastar;\n\nstruct expected_exception : public std::exception {};\n\nstruct A {\n    static bool destroyed;\n    A() {\n        destroyed = false;\n    }\n    virtual ~A() {\n        destroyed = true;\n    }\n};\n\nstruct A_esft : public A, public enable_lw_shared_from_this<A_esft> {\n};\n\nstruct B {\n    virtual void x() {}\n};\n\nbool A::destroyed = false;\n\nBOOST_AUTO_TEST_CASE(explot_dynamic_cast_use_after_free_problem) {\n    shared_ptr<A> p = ::make_shared<A>();\n    {\n        auto p2 = dynamic_pointer_cast<B>(p);\n        SEASTAR_ASSERT(!p2);\n    }\n    SEASTAR_ASSERT(!A::destroyed);\n}\n\nclass C : public enable_shared_from_this<C> {\npublic:\n    shared_ptr<const C> get() const { return shared_from_this(); }\n};\n\nBOOST_AUTO_TEST_CASE(test_const_ptr) {\n    shared_ptr<C> a = make_shared<C>();\n    shared_ptr<const C> ca = a;\n    BOOST_REQUIRE(ca == a);\n    shared_ptr<const C> cca = ca->get();\n    BOOST_REQUIRE(cca == ca);\n}\n\nstruct D {};\n\nBOOST_AUTO_TEST_CASE(test_lw_const_ptr_1) {\n    auto pd1 = make_lw_shared<const D>(D());\n    auto pd2 = make_lw_shared(D());\n    lw_shared_ptr<const D> pd3 = pd2;\n    BOOST_REQUIRE(pd2 == pd3);\n}\n\nstruct E : enable_lw_shared_from_this<E> {};\n\nBOOST_AUTO_TEST_CASE(test_lw_const_ptr_2) {\n    auto pe1 = make_lw_shared<const E>();\n    auto pe2 = make_lw_shared<E>();\n    lw_shared_ptr<const E> pe3 = pe2;\n    BOOST_REQUIRE(pe2 == pe3);\n}\n\nstruct F : enable_lw_shared_from_this<F> {\n    auto const_method() const {\n        return shared_from_this();\n    }\n};\n\nBOOST_AUTO_TEST_CASE(test_shared_from_this_called_on_const_object) {\n    auto ptr = make_lw_shared<F>();\n    ptr->const_method();\n}\n\nBOOST_AUTO_TEST_CASE(test_exception_thrown_from_constructor_is_propagated) {\n    struct X {\n        X() {\n            throw expected_exception();\n        }\n    };\n    try {\n        auto ptr = make_lw_shared<X>();\n        BOOST_FAIL(\"Constructor should have thrown\");\n    } catch (const expected_exception& e) {\n        BOOST_TEST_MESSAGE(\"Expected exception caught\");\n    }\n    try {\n        auto ptr = ::make_shared<X>();\n        BOOST_FAIL(\"Constructor should have thrown\");\n    } catch (const expected_exception& e) {\n        BOOST_TEST_MESSAGE(\"Expected exception caught\");\n    }\n}\n\nBOOST_AUTO_TEST_CASE(test_indirect_functors) {\n    {\n        std::multiset<shared_ptr<sstring>, indirect_less<shared_ptr<sstring>>> a_set;\n\n        a_set.insert(make_shared<sstring>(\"k3\"));\n        a_set.insert(make_shared<sstring>(\"k1\"));\n        a_set.insert(make_shared<sstring>(\"k2\"));\n        a_set.insert(make_shared<sstring>(\"k4\"));\n        a_set.insert(make_shared<sstring>(\"k0\"));\n\n\n        auto i = a_set.begin();\n        BOOST_REQUIRE_EQUAL(sstring(\"k0\"), *(*i++));\n        BOOST_REQUIRE_EQUAL(sstring(\"k1\"), *(*i++));\n        BOOST_REQUIRE_EQUAL(sstring(\"k2\"), *(*i++));\n        BOOST_REQUIRE_EQUAL(sstring(\"k3\"), *(*i++));\n        BOOST_REQUIRE_EQUAL(sstring(\"k4\"), *(*i++));\n    }\n\n    {\n        std::unordered_map<shared_ptr<sstring>, bool,\n                indirect_hash<shared_ptr<sstring>>, indirect_equal_to<shared_ptr<sstring>>> a_map;\n\n        a_map.emplace(make_shared<sstring>(\"k3\"), true);\n        a_map.emplace(make_shared<sstring>(\"k1\"), true);\n        a_map.emplace(make_shared<sstring>(\"k2\"), true);\n        a_map.emplace(make_shared<sstring>(\"k4\"), true);\n        a_map.emplace(make_shared<sstring>(\"k0\"), true);\n\n        BOOST_REQUIRE(a_map.count(make_shared<sstring>(\"k0\")));\n        BOOST_REQUIRE(a_map.count(make_shared<sstring>(\"k1\")));\n        BOOST_REQUIRE(a_map.count(make_shared<sstring>(\"k2\")));\n        BOOST_REQUIRE(a_map.count(make_shared<sstring>(\"k3\")));\n        BOOST_REQUIRE(a_map.count(make_shared<sstring>(\"k4\")));\n        BOOST_REQUIRE(!a_map.count(make_shared<sstring>(\"k5\")));\n    }\n}\n\ntemplate<typename T>\nvoid do_test_release() {\n    auto ptr = make_lw_shared<T>();\n    BOOST_REQUIRE(!T::destroyed);\n\n    auto ptr2 = ptr;\n\n    BOOST_REQUIRE(!ptr.release());\n    BOOST_REQUIRE(!ptr);\n    BOOST_REQUIRE(ptr2.use_count() == 1);\n\n    auto uptr2 = ptr2.release();\n    BOOST_REQUIRE(uptr2);\n    BOOST_REQUIRE(!ptr2);\n    ptr2 = {};\n\n    BOOST_REQUIRE(!T::destroyed);\n    uptr2 = {};\n\n    BOOST_REQUIRE(T::destroyed);\n\n    // Check destroying via disposer\n    auto ptr3 = make_lw_shared<T>();\n    auto uptr3 = ptr3.release();\n    BOOST_REQUIRE(uptr3);\n    BOOST_REQUIRE(!T::destroyed);\n\n    auto raw_ptr3 = uptr3.release();\n    lw_shared_ptr<T>::dispose(raw_ptr3);\n    BOOST_REQUIRE(T::destroyed);\n}\n\nBOOST_AUTO_TEST_CASE(test_release) {\n    do_test_release<A>();\n    do_test_release<A_esft>();\n}\n\nBOOST_AUTO_TEST_CASE(test_const_release) {\n    do_test_release<const A>();\n    do_test_release<const A_esft>();\n}\n\nBOOST_AUTO_TEST_CASE(test_nullptr_compare) {\n    seastar::shared_ptr<int> ptr;\n    BOOST_REQUIRE(ptr == nullptr);\n    BOOST_REQUIRE(nullptr == ptr);\n    ptr = seastar::make_shared<int>(0);\n    BOOST_REQUIRE(ptr != nullptr);\n    BOOST_REQUIRE(nullptr != ptr);\n\n    seastar::lw_shared_ptr<int> lptr;\n    BOOST_REQUIRE(lptr == nullptr);\n    BOOST_REQUIRE(nullptr == lptr);\n    lptr = seastar::make_lw_shared<int>(0);\n    BOOST_REQUIRE(lptr != nullptr);\n    BOOST_REQUIRE(nullptr != lptr);\n}\n"
  },
  {
    "path": "tests/unit/shared_token_bucket_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 ScyllaDB\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/manual_clock.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/shared_token_bucket.hh>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_basic_non_capped_loop) {\n    internal::shared_token_bucket<uint64_t, std::ratio<1>, internal::capped_release::no, manual_clock> tb(1, 1, 0, false);\n\n    // Grab one token and make sure it's only available in 1s\n    auto th = tb.grab(1);\n    BOOST_REQUIRE(tb.deficiency(th) > 0);\n    manual_clock::advance(1s);\n    tb.replenish(manual_clock::now());\n    BOOST_REQUIRE(tb.deficiency(th) == 0);\n\n    // Grab one more token and check the same\n    th = tb.grab(1);\n    BOOST_REQUIRE(tb.deficiency(th) > 0);\n    manual_clock::advance(1s);\n    tb.replenish(manual_clock::now());\n    BOOST_REQUIRE(tb.deficiency(th) == 0);\n\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_basic_capped_loop) {\n    internal::shared_token_bucket<uint64_t, std::ratio<1>, internal::capped_release::yes, manual_clock> tb(1, 1, 0, false);\n\n    // Grab on token and make sure it's only available in 1s\n    auto th = tb.grab(1);\n    BOOST_REQUIRE(tb.deficiency(th) > 0);\n    manual_clock::advance(1s);\n    tb.replenish(manual_clock::now());\n    BOOST_REQUIRE(tb.deficiency(th) == 0);\n\n    // The 2nd time this trick only works after the 1st token is explicitly released\n    th = tb.grab(1);\n    BOOST_REQUIRE(tb.deficiency(th) > 0);\n    manual_clock::advance(1s);\n    tb.replenish(manual_clock::now());\n    BOOST_REQUIRE(tb.deficiency(th) > 0);\n    manual_clock::advance(1s);\n    tb.release(1);\n    tb.replenish(manual_clock::now());\n    BOOST_REQUIRE(tb.deficiency(th) == 0);\n\n    return make_ready_future<>();\n}\n"
  },
  {
    "path": "tests/unit/signal_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/signal.hh>\n#include <seastar/core/shared_ptr.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n\nusing namespace seastar;\n\nextern \"C\" {\n#include <signal.h>\n#include <sys/types.h>\n#include <unistd.h>\n}\n\nSEASTAR_TEST_CASE(test_sighup) {\n    return do_with(make_lw_shared<promise<>>(), false, [](auto const& p, bool& signaled) {\n        seastar::handle_signal(SIGHUP, [p, &signaled] {\n            signaled = true;\n            p->set_value();\n        });\n\n        kill(getpid(), SIGHUP);\n\n        return p->get_future().then([&] {\n            BOOST_REQUIRE_EQUAL(signaled, true);\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/simple_stream_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#define BOOST_TEST_MODULE simple_stream\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/simple-stream.hh>\n\nusing namespace seastar;\n\ntemplate<typename Input, typename Output>\nstatic void write_read_test(Input in, Output out)\n{\n    auto aa = std::vector<char>(4, 'a');\n    auto bb = std::vector<char>(3, 'b');\n    auto cc = std::vector<char>(2, 'c');\n\n    out.write(aa.data(), aa.size());\n    out.fill('b', 3);\n\n    BOOST_CHECK_THROW(out.fill(' ', 3), std::out_of_range);\n    BOOST_CHECK_THROW(out.write(\"   \", 3), std::out_of_range);\n\n    out.write(cc.data(), cc.size());\n\n    BOOST_CHECK_THROW(out.fill(' ', 1), std::out_of_range);\n    BOOST_CHECK_THROW(out.write(\" \", 1), std::out_of_range);\n\n    auto actual_aa = std::vector<char>(4);\n    in.read(actual_aa.data(), actual_aa.size());\n    BOOST_CHECK_EQUAL(aa, actual_aa);\n\n    auto actual_bb = std::vector<char>(3);\n    in.read(actual_bb.data(), actual_bb.size());\n    BOOST_CHECK_EQUAL(bb, actual_bb);\n\n    actual_aa.resize(1024);\n    BOOST_CHECK_THROW(in.read(actual_aa.data(), actual_aa.size()), std::out_of_range);\n\n    auto actual_cc = std::vector<char>(2);\n    in.read(actual_cc.data(), actual_cc.size());\n    BOOST_CHECK_EQUAL(cc, actual_cc);\n\n    BOOST_CHECK_THROW(in.read(actual_aa.data(), 1), std::out_of_range);\n}\n\nBOOST_AUTO_TEST_CASE(simple_write_read_test) {\n    auto buf = temporary_buffer<char>(9);\n\n    write_read_test(simple_memory_input_stream(buf.get(), buf.size()),\n                    simple_memory_output_stream(buf.get_write(), buf.size()));\n\n    std::fill_n(buf.get_write(), buf.size(), 'x');\n\n    auto out = simple_memory_output_stream(buf.get_write(), buf.size());\n    write_read_test(out.to_input_stream(), out);\n}\n\nBOOST_AUTO_TEST_CASE(fragmented_write_read_test) {\n    static constexpr size_t total_size = 9;\n\n    auto bufs = std::vector<temporary_buffer<char>>();\n    using iterator_type = std::vector<temporary_buffer<char>>::iterator;\n\n    auto test = [&] {\n        write_read_test(fragmented_memory_input_stream<iterator_type>(bufs.begin(), total_size),\n                        fragmented_memory_output_stream<iterator_type>(bufs.begin(), total_size));\n\n        auto out = fragmented_memory_output_stream<iterator_type>(bufs.begin(), total_size);\n        write_read_test(out.to_input_stream(), out);\n    };\n\n    bufs.emplace_back(total_size);\n    test();\n\n    bufs.clear();\n    for (auto i = 0u; i < total_size; i++) {\n        bufs.emplace_back(1);\n    }\n    test();\n}\n"
  },
  {
    "path": "tests/unit/slab_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n *\n */\n\n#define BOOST_TEST_MODULE slab\n\n#include <boost/test/unit_test.hpp>\n#include <iostream>\n#include <seastar/core/slab.hh>\n\nusing namespace seastar;\n\nnamespace bi = boost::intrusive;\n\nstatic constexpr size_t max_object_size = 1024*1024;\n\nclass item : public slab_item_base {\npublic:\n    bi::list_member_hook<> _cache_link;\n    uint32_t _slab_page_index;\n\n    item(uint32_t slab_page_index) : _slab_page_index(slab_page_index) {}\n\n    const uint32_t get_slab_page_index() {\n        return _slab_page_index;\n    }\n    const bool is_unlocked() {\n        return true;\n    }\n};\n\ntemplate<typename Item>\nstatic void free_vector(slab_allocator<Item>& slab, std::vector<item *>& items) {\n    for (auto item : items) {\n        slab.free(item);\n    }\n}\n\nBOOST_AUTO_TEST_CASE(test_allocation_1) {\n    constexpr double growth_factor = 1.25;\n    constexpr unsigned slab_limit_size = 5*1024*1024;\n\n    slab_allocator<item> slab(growth_factor, slab_limit_size, max_object_size);\n    constexpr size_t size = max_object_size;\n\n    slab.print_slab_classes();\n\n    std::vector<item *> items;\n\n    static_assert(slab_limit_size % size == 0);\n    for (auto i = 0u; i < (slab_limit_size / size); i++) {\n        auto item = slab.create(size);\n        items.push_back(item);\n    }\n    BOOST_REQUIRE(slab.create(size) == nullptr);\n\n    free_vector<item>(slab, items);\n    std::cout << __FUNCTION__ << \" done!\\n\";\n}\n\nBOOST_AUTO_TEST_CASE(test_allocation_2) {\n    constexpr double growth_factor = 1.07; // it's growth_factor used by facebook\n    constexpr unsigned slab_limit_size = 5*1024*1024;\n\n    slab_allocator<item> slab(growth_factor, slab_limit_size, max_object_size);\n    size_t size = 1024;\n\n    std::vector<item *> items;\n\n    auto allocations = 0u;\n    for (;;) {\n        auto item = slab.create(size);\n        if (!item) {\n            break;\n        }\n        items.push_back(item);\n        allocations++;\n    }\n\n    auto class_size = slab.class_size(size);\n    auto per_slab_page = max_object_size / class_size;\n    auto available_slab_pages = slab_limit_size / max_object_size;\n    BOOST_REQUIRE_EQUAL(allocations, per_slab_page * available_slab_pages);\n\n    free_vector<item>(slab, items);\n    std::cout << __FUNCTION__ << \" done!\\n\";\n}\n\nBOOST_AUTO_TEST_CASE(test_allocation_with_lru) {\n    constexpr double growth_factor = 1.25;\n    constexpr unsigned slab_limit_size = 5*1024*1024;\n\n    bi::list<item, bi::member_hook<item, bi::list_member_hook<>, &item::_cache_link>> _cache;\n    unsigned evictions = 0;\n\n    slab_allocator<item> slab(growth_factor, slab_limit_size, max_object_size,\n        [&](item& item_ref) { _cache.erase(_cache.iterator_to(item_ref)); evictions++; });\n    size_t size = max_object_size;\n\n    auto max = slab_limit_size / max_object_size;\n    for (auto i = 0u; i < max * 1000; i++) {\n        auto item = slab.create(size);\n        BOOST_REQUIRE(item != nullptr);\n        _cache.push_front(*item);\n    }\n    BOOST_REQUIRE_EQUAL(evictions, max * 999);\n\n    _cache.clear();\n\n    std::cout << __FUNCTION__ << \" done!\\n\";\n}\n\nBOOST_AUTO_TEST_CASE(test_limit_is_violated_by_new_class) {\n    constexpr double growth_factor = 1.25;\n    constexpr unsigned slab_limit_size = 5*1024*1024;\n\n    slab_allocator<item> slab(growth_factor, slab_limit_size, max_object_size);\n\n    // Exhaust the slab page limit using the largest possible object size.\n    // This will consume all `_available_slab_pages`.\n    constexpr size_t large_size = max_object_size;\n    static_assert(slab_limit_size % large_size == 0);\n    auto pages_in_limit = slab_limit_size / large_size;\n\n    std::vector<item *> items;\n    for (auto i = 0u; i < pages_in_limit; i++) {\n        auto item = slab.create(large_size);\n        BOOST_REQUIRE(item != nullptr); // These should succeed\n        items.push_back(item);\n    }\n\n    // Verify that the limit is enforced for the same slab class.\n    // This assertion should pass, as it does in test_allocation_1.\n    BOOST_REQUIRE(slab.create(large_size) == nullptr);\n\n    // According to the memory limit, this should fail.\n    const size_t medium_size = 1024;\n    item* leaky_item = slab.create(medium_size);\n\n    // Assert that the allocation FAILED, proving the limit is now correctly enforced.\n    BOOST_REQUIRE(leaky_item == nullptr);\n\n    free_vector<item>(slab, items);\n    std::cout << __FUNCTION__ << \" done!\\n\";\n}\n"
  },
  {
    "path": "tests/unit/smp_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/print.hh>\n\nusing namespace seastar;\n\nfuture<bool> test_smp_call() {\n    return smp::submit_to(1, [] {\n        return make_ready_future<int>(3);\n    }).then([] (int ret) {\n        return make_ready_future<bool>(ret == 3);\n    });\n}\n\nstruct nasty_exception {};\n\nfuture<bool> test_smp_exception() {\n    fmt::print(\"1\\n\");\n    return smp::submit_to(1, [] {\n        fmt::print(\"2\\n\");\n        auto x = make_exception_future<int>(nasty_exception());\n        fmt::print(\"3\\n\");\n        return x;\n    }).then_wrapped([] (future<int> result) {\n        fmt::print(\"4\\n\");\n        try {\n            result.get();\n            return make_ready_future<bool>(false); // expected an exception\n        } catch (nasty_exception&) {\n            // all is well\n            return make_ready_future<bool>(true);\n        } catch (...) {\n            // incorrect exception type\n            return make_ready_future<bool>(false);\n        }\n    });\n}\n\nint tests, fails;\n\nfuture<>\nreport(sstring msg, future<bool>&& result) {\n    return std::move(result).then([msg] (bool result) {\n        fmt::print(\"{}: {}\\n\", (result ? \"PASS\" : \"FAIL\"), msg);\n        tests += 1;\n        fails += !result;\n    });\n}\n\nint main(int ac, char** av) {\n    return app_template().run_deprecated(ac, av, [] {\n       return report(\"smp call\", test_smp_call()).then([] {\n           return report(\"smp exception\", test_smp_exception());\n       }).then([] {\n           fmt::print(\"\\n{:d} tests / {:d} failures\\n\", tests, fails);\n           engine().exit(fails ? 1 : 0);\n       });\n    });\n}\n"
  },
  {
    "path": "tests/unit/socket_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Elazar Leibovich\n */\n\n#include <seastar/core/reactor.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/core/app-template.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/byteorder.hh>\n#include <seastar/core/alien.hh>\n#include <seastar/core/smp.hh>\n#include <seastar/util/assert.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/later.hh>\n#include <seastar/util/defer.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/abort_source.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/posix-stack.hh>\n\n#include <optional>\n#include <tuple>\n#include <future>\n\nusing namespace seastar;\n\nfuture<> handle_connection(connected_socket s) {\n    auto in = s.input();\n    auto out = s.output();\n    return do_with(std::move(in), std::move(out), [](auto& in, auto& out) {\n        return do_until([&in]() { return in.eof(); },\n            [&in, &out] {\n                return in.read().then([&out](auto buf) {\n                    return out.write(std::move(buf)).then([&out]() { return out.close(); });\n                });\n            });\n    });\n}\n\nfuture<> echo_server_loop() {\n    return do_with(\n        server_socket(listen(make_ipv4_address({1234}), listen_options{.reuse_address = true})), [](auto& listener) {\n              // Connect asynchronously in background.\n              (void)connect(make_ipv4_address({\"127.0.0.1\", 1234})).then([](connected_socket&& socket) {\n                  socket.shutdown_output();\n              });\n              return listener.accept().then(\n                  [](accept_result ar) {\n                      connected_socket s = std::move(ar.connection);\n                      return handle_connection(std::move(s));\n                  }).then([l = std::move(listener)]() mutable { return l.abort_accept(); });\n        });\n}\n\nclass my_malloc_allocator : public std::pmr::memory_resource {\npublic:\n    int allocs;\n    int frees;\n    void* do_allocate(std::size_t bytes, std::size_t alignment) override { allocs++; return malloc(bytes); }\n    void do_deallocate(void *ptr, std::size_t bytes, std::size_t alignment) override { frees++; return free(ptr); }\n    virtual bool do_is_equal(const std::pmr::memory_resource& __other) const noexcept override { abort(); }\n};\n\nmy_malloc_allocator malloc_allocator;\nstd::pmr::polymorphic_allocator<char> allocator{&malloc_allocator};\n\nSEASTAR_TEST_CASE(socket_allocation_test) {\n    return echo_server_loop().finally([](){ engine().exit((malloc_allocator.allocs == malloc_allocator.frees) ? 0 : 1); });\n}\n\nSEASTAR_TEST_CASE(socket_skip_test) {\n    return seastar::async([&] {\n        listen_options lo;\n        lo.reuse_address = true;\n        server_socket ss = seastar::listen(ipv4_addr(\"127.0.0.1\", 1234), lo);\n\n        abort_source as;\n        auto client = async([&as] {\n            connected_socket socket = connect(ipv4_addr(\"127.0.0.1\", 1234)).get();\n            socket.output().write(\"abc\").get();\n            socket.shutdown_output();\n            try {\n                sleep_abortable(std::chrono::seconds(10), as).get();\n            } catch (const sleep_aborted&) {\n                // expected\n                return;\n            }\n            SEASTAR_ASSERT(!\"Skipping data from socket is likely stuck\");\n        });\n\n        accept_result accepted = ss.accept().get();\n        input_stream<char> input = accepted.connection.input();\n        input.skip(16).get();\n        as.request_abort();\n        client.get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_file_desc_fdinfo) {\n    auto fd = file_desc::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);\n    auto info = fd.fdinfo();\n    BOOST_REQUIRE_EQUAL(info.substr(0, 8), \"socket:[\");\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(socket_on_close_test) {\n    return seastar::async([&] {\n        listen_options lo;\n        lo.reuse_address = true;\n        server_socket ss = seastar::listen(ipv4_addr(\"127.0.0.1\", 12345), lo);\n\n        bool server_closed = false;\n        bool client_notified = false;\n\n        auto client = seastar::async([&] {\n            connected_socket cln = connect(ipv4_addr(\"127.0.0.1\", 12345)).get();\n\n            auto close_wait_fiber = cln.wait_input_shutdown().then([&] {\n                BOOST_REQUIRE_EQUAL(server_closed, true);\n                client_notified = true;\n                fmt::print(\"Client: server closed\\n\");\n            });\n\n            auto out = cln.output();\n            auto in = cln.input();\n\n            while (!client_notified) {\n                fmt::print(\"Client: -> message\\n\");\n                out.write(\"hello\").get();\n                out.flush().get();\n                seastar::sleep(std::chrono::milliseconds(250)).get();\n                fmt::print(\"Client: <- message\\n\");\n                auto buf = in.read().get();\n                if (!buf) {\n                    fmt::print(\"Client: server eof\\n\");\n                    break;\n                }\n                seastar::sleep(std::chrono::milliseconds(250)).get();\n            }\n\n            out.close().get();\n            in.close().get();\n            close_wait_fiber.get();\n        });\n\n        auto server = seastar::async([&] {\n            accept_result acc = ss.accept().get();\n            auto out = acc.connection.output();\n            auto in = acc.connection.input();\n\n            for (int i = 0; i < 3; i++) {\n                auto buf = in.read().get();\n                BOOST_REQUIRE_EQUAL(client_notified, false);\n                out.write(std::move(buf)).get();\n                out.flush().get();\n                fmt::print(\"Server: served\\n\");\n            }\n\n            server_closed = true;\n            fmt::print(\"Server: closing\\n\");\n            out.close().get();\n            in.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\nSEASTAR_TEST_CASE(socket_on_close_local_shutdown_test) {\n    return seastar::async([&] {\n        listen_options lo;\n        lo.reuse_address = true;\n        server_socket ss = seastar::listen(ipv4_addr(\"127.0.0.1\", 12345), lo);\n\n        bool server_closed = false;\n        bool client_notified = false;\n\n        auto client = seastar::async([&] {\n            connected_socket cln = connect(ipv4_addr(\"127.0.0.1\", 12345)).get();\n\n            auto close_wait_fiber = cln.wait_input_shutdown().then([&] {\n                BOOST_REQUIRE_EQUAL(server_closed, false);\n                client_notified = true;\n                fmt::print(\"Client: socket closed\\n\");\n            });\n\n            auto out = cln.output();\n            cln.shutdown_input();\n\n            auto fin = std::chrono::steady_clock::now() + std::chrono::seconds(1);\n            do {\n                seastar::yield().get();\n            } while (!client_notified && std::chrono::steady_clock::now() < fin);\n            BOOST_REQUIRE_EQUAL(client_notified, true);\n\n            out.write(\"hello\").get();\n            out.flush().get();\n            out.close().get();\n\n            close_wait_fiber.get();\n        });\n\n        auto server = seastar::async([&] {\n            accept_result acc = ss.accept().get();\n            auto in = acc.connection.input();\n            auto buf = in.read().get();\n            server_closed = true;\n            fmt::print(\"Server: closing\\n\");\n            in.close().get();\n        });\n\n        when_all(std::move(client), std::move(server)).discard_result().get();\n    });\n}\n\n// The test makes sure it's possible to abort connect()-ing a socket before\n// it succeeds or fails. The way to abort the in-flight connection is to call\n// shutdown() on the socket. The connect()'s future<> must resolve shortly\n// after that with exception.\n//\n// The test currently fails on io_uring backend -- calling shutdown() doesn't\n// make connect() future<> to resolve, instead it resolves after kernel times\n// out the socket, which's not what test expects (see scylladb/seastar#2303)\nSEASTAR_TEST_CASE(socket_connect_abort_test) {\n    return seastar::async([&] {\n        bool too_late = false;\n        auto sk = make_socket();\n        auto cf = sk.connect(ipv4_addr(\"192.0.2.1\", 12345)).then([] (auto cs) {\n            fmt::print(\"Connected\\n\");\n            BOOST_REQUIRE(false);\n        }).handle_exception([&too_late] (auto ex) {\n            fmt::print(\"Cannot connect {}\\n\", ex);\n            BOOST_REQUIRE(!too_late);\n        });\n\n        auto abort = sleep(std::chrono::milliseconds(500)).then([&sk] {\n            fmt::print(\"Abort connect\\n\");\n            sk.shutdown();\n        });\n\n        auto check = sleep(std::chrono::seconds(2)).then([&too_late] {\n            fmt::print(\"Connection must have been aborted already\\n\");\n            too_late = true;\n        });\n\n        when_all(std::move(cf), std::move(check), std::move(abort)).get();\n    });\n}\n\n// Check that server_socket::accept() is abortable by server_socket::abort_accept()\nSEASTAR_THREAD_TEST_CASE(socket_accept_abort_test) {\n    ipv4_addr addr(\"127.0.0.1\", 3174);\n    server_socket ss = seastar::listen(addr, listen_options{ .reuse_address = true });\n    bool too_late = false;\n    auto f = async([&] {\n        try {\n            ss.accept().get();\n            BOOST_FAIL(\"Accept didn't resolve into exception\");\n        } catch (const std::system_error& e) {\n            BOOST_REQUIRE(!too_late);\n            BOOST_REQUIRE_EQUAL(e.code(), std::error_code(ECONNABORTED, std::system_category()));\n        }\n    });\n\n    auto abort = sleep(std::chrono::milliseconds(500)).then([&ss] {\n        fmt::print(\"Abort accept\\n\");\n        ss.abort_accept();\n    });\n\n    auto check = sleep(std::chrono::seconds(2)).then([&too_late] {\n        fmt::print(\"Accept must have been aborted already\\n\");\n        too_late = true;\n    });\n\n    when_all(std::move(f), std::move(abort), std::move(check)).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(socket_bufsize) {\n\n    // Test that setting the send and recv buffer sizes on the listening\n    // socket is propagated to the socket returned by accept().\n\n    auto buf_size = [](std::optional<int> snd_size, std::optional<int> rcv_size) {\n        listen_options lo{\n            .reuse_address = true,\n            .lba = server_socket::load_balancing_algorithm::fixed,\n            .so_sndbuf = snd_size,\n            .so_rcvbuf = rcv_size\n        };\n\n        ipv4_addr addr(\"127.0.0.1\", 1234);\n        server_socket ss = seastar::listen(addr, lo);\n        connected_socket client = connect(addr).get();\n        connected_socket server = ss.accept().get().connection;\n\n        auto sockopt = [&](int option) {\n            int val{};\n            int ret = server.get_sockopt(SOL_SOCKET, option, &val, sizeof(val));\n            BOOST_REQUIRE_EQUAL(ret, 0);\n            return val;\n        };\n\n        int send = sockopt(SO_SNDBUF);\n        int recv = sockopt(SO_RCVBUF);\n\n        ss.abort_accept();\n        client.shutdown_output();\n        server.shutdown_output();\n\n\n        return std::make_tuple(send, recv);\n    };\n\n    constexpr int small_size = 8192, big_size = 128 * 1024;\n\n    // we pass different sizes for send and recv to catch any copy/paste\n    // style bugs\n    auto [send_small, recv_small] = buf_size(small_size, small_size * 2);\n    auto [send_big, recv_big] = buf_size(big_size, big_size * 2);\n\n    // Setting socket buffer sizes isn't an exact science: the kernel does\n    // some rounding, and also (currently) doubles the requested size and\n    // also applies so limits. So as a basic check, assert simply that the\n    // explicit small buffer ends up smaller than the explicit big buffer,\n    // and that both results are at least as large as the requested amount.\n    // The latter condition could plausibly fail if the OS clamped the size\n    // at a small amount, but this is unlikely for the chosen buffer sizes.\n\n    BOOST_CHECK_LT(send_small, send_big);\n    BOOST_CHECK_LT(recv_small, recv_big);\n\n    BOOST_CHECK_GE(send_small, small_size);\n    BOOST_CHECK_GE(send_big, big_size);\n\n    BOOST_CHECK_GE(recv_small, small_size * 2);\n    BOOST_CHECK_GE(recv_big, big_size * 2);\n\n    // not much to check here with \"default\" sizes, but let's at least call it\n    // and check that we get a reasonable answer\n    auto [send_default, recv_default] = buf_size({}, {});\n\n    BOOST_CHECK_GE(send_default, 4096);\n    BOOST_CHECK_GE(recv_default, 4096);\n\n    // we don't really know the default socket size and it can vary by kernel\n    // config, but 20 MB should be enough for everyone.\n    BOOST_CHECK_LT(send_default, 20'000'000);\n    BOOST_CHECK_LT(recv_default, 20'000'000);\n}\n\nstatic\nvoid\ntest_load_balancing_algorithm_port(socket_address listen_addr, bool proxy_protocol) {\n    auto& alien = engine().alien();\n    listen_options lo;\n    lo.reuse_address = true;\n    lo.lba = server_socket::load_balancing_algorithm::port;\n    lo.proxy_protocol = proxy_protocol;\n\n    struct client_results {\n        int attempts = 0;\n        int bad_socket = 0;\n        int bad_bind = 0;\n        int bad_connect = 0;\n        int bad_proxy_send = 0;\n        int bad_recv = 0;\n        int bad_shard = 0;\n        int good = 0;\n    };\n\n    struct shard_number_server {\n        server_socket ss;\n        future<> runner;\n        shard_number_server(socket_address addr, listen_options lo)\n                : ss(seastar::listen(addr, lo))\n                , runner(run()) {\n        }\n        future<> run() {\n            try {\n                while (true) {\n                    auto [cs, _] = co_await ss.accept();\n                    auto out = cs.output();\n                    auto in = cs.input();\n                    unsigned shard_id = this_shard_id();\n                    char buf[4];\n                    write_be<unsigned>(buf, shard_id);\n                    co_await out.write(buf, sizeof(buf));\n                    co_await out.close();\n                    co_await in.close();\n                }\n            } catch (...) {\n                // expected on abort_accept\n            }\n        }\n        future<> stop() {\n            ss.abort_accept();\n            return std::move(runner);\n        }\n    };\n    auto server = sharded<shard_number_server>();\n    server.start(listen_addr, lo).get();\n    auto smp_count = smp::count;\n    promise<> client_done;\n    auto client = std::async(std::launch::async, [&] {\n        auto r = client_results{};\n        for (unsigned i = 0; i < 100u; ++i) {\n            ++r.attempts;\n            uint16_t port = 20000 + i;\n            unsigned expected_shard = port % smp_count;\n            auto client_addr = listen_addr;\n            switch (client_addr.family()) {\n            case AF_INET: {\n                auto& sa_in = client_addr.as_posix_sockaddr_in();\n                sa_in.sin_port = htons(port);\n                break;\n            }\n            case AF_INET6: {\n                auto& sa_in6 = client_addr.as_posix_sockaddr_in6();\n                sa_in6.sin6_port = htons(port);\n                break;\n            }\n            default:\n                std::abort();\n            }\n            int conn = ::socket(client_addr.family(), SOCK_STREAM, IPPROTO_TCP);\n            if (conn == -1) {\n                ++r.bad_socket;\n                continue;\n            }\n            // set REUSEADDR to avoid port exhaustion in case of test failures\n            int opt = 1;\n            ::setsockopt(conn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));\n            auto do_close = defer([conn] () noexcept { ::close(conn); });\n            if (!proxy_protocol) {\n                int bind_result = ::bind(conn, &client_addr.as_posix_sockaddr(), client_addr.length());\n                if (bind_result == -1) {\n                    ++r.bad_bind; // port may be busy;\n                    continue;\n                }\n            }\n            int conn_result = ::connect(conn, &listen_addr.as_posix_sockaddr(), listen_addr.length());\n            if (conn_result == -1) {\n                ++r.bad_connect;\n                continue;\n            }\n            if (proxy_protocol) {\n                // send a minimal PROXY protocol v2 header with correct source port\n                char buf[16 + 36] = {\n                    0x0d, 0x0a, 0x0d, 0x0a, 0x00, 0x0d, 0x0a, 0x51, 0x55, 0x49, 0x54, 0x0a, // signature\n                };\n                buf[12] = 0x21;                   // version and command (PROXY)\n                buf[13] = (listen_addr.family() == AF_INET) ? 0x11 : 0x21; // family and protocol\n                uint16_t xlen = (listen_addr.family() == AF_INET) ? 12 : 36;\n                write_be<uint16_t>(buf + 14, xlen); // length\n                if (listen_addr.family() == AF_INET) {\n                    buf[16] = 127;\n                    buf[17] = 0;\n                    buf[18] = 0;\n                    buf[19] = 1;\n                    buf[20] = 127;\n                    buf[21] = 0;\n                    buf[22] = 0;\n                    buf[23] = 1;\n                    write_be<uint16_t>(buf + 24, client_addr.port()); // source port\n                    write_be<uint16_t>(buf + 26, listen_addr.port()); // destination port\n                } else {\n                    buf[31] = 1;\n                    buf[47] = 1;\n                    write_be<uint16_t>(buf + 48, client_addr.port()); // source port\n                    write_be<uint16_t>(buf + 50, listen_addr.port()); // destination port\n                }\n                auto proxy_send_result = ::send(conn, buf, 16 + xlen, 0);\n                if (proxy_send_result != 16 + xlen) {\n                    ++r.bad_proxy_send;\n                    continue;\n                }\n            }\n            char buf[4];\n            auto recv_result = ::recv(conn, buf, sizeof(buf), 0);\n            if (recv_result != 4) {\n                ++r.bad_recv;\n                continue;\n            }\n            auto actual_shard = read_be<unsigned>(buf);\n            if (actual_shard != expected_shard) {\n                ++r.bad_shard;\n                continue;\n            }\n            ++r.good;\n        }\n        alien::submit_to(alien, 0, [&] { client_done.set_value(); return make_ready_future<>(); });\n        return r;\n    });\n    client_done.get_future().get();\n    server.stop().get();\n    auto results = client.get();\n    BOOST_REQUIRE_EQUAL(results.attempts, 100);\n    BOOST_REQUIRE_EQUAL(results.bad_socket, 0);\n    // We can have bad binds due to other connection using the ports.\n    // 20 should be plenty of margin.\n    BOOST_REQUIRE_LE(results.bad_bind, 20);\n    BOOST_REQUIRE_EQUAL(results.bad_connect, 0);\n    BOOST_REQUIRE_EQUAL(results.bad_proxy_send, 0);\n    BOOST_REQUIRE_EQUAL(results.bad_recv, 0);\n    BOOST_REQUIRE_EQUAL(results.bad_shard, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(load_balancing_algorithm_port_ipv4_test) {\n    test_load_balancing_algorithm_port(ipv4_addr(\"127.0.0.1\", 11001), false);\n}\n\nSEASTAR_THREAD_TEST_CASE(load_balancing_algorithm_port_ipv6_test) {\n    test_load_balancing_algorithm_port(ipv6_addr(\"::1\", 11001), false);\n}\n\nSEASTAR_THREAD_TEST_CASE(load_balancing_algorithm_port_ipv4_proxy_test) {\n    test_load_balancing_algorithm_port(ipv4_addr(\"127.0.0.1\", 11001), true);\n}\n\nSEASTAR_THREAD_TEST_CASE(load_balancing_algorithm_port_ipv6_proxy_test) {\n    test_load_balancing_algorithm_port(ipv6_addr(\"::1\", 11001), true);\n}\n\nSEASTAR_THREAD_TEST_CASE(inet_local_remote_address_sanity) {\n    auto addr = make_ipv4_address(11003);\n    auto ls = listen(addr);\n    auto ar_f = ls.accept();\n\n    std::optional<connected_socket> cs = connect(addr).get();\n    auto ar = ar_f.get();\n    auto ss = std::move(ar.connection);\n\n    BOOST_CHECK_EQUAL(cs->local_address(), ss.remote_address());\n    BOOST_CHECK_EQUAL(cs->remote_address(), ss.local_address());\n\n    // Now disconnect the server socket on the kernel level\n    // For that -- write some data into client, then close the client. When\n    // it happens, kernel forces connection reset and server socket will\n    // get into unconnected state\n    auto sout = ss.output();\n    sout.write(\"data\").get();\n    sout.flush().get();\n    sout.close().get();\n    // Sockets are batch-flushed, so we need to give it a time to get\n    // flush-polled and also let kernel transfer the data into client\n    // socket\n    seastar::sleep(std::chrono::milliseconds(500)).get();\n    // Close the socket. Closing in/out streams won't work, it will shutdown\n    // the socket and shutting down doesn't send RST-s\n    cs.reset();\n\n    ss.wait_input_shutdown().get();\n    BOOST_CHECK(ss.remote_address().is_unspecified());\n}\n"
  },
  {
    "path": "tests/unit/source_location_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2021 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n\n#include <seastar/util/std-compat.hh>\n#include <source_location>\n\nusing namespace seastar;\n\nstatic void test_source_location_callee(const char* ref_file, const char* ref_func, int ref_line, std::source_location loc = std::source_location::current()) {\n    BOOST_REQUIRE_EQUAL(loc.file_name(), ref_file);\n    BOOST_REQUIRE_EQUAL(loc.line(), ref_line);\n    BOOST_REQUIRE(std::string(loc.function_name()).find(ref_func) != std::string::npos);\n}\n\nstatic void test_source_location_caller() {\n    test_source_location_callee(__FILE__, __func__, __LINE__);\n}\n\nBOOST_AUTO_TEST_CASE(test_source_location) {\n    test_source_location_caller();\n}\n"
  },
  {
    "path": "tests/unit/spawn_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2022 Kefu Chai ( tchaikov@gmail.com )\n */\n#include <seastar/core/seastar.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/process.hh>\n\nusing namespace seastar;\nusing namespace seastar::experimental;\n\nstatic seastar::logger testlog(\"testlog\");\n\nSEASTAR_TEST_CASE(test_spawn_success) {\n    return spawn_process(\"/bin/true\").then([] (auto process) {\n        return process.wait();\n    }).then([] (auto wstatus) {\n        auto* exit_status = std::get_if<process::wait_exited>(&wstatus);\n        BOOST_REQUIRE(exit_status != nullptr);\n        BOOST_CHECK_EQUAL(exit_status->exit_code, EXIT_SUCCESS);\n    });\n}\n\nSEASTAR_TEST_CASE(test_spawn_failure) {\n    return spawn_process(\"/bin/false\").then([] (auto process) {\n        return process.wait();\n    }).then([] (auto wstatus) {\n        auto* exit_status = std::get_if<process::wait_exited>(&wstatus);\n        BOOST_REQUIRE(exit_status != nullptr);\n        BOOST_CHECK_EQUAL(exit_status->exit_code, EXIT_FAILURE);\n    });\n}\n\nSEASTAR_TEST_CASE(test_spawn_program_does_not_exist) {\n    return spawn_process(\"non/existent/path\").then_wrapped([] (future<process> fut) {\n        BOOST_REQUIRE(fut.failed());\n        BOOST_CHECK_EXCEPTION(std::rethrow_exception(fut.get_exception()),\n                              std::system_error,\n                              [](const auto& e) {\n                                  return e.code().value() == ENOENT;\n                              });\n    });\n}\n\nSEASTAR_TEST_CASE(test_spawn_echo) {\n    const char* echo_cmd = \"/bin/echo\";\n    return spawn_process(echo_cmd, {.argv = {echo_cmd, \"-n\", \"hello\", \"world\"}}).then([] (auto process) {\n        auto cout = process.cout();\n        return do_with(std::move(process), std::move(cout), bool(false), [](auto& p, auto& cout, auto& matched) {\n            using consumption_result_type = typename input_stream<char>::consumption_result_type;\n            using stop_consuming_type = typename consumption_result_type::stop_consuming_type;\n            using tmp_buf = stop_consuming_type::tmp_buf;\n            struct consumer {\n                consumer(std::string_view expected, bool& matched)\n                    : _expected(expected), _matched(matched) {}\n                future<consumption_result_type> operator()(tmp_buf buf) {\n                    if (!std::equal(buf.begin(), buf.end(), _expected.begin())) {\n                        _matched = false;\n                        return make_ready_future<consumption_result_type>(stop_consuming_type({}));\n                    }\n                    _expected.remove_prefix(buf.size());\n                    if (_expected.empty()) {\n                        _matched = true;\n                        return make_ready_future<consumption_result_type>(stop_consuming_type({}));\n                    }\n                    return make_ready_future<consumption_result_type>(continue_consuming{});\n                }\n                std::string_view _expected;\n                bool& _matched;\n            };\n            return cout.consume(consumer(\"hello world\", matched)).then([&matched] {\n                BOOST_CHECK(matched);\n            }).finally([&p] {\n                return p.wait().discard_result();\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_spawn_input) {\n    static const sstring text = \"hello world\\n\";\n    return spawn_process(\"/bin/cat\").then([] (auto process) {\n        auto cin = process.cin();\n        auto cout = process.cout();\n        return do_with(std::move(process), std::move(cin), std::move(cout), [](auto& p, auto& cin, auto& cout) {\n            return cin.write(text).then([&cin] {\n                return cin.close();\n            }).handle_exception_type([] (std::system_error& e) {\n                BOOST_TEST_ERROR(fmt::format(\"failed to write to stdin: {}\", e));\n            }).then([&cout] {\n                return cout.read_exactly(text.size());\n            }).handle_exception_type([] (std::system_error& e) {\n                BOOST_TEST_ERROR(fmt::format(\"failed to read from cout: {}\", e));\n                return make_ready_future<temporary_buffer<char>>();\n            }).then([] (temporary_buffer<char> echo) {\n                BOOST_CHECK_EQUAL(sstring(echo.get(), echo.size()), text);\n            }).finally([&p] {\n                return p.wait().then([](process::wait_status wstatus) {\n                    auto* exit_status = std::get_if<process::wait_exited>(&wstatus);\n                    BOOST_REQUIRE(exit_status != nullptr);\n                    BOOST_CHECK_EQUAL(exit_status->exit_code, EXIT_SUCCESS);\n                 });\n            });\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_spawn_kill) {\n    const char* sleep_cmd = \"/bin/sleep\";\n    // sleep for 10s, but terminate it right away.\n    return spawn_process(sleep_cmd, {.argv = {sleep_cmd, \"10\"}}).then([] (auto process) {\n        auto start = std::chrono::high_resolution_clock::now();\n        return do_with(std::move(process), [](auto& p) {\n            p.terminate();\n            return p.wait();\n        }).then([start](experimental::process::wait_status wait_status) {\n            auto* wait_signaled = std::get_if<experimental::process::wait_signaled>(&wait_status);\n            BOOST_REQUIRE(wait_signaled != nullptr);\n            BOOST_CHECK_EQUAL(wait_signaled->terminating_signal, SIGTERM);\n            auto end = std::chrono::high_resolution_clock::now();\n            auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();\n            // sleep should be terminated in 10ms.\n            // pidfd_open(2) may fail and thus p.wait() falls back to\n            // waitpid(2) with backoff (at least 20ms).\n            // the minimal backoff is added to 10ms, so the test can pass on\n            // older kernels as well.\n            BOOST_CHECK_LE(ms, 10 + 20);\n        });\n    });\n}\n"
  },
  {
    "path": "tests/unit/sstring_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2014 Cloudius Systems\n */\n\n#define BOOST_TEST_MODULE core\n// formatting of std::optional was introduced in fmt 10\n#define FMT_VERSION_OPTIONAL_FORMAT 100000\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/sstring.hh>\n#include <list>\n#include <fmt/ranges.h>\n#if FMT_VERSION >= FMT_VERSION_OPTIONAL_FORMAT\n#include <fmt/std.h>\n#endif\n\nusing namespace std::literals;\nusing namespace seastar;\n\nBOOST_AUTO_TEST_CASE(test_make_sstring) {\n    std::string_view foo = \"foo\";\n    std::string bar = \"bar\";\n    sstring zed = \"zed\";\n    const char* baz = \"baz\";\n    BOOST_REQUIRE_EQUAL(make_sstring(foo, bar, zed, baz, \"bah\"), sstring(\"foobarzedbazbah\"));\n}\n\nBOOST_AUTO_TEST_CASE(test_construction) {\n    BOOST_REQUIRE_EQUAL(sstring(std::string_view(\"abc\")), sstring(\"abc\"));\n}\n\nBOOST_AUTO_TEST_CASE(test_equality) {\n    BOOST_REQUIRE_EQUAL(sstring(\"aaa\"), sstring(\"aaa\"));\n}\n\nBOOST_AUTO_TEST_CASE(test_to_sstring) {\n    BOOST_REQUIRE_EQUAL(to_sstring(1234567), sstring(\"1234567\"));\n}\n\nBOOST_AUTO_TEST_CASE(test_add_literal_to_sstring) {\n    BOOST_REQUIRE_EQUAL(\"x\" + sstring(\"y\"), sstring(\"xy\"));\n}\n\nBOOST_AUTO_TEST_CASE(test_front) {\n    sstring s(\"abcde\");\n    BOOST_CHECK_EQUAL(s.front(), 'a');\n    BOOST_CHECK_EQUAL(std::as_const(s).front(), 'a');\n}\n\nBOOST_AUTO_TEST_CASE(test_find_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find('b'), 1u);\n    BOOST_REQUIRE_EQUAL(sstring(\"babcde\").find('b',1), 2u);\n}\n\nBOOST_AUTO_TEST_CASE(test_find_sstring_compatible) {\n    auto check_find = [](const char* s1, const char* s2, size_t pos) {\n        const auto xpos_ss = sstring(s1).find(s2, pos);\n        const auto xpos_std = std::string(s1).find(s2, pos);\n\n        // verify that std::string really has the same behavior as we just tested for sstring\n        if (xpos_ss == sstring::npos) {  // sstring::npos may not equal std::string::npos ?\n            BOOST_REQUIRE_EQUAL(xpos_std, std::string::npos);\n        } else {\n            BOOST_REQUIRE_EQUAL(xpos_ss, xpos_std);\n        }\n    };\n\n    check_find(\"\", \"\", 0);\n    check_find(\"\", \"\", 1);\n    check_find(\"abcde\", \"\", 0);\n    check_find(\"abcde\", \"\", 1);\n    check_find(\"abcde\", \"\", 5);\n    check_find(\"abcde\", \"\", 6);\n}\n\nBOOST_AUTO_TEST_CASE(test_not_find_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find('x'), sstring::npos);\n}\n\nBOOST_AUTO_TEST_CASE(test_str_find_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find(\"bc\"), 1u);\n    BOOST_REQUIRE_EQUAL(sstring(\"abcbcde\").find(\"bc\", 2), 3u);\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find(\"abcde\"), 0u);\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find(\"\", 5), 5u);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababcbdbef\").find(\"bef\"), 7u);\n    BOOST_REQUIRE_EQUAL(sstring(\"\").find(\"\", 0), 0u);\n}\n\nBOOST_AUTO_TEST_CASE(test_str_not_find_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").find(\"x\"), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"abcdefg\").find(\"cde\", 6), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"abcdefg\").find(\"cde\", 4), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababcbdbe\").find(\"bcd\"), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"\").find(\"\", 1), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").find(\"abcde\"), sstring::npos);\n}\n\nBOOST_AUTO_TEST_CASE(test_str_starts_with) {\n    BOOST_CHECK(std::string(\"abcdefg\").starts_with(\"abcdefg\"));\n    BOOST_CHECK(sstring(\"abcdefg\").starts_with(\"abcdefg\"));\n    BOOST_CHECK(sstring(\"abcdefg\").starts_with(\"ab\"sv));\n    BOOST_CHECK(sstring(\"abcde\").starts_with('a'));\n    BOOST_CHECK(sstring(\"abcde\").starts_with(\"ab\"));\n    BOOST_CHECK(sstring(\"abcdefg\").starts_with(\"\"));\n\n    BOOST_CHECK(!sstring(\"abcde\").starts_with(\"cde\"));\n    BOOST_CHECK(!sstring(\"abcde\").starts_with('b'));\n    BOOST_CHECK(!sstring(\"abcdefg\").starts_with(\"cde\"sv));\n    BOOST_CHECK(!sstring(\"abcdefg\").starts_with(\"ab\\0\"sv));\n}\n\nBOOST_AUTO_TEST_CASE(test_str_ends_with) {\n    BOOST_CHECK(std::string(\"abcdefg\").ends_with(\"abcdefg\"));\n    BOOST_CHECK(sstring(\"abcdefg\").ends_with(\"abcdefg\"));\n    BOOST_CHECK(sstring(\"abcdefg\").ends_with(\"efg\"sv));\n    BOOST_CHECK(sstring(\"abcde\").ends_with('e'));\n    BOOST_CHECK(sstring(\"abcde\").ends_with(\"de\"));\n    BOOST_CHECK(sstring(\"abcdefg\").ends_with(\"\"));\n\n    BOOST_CHECK(!sstring(\"abcde\").ends_with(\"abc\"));\n    BOOST_CHECK(!sstring(\"abcde\").ends_with('b'));\n    BOOST_CHECK(!sstring(\"abcdefg\").ends_with(\"abc\"sv));\n    BOOST_CHECK(!sstring(\"abcdefg\").ends_with(\"efg\\0\"sv));\n}\n\nBOOST_AUTO_TEST_CASE(test_str_contains) {\n    BOOST_CHECK(sstring(\"abcde\").starts_with(\"ab\"sv));\n    BOOST_CHECK(sstring(\"abcde\").starts_with('a'));\n    BOOST_CHECK(sstring(\"abcde\").starts_with(\"ab\"));\n    BOOST_CHECK(sstring(\"abcde\").starts_with(\"\"));\n\n    BOOST_CHECK(sstring(\"abcde\").contains(\"bc\"sv));\n    BOOST_CHECK(sstring(\"abcde\").contains(\"bc\"));\n    BOOST_CHECK(sstring(\"abcde\").contains('c'));\n\n    BOOST_CHECK(sstring(\"abcde\").contains(\"de\"sv));\n    BOOST_CHECK(sstring(\"abcde\").contains(\"de\"));\n\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"bad\"));\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"bce\"));\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"x\"));\n    BOOST_CHECK(!sstring(\"abcde\").contains('x'));\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"ab\\0\"sv));\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"bc\\0\"sv));\n    BOOST_CHECK(!sstring(\"abcde\").contains(\"de\\0\"sv));\n}\n\nBOOST_AUTO_TEST_CASE(test_substr_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").substr(1,2), \"bc\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").substr(1,2), \"bc\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").substr(1,3), \"bc\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").substr(0, 2), \"ab\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").substr(3, 2), \"\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").substr(1), \"bc\");\n}\n\nBOOST_AUTO_TEST_CASE(test_substr_eor_sstring) {\n    BOOST_REQUIRE_THROW(sstring(\"abcde\").substr(6,1), std::out_of_range);\n}\n\nBOOST_AUTO_TEST_CASE(test_at_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abcde\").at(1), 'b');\n    BOOST_REQUIRE_THROW(sstring(\"abcde\").at(6), std::out_of_range);\n    sstring s(\"abcde\");\n    s.at(1) = 'd';\n    BOOST_REQUIRE_EQUAL(s, \"adcde\");\n}\n\nBOOST_AUTO_TEST_CASE(test_find_last_sstring) {\n    BOOST_REQUIRE_EQUAL(sstring(\"ababa\").find_last_of('a'), 4u);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababa\").find_last_of('a',5), 4u);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababa\").find_last_of('a',4), 4u);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababa\").find_last_of('a',3), 2u);\n    BOOST_REQUIRE_EQUAL(sstring(\"ababa\").find_last_of('x'), sstring::npos);\n    BOOST_REQUIRE_EQUAL(sstring(\"\").find_last_of('a'), sstring::npos);\n}\n\n\nBOOST_AUTO_TEST_CASE(test_append) {\n    BOOST_REQUIRE_EQUAL(sstring(\"aba\").append(\"1234\", 3), \"aba123\");\n    BOOST_REQUIRE_EQUAL(sstring(\"aba\").append(\"1234\", 4), \"aba1234\");\n    BOOST_REQUIRE_EQUAL(sstring(\"aba\").append(\"1234\", 0), \"aba\");\n}\n\nBOOST_AUTO_TEST_CASE(test_replace) {\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").replace(1,1, \"xyz\", 1), \"axc\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").replace(3,2, \"xyz\", 2), \"abcxy\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").replace(2,2, \"xyz\", 2), \"abxy\");\n    BOOST_REQUIRE_EQUAL(sstring(\"abc\").replace(0,2, \"\", 0), \"c\");\n    BOOST_REQUIRE_THROW(sstring(\"abc\").replace(4,1, \"xyz\", 1), std::out_of_range);\n    const char* s = \"xyz\";\n    sstring str(\"abcdef\");\n    BOOST_REQUIRE_EQUAL(str.replace(str.begin() + 1 , str.begin() + 3, s + 1, s + 3), \"ayzdef\");\n    BOOST_REQUIRE_THROW(sstring(\"abc\").replace(4,1, \"xyz\", 1), std::out_of_range);\n\n}\n\nBOOST_AUTO_TEST_CASE(test_insert) {\n    sstring str(\"abc\");\n    const char* s = \"xyz\";\n    str.insert(str.begin() +1, s + 1, s + 2);\n    BOOST_REQUIRE_EQUAL(str, \"aybc\");\n    str = \"abc\";\n    BOOST_REQUIRE_THROW(str.insert(str.begin() + 5, s + 1, s + 2), std::out_of_range);\n}\n\nBOOST_AUTO_TEST_CASE(test_erase) {\n    sstring str(\"abcdef\");\n    auto i = str.erase(str.begin() + 1, str.begin() + 3);\n    BOOST_REQUIRE_EQUAL(*i, 'd');\n    BOOST_REQUIRE_EQUAL(str, \"adef\");\n}\n\nBOOST_AUTO_TEST_CASE(test_ctor_iterator) {\n    std::list<char> data{{'a', 'b', 'c'}};\n    sstring s(data.begin(), data.end());\n    BOOST_REQUIRE_EQUAL(s, \"abc\");\n}\n\nBOOST_AUTO_TEST_CASE(test_nul_termination) {\n    using stype = basic_sstring<char, uint32_t, 15, true>;\n\n    for (int size = 1; size <= 32; size *= 2) {\n        auto s1 = uninitialized_string<stype>(size - 1);\n        BOOST_REQUIRE_EQUAL(s1.c_str()[size - 1], '\\0');\n        auto s2 = uninitialized_string<stype>(size);\n        BOOST_REQUIRE_EQUAL(s2.c_str()[size], '\\0');\n\n        s1 = stype(\"01234567890123456789012345678901\", size - 1);\n        BOOST_REQUIRE_EQUAL(s1.c_str()[size - 1], '\\0');\n        s2 = stype(\"01234567890123456789012345678901\", size);\n        BOOST_REQUIRE_EQUAL(s2.c_str()[size], '\\0');\n\n        s1 = stype(size - 1, ' ');\n        BOOST_REQUIRE_EQUAL(s1.c_str()[size - 1], '\\0');\n        s2 = stype(size, ' ');\n        BOOST_REQUIRE_EQUAL(s2.c_str()[size], '\\0');\n\n        s2 = s1;\n        BOOST_REQUIRE_EQUAL(s2.c_str()[s1.size()], '\\0');\n        s2.resize(s1.size());\n        BOOST_REQUIRE_EQUAL(s2.c_str()[s1.size()], '\\0');\n        BOOST_REQUIRE_EQUAL(s1, s2);\n\n        auto new_size = size / 2;\n        s2 = s1;\n        s2.resize(new_size);\n        BOOST_REQUIRE_EQUAL(s2.c_str()[new_size], '\\0');\n        BOOST_REQUIRE(!strncmp(s1.c_str(), s2.c_str(), new_size));\n\n        new_size = size * 2;\n        s2 = s1;\n        s2.resize(new_size);\n        BOOST_REQUIRE_EQUAL(s2.c_str()[new_size], '\\0');\n        BOOST_REQUIRE(!strncmp(s1.c_str(), s2.c_str(), std::min(s1.size(), s2.size())));\n\n        new_size = size * 2;\n        s2 = s1 + s1;\n        BOOST_REQUIRE_EQUAL(s2.c_str()[s2.size()], '\\0');\n        BOOST_REQUIRE(!strncmp(s1.c_str(), s2.c_str(), std::min(s1.size(), s2.size())));\n    }\n}\n\nBOOST_AUTO_TEST_CASE(test_resize_and_overwrite) {\n    static constexpr size_t new_size = 42;\n    static constexpr char pattern = 's';\n    {\n        // the size of new content is identical to the specified count\n        sstring s;\n        s.resize_and_overwrite(new_size, [](char* buf, size_t n) {\n            memset(buf, pattern, n);\n            return n;\n        });\n        BOOST_CHECK_EQUAL(s, sstring(new_size, pattern));\n    }\n    {\n        // the size of new content is smaller than the specified count\n        static constexpr size_t smaller_size = new_size / 2;\n        sstring s;\n        s.resize_and_overwrite(new_size, [](char* buf, size_t n) {\n            memset(buf, pattern, smaller_size);\n            return smaller_size;\n        });\n        BOOST_CHECK_EQUAL(s, sstring(smaller_size, pattern));\n    }\n}\n\n\nBOOST_AUTO_TEST_CASE(test_compares_left_hand_not_string) {\n    // mostly a compile test for non-sstring left-hand-side\n    BOOST_REQUIRE(\"a\" == sstring(\"a\"));\n    BOOST_REQUIRE(std::string(\"a\") == sstring(\"a\"));\n    BOOST_REQUIRE(std::string_view(\"a\") == sstring(\"a\"));\n\n#ifdef __cpp_lib_three_way_comparison\n    BOOST_REQUIRE(\"a\" < sstring(\"b\"));\n    BOOST_REQUIRE(std::string(\"a\") < sstring(\"b\"));\n#endif\n}\n\n\nBOOST_AUTO_TEST_CASE(test_fmt) {\n#if FMT_VERSION >= FMT_VERSION_OPTIONAL_FORMAT\n    // https://github.com/llvm/llvm-project/issues/68849\n    std::ignore = fmt::format(\"{}\", std::optional(sstring{\"hello\"}));\n#endif\n    std::vector<sstring> strings;\n    std::ignore = fmt::format(\"{}\", strings);\n}\n\n"
  },
  {
    "path": "tests/unit/stall_detector_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2018 ScyllaDB Ltd.\n */\n\n#include <boost/test/tools/old/interface.hpp>\n#include <cstddef>\n#include <seastar/core/internal/stall_detector.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/thread_cputime_clock.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/util/later.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <atomic>\n#include <chrono>\n#include <ranges>\n\n#include <sys/mman.h>\n\n#ifndef SEASTAR_DEBUG\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nstatic seastar::logger testlog(\"testlog\");\n\nclass temporary_stall_detector_settings {\n    std::chrono::milliseconds _old_threshold;\n    std::function<void ()> _old_report;\npublic:\n    /**\n     * Temporarily (until destructor) overload the stall detector threshold and reporting function.\n     *\n     * Also resets the reported stalls counter to zero, so the next backtraces will not be supressed.\n     */\n    temporary_stall_detector_settings(std::chrono::duration<double> threshold, std::function<void ()> report = {})\n            : _old_threshold(engine().get_blocked_reactor_notify_ms())\n            , _old_report(reactor::test::get_stall_detector_report_function()) {\n        engine().update_blocked_reactor_notify_ms(std::chrono::duration_cast<std::chrono::milliseconds>(threshold));\n        reactor::test::set_stall_detector_report_function(std::move(report));\n    }\n\n    ~temporary_stall_detector_settings() {\n        engine().update_blocked_reactor_notify_ms(_old_threshold);\n        reactor::test::set_stall_detector_report_function(std::move(_old_report));\n    }\n};\n\nusing void_fn = std::function<void()>;\n\nvoid spin(std::chrono::duration<double> how_much, void_fn body = []{}) {\n    auto end = internal::cpu_stall_detector::clock_type::now() + how_much;\n    while (internal::cpu_stall_detector::clock_type::now() < end) {\n        body(); // spin!\n    }\n}\n\nstatic void spin_user_hires(std::chrono::duration<double> how_much) {\n    auto end = std::chrono::high_resolution_clock::now() + how_much;\n    while (std::chrono::high_resolution_clock::now() < end) {\n\n    }\n}\n\nvoid spin_some_cooperatively(std::chrono::duration<double> how_much, void_fn body = []{}) {\n    auto end = std::chrono::steady_clock::now() + how_much;\n    while (std::chrono::steady_clock::now() < end) {\n        spin(200us, body);\n        if (need_preempt()) {\n            thread::yield();\n        }\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(normal_case) {\n    std::atomic<unsigned> reports{};\n    temporary_stall_detector_settings tsds(10ms, [&] { ++reports; });\n    spin_some_cooperatively(1s);\n    BOOST_REQUIRE_EQUAL(reports, 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(simple_stalls) {\n    std::atomic<unsigned> reports{};\n    temporary_stall_detector_settings tsds(10ms, [&] { ++reports; });\n    unsigned nr = 10;\n    for (unsigned i = 0; i < nr; ++i) {\n        spin_some_cooperatively(100ms);\n        spin(20ms);\n    }\n    spin_some_cooperatively(100ms);\n\n    // blocked-reactor-reports-per-minute defaults to 5, so we don't\n    // get all 10 reports.\n    BOOST_REQUIRE_EQUAL(reports, 5);\n}\n\nSEASTAR_THREAD_TEST_CASE(no_poll_no_stall) {\n    std::atomic<unsigned> reports{};\n    temporary_stall_detector_settings tsds(10ms, [&] { ++reports; });\n    spin_some_cooperatively(1ms); // need to yield so that stall detector change from above take effect\n    static constexpr unsigned tasks = 2000;\n    promise<> p;\n    auto f = p.get_future();\n    parallel_for_each(std::views::iota(0u, tasks), [&p] (unsigned int i) {\n        (void)yield().then([i, &p] {\n            spin(500us);\n            if (i == tasks - 1) {\n                p.set_value();\n            }\n        });\n        return make_ready_future<>();\n    }).get();\n    f.get();\n    BOOST_REQUIRE_EQUAL(reports, 0);\n}\n\n// Triggers stalls by spinning with a specify \"body\" function\n// which takes most of the spin time.\nstatic void test_spin_with_body(const char* what, void_fn body) {\n    // The !count_stacks mode outputs stall notification to stderr as usual\n    // and do not assert anything, but are intended for diagnosing\n    // stall problems by inspecting the output. We expect the userspace\n    // spin test to show no kernel callstack, and the kernel test to\n    // show kernel backtraces in the mmap or munmap path, but this is\n    // not exact since neither test spends 100% of its time in the\n    // selected mode (of course, kernel stacks only appear if the\n    // perf-based stall detected could be enabled).\n    //\n    // Then the count_stacks mode tests that the right number of stacks\n    // were output.\n    for (auto count_stacks : {false, true}) {\n        testlog.info(\"Starting spin test: {}\", what);\n        std::atomic<unsigned> reports{};\n        std::function<void()> reporter = count_stacks ? std::function<void()>{[&]{ ++reports; }} : nullptr;\n        temporary_stall_detector_settings tsds(10ms, std::move(reporter));\n        constexpr unsigned nr = 5;\n        for (unsigned i = 0; i < nr; ++i) {\n            spin_some_cooperatively(100ms, body);\n            spin(20ms, body);\n        }\n        testlog.info(\"Ending spin test: {}\", what);\n        BOOST_CHECK_EQUAL(reports, count_stacks ? 5 : 0);\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(spin_in_userspace) {\n    // a body which spends almost all of its time in userspace\n    test_spin_with_body(\"userspace\", [] { spin_user_hires(1ms); });\n}\n\nstatic void mmap_populate(size_t len) {\n    void *p = mmap(nullptr, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, 0, 0);\n    BOOST_REQUIRE(p != MAP_FAILED);\n    BOOST_REQUIRE(munmap(p, len) == 0);\n}\n\nSEASTAR_THREAD_TEST_CASE(spin_in_kernel) {\n    // a body which spends almost all of its time in the kernel\n    // doing 128K mmaps\n    test_spin_with_body(\"kernel\", [] { mmap_populate(128 * 1024); });\n}\n\n// This test reproduces the issue described in https://github.com/scylladb/seastar/issues/2697\n// The issue is reproduced most quickly using the following arguments:\n// --blocked-reactor-notify-ms=1\n// but it will happen with default arguments as well.\nSEASTAR_THREAD_TEST_CASE(stall_detector_crash) {\n    // increase total_iters to increase chance of failure\n    // the value below is tuned to take about 1 second in\n    // release builds\n    constexpr auto total_iters = 100000;\n    constexpr int max_depth = 20;\n\n    auto now = [] { return std::chrono::high_resolution_clock::now(); };\n\n    auto recursive_thrower = [](auto self, int x) -> void {\n        if (x <= 0) {\n            throw std::runtime_error(\"foo\");\n        } else {\n            try {\n                self(self, x - 1);\n            } catch (...) {\n                if (x & 0xF) {\n                    throw;\n                }\n            }\n        }\n    };\n\n    auto next_yield = now();\n    for (int a = 1; a < total_iters; a++) {\n        if (now() > next_yield) {\n            // we need to periodically yield or else the stall reports will become\n            // less and less frequent as exponentially grow the report interval while\n            // the same task is running\n            thread::yield();\n            next_yield = now() + 40ms;\n            // the next line resets the suppression state which allows many more reports\n            // per second, increasing the chance of a failure\n            reactor::test::set_stall_detector_report_function({});\n        }\n\n        try {\n            recursive_thrower(recursive_thrower, a % max_depth);\n        } catch (...) {\n        }\n\n        if (a % 100000 == 0) {\n            fmt::print(\"Making progress: {:6.3f}%\\n\", 100. * a / total_iters);\n        }\n  }\n}\n\n\n\n#else\n\nSEASTAR_THREAD_TEST_CASE(stall_detector_test_not_valid_in_debug_mode) {\n}\n\n#endif\n"
  },
  {
    "path": "tests/unit/stream_reader_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB.\n */\n\n#include <seastar/core/future.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/http/request.hh>\n#include <seastar/util/short_streams.hh>\n#include <string>\n\nusing namespace seastar;\nusing namespace util;\n\n/*\n * Simple data source producing up to total_size bytes\n * in buffer_size-byte chunks.\n * */\nclass test_source_impl : public data_source_impl {\n    short _current_letter = 0; // a-z corresponds to 0-25\n    size_t _buffer_size;\n    size_t _remaining_size;\npublic:\n    test_source_impl(size_t buffer_size, size_t total_size)\n        : _buffer_size(buffer_size), _remaining_size(total_size) {\n    }\n    virtual future<temporary_buffer<char>> get() override {\n        size_t len = std::min(_buffer_size, _remaining_size);\n        temporary_buffer<char> tmp(len);\n        for (size_t i = 0; i < len; i++) {\n            tmp.get_write()[i] = 'a' + _current_letter;\n            ++_current_letter %= 26;\n        }\n        _remaining_size -= len;\n        return make_ready_future<temporary_buffer<char>>(std::move(tmp));\n    }\n    virtual future<temporary_buffer<char>> skip(uint64_t n) override {\n        _remaining_size -= std::min(_remaining_size, n);\n        _current_letter += n %= 26;\n        return make_ready_future<temporary_buffer<char>>();\n    }\n};\n\nSEASTAR_TEST_CASE(test_read_all) {\n    return async([] {\n        auto check_read_all = [] (input_stream<char>& strm, const char* test) {\n            auto all = read_entire_stream(strm).get();\n            sstring s;\n            for (auto&& buf: all) {\n                s += seastar::to_sstring(std::move(buf));\n            };\n            BOOST_REQUIRE_EQUAL(s, test);\n        };\n        input_stream<char> inp(data_source(std::make_unique<test_source_impl>(5, 15)));\n        check_read_all(inp, \"abcdefghijklmno\");\n        BOOST_REQUIRE(inp.eof());\n        input_stream<char> inp2(data_source(std::make_unique<test_source_impl>(5, 16)));\n        check_read_all(inp2, \"abcdefghijklmnop\");\n        BOOST_REQUIRE(inp2.eof());\n        input_stream<char> empty_inp(data_source(std::make_unique<test_source_impl>(5, 0)));\n        check_read_all(empty_inp, \"\");\n        BOOST_REQUIRE(empty_inp.eof());\n\n        input_stream<char> inp_cont(data_source(std::make_unique<test_source_impl>(5, 15)));\n        BOOST_REQUIRE_EQUAL(to_sstring(read_entire_stream_contiguous(inp_cont).get()), \"abcdefghijklmno\");\n        BOOST_REQUIRE(inp_cont.eof());\n        input_stream<char> inp_cont2(data_source(std::make_unique<test_source_impl>(5, 16)));\n        BOOST_REQUIRE_EQUAL(to_sstring(read_entire_stream_contiguous(inp_cont2).get()), \"abcdefghijklmnop\");\n        BOOST_REQUIRE(inp_cont2.eof());\n        input_stream<char> empty_inp_cont(data_source(std::make_unique<test_source_impl>(5, 0)));\n        BOOST_REQUIRE_EQUAL(to_sstring(read_entire_stream_contiguous(empty_inp_cont).get()), \"\");\n        BOOST_REQUIRE(empty_inp_cont.eof());\n    });\n}\n\nSEASTAR_TEST_CASE(test_skip_all) {\n    return async([] {\n        input_stream<char> inp(data_source(std::make_unique<test_source_impl>(5, 15)));\n        skip_entire_stream(inp).get();\n        BOOST_REQUIRE(inp.eof());\n        BOOST_REQUIRE(to_sstring(inp.read().get()).empty());\n        input_stream<char> inp2(data_source(std::make_unique<test_source_impl>(5, 16)));\n        skip_entire_stream(inp2).get();\n        BOOST_REQUIRE(inp2.eof());\n        BOOST_REQUIRE(to_sstring(inp2.read().get()).empty());\n        input_stream<char> empty_inp(data_source(std::make_unique<test_source_impl>(5, 0)));\n        skip_entire_stream(empty_inp).get();\n        BOOST_REQUIRE(empty_inp.eof());\n        BOOST_REQUIRE(to_sstring(empty_inp.read().get()).empty());\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_read_exactly) {\n    const size_t total_size = 22;\n    for (size_t bs = 3; bs < total_size; bs++) {\n        input_stream<char> in(data_source(std::make_unique<test_source_impl>(5, total_size)));\n        size_t total = 0;\n        while (true) {\n            auto buf = in.read_exactly(bs).get();\n            total += buf.size();\n            if (buf.size() != bs) {\n                BOOST_REQUIRE_LT(buf.size(), bs);\n                if (buf.size() != 0) {\n                    buf = in.read_exactly(bs).get();\n                    BOOST_REQUIRE_EQUAL(buf.size(), 0);\n                }\n                break;\n            }\n        }\n        BOOST_REQUIRE_EQUAL(total, total_size);\n    }\n}\n"
  },
  {
    "path": "tests/unit/temporary_buffer_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2025 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <numeric>\n#include <sys/uio.h>\n#include <fmt/core.h>\n#include <fmt/ranges.h>\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/util/internal/iovec_utils.hh>\n\nusing namespace seastar;\n\nstatic void do_test_detach_buffers(size_t initial_buffers, std::vector<size_t> initial_sizes) {\n    std::vector<temporary_buffer<char>> bufs;\n    bufs.reserve(initial_buffers);\n    char letter = 'a';\n    size_t total_len = 0;\n    for (auto size : initial_sizes) {\n        bufs.emplace_back(temporary_buffer<char>::copy_of(sstring(size, letter++)));\n        total_len += size;\n    }\n\n    auto show_buffers = [] (const std::vector<temporary_buffer<char>>& bufs) {\n        for (auto& b : bufs) {\n            fmt::print(\" [{}]\", internal::to_sstring<sstring>(b));\n        }\n    };\n\n    fmt::print(\"Detaching from {} buffers ({} chars):\", initial_buffers, total_len);\n    show_buffers(bufs);\n    fmt::print(\"\\n\");\n\n    auto merge_buffers = [] (const std::vector<temporary_buffer<char>>& bufs) {\n        size_t len = 0;\n        for (auto& b : bufs) {\n            BOOST_REQUIRE_NE(b.size(), 0);\n            len += b.size();\n        }\n        temporary_buffer<char> res(len);\n        len = 0;\n        for (auto& b : bufs) {\n            std::copy_n(b.get(), b.size(), res.get_write() + len);\n            len += b.size();\n        }\n        return res;\n    };\n\n    auto bufs_m = merge_buffers(bufs);\n\n    for (size_t off = 1; off < total_len; off++) {\n        std::vector<temporary_buffer<char>> copy_of_bufs;\n        copy_of_bufs.reserve(bufs.size());\n        for (auto& b : bufs) {\n            copy_of_bufs.emplace_back(b.get(), b.size());\n        }\n\n        auto res = internal::detach_front(copy_of_bufs, off);\n\n        fmt::print(\"/{} -> {}/{}:\", off, res.size(), copy_of_bufs.size());\n        show_buffers(res);\n        fmt::print(\" +\");\n        show_buffers(copy_of_bufs);\n        fmt::print(\"\\n\");\n\n        for (auto& b : copy_of_bufs) {\n            res.emplace_back(std::move(b));\n        }\n\n        auto res_m = merge_buffers(res);\n        BOOST_REQUIRE(res_m == bufs_m);\n    }\n}\n\nstatic void do_test_detach_buffers(std::vector<size_t> sizes) {\n    for (size_t s : {1, 3, 8}) {\n        sizes.back() = s;\n        do_test_detach_buffers(sizes.size(), sizes);\n\n        if (sizes.size() < 4) {\n            auto copy_of_sizes = sizes;\n            copy_of_sizes.emplace_back(0);\n            do_test_detach_buffers(std::move(copy_of_sizes));\n        }\n    }\n}\n\nBOOST_AUTO_TEST_CASE(test_detach_buffers) {\n    do_test_detach_buffers({0});\n}\n\nBOOST_AUTO_TEST_CASE(test_iovec_trim_front) {\n    const char* data = \"abcdefghijklmno\";\n\n    for (size_t l = 0; l < 17; l++) {\n        std::vector<iovec> iovs;\n        iovs.push_back(iovec{ (void*)(data + 0), 5 });\n        iovs.push_back(iovec{ (void*)(data + 5), 3 });\n        iovs.push_back(iovec{ (void*)(data + 8), 7 });\n        auto res = internal::iovec_trim_front(std::span(iovs), l);\n        if (l >= 15) {\n            BOOST_REQUIRE(res.empty());\n        } else {\n            BOOST_REQUIRE(res.size() > 0);\n            BOOST_REQUIRE_EQUAL(*reinterpret_cast<char*>(res[0].iov_base), data[l]);\n            BOOST_REQUIRE_NE(res[0].iov_len, 0);\n            size_t total = std::accumulate(res.begin(), res.end(), size_t(0), [] (size_t s, const auto& b) { return s + b.iov_len; });\n            BOOST_REQUIRE_EQUAL(total, 15 - l);\n            if (res.size() > 1) {\n                BOOST_REQUIRE_EQUAL(*reinterpret_cast<char*>(res[1].iov_base), data[l + res[0].iov_len]);\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "tests/unit/test_fixture_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2025 ScyllaDB Ltd.\n */\n\n#include <boost/test/execution_monitor.hpp>\n\n#include <seastar/core/sleep.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/test_fixture.hh>\n\nusing namespace seastar;\nusing namespace seastar::testing;\n\nstruct AsyncTestFixture {\n    bool inited = false;\n    bool destroyed = false;\n\n    ~AsyncTestFixture() {\n        BOOST_REQUIRE(destroyed);\n    }\n    future<> setup() {\n        inited = true;\n        return make_ready_future<>();\n    }\n    future<> teardown() {\n        destroyed = true;\n        return make_ready_future<>();\n    }\n};\n\nSEASTAR_FIXTURE_TEST_CASE(test_single_test_fixture, AsyncTestFixture) {\n    BOOST_REQUIRE(inited);\n    return make_ready_future<>();\n}\n\nstruct SyncTestFixture {\n    bool inited = false;\n    bool destroyed = false;\n\n    ~SyncTestFixture() {\n        BOOST_REQUIRE(destroyed);\n    }\n    void setup() {\n        inited = true;\n    }\n    void teardown() {\n        destroyed = true;\n    }\n};\n\nusing namespace std::chrono_literals;\n\nSEASTAR_FIXTURE_TEST_CASE(test_single_test_fixture_void_ret, SyncTestFixture) {\n    BOOST_REQUIRE(inited);\n    return make_ready_future<>();\n}\n\nSEASTAR_FIXTURE_THREAD_TEST_CASE(test_single_thread_test_fixture_void_ret, SyncTestFixture) {\n    BOOST_REQUIRE(inited);\n    seastar::sleep(1ms).get();\n}\n\nstruct ThreadTestFixture {\n    bool inited = false;\n    bool destroyed = false;\n\n    ~ThreadTestFixture() {\n        BOOST_REQUIRE(destroyed);\n    }\n    void setup() {\n        seastar::sleep(1ms).get();\n        inited = true;\n    }\n    void teardown() {\n        seastar::sleep(1ms).get();\n        destroyed = true;\n    }\n};\n\nSEASTAR_FIXTURE_THREAD_TEST_CASE(test_single_thread_test_fixture_thread_fixt, ThreadTestFixture) {\n    BOOST_REQUIRE(inited);\n    seastar::sleep(1ms).get();\n}\n\n// having these thread local subtly verifies that the fixture\n// is run on the proper shard.\nstatic thread_local int num_shared_test_fixts_setup = 0;\nstatic thread_local int num_shared_test_fixts_teardown = 0;\nstatic thread_local std::string shared_test_fixts_string;\n\nstruct SharedTestFixture {\n    SharedTestFixture()\n    {}\n    SharedTestFixture(const std::string& s)\n    {\n        shared_test_fixts_string = s;\n    }\n    future<> setup() {\n        ++num_shared_test_fixts_setup;\n        return make_ready_future<>();\n    }\n    future<> teardown() {\n        ++num_shared_test_fixts_teardown;\n        shared_test_fixts_string = {};\n        return make_ready_future<>();\n    }\n};\n\nBOOST_AUTO_TEST_SUITE(shared_fixtures,\n    *async_fixture<SharedTestFixture>()\n    *async_fixture<SharedTestFixture>(\"los lobos\")\n    *async_fixture(\n        [] { ++num_shared_test_fixts_setup; return make_ready_future<>(); },\n        [] { ++num_shared_test_fixts_teardown; return make_ready_future<>(); }\n    )\n)\n\nSEASTAR_TEST_CASE(test_shared_fixture1) {\n    BOOST_REQUIRE(num_shared_test_fixts_setup == 3);\n    BOOST_REQUIRE(num_shared_test_fixts_teardown == 0);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_shared_fixture2) {\n    BOOST_REQUIRE(num_shared_test_fixts_setup == 3);\n    BOOST_REQUIRE(num_shared_test_fixts_teardown == 0);\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(test_shared_fixture_init_value) {\n    BOOST_REQUIRE(shared_test_fixts_string != \"\");\n    BOOST_TEST_MESSAGE(shared_test_fixts_string);\n    return make_ready_future<>();\n}\n\nBOOST_AUTO_TEST_SUITE_END()\n\nstatic bool do_throw = false;\n\n// Dummy test that on its own does nothing.\nSEASTAR_TEST_CASE(test_nested_throw_helper) {\n    if (std::exchange(do_throw, false)) {\n        try {\n            throw std::invalid_argument(\"Message 1\");\n        } catch (...) {\n            std::throw_with_nested(std::out_of_range(\"Message 2\"));\n        }\n    }\n    return make_ready_future<>();\n}\n\n// Cannot be a SEASTAR_TEST_CASE, because of recursion of seastar::test::run.\nBOOST_AUTO_TEST_CASE(test_nested_throw) {\n    // make helper throw its nested test failure\n    do_throw = true;\n    try {\n        // fake \"normal\" test invoke. This will push the test into the main\n        // runner etc. The exception will be thrown.\n        const_cast<test_nested_throw_helper&>(test_nested_throw_helper_instance).run();\n    } catch (boost::execution_exception& e) {\n        // Should get a cpp exception failure\n        BOOST_REQUIRE_EQUAL(e.code(), boost::execution_exception::error_code::cpp_exception_error);\n        // Should have the nested message\n        BOOST_REQUIRE(e.what().find(\"Message 1\") != std::string::npos);\n        // Should have the outer message\n        BOOST_REQUIRE(e.what().find(\"Message 2\") != std::string::npos);\n    }\n}\n\n"
  },
  {
    "path": "tests/unit/thread_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <seastar/core/thread.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/semaphore.hh>\n#include <seastar/core/do_with.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/assert.hh>\n#include <sys/mman.h>\n#include <signal.h>\n\n#include <valgrind/valgrind.h>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\nSEASTAR_TEST_CASE(test_thread_1) {\n    return do_with(sstring(), [] (sstring& x) {\n        auto t1 = new thread([&x] {\n            x = \"abc\";\n        });\n        return t1->join().then([&x, t1] {\n            BOOST_REQUIRE_EQUAL(x, \"abc\");\n            delete t1;\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_thread_2) {\n    struct tmp {\n        std::vector<thread> threads;\n        semaphore sem1{0};\n        semaphore sem2{0};\n        int counter = 0;\n        void thread_fn() {\n            sem1.wait(1).get();\n            ++counter;\n            sem2.signal(1);\n        }\n    };\n    return do_with(tmp(), [] (tmp& x) {\n        auto n = 10;\n        for (int i = 0; i < n; ++i) {\n            x.threads.emplace_back(std::bind(&tmp::thread_fn, &x));\n        }\n        BOOST_REQUIRE_EQUAL(x.counter, 0);\n        x.sem1.signal(n);\n        return x.sem2.wait(n).then([&x, n] {\n            BOOST_REQUIRE_EQUAL(x.counter, n);\n            return parallel_for_each(x.threads.begin(), x.threads.end(), std::mem_fn(&thread::join));\n        });\n    });\n}\n\nSEASTAR_TEST_CASE(test_thread_async) {\n    sstring x = \"x\";\n    sstring y = \"y\";\n    auto concat = [] (sstring x, sstring y) {\n        sleep(10ms).get();\n        return x + y;\n    };\n    return async(concat, x, y).then([] (sstring xy) {\n        BOOST_REQUIRE_EQUAL(xy, \"xy\");\n    });\n}\n\nSEASTAR_TEST_CASE(test_thread_async_immed) {\n    return async([] { return 3; }).then([] (int three) {\n        BOOST_REQUIRE_EQUAL(three, 3);\n    });\n}\n\nSEASTAR_TEST_CASE(test_thread_async_nested) {\n    return async([] {\n        return async([] {\n            return 3;\n        }).get();\n    }).then([] (int three) {\n        BOOST_REQUIRE_EQUAL(three, 3);\n    });\n}\n\nvoid compute(float& result, bool& done, uint64_t& ctr) {\n    while (!done) {\n        for (int n = 0; n < 10000; ++n) {\n            result += 1 / (result + 1);\n            ++ctr;\n        }\n        thread::yield();\n    }\n}\n\n#if defined(SEASTAR_ASAN_ENABLED) && defined(SEASTAR_HAVE_ASAN_FIBER_SUPPORT)\nvolatile int force_write;\nvolatile void* shut_up_gcc;\n\n[[gnu::noinline]]\nvoid throw_exception() {\n    volatile char buf[1024];\n    shut_up_gcc = &buf;\n    for (int i = 0; i < 1024; i++) {\n        buf[i] = force_write;\n    }\n    throw 1;\n}\n\n[[gnu::noinline]]\nvoid use_stack() {\n    volatile char buf[2 * 1024];\n    shut_up_gcc = &buf;\n    for (int i = 0; i < 2 * 1024; i++) {\n        buf[i] = force_write;\n    }\n}\n\nSEASTAR_TEST_CASE(test_asan_false_positive) {\n    return async([] {\n        try {\n            throw_exception();\n        } catch (...) {\n            use_stack();\n        }\n    });\n}\n#endif\n\nSEASTAR_THREAD_TEST_CASE(abc, *boost::unit_test::expected_failures(2)) {\n    BOOST_TEST(false);\n    BOOST_TEST(false);\n}\n\nSEASTAR_TEST_CASE(test_thread_custom_stack_size) {\n    sstring x = \"x\";\n    sstring y = \"y\";\n    auto concat = [] (sstring x, sstring y) {\n        sleep(10ms).get();\n        return x + y;\n    };\n    thread_attributes attr;\n    attr.stack_size = 16384;\n    return async(attr, concat, x, y).then([] (sstring xy) {\n        BOOST_REQUIRE_EQUAL(xy, \"xy\");\n    });\n}\n\n// The test case uses x86_64 specific signal handler info. The test\n// fails with detect_stack_use_after_return=1. We could put it behind\n// a command line option and fork/exec to run it after removing\n// detect_stack_use_after_return=1 from the environment.\n#if defined(SEASTAR_THREAD_STACK_GUARDS) && defined(__x86_64__) && !defined(SEASTAR_ASAN_ENABLED)\nstruct test_thread_custom_stack_size_failure : public seastar::testing::seastar_test {\n    using seastar::testing::seastar_test::seastar_test;\n    seastar::future<> run_test_case() const override;\n};\n\nstatic test_thread_custom_stack_size_failure test_thread_custom_stack_size_failure_instance(\n    \"test_thread_custom_stack_size_failure\",\n    __FILE__, __LINE__);\nstatic thread_local volatile bool stack_guard_bypassed = false;\n\nstatic int get_mprotect_flags(void* ctx) {\n    int flags;\n    ucontext_t* context = reinterpret_cast<ucontext_t*>(ctx);\n    if (context->uc_mcontext.gregs[REG_ERR] & 0x2) {\n        flags = PROT_READ | PROT_WRITE;\n    } else {\n        flags = PROT_READ;\n    }\n    return flags;\n}\n\nstatic void* pagealign(void* ptr, size_t page_size) {\n    static const int pageshift = ffs(page_size) - 1;\n    return reinterpret_cast<void*>(((reinterpret_cast<intptr_t>((ptr)) >> pageshift) << pageshift));\n}\n\nstatic thread_local struct sigaction default_old_sigsegv_handler;\n\nstatic void bypass_stack_guard(int sig, siginfo_t* si, void* ctx) {\n    SEASTAR_ASSERT(sig == SIGSEGV);\n    int flags = get_mprotect_flags(ctx);\n    stack_guard_bypassed = (flags & PROT_WRITE);\n    if (!stack_guard_bypassed) {\n        return;\n    }\n    size_t page_size = getpagesize();\n    auto mp_result = mprotect(pagealign(si->si_addr, page_size), page_size, PROT_READ | PROT_WRITE);\n    SEASTAR_ASSERT(mp_result == 0);\n}\n\n// This test will fail with a regular stack size, because we only probe\n// around 10KiB of data, and the stack guard resides after 128'th KiB.\nseastar::future<> test_thread_custom_stack_size_failure::run_test_case() const {\n    if (RUNNING_ON_VALGRIND) {\n        return make_ready_future<>();\n    }\n\n    sstring x = \"x\";\n    sstring y = \"y\";\n\n    // Catch segmentation fault once:\n    struct sigaction sa{};\n    sa.sa_sigaction = &bypass_stack_guard;\n    sa.sa_flags = SA_SIGINFO;\n    auto ret = sigaction(SIGSEGV, &sa, &default_old_sigsegv_handler);\n    if (ret) {\n        throw std::system_error(ret, std::system_category());\n    }\n\n    auto concat = [] (sstring x, sstring y) {\n        sleep(10ms).get();\n        // Probe the stack by writing to it in intervals of 1024,\n        // until we hit a write fault. In order not to ruin anything,\n        // the \"write\" uses data it just read from the address.\n        volatile char* mem = reinterpret_cast<volatile char*>(&x);\n        for (int i = 0; i < 20; ++i) {\n            mem[i*-1024] = char(mem[i*-1024]);\n            if (stack_guard_bypassed) {\n                break;\n            }\n        }\n        return x + y;\n    };\n    thread_attributes attr;\n    attr.stack_size = 16384;\n    return async(attr, concat, x, y).then([] (sstring xy) {\n        BOOST_REQUIRE_EQUAL(xy, \"xy\");\n        BOOST_REQUIRE(stack_guard_bypassed);\n        auto ret = sigaction(SIGSEGV, &default_old_sigsegv_handler, nullptr);\n        if (ret) {\n            throw std::system_error(ret, std::system_category());\n        }\n    }).then([concat, x, y] {\n        // The same function with a default stack will not trigger\n        // a segfault, because its stack is much bigger than 10KiB\n        return async(concat, x, y).then([] (sstring xy) {\n            BOOST_REQUIRE_EQUAL(xy, \"xy\");\n        });\n    });\n}\n#endif // SEASTAR_THREAD_STACK_GUARDS && __x86_64__\n"
  },
  {
    "path": "tests/unit/timer_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2014 Cloudius Systems, Ltd.\n */\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n\n#include <seastar/core/timer.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/sleep.hh>\n#include <seastar/util/later.hh>\n#include <chrono>\n#include <iostream>\n\nusing namespace seastar;\nusing namespace std::chrono_literals;\n\ntemplate <typename Clock>\nvoid test_timer_basic() {\n    timer<Clock> t1;\n    timer<Clock> t2;\n    timer<Clock> t3;\n    timer<Clock> t4;\n    timer<Clock> t5;\n    promise<> pr1;\n\n    t1.set_callback([&] {\n        fmt::print(\" 500ms timer expired\\n\");\n        BOOST_REQUIRE(t4.cancel());\n        BOOST_REQUIRE(t5.cancel());\n        t5.arm(1100ms);\n    });\n    t2.set_callback([] { fmt::print(\" 900ms timer expired\\n\"); });\n    t3.set_callback([] { fmt::print(\"1000ms timer expired\\n\"); });\n    t4.set_callback([] { BOOST_FAIL(\"cancelled timer expired\\n\"); });\n    t5.set_callback([&pr1] { fmt::print(\"1600ms rearmed timer expired\\n\"); pr1.set_value(); });\n\n    t1.arm(500ms);\n    t2.arm(900ms);\n    t3.arm(1000ms);\n    t4.arm(700ms);\n    t5.arm(800ms);\n\n    pr1.get_future().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_basic_steady) {\n    test_timer_basic<steady_clock_type>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_basic_lowres) {\n    test_timer_basic<lowres_clock>();\n}\n\ntemplate <typename Clock>\nvoid test_timer_cancelling() {\n    promise<> pr2;\n\n    timer<Clock> t1;\n    t1.set_callback([] { BOOST_FAIL(\"canceled timer expired\"); });\n    t1.arm(100ms);\n    t1.cancel();\n\n    t1.arm(100ms);\n    t1.cancel();\n\n    t1.set_callback([&pr2] { pr2.set_value(); });\n    t1.arm(100ms);\n\n    pr2.get_future().get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_cancelling_steady) {\n    test_timer_cancelling<steady_clock_type>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_cancelling_lowres) {\n    test_timer_cancelling<lowres_clock>();\n}\n\ntemplate <typename Clock>\nvoid test_timer_with_scheduling_groups() {\n    auto sg1 = create_scheduling_group(\"sg1\", 100).get();\n    auto sg2 = create_scheduling_group(\"sg2\", 100).get();\n    thread_attributes t1attr;\n    t1attr.sched_group = sg1;\n    auto expirations = 0;\n    async(t1attr, [&] {\n        auto make_callback_checking_sg = [&] (scheduling_group sg_to_check) {\n            return [sg_to_check, &expirations] {\n                ++expirations;\n                BOOST_REQUIRE(current_scheduling_group() == sg_to_check);\n            };\n        };\n        timer<Clock> t1(make_callback_checking_sg(sg1));\n        t1.arm(10ms);\n        timer<Clock> t2(sg2, make_callback_checking_sg(sg2));\n        t2.arm(10ms);\n        sleep(500ms).get();\n        BOOST_REQUIRE_EQUAL(expirations, 2);\n    }).get();\n    destroy_scheduling_group(sg1).get();\n    destroy_scheduling_group(sg2).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_with_scheduling_groups_steady) {\n    test_timer_with_scheduling_groups<steady_clock_type>();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_timer_with_scheduling_groups_lowres) {\n    test_timer_with_scheduling_groups<lowres_clock>();\n}\n\n// Regression test for #3013\nSEASTAR_THREAD_TEST_CASE(test_highres_periodic_cancel_in_callback) {\n    timer<steady_clock_type> t1;\n    timer<steady_clock_type> t2;\n    bool t1_fired = false;\n    bool t2_fired = false;\n\n    t1.set_callback([&] {\n        t1_fired = true;\n        t1.cancel();\n        (void)yield().then([&] {\n            t2.arm(10ms);\n        });\n    });\n\n    t2.set_callback([&] {\n        t2_fired = true;\n    });\n\n    t1.arm_periodic(5ms);\n\n    auto end = std::chrono::steady_clock::now() + 1s;\n    do {\n        yield().get();\n    } while (std::chrono::steady_clock::now() < end);\n\n    BOOST_REQUIRE(t1_fired);\n    BOOST_REQUIRE(t2_fired);\n}\n"
  },
  {
    "path": "tests/unit/tl-generator.hh",
    "content": "///\n// generator - Single-header, ranges-compatible generator type built\n// on C++20 coroutines\n// Written in 2021 by Sy Brand (tartanllama@gmail.com, @TartanLlama)\n//\n// Documentation available at https://tl.tartanllama.xyz/\n//\n// To the extent possible under law, the author(s) have dedicated all\n// copyright and related and neighboring rights to this software to the\n// public domain worldwide. This software is distributed without any warranty.\n//\n// You should have received a copy of the CC0 Public Domain Dedication\n// along with this software. If not, see\n// <http://creativecommons.org/publicdomain/zero/1.0/>.\n///\n\n#ifndef TL_GENERATOR_HPP\n#define TL_GENERATOR_HPP\n\n#define TL_GENERATOR_VERSION_MAJOR 0\n#define TL_GENERATOR_VERSION_MINOR 3\n#define TL_GENERATOR_VERSION_PATCH 0\n\n#include <coroutine>\n#include <exception>\n#include <utility>\n#include <type_traits>\n#include <ranges>\n\nnamespace tl {\n   template <class T>\n   class generator {\n      struct promise {\n         using value_type = std::remove_reference_t<T>;\n         using reference_type = value_type&;\n         using pointer_type = value_type*;\n\n         promise() = default;\n\n         generator get_return_object() {\n            return generator(std::coroutine_handle<promise>::from_promise(*this));\n         }\n\n         std::suspend_always initial_suspend() const { return {}; }\n         std::suspend_always final_suspend() const noexcept { return {}; }\n\n         void return_void() const noexcept { return; }\n\n         void unhandled_exception() noexcept {\n            exception_ = std::current_exception();\n         }\n\n         void rethrow_if_exception() {\n            if (exception_) {\n               std::rethrow_exception(exception_);\n            }\n         }\n\n         std::suspend_always yield_value(reference_type v) noexcept {\n            value_ = std::addressof(v);\n            return {};\n         }\n\n         std::exception_ptr exception_;\n         pointer_type value_;\n      };\n\n   public:\n      using promise_type = promise;\n      class sentinel {};\n\n      class iterator {\n         using handle_type = std::coroutine_handle<promise_type>;\n\n      public:\n         using value_type = typename promise_type::value_type;\n         using reference_type = typename promise_type::reference_type;\n         using pointer_type = typename promise_type::pointer_type;\n         using difference_type = std::ptrdiff_t;\n\n         iterator() = default;\n         ~iterator() {\n            if (handle_) handle_.destroy();\n         }\n\n         //Non-copyable because coroutine handles point to a unique resource\n         iterator(iterator const&) = delete;\n         iterator(iterator&& rhs) noexcept : handle_(std::exchange(rhs.handle_, nullptr)) {}\n         iterator& operator=(iterator const&) = delete;\n         iterator& operator=(iterator&& rhs) noexcept {\n            handle_ = std::exchange(rhs.handle_, nullptr);\n            return *this;\n         }\n\n         friend bool operator==(iterator const& it, sentinel) noexcept {\n            return (!it.handle_ || it.handle_.done());\n         }\n\n         iterator& operator++() {\n            handle_.resume();\n            if (handle_.done()) {\n               handle_.promise().rethrow_if_exception();\n            }\n            return *this;\n         }\n\n         void operator++(int) {\n            (void)this->operator++();\n         }\n\n         reference_type operator*() const\n            noexcept(noexcept(std::is_nothrow_copy_constructible_v<reference_type>)){\n            return *handle_.promise().value_;\n         }\n\n      private:\n         friend class generator;\n         iterator(handle_type handle) : handle_(handle) {}\n\n         handle_type handle_;\n      };\n\n      using handle_type = std::coroutine_handle<promise_type>;\n\n      generator() noexcept = default;\n      ~generator() {\n         if (handle_) handle_.destroy();\n      }\n\n      generator(generator const&) = delete;\n      generator(generator&& rhs) noexcept : handle_(std::exchange(rhs.handle_, nullptr)) {}\n      generator& operator=(generator const&) = delete;\n      generator& operator=(generator&& rhs) noexcept {\n         swap(rhs);\n         return *this;\n      }\n\n      iterator begin() {\n         handle_.resume();\n         if (handle_.done()) {\n            handle_.promise().rethrow_if_exception();\n         }\n         return {std::exchange(handle_, nullptr)};\n      }\n\n      sentinel end() const noexcept {\n         return {};\n      }\n\n      void swap(generator& other) noexcept {\n         std::swap(handle_, other.handle_);\n      }\n\n   private:\n      friend class iterator;\n      explicit generator(handle_type handle) noexcept : handle_(handle) {}\n\n      handle_type handle_ = nullptr;\n   };\n}\n\ntemplate<class T>\ninline constexpr bool std::ranges::enable_view<tl::generator<T>> = true;\n\n#endif\n"
  },
  {
    "path": "tests/unit/tls-ca-bundle.pem",
    "content": "-----BEGIN CERTIFICATE-----\nMIIDpDCCAoygAwIBAgIBATANBgkqhkiG9w0BAQUFADBjMQswCQYDVQQGEwJVUzEc\nMBoGA1UEChMTQW1lcmljYSBPbmxpbmUgSW5jLjE2MDQGA1UEAxMtQW1lcmljYSBP\nbmxpbmUgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAxMB4XDTAyMDUyODA2\nMDAwMFoXDTM3MTExOTIwNDMwMFowYzELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE0Ft\nZXJpY2EgT25saW5lIEluYy4xNjA0BgNVBAMTLUFtZXJpY2EgT25saW5lIFJvb3Qg\nQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgMTCCASIwDQYJKoZIhvcNAQEBBQADggEP\nADCCAQoCggEBAKgv6KRpBgNHw+kqmP8ZonCaxlCyfqXfaE0bfA+2l2h9LaaLl+lk\nhsmj76CGv2BlnEtUiMJIxUo5vxTjWVXlGbR0yLQFOVwWpeKVBeASrlmLojNoWBym\n1BW32J/X3HGrfpq/m44zDyL9Hy7nBzbvYjnF3cu6JRQj3gzGPTzOggjmZj7aUTsW\nOqMFf6Dch9Wc/HKpoH145LcxVR5lu9RhsCFg7RAycsWSJR74kEoYeEfffjA3PlAb\n2xzTa5qGUwew76wGePiEmf4hjUyAtgyC9mZweRrTT6PP8c9GsEsPPt2IYriMqQko\nO3rHl+Ee5fSfwMCuJKDIodkP1nsmgmkyPacCAwEAAaNjMGEwDwYDVR0TAQH/BAUw\nAwEB/zAdBgNVHQ4EFgQUAK3Zo/Z59m50qX8zPYEX10zPM94wHwYDVR0jBBgwFoAU\nAK3Zo/Z59m50qX8zPYEX10zPM94wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEB\nBQUAA4IBAQB8itEfGDeC4Liwo+1WlchiYZwFos3CYiZhzRAW18y0ZTTQEYqtqKkF\nZu90821fnZmv9ov761KyBZiibyrFVL0lvV+uyIbqRizBs73B6UlwGBaXCBOMIOAb\nLjpHyx7kADCVW/RFo8AasAFOq73AI25jP4BKxQft3OJvx8Fi8eNy1gTIdGcL+oir\noQHIb/AUr9KZzVGTfu0uOMe9zkZQPXLjeSWdm4grECDdpbgyn43gKd8hdIaC2y+C\nMMbHNYaz+ZZfRtsMRf3zUMNvxsNIrUam4SdHCh0Om7bCd39j8uB9Gr784N/Xx6ds\nsPmuujz9dLQR6FgNgLzTqIA6me11zEZ7\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFpDCCA4ygAwIBAgIBATANBgkqhkiG9w0BAQUFADBjMQswCQYDVQQGEwJVUzEc\nMBoGA1UEChMTQW1lcmljYSBPbmxpbmUgSW5jLjE2MDQGA1UEAxMtQW1lcmljYSBP\nbmxpbmUgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAyMB4XDTAyMDUyODA2\nMDAwMFoXDTM3MDkyOTE0MDgwMFowYzELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE0Ft\nZXJpY2EgT25saW5lIEluYy4xNjA0BgNVBAMTLUFtZXJpY2EgT25saW5lIFJvb3Qg\nQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgMjCCAiIwDQYJKoZIhvcNAQEBBQADggIP\nADCCAgoCggIBAMxBRR3pPU0Q9oyxQcngXssNt79Hc9PwVU3dxgz6sWYFas14tNwC\n206B89enfHG8dWOgXeMHDEjsJcQDIPT/DjsS/5uN4cbVG7RtIuOx238hZK+GvFci\nKtZHgVdEglZTvYYUAQv8f3SkWq7xuhG1m1hagLQ3eAkzfDJHA1zEpYNI9FdWboE2\nJxhP7JsowtS013wMPgwr38oE18aO6lhOqKSlGBxsRZijQdEt0sdtjRnxrXm3gT+9\nBoInLRBYBbV4Bbkv2wxrkJB+FFk4u5QkE+XRnRTf04JNRvCAOVIyD+OEsnpD8l7e\nXz8d3eOyG6ChKiMDbi4BFYdcpnV1x5dhvt6G3NRI270qv0pV2uh9UPu0gBe4lL8B\nPeraunzgWGcXuVjgiIZGZ2ydEEdYMtA1fHkqkKJaEBEjNa0vzORKW6fIJ/KD3l67\nXnfn6KVuY8INXWHQjNJsWiEOyiijzirplcdIz5ZvHZIlyMbGwcEMBawmxNJ10uEq\nZ8A9W6Wa6897GqidFEXlD6CaZd4vKL3Ob5Rmg0gp2OpljK+T2WSfVVcmv2/LNzGZ\no2C7HK2JNDJiuEMhBnIMoVxtRsX6Kc8w3onccVvdtjc+31D1uAclJuW8tf48ArO3\n+L5DwYcRlJ4jbBeKuIonDFRH8KmzwICMoCfrHRnjB453cMor9H124HhnAgMBAAGj\nYzBhMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFE1FwWg4u3OpaaEg5+31IqEj\nFNeeMB8GA1UdIwQYMBaAFE1FwWg4u3OpaaEg5+31IqEjFNeeMA4GA1UdDwEB/wQE\nAwIBhjANBgkqhkiG9w0BAQUFAAOCAgEAZ2sGuV9FOypLM7PmG2tZTiLMubekJcmn\nxPBUlgtk87FYT15R/LKXeydlwuXK5w0MJXti4/qftIe3RUavg6WXSIylvfEWK5t2\nLHo1YGwRgJfMqZJS5ivmae2p+DYtLHe/YUjRYwu5W1LtGLBDQiKmsXeu3mnFzccc\nobGlHBD7GL4acN3Bkku+KVqdPzW+5X1R+FXgJXUjhx5c3LqdsKyzadsXg8n33gy8\nCNyRnqjQ1xU3c6U1uPx+xURABsPr+CKAXEfOAuMRn0T//ZoyzH1kUQ7rVyZ2OuMe\nIjzCpjbdGe+n/BLzJsBZMYVMnNjP36TMzCmT/5RtdlwTCJfy7aULTd3oyWgOZtMA\nDjMSW7yV5TKQqLPGbIOtd+6Lfn6xqavT4fG2wLHqiMDn05DpKJKUe2h7lyoKZy2F\nAjgQ5ANh1NolNscIWC2hp1GvMApJ9aZphwctREZ2jirlmjvXGKL8nDgQzMY70rUX\nOm/9riW99XJZZLF0KjhfGEzfz3EEWjbUvy+ZnOjZurGV5gJLIaFb1cFPj65pbVPb\nAZO1XB4Y3WRayhgoPmMEEf0cjQAPuDffZ4qdZqkCapH/E8ovXYO8h5Ns3CRRFgQl\nZvqz2cK6Kb6aSDiCmfS/O0oxGfm/jiEzFMpPVF/7zvuPcX/9XhmgD0uRuMRUvAaw\nRY8mkaKO/qk=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIE2DCCBEGgAwIBAgIEN0rSQzANBgkqhkiG9w0BAQUFADCBwzELMAkGA1UEBhMC\nVVMxFDASBgNVBAoTC0VudHJ1c3QubmV0MTswOQYDVQQLEzJ3d3cuZW50cnVzdC5u\nZXQvQ1BTIGluY29ycC4gYnkgcmVmLiAobGltaXRzIGxpYWIuKTElMCMGA1UECxMc\nKGMpIDE5OTkgRW50cnVzdC5uZXQgTGltaXRlZDE6MDgGA1UEAxMxRW50cnVzdC5u\nZXQgU2VjdXJlIFNlcnZlciBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05OTA1\nMjUxNjA5NDBaFw0xOTA1MjUxNjM5NDBaMIHDMQswCQYDVQQGEwJVUzEUMBIGA1UE\nChMLRW50cnVzdC5uZXQxOzA5BgNVBAsTMnd3dy5lbnRydXN0Lm5ldC9DUFMgaW5j\nb3JwLiBieSByZWYuIChsaW1pdHMgbGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBF\nbnRydXN0Lm5ldCBMaW1pdGVkMTowOAYDVQQDEzFFbnRydXN0Lm5ldCBTZWN1cmUg\nU2VydmVyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGdMA0GCSqGSIb3DQEBAQUA\nA4GLADCBhwKBgQDNKIM0VBuJ8w+vN5Ex/68xYMmo6LIQaO2f55M28Qpku0f1BBc/\nI0dNxScZgSYMVHINiC3ZH5oSn7yzcdOAGT9HZnuMNSjSuQrfJNqc1lB5gXpa0zf3\nwkrYKZImZNHkmGw6AIr1NJtl+O3jEP/9uElY3KDegjlrgbEWGWG5VLbmQwIBA6OC\nAdcwggHTMBEGCWCGSAGG+EIBAQQEAwIABzCCARkGA1UdHwSCARAwggEMMIHeoIHb\noIHYpIHVMIHSMQswCQYDVQQGEwJVUzEUMBIGA1UEChMLRW50cnVzdC5uZXQxOzA5\nBgNVBAsTMnd3dy5lbnRydXN0Lm5ldC9DUFMgaW5jb3JwLiBieSByZWYuIChsaW1p\ndHMgbGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVk\nMTowOAYDVQQDEzFFbnRydXN0Lm5ldCBTZWN1cmUgU2VydmVyIENlcnRpZmljYXRp\nb24gQXV0aG9yaXR5MQ0wCwYDVQQDEwRDUkwxMCmgJ6AlhiNodHRwOi8vd3d3LmVu\ndHJ1c3QubmV0L0NSTC9uZXQxLmNybDArBgNVHRAEJDAigA8xOTk5MDUyNTE2MDk0\nMFqBDzIwMTkwNTI1MTYwOTQwWjALBgNVHQ8EBAMCAQYwHwYDVR0jBBgwFoAU8Bdi\nE1U9s/8KAGv7UISX8+1i0BowHQYDVR0OBBYEFPAXYhNVPbP/CgBr+1CEl/PtYtAa\nMAwGA1UdEwQFMAMBAf8wGQYJKoZIhvZ9B0EABAwwChsEVjQuMAMCBJAwDQYJKoZI\nhvcNAQEFBQADgYEAkNwwAvpkdMKnCqV8IY00F6j7Rw7/JXyNEwr75Ji174z4xRAN\n95K+8cPV1ZVqBLssziY2ZcgxxufuP+NXdYR6Ee9GTxj005i7qIcyunL2POI9n9cd\n2cNgQ4xYDiKWL2KjLB+6rQXvqzJ4h6BUcxm1XAX5Uj5tLUUL9wqT6u0G+bI=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICgjCCAeugAwIBAgIBBDANBgkqhkiG9w0BAQQFADBTMQswCQYDVQQGEwJVUzEc\nMBoGA1UEChMTRXF1aWZheCBTZWN1cmUgSW5jLjEmMCQGA1UEAxMdRXF1aWZheCBT\nZWN1cmUgZUJ1c2luZXNzIENBLTEwHhcNOTkwNjIxMDQwMDAwWhcNMjAwNjIxMDQw\nMDAwWjBTMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTRXF1aWZheCBTZWN1cmUgSW5j\nLjEmMCQGA1UEAxMdRXF1aWZheCBTZWN1cmUgZUJ1c2luZXNzIENBLTEwgZ8wDQYJ\nKoZIhvcNAQEBBQADgY0AMIGJAoGBAM4vGbwXt3fek6lfWg0XTzQaDJj0ItlZ1MRo\nRvC0NcWFAyDGr0WlIVFFQesWWDYyb+JQYmT5/VGcqiTZ9J2DKocKIdMSODRsjQBu\nWqDZQu4aIZX5UkxVWsUPOE9G+m34LjXWHXzr4vCwdYDIqROsvojvOm6rXyo4YgKw\nEnv+j6YDAgMBAAGjZjBkMBEGCWCGSAGG+EIBAQQEAwIABzAPBgNVHRMBAf8EBTAD\nAQH/MB8GA1UdIwQYMBaAFEp4MlIR21kWNl7fwRQ2QGpHfEyhMB0GA1UdDgQWBBRK\neDJSEdtZFjZe38EUNkBqR3xMoTANBgkqhkiG9w0BAQQFAAOBgQB1W6ibAxHm6VZM\nzfmpTMANmvPMZWnmJXbMWbfWVMMdzZmsGd20hdXgPfxiIKeES1hl8eL5lSE/9dR+\nWB5Hh1Q+WKG1tfgq73HnvMP2sUlG4tega+VWeponmHxGYhTnyfxuAxJ5gDgdSIKN\n/Bf+KpYrtWKmpj29f5JZzVoqgrI3eQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYD\nVQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNv\nbHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJv\nb3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEzMjM1OTAwWjB1MQswCQYDVQQGEwJV\nUzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU\ncnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds\nb2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrH\niM3dFw4usJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTS\nr41tiGeA5u2ylc9yMcqlHHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X4\n04Wqk2kmhXBIgD8SFcd5tB8FLztimQIDAQABMA0GCSqGSIb3DQEBBAUAA4GBAG3r\nGwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMWM4ETCJ57NE7fQMh017l9\n3PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OFNMQkpw0P\nlZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIC5zCCAlACAQEwDQYJKoZIhvcNAQEFBQAwgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0\nIFZhbGlkYXRpb24gTmV0d29yazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAz\nBgNVBAsTLFZhbGlDZXJ0IENsYXNzIDMgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9y\naXR5MSEwHwYDVQQDExhodHRwOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG\n9w0BCQEWEWluZm9AdmFsaWNlcnQuY29tMB4XDTk5MDYyNjAwMjIzM1oXDTE5MDYy\nNjAwMjIzM1owgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0IFZhbGlkYXRpb24gTmV0d29y\nazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAzBgNVBAsTLFZhbGlDZXJ0IENs\nYXNzIDMgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9yaXR5MSEwHwYDVQQDExhodHRw\nOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG9w0BCQEWEWluZm9AdmFsaWNl\ncnQuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDjmFGWHOjVsQaBalfD\ncnWTq8+epvzzFlLWLU2fNUSoLgRNB0mKOCn1dzfnt6td3zZxFJmP3MKS8edgkpfs\n2Ejcv8ECIMYkpChMMFp2bbFc893enhBxoYjHW5tBbcqwuI4V7q0zK89HBFx1cQqY\nJJgpp0lZpd34t0NiYfPT4tBVPwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBAFa7AliE\nZwgs3x/be0kz9dNnnfS0ChCzycUs4pJqcXgn8nCDQtM+z6lU9PHYkhaM0QTLS6vJ\nn0WuPIqpsHEzXcjFV9+vqDWzf4mH6eglkrh/hXqu1rweN1gqZ8mRzyqBPu3GOd/A\nPhmcGcwTTYJBtYze4D1gCCAPRX5ron+jjBXu\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDJzCCApCgAwIBAgIBATANBgkqhkiG9w0BAQQFADCBzjELMAkGA1UEBhMCWkEx\nFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD\nVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv\nbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhhd3RlIFByZW1pdW0gU2Vy\ndmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNlcnZlckB0aGF3dGUuY29t\nMB4XDTk2MDgwMTAwMDAwMFoXDTIwMTIzMTIzNTk1OVowgc4xCzAJBgNVBAYTAlpB\nMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEdMBsG\nA1UEChMUVGhhd3RlIENvbnN1bHRpbmcgY2MxKDAmBgNVBAsTH0NlcnRpZmljYXRp\nb24gU2VydmljZXMgRGl2aXNpb24xITAfBgNVBAMTGFRoYXd0ZSBQcmVtaXVtIFNl\ncnZlciBDQTEoMCYGCSqGSIb3DQEJARYZcHJlbWl1bS1zZXJ2ZXJAdGhhd3RlLmNv\nbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0jY2aovXwlue2oFBYo847kkE\nVdbQ7xwblRZH7xhINTpS9CtqBo87L+pW46+GjZ4X9560ZXUCTe/LCaIhUdib0GfQ\nug2SBhRz1JPLlyoAnFxODLz6FVL88kRu2hFKbgifLy3j+ao6hnO2RlNYyIkFvYMR\nuHM/qgeN9EJN50CdHDcCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG\n9w0BAQQFAAOBgQAmSCwWwlj66BZ0DKqqX1Q/8tfJeGBeXm43YyJ3Nn6yF8Q0ufUI\nhfzJATj/Tb7yFkJD57taRvvBxhEf8UqwKEbJw8RCfbz6q1lu1bdRiBHjpIUZa4JM\npAwSremkrj/xw0llmozFyD4lt5SZu5IycQfwhl7tUCemDaYj+bvLpgcUQg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDEzCCAnygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBxDELMAkGA1UEBhMCWkEx\nFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD\nVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv\nbiBTZXJ2aWNlcyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEm\nMCQGCSqGSIb3DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wHhcNOTYwODAx\nMDAwMDAwWhcNMjAxMjMxMjM1OTU5WjCBxDELMAkGA1UEBhMCWkExFTATBgNVBAgT\nDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYDVQQKExRUaGF3\ndGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl\ncyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEmMCQGCSqGSIb3\nDQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQAD\ngY0AMIGJAoGBANOkUG7I/1Zr5s9dtuoMaHVHoqrC2oQl/Kj0R1HahbUgdJSGHg91\nyekIYfUGbTBuFRkC6VLAYttNmZ7iagxEOM3+vuNkCXDF/rFrKbYvScg71CcEJRCX\nL+eQbcAoQpnXTEPew/UhbVSfXcNY4cDk2VuwuNy0e982OsK1ZiIS1ocNAgMBAAGj\nEzARMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEEBQADgYEAB/pMaVz7lcxG\n7oWDTSEwjsrZqG9JGubaUeNgcGyEYRGhGshIPllDfU+VPaGLtwtimHp1it2ITk6e\nQNuozDJ0uW8NxuOzRAvZim+aKZuZGCg70eNAKJpaPNW15yAbi8qkq43pUdniTCxZ\nqdq5snUb9kLy78fyGPmJvKP/iiMucEc=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIC5zCCAlACAQEwDQYJKoZIhvcNAQEFBQAwgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0\nIFZhbGlkYXRpb24gTmV0d29yazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAz\nBgNVBAsTLFZhbGlDZXJ0IENsYXNzIDEgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9y\naXR5MSEwHwYDVQQDExhodHRwOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG\n9w0BCQEWEWluZm9AdmFsaWNlcnQuY29tMB4XDTk5MDYyNTIyMjM0OFoXDTE5MDYy\nNTIyMjM0OFowgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0IFZhbGlkYXRpb24gTmV0d29y\nazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAzBgNVBAsTLFZhbGlDZXJ0IENs\nYXNzIDEgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9yaXR5MSEwHwYDVQQDExhodHRw\nOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG9w0BCQEWEWluZm9AdmFsaWNl\ncnQuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDYWYJ6ibiWuqYvaG9Y\nLqdUHAZu9OqNSLwxlBfw8068srg1knaw0KWlAdcAAxIiGQj4/xEjm84H9b9pGib+\nTunRf50sQB1ZaG6m+FiwnRqP0z/x3BkGgagO4DrdyFNFCQbmD3DD+kCmDuJWBQ8Y\nTfwggtFzVXSNdnKgHZ0dwN0/cQIDAQABMA0GCSqGSIb3DQEBBQUAA4GBAFBoPUn0\nLBwGlN+VYH+Wexf+T3GtZMjdd9LvWVXoP+iOBSoh8gfStadS/pyxtuJbdxdA6nLW\nI8sogTLDAHkY7FkXicnGah5xyf23dKUlRWnFSKsZ4UWKJWsZ7uW7EvV/96aNUcPw\nnXS3qT6gpf+2SQMT2iLM7XGCK5nPOrf1LXLI\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIC5zCCAlACAQEwDQYJKoZIhvcNAQEFBQAwgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0\nIFZhbGlkYXRpb24gTmV0d29yazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAz\nBgNVBAsTLFZhbGlDZXJ0IENsYXNzIDIgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9y\naXR5MSEwHwYDVQQDExhodHRwOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG\n9w0BCQEWEWluZm9AdmFsaWNlcnQuY29tMB4XDTk5MDYyNjAwMTk1NFoXDTE5MDYy\nNjAwMTk1NFowgbsxJDAiBgNVBAcTG1ZhbGlDZXJ0IFZhbGlkYXRpb24gTmV0d29y\nazEXMBUGA1UEChMOVmFsaUNlcnQsIEluYy4xNTAzBgNVBAsTLFZhbGlDZXJ0IENs\nYXNzIDIgUG9saWN5IFZhbGlkYXRpb24gQXV0aG9yaXR5MSEwHwYDVQQDExhodHRw\nOi8vd3d3LnZhbGljZXJ0LmNvbS8xIDAeBgkqhkiG9w0BCQEWEWluZm9AdmFsaWNl\ncnQuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOOnHK5avIWZJV16vY\ndA757tn2VUdZZUcOBVXc65g2PFxTXdMwzzjsvUGJ7SVCCSRrCl6zfN1SLUzm1NZ9\nWlmpZdRJEy0kTRxQb7XBhVQ7/nHk01xC+YDgkRoKWzk2Z/M/VXwbP7RfZHM047QS\nv4dk+NoS/zcnwbNDu+97bi5p9wIDAQABMA0GCSqGSIb3DQEBBQUAA4GBADt/UG9v\nUJSZSWI4OB9L+KXIPqeCgfYrx+jFzug6EILLGACOTb2oWH+heQC1u+mNr0HZDzTu\nIYEZoDJJKPTEjlbVUjP9UNV+mWwD5MlM/Mtsq2azSiGM5bUMMj4QssxsodyamEwC\nW/POuZ6lcg5Ktz885hZo+L7tdEy8W9ViH0Pd\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICPDCCAaUCEHC65B0Q2Sk0tjjKewPMur8wDQYJKoZIhvcNAQECBQAwXzELMAkG\nA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz\ncyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2\nMDEyOTAwMDAwMFoXDTI4MDgwMTIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV\nBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmlt\nYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN\nADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhE\nBarsAx94f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/is\nI19wKTakyYbnsZogy1Olhec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0G\nCSqGSIb3DQEBAgUAA4GBALtMEivPLCYATxQT3ab7/AoRhIzzKBxnki98tsX63/Do\nlbwdj2wsqFHMc9ikwFPwTtYmwHYBV4GSXiHx0bH/59AhWM1pF+NEHJwZRDmJXNyc\nAA9WjQKZ7aKQRUzkuxCkPfAyAw7xzvjoyVGM5mKf5p/AfbdynMk2OmufTqj/ZA1k\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICPDCCAaUCEDyRMcsf9tAbDpq40ES/Er4wDQYJKoZIhvcNAQEFBQAwXzELMAkG\nA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz\ncyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2\nMDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV\nBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmlt\nYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN\nADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhE\nBarsAx94f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/is\nI19wKTakyYbnsZogy1Olhec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0G\nCSqGSIb3DQEBBQUAA4GBABByUqkFFBkyCEHwxWsKzH4PIRnN5GfcX6kb5sroc50i\n2JhucwNhkcV8sEVAbkSdjbCxlnRhLQ2pRdKkkirWmnWXbj9T/UWZYB2oK0z5XqcJ\n2HUw19JlYD1n1khVdWk/kfVIC0dpImmClr7JyDiGSnoscxlIaU5rfGW/D/xwzoiQ\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDAjCCAmsCEH3Z/gfPqB63EHln+6eJNMYwDQYJKoZIhvcNAQEFBQAwgcExCzAJ\nBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE8MDoGA1UECxMzQ2xh\nc3MgMyBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEcy\nMTowOAYDVQQLEzEoYykgMTk5OCBWZXJpU2lnbiwgSW5jLiAtIEZvciBhdXRob3Jp\nemVkIHVzZSBvbmx5MR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMB4X\nDTk4MDUxODAwMDAwMFoXDTI4MDgwMTIzNTk1OVowgcExCzAJBgNVBAYTAlVTMRcw\nFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE8MDoGA1UECxMzQ2xhc3MgMyBQdWJsaWMg\nUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEcyMTowOAYDVQQLEzEo\nYykgMTk5OCBWZXJpU2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5\nMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMIGfMA0GCSqGSIb3DQEB\nAQUAA4GNADCBiQKBgQDMXtERXVxp0KvTuWpMmR9ZmDCOFoUgRm1HP9SFIIThbbP4\npO0M8RcPO/mn+SXXwc+EY/J8Y8+iR/LGWzOOZEAEaMGAuWQcRXfH2G71lSk8UOg0\n13gfqLptQ5GVj0VXXn7F+8qkBOvqlzdUMG+7AUcyM83cV5tkaWH4mx0ciU9cZwID\nAQABMA0GCSqGSIb3DQEBBQUAA4GBAFFNzb5cy5gZnBWyATl4Lk0PZ3BwmcYQWpSk\nU01UbSuvDV1Ai2TT1+7eVmGSX6bEHRBhNtMsJzzoKQm5EWR0zLVznxxIqbxhAe7i\nF6YM40AIOw7n60RzKprxaZLvcRTDOaxxp5EJb+RxBrO6WVcmeQD2+A2iMzAo1KpY\noJ2daZH9\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDzzCCAregAwIBAgIDAWweMA0GCSqGSIb3DQEBBQUAMIGNMQswCQYDVQQGEwJB\nVDFIMEYGA1UECgw/QS1UcnVzdCBHZXMuIGYuIFNpY2hlcmhlaXRzc3lzdGVtZSBp\nbSBlbGVrdHIuIERhdGVudmVya2VociBHbWJIMRkwFwYDVQQLDBBBLVRydXN0LW5R\ndWFsLTAzMRkwFwYDVQQDDBBBLVRydXN0LW5RdWFsLTAzMB4XDTA1MDgxNzIyMDAw\nMFoXDTE1MDgxNzIyMDAwMFowgY0xCzAJBgNVBAYTAkFUMUgwRgYDVQQKDD9BLVRy\ndXN0IEdlcy4gZi4gU2ljaGVyaGVpdHNzeXN0ZW1lIGltIGVsZWt0ci4gRGF0ZW52\nZXJrZWhyIEdtYkgxGTAXBgNVBAsMEEEtVHJ1c3QtblF1YWwtMDMxGTAXBgNVBAMM\nEEEtVHJ1c3QtblF1YWwtMDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQCtPWFuA/OQO8BBC4SAzewqo51ru27CQoT3URThoKgtUaNR8t4j8DRE/5TrzAUj\nlUC5B3ilJfYKvUWG6Nm9wASOhURh73+nyfrBJcyFLGM/BWBzSQXgYHiVEEvc+RFZ\nznF/QJuKqiTfC0Li21a8StKlDJu3Qz7dg9MmEALP6iPESU7l0+m0iKsMrmKS1GWH\n2WrX9IWf5DMiJaXlyDO6w8dB3F/GaswADm0yqLaHNgBid5seHzTLkDx4iHQF63n1\nk3Flyp3HaxgtPVxO59X4PzF9j4fsCiIvI+n+u33J4PTs63zEsMMtYrWacdaxaujs\n2e3Vcuy+VwHOBVWf3tFgiBCzAgMBAAGjNjA0MA8GA1UdEwEB/wQFMAMBAf8wEQYD\nVR0OBAoECERqlWdVeRFPMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOC\nAQEAVdRU0VlIXLOThaq/Yy/kgM40ozRiPvbY7meIMQQDbwvUB/tOdQ/TLtPAF8fG\nKOwGDREkDg6lXb+MshOWcdzUzg4NCmgybLlBMRmrsQd7TZjTXLDR8KdCoLXEjq/+\n8T/0709GAHbrAvv5ndJAlseIOrifEXnzgGWovR/TeIGgUUw3tKZdJXDRZslo+S4R\nFGjxVJgIrCaSD96JntT6s3kr0qN51OyLrIdTaEJMUVF0HhsnLuP1Hyl0Te2v9+GS\nmYHovjrHF1D2t8b8m7CKa9aIA5GPBnc6hQLdmNVDeD/GMBWsm2vLV7eJUYs66MmE\nDNuxUCAKGkq6ahq97BvIxYSazQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE\nAwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw\nCQYDVQQGEwJFUzAeFw0xMTA1MDUwOTM3MzdaFw0zMDEyMzEwOTM3MzdaMEIxEjAQ\nBgNVBAMMCUFDQ1ZSQUlaMTEQMA4GA1UECwwHUEtJQUNDVjENMAsGA1UECgwEQUND\nVjELMAkGA1UEBhMCRVMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCb\nqau/YUqXry+XZpp0X9DZlv3P4uRm7x8fRzPCRKPfmt4ftVTdFXxpNRFvu8gMjmoY\nHtiP2Ra8EEg2XPBjs5BaXCQ316PWywlxufEBcoSwfdtNgM3802/J+Nq2DoLSRYWo\nG2ioPej0RGy9ocLLA76MPhMAhN9KSMDjIgro6TenGEyxCQ0jVn8ETdkXhBilyNpA\nlHPrzg5XPAOBOp0KoVdDaaxXbXmQeOW1tDvYvEyNKKGno6e6Ak4l0Squ7a4DIrhr\nIA8wKFSVf+DuzgpmndFALW4ir50awQUZ0m/A8p/4e7MCQvtQqR0tkw8jq8bBD5L/\n0KIV9VMJcRz/RROE5iZe+OCIHAr8Fraocwa48GOEAqDGWuzndN9wrqODJerWx5eH\nk6fGioozl2A3ED6XPm4pFdahD9GILBKfb6qkxkLrQaLjlUPTAYVtjrs78yM2x/47\n4KElB0iryYl0/wiPgL/AlmXz7uxLaL2diMMxs0Dx6M/2OLuc5NF/1OVYm3z61PMO\nm3WR5LpSLhl+0fXNWhn8ugb2+1KoS5kE3fj5tItQo05iifCHJPqDQsGH+tUtKSpa\ncXpkatcnYGMN285J9Y0fkIkyF/hzQ7jSWpOGYdbhdQrqeWZ2iE9x6wQl1gpaepPl\nuUsXQA+xtrn13k/c4LOsOxFwYIRKQ26ZIMApcQrAZQIDAQABo4ICyzCCAscwfQYI\nKwYBBQUHAQEEcTBvMEwGCCsGAQUFBzAChkBodHRwOi8vd3d3LmFjY3YuZXMvZmls\nZWFkbWluL0FyY2hpdm9zL2NlcnRpZmljYWRvcy9yYWl6YWNjdjEuY3J0MB8GCCsG\nAQUFBzABhhNodHRwOi8vb2NzcC5hY2N2LmVzMB0GA1UdDgQWBBTSh7Tj3zcnk1X2\nVuqB5TbMjB4/vTAPBgNVHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFNKHtOPfNyeT\nVfZW6oHlNsyMHj+9MIIBcwYDVR0gBIIBajCCAWYwggFiBgRVHSAAMIIBWDCCASIG\nCCsGAQUFBwICMIIBFB6CARAAQQB1AHQAbwByAGkAZABhAGQAIABkAGUAIABDAGUA\ncgB0AGkAZgBpAGMAYQBjAGkA8wBuACAAUgBhAO0AegAgAGQAZQAgAGwAYQAgAEEA\nQwBDAFYAIAAoAEEAZwBlAG4AYwBpAGEAIABkAGUAIABUAGUAYwBuAG8AbABvAGcA\n7QBhACAAeQAgAEMAZQByAHQAaQBmAGkAYwBhAGMAaQDzAG4AIABFAGwAZQBjAHQA\ncgDzAG4AaQBjAGEALAAgAEMASQBGACAAUQA0ADYAMAAxADEANQA2AEUAKQAuACAA\nQwBQAFMAIABlAG4AIABoAHQAdABwADoALwAvAHcAdwB3AC4AYQBjAGMAdgAuAGUA\nczAwBggrBgEFBQcCARYkaHR0cDovL3d3dy5hY2N2LmVzL2xlZ2lzbGFjaW9uX2Mu\naHRtMFUGA1UdHwROMEwwSqBIoEaGRGh0dHA6Ly93d3cuYWNjdi5lcy9maWxlYWRt\naW4vQXJjaGl2b3MvY2VydGlmaWNhZG9zL3JhaXphY2N2MV9kZXIuY3JsMA4GA1Ud\nDwEB/wQEAwIBBjAXBgNVHREEEDAOgQxhY2N2QGFjY3YuZXMwDQYJKoZIhvcNAQEF\nBQADggIBAJcxAp/n/UNnSEQU5CmH7UwoZtCPNdpNYbdKl02125DgBS4OxnnQ8pdp\nD70ER9m+27Up2pvZrqmZ1dM8MJP1jaGo/AaNRPTKFpV8M9xii6g3+CfYCS0b78gU\nJyCpZET/LtZ1qmxNYEAZSUNUY9rizLpm5U9EelvZaoErQNV/+QEnWCzI7UiRfD+m\nAM/EKXMRNt6GGT6d7hmKG9Ww7Y49nCrADdg9ZuM8Db3VlFzi4qc1GwQA9j9ajepD\nvV+JHanBsMyZ4k0ACtrJJ1vnE5Bc5PUzolVt3OAJTS+xJlsndQAJxGJ3KQhfnlms\ntn6tn1QwIgPBHnFk/vk4CpYY3QIUrCPLBhwepH2NDd4nQeit2hW3sCPdK6jT2iWH\n7ehVRE2I9DZ+hJp4rPcOVkkO1jMl1oRQQmwgEh0q1b688nCBpHBgvgW1m54ERL5h\nI6zppSSMEYCUWqKiuUnSwdzRp+0xESyeGabu4VXhwOrPDYTkF7eifKXeVSUG7szA\nh1xA2syVP1XgNce4hL60Xc16gwFy7ofmXx2utYXGJt/mwZrpHgJHnyqobalbz+xF\nd3+YJ5oyXSrjhO7FmGYvliAd3djDJ9ew+f7Zfc3Qn48LFFhRny+Lwzgt3uiP1o2H\npPVWQxaZLPSkVrQ0uGE3ycJYgBugl6H8WY3pEfbRD0tVNEYqi4Y7\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFtTCCA52gAwIBAgIIYY3HhjsBggUwDQYJKoZIhvcNAQEFBQAwRDEWMBQGA1UE\nAwwNQUNFRElDT00gUm9vdDEMMAoGA1UECwwDUEtJMQ8wDQYDVQQKDAZFRElDT00x\nCzAJBgNVBAYTAkVTMB4XDTA4MDQxODE2MjQyMloXDTI4MDQxMzE2MjQyMlowRDEW\nMBQGA1UEAwwNQUNFRElDT00gUm9vdDEMMAoGA1UECwwDUEtJMQ8wDQYDVQQKDAZF\nRElDT00xCzAJBgNVBAYTAkVTMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKC\nAgEA/5KV4WgGdrQsyFhIyv2AVClVYyT/kGWbEHV7w2rbYgIB8hiGtXxaOLHkWLn7\n09gtn70yN78sFW2+tfQh0hOR2QetAQXW8713zl9CgQr5auODAKgrLlUTY4HKRxx7\nXBZXehuDYAQ6PmXDzQHe3qTWDLqO3tkE7hdWIpuPY/1NFgu3e3eM+SW10W2ZEi5P\nGrjm6gSSrj0RuVFCPYewMYWveVqc/udOXpJPQ/yrOq2lEiZmueIM15jO1FillUAK\nt0SdE3QrwqXrIhWYENiLxQSfHY9g5QYbm8+5eaA9oiM/Qj9r+hwDezCNzmzAv+Yb\nX79nuIQZ1RXve8uQNjFiybwCq0Zfm/4aaJQ0PZCOrfbkHQl/Sog4P75n/TSW9R28\nMHTLOO7VbKvU/PQAtwBbhTIWdjPp2KOZnQUAqhbm84F9b32qhm2tFXTTxKJxqvQU\nfecyuB+81fFOvW8XAjnXDpVCOscAPukmYxHqC9FK/xidstd7LzrZlvvoHpKuE1XI\n2Sf23EgbsCTBheN3nZqk8wwRHQ3ItBTutYJXCb8gWH8vIiPYcMt5bMlL8qkqyPyH\nK9caUPgn6C9D4zq92Fdx/c6mUlv53U3t5fZvie27k5x2IXXwkkwp9y+cAS7+UEae\nZAwUswdbxcJzbPEHXEUkFDWug/FqTYl6+rPYLWbwNof1K1MCAwEAAaOBqjCBpzAP\nBgNVHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFKaz4SsrSbbXc6GqlPUB53NlTKxQ\nMA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUprPhKytJttdzoaqU9QHnc2VMrFAw\nRAYDVR0gBD0wOzA5BgRVHSAAMDEwLwYIKwYBBQUHAgEWI2h0dHA6Ly9hY2VkaWNv\nbS5lZGljb21ncm91cC5jb20vZG9jMA0GCSqGSIb3DQEBBQUAA4ICAQDOLAtSUWIm\nfQwng4/F9tqgaHtPkl7qpHMyEVNEskTLnewPeUKzEKbHDZ3Ltvo/Onzqv4hTGzz3\ngvoFNTPhNahXwOf9jU8/kzJPeGYDdwdY6ZXIfj7QeQCM8htRM5u8lOk6e25SLTKe\nI6RF+7YuE7CLGLHdztUdp0J/Vb77W7tH1PwkzQSulgUV1qzOMPPKC8W64iLgpq0i\n5ALudBF/TP94HTXa5gI06xgSYXcGCRZj6hitoocf8seACQl1ThCojz2GuHURwCRi\nipZ7SkXp7FnFvmuD5uHorLUwHv4FB4D54SMNUI8FmP8sX+g7tq3PgbUhh8oIKiMn\nMCArz+2UW6yyetLHKKGKC5tNSixthT8Jcjxn4tncB7rrZXtaAWPWkFtPF2Y9fwsZ\no5NjEFIqnxQWWOLcpfShFosOkYuByptZ+thrkQdlVV9SH686+5DdaaVbnG0OLLb6\nzqylfDJKZ0DcMDQj3dcEI2bw/FWAp/tmGYI1Z2JwOV5vx+qQQEQIHriy1tvuWacN\nGHk0vFQYXlPKNFHtRQrmjseCNj6nOGOpMCwXEGCSn1WHElkQwg9naRHMTh5+Spqt\nr0CodaxWkHS4oJyleW/c6RrIaQXpuvoDs3zk4E7Czp3otkYNbn5XOmeUwssfnHdK\nZ05phkOTOPu220+DkdRgfks+KzgHVZhepA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFuzCCA6OgAwIBAgIIVwoRl0LE48wwDQYJKoZIhvcNAQELBQAwazELMAkGA1UE\nBhMCSVQxDjAMBgNVBAcMBU1pbGFuMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5BLi8w\nMzM1ODUyMDk2NzEnMCUGA1UEAwweQWN0YWxpcyBBdXRoZW50aWNhdGlvbiBSb290\nIENBMB4XDTExMDkyMjExMjIwMloXDTMwMDkyMjExMjIwMlowazELMAkGA1UEBhMC\nSVQxDjAMBgNVBAcMBU1pbGFuMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5BLi8wMzM1\nODUyMDk2NzEnMCUGA1UEAwweQWN0YWxpcyBBdXRoZW50aWNhdGlvbiBSb290IENB\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAp8bEpSmkLO/lGMWwUKNv\nUTufClrJwkg4CsIcoBh/kbWHuUA/3R1oHwiD1S0eiKD4j1aPbZkCkpAW1V8IbInX\n4ay8IMKx4INRimlNAJZaby/ARH6jDuSRzVju3PvHHkVH3Se5CAGfpiEd9UEtL0z9\nKK3giq0itFZljoZUj5NDKd45RnijMCO6zfB9E1fAXdKDa0hMxKufgFpbOr3JpyI/\ngCczWw63igxdBzcIy2zSekciRDXFzMwujt0q7bd9Zg1fYVEiVRvjRuPjPdA1Yprb\nrxTIW6HMiRvhMCb8oJsfgadHHwTrozmSBp+Z07/T6k9QnBn+locePGX2oxgkg4YQ\n51Q+qDp2JE+BIcXjDwL4k5RHILv+1A7TaLndxHqEguNTVHnd25zS8gebLra8Pu2F\nbe8lEfKXGkJh90qX6IuxEAf6ZYGyojnP9zz/GPvG8VqLWeICrHuS0E4UT1lF9gxe\nKF+w6D9Fz8+vm2/7hNN3WpVvrJSEnu68wEqPSpP4RCHiMUVhUE4Q2OM1fEwZtN4F\nv6MGn8i1zeQf1xcGDXqVdFUNaBr8EBtiZJ1t4JWgw5QHVw0U5r0F+7if5t+L4sbn\nfpb2U8WANFAoWPASUHEXMLrmeGO89LKtmyuy/uE5jF66CyCU3nuDuP/jVo23Eek7\njPKxwV2dpAtMK9myGPW1n0sCAwEAAaNjMGEwHQYDVR0OBBYEFFLYiDrIn3hm7Ynz\nezhwlMkCAjbQMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUUtiIOsifeGbt\nifN7OHCUyQICNtAwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4ICAQAL\ne3KHwGCmSUyIWOYdiPcUZEim2FgKDk8TNd81HdTtBjHIgT5q1d07GjLukD0R0i70\njsNjLiNmsGe+b7bAEzlgqqI0JZN1Ut6nna0Oh4lScWoWPBkdg/iaKWW+9D+a2fDz\nWochcYBNy+A4mz+7+uAwTc+G02UQGRjRlwKxK3JCaKygvU5a2hi/a5iB0P2avl4V\nSM0RFbnAKVy06Ij3Pjaut2L9HmLecHgQHEhb2rykOLpn7VU+Xlff1ANATIGk0k9j\npwlCCRT8AKnCgHNPLsBA2RF7SOp6AsDT6ygBJlh0wcBzIm2Tlf05fbsq4/aC4yyX\nX04fkZT6/iyj2HYauE2yOE+b+h1IYHkm4vP9qdCa6HCPSXrW5b0KDtst842/6+Ok\nfcvHlXHo2qN8xcL4dJIEG4aspCJTQLas/kx2z/uUMsA1n3Y/buWQbqCmJqK4LL7R\nK4X9p2jIugErsWx0Hbhzlefut8cl8ABMALJ+tguLHPPAUJ4lueAI3jZm/zel0btU\nZCzJJ7VLkn5l/9Mt4blOvH+kQSGQQXemOR/qnuOf0GZvBeyqdn6/axag67XH/JJU\nLysRJyU3eExRarDzzFhdFPFqSBX/wge2sY0PjlxQRrM9vwGYT7JZVEc+NHt4bVaT\nLnPqZih4zR0Uv6CPLy64Lo7yFIrM6bV8+2ydDKXhlg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU\nMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs\nIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290\nMB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux\nFDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h\nbCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v\ndDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt\nH7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9\nuMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX\nmk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX\na0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN\nE0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0\nWicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD\nVR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0\nJvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU\ncnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx\nIjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN\nAQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH\nYINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5\n6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC\nNr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX\nc4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a\nmnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEGDCCAwCgAwIBAgIBATANBgkqhkiG9w0BAQUFADBlMQswCQYDVQQGEwJTRTEU\nMBIGA1UEChMLQWRkVHJ1c3QgQUIxHTAbBgNVBAsTFEFkZFRydXN0IFRUUCBOZXR3\nb3JrMSEwHwYDVQQDExhBZGRUcnVzdCBDbGFzcyAxIENBIFJvb3QwHhcNMDAwNTMw\nMTAzODMxWhcNMjAwNTMwMTAzODMxWjBlMQswCQYDVQQGEwJTRTEUMBIGA1UEChML\nQWRkVHJ1c3QgQUIxHTAbBgNVBAsTFEFkZFRydXN0IFRUUCBOZXR3b3JrMSEwHwYD\nVQQDExhBZGRUcnVzdCBDbGFzcyAxIENBIFJvb3QwggEiMA0GCSqGSIb3DQEBAQUA\nA4IBDwAwggEKAoIBAQCWltQhSWDia+hBBwzexODcEyPNwTXH+9ZOEQpnXvUGW2ul\nCDtbKRY654eyNAbFvAWlA3yCyykQruGIgb3WntP+LVbBFc7jJp0VLhD7Bo8wBN6n\ntGO0/7Gcrjyvd7ZWxbWroulpOj0OM3kyP3CCkplhbY0wCI9xP6ZIVxn4JdxLZlyl\ndI+Yrsj5wAYi56xz36Uu+1LcsRVlIPo1Zmne3yzxbrww2ywkEtvrNTVokMsAsJch\nPXQhI2U0K7t4WaPW4XY5mqRJjox0r26kmqPZm9I4XJuiGMx1I4S+6+JNM3GOGvDC\n+Mcdoq0Dlyz4zyXG9rgkMbFjXZJ/Y/AlyVMuH79NAgMBAAGjgdIwgc8wHQYDVR0O\nBBYEFJWxtPCUtr3H2tERCSG+wa9J/RB7MAsGA1UdDwQEAwIBBjAPBgNVHRMBAf8E\nBTADAQH/MIGPBgNVHSMEgYcwgYSAFJWxtPCUtr3H2tERCSG+wa9J/RB7oWmkZzBl\nMQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxHTAbBgNVBAsTFEFk\nZFRydXN0IFRUUCBOZXR3b3JrMSEwHwYDVQQDExhBZGRUcnVzdCBDbGFzcyAxIENB\nIFJvb3SCAQEwDQYJKoZIhvcNAQEFBQADggEBACxtZBsfzQ3duQH6lmM0MkhHma6X\n7f1yFqZzR1r0693p9db7RcwpiURdv0Y5PejuvE1Uhh4dbOMXJ0PhiVYrqW9yTkkz\n43J8KiOavD7/KCrto/8cI7pDVwlnTUtiBi34/2ydYB7YHEt9tTEv2dB8Xfjea4MY\neDdXL+gzB2ffHsdrKpV2ro9Xo/D0UrSpUwjP4E/TelOL/bscVjby/rK25Xa71SJl\npz/+0WatC7xrmYbvP33zGDLKe8bjq2RGlfgmadlVg3sslgf/WSxEo8bl6ancoWOA\nWiFeIc9TVPC6b4nbqKqVz4vjccweGyBECMB6tkD9xOQ14R0WHNC8K47Wcdk=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEFTCCAv2gAwIBAgIBATANBgkqhkiG9w0BAQUFADBkMQswCQYDVQQGEwJTRTEU\nMBIGA1UEChMLQWRkVHJ1c3QgQUIxHTAbBgNVBAsTFEFkZFRydXN0IFRUUCBOZXR3\nb3JrMSAwHgYDVQQDExdBZGRUcnVzdCBQdWJsaWMgQ0EgUm9vdDAeFw0wMDA1MzAx\nMDQxNTBaFw0yMDA1MzAxMDQxNTBaMGQxCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtB\nZGRUcnVzdCBBQjEdMBsGA1UECxMUQWRkVHJ1c3QgVFRQIE5ldHdvcmsxIDAeBgNV\nBAMTF0FkZFRydXN0IFB1YmxpYyBDQSBSb290MIIBIjANBgkqhkiG9w0BAQEFAAOC\nAQ8AMIIBCgKCAQEA6Rowj4OIFMEg2Dybjxt+A3S72mnTRqX4jsIMEZBRpS9mVEBV\n6tsfSlbunyNu9DnLoblv8n75XYcmYZ4c+OLspoH4IcUkzBEMP9smcnrHAZcHF/nX\nGCwwfQ56HmIexkvA/X1id9NEHif2P0tEs7c42TkfYNVRknMDtABp4/MUTu7R3AnP\ndzRGULD4EfL+OHn3Bzn+UZKXC1sIXzSGAa2Il+tmzV7R/9x98oTaunet3IAIx6eH\n1lWfl2royBFkuucZKT8Rs3iQhCBSWxHveNCD9tVIkNAwHM+A+WD+eeSI8t0A65RF\n62WUaUC6wNW0uLp9BBGo6zEFlpROWCGOn9Bg/QIDAQABo4HRMIHOMB0GA1UdDgQW\nBBSBPjfYkrAfd59ctKtzquf2NGAv+jALBgNVHQ8EBAMCAQYwDwYDVR0TAQH/BAUw\nAwEB/zCBjgYDVR0jBIGGMIGDgBSBPjfYkrAfd59ctKtzquf2NGAv+qFopGYwZDEL\nMAkGA1UEBhMCU0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMR0wGwYDVQQLExRBZGRU\ncnVzdCBUVFAgTmV0d29yazEgMB4GA1UEAxMXQWRkVHJ1c3QgUHVibGljIENBIFJv\nb3SCAQEwDQYJKoZIhvcNAQEFBQADggEBAAP3FUr4JNojVhaTdt02KLmuG7jD8WS6\nIBh4lSknVwW8fCr0uVFV2ocC3g8WFzH4qnkuCRO7r7IgGRLlk/lL+YPoRNWyQSW/\niHVv/xD8SlTQX/D67zZzfRs2RcYhbbQVuE7PnFylPVoAjgbjPGsye/Kf8Lb93/Ao\nGEjwxrzQvzSAlsJKsW2Ox5BF3i9nrEUEo3rcVZLJR2bYGozH7ZxOmuASu7VqTITh\n4SINhwBk/ox9Yjllpu9CtoAlEmEBqCQTcAARJl/6NVDFSMwGR+gn2HCNX2TmoUQm\nXiLsks3/QppEIW1cxeMiHV9HEufOX1362KqxMy3ZdvJOOjMMK7MtkAY=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEHjCCAwagAwIBAgIBATANBgkqhkiG9w0BAQUFADBnMQswCQYDVQQGEwJTRTEU\nMBIGA1UEChMLQWRkVHJ1c3QgQUIxHTAbBgNVBAsTFEFkZFRydXN0IFRUUCBOZXR3\nb3JrMSMwIQYDVQQDExpBZGRUcnVzdCBRdWFsaWZpZWQgQ0EgUm9vdDAeFw0wMDA1\nMzAxMDQ0NTBaFw0yMDA1MzAxMDQ0NTBaMGcxCzAJBgNVBAYTAlNFMRQwEgYDVQQK\nEwtBZGRUcnVzdCBBQjEdMBsGA1UECxMUQWRkVHJ1c3QgVFRQIE5ldHdvcmsxIzAh\nBgNVBAMTGkFkZFRydXN0IFF1YWxpZmllZCBDQSBSb290MIIBIjANBgkqhkiG9w0B\nAQEFAAOCAQ8AMIIBCgKCAQEA5B6a/twJWoekn0e+EV+vhDTbYjx5eLfpMLXsDBwq\nxBb/4Oxx64r1EW7tTw2R0hIYLUkVAcKkIhPHEWT/IhKauY5cLwjPcWqzZwFZ8V1G\n87B4pfYOQnrjfxvM0PC3KP0q6p6zsLkEqv32x7SxuCqg+1jxGaBvcCV+PmlKfw8i\n2O+tCBGaKZnhqkRFmhJePp1tUvznoD1oL/BLcHwTOK28FSXx1s6rosAx1i+f4P8U\nWfyEk9mHfExUE+uf0S0R+Bg6Ot4l2ffTQO2kBhLEO+GRwVY18BTcZTYJbqukB8c1\n0cIDMzZbdSZtQvESa0NvS3GU+jQd7RNuyoB/mC9suWXY6QIDAQABo4HUMIHRMB0G\nA1UdDgQWBBQ5lYtii1zJ1IC6WA+XPxUIQ8yYpzALBgNVHQ8EBAMCAQYwDwYDVR0T\nAQH/BAUwAwEB/zCBkQYDVR0jBIGJMIGGgBQ5lYtii1zJ1IC6WA+XPxUIQ8yYp6Fr\npGkwZzELMAkGA1UEBhMCU0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMR0wGwYDVQQL\nExRBZGRUcnVzdCBUVFAgTmV0d29yazEjMCEGA1UEAxMaQWRkVHJ1c3QgUXVhbGlm\naWVkIENBIFJvb3SCAQEwDQYJKoZIhvcNAQEFBQADggEBABmrder4i2VhlRO6aQTv\nhsoToMeqT2QbPxj2qC0sVY8FtzDqQmodwCVRLae/DLPt7wh/bDxGGuoYQ992zPlm\nhpwsaPXpF/gxsxjE1kh9I0xowX67ARRvxdlu3rsEQmr49lx95dr6h+sNNVJn0J6X\ndgWTP5XHAeZpVTh/EGGZyeNfpso+gmNIquIISD6q8rKFYqa0p9m9N5xotS1WfbC3\nP6CxB9bpT9zeRXEwMn8bLgn5v1Kh7sKAPgZcLlVAwRv1cEWw3F369nJad9Jjzc9Y\niQBCYz95OdBEsIJuQRno3eDBiFrRHnGTHyQwdOUeqN48Jzd/g66ed8/wMLH/S5no\nxqE=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDTDCCAjSgAwIBAgIId3cGJyapsXwwDQYJKoZIhvcNAQELBQAwRDELMAkGA1UE\nBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZpcm1UcnVz\ndCBDb21tZXJjaWFsMB4XDTEwMDEyOTE0MDYwNloXDTMwMTIzMTE0MDYwNlowRDEL\nMAkGA1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZp\ncm1UcnVzdCBDb21tZXJjaWFsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEA9htPZwcroRX1BiLLHwGy43NFBkRJLLtJJRTWzsO3qyxPxkEylFf6EqdbDuKP\nHx6GGaeqtS25Xw2Kwq+FNXkyLbscYjfysVtKPcrNcV/pQr6U6Mje+SJIZMblq8Yr\nba0F8PrVC8+a5fBQpIs7R6UjW3p6+DM/uO+Zl+MgwdYoic+U+7lF7eNAFxHUdPAL\nMeIrJmqbTFeurCA+ukV6BfO9m2kVrn1OIGPENXY6BwLJN/3HR+7o8XYdcxXyl6S1\nyHp52UKqK39c/s4mT6NmgTWvRLpUHhwwMmWd5jyTXlBOeuM61G7MGvv50jeuJCqr\nVwMiKA1JdX+3KNp1v47j3A55MQIDAQABo0IwQDAdBgNVHQ4EFgQUnZPGU4teyq8/\nnx4P5ZmVvCT2lI8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwDQYJ\nKoZIhvcNAQELBQADggEBAFis9AQOzcAN/wr91LoWXym9e2iZWEnStB03TX8nfUYG\nXUPGhi4+c7ImfU+TqbbEKpqrIZcUsd6M06uJFdhrJNTxFq7YpFzUf1GO7RgBsZNj\nvbz4YYCanrHOQnDiqX0GJX0nof5v7LMeJNrjS1UaADs1tDvZ110w/YETifLCBivt\nZ8SOyUOyXGsViQK8YvxO8rUzqrJv0wqiUOP2O+guRMLbZjipM1ZI8W0bM40NjD9g\nN53Tym1+NH4Nn3J2ixufcv1SNUFFApYvHLKac0khsUlHRUe072o0EclNmsxZt9YC\nnlpOZbWUrhvfKbAW8b8Angc6F2S1BLUjIZkKlTuXfO8=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDTDCCAjSgAwIBAgIIfE8EORzUmS0wDQYJKoZIhvcNAQEFBQAwRDELMAkGA1UE\nBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZpcm1UcnVz\ndCBOZXR3b3JraW5nMB4XDTEwMDEyOTE0MDgyNFoXDTMwMTIzMTE0MDgyNFowRDEL\nMAkGA1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MR8wHQYDVQQDDBZBZmZp\ncm1UcnVzdCBOZXR3b3JraW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEAtITMMxcua5Rsa2FSoOujz3mUTOWUgJnLVWREZY9nZOIG41w3SfYvm4SEHi3y\nYJ0wTsyEheIszx6e/jarM3c1RNg1lho9Nuh6DtjVR6FqaYvZ/Ls6rnla1fTWcbua\nkCNrmreIdIcMHl+5ni36q1Mr3Lt2PpNMCAiMHqIjHNRqrSK6mQEubWXLviRmVSRL\nQESxG9fhwoXA3hA/Pe24/PHxI1Pcv2WXb9n5QHGNfb2V1M6+oF4nI979ptAmDgAp\n6zxG8D1gvz9Q0twmQVGeFDdCBKNwV6gbh+0t+nvujArjqWaJGctB+d1ENmHP4ndG\nyH329JKBNv3bNPFyfvMMFr20FQIDAQABo0IwQDAdBgNVHQ4EFgQUBx/S55zawm6i\nQLSwelAQUHTEyL0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwDQYJ\nKoZIhvcNAQEFBQADggEBAIlXshZ6qML91tmbmzTCnLQyFE2npN/svqe++EPbkTfO\ntDIuUFUaNU52Q3Eg75N3ThVwLofDwR1t3Mu1J9QsVtFSUzpE0nPIxBsFZVpikpzu\nQY0x2+c06lkh1QF612S4ZDnNye2v7UsDSKegmQGA3GWjNq5lWUhPgkvIZfFXHeVZ\nLgo/bNjR9eUJtGxUAArgFU2HdW23WJZa3W3SAKD0m0i+wzekujbgfIeFlxoVot4u\nolu9rxj5kFDNcFn4J2dHy8egBzp90SxdbBk6ZrV9/ZFvgrG+CJPbFEfxojfHRZ48\nx3evZKiT3/Zpg4Jg8klCNO1aAFSFHBY2kgxc+qatv9s=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFRjCCAy6gAwIBAgIIbYwURrGmCu4wDQYJKoZIhvcNAQEMBQAwQTELMAkGA1UE\nBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MRwwGgYDVQQDDBNBZmZpcm1UcnVz\ndCBQcmVtaXVtMB4XDTEwMDEyOTE0MTAzNloXDTQwMTIzMTE0MTAzNlowQTELMAkG\nA1UEBhMCVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MRwwGgYDVQQDDBNBZmZpcm1U\ncnVzdCBQcmVtaXVtMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxBLf\nqV/+Qd3d9Z+K4/as4Tx4mrzY8H96oDMq3I0gW64tb+eT2TZwamjPjlGjhVtnBKAQ\nJG9dKILBl1fYSCkTtuG+kU3fhQxTGJoeJKJPj/CihQvL9Cl/0qRY7iZNyaqoe5rZ\n+jjeRFcV5fiMyNlI4g0WJx0eyIOFJbe6qlVBzAMiSy2RjYvmia9mx+n/K+k8rNrS\ns8PhaJyJ+HoAVt70VZVs+7pk3WKL3wt3MutizCaam7uqYoNMtAZ6MMgpv+0GTZe5\nHMQxK9VfvFMSF5yZVylmd2EhMQcuJUmdGPLu8ytxjLW6OQdJd/zvLpKQBY0tL3d7\n70O/Nbua2Plzpyzy0FfuKE4mX4+QaAkvuPjcBukumj5Rp9EixAqnOEhss/n/fauG\nV+O61oV4d7pD6kh/9ti+I20ev9E2bFhc8e6kGVQa9QPSdubhjL08s9NIS+LI+H+S\nqHZGnEJlPqQewQcDWkYtuJfzt9WyVSHvutxMAJf7FJUnM7/oQ0dG0giZFmA7mn7S\n5u046uwBHjxIVkkJx0w3AJ6IDsBz4W9m6XJHMD4Q5QsDyZpCAGzFlH5hxIrff4Ia\nC1nEWTJ3s7xgaVY5/bQGeyzWZDbZvUjthB9+pSKPKrhC9IK31FOQeE4tGv2Bb0TX\nOwF0lkLgAOIua+rF7nKsu7/+6qqo+Nz2snmKtmcCAwEAAaNCMEAwHQYDVR0OBBYE\nFJ3AZ6YMItkm9UWrpmVSESfYRaxjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/\nBAQDAgEGMA0GCSqGSIb3DQEBDAUAA4ICAQCzV00QYk465KzquByvMiPIs0laUZx2\nKI15qldGF9X1Uva3ROgIRL8YhNILgM3FEv0AVQVhh0HctSSePMTYyPtwni94loMg\nNt58D2kTiKV1NpgIpsbfrM7jWNa3Pt668+s0QNiigfV4Py/VpfzZotReBA4Xrf5B\n8OWycvpEgjNC6C1Y91aMYj+6QrCcDFx+LmUmXFNPALJ4fqENmS2NuB2OosSw/WDQ\nMKSOyARiqcTtNd56l+0OOF6SL5Nwpamcb6d9Ex1+xghIsV5n61EIJenmJWtSKZGc\n0jlzCFfemQa0W50QBuHCAKi4HEoCChTQwUHK+4w1IX2COPKpVJEZNZOUbWo6xbLQ\nu4mGk+ibyQ86p3q4ofB4Rvr8Ny/lioTz3/4E2aFooC8k4gmVBtWVyuEklut89pMF\nu+1z6S3RdTnX5yTb2E5fQ4+e0BQ5v1VwSJlXMbSc7kqYA5YwH2AG7hsj/oFgIxpH\nYoWlzBk0gG+zrBrjn/B7SK3VAdlntqlyk+otZrWyuOQ9PLLvTIzq6we/qzWaVYa8\nGKa1qF60g2xraUDTn9zxw2lrueFtCfTxqlB2Cnp9ehehVZZCmTEJ3WARjQUwfuaO\nRtGdFNrHF+QFlozEJLUbzxQHskD4o55BhrwE0GuWyCqANP2/7waj3VjFhT0+j/6e\nKeC2uAloGRwYQw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIB/jCCAYWgAwIBAgIIdJclisc/elQwCgYIKoZIzj0EAwMwRTELMAkGA1UEBhMC\nVVMxFDASBgNVBAoMC0FmZmlybVRydXN0MSAwHgYDVQQDDBdBZmZpcm1UcnVzdCBQ\ncmVtaXVtIEVDQzAeFw0xMDAxMjkxNDIwMjRaFw00MDEyMzExNDIwMjRaMEUxCzAJ\nBgNVBAYTAlVTMRQwEgYDVQQKDAtBZmZpcm1UcnVzdDEgMB4GA1UEAwwXQWZmaXJt\nVHJ1c3QgUHJlbWl1bSBFQ0MwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQNMF4bFZ0D\n0KF5Nbc6PJJ6yhUczWLznCZcBz3lVPqj1swS6vQUX+iOGasvLkjmrBhDeKzQN8O9\nss0s5kfiGuZjuD0uL3jET9v0D6RoTFVya5UdThhClXjMNzyR4ptlKymjQjBAMB0G\nA1UdDgQWBBSaryl6wBE1NSZRMADDav5A1a7WPDAPBgNVHRMBAf8EBTADAQH/MA4G\nA1UdDwEB/wQEAwIBBjAKBggqhkjOPQQDAwNnADBkAjAXCfOHiFBar8jAQr9HX/Vs\naobgxCd05DhT1wV/GzTjxi+zygk8N53X57hG8f2h4nECMEJZh0PUUd+60wkyWs6I\nflc9nF9Ca/UHLbXwgpP5WW+uZPpY5Yse42O+tYHNbwKMeQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDoDCCAoigAwIBAgIBMTANBgkqhkiG9w0BAQUFADBDMQswCQYDVQQGEwJKUDEc\nMBoGA1UEChMTSmFwYW5lc2UgR292ZXJubWVudDEWMBQGA1UECxMNQXBwbGljYXRp\nb25DQTAeFw0wNzEyMTIxNTAwMDBaFw0xNzEyMTIxNTAwMDBaMEMxCzAJBgNVBAYT\nAkpQMRwwGgYDVQQKExNKYXBhbmVzZSBHb3Zlcm5tZW50MRYwFAYDVQQLEw1BcHBs\naWNhdGlvbkNBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp23gdE6H\nj6UG3mii24aZS2QNcfAKBZuOquHMLtJqO8F6tJdhjYq+xpqcBrSGUeQ3DnR4fl+K\nf5Sk10cI/VBaVuRorChzoHvpfxiSQE8tnfWuREhzNgaeZCw7NCPbXCbkcXmP1G55\nIrmTwcrNwVbtiGrXoDkhBFcsovW8R0FPXjQilbUfKW1eSvNNcr5BViCH/OlQR9cw\nFO5cjFW6WY2H/CPek9AEjP3vbb3QesmlOmpyM8ZKDQUXKi17safY1vC+9D/qDiht\nQWEjdnjDuGWk81quzMKq2edY3rZ+nYVunyoKb58DKTCXKB28t89UKU5RMfkntigm\n/qJj5kEW8DOYRwIDAQABo4GeMIGbMB0GA1UdDgQWBBRUWssmP3HMlEYNllPqa0jQ\nk/5CdTAOBgNVHQ8BAf8EBAMCAQYwWQYDVR0RBFIwUKROMEwxCzAJBgNVBAYTAkpQ\nMRgwFgYDVQQKDA/ml6XmnKzlm73mlL/lupwxIzAhBgNVBAsMGuOCouODl+ODquOC\nseODvOOCt+ODp+ODs0NBMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEFBQAD\nggEBADlqRHZ3ODrso2dGD/mLBqj7apAxzn7s2tGJfHrrLgy9mTLnsCTWw//1sogJ\nhyzjVOGjprIIC8CFqMjSnHH2HZ9g/DgzE+Ge3Atf2hZQKXsvcJEPmbo0NI2VdMV+\neKlmXb3KIXdCEKxmJj3ekav9FfBv7WxfEPjzFvYDio+nEhEMy/0/ecGc/WLuo89U\nDNErXxc+4z6/wCs+CZv+iKZ+tJIX/COUgb1up8WMwusRRdv4QcmWdupwX3kSa+Sj\nB1oF7ydJzyGfikwJcGapJsErEU4z0g781mzSDjJkaP+tBXhfAx2o45CsJOAPQKdL\nrosot4LKGAfmt1t06SAZf7IbiVQ=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDdzCCAl+gAwIBAgIIXDPLYixfszIwDQYJKoZIhvcNAQELBQAwPDEeMBwGA1UE\nAwwVQXRvcyBUcnVzdGVkUm9vdCAyMDExMQ0wCwYDVQQKDARBdG9zMQswCQYDVQQG\nEwJERTAeFw0xMTA3MDcxNDU4MzBaFw0zMDEyMzEyMzU5NTlaMDwxHjAcBgNVBAMM\nFUF0b3MgVHJ1c3RlZFJvb3QgMjAxMTENMAsGA1UECgwEQXRvczELMAkGA1UEBhMC\nREUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCVhTuXbyo7LjvPpvMp\nNb7PGKw+qtn4TaA+Gke5vJrf8v7MPkfoepbCJI419KkM/IL9bcFyYie96mvr54rM\nVD6QUM+A1JX76LWC1BTFtqlVJVfbsVD2sGBkWXppzwO3bw2+yj5vdHLqqjAqc2K+\nSZFhyBH+DgMq92og3AIVDV4VavzjgsG1xZ1kCWyjWZgHJ8cblithdHFsQ/H3NYkQ\n4J7sVaE3IqKHBAUsR320HLliKWYoyrfhk/WklAOZuXCFteZI6o1Q/NnezG8HDt0L\ncp2AMBYHlT8oDv3FdU9T1nSatCQujgKRz3bFmx5VdJx4IbHwLfELn8LVlhgf8FQi\neowHAgMBAAGjfTB7MB0GA1UdDgQWBBSnpQaxLKYJYO7Rl+lwrrw7GWzbITAPBgNV\nHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFKelBrEspglg7tGX6XCuvDsZbNshMBgG\nA1UdIAQRMA8wDQYLKwYBBAGwLQMEAQEwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3\nDQEBCwUAA4IBAQAmdzTblEiGKkGdLD4GkGDEjKwLVLgfuXvTBznk+j57sj1O7Z8j\nvZfza1zv7v1Apt+hk6EKhqzvINB5Ab149xnYJDE0BAGmuhWawyfc2E8PzBhj/5kP\nDpFrdRbhIfzYJsdHt6bPWHJxfrrhTZVHO8mvbaG0weyJ9rQPOLXiZNwlz6bb65pc\nmaHFCN795trV1lpFDMS3wrUU77QR/w4VtfX128a961qn8FYiqTxlVMYVqL2Gns2D\nlmh6cYGJ4Qvh6hEbaAjMaZ7snkGeRDImeuKHCnE96+RapNLbxc3G3mB/ufNPRJLv\nKrcYPqcZ2Qt9sTdBQrC6YB3y/gkRsPCHe6ed\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIGFDCCA/ygAwIBAgIIU+w77vuySF8wDQYJKoZIhvcNAQEFBQAwUTELMAkGA1UE\nBhMCRVMxQjBABgNVBAMMOUF1dG9yaWRhZCBkZSBDZXJ0aWZpY2FjaW9uIEZpcm1h\ncHJvZmVzaW9uYWwgQ0lGIEE2MjYzNDA2ODAeFw0wOTA1MjAwODM4MTVaFw0zMDEy\nMzEwODM4MTVaMFExCzAJBgNVBAYTAkVTMUIwQAYDVQQDDDlBdXRvcmlkYWQgZGUg\nQ2VydGlmaWNhY2lvbiBGaXJtYXByb2Zlc2lvbmFsIENJRiBBNjI2MzQwNjgwggIi\nMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKlmuO6vj78aI14H9M2uDDUtd9\nthDIAl6zQyrET2qyyhxdKJp4ERppWVevtSBC5IsP5t9bpgOSL/UR5GLXMnE42QQM\ncas9UX4PB99jBVzpv5RvwSmCwLTaUbDBPLutN0pcyvFLNg4kq7/DhHf9qFD0sefG\nL9ItWY16Ck6WaVICqjaY7Pz6FIMMNx/Jkjd/14Et5cS54D40/mf0PmbR0/RAz15i\nNA9wBj4gGFrO93IbJWyTdBSTo3OxDqqHECNZXyAFGUftaI6SEspd/NYrspI8IM/h\nX68gvqB2f3bl7BqGYTM+53u0P6APjqK5am+5hyZvQWyIplD9amML9ZMWGxmPsu2b\nm8mQ9QEM3xk9Dz44I8kvjwzRAv4bVdZO0I08r0+k8/6vKtMFnXkIoctXMbScyJCy\nZ/QYFpM6/EfY0XiWMR+6KwxfXZmtY4laJCB22N/9q06mIqqdXuYnin1oKaPnirja\nEbsXLZmdEyRG98Xi2J+Of8ePdG1asuhy9azuJBCtLxTa/y2aRnFHvkLfuwHb9H/T\nKI8xWVvTyQKmtFLKbpf7Q8UIJm+K9Lv9nyiqDdVF8xM6HdjAeI9BZzwelGSuewvF\n6NkBiDkal4ZkQdU7hwxu+g/GvUgUvzlN1J5Bto+WHWOWk9mVBngxaJ43BjuAiUVh\nOSPHG0SjFeUc+JIwuwIDAQABo4HvMIHsMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYD\nVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRlzeurNR4APn7VdMActHNHDhpkLzCBpgYD\nVR0gBIGeMIGbMIGYBgRVHSAAMIGPMC8GCCsGAQUFBwIBFiNodHRwOi8vd3d3LmZp\ncm1hcHJvZmVzaW9uYWwuY29tL2NwczBcBggrBgEFBQcCAjBQHk4AUABhAHMAZQBv\nACAAZABlACAAbABhACAAQgBvAG4AYQBuAG8AdgBhACAANAA3ACAAQgBhAHIAYwBl\nAGwAbwBuAGEAIAAwADgAMAAxADcwDQYJKoZIhvcNAQEFBQADggIBABd9oPm03cXF\n661LJLWhAqvdpYhKsg9VSytXjDvlMd3+xDLx51tkljYyGOylMnfX40S2wBEqgLk9\nam58m9Ot/MPWo+ZkKXzR4Tgegiv/J2Wv+xYVxC5xhOW1//qkR71kMrv2JYSiJ0L1\nILDCExARzRAVukKQKtJE4ZYm6zFIEv0q2skGz3QeqUvVhyj5eTSSPi5E6PaPT481\nPyWzOdxjKpBrIF/EUhJOlywqrJ2X3kjyo2bbwtKDlaZmp54lD+kLM5FlClrD2VQS\n3a/DTg4fJl4N3LON7NWBcN7STyQF82xO9UxJZo3R/9ILJUFI/lGExkKvgATP0H5k\nSeTy36LssUzAKh3ntLFlosS88Zj0qnAHY7S42jtM+kAiMFsRpvAFDsYCA0irhpuF\n3dvd6qJ2gHN99ZwExEWN57kci57q13XRcrHedUTnQn3iV2t93Jm8PYMo6oCTjcVM\nZcFwgbg4/EMxsvYDNEeyrPsiBsse3RdHHF9mudMaotoRsaS8I8nkvof/uZS2+F0g\nStRf571oe2XyFR7SOqkt6dhrJKyXWERHrVkY8SFlcN7ONGCoQPHzPKTDKCOM/icz\nQ0CgFzzr6juwcqajuUpLXhZI9LK8yIySxZ2frHI2vDSANGupi5LAuBft7HZT9SQB\njLMi6Et8Vcad+qMUu2WFbm5PEn4KPJ2V\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ\nRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD\nVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX\nDTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y\nZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy\nVHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr\nmD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr\nIZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK\nmpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu\nXmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy\ndc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye\njl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1\nBE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3\nDQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92\n9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx\njkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0\nEpn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz\nksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS\nR9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDUzCCAjugAwIBAgIBATANBgkqhkiG9w0BAQUFADBLMQswCQYDVQQGEwJOTzEd\nMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxHTAbBgNVBAMMFEJ1eXBhc3Mg\nQ2xhc3MgMiBDQSAxMB4XDTA2MTAxMzEwMjUwOVoXDTE2MTAxMzEwMjUwOVowSzEL\nMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MR0wGwYD\nVQQDDBRCdXlwYXNzIENsYXNzIDIgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEP\nADCCAQoCggEBAIs8B0XY9t/mx8q6jUPFR42wWsE425KEHK8T1A9vNkYgxC7McXA0\nojTTNy7Y3Tp3L8DrKehc0rWpkTSHIln+zNvnma+WwajHQN2lFYxuyHyXA8vmIPLX\nl18xoS830r7uvqmtqEyeIWZDO6i88wmjONVZJMHCR3axiFyCO7srpgTXjAePzdVB\nHfCuuCkslFJgNJQ72uA40Z0zPhX0kzLFANq1KWYOOngPIVJfAuWSeyXTkh4vFZ2B\n5J2O6O+JzhRMVB0cgRJNcKi+EAUXfh/RuFdV7c27UsKwHnjCTTZoy1YmwVLBvXb3\nWNVyfh9EdrsAiR0WnVE1703CVu9r4Iw7DekCAwEAAaNCMEAwDwYDVR0TAQH/BAUw\nAwEB/zAdBgNVHQ4EFgQUP42aWYv8e3uco684sDntkHGA1sgwDgYDVR0PAQH/BAQD\nAgEGMA0GCSqGSIb3DQEBBQUAA4IBAQAVGn4TirnoB6NLJzKyQJHyIdFkhb5jatLP\ngcIV1Xp+DCmsNx4cfHZSldq1fyOhKXdlyTKdqC5Wq2B2zha0jX94wNWZUYN/Xtm+\nDKhQ7SLHrQVMdvvt7h5HZPb3J31cKA9FxVxiXqaakZG3Uxcu3K1gnZZkOb1naLKu\nBctN518fV4bVIJwo+28TOPX2EZL2fZleHwzoq0QkKXJAPTZSr4xYkHPB7GEseaHs\nh7U/2k3ZIQAw3pDaDtMaSKk+hQsUi4y8QZ5q9w5wwDX3OaJdZtB7WZ+oRxKaJyOk\nLY4ng5IgodcVf/EuGO70SH8vf/GhGLWhC5SgYiAynB321O+/TIho\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFWTCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQsFADBOMQswCQYDVQQGEwJOTzEd\nMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIDAeBgNVBAMMF0J1eXBhc3Mg\nQ2xhc3MgMiBSb290IENBMB4XDTEwMTAyNjA4MzgwM1oXDTQwMTAyNjA4MzgwM1ow\nTjELMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MSAw\nHgYDVQQDDBdCdXlwYXNzIENsYXNzIDIgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEB\nBQADggIPADCCAgoCggIBANfHXvfBB9R3+0Mh9PT1aeTuMgHbo4Yf5FkNuud1g1Lr\n6hxhFUi7HQfKjK6w3Jad6sNgkoaCKHOcVgb/S2TwDCo3SbXlzwx87vFKu3MwZfPV\nL4O2fuPn9Z6rYPnT8Z2SdIrkHJasW4DptfQxh6NR/Md+oW+OU3fUl8FVM5I+GC91\n1K2GScuVr1QGbNgGE41b/+EmGVnAJLqBcXmQRFBoJJRfuLMR8SlBYaNByyM21cHx\nMlAQTn/0hpPshNOOvEu/XAFOBz3cFIqUCqTqc/sLUegTBxj6DvEr0VQVfTzh97QZ\nQmdiXnfgolXsttlpF9U6r0TtSsWe5HonfOV116rLJeffawrbD02TTqigzXsu8lkB\narcNuAeBfos4GzjmCleZPe4h6KP1DBbdi+w0jpwqHAAVF41og9JwnxgIzRFo1clr\nUs3ERo/ctfPYV3Me6ZQ5BL/T3jjetFPsaRyifsSP5BtwrfKi+fv3FmRmaZ9JUaLi\nFRhnBkp/1Wy1TbMz4GHrXb7pmA8y1x1LPC5aAVKRCfLf6o3YBkBjqhHk/sM3nhRS\nP/TizPJhk9H9Z2vXUq6/aKtAQ6BXNVN48FP4YUIHZMbXb5tMOA1jrGKvNouicwoN\n9SG9dKpN6nIDSdvHXx1iY8f93ZHsM+71bbRuMGjeyNYmsHVee7QHIJihdjK4TWxP\nAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMmAd+BikoL1Rpzz\nuvdMw964o605MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAU18h\n9bqwOlI5LJKwbADJ784g7wbylp7ppHR/ehb8t/W2+xUbP6umwHJdELFx7rxP462s\nA20ucS6vxOOto70MEae0/0qyexAQH6dXQbLArvQsWdZHEIjzIVEpMMpghq9Gqx3t\nOluwlN5E40EIosHsHdb9T7bWR9AUC8rmyrV7d35BH16Dx7aMOZawP5aBQW9gkOLo\n+fsicdl9sz1Gv7SEr5AcD48Saq/v7h56rgJKihcrdv6sVIkkLE8/trKnToyokZf7\nKcZ7XC25y2a2t6hbElGFtQl+Ynhw/qlqYLYdDnkM/crqJIByw5c/8nerQyIKx+u2\nDISCLIBrQYoIwOula9+ZEsuK1V6ADJHgJgg2SMX6OBE1/yWDLfJ6v9r9jv6ly0Us\nH8SIU653DtmadsWOLB2jutXsMq7Aqqz30XpN69QH4kj3Io6wpJ9qzo6ysmD0oyLQ\nI+uUWnpp3Q+/QFesa1lQ2aOZ4W7+jQF5JyMV3pKdewlNWudLSDBaGOYKbeaP4NK7\n5t98biGCwWg5TbSYWGZizEqQXsP6JwSxeRV0mcy+rSDeJmAc61ZRpqPq5KM/p/9h\n3PFaTWwyI0PurKju7koSCTxdccK+efrCh2gdC/1cacwG0Jp9VJkqyTkaGa9LKkPz\nY11aWOIv4x3kqdbQCtCev9eBCfHJxyYNrJgWVqA=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDUzCCAjugAwIBAgIBAjANBgkqhkiG9w0BAQUFADBLMQswCQYDVQQGEwJOTzEd\nMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxHTAbBgNVBAMMFEJ1eXBhc3Mg\nQ2xhc3MgMyBDQSAxMB4XDTA1MDUwOTE0MTMwM1oXDTE1MDUwOTE0MTMwM1owSzEL\nMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MR0wGwYD\nVQQDDBRCdXlwYXNzIENsYXNzIDMgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEP\nADCCAQoCggEBAKSO13TZKWTeXx+HgJHqTjnmGcZEC4DVC69TB4sSveZn8AKxifZg\nisRbsELRwCGoy+Gb72RRtqfPFfV0gGgEkKBYouZ0plNTVUhjP5JW3SROjvi6K//z\nNIqeKNc0n6wv1g/xpC+9UrJJhW05NfBEMJNGJPO251P7vGGvqaMU+8IXF4Rs4HyI\n+MkcVyzwPX6UvCWThOiaAJpFBUJXgPROztmuOfbIUxAMZTpHe2DC1vqRycZxbL2R\nhzyRhkmr8w+gbCZ2Xhysm3HljbybIR6c1jh+JIAVMYKWsUnTYjdbiAwKYjT+p0h+\nmbEwi5A3lRyoH6UsjfRVyNvdWQrCrXig9IsCAwEAAaNCMEAwDwYDVR0TAQH/BAUw\nAwEB/zAdBgNVHQ4EFgQUOBTmyPCppAP0Tj4io1vy1uCtQHQwDgYDVR0PAQH/BAQD\nAgEGMA0GCSqGSIb3DQEBBQUAA4IBAQABZ6OMySU9E2NdFm/soT4JXJEVKirZgCFP\nBdy7pYmrEzMqnji3jG8CcmPHc3ceCQa6Oyh7pEfJYWsICCD8igWKH7y6xsL+z27s\nEzNxZy5p+qksP2bAEllNC1QCkoS72xLvg3BweMhT+t/Gxv/ciC8HwEmdMldg0/L2\nmSlf56oBzKwzqBwKu5HEA6BvtjT5htOzdlSY9EqBs1OdTUDs5XcTRa9bqh/YL0yC\ne/4qxFi7T/ye/QNlGioOw6UgFpRreaaiErS7GqQjel/wroQk5PMr+4okoyeYZdow\ndXb8GZHo2+ubPzK/QJcHJrrM85SFSnonk8+QQtS4Wxam58tAA915\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFWTCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQsFADBOMQswCQYDVQQGEwJOTzEd\nMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIDAeBgNVBAMMF0J1eXBhc3Mg\nQ2xhc3MgMyBSb290IENBMB4XDTEwMTAyNjA4Mjg1OFoXDTQwMTAyNjA4Mjg1OFow\nTjELMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3MgQVMtOTgzMTYzMzI3MSAw\nHgYDVQQDDBdCdXlwYXNzIENsYXNzIDMgUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEB\nBQADggIPADCCAgoCggIBAKXaCpUWUOOV8l6ddjEGMnqb8RB2uACatVI2zSRHsJ8Y\nZLya9vrVediQYkwiL944PdbgqOkcLNt4EemOaFEVcsfzM4fkoF0LXOBXByow9c3E\nN3coTRiR5r/VUv1xLXA+58bEiuPwKAv0dpihi4dVsjoT/Lc+JzeOIuOoTyrvYLs9\ntznDDgFHmV0ST9tD+leh7fmdvhFHJlsTmKtdFoqwNxxXnUX/iJY2v7vKB3tvh2PX\n0DJq1l1sDPGzbjniazEuOQAnFN44wOwZZoYS6J1yFhNkUsepNxz9gjDthBgd9K5c\n/3ATAOux9TN6S9ZV+AWNS2mw9bMoNlwUxFFzTWsL8TQH2xc519woe2v1n/MuwU8X\nKhDzzMro6/1rqy6any2CbgTUUgGTLT2G/H783+9CHaZr77kgxve9oKeV/afmiSTY\nzIw0bOIjL9kSGiG5VZFvC5F5GQytQIgLcOJ60g7YaEi7ghM5EFjp2CoHxhLbWNvS\nO1UQRwUVZ2J+GGOmRj8JDlQyXr8NYnon74Do29lLBlo3WiXQCBJ31G8JUJc9yB3D\n34xFMFbG02SrZvPAXpacw8Tvw3xrizp5f7NJzz3iiZ+gMEuFuZyUJHmPfWupRWgP\nK9Dx2hzLabjKSWJtyNBjYt1gD1iqj6G8BaVmos8bdrKEZLFMOVLAMLrwjEsCsLa3\nAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEe4zf/lb+74suwv\nTg75JbCOPGvDMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAACAj\nQTUEkMJAYmDv4jVM1z+s4jSQuKFvdvoWFqRINyzpkMLyPPgKn9iB5btb2iUspKdV\ncSQy9sgL8rxq+JOssgfCX5/bzMiKqr5qb+FJEMwx14C7u8jYog5kV+qi9cKpMRXS\nIGrs/CIBKM+GuIAeqcwRpTzyFrNHnfzSgCHEy9BHcEGhyoMZCCxt8l13nIoUE9Q2\nHJLw5QY33KbmkJs4j1xrG0aGQ0JfPgEHU1RdZX33inOhmlRaHylDFCfChQ+1iHsa\nO5S3HWCntZznKWlXWpuTekMwGwPXYshApqr8ZORK15FTAaggiG6cX0S5y2CBNOxv\n033aSF/rtJC8LakcC6wc1aJoIIAE1vyxjy+7SjENSoYc6+I2KSb12tjE8nVhz36u\ndmNKekBlk4f4HoCMhuWG1o8O/FMsYOgWYRqiPkN7zTlgVGr18okmAWiDSKIz6MkE\nkbIRNBE+6tBDGR8Dk5AM/1E9V/RBbuHLoL7ryWPNbczk+DaqaJ3tvV2XcEQNtg41\n3OEMXbugUZTLfhbrES+jkkXITHHZvMmZUldGL1DPvTVp9D0VzgalLA8+9oG6lLvD\nu79leNKGef9JOxqDDPDeeOzI8k1MGt6CKfjBWtrt7uYnXuhF0J0cUahoq0Tj0Itq\n4/g7u9xN12TyUb7mqqta6THuBrxzvxNiCp/HuZc=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEDzCCAvegAwIBAgIBATANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJTSzET\nMBEGA1UEBxMKQnJhdGlzbGF2YTETMBEGA1UEChMKRGlzaWcgYS5zLjERMA8GA1UE\nAxMIQ0EgRGlzaWcwHhcNMDYwMzIyMDEzOTM0WhcNMTYwMzIyMDEzOTM0WjBKMQsw\nCQYDVQQGEwJTSzETMBEGA1UEBxMKQnJhdGlzbGF2YTETMBEGA1UEChMKRGlzaWcg\nYS5zLjERMA8GA1UEAxMIQ0EgRGlzaWcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\nggEKAoIBAQCS9jHBfYj9mQGp2HvycXXxMcbzdWb6UShGhJd4NLxs/LxFWYgmGErE\nNx+hSkS943EE9UQX4j/8SFhvXJ56CbpRNyIjZkMhsDxkovhqFQ4/61HhVKndBpnX\nmjxUizkDPw/Fzsbrg3ICqB9x8y34dQjbYkzo+s7552oftms1grrijxaSfQUMbEYD\nXcDtab86wYqg6I7ZuUUohwjstMoVvoLdtUSLLa2GDGhibYVW8qwUYzrG0ZmsNHhW\nS8+2rT+MitcE5eN4TPWGqvWP+j1scaMtymfraHtuM6kMgiioTGohQBUgDCZbg8Kp\nFhXAJIJdKxatymP2dACw30PEEGBWZ2NFAgMBAAGjgf8wgfwwDwYDVR0TAQH/BAUw\nAwEB/zAdBgNVHQ4EFgQUjbJJaJ1yCCW5wCf1UJNWSEZx+Y8wDgYDVR0PAQH/BAQD\nAgEGMDYGA1UdEQQvMC2BE2Nhb3BlcmF0b3JAZGlzaWcuc2uGFmh0dHA6Ly93d3cu\nZGlzaWcuc2svY2EwZgYDVR0fBF8wXTAtoCugKYYnaHR0cDovL3d3dy5kaXNpZy5z\nay9jYS9jcmwvY2FfZGlzaWcuY3JsMCygKqAohiZodHRwOi8vY2EuZGlzaWcuc2sv\nY2EvY3JsL2NhX2Rpc2lnLmNybDAaBgNVHSAEEzARMA8GDSuBHpGT5goAAAABAQEw\nDQYJKoZIhvcNAQEFBQADggEBAF00dGFMrzvY/59tWDYcPQuBDRIrRhCA/ec8J9B6\nyKm2fnQwM6M6int0wHl5QpNt/7EpFIKrIYwvF/k/Ji/1WcbvgAa3mkkp7M5+cTxq\nEEHA9tOasnxakZzArFvITV734VP/Q3f8nktnbNfzg9Gg4H8l37iYC5oyOGwwoPP/\nCBUz91BKez6jPiCp3C9WgArtQVCwyfTssuMmRAAOb54GvCKWU3BlxFAKRmukLyeB\nEicTXxChds6KezfqwzlhA5WYOudsiCUI/HloDYd9Yvi0X/vF2Ey9WLw/Q1vUHgFN\nPGO+I++MzVpQuGhU+QqZMxEA4Z7CRneC9VkGjCFMhwnN5ag=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFaTCCA1GgAwIBAgIJAMMDmu5QkG4oMA0GCSqGSIb3DQEBBQUAMFIxCzAJBgNV\nBAYTAlNLMRMwEQYDVQQHEwpCcmF0aXNsYXZhMRMwEQYDVQQKEwpEaXNpZyBhLnMu\nMRkwFwYDVQQDExBDQSBEaXNpZyBSb290IFIxMB4XDTEyMDcxOTA5MDY1NloXDTQy\nMDcxOTA5MDY1NlowUjELMAkGA1UEBhMCU0sxEzARBgNVBAcTCkJyYXRpc2xhdmEx\nEzARBgNVBAoTCkRpc2lnIGEucy4xGTAXBgNVBAMTEENBIERpc2lnIFJvb3QgUjEw\nggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCqw3j33Jijp1pedxiy3QRk\nD2P9m5YJgNXoqqXinCaUOuiZc4yd39ffg/N4T0Dhf9Kn0uXKE5Pn7cZ3Xza1lK/o\nOI7bm+V8u8yN63Vz4STN5qctGS7Y1oprFOsIYgrY3LMATcMjfF9DCCMyEtztDK3A\nfQ+lekLZWnDZv6fXARz2m6uOt0qGeKAeVjGu74IKgEH3G8muqzIm1Cxr7X1r5OJe\nIgpFy4QxTaz+29FHuvlglzmxZcfe+5nkCiKxLU3lSCZpq+Kq8/v8kiky6bM+TR8n\noc2OuRf7JT7JbvN32g0S9l3HuzYQ1VTW8+DiR0jm3hTaYVKvJrT1cU/J19IG32PK\n/yHoWQbgCNWEFVP3Q+V8xaCJmGtzxmjOZd69fwX3se72V6FglcXM6pM6vpmumwKj\nrckWtc7dXpl4fho5frLABaTAgqWjR56M6ly2vGfb5ipN0gTco65F97yLnByn1tUD\n3AjLLhbKXEAz6GfDLuemROoRRRw1ZS0eRWEkG4IupZ0zXWX4Qfkuy5Q/H6MMMSRE\n7cderVC6xkGbrPAXZcD4XW9boAo0PO7X6oifmPmvTiT6l7Jkdtqr9O3jw2Dv1fkC\nyC2fg69naQanMVXVz0tv/wQFx1isXxYb5dKj6zHbHzMVTdDypVP1y+E9Tmgt2BLd\nqvLmTZtJ5cUoobqwWsagtQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1Ud\nDwEB/wQEAwIBBjAdBgNVHQ4EFgQUiQq0OJMa5qvum5EY+fU8PjXQ04IwDQYJKoZI\nhvcNAQEFBQADggIBADKL9p1Kyb4U5YysOMo6CdQbzoaz3evUuii+Eq5FLAR0rBNR\nxVgYZk2C2tXck8An4b58n1KeElb21Zyp9HWc+jcSjxyT7Ff+Bw+r1RL3D65hXlaA\nSfX8MPWbTx9BLxyE04nH4toCdu0Jz2zBuByDHBb6lM19oMgY0sidbvW9adRtPTXo\nHqJPYNcHKfyyo6SdbhWSVhlMCrDpfNIZTUJG7L399ldb3Zh+pE3McgODWF3vkzpB\nemOqfDqo9ayk0d2iLbYq/J8BjuIQscTK5GfbVSUZP/3oNn6z4eGBrxEWi1CXYBmC\nAMBrTXO40RMHPuq2MU/wQppt4hF05ZSsjYSVPCGvxdpHyN85YmLLW1AL14FABZyb\n7bq2ix4Eb5YgOe2kfSnbSM6C3NQCjR0EMVrHS/BsYVLXtFHCgWzN4funodKSds+x\nDzdYpPJScWc/DIh4gInByLUfkmO+p3qKViwaqKactV2zY9ATIKHrkWzQjX2v3wvk\nF7mGnjixlAxYjOBVqjtjbZqJYLhkKpLGN/R+Q0O3c+gB53+XD9fyexn9GtePyfqF\na3qdnom2piiZk4hA9z7NUaPK6u95RyG1/jLix8NRb76AdPCkwzryT+lf3xkK8jsT\nQ6wxpLPn6/wY1gGp8yqPNg7rtLG8t0zJa7+h89n07eLw4+1knj0vllJPgFOL\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFaTCCA1GgAwIBAgIJAJK4iNuwisFjMA0GCSqGSIb3DQEBCwUAMFIxCzAJBgNV\nBAYTAlNLMRMwEQYDVQQHEwpCcmF0aXNsYXZhMRMwEQYDVQQKEwpEaXNpZyBhLnMu\nMRkwFwYDVQQDExBDQSBEaXNpZyBSb290IFIyMB4XDTEyMDcxOTA5MTUzMFoXDTQy\nMDcxOTA5MTUzMFowUjELMAkGA1UEBhMCU0sxEzARBgNVBAcTCkJyYXRpc2xhdmEx\nEzARBgNVBAoTCkRpc2lnIGEucy4xGTAXBgNVBAMTEENBIERpc2lnIFJvb3QgUjIw\nggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCio8QACdaFXS1tFPbCw3Oe\nNcJxVX6B+6tGUODBfEl45qt5WDza/3wcn9iXAng+a0EE6UG9vgMsRfYvZNSrXaNH\nPWSb6WiaxswbP7q+sos0Ai6YVRn8jG+qX9pMzk0DIaPY0jSTVpbLTAwAFjxfGs3I\nx2ymrdMxp7zo5eFm1tL7A7RBZckQrg4FY8aAamkw/dLukO8NJ9+flXP04SXabBbe\nQTg06ov80egEFGEtQX6sx3dOy1FU+16SGBsEWmjGycT6txOgmLcRK7fWV8x8nhfR\nyyX+hk4kLlYMeE2eARKmK6cBZW58Yh2EhN/qwGu1pSqVg8NTEQxzHQuyRpDRQjrO\nQG6Vrf/GlK1ul4SOfW+eioANSW1z4nuSHsPzwfPrLgVv2RvPN3YEyLRa5Beny912\nH9AZdugsBbPWnDTYltxhh5EF5EQIM8HauQhl1K6yNg3ruji6DOWbnuuNZt2Zz9aJ\nQfYEkoopKW1rOhzndX0CcQ7zwOe9yxndnWCywmZgtrEE7snmhrmaZkCo5xHtgUUD\ni/ZnWejBBhG93c+AAk9lQHhcR1DIm+YfgXvkRKhbhZri3lrVx/k6RGZL5DJUfORs\nnLMOPReisjQS1n6yqEm70XooQL6iFh/f5DcfEXP7kAplQ6INfPgGAVUzfbANuPT1\nrqVCV3w2EYx7XsQDnYx5nQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1Ud\nDwEB/wQEAwIBBjAdBgNVHQ4EFgQUtZn4r7CU9eMg1gqtzk5WpC5uQu0wDQYJKoZI\nhvcNAQELBQADggIBACYGXnDnZTPIgm7ZnBc6G3pmsgH2eDtpXi/q/075KMOYKmFM\ntCQSin1tERT3nLXK5ryeJ45MGcipvXrA1zYObYVybqjGom32+nNjf7xueQgcnYqf\nGopTpti72TVVsRHFqQOzVju5hJMiXn7B9hJSi+osZ7z+Nkz1uM/Rs0mSO9MpDpkb\nlvdhuDvEK7Z4bLQjb/D907JedR+Zlais9trhxTF7+9FGs9K8Z7RiVLoJ92Owk6Ka\n+elSLotgEqv89WBW7xBci8QaQtyDW2QOy7W81k/BfDxujRNt+3vrMNDcTa/F1bal\nTFtxyegxvug4BkihGuLq0t4SOVga/4AOgnXmt8kHbA7v/zjxmHHEt38OFdAlab0i\nnSvtBfZGR6ztwPDUO+Ls7pZbkBNOHlY667DvlruWIxG68kOGdGSVyCh13x01utI3\ngzhTODY7z2zp+WsO0PsE6E9312UBeIYMej4hYvF/Y3EMyZ9E26gnonW+boE+18Dr\nG5gPcFw0sorMwIUY6256s/daoQe/qUKS82Ail+QUoQebTnbAjn39pCXHR+3/H3Os\nzMOl6W8KjptlwlCFtaOgUxLMVYdh84GuEEZhvUQhuMI9dM9+JDX6HAcOmz0iyu8x\nL4ysEr3vQCj8KWefshNPZiTEUxnpHikV7+ZtsH8tZ/3zbBt1RqPlShfppNcL\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDVTCCAj2gAwIBAgIESTMAATANBgkqhkiG9w0BAQUFADAyMQswCQYDVQQGEwJD\nTjEOMAwGA1UEChMFQ05OSUMxEzARBgNVBAMTCkNOTklDIFJPT1QwHhcNMDcwNDE2\nMDcwOTE0WhcNMjcwNDE2MDcwOTE0WjAyMQswCQYDVQQGEwJDTjEOMAwGA1UEChMF\nQ05OSUMxEzARBgNVBAMTCkNOTklDIFJPT1QwggEiMA0GCSqGSIb3DQEBAQUAA4IB\nDwAwggEKAoIBAQDTNfc/c3et6FtzF8LRb+1VvG7q6KR5smzDo+/hn7E7SIX1mlwh\nIhAsxYLO2uOabjfhhyzcuQxauohV3/2q2x8x6gHx3zkBwRP9SFIhxFXf2tizVHa6\ndLG3fdfA6PZZxU3Iva0fFNrfWEQlMhkqx35+jq44sDB7R3IJMfAw28Mbdim7aXZO\nV/kbZKKTVrdvmW7bCgScEeOAH8tjlBAKqeFkgjH5jCftppkA9nCTGPihNIaj3XrC\nGHn2emU1z5DrvTOTn1OrczvmmzQgLx3vqR1jGqCA2wMv+SYahtKNu6m+UjqHZ0gN\nv7Sg2Ca+I19zN38m5pIEo3/PIKe38zrKy5nLAgMBAAGjczBxMBEGCWCGSAGG+EIB\nAQQEAwIABzAfBgNVHSMEGDAWgBRl8jGtKvf33VKWCscCwQ7vptU7ETAPBgNVHRMB\nAf8EBTADAQH/MAsGA1UdDwQEAwIB/jAdBgNVHQ4EFgQUZfIxrSr3991SlgrHAsEO\n76bVOxEwDQYJKoZIhvcNAQEFBQADggEBAEs17szkrr/Dbq2flTtLP1se31cpolnK\nOOK5Gv+e5m4y3R6u6jW39ZORTtpC4cMXYFDy0VwmuYK36m3knITnA3kXr5g9lNvH\nugDnuL8BV8F3RTIMO/G0HAiw/VGgod2aHRM2mm23xzy54cXZF/qD1T0VoDy7Hgvi\nyJA/qIYM/PmLXoXLT1tLYhFHxUV8BS9BsZ4QaRuZluBVeftOhpm4lNqGOGqTo+fL\nbuXf6iFViZx9fX+Y9QCJ7uOEwFyWtcVG6kbghVW2G8kS1sHNzYDzAgE8yGnLRUhj\n2JTQ7IUOO04RZfSCjKY9ri4ilAnIXOo8gV0WKgOXFlUJ24pBgp5mmxE=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEHTCCAwWgAwIBAgIQToEtioJl4AsC7j41AkblPTANBgkqhkiG9w0BAQUFADCB\ngTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G\nA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxJzAlBgNV\nBAMTHkNPTU9ETyBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNjEyMDEwMDAw\nMDBaFw0yOTEyMzEyMzU5NTlaMIGBMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3Jl\nYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P\nRE8gQ0EgTGltaXRlZDEnMCUGA1UEAxMeQ09NT0RPIENlcnRpZmljYXRpb24gQXV0\naG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0ECLi3LjkRv3\nUcEbVASY06m/weaKXTuH+7uIzg3jLz8GlvCiKVCZrts7oVewdFFxze1CkU1B/qnI\n2GqGd0S7WWaXUF601CxwRM/aN5VCaTwwxHGzUvAhTaHYujl8HJ6jJJ3ygxaYqhZ8\nQ5sVW7euNJH+1GImGEaaP+vB+fGQV+useg2L23IwambV4EajcNxo2f8ESIl33rXp\n+2dtQem8Ob0y2WIC8bGoPW43nOIv4tOiJovGuFVDiOEjPqXSJDlqR6sA1KGzqSX+\nDT+nHbrTUcELpNqsOO9VUCQFZUaTNE8tja3G1CEZ0o7KBWFxB3NH5YoZEr0ETc5O\nnKVIrLsm9wIDAQABo4GOMIGLMB0GA1UdDgQWBBQLWOWLxkwVN6RAqTCpIb5HNlpW\n/zAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zBJBgNVHR8EQjBAMD6g\nPKA6hjhodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9DZXJ0aWZpY2F0aW9u\nQXV0aG9yaXR5LmNybDANBgkqhkiG9w0BAQUFAAOCAQEAPpiem/Yb6dc5t3iuHXIY\nSdOH5EOC6z/JqvWote9VfCFSZfnVDeFs9D6Mk3ORLgLETgdxb8CPOGEIqB6BCsAv\nIC9Bi5HcSEW88cbeunZrM8gALTFGTO3nnc+IlP8zwFboJIYmuNg4ON8qa90SzMc/\nRxdMosIGlgnW2/4/PEZB31jiVg88O8EckzXZOFKs7sjsLjBOlDW0JB9LeGna8gI4\nzJVSk/BwJVmcIGfE7vmLV2H0knZ9P4SNVbfo5azV8fUZVqZa+5Acr5Pr5RzUZ5dd\nBA6+C4OmF4O5MBKgxTMVBbkN+8cFduPYSo38NBejxiEovjBFMR7HeL5YYTisO+IB\nZQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICiTCCAg+gAwIBAgIQH0evqmIAcFBUTAGem2OZKjAKBggqhkjOPQQDAzCBhTEL\nMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE\nBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxKzApBgNVBAMT\nIkNPTU9ETyBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDgwMzA2MDAw\nMDAwWhcNMzgwMTE4MjM1OTU5WjCBhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdy\nZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09N\nT0RPIENBIExpbWl0ZWQxKzApBgNVBAMTIkNPTU9ETyBFQ0MgQ2VydGlmaWNhdGlv\nbiBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQDR3svdcmCFYX7deSR\nFtSrYpn1PlILBs5BAH+X4QokPB0BBO490o0JlwzgdeT6+3eKKvUDYEs2ixYjFq0J\ncfRK9ChQtP6IHG4/bC8vCVlbpVsLM5niwz2J+Wos77LTBumjQjBAMB0GA1UdDgQW\nBBR1cacZSBm8nZ3qQUfflMRId5nTeTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/\nBAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjEA7wNbeqy3eApyt4jf/7VGFAkK+qDm\nfQjGGoe9GKhzvSbKYAydzpmfz1wPMOG+FDHqAjAU9JM8SaczepBGR7NjfRObTrdv\nGDeAU/7dIOA1mjbRxwG55tzd8/8dLDoWV9mSOdY=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF2DCCA8CgAwIBAgIQTKr5yttjb+Af907YWwOGnTANBgkqhkiG9w0BAQwFADCB\nhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G\nA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxKzApBgNV\nBAMTIkNPTU9ETyBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAwMTE5\nMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgT\nEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMR\nQ09NT0RPIENBIExpbWl0ZWQxKzApBgNVBAMTIkNPTU9ETyBSU0EgQ2VydGlmaWNh\ndGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCR\n6FSS0gpWsawNJN3Fz0RndJkrN6N9I3AAcbxT38T6KhKPS38QVr2fcHK3YX/JSw8X\npz3jsARh7v8Rl8f0hj4K+j5c+ZPmNHrZFGvnnLOFoIJ6dq9xkNfs/Q36nGz637CC\n9BR++b7Epi9Pf5l/tfxnQ3K9DADWietrLNPtj5gcFKt+5eNu/Nio5JIk2kNrYrhV\n/erBvGy2i/MOjZrkm2xpmfh4SDBF1a3hDTxFYPwyllEnvGfDyi62a+pGx8cgoLEf\nZd5ICLqkTqnyg0Y3hOvozIFIQ2dOciqbXL1MGyiKXCJ7tKuY2e7gUYPDCUZObT6Z\n+pUX2nwzV0E8jVHtC7ZcryxjGt9XyD+86V3Em69FmeKjWiS0uqlWPc9vqv9JWL7w\nqP/0uK3pN/u6uPQLOvnoQ0IeidiEyxPx2bvhiWC4jChWrBQdnArncevPDt09qZah\nSL0896+1DSJMwBGB7FY79tOi4lu3sgQiUpWAk2nojkxl8ZEDLXB0AuqLZxUpaVIC\nu9ffUGpVRr+goyhhf3DQw6KqLCGqR84onAZFdr+CGCe01a60y1Dma/RMhnEw6abf\nFobg2P9A3fvQQoh/ozM6LlweQRGBY84YcWsr7KaKtzFcOmpH4MN5WdYgGq/yapiq\ncrxXStJLnbsQ/LBMQeXtHT1eKJ2czL+zUdqnR+WEUwIDAQABo0IwQDAdBgNVHQ4E\nFgQUu69+Aj36pvE8hI6t7jiY7NkyMtQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB\n/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAArx1UaEt65Ru2yyTUEUAJNMnMvl\nwFTPoCWOAvn9sKIN9SCYPBMtrFaisNZ+EZLpLrqeLppysb0ZRGxhNaKatBYSaVqM\n4dc+pBroLwP0rmEdEBsqpIt6xf4FpuHA1sj+nq6PK7o9mfjYcwlYRm6mnPTXJ9OV\n2jeDchzTc+CiR5kDOF3VSXkAKRzH7JsgHAckaVd4sjn8OoSgtZx8jb8uk2Intzna\nFxiuvTwJaP+EmzzV1gsD41eeFPfR60/IvYcjt7ZJQ3mFXLrrkguhxuhoqEwWsRqZ\nCuhTLJK7oQkYdQxlqHvLI7cawiiFwxv/0Cti76R7CZGYZ4wUAc1oBmpjIXUDgIiK\nboHGhfKppC3n9KUkEEeDys30jXlYsQab5xoq2Z0B15R97QNKyvDb6KkBPvVWmcke\njkk9u+UJueBPSZI9FoJAzMxZxuY67RIuaTxslbH9qh17f4a+Hg4yRvv7E491f0yL\nS0Zj/gA0QHDBw7mh3aZw4gSzQbzpgJHqZJx64SIDqZxubw5lT2yHh17zbqD5daWb\nQOhTsiedSrnAdyGN/4fy3ryM7xfft0kL0fJuMAsaDk527RH89elWsn2/x20Kk4yl\n0MC2Hb46TpSi125sC8KKfPog88Tk5c0NqMuRkrF8hey1FGlmDoLnzc7ILaZRfyHB\nNVOFBkpdn627G190\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEvTCCA6WgAwIBAgIBADANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJFVTEn\nMCUGA1UEChMeQUMgQ2FtZXJmaXJtYSBTQSBDSUYgQTgyNzQzMjg3MSMwIQYDVQQL\nExpodHRwOi8vd3d3LmNoYW1iZXJzaWduLm9yZzEiMCAGA1UEAxMZQ2hhbWJlcnMg\nb2YgQ29tbWVyY2UgUm9vdDAeFw0wMzA5MzAxNjEzNDNaFw0zNzA5MzAxNjEzNDRa\nMH8xCzAJBgNVBAYTAkVVMScwJQYDVQQKEx5BQyBDYW1lcmZpcm1hIFNBIENJRiBB\nODI3NDMyODcxIzAhBgNVBAsTGmh0dHA6Ly93d3cuY2hhbWJlcnNpZ24ub3JnMSIw\nIAYDVQQDExlDaGFtYmVycyBvZiBDb21tZXJjZSBSb290MIIBIDANBgkqhkiG9w0B\nAQEFAAOCAQ0AMIIBCAKCAQEAtzZV5aVdGDDg2olUkfzIx1L4L1DZ77F1c2VHfRtb\nunXF/KGIJPov7coISjlUxFF6tdpg6jg8gbLL8bvZkSM/SAFwdakFKq0fcfPJVD0d\nBmpAPrMMhe5cG3nCYsS4No41XQEMIwRHNaqbYE6gZj3LJgqcQKH0XZi/caulAGgq\n7YN6D6IUtdQis4CwPAxaUWktWBiP7Zme8a7ileb2R6jWDA+wWFjbw2Y3npuRVDM3\n0pQcakjJyfKl2qUMI/cjDpwyVV5xnIQFUZot/eZOKjRa3spAN2cMVCFVd9oKDMyX\nroDclDZK9D7ONhMeU+SsTjoF7Nuucpw4i9A5O4kKPnf+dQIBA6OCAUQwggFAMBIG\nA1UdEwEB/wQIMAYBAf8CAQwwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybC5j\naGFtYmVyc2lnbi5vcmcvY2hhbWJlcnNyb290LmNybDAdBgNVHQ4EFgQU45T1sU3p\n26EpW1eLTXYGduHRooowDgYDVR0PAQH/BAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIA\nBzAnBgNVHREEIDAegRxjaGFtYmVyc3Jvb3RAY2hhbWJlcnNpZ24ub3JnMCcGA1Ud\nEgQgMB6BHGNoYW1iZXJzcm9vdEBjaGFtYmVyc2lnbi5vcmcwWAYDVR0gBFEwTzBN\nBgsrBgEEAYGHLgoDATA+MDwGCCsGAQUFBwIBFjBodHRwOi8vY3BzLmNoYW1iZXJz\naWduLm9yZy9jcHMvY2hhbWJlcnNyb290Lmh0bWwwDQYJKoZIhvcNAQEFBQADggEB\nAAxBl8IahsAifJ/7kPMa0QOx7xP5IV8EnNrJpY0nbJaHkb5BkAFyk+cefV/2icZd\np0AJPaxJRUXcLo0waLIJuvvDL8y6C98/d3tGfToSJI6WjzwFCm/SlCgdbQzALogi\n1djPHRPH8EjX1wWnz8dHnjs8NMiAT9QUu/wNUPf6s+xCX6ndbcj0dc97wXImsQEc\nXCz9ek60AcUFV7nnPKoF2YjpB0ZBzu9Bga5Y34OirsrXdx/nADydb47kMgkdTXg0\neDQ8lJsm7U9xxhl6vSAiSFr+S30Dt+dYvsYyTnQeaN2oaFuzPu5ifdmA6Ap1erfu\ntGWaIZDgqtCYvDi1czyL+Nw=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIExTCCA62gAwIBAgIBADANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJFVTEn\nMCUGA1UEChMeQUMgQ2FtZXJmaXJtYSBTQSBDSUYgQTgyNzQzMjg3MSMwIQYDVQQL\nExpodHRwOi8vd3d3LmNoYW1iZXJzaWduLm9yZzEgMB4GA1UEAxMXR2xvYmFsIENo\nYW1iZXJzaWduIFJvb3QwHhcNMDMwOTMwMTYxNDE4WhcNMzcwOTMwMTYxNDE4WjB9\nMQswCQYDVQQGEwJFVTEnMCUGA1UEChMeQUMgQ2FtZXJmaXJtYSBTQSBDSUYgQTgy\nNzQzMjg3MSMwIQYDVQQLExpodHRwOi8vd3d3LmNoYW1iZXJzaWduLm9yZzEgMB4G\nA1UEAxMXR2xvYmFsIENoYW1iZXJzaWduIFJvb3QwggEgMA0GCSqGSIb3DQEBAQUA\nA4IBDQAwggEIAoIBAQCicKLQn0KuWxfH2H3PFIP8T8mhtxOviteePgQKkotgVvq0\nMi+ITaFgCPS3CU6gSS9J1tPfnZdan5QEcOw/Wdm3zGaLmFIoCQLfxS+EjXqXd7/s\nQJ0lcqu1PzKY+7e3/HKE5TWH+VX6ox8Oby4o3Wmg2UIQxvi1RMLQQ3/bvOSiPGpV\neAp3qdjqGTK3L/5cPxvusZjsyq16aUXjlg9V9ubtdepl6DJWk0aJqCWKZQbua795\nB9Dxt6/tLE2Su8CoX6dnfQTyFQhwrJLWfQTSM/tMtgsL+xrJxI0DqX5c8lCrEqWh\nz0hQpe/SyBoT+rB/sYIcd2oPX9wLlY/vQ37mRQklAgEDo4IBUDCCAUwwEgYDVR0T\nAQH/BAgwBgEB/wIBDDA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3JsLmNoYW1i\nZXJzaWduLm9yZy9jaGFtYmVyc2lnbnJvb3QuY3JsMB0GA1UdDgQWBBRDnDafsJ4w\nTcbOX60Qq+UDpfqpFDAOBgNVHQ8BAf8EBAMCAQYwEQYJYIZIAYb4QgEBBAQDAgAH\nMCoGA1UdEQQjMCGBH2NoYW1iZXJzaWducm9vdEBjaGFtYmVyc2lnbi5vcmcwKgYD\nVR0SBCMwIYEfY2hhbWJlcnNpZ25yb290QGNoYW1iZXJzaWduLm9yZzBbBgNVHSAE\nVDBSMFAGCysGAQQBgYcuCgEBMEEwPwYIKwYBBQUHAgEWM2h0dHA6Ly9jcHMuY2hh\nbWJlcnNpZ24ub3JnL2Nwcy9jaGFtYmVyc2lnbnJvb3QuaHRtbDANBgkqhkiG9w0B\nAQUFAAOCAQEAPDtwkfkEVCeR4e3t/mh/YV3lQWVPMvEYBZRqHN4fcNs+ezICNLUM\nbKGKfKX0j//U2K0X1S0E0T9YgOKBWYi+wONGkyT+kL0mojAt6JcmVzWJdJYY9hXi\nryQZVgICsroPFOrGimbBhkVVi76SvpykBMdJPJ7oKXqJ1/6v/2j1pReQvayZzKWG\nVwlnRtvWFsJG8eSpUPWP0ZIV018+xgBJOm5YstHRJw0lyDL4IBHNfTIzSJRUTN3c\necQwn+uOuFW114hcxWokPbLTBQNRxgfvzBRydD1ucs4YKIxKoHflCStFREest2d/\nAYoFWpO+ocH/+OcOZ6RHSXZddZAa9SaP8A==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDqDCCApCgAwIBAgIJAP7c4wEPyUj/MA0GCSqGSIb3DQEBBQUAMDQxCzAJBgNV\nBAYTAkZSMRIwEAYDVQQKDAlEaGlteW90aXMxETAPBgNVBAMMCENlcnRpZ25hMB4X\nDTA3MDYyOTE1MTMwNVoXDTI3MDYyOTE1MTMwNVowNDELMAkGA1UEBhMCRlIxEjAQ\nBgNVBAoMCURoaW15b3RpczERMA8GA1UEAwwIQ2VydGlnbmEwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQDIaPHJ1tazNHUmgh7stL7qXOEm7RFHYeGifBZ4\nQCHkYJ5ayGPhxLGWkv8YbWkj4Sti993iNi+RB7lIzw7sebYs5zRLcAglozyHGxny\ngQcPOJAZ0xH+hrTy0V4eHpbNgGzOOzGTtvKg0KmVEn2lmsxryIRWijOp5yIVUxbw\nzBfsV1/pogqYCd7jX5xv3EjjhQsVWqa6n6xI4wmy9/Qy3l40vhx4XUJbzg4ij02Q\n130yGLMLLGq/jj8UEYkgDncUtT2UCIf3JR7VsmAA7G8qKCVuKj4YYxclPz5EIBb2\nJsglrgVKtOdjLPOMFlN+XPsRGgjBRmKfIrjxwo1p3Po6WAbfAgMBAAGjgbwwgbkw\nDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUGu3+QTmQtCRZvgHyUtVF9lo53BEw\nZAYDVR0jBF0wW4AUGu3+QTmQtCRZvgHyUtVF9lo53BGhOKQ2MDQxCzAJBgNVBAYT\nAkZSMRIwEAYDVQQKDAlEaGlteW90aXMxETAPBgNVBAMMCENlcnRpZ25hggkA/tzj\nAQ/JSP8wDgYDVR0PAQH/BAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIABzANBgkqhkiG\n9w0BAQUFAAOCAQEAhQMeknH2Qq/ho2Ge6/PAD/Kl1NqV5ta+aDY9fm4fTIrv0Q8h\nbV6lUmPOEvjvKtpv6zf+EwLHyzs+ImvaYS5/1HI93TDhHkxAGYwP15zRgzB7mFnc\nfca5DClMoTOi62c6ZYTTluLtdkVwj7Ur3vkj1kluPBS1xp81HlDQwY9qcEQCYsuu\nHWhBp6pX6FOqB9IG9tUUBguRA3UsbHK1YZWaDYu5Def131TN3ubY1gkIl2PlwS6w\nt0QmwCbAr1UwnjvVNioZBPRcHv/PLLf/0P2HQBHVESO7SMAhqaQoLf0V+LBOK/Qw\nWyH8EZE0vkHve52Xdf+XlcCWWC/qu0bXu+TZLg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFnDCCA4SgAwIBAgIBATANBgkqhkiG9w0BAQUFADBjMQswCQYDVQQGEwJGUjET\nMBEGA1UEChMKQ2VydGlub21pczEXMBUGA1UECxMOMDAwMiA0MzM5OTg5MDMxJjAk\nBgNVBAMMHUNlcnRpbm9taXMgLSBBdXRvcml0w6kgUmFjaW5lMB4XDTA4MDkxNzA4\nMjg1OVoXDTI4MDkxNzA4Mjg1OVowYzELMAkGA1UEBhMCRlIxEzARBgNVBAoTCkNl\ncnRpbm9taXMxFzAVBgNVBAsTDjAwMDIgNDMzOTk4OTAzMSYwJAYDVQQDDB1DZXJ0\naW5vbWlzIC0gQXV0b3JpdMOpIFJhY2luZTCCAiIwDQYJKoZIhvcNAQEBBQADggIP\nADCCAgoCggIBAJ2Fn4bT46/HsmtuM+Cet0I0VZ35gb5j2CN2DpdUzZlMGvE5x4jY\nF1AMnmHawE5V3udauHpOd4cN5bjr+p5eex7Ezyh0x5P1FMYiKAT5kcOrJ3NqDi5N\n8y4oH3DfVS9O7cdxbwlyLu3VMpfQ8Vh30WC8Tl7bmoT2R2FFK/ZQpn9qcSdIhDWe\nrP5pqZ56XjUl+rSnSTV3lqc2W+HN3yNw2F1MpQiD8aYkOBOo7C+ooWfHpi2GR+6K\n/OybDnT0K0kCe5B1jPyZOQE51kqJ5Z52qz6WKDgmi92NjMD2AR5vpTESOH2VwnHu\n7XSu5DaiQ3XV8QCb4uTXzEIDS3h65X27uK4uIJPT5GHfceF2Z5c/tt9qc1pkIuVC\n28+BA5PY9OMQ4HL2AHCs8MF6DwV/zzRpRbWT5BnbUhYjBYkOjUjkJW+zeL9i9Qf6\nlSTClrLooyPCXQP8w9PlfMl1I9f09bze5N/NgL+RiH2nE7Q5uiy6vdFrzPOlKO1E\nnn1So2+WLhl+HPNbxxaOu2B9d2ZHVIIAEWBsMsGoOBvrbpgT1u449fCfDu/+MYHB\n0iSVL1N6aaLwD4ZFjliCK0wi1F6g530mJ0jfJUaNSih8hp75mxpZuWW/Bd22Ql09\n5gBIgl4g9xGC3srYn+Y3RyYe63j3YcNBZFgCQfna4NH4+ej9Uji29YnfAgMBAAGj\nWzBZMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBQN\njLZh2kS40RR9w759XkjwzspqsDAXBgNVHSAEEDAOMAwGCiqBegFWAgIAAQEwDQYJ\nKoZIhvcNAQEFBQADggIBACQ+YAZ+He86PtvqrxyaLAEL9MW12Ukx9F1BjYkMTv9s\nov3/4gbIOZ/xWqndIlgVqIrTseYyCYIDbNc/CMf4uboAbbnW/FIyXaR/pDGUu7ZM\nOH8oMDX/nyNTt7buFHAAQCvaR6s0fl6nVjBhK4tDrP22iCj1a7Y+YEq6QpA0Z43q\n619FVDsXrIvkxmUP7tCMXWY5zjKn2BCXwH40nJ+U8/aGH88bc62UeYdocMMzpXDn\n2NU4lG9jeeu/Cg4I58UvD0KgKxRA/yHgBcUn4YQRE7rWhh1BCxMjidPJC+iKunqj\no3M3NYB9Ergzd0A4wPpeMNLytqOx1qKVl4GbUu1pTP+A5FPbVFsDbVRfsbjvJL1v\nnxHDx2TCDyhihWZeGnuyt++uNckZM6i4J9szVb9o4XVIRFb7zdNIu0eJOqxp9YDG\n5ERQL1TEqkPFMTFYvZbF6nVsmnWxTfj3l/+WFvKXTej28xH5On2KOG4Ey+HTRRWq\npdEdnV1j6CTmNhTih60bWfVEm/vXd3wfAXBioSAaosUaKPQhA+4u2cGA6rnZgtZb\ndsLLO7XSAPCjDuGtbkD326C00EauFddEwk01+dIL8hf2rGbVJLJP0RyZwG71fet0\nBLj5TXcJ17TPBzAJ8bgAVtkXFhYKK4bfjwEZGuW7gmP/vgt2Fl43N+bYdJeimUV5\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDkjCCAnqgAwIBAgIRAIW9S/PY2uNp9pTXX8OlRCMwDQYJKoZIhvcNAQEFBQAw\nPTELMAkGA1UEBhMCRlIxETAPBgNVBAoTCENlcnRwbHVzMRswGQYDVQQDExJDbGFz\ncyAyIFByaW1hcnkgQ0EwHhcNOTkwNzA3MTcwNTAwWhcNMTkwNzA2MjM1OTU5WjA9\nMQswCQYDVQQGEwJGUjERMA8GA1UEChMIQ2VydHBsdXMxGzAZBgNVBAMTEkNsYXNz\nIDIgUHJpbWFyeSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANxQ\nltAS+DXSCHh6tlJw/W/uz7kRy1134ezpfgSN1sxvc0NXYKwzCkTsA18cgCSR5aiR\nVhKC9+Ar9NuuYS6JEI1rbLqzAr3VNsVINyPi8Fo3UjMXEuLRYE2+L0ER4/YXJQyL\nkcAbmXuZVg2v7tK8R1fjeUl7NIknJITesezpWE7+Tt9avkGtrAjFGA7v0lPubNCd\nEgETjdyAYveVqUSISnFOYFWe2yMZeVYHDD9jC1yw4r5+FfyUM1hBOHTE4Y+L3yas\nH7WLO7dDWWuwJKZtkIvEcupdM5i3y95ee++U8Rs+yskhwcWYAqqi9lt3m/V+llU0\nHGdpwPFC40es/CgcZlUCAwEAAaOBjDCBiTAPBgNVHRMECDAGAQH/AgEKMAsGA1Ud\nDwQEAwIBBjAdBgNVHQ4EFgQU43Mt38sOKAze3bOkynm4jrvoMIkwEQYJYIZIAYb4\nQgEBBAQDAgEGMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly93d3cuY2VydHBsdXMu\nY29tL0NSTC9jbGFzczIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQCnVM+IRBnL39R/\nAN9WM2K191EBkOvDP9GIROkkXe/nFL0gt5o8AP5tn9uQ3Nf0YtaLcF3n5QRIqWh8\nyfFC82x/xXp8HVGIutIKPidd3i1RTtMTZGnkLuPT55sJmabglZvOGtd/vjzOUrMR\nFcEPF80Du5wlFbqidon8BvEY0JNLDnyCt6X09l/+7UCmnYR0ObncHoUW2ikbhiMA\nybuJfm6AiB4vFLQDJKgybwOaRywwvlbGp0ICcBvqQNi6BQNwB6SW//1IMwrh3KWB\nkJtN3X3n57LNXMhqlfil9o3EXXgIvnsG1knPGTZQIy4I5p4FTUcY1Rbpsda2ENW7\nl7+ijrRU\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDDDCCAfSgAwIBAgIDAQAgMA0GCSqGSIb3DQEBBQUAMD4xCzAJBgNVBAYTAlBM\nMRswGQYDVQQKExJVbml6ZXRvIFNwLiB6IG8uby4xEjAQBgNVBAMTCUNlcnR1bSBD\nQTAeFw0wMjA2MTExMDQ2MzlaFw0yNzA2MTExMDQ2MzlaMD4xCzAJBgNVBAYTAlBM\nMRswGQYDVQQKExJVbml6ZXRvIFNwLiB6IG8uby4xEjAQBgNVBAMTCUNlcnR1bSBD\nQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM6xwS7TT3zNJc4YPk/E\njG+AanPIW1H4m9LcuwBcsaD8dQPugfCI7iNS6eYVM42sLQnFdvkrOYCJ5JdLkKWo\nePhzQ3ukYbDYWMzhbGZ+nPMJXlVjhNWo7/OxLjBos8Q82KxujZlakE403Daaj4GI\nULdtlkIJ89eVgw1BS7Bqa/j8D35in2fE7SZfECYPCE/wpFcozo+47UX2bu4lXapu\nOb7kky/ZR6By6/qmW6/KUz/iDsaWVhFu9+lmqSbYf5VT7QqFiLpPKaVCjF62/IUg\nAKpoC6EahQGcxEZjgoi2IrHu/qpGWX7PNSzVttpd90gzFFS269lvzs2I1qsb2pY7\nHVkCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEA\nuI3O7+cUus/usESSbLQ5PqKEbq24IXfS1HeCh+YgQYHu4vgRt2PRFze+GXYkHAQa\nTOs9qmdvLdTN/mUxcMUbpgIKumB7bVjCmkn+YzILa+M6wKyrO7Do0wlRjBCDxjTg\nxSvgGrZgFCdsMneMvLJymM/NzD+5yCRCFNZX/OYmQ6kd5YCQzgNUKD73P9P4Te1q\nCjqTE5s7FCMTY5w/0YcneeVMUeMBrYVdGjux1XMQpNPyvG5k9VpWkKjHDkx0Dy5x\nO/fIR/RpbxXyEV6DHpx8Uq79AtoSqFlnGNu8cN2bsWntgM6JQEhqDjXKKWYVIZQs\n6GAqm4VKQPNriiTsBhYscw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDuzCCAqOgAwIBAgIDBETAMA0GCSqGSIb3DQEBBQUAMH4xCzAJBgNVBAYTAlBM\nMSIwIAYDVQQKExlVbml6ZXRvIFRlY2hub2xvZ2llcyBTLkEuMScwJQYDVQQLEx5D\nZXJ0dW0gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIjAgBgNVBAMTGUNlcnR1bSBU\ncnVzdGVkIE5ldHdvcmsgQ0EwHhcNMDgxMDIyMTIwNzM3WhcNMjkxMjMxMTIwNzM3\nWjB+MQswCQYDVQQGEwJQTDEiMCAGA1UEChMZVW5pemV0byBUZWNobm9sb2dpZXMg\nUy5BLjEnMCUGA1UECxMeQ2VydHVtIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSIw\nIAYDVQQDExlDZXJ0dW0gVHJ1c3RlZCBOZXR3b3JrIENBMIIBIjANBgkqhkiG9w0B\nAQEFAAOCAQ8AMIIBCgKCAQEA4/t9o3K6wvDJFIf1awFO4W5AB7ptJ11/91sts1rH\nUV+rpDKmYYe2bg+G0jACl/jXaVehGDldamR5xgFZrDwxSjh80gTSSyjoIF87B6LM\nTXPb865Px1bVWqeWifrzq2jUI4ZZJ88JJ7ysbnKDHDBy3+Ci6dLhdHUZvSqeexVU\nBBvXQzmtVSjF4hq79MDkrjhJM8x2hZ85RdKknvISjFH4fOQtf/WsX+sWn7Et0brM\nkUJ3TCXJkDhv2/DM+44el1k+1WBO5gUo7Ul5E0u6SNsv+XLTOcr+H9g0cvW0QM8x\nAcPs3hEtF10fuFDRXhmnad4HMyjKUJX5p1TLVIZQRan5SQIDAQABo0IwQDAPBgNV\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQIds3LB/8k9sXN7buQvOKEN0Z19zAOBgNV\nHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQEFBQADggEBAKaorSLOAT2mo/9i0Eidi15y\nsHhE49wcrwn9I0j6vSrEuVUEtRCjjSfeC4Jj0O7eDDd5QVsisrCaQVymcODU0HfL\nI9MA4GxWL+FpDQ3Zqr8hgVDZBqWo/5U30Kr+4rP1mS1FhIrlQgnXdAIv94nYmem8\nJ9RHjboNRhx3zxSkHLmkMcScKHQDNP8zGSal6Q10tz6XxnboJ5ajZt3hrvJBW8qY\nVoNzcOSGGtIxQbovvi0TWnZvTuhOgQ4/WwMioBK+ZlgRSssDxLQqKi2WF+A5VLxI\n03YnnZotBqbJ7DnSq9ufmgsnAjUpsUCV5/nonFWIGUbWtzT1fs45mtk48VH3Tyw=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIHTzCCBTegAwIBAgIJAKPaQn6ksa7aMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYD\nVQQGEwJFVTFDMEEGA1UEBxM6TWFkcmlkIChzZWUgY3VycmVudCBhZGRyZXNzIGF0\nIHd3dy5jYW1lcmZpcm1hLmNvbS9hZGRyZXNzKTESMBAGA1UEBRMJQTgyNzQzMjg3\nMRswGQYDVQQKExJBQyBDYW1lcmZpcm1hIFMuQS4xKTAnBgNVBAMTIENoYW1iZXJz\nIG9mIENvbW1lcmNlIFJvb3QgLSAyMDA4MB4XDTA4MDgwMTEyMjk1MFoXDTM4MDcz\nMTEyMjk1MFowga4xCzAJBgNVBAYTAkVVMUMwQQYDVQQHEzpNYWRyaWQgKHNlZSBj\ndXJyZW50IGFkZHJlc3MgYXQgd3d3LmNhbWVyZmlybWEuY29tL2FkZHJlc3MpMRIw\nEAYDVQQFEwlBODI3NDMyODcxGzAZBgNVBAoTEkFDIENhbWVyZmlybWEgUy5BLjEp\nMCcGA1UEAxMgQ2hhbWJlcnMgb2YgQ29tbWVyY2UgUm9vdCAtIDIwMDgwggIiMA0G\nCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCvAMtwNyuAWko6bHiUfaN/Gh/2NdW9\n28sNRHI+JrKQUrpjOyhYb6WzbZSm891kDFX29ufyIiKAXuFixrYp4YFs8r/lfTJq\nVKAyGVn+H4vXPWCGhSRv4xGzdz4gljUha7MI2XAuZPeEklPWDrCQiorjh40G072Q\nDuKZoRuGDtqaCrsLYVAGUvGef3bsyw/QHg3PmTA9HMRFEFis1tPo1+XqxQEHd9ZR\n5gN/ikilTWh1uem8nk4ZcfUyS5xtYBkL+8ydddy/Js2Pk3g5eXNeJQ7KXOt3EgfL\nZEFHcpOrUMPrCXZkNNI5t3YRCQ12RcSprj1qr7V9ZS+UWBDsXHyvfuK2GNnQm05a\nSd+pZgvMPMZ4fKecHePOjlO+Bd5gD2vlGts/4+EhySnB8esHnFIbAURRPHsl18Tl\nUlRdJQfKFiC4reRB7noI/plvg6aRArBsNlVq5331lubKgdaX8ZSD6e2wsWsSaR6s\n+12pxZjptFtYer49okQ6Y1nUCyXeG0+95QGezdIp1Z8XGQpvvwyQ0wlf2eOKNcx5\nWk0ZN5K3xMGtr/R5JJqyAQuxr1yW84Ay+1w9mPGgP0revq+ULtlVmhduYJ1jbLhj\nya6BXBg14JC7vjxPNyK5fuvPnnchpj04gftI2jE9K+OJ9dC1vX7gUMQSibMjmhAx\nhduub+84Mxh2EQIDAQABo4IBbDCCAWgwEgYDVR0TAQH/BAgwBgEB/wIBDDAdBgNV\nHQ4EFgQU+SSsD7K1+HnA+mCIG8TZTQKeFxkwgeMGA1UdIwSB2zCB2IAU+SSsD7K1\n+HnA+mCIG8TZTQKeFxmhgbSkgbEwga4xCzAJBgNVBAYTAkVVMUMwQQYDVQQHEzpN\nYWRyaWQgKHNlZSBjdXJyZW50IGFkZHJlc3MgYXQgd3d3LmNhbWVyZmlybWEuY29t\nL2FkZHJlc3MpMRIwEAYDVQQFEwlBODI3NDMyODcxGzAZBgNVBAoTEkFDIENhbWVy\nZmlybWEgUy5BLjEpMCcGA1UEAxMgQ2hhbWJlcnMgb2YgQ29tbWVyY2UgUm9vdCAt\nIDIwMDiCCQCj2kJ+pLGu2jAOBgNVHQ8BAf8EBAMCAQYwPQYDVR0gBDYwNDAyBgRV\nHSAAMCowKAYIKwYBBQUHAgEWHGh0dHA6Ly9wb2xpY3kuY2FtZXJmaXJtYS5jb20w\nDQYJKoZIhvcNAQEFBQADggIBAJASryI1wqM58C7e6bXpeHxIvj99RZJe6dqxGfwW\nPJ+0W2aeaufDuV2I6A+tzyMP3iU6XsxPpcG1Lawk0lgH3qLPaYRgM+gQDROpI9CF\n5Y57pp49chNyM/WqfcZjHwj0/gF/JM8rLFQJ3uIrbZLGOU8W6jx+ekbURWpGqOt1\nglanq6B8aBMz9p0w8G8nOSQjKpD9kCk18pPfNKXG9/jvjA9iSnyu0/VU+I22mlaH\nFoI6M6taIgj3grrqLuBHmrS1RaMFO9ncLkVAO+rcf+g769HsJtg1pDDFOqxXnrN2\npSB7+R5KBWIBpih1YJeSDW4+TTdDDZIVnBgizVGZoCkaPF+KMjNbMMeJL0eYD6MD\nxvbxrN8y8NmBGuScvfaAFPDRLLmF9dijscilIeUcE5fuDr3fKanvNFNb0+RqE4QG\ntjICxFKuItLcsiFCGtpA8CnJ7AoMXOLQusxI0zcKzBIKinmwPQN/aUv0NCB9szTq\njktk9T79syNnFQ0EuPAtwQlRPLJsFfClI9eDdOTlLsn+mCdCxqvGnrDQWzilm1De\nfhiYtUU79nm06PcaewaD+9CL2rvHvRirCG88gGtAPxkZumWK5r7VXNM21+9AUiRg\nOGcEMeyP84LG3rlV8zsxkVrctQgVrXYlCg17LofiDKYGvCYQbTed7N14jHyAxfDZ\nd0jQ\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC\nQ04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g\nQ2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0\naW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa\nFw0zMDA4MzEwNzExMjVaMIGKMQswCQYDVQQGEwJDTjEyMDAGA1UECgwpQ2hpbmEg\nSW50ZXJuZXQgTmV0d29yayBJbmZvcm1hdGlvbiBDZW50ZXIxRzBFBgNVBAMMPkNo\naW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24gQ2VudGVyIEVWIENlcnRp\nZmljYXRlcyBSb290MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm35z\n7r07eKpkQ0H1UN+U8i6yjUqORlTSIRLIOTJCBumD1Z9S7eVnAztUwYyZmczpwA//\nDdmEEbK40ctb3B75aDFk4Zv6dOtouSCV98YPjUesWgbdYavi7NifFy2cyjw1l1Vx\nzUOFsUcW9SxTgHbP0wBkvUCZ3czY28Sf1hNfQYOL+Q2HklY0bBoQCxfVWhyXWIQ8\nhBouXJE0bhlffxdpxWXvayHG1VA6v2G5BY3vbzQ6sm8UY78WO5upKv23KzhmBsUs\n4qpnHkWnjQRmQvaPK++IIGmPMowUc9orhpFjIpryp9vOiYurXccUwVswah+xt54u\ngQEC7c+WXmPbqOY4twIDAQABo2MwYTAfBgNVHSMEGDAWgBR8cks5x8DbYqVPm6oY\nNJKiyoOCWTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4E\nFgQUfHJLOcfA22KlT5uqGDSSosqDglkwDQYJKoZIhvcNAQEFBQADggEBACrDx0M3\nj92tpLIM7twUbY8opJhJywyA6vPtI2Z1fcXTIWd50XPFtQO3WKwMVC/GVhMPMdoG\n52U7HW8228gd+f2ABsqjPWYWqJ1MFn3AlUa1UeTiH9fqBk1jjZaM7+czV0I664zB\nechNdn3e9rG3geCg+aF4RhcaVpjwTj2rHO3sOdwHSPdj/gauwqRcalsyiMXHM4Ws\nZkJHwlgkmeHlPuV1LI5D1l08eB6olYIpUNHRFrrvwb562bTYzB5MRuF3sTGrvSrI\nzo9uoV1/A3U05K2JRVRevq4opbs/eHnrc7MKDf2+yfdWrPa37S+bISnHOLaVxATy\nwy39FCqQmbkHzJ8=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDqzCCApOgAwIBAgIRAMcoRwmzuGxFjB36JPU2TukwDQYJKoZIhvcNAQEFBQAw\nPDEbMBkGA1UEAxMSQ29tU2lnbiBTZWN1cmVkIENBMRAwDgYDVQQKEwdDb21TaWdu\nMQswCQYDVQQGEwJJTDAeFw0wNDAzMjQxMTM3MjBaFw0yOTAzMTYxNTA0NTZaMDwx\nGzAZBgNVBAMTEkNvbVNpZ24gU2VjdXJlZCBDQTEQMA4GA1UEChMHQ29tU2lnbjEL\nMAkGA1UEBhMCSUwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDGtWhf\nHZQVw6QIVS3joFd67+l0Kru5fFdJGhFeTymHDEjWaueP1H5XJLkGieQcPOqs49oh\ngHMhCu95mGwfCP+hUH3ymBvJVG8+pSjsIQQPRbsHPaHA+iqYHU4Gk/v1iDurX8sW\nv+bznkqH7Rnqwp9D5PGBpX8QTz7RSmKtUxvLg/8HZaWSLWapW7ha9B20IZFKF3ue\nMv5WJDmyVIRD9YTC2LxBkMyd1mja6YJQqTtoz7VdApRgFrFD2UNd3V2Hbuq7s8lr\n9gOUCXDeFhF6K+h2j0kQmHe5Y1yLM5d19guMsqtb3nQgJT/j8xH5h2iGNXHDHYwt\n6+UarA9z1YJZQIDTAgMBAAGjgacwgaQwDAYDVR0TBAUwAwEB/zBEBgNVHR8EPTA7\nMDmgN6A1hjNodHRwOi8vZmVkaXIuY29tc2lnbi5jby5pbC9jcmwvQ29tU2lnblNl\nY3VyZWRDQS5jcmwwDgYDVR0PAQH/BAQDAgGGMB8GA1UdIwQYMBaAFMFL7XC29z58\nADsAj8c+DkWfHl3sMB0GA1UdDgQWBBTBS+1wtvc+fAA7AI/HPg5Fnx5d7DANBgkq\nhkiG9w0BAQUFAAOCAQEAFs/ukhNQq3sUnjO2QiBq1BW9Cav8cujvR3qQrFHBZE7p\niL1DRYHjZiM/EoZNGeQFsOY3wo3aBijJD4mkU6l1P7CW+6tMM1X5eCZGbxs2mPtC\ndsGCuY7e+0X5YxtiOzkGynd6qDwJz2w2PQ8KRUtpFhpFfTMDZflScZAmlaxMDPWL\nkz/MdXSFmLr/YnpNH4n+rr2UAJm/EaXc4HnFFgt9AmEd6oX5AhVP51qJThRv4zdL\nhfXBPGHg/QVBspJ/wx2g0K5SZGBrGMYmnNj1ZOQ2GmKfig8+/21OGVZOIJFsnzQz\nOjRXUDpvgV4GxvU+fE6OK85lBi5d0ipTdF7Tbieejw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb\nMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow\nGAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj\nYXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL\nMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE\nBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM\nGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP\nADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua\nBtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe\n3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4\nYgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR\nrOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm\nez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU\noBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF\nMAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v\nQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t\nb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF\nAAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q\nGE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz\nRt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2\nG9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi\nl2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3\nsmPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEPzCCAyegAwIBAgIBATANBgkqhkiG9w0BAQUFADB+MQswCQYDVQQGEwJHQjEb\nMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow\nGAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEkMCIGA1UEAwwbU2VjdXJlIENlcnRp\nZmljYXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVow\nfjELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G\nA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxJDAiBgNV\nBAMMG1NlY3VyZSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMBxM4KK0HDrc4eCQNUd5MvJDkKQ+d40uaG6EfQlhfPM\ncm3ye5drswfxdySRXyWP9nQ95IDC+DwN879A6vfIUtFyb+/Iq0G4bi4XKpVpDM3S\nHpR7LZQdqnXXs5jLrLxkU0C8j6ysNstcrbvd4JQX7NFc0L/vpZXJkMWwrPsbQ996\nCF23uPJAGysnnlDOXmWCiIxe004MeuoIkbY2qitC++rCoznl2yY4rYsK7hljxxwk\n3wN42ubqwUcaCwtGCd0C/N7Lh1/XMGNooa7cMqG6vv5Eq2i2pRcV/b3Vp6ea5EQz\n6YiO/O1R65NxTq0B50SOqy3LqP4BSUjwwN3HaNiS/j0CAwEAAaOBxzCBxDAdBgNV\nHQ4EFgQUPNiTiMLAggnMAZkGkyDpnnAJY08wDgYDVR0PAQH/BAQDAgEGMA8GA1Ud\nEwEB/wQFMAMBAf8wgYEGA1UdHwR6MHgwO6A5oDeGNWh0dHA6Ly9jcmwuY29tb2Rv\nY2EuY29tL1NlY3VyZUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDmgN6A1hjNodHRw\nOi8vY3JsLmNvbW9kby5uZXQvU2VjdXJlQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww\nDQYJKoZIhvcNAQEFBQADggEBAIcBbSMdflsXfcFhMs+P5/OKlFlm4J4oqF7Tt/Q0\n5qo5spcWxYJvMqTpjOev/e/C6LlLqqP05tqNZSH7uoDrJiiFGv45jN5bBAS0VPmj\nZ55B+glSzAVIqMk/IQQezkhr/IXownuvf7fM+F86/TXGDe+X3EyrEeFryzHRbPtI\ngKvcnDe4IRRLDXE97IMzbtFuMhbsmMcWi1mmNKsFVy2T96oTy9IT4rcuO81rUBcJ\naD61JlfutuC23bkpgHl9j6PwpCikFcSF9CfUa7/lXORlAnZUtOM3ZiTTGWHIUhDl\nizeauan5Hb/qmZJhlv8BzaFfDbxxvA6sCx1HRR3B7Hzs/Sk=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEQzCCAyugAwIBAgIBATANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJHQjEb\nMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow\nGAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDElMCMGA1UEAwwcVHJ1c3RlZCBDZXJ0\naWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTla\nMH8xCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO\nBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSUwIwYD\nVQQDDBxUcnVzdGVkIENlcnRpZmljYXRlIFNlcnZpY2VzMIIBIjANBgkqhkiG9w0B\nAQEFAAOCAQ8AMIIBCgKCAQEA33FvNlhTWvI2VFeAxHQIIO0Yfyod5jWaHiWsnOWW\nfnJSoBVC21ndZHoa0Lh73TkVvFVIxO06AOoxEbrycXQaZ7jPM8yoMa+j49d/vzMt\nTGo87IvDktJTdyR0nAducPy9C1t2ul/y/9c3S0pgePfw+spwtOpZqqPOSC+pw7IL\nfhdyFgymBwwbOM/JYrc/oJOlh0Hyt3BAd9i+FHzjqMB6juljatEPmsbS9Is6FARW\n1O24zG71++IsWL1/T2sr92AkWCTOJu80kTrV44HQsvAEAtdbtz6SrGsSivnkBbA7\nkUlcsutT6vifR4buv5XAwAaf0lteERv0xwQ1KdJVXOTt6wIDAQABo4HJMIHGMB0G\nA1UdDgQWBBTFe1i97doladL3WRaoszLAeydb9DAOBgNVHQ8BAf8EBAMCAQYwDwYD\nVR0TAQH/BAUwAwEB/zCBgwYDVR0fBHwwejA8oDqgOIY2aHR0cDovL2NybC5jb21v\nZG9jYS5jb20vVHJ1c3RlZENlcnRpZmljYXRlU2VydmljZXMuY3JsMDqgOKA2hjRo\ndHRwOi8vY3JsLmNvbW9kby5uZXQvVHJ1c3RlZENlcnRpZmljYXRlU2VydmljZXMu\nY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQDIk4E7ibSvuIQSTI3S8NtwuleGFTQQuS9/\nHrCoiWChisJ3DFBKmwCL2Iv0QeLQg4pKHBQGsKNoBXAxMKdTmw7pSqBYaWcOrp32\npSxBvzwGa+RZzG0Q8ZZvH9/0BAKkn0U+yNj6NkZEUD+Cl5EfKNsYEYwq5GWDVxIS\njBc/lDb+XbDABHcTuPQV1T84zJQ6VdCsmPW6AF/ghhmBeC8owH7TzEIK9a5QoNE+\nxqFx7D+gIIxmOom0jtTYsU0lR+4viMi14QVFwL4Ucd56/Y57fU0IlqUSc/Atyjcn\ndBInTMu2l+nZrghtWjlA3QVHdWpaIbOjGM9O9y5Xt5hwXsjEeLBi\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDoTCCAomgAwIBAgILBAAAAAABD4WqLUgwDQYJKoZIhvcNAQEFBQAwOzEYMBYG\nA1UEChMPQ3liZXJ0cnVzdCwgSW5jMR8wHQYDVQQDExZDeWJlcnRydXN0IEdsb2Jh\nbCBSb290MB4XDTA2MTIxNTA4MDAwMFoXDTIxMTIxNTA4MDAwMFowOzEYMBYGA1UE\nChMPQ3liZXJ0cnVzdCwgSW5jMR8wHQYDVQQDExZDeWJlcnRydXN0IEdsb2JhbCBS\nb290MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA+Mi8vRRQZhP/8NN5\n7CPytxrHjoXxEnOmGaoQ25yiZXRadz5RfVb23CO21O1fWLE3TdVJDm71aofW0ozS\nJ8bi/zafmGWgE07GKmSb1ZASzxQG9Dvj1Ci+6A74q05IlG2OlTEQXO2iLb3VOm2y\nHLtgwEZLAfVJrn5GitB0jaEMAs7u/OePuGtm839EAL9mJRQr3RAwHQeWP032a7iP\nt3sMpTjr3kfb1V05/Iin89cqdPHoWqI7n1C6poxFNcJQZZXcY4Lv3b93TZxiyWNz\nFtApD0mpSPCzqrdsxacwOUBdrsTiXSZT8M4cIwhhqJQZugRiQOwfOHB3EgZxpzAY\nXSUnpQIDAQABo4GlMIGiMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/\nMB0GA1UdDgQWBBS2CHsNesysIEyGVjJez6tuhS1wVzA/BgNVHR8EODA2MDSgMqAw\nhi5odHRwOi8vd3d3Mi5wdWJsaWMtdHJ1c3QuY29tL2NybC9jdC9jdHJvb3QuY3Js\nMB8GA1UdIwQYMBaAFLYIew16zKwgTIZWMl7Pq26FLXBXMA0GCSqGSIb3DQEBBQUA\nA4IBAQBW7wojoFROlZfJ+InaRcHUowAl9B8Tq7ejhVhpwjCt2BWKLePJzYFa+HMj\nWqd8BfP9IjsO0QbE2zZMcwSO5bAi5MXzLqXZI+O4Tkogp24CJJ8iYGd7ix1yCcUx\nXOl5n4BHPa2hCwcUPUf/A2kaDAtE52Mlp3+yybh2hO0j9n0Hq0V+09+zv+mKts2o\nomcrUtW3ZfA5TGOgkXmTUg9U3YO7n9GPp1Nzw8v/MOx8BLjYRB+TX3EJIrduPuoc\nA06dGiBh+4E37F78CkWr1+cXVdCg6mCbpvbjjFspwgZgFJ0tl0ypkxWdYcQBX0jW\nWL1WMRJOEcgh4LMRkWXbtKaIOM5V\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEMzCCAxugAwIBAgIDCYPzMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNVBAYTAkRF\nMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxJzAlBgNVBAMMHkQtVFJVU1QgUm9vdCBD\nbGFzcyAzIENBIDIgMjAwOTAeFw0wOTExMDUwODM1NThaFw0yOTExMDUwODM1NTha\nME0xCzAJBgNVBAYTAkRFMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxJzAlBgNVBAMM\nHkQtVFJVU1QgUm9vdCBDbGFzcyAzIENBIDIgMjAwOTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBANOySs96R+91myP6Oi/WUEWJNTrGa9v+2wBoqOADER03\nUAifTUpolDWzU9GUY6cgVq/eUXjsKj3zSEhQPgrfRlWLJ23DEE0NkVJD2IfgXU42\ntSHKXzlABF9bfsyjxiupQB7ZNoTWSPOSHjRGICTBpFGOShrvUD9pXRl/RcPHAY9R\nySPocq60vFYJfxLLHLGvKZAKyVXMD9O0Gu1HNVpK7ZxzBCHQqr0ME7UAyiZsxGsM\nlFqVlNpQmvH/pStmMaTJOKDfHR+4CS7zp+hnUquVH+BGPtikw8paxTGA6Eian5Rp\n/hnd2HN8gcqW3o7tszIFZYQ05ub9VxC1X3a/L7AQDcUCAwEAAaOCARowggEWMA8G\nA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFP3aFMSfMN4hvR5COfyrYyNJ4PGEMA4G\nA1UdDwEB/wQEAwIBBjCB0wYDVR0fBIHLMIHIMIGAoH6gfIZ6bGRhcDovL2RpcmVj\ndG9yeS5kLXRydXN0Lm5ldC9DTj1ELVRSVVNUJTIwUm9vdCUyMENsYXNzJTIwMyUy\nMENBJTIwMiUyMDIwMDksTz1ELVRydXN0JTIwR21iSCxDPURFP2NlcnRpZmljYXRl\ncmV2b2NhdGlvbmxpc3QwQ6BBoD+GPWh0dHA6Ly93d3cuZC10cnVzdC5uZXQvY3Js\nL2QtdHJ1c3Rfcm9vdF9jbGFzc18zX2NhXzJfMjAwOS5jcmwwDQYJKoZIhvcNAQEL\nBQADggEBAH+X2zDI36ScfSF6gHDOFBJpiBSVYEQBrLLpME+bUMJm2H6NMLVwMeni\nacfzcNsgFYbQDfC+rAF1hM5+n02/t2A7nPPKHeJeaNijnZflQGDSNiH+0LS4F9p0\no3/U37CYAqxva2ssJSRyoWXuJVrl5jLn8t+rSfrzkGkj2wTZ51xY/GXUl77M/C4K\nzCUqNQT4YJEVdT1B/yMfGchs64JTBKbkTCJNjYy6zltz7GRUUG3RnFX7acM2w4y8\nPIWmawomDeCTmGCufsYkl4phX5GOZpIJhzbNi5stPvZR1FDUWSi9g/LMKHtThm3Y\nJohw1+qRzT65ysCQblrGXnRl11z+o+I=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEQzCCAyugAwIBAgIDCYP0MA0GCSqGSIb3DQEBCwUAMFAxCzAJBgNVBAYTAkRF\nMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxKjAoBgNVBAMMIUQtVFJVU1QgUm9vdCBD\nbGFzcyAzIENBIDIgRVYgMjAwOTAeFw0wOTExMDUwODUwNDZaFw0yOTExMDUwODUw\nNDZaMFAxCzAJBgNVBAYTAkRFMRUwEwYDVQQKDAxELVRydXN0IEdtYkgxKjAoBgNV\nBAMMIUQtVFJVU1QgUm9vdCBDbGFzcyAzIENBIDIgRVYgMjAwOTCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAJnxhDRwui+3MKCOvXwEz75ivJn9gpfSegpn\nljgJ9hBOlSJzmY3aFS3nBfwZcyK3jpgAvDw9rKFs+9Z5JUut8Mxk2og+KbgPCdM0\n3TP1YtHhzRnp7hhPTFiu4h7WDFsVWtg6uMQYZB7jM7K1iXdODL/ZlGsTl28So/6Z\nqQTMFexgaDbtCHu39b+T7WYxg4zGcTSHThfqr4uRjRxWQa4iN1438h3Z0S0NL2lR\np75mpoo6Kr3HGrHhFPC+Oh25z1uxav60sUYgovseO3Dvk5h9jHOW8sXvhXCtKSb8\nHgQ+HKDYD8tSg2J87otTlZCpV6LqYQXY+U3EJ/pure3511H3a6UCAwEAAaOCASQw\nggEgMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNOUikxiEyoZLsyvcop9Ntea\nHNxnMA4GA1UdDwEB/wQEAwIBBjCB3QYDVR0fBIHVMIHSMIGHoIGEoIGBhn9sZGFw\nOi8vZGlyZWN0b3J5LmQtdHJ1c3QubmV0L0NOPUQtVFJVU1QlMjBSb290JTIwQ2xh\nc3MlMjAzJTIwQ0ElMjAyJTIwRVYlMjAyMDA5LE89RC1UcnVzdCUyMEdtYkgsQz1E\nRT9jZXJ0aWZpY2F0ZXJldm9jYXRpb25saXN0MEagRKBChkBodHRwOi8vd3d3LmQt\ndHJ1c3QubmV0L2NybC9kLXRydXN0X3Jvb3RfY2xhc3NfM19jYV8yX2V2XzIwMDku\nY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQA07XtaPKSUiO8aEXUHL7P+PPoeUSbrh/Yp\n3uDx1MYkCenBz1UbtDDZzhr+BlGmFaQt77JLvyAoJUnRpjZ3NOhk31KxEcdzes05\nnsKtjHEh8lprr988TlWvsoRlFIm5d8sqMb7Po23Pb0iUMkZv53GMoKaEGTcH8gNF\nCSuGdXzfX2lXANtu2KZyIktQ1HWYVt+3GP9DQ1CuekR78HlR10M9p9OB0/DJT7na\nxpeG0ILD5EJt/rDiZE4OJudANCa1CInXCGNjOCd1HjPqbqjdn5lPdE2BiYBL3ZqX\nKVwvvoFBuYz/6n1gBp7N1z3TLqMVvKjmJuVvw9y4AyHqnxbxLFS1\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECTCCAvGgAwIBAgIQDV6ZCtadt3js2AdWO4YV2TANBgkqhkiG9w0BAQUFADBb\nMQswCQYDVQQGEwJVUzEgMB4GA1UEChMXRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3Qx\nETAPBgNVBAsTCERTVCBBQ0VTMRcwFQYDVQQDEw5EU1QgQUNFUyBDQSBYNjAeFw0w\nMzExMjAyMTE5NThaFw0xNzExMjAyMTE5NThaMFsxCzAJBgNVBAYTAlVTMSAwHgYD\nVQQKExdEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdDERMA8GA1UECxMIRFNUIEFDRVMx\nFzAVBgNVBAMTDkRTVCBBQ0VTIENBIFg2MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A\nMIIBCgKCAQEAuT31LMmU3HWKlV1j6IR3dma5WZFcRt2SPp/5DgO0PWGSvSMmtWPu\nktKe1jzIDZBfZIGxqAgNTNj50wUoUrQBJcWVHAx+PhCEdc/BGZFjz+iokYi5Q1K7\ngLFViYsx+tC3dr5BPTCapCIlF3PoHuLTrCq9Wzgh1SpL11V94zpVvddtawJXa+ZH\nfAjIgrrep4c9oW24MFbCswKBXy314powGCi4ZtPLAZZv6opFVdbgnf9nKxcCpk4a\nahELfrd755jWjHZvwTvbUJN+5dCOHze4vbrGn2zpfDPyMjwmR/onJALJfh1biEIT\najV8fTXpLmaRcpPVMibEdPVTo7NdmvYJywIDAQABo4HIMIHFMA8GA1UdEwEB/wQF\nMAMBAf8wDgYDVR0PAQH/BAQDAgHGMB8GA1UdEQQYMBaBFHBraS1vcHNAdHJ1c3Rk\nc3QuY29tMGIGA1UdIARbMFkwVwYKYIZIAWUDAgEBATBJMEcGCCsGAQUFBwIBFjto\ndHRwOi8vd3d3LnRydXN0ZHN0LmNvbS9jZXJ0aWZpY2F0ZXMvcG9saWN5L0FDRVMt\naW5kZXguaHRtbDAdBgNVHQ4EFgQUCXIGThhDD+XWzMNqizF7eI+og7gwDQYJKoZI\nhvcNAQEFBQADggEBAKPYjtay284F5zLNAdMEA+V25FYrnJmQ6AgwbN99Pe7lv7Uk\nQIRJ4dEorsTCOlMwiPH1d25Ryvr/ma8kXxug/fKshMrfqfBfBC6tFr8hlxCBPeP/\nh40y3JTlR4peahPJlJU90u7INJXQgNStMgiAVDzgvVJT11J8smk/f3rPanTK+gQq\nnExaBqXpIK1FZg9p8d2/6eMyi/rgwYZNcjwu2JN4Cir42NInPRmJX1p7ijvMDNpR\nrscL9yuwNwXsvFcj4jjSm2jzVhKIT0J8uDHEtdvkyCE06UgRNe76x5JXxZ805Mf2\n9w4LTJxoeHtxMcfrHuBnQfO3oKfN5XozNmr6mis=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/\nMSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\nDkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow\nPzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD\nEw5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB\nAN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O\nrz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq\nOLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b\nxiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw\n7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD\naeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV\nHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG\nSIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69\nikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr\nAvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz\nR8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5\nJDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo\nOb8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDnzCCAoegAwIBAgIBJjANBgkqhkiG9w0BAQUFADBxMQswCQYDVQQGEwJERTEc\nMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2Vj\nIFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENB\nIDIwHhcNOTkwNzA5MTIxMTAwWhcNMTkwNzA5MjM1OTAwWjBxMQswCQYDVQQGEwJE\nRTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxl\nU2VjIFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290\nIENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrC6M14IspFLEU\nha88EOQ5bzVdSq7d6mGNlUn0b2SjGmBmpKlAIoTZ1KXleJMOaAGtuU1cOs7TuKhC\nQN/Po7qCWWqSG6wcmtoIKyUn+WkjR/Hg6yx6m/UTAtB+NHzCnjwAWav12gz1Mjwr\nrFDa1sPeg5TKqAyZMg4ISFZbavva4VhYAUlfckE8FQYBjl2tqriTtM2e66foai1S\nNNs671x1Udrb8zH57nGYMsRUFUQM+ZtV7a3fGAigo4aKSe5TBY8ZTNXeWHmb0moc\nQqvF1afPaA+W5OFhmHZhyJF81j4A4pFQh+GdCuatl9Idxjp9y7zaAzTVjlsB9WoH\ntxa2bkp/AgMBAAGjQjBAMB0GA1UdDgQWBBQxw3kbuvVT1xfgiXotF2wKsyudMzAP\nBgNVHRMECDAGAQH/AgEFMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOC\nAQEAlGRZrTlk5ynrE/5aw4sTV8gEJPB0d8Bg42f76Ymmg7+Wgnxu1MM9756Abrsp\ntJh6sTtU6zkXR34ajgv8HzFZMQSyzhfzLMdiNlXiItiJVbSYSKpk+tYcNthEeFpa\nIzpXl/V6ME+un2pMSyuOoAPjPuCp1NJ70rOo4nI8rZ7/gFnkm0W09juwzTkZmDLl\n6iFhkOQxIY40sfcvNUqFENrnijchvllj4PKFiDFT1FQUhXB59C4Gdyd1Lx+4ivn+\nxbrYNuSD7Odlt79jWvNGr4GUN9RBjNYj1h7P9WgbRGOiWrqnNVmh5XAFmw4jV5mU\nCm26OWMohpLzGITY+9HPBVZkVw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDtzCCAp+gAwIBAgIQDOfg5RfYRv6P5WD8G/AwOTANBgkqhkiG9w0BAQUFADBl\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJv\nb3QgQ0EwHhcNMDYxMTEwMDAwMDAwWhcNMzExMTEwMDAwMDAwWjBlMQswCQYDVQQG\nEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNl\ncnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgQ0EwggEi\nMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCtDhXO5EOAXLGH87dg+XESpa7c\nJpSIqvTO9SA5KFhgDPiA2qkVlTJhPLWxKISKityfCgyDF3qPkKyK53lTXDGEKvYP\nmDI2dsze3Tyoou9q+yHyUmHfnyDXH+Kx2f4YZNISW1/5WBg1vEfNoTb5a3/UsDg+\nwRvDjDPZ2C8Y/igPs6eD1sNuRMBhNZYW/lmci3Zt1/GiSw0r/wty2p5g0I6QNcZ4\nVYcgoc/lbQrISXwxmDNsIumH0DJaoroTghHtORedmTpyoeb6pNnVFzF1roV9Iq4/\nAUaG9ih5yLHa5FcXxH4cDrC0kqZWs72yl+2qp/C3xag/lRbQ/6GW6whfGHdPAgMB\nAAGjYzBhMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW\nBBRF66Kv9JLLgjEtUYunpyGd823IDzAfBgNVHSMEGDAWgBRF66Kv9JLLgjEtUYun\npyGd823IDzANBgkqhkiG9w0BAQUFAAOCAQEAog683+Lt8ONyc3pklL/3cmbYMuRC\ndWKuh+vy1dneVrOfzM4UKLkNl2BcEkxY5NM9g0lFWJc1aRqoR+pWxnmrEthngYTf\nfwk8lOa4JiwgvT2zKIn3X/8i4peEH+ll74fg38FnSbNd67IJKusm7Xi+fT8r87cm\nNW1fiQG2SVufAQWbqz0lwcy2f8Lxb4bG+mRo64EtlOtCt/qMHt1i8b5QZ7dsvfPx\nH2sMNgcWfzd8qVttevESRmCD1ycEvkvOl77DZypoEd+A5wwzZr8TDRRu838fYxAe\n+o0bJW1sj6W3YQGx0qMmoRBxna3iw/nDmVG3KwcIzi7mULKn+gpFL6Lw8g==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDljCCAn6gAwIBAgIQC5McOtY5Z+pnI7/Dr5r0SzANBgkqhkiG9w0BAQsFADBl\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJv\nb3QgRzIwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBlMQswCQYDVQQG\nEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNl\ncnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgRzIwggEi\nMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDZ5ygvUj82ckmIkzTz+GoeMVSA\nn61UQbVH35ao1K+ALbkKz3X9iaV9JPrjIgwrvJUXCzO/GU1BBpAAvQxNEP4Htecc\nbiJVMWWXvdMX0h5i89vqbFCMP4QMls+3ywPgym2hFEwbid3tALBSfK+RbLE4E9Hp\nEgjAALAcKxHad3A2m67OeYfcgnDmCXRwVWmvo2ifv922ebPynXApVfSr/5Vh88lA\nbx3RvpO704gqu52/clpWcTs/1PPRCv4o76Pu2ZmvA9OPYLfykqGxvYmJHzDNw6Yu\nYjOuFgJ3RFrngQo8p0Quebg/BLxcoIfhG69Rjs3sLPr4/m3wOnyqi+RnlTGNAgMB\nAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQW\nBBTOw0q5mVXyuNtgv6l+vVa1lzan1jANBgkqhkiG9w0BAQsFAAOCAQEAyqVVjOPI\nQW5pJ6d1Ee88hjZv0p3GeDgdaZaikmkuOGybfQTUiaWxMTeKySHMq2zNixya1r9I\n0jJmwYrA8y8678Dj1JGG0VDjA9tzd29KOVPt3ibHtX2vK0LRdWLjSisCx1BL4Gni\nlmwORGYQRI+tBev4eaymG+g3NJ1TyWGqolKvSnAWhsI6yLETcDbYz+70CjTVW0z9\nB5yiutkBclzzTcHdDrEcDcRjvq30FPuJ7KJBDkzMyFdA0G4Dqs0MjomZmWzwPDCv\nON9vvKO+KSAnq3T/EyJ43pdSVR6DtVQgA+6uwE9W3jfMw3+qBCe703e4YtsXfJwo\nIhNzbM8m9Yop5w==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICRjCCAc2gAwIBAgIQC6Fa+h3foLVJRK/NJKBs7DAKBggqhkjOPQQDAzBlMQsw\nCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cu\nZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3Qg\nRzMwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBlMQswCQYDVQQGEwJV\nUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQu\nY29tMSQwIgYDVQQDExtEaWdpQ2VydCBBc3N1cmVkIElEIFJvb3QgRzMwdjAQBgcq\nhkjOPQIBBgUrgQQAIgNiAAQZ57ysRGXtzbg/WPuNsVepRC0FFfLvC/8QdJ+1YlJf\nZn4f5dwbRXkLzMZTCp2NXQLZqVneAlr2lSoOjThKiknGvMYDOAdfVdp+CW7if17Q\nRSAPWXYQ1qAk8C3eNvJsKTmjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/\nBAQDAgGGMB0GA1UdDgQWBBTL0L2p4ZgFUaFNN6KDec6NHSrkhDAKBggqhkjOPQQD\nAwNnADBkAjAlpIFFAmsSS3V0T8gj43DydXLefInwz5FyYZ5eEJJZVrmDxxDnOOlY\nJjZ91eQ0hjkCMHw2U/Aw5WJjOpnitqM7mzT6HtoQknFekROn3aRukswy1vUhZscv\n6pZjamVFkpUBtA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD\nQTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT\nMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j\nb20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB\nCSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97\nnh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt\n43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P\nT19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4\ngdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO\nBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR\nTLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw\nDQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr\nhMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg\n06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF\nPnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls\nYSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk\nCAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH\nMjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT\nMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j\nb20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI\n2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx\n1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ\nq2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz\ntCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ\nvIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP\nBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV\n5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY\n1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4\nNeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG\nFdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91\n8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe\npLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl\nMrY=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICPzCCAcWgAwIBAgIQBVVWvPJepDU1w6QP1atFcjAKBggqhkjOPQQDAzBhMQsw\nCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cu\nZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBHMzAe\nFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVTMRUw\nEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20x\nIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEczMHYwEAYHKoZIzj0CAQYF\nK4EEACIDYgAE3afZu4q4C/sLfyHS8L6+c/MzXRq8NOrexpu80JX28MzQC7phW1FG\nfp4tn+6OYwwX7Adw9c+ELkCDnOg/QW07rdOkFFk2eJ0DQ+4QE2xy3q6Ip6FrtUPO\nZ9wj/wMco+I+o0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAd\nBgNVHQ4EFgQUs9tIpPmhxdiuNkHMEWNpYim8S8YwCgYIKoZIzj0EAwMDaAAwZQIx\nAK288mw/EkrRLTnDCgmXc/SINoyIJ7vmiI1Qhadj+Z4y3maTD/HMsQmP3Wyr+mt/\noAIwOWZbwmSNuJ5Q3KjVSaLtx9zRSX8XAbjIho9OjIgrqJqpisXRAL34VOKa5Vt8\nsycX\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDxTCCAq2gAwIBAgIQAqxcJmoLQJuPC3nyrkYldzANBgkqhkiG9w0BAQUFADBs\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSswKQYDVQQDEyJEaWdpQ2VydCBIaWdoIEFzc3VyYW5j\nZSBFViBSb290IENBMB4XDTA2MTExMDAwMDAwMFoXDTMxMTExMDAwMDAwMFowbDEL\nMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3\nLmRpZ2ljZXJ0LmNvbTErMCkGA1UEAxMiRGlnaUNlcnQgSGlnaCBBc3N1cmFuY2Ug\nRVYgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMbM5XPm\n+9S75S0tMqbf5YE/yc0lSbZxKsPVlDRnogocsF9ppkCxxLeyj9CYpKlBWTrT3JTW\nPNt0OKRKzE0lgvdKpVMSOO7zSW1xkX5jtqumX8OkhPhPYlG++MXs2ziS4wblCJEM\nxChBVfvLWokVfnHoNb9Ncgk9vjo4UFt3MRuNs8ckRZqnrG0AFFoEt7oT61EKmEFB\nIk5lYYeBQVCmeVyJ3hlKV9Uu5l0cUyx+mM0aBhakaHPQNAQTXKFx01p8VdteZOE3\nhzBWBOURtCmAEvF5OYiiAhF8J2a3iLd48soKqDirCmTCv2ZdlYTBoSUeh10aUAsg\nEsxBu24LUTi4S8sCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQF\nMAMBAf8wHQYDVR0OBBYEFLE+w2kD+L9HAdSYJhoIAu9jZCvDMB8GA1UdIwQYMBaA\nFLE+w2kD+L9HAdSYJhoIAu9jZCvDMA0GCSqGSIb3DQEBBQUAA4IBAQAcGgaX3Nec\nnzyIZgYIVyHbIUf4KmeqvxgydkAQV8GK83rZEWWONfqe/EW1ntlMMUu4kehDLI6z\neM7b41N5cdblIZQB2lWHmiRk9opmzN6cN82oNLFpmyPInngiK3BD41VHMWEZ71jF\nhS9OMPagMRYjyOfiZRYzy78aG6A9+MpeizGLYAiJLQwGXFK3xPkKmNEVX58Svnw2\nYzi9RKR/5CYrCsSXaQ3pjOLAEFe4yHYSkVXySGnYvCoCWw9E1CAx2/S6cCZdkGCe\nvEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep\n+OkuE6N36B9K\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFkDCCA3igAwIBAgIQBZsbV56OITLiOQe9p3d1XDANBgkqhkiG9w0BAQwFADBi\nMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\nd3cuZGlnaWNlcnQuY29tMSEwHwYDVQQDExhEaWdpQ2VydCBUcnVzdGVkIFJvb3Qg\nRzQwHhcNMTMwODAxMTIwMDAwWhcNMzgwMTE1MTIwMDAwWjBiMQswCQYDVQQGEwJV\nUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQu\nY29tMSEwHwYDVQQDExhEaWdpQ2VydCBUcnVzdGVkIFJvb3QgRzQwggIiMA0GCSqG\nSIb3DQEBAQUAA4ICDwAwggIKAoICAQC/5pBzaN675F1KPDAiMGkz7MKnJS7JIT3y\nithZwuEppz1Yq3aaza57G4QNxDAf8xukOBbrVsaXbR2rsnnyyhHS5F/WBTxSD1If\nxp4VpX6+n6lXFllVcq9ok3DCsrp1mWpzMpTREEQQLt+C8weE5nQ7bXHiLQwb7iDV\nySAdYyktzuxeTsiT+CFhmzTrBcZe7FsavOvJz82sNEBfsXpm7nfISKhmV1efVFiO\nDCu3T6cw2Vbuyntd463JT17lNecxy9qTXtyOj4DatpGYQJB5w3jHtrHEtWoYOAMQ\njdjUN6QuBX2I9YI+EJFwq1WCQTLX2wRzKm6RAXwhTNS8rhsDdV14Ztk6MUSaM0C/\nCNdaSaTC5qmgZ92kJ7yhTzm1EVgX9yRcRo9k98FpiHaYdj1ZXUJ2h4mXaXpI8OCi\nEhtmmnTK3kse5w5jrubU75KSOp493ADkRSWJtppEGSt+wJS00mFt6zPZxd9LBADM\nfRyVw4/3IbKyEbe7f/LVjHAsQWCqsWMYRJUadmJ+9oCw++hkpjPRiQfhvbfmQ6QY\nuKZ3AeEPlAwhHbJUKSWJbOUOUlFHdL4mrLZBdd56rF+NP8m800ERElvlEFDrMcXK\nchYiCd98THU/Y+whX8QgUWtvsauGi0/C1kVfnSD8oR7FwI+isX4KJpn15GkvmB0t\n9dmpsh3lGwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nhjAdBgNVHQ4EFgQU7NfjgtJxXWRM3y5nP+e6mK4cD08wDQYJKoZIhvcNAQEMBQAD\nggIBALth2X2pbL4XxJEbw6GiAI3jZGgPVs93rnD5/ZpKmbnJeFwMDF/k5hQpVgs2\nSV1EY+CtnJYYZhsjDT156W1r1lT40jzBQ0CuHVD1UvyQO7uYmWlrx8GnqGikJ9yd\n+SeuMIW59mdNOj6PWTkiU0TryF0Dyu1Qen1iIQqAyHNm0aAFYF/opbSnr6j3bTWc\nfFqK1qI4mfN4i/RN0iAL3gTujJtHgXINwBQy7zBZLq7gcfJW5GqXb5JQbZaNaHqa\nsjYUegbyJLkJEVDXCLG4iXqEI2FCKeWjzaIgQdfRnGTZ6iahixTXTBmyUEFxPT9N\ncCOGDErcgdLMMpSEDQgJlxxPwO5rIHQw0uA5NBCFIRUBCOhVMt5xSdkoF1BN5r5N\n0XWs0Mr7QbhDparTwwVETyw2m+L64kW4I1NsBm9nVX9GtUw/bihaeSbSpKhil9Ie\n4u1Ki7wb/UdKDd9nZn6yW0HQO+T0O/QEY+nvwlQAUaCKKsnOeMzV6ocEGLPOr0mI\nr/OSmbaz5mEP0oUA51Aa5BuVnRmhuZyxm7EAHu/QD09CbMkKvO5D+jpxpchNJqU1\n/YldvIViHTLSoCtU7ZpXwdv6EM8Zt4tKG48BtieVU+i2iW1bvGjUI+iLUaJW+fCm\ngKDWHrO8Dw9TdSmq6hN35N6MgSGtBxBHEa2HPQfRdbzP82Z+\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDtjCCAp6gAwIBAgIQRJmNPMADJ72cdpW56tustTANBgkqhkiG9w0BAQUFADB1\nMQswCQYDVQQGEwJUUjEoMCYGA1UEChMfRWxla3Ryb25payBCaWxnaSBHdXZlbmxp\nZ2kgQS5TLjE8MDoGA1UEAxMzZS1HdXZlbiBLb2sgRWxla3Ryb25payBTZXJ0aWZp\na2EgSGl6bWV0IFNhZ2xheWljaXNpMB4XDTA3MDEwNDExMzI0OFoXDTE3MDEwNDEx\nMzI0OFowdTELMAkGA1UEBhMCVFIxKDAmBgNVBAoTH0VsZWt0cm9uaWsgQmlsZ2kg\nR3V2ZW5saWdpIEEuUy4xPDA6BgNVBAMTM2UtR3V2ZW4gS29rIEVsZWt0cm9uaWsg\nU2VydGlmaWthIEhpem1ldCBTYWdsYXlpY2lzaTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAMMSIJ6wXgBljU5Gu4Bc6SwGl9XzcslwuedLZYDBS75+PNdU\nMZTe1RK6UxYC6lhj71vY8+0qGqpxSKPcEC1fX+tcS5yWCEIlKBHMilpiAVDV6wlT\nL/jDj/6z/P2douNffb7tC+Bg62nsM+3YjfsSSYMAyYuXjDtzKjKzEve5TfL0TW3H\n5tYmNwjy2f1rXKPlSFxYvEK+A1qBuhw1DADT9SN+cTAIJjjcJRFHLfO6IxClv7wC\n90Nex/6wN1CZew+TzuZDLMN+DfIcQ2Zgy2ExR4ejT669VmxMvLz4Bcpk9Ok0oSy1\nc+HCPujIyTQlCFzz7abHlJ+tiEMl1+E5YP6sOVkCAwEAAaNCMEAwDgYDVR0PAQH/\nBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFJ/uRLOU1fqRTy7ZVZoE\nVtstxNulMA0GCSqGSIb3DQEBBQUAA4IBAQB/X7lTW2M9dTLn+sR0GstG30ZpHFLP\nqk/CaOv/gKlR6D1id4k9CnU58W5dF4dvaAXBlGzZXd/aslnLpRCKysw5zZ/rTt5S\n/wzw9JKp8mxTq5vSR6AfdPebmvEvFZ96ZDAYBzwqD2fK/A+JYZ1lpTzlvBNbCNvj\n/+27BrtqBrF6T2XGgv0enIu1De5Iu7i9qgi0+6N8y5/NkHZchpZ4Vwpm+Vganf2X\nKWDeEaaQHBkc7gGWIjQ0LpH5t8Qn0Xvmv/uARFoW5evg1Ao4vOSR49XrXMGs3xtq\nfJ7lddK2l4fbzIcrQzqECK+rPNv3PGYxhrCdU3nt+CPeQuMtgvEP5fqX\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIGSzCCBDOgAwIBAgIIamg+nFGby1MwDQYJKoZIhvcNAQELBQAwgbIxCzAJBgNV\nBAYTAlRSMQ8wDQYDVQQHDAZBbmthcmExQDA+BgNVBAoMN0UtVHXEn3JhIEVCRyBC\naWxpxZ9pbSBUZWtub2xvamlsZXJpIHZlIEhpem1ldGxlcmkgQS7Fni4xJjAkBgNV\nBAsMHUUtVHVncmEgU2VydGlmaWthc3lvbiBNZXJrZXppMSgwJgYDVQQDDB9FLVR1\nZ3JhIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTEzMDMwNTEyMDk0OFoXDTIz\nMDMwMzEyMDk0OFowgbIxCzAJBgNVBAYTAlRSMQ8wDQYDVQQHDAZBbmthcmExQDA+\nBgNVBAoMN0UtVHXEn3JhIEVCRyBCaWxpxZ9pbSBUZWtub2xvamlsZXJpIHZlIEhp\nem1ldGxlcmkgQS7Fni4xJjAkBgNVBAsMHUUtVHVncmEgU2VydGlmaWthc3lvbiBN\nZXJrZXppMSgwJgYDVQQDDB9FLVR1Z3JhIENlcnRpZmljYXRpb24gQXV0aG9yaXR5\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4vU/kwVRHoViVF56C/UY\nB4Oufq9899SKa6VjQzm5S/fDxmSJPZQuVIBSOTkHS0vdhQd2h8y/L5VMzH2nPbxH\nD5hw+IyFHnSOkm0bQNGZDbt1bsipa5rAhDGvykPL6ys06I+XawGb1Q5KCKpbknSF\nQ9OArqGIW66z6l7LFpp3RMih9lRozt6Plyu6W0ACDGQXwLWTzeHxE2bODHnv0ZEo\nq1+gElIwcxmOj+GMB6LDu0rw6h8VqO4lzKRG+Bsi77MOQ7osJLjFLFzUHPhdZL3D\nk14opz8n8Y4e0ypQBaNV2cvnOVPAmJ6MVGKLJrD3fY185MaeZkJVgkfnsliNZvcH\nfC425lAcP9tDJMW/hkd5s3kc91r0E+xs+D/iWR+V7kI+ua2oMoVJl0b+SzGPWsut\ndEcf6ZG33ygEIqDUD13ieU/qbIWGvaimzuT6w+Gzrt48Ue7LE3wBf4QOXVGUnhMM\nti6lTPk5cDZvlsouDERVxcr6XQKj39ZkjFqzAQqptQpHF//vkUAqjqFGOjGY5RH8\nzLtJVor8udBhmm9lbObDyz51Sf6Pp+KJxWfXnUYTTjF2OySznhFlhqt/7x3U+Lzn\nrFpct1pHXFXOVbQicVtbC/DP3KBhZOqp12gKY6fgDT+gr9Oq0n7vUaDmUStVkhUX\nU8u3Zg5mTPj5dUyQ5xJwx0UCAwEAAaNjMGEwHQYDVR0OBBYEFC7j27JJ0JxUeVz6\nJyr+zE7S6E5UMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAULuPbsknQnFR5\nXPonKv7MTtLoTlQwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4ICAQAF\nNzr0TbdF4kV1JI+2d1LoHNgQk2Xz8lkGpD4eKexd0dCrfOAKkEh47U6YA5n+KGCR\nHTAduGN8qOY1tfrTYXbm1gdLymmasoR6d5NFFxWfJNCYExL/u6Au/U5Mh/jOXKqY\nGwXgAEZKgoClM4so3O0409/lPun++1ndYYRP0lSWE2ETPo+Aab6TR7U1Q9Jauz1c\n77NCR807VRMGsAnb/WP2OogKmW9+4c4bU2pEZiNRCHu8W1Ki/QY3OEBhj0qWuJA3\n+GbHeJAAFS6LrVE1Uweoa2iu+U48BybNCAVwzDk/dr2l02cmAYamU9JgO3xDf1WK\nvJUawSg5TB9D0pH0clmKuVb8P7Sd2nCcdlqMQ1DujjByTd//SffGqWfZbawCEeI6\nFiWnWAjLb1NBnEg4R2gz0dfHj9R0IdTDBZB6/86WiLEVKV0jq9BgoRJP3vQXzTLl\nyb/IQ639Lo7xr+L0mPoSHyDYwKcMhcWQ9DstliaxLL5Mq+ux0orJ23gTDx4JnW2P\nAJ8C2sH6H3p6CcRK5ogql5+Ji/03X186zjhZhkuvcQu02PJwT58yE+Owp1fl2tpD\ny4Q08ijE6m30Ku/Ba3ba+367hTzSU8JNvnHhRdH9I2cNE3X7z2VnIp2usAnRCf8d\nNL/+I5c30jn6PQ0GC7TbO6Orb1wdtn7os4I07QZcJA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF5zCCA8+gAwIBAgIITK9zQhyOdAIwDQYJKoZIhvcNAQEFBQAwgYAxODA2BgNV\nBAMML0VCRyBFbGVrdHJvbmlrIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sx\nc8SxMTcwNQYDVQQKDC5FQkcgQmlsacWfaW0gVGVrbm9sb2ppbGVyaSB2ZSBIaXpt\nZXRsZXJpIEEuxZ4uMQswCQYDVQQGEwJUUjAeFw0wNjA4MTcwMDIxMDlaFw0xNjA4\nMTQwMDMxMDlaMIGAMTgwNgYDVQQDDC9FQkcgRWxla3Ryb25payBTZXJ0aWZpa2Eg\nSGl6bWV0IFNhxJ9sYXnEsWPEsXPEsTE3MDUGA1UECgwuRUJHIEJpbGnFn2ltIFRl\na25vbG9qaWxlcmkgdmUgSGl6bWV0bGVyaSBBLsWeLjELMAkGA1UEBhMCVFIwggIi\nMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDuoIRh0DpqZhAy2DE4f6en5f2h\n4fuXd7hxlugTlkaDT7byX3JWbhNgpQGR4lvFzVcfd2NR/y8927k/qqk153nQ9dAk\ntiHq6yOU/im/+4mRDGSaBUorzAzu8T2bgmmkTPiab+ci2hC6X5L8GCcKqKpE+i4s\ntPtGmggDg3KriORqcsnlZR9uKg+ds+g75AxuetpX/dfreYteIAbTdgtsApWjluTL\ndlHRKJ2hGvxEok3MenaoDT2/F08iiFD9rrbskFBKW5+VQarKD7JK/oCZTqNGFav4\nc0JqwmZ2sQomFd2TkuzbqV9UIlKRcF0T6kjsbgNs2d1s/OsNA/+mgxKb8amTD8Um\nTDGyY5lhcucqZJnSuOl14nypqZoaqsNW2xCaPINStnuWt6yHd6i58mcLlEOzrz5z\n+kI2sSXFCjEmN1ZnuqMLfdb3ic1nobc6HmZP9qBVFCVMLDMNpkGMvQQxahByCp0O\nLna9XvNRiYuoP1Vzv9s6xiQFlpJIqkuNKgPlV5EQ9GooFW5Hd4RcUXSfGenmHmMW\nOeMRFeNYGkS9y8RsZteEBt8w9DeiQyJ50hBs37vmExH8nYQKE3vwO9D8owrXieqW\nfo1IhR5kX9tUoqzVegJ5a9KK8GfaZXINFHDk6Y54jzJ0fFfy1tb0Nokb+Clsi7n2\nl9GkLqq+CxnCRelwXQIDAJ3Zo2MwYTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB\n/wQEAwIBBjAdBgNVHQ4EFgQU587GT/wWZ5b6SqMHwQSny2re2kcwHwYDVR0jBBgw\nFoAU587GT/wWZ5b6SqMHwQSny2re2kcwDQYJKoZIhvcNAQEFBQADggIBAJuYml2+\n8ygjdsZs93/mQJ7ANtyVDR2tFcU22NU57/IeIl6zgrRdu0waypIN30ckHrMk2pGI\n6YNw3ZPX6bqz3xZaPt7gyPvT/Wwp+BVGoGgmzJNSroIBk5DKd8pNSe/iWtkqvTDO\nTLKBtjDOWU/aWR1qeqRFsIImgYZ29fUQALjuswnoT4cCB64kXPBfrAowzIpAoHME\nwfuJJPaaHFy3PApnNgUIMbOv2AFoKuB4j3TeuFGkjGwgPaL7s9QJ/XvCgKqTbCmY\nIai7FvOpEl90tYeY8pUm3zTvilORiF0alKM/fCL414i6poyWqD1SNGKfAB5UVUJn\nxk1Gj7sURT0KlhaOEKGXmdXTMIXM3rRyt7yKPBgpaP3ccQfuJDlq+u2lrDgv+R4Q\nDgZxGhBM/nV+/x5XOULK1+EVoVZVWRvRo68R2E7DpSvvkL/A7IITW43WciyTTo9q\nKd+FPNMN4KIYEsxVL0e3p5sC/kH2iExt2qkBR4NkJ2IQgtYSe14DHzSpyZH+r11t\nhie3I6p1GMog57AP14kOpmciY/SDQSsGS7tY1dHXt7kQY9iJSrSq3RZj9W6+YKH4\n7ejWkE8axsWgKdOnIaj1Wjz3x0miIZpKlVIglnKaZsv30oZDfCK+lvm9AahH3eU7\nQPl1K5srRmSGjR70j/sHd9DqSaIcjVIUpgqT\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFVjCCBD6gAwIBAgIQ7is969Qh3hSoYqwE893EATANBgkqhkiG9w0BAQUFADCB\n8zELMAkGA1UEBhMCRVMxOzA5BgNVBAoTMkFnZW5jaWEgQ2F0YWxhbmEgZGUgQ2Vy\ndGlmaWNhY2lvIChOSUYgUS0wODAxMTc2LUkpMSgwJgYDVQQLEx9TZXJ2ZWlzIFB1\nYmxpY3MgZGUgQ2VydGlmaWNhY2lvMTUwMwYDVQQLEyxWZWdldSBodHRwczovL3d3\ndy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAoYykwMzE1MDMGA1UECxMsSmVyYXJxdWlh\nIEVudGl0YXRzIGRlIENlcnRpZmljYWNpbyBDYXRhbGFuZXMxDzANBgNVBAMTBkVD\nLUFDQzAeFw0wMzAxMDcyMzAwMDBaFw0zMTAxMDcyMjU5NTlaMIHzMQswCQYDVQQG\nEwJFUzE7MDkGA1UEChMyQWdlbmNpYSBDYXRhbGFuYSBkZSBDZXJ0aWZpY2FjaW8g\nKE5JRiBRLTA4MDExNzYtSSkxKDAmBgNVBAsTH1NlcnZlaXMgUHVibGljcyBkZSBD\nZXJ0aWZpY2FjaW8xNTAzBgNVBAsTLFZlZ2V1IGh0dHBzOi8vd3d3LmNhdGNlcnQu\nbmV0L3ZlcmFycmVsIChjKTAzMTUwMwYDVQQLEyxKZXJhcnF1aWEgRW50aXRhdHMg\nZGUgQ2VydGlmaWNhY2lvIENhdGFsYW5lczEPMA0GA1UEAxMGRUMtQUNDMIIBIjAN\nBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsyLHT+KXQpWIR4NA9h0X84NzJB5R\n85iKw5K4/0CQBXCHYMkAqbWUZRkiFRfCQ2xmRJoNBD45b6VLeqpjt4pEndljkYRm\n4CgPukLjbo73FCeTae6RDqNfDrHrZqJyTxIThmV6PttPB/SnCWDaOkKZx7J/sxaV\nHMf5NLWUhdWZXqBIoH7nF2W4onW4HvPlQn2v7fOKSGRdghST2MDk/7NQcvJ29rNd\nQlB50JQ+awwAvthrDk4q7D7SzIKiGGUzE3eeml0aE9jD2z3Il3rucO2n5nzbcc8t\nlGLfbdb1OL4/pYUKGbio2Al1QnDE6u/LDsg0qBIimAy4E5S2S+zw0JDnJwIDAQAB\no4HjMIHgMB0GA1UdEQQWMBSBEmVjX2FjY0BjYXRjZXJ0Lm5ldDAPBgNVHRMBAf8E\nBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUoMOLRKo3pUW/l4Ba0fF4\nopvpXY0wfwYDVR0gBHgwdjB0BgsrBgEEAfV4AQMBCjBlMCwGCCsGAQUFBwIBFiBo\ndHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbDA1BggrBgEFBQcCAjApGidW\nZWdldSBodHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAwDQYJKoZIhvcN\nAQEFBQADggEBAKBIW4IB9k1IuDlVNZyAelOZ1Vr/sXE7zDkJlF7W2u++AVtd0x7Y\n/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k\nSBg6hW/cnbw/nZzBh7h6YQjpdwt/cKt63dmXLGQehb+8dJahw3oS7AwaboMMPOhy\nRp/7SNVel+axofjk70YllJyJ22k4vuxcDlbHZVHlUIiIv0LVKz3l+bqeLrPK9HOS\nAgu+TGbrIP65y7WZf+a2E/rKS03Z7lNGBjvGTq2TWoF+bCpLagVFjPIhpDGQh2xl\nnJ2lYJU6Un/10asIbvPuW/mIPX64b24D5EI=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEAzCCAuugAwIBAgIQVID5oHPtPwBMyonY43HmSjANBgkqhkiG9w0BAQUFADB1\nMQswCQYDVQQGEwJFRTEiMCAGA1UECgwZQVMgU2VydGlmaXRzZWVyaW1pc2tlc2t1\nczEoMCYGA1UEAwwfRUUgQ2VydGlmaWNhdGlvbiBDZW50cmUgUm9vdCBDQTEYMBYG\nCSqGSIb3DQEJARYJcGtpQHNrLmVlMCIYDzIwMTAxMDMwMTAxMDMwWhgPMjAzMDEy\nMTcyMzU5NTlaMHUxCzAJBgNVBAYTAkVFMSIwIAYDVQQKDBlBUyBTZXJ0aWZpdHNl\nZXJpbWlza2Vza3VzMSgwJgYDVQQDDB9FRSBDZXJ0aWZpY2F0aW9uIENlbnRyZSBS\nb290IENBMRgwFgYJKoZIhvcNAQkBFglwa2lAc2suZWUwggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQDIIMDs4MVLqwd4lfNE7vsLDP90jmG7sWLqI9iroWUy\neuuOF0+W2Ap7kaJjbMeMTC55v6kF/GlclY1i+blw7cNRfdCT5mzrMEvhvH2/UpvO\nbntl8jixwKIy72KyaOBhU8E2lf/slLo2rpwcpzIP5Xy0xm90/XsY6KxX7QYgSzIw\nWFv9zajmofxwvI6Sc9uXp3whrj3B9UiHbCe9nyV0gVWw93X2PaRka9ZP585ArQ/d\nMtO8ihJTmMmJ+xAdTX7Nfh9WDSFwhfYggx/2uh8Ej+p3iDXE/+pOoYtNP2MbRMNE\n1CV2yreN1x5KZmTNXMWcg+HCCIia7E6j8T4cLNlsHaFLAgMBAAGjgYowgYcwDwYD\nVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFBLyWj7qVhy/\nzQas8fElyalL1BSZMEUGA1UdJQQ+MDwGCCsGAQUFBwMCBggrBgEFBQcDAQYIKwYB\nBQUHAwMGCCsGAQUFBwMEBggrBgEFBQcDCAYIKwYBBQUHAwkwDQYJKoZIhvcNAQEF\nBQADggEBAHv25MANqhlHt01Xo/6tu7Fq1Q+e2+RjxY6hUFaTlrg4wCQiZrxTFGGV\nv9DHKpY5P30osxBAIWrEr7BSdxjhlthWXePdNl4dp1BUoMUq5KqMlIpPnTX/dqQG\nE5Gion0ARD9V04I8GtVbvFZMIi5GQ4okQC3zErg7cBqklrkar4dBGmoYDQZPxz5u\nuSlNDUmJEYcyW+ZLBMjkXOZ0c5RdFpgTlf7727FE5TpwrDdr5rMzcijJs1eg9gIW\niAYLtqZLICjU3j2LrTcFU3T+bsy8QxdxXvnFzBqpYe73dgzzcvRyrc9yAjYHR8/v\nGVCJYMzpJJUPwssd8m92kMfMdcGWxZ0=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEKjCCAxKgAwIBAgIEOGPe+DANBgkqhkiG9w0BAQUFADCBtDEUMBIGA1UEChML\nRW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5lbnRydXN0Lm5ldC9DUFNfMjA0OCBp\nbmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNVBAsTHChjKSAxOTk5\nIEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENlcnRp\nZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTAeFw05OTEyMjQxNzUwNTFaFw0yOTA3\nMjQxNDE1MTJaMIG0MRQwEgYDVQQKEwtFbnRydXN0Lm5ldDFAMD4GA1UECxQ3d3d3\nLmVudHJ1c3QubmV0L0NQU18yMDQ4IGluY29ycC4gYnkgcmVmLiAobGltaXRzIGxp\nYWIuKTElMCMGA1UECxMcKGMpIDE5OTkgRW50cnVzdC5uZXQgTGltaXRlZDEzMDEG\nA1UEAxMqRW50cnVzdC5uZXQgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgKDIwNDgp\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArU1LqRKGsuqjIAcVFmQq\nK0vRvwtKTY7tgHalZ7d4QMBzQshowNtTK91euHaYNZOLGp18EzoOH1u3Hs/lJBQe\nsYGpjX24zGtLA/ECDNyrpUAkAH90lKGdCCmziAv1h3edVc3kw37XamSrhRSGlVuX\nMlBvPci6Zgzj/L24ScF2iUkZ/cCovYmjZy/Gn7xxGWC4LeksyZB2ZnuU4q941mVT\nXTzWnLLPKQP5L6RQstRIzgUyVYr9smRMDuSYB3Xbf9+5CFVghTAp+XtIpGmG4zU/\nHoZdenoVve8AjhUiVBcAkCaTvA5JaJG/+EfTnZVCwQ5N328mz8MYIWJmQ3DW1cAH\n4QIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV\nHQ4EFgQUVeSB0RGAvtiJuQijMfmhJAkWuXAwDQYJKoZIhvcNAQEFBQADggEBADub\nj1abMOdTmXx6eadNl9cZlZD7Bh/KM3xGY4+WZiT6QBshJ8rmcnPyT/4xmf3IDExo\nU8aAghOY+rat2l098c5u9hURlIIM7j+VrxGrD9cv3h8Dj1csHsm7mhpElesYT6Yf\nzX1XEC+bBAlahLVu2B064dae0Wx5XnkcFMXj0EyTO2U87d89vqbllRrDtRnDvV5b\nu/8j72gZyxKTJ1wDLW8w0B62GqzeWvfRqqgnpv55gcR5mTNXuhKwqeBCbJPKVt7+\nbYQLCIt+jerXmCHG8+c8eS9enNFMFY3h7CI3zJpDC5fcgJCNs2ebb0gIFVbPv/Er\nfF6adulZkMV8gzURZVE=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEkTCCA3mgAwIBAgIERWtQVDANBgkqhkiG9w0BAQUFADCBsDELMAkGA1UEBhMC\nVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0\nLm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5jZTEfMB0GA1UECxMW\nKGMpIDIwMDYgRW50cnVzdCwgSW5jLjEtMCsGA1UEAxMkRW50cnVzdCBSb290IENl\ncnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA2MTEyNzIwMjM0MloXDTI2MTEyNzIw\nNTM0MlowgbAxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMTkw\nNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BTIGlzIGluY29ycG9yYXRlZCBieSBy\nZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDA2IEVudHJ1c3QsIEluYy4xLTArBgNV\nBAMTJEVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTCCASIwDQYJ\nKoZIhvcNAQEBBQADggEPADCCAQoCggEBALaVtkNC+sZtKm9I35RMOVcF7sN5EUFo\nNu3s/poBj6E4KPz3EEZmLk0eGrEaTsbRwJWIsMn/MYszA9u3g3s+IIRe7bJWKKf4\n4LlAcTfFy0cOlypowCKVYhXbR9n10Cv/gkvJrT7eTNuQgFA/CYqEAOwwCj0Yzfv9\nKlmaI5UXLEWeH25DeW0MXJj+SKfFI0dcXv1u5x609mhF0YaDW6KKjbHjKYD+JXGI\nrb68j6xSlkuqUY3kEzEZ6E5Nn9uss2rVvDlUccp6en+Q3X0dgNmBu1kmwhH+5pPi\n94DkZfs0Nw4pgHBNrziGLp5/V6+eF67rHMsoIV+2HNjnogQi+dPa2MsCAwEAAaOB\nsDCBrTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zArBgNVHRAEJDAi\ngA8yMDA2MTEyNzIwMjM0MlqBDzIwMjYxMTI3MjA1MzQyWjAfBgNVHSMEGDAWgBRo\nkORnpKZTgMeGZqTx90tD+4S9bTAdBgNVHQ4EFgQUaJDkZ6SmU4DHhmak8fdLQ/uE\nvW0wHQYJKoZIhvZ9B0EABBAwDhsIVjcuMTo0LjADAgSQMA0GCSqGSIb3DQEBBQUA\nA4IBAQCT1DCw1wMgKtD5Y+iRDAUgqV8ZyntyTtSx29CW+1RaGSwMCPeyvIWonX9t\nO1KzKtvn1ISMY/YPyyYBkVBs9F8U4pN0wBOeMDpQ47RgxRzwIkSNcUesyBrJ6Zua\nAGAT/3B+XxFNSRuzFVJ7yVTav52Vr2ua2J7p8eRDjeIRRDq/r72DQnNSi6q7pynP\n9WQcCk3RvKqsnyrQ/39/2n3qse0wJcGE2jTSW3iDVuycNsMm4hH2Z0kdkquM++v/\neu6FSqdQgPCnXEqULl8FmTxSQeDNtGPPAUO6nIPcj2A781q0tHuu2guQOHXvgR1m\n0vdXcDazv/wor3ElhVsT/h5/WrQ8\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDIDCCAomgAwIBAgIENd70zzANBgkqhkiG9w0BAQUFADBOMQswCQYDVQQGEwJV\nUzEQMA4GA1UEChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2Vy\ndGlmaWNhdGUgQXV0aG9yaXR5MB4XDTk4MDgyMjE2NDE1MVoXDTE4MDgyMjE2NDE1\nMVowTjELMAkGA1UEBhMCVVMxEDAOBgNVBAoTB0VxdWlmYXgxLTArBgNVBAsTJEVx\ndWlmYXggU2VjdXJlIENlcnRpZmljYXRlIEF1dGhvcml0eTCBnzANBgkqhkiG9w0B\nAQEFAAOBjQAwgYkCgYEAwV2xWGcIYu6gmi0fCG2RFGiYCh7+2gRvE4RiIcPRfM6f\nBeC4AfBONOziipUEZKzxa1NfBbPLZ4C/QgKO/t0BCezhABRP/PvwDN1Dulsr4R+A\ncJkVV5MW8Q+XarfCaCMczE1ZMKxRHjuvK9buY0V7xdlfUNLjUA86iOe/FP3gx7kC\nAwEAAaOCAQkwggEFMHAGA1UdHwRpMGcwZaBjoGGkXzBdMQswCQYDVQQGEwJVUzEQ\nMA4GA1UEChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2VydGlm\naWNhdGUgQXV0aG9yaXR5MQ0wCwYDVQQDEwRDUkwxMBoGA1UdEAQTMBGBDzIwMTgw\nODIyMTY0MTUxWjALBgNVHQ8EBAMCAQYwHwYDVR0jBBgwFoAUSOZo+SvSspXXR9gj\nIBBPM5iQn9QwHQYDVR0OBBYEFEjmaPkr0rKV10fYIyAQTzOYkJ/UMAwGA1UdEwQF\nMAMBAf8wGgYJKoZIhvZ9B0EABA0wCxsFVjMuMGMDAgbAMA0GCSqGSIb3DQEBBQUA\nA4GBAFjOKer89961zgK5F7WF0bnj4JXMJTENAKaSbn+2kmOeUJXRmm/kEd5jhW6Y\n7qj/WsjTVbJmcVfewCHrPSqnI0kBBIZCe/zuf6IWUrVnZ9NA2zsmWLIodz2uFHdh\n1voqZiegDfqnc1zqcPGUIWVEX/r87yloqaKHee9570+sB3c4\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICkDCCAfmgAwIBAgIBATANBgkqhkiG9w0BAQQFADBaMQswCQYDVQQGEwJVUzEc\nMBoGA1UEChMTRXF1aWZheCBTZWN1cmUgSW5jLjEtMCsGA1UEAxMkRXF1aWZheCBT\nZWN1cmUgR2xvYmFsIGVCdXNpbmVzcyBDQS0xMB4XDTk5MDYyMTA0MDAwMFoXDTIw\nMDYyMTA0MDAwMFowWjELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE0VxdWlmYXggU2Vj\ndXJlIEluYy4xLTArBgNVBAMTJEVxdWlmYXggU2VjdXJlIEdsb2JhbCBlQnVzaW5l\nc3MgQ0EtMTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAuucXkAJlsTRVPEnC\nUdXfp9E3j9HngXNBUmCbnaEXJnitx7HoJpQytd4zjTov2/KaelpzmKNc6fuKcxtc\n58O/gGzNqfTWK8D3+ZmqY6KxRwIP1ORROhI8bIpaVIRw28HFkM9yRcuoWcDNM50/\no5brhTMhHD4ePmBudpxnhcXIw2ECAwEAAaNmMGQwEQYJYIZIAYb4QgEBBAQDAgAH\nMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUvqigdHJQa0S3ySPY+6j/s1dr\naGwwHQYDVR0OBBYEFL6ooHRyUGtEt8kj2Puo/7NXa2hsMA0GCSqGSIb3DQEBBAUA\nA4GBADDiAVGqx+pf2rnQZQ8w1j7aDRRJbpGTJxQx78T3LUX47Me/okENI7SS+RkA\nZ70Br83gcfxaz2TE4JaY0KNA4gGK7ycH8WUBikQtBmV1UsCGECAhX2xrD2yuCRyv\n8qIYNMR1pHMc8Y3c7635s3a0kr/clRAevsvIO1qEYBlWlKlV\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDVDCCAjygAwIBAgIDAjRWMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT\nMRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i\nYWwgQ0EwHhcNMDIwNTIxMDQwMDAwWhcNMjIwNTIxMDQwMDAwWjBCMQswCQYDVQQG\nEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMSR2VvVHJ1c3Qg\nR2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2swYYzD9\n9BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9mOSm9BXiLnTjoBbdq\nfnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIuT8rxh0PBFpVXLVDv\niS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6cJmTM386DGXHKTubU\n1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmRCw7+OC7RHQWa9k0+\nbw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5aszPeE4uwc2hGKceeoW\nMPRfwCvocWvk+QIDAQABo1MwUTAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTA\nephojYn7qwVkDBF9qn1luMrMTjAfBgNVHSMEGDAWgBTAephojYn7qwVkDBF9qn1l\nuMrMTjANBgkqhkiG9w0BAQUFAAOCAQEANeMpauUvXVSOKVCUn5kaFOSPeCpilKIn\nZ57QzxpeR+nBsqTP3UEaBU6bS+5Kb1VSsyShNwrrZHYqLizz/Tt1kL/6cdjHPTfS\ntQWVYrmm3ok9Nns4d0iXrKYgjy6myQzCsplFAMfOEVEiIuCl6rYVSAlk6l5PdPcF\nPseKUgzbFbS9bZvlxrFUaKnjaZC2mqUPuLk/IH2uSrW4nOQdtqvmlKXBx4Ot2/Un\nhw4EbNX/3aBd7YdStysVAq45pmp06drE57xNNB6pXE0zX5IJL4hmXXeXxx12E6nV\n5fEWCRE11azbJHFwLJhWC9kXtNHjUStedejV0NxPNO3CBWaAocvmMw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDZjCCAk6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBEMQswCQYDVQQGEwJVUzEW\nMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEdMBsGA1UEAxMUR2VvVHJ1c3QgR2xvYmFs\nIENBIDIwHhcNMDQwMzA0MDUwMDAwWhcNMTkwMzA0MDUwMDAwWjBEMQswCQYDVQQG\nEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEdMBsGA1UEAxMUR2VvVHJ1c3Qg\nR2xvYmFsIENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDvPE1A\nPRDfO1MA4Wf+lGAVPoWI8YkNkMgoI5kF6CsgncbzYEbYwbLVjDHZ3CB5JIG/NTL8\nY2nbsSpr7iFY8gjpeMtvy/wWUsiRxP89c96xPqfCfWbB9X5SJBri1WeR0IIQ13hL\nTytCOb1kLUCgsBDTOEhGiKEMuzozKmKY+wCdE1l/bztyqu6mD4b5BWHqZ38MN5aL\n5mkWRxHCJ1kDs6ZgwiFAVvqgx306E+PsV8ez1q6diYD3Aecs9pYrEw15LNnA5IZ7\nS4wMcoKK+xfNAGw6EzywhIdLFnopsk/bHdQL82Y3vdj2V7teJHq4PIu5+pIaGoSe\n2HSPqht/XvT+RSIhAgMBAAGjYzBhMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYE\nFHE4NvICMVNHK266ZUapEBVYIAUJMB8GA1UdIwQYMBaAFHE4NvICMVNHK266ZUap\nEBVYIAUJMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQUFAAOCAQEAA/e1K6td\nEPx7srJerJsOflN4WT5CBP51o62sgU7XAotexC3IUnbHLB/8gTKY0UvGkpMzNTEv\n/NgdRN3ggX+d6YvhZJFiCzkIjKx0nVnZellSlxG5FntvRdOW2TF9AjYPnDtuzywN\nA0ZF66D0f0hExghAzN4bcLUprbqLOzRldRtxIR0sFAqwlpW41uryZfspuk/qkZN0\nabby/+Ea0AzRdoXLiiW9l14sbxWZJue2Kf8i7MkCx1YAzUm5s2x7UwQa4qjJqhIF\nI8LO57sEAszAR6LkxCkvW0VXiVHuPOtSCP8HNR6fNWpHSlaY0VqFH4z1Ir+rzoPz\n4iIprn2DQKi6bA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDfDCCAmSgAwIBAgIQGKy1av1pthU6Y2yv2vrEoTANBgkqhkiG9w0BAQUFADBY\nMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjExMC8GA1UEAxMo\nR2VvVHJ1c3QgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNjEx\nMjcwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMFgxCzAJBgNVBAYTAlVTMRYwFAYDVQQK\nEw1HZW9UcnVzdCBJbmMuMTEwLwYDVQQDEyhHZW9UcnVzdCBQcmltYXJ5IENlcnRp\nZmljYXRpb24gQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEAvrgVe//UfH1nrYNke8hCUy3f9oQIIGHWAVlqnEQRr+92/ZV+zmEwu3qDXwK9\nAWbK7hWNb6EwnL2hhZ6UOvNWiAAxz9juapYC2e0DjPt1befquFUWBRaa9OBesYjA\nZIVcFU2Ix7e64HXprQU9nceJSOC7KMgD4TCTZF5SwFlwIjVXiIrxlQqD17wxcwE0\n7e9GceBrAqg1cmuXm2bgyxx5X9gaBGgeRwLmnWDiNpcB3841kt++Z8dtd1k7j53W\nkBWUvEI0EME5+bEnPn7WinXFsq+W06Lem+SYvn3h6YGttm/81w7a4DSwDRp35+MI\nmO9Y+pyEtzavwt+s0vQQBnBxNQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4G\nA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQULNVQQZcVi/CPNmFbSvtr2ZnJM5IwDQYJ\nKoZIhvcNAQEFBQADggEBAFpwfyzdtzRP9YZRqSa+S7iq8XEN3GHHoOo0Hnp3DwQ1\n6CePbJC/kRYkRj5KTs4rFtULUh38H2eiAkUxT87z+gOneZ1TatnaYzr4gNfTmeGl\n4b7UVXGYNTq+k+qurUKykG/g/CFNNWMziUnWm07Kx+dOCQD32sfvmWKZd7aVIl6K\noKv0uHiYyjgZmclynnjNS6yvGaBzEi38wkG6gZHaFloxt/m0cYASSJlyc1pZU8Fj\nUjPtp8nSOQJw+uCxQmYpqptR7TBUIhRf2asdweSU8Pj1K/fqynhG1riR/aYNKxoU\nAT6A8EKglQdebc3MS6RFjasS6LPeWuWgfOgPIh1a6Vk=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICrjCCAjWgAwIBAgIQPLL0SAoA4v7rJDteYD7DazAKBggqhkjOPQQDAzCBmDEL\nMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUdlb1RydXN0IEluYy4xOTA3BgNVBAsTMChj\nKSAyMDA3IEdlb1RydXN0IEluYy4gLSBGb3IgYXV0aG9yaXplZCB1c2Ugb25seTE2\nMDQGA1UEAxMtR2VvVHJ1c3QgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0\neSAtIEcyMB4XDTA3MTEwNTAwMDAwMFoXDTM4MDExODIzNTk1OVowgZgxCzAJBgNV\nBAYTAlVTMRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMTkwNwYDVQQLEzAoYykgMjAw\nNyBHZW9UcnVzdCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxNjA0BgNV\nBAMTLUdlb1RydXN0IFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBH\nMjB2MBAGByqGSM49AgEGBSuBBAAiA2IABBWx6P0DFUPlrOuHNxFi79KDNlJ9RVcL\nSo17VDs6bl8VAsBQps8lL33KSLjHUGMcKiEIfJo22Av+0SbFWDEwKCXzXV2juLal\ntJLtbCyf691DiaI8S0iRHVDsJt/WYC69IaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAO\nBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFBVfNVdRVfslsq0DafwBo/q+EVXVMAoG\nCCqGSM49BAMDA2cAMGQCMGSWWaboCd6LuvpaiIjwH5HTRqjySkwCY/tsXzjbLkGT\nqQ7mndwxHLKgpxgceeHHNgIwOlavmnRs9vuD4DPTCF+hnMJbn0bWtsuRBmOiBucz\nrD6ogRLQy7rQkgu2npaqBA+K\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID/jCCAuagAwIBAgIQFaxulBmyeUtB9iepwxgPHzANBgkqhkiG9w0BAQsFADCB\nmDELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUdlb1RydXN0IEluYy4xOTA3BgNVBAsT\nMChjKSAyMDA4IEdlb1RydXN0IEluYy4gLSBGb3IgYXV0aG9yaXplZCB1c2Ugb25s\neTE2MDQGA1UEAxMtR2VvVHJ1c3QgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhv\ncml0eSAtIEczMB4XDTA4MDQwMjAwMDAwMFoXDTM3MTIwMTIzNTk1OVowgZgxCzAJ\nBgNVBAYTAlVTMRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMTkwNwYDVQQLEzAoYykg\nMjAwOCBHZW9UcnVzdCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxNjA0\nBgNVBAMTLUdlb1RydXN0IFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg\nLSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANziXmJYHTNXOTIz\n+uvLh4yn1ErdBojqZI4xmKU4kB6Yzy5jK/BGvESyiaHAKAxJcCGVn2TAppMSAmUm\nhsalifD614SgcK9PGpc/BkTVyetyEH3kMSj7HGHmKAdEc5IiaacDiGydY8hS2pgn\n5whMcD60yRLBxWeDXTPzAxHsatBT4tG6NmCUgLthY2xbF37fQJQeqw3CIShwiP/W\nJmxsYAQlTlV+fe+/lEjetx3dcI0FX4ilm/LC7urRQEFtYjgdVgbFA0dRIBn8exAL\nDmKudlW/X3e+PkkBUz2YJQN2JFodtNuJ6nnltrM7P7pMKEF/BqxqjsHQ9gUdfeZC\nhuOl1UcCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw\nHQYDVR0OBBYEFMR5yo6hTgMdHNxr2zFblD4/MH8tMA0GCSqGSIb3DQEBCwUAA4IB\nAQAtxRPPVoB7eni9n64smefv2t+UXglpp+duaIy9cr5HqQ6XErhK8WTTOd8lNNTB\nzU6B8A8ExCSzNJbGpqow32hhc9f5joWJ7w5elShKKiePEI4ufIbEAp7aDHdlDkQN\nkv39sxY2+hENHYwOB4lqKVb3cvTdFZx3NWZXqxNT2I7BQMXXExZacse3aQHEerGD\nAWh9jUGhlBjBJVz88P6DAod8DQ3PLghcSkANPuyBYeYk28rgDi0Hsj5W3I31QYUH\nSJsMC8tJP33st/3LjWeJGqvtux6jAAgIFyqCXDFdRootD4abdNlF+9RAsXqqaC2G\nspki4cErx5z481+oghLrGREt\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFaDCCA1CgAwIBAgIBATANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJVUzEW\nMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEeMBwGA1UEAxMVR2VvVHJ1c3QgVW5pdmVy\nc2FsIENBMB4XDTA0MDMwNDA1MDAwMFoXDTI5MDMwNDA1MDAwMFowRTELMAkGA1UE\nBhMCVVMxFjAUBgNVBAoTDUdlb1RydXN0IEluYy4xHjAcBgNVBAMTFUdlb1RydXN0\nIFVuaXZlcnNhbCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAKYV\nVaCjxuAfjJ0hUNfBvitbtaSeodlyWL0AG0y/YckUHUWCq8YdgNY96xCcOq9tJPi8\ncQGeBvV8Xx7BDlXKg5pZMK4ZyzBIle0iN430SppyZj6tlcDgFgDgEB8rMQ7XlFTT\nQjOgNB0eRXbdT8oYN+yFFXoZCPzVx5zw8qkuEKmS5j1YPakWaDwvdSEYfyh3peFh\nF7em6fgemdtzbvQKoiFs7tqqhZJmr/Z6a4LauiIINQ/PQvE1+mrufislzDoR5G2v\nc7J2Ha3QsnhnGqQ5HFELZ1aD/ThdDc7d8Lsrlh/eezJS/R27tQahsiFepdaVaH/w\nmZ7cRQg+59IJDTWU3YBOU5fXtQlEIGQWFwMCTFMNaN7VqnJNk22CDtucvc+081xd\nVHppCZbW2xHBjXWotM85yM48vCR85mLK4b19p71XZQvk/iXttmkQ3CgaRr0BHdCX\nteGYO8A3ZNY9lO4L4fUorgtWv3GLIylBjobFS1J72HGrH4oVpjuDWtdYAVHGTEHZ\nf9hBZ3KiKN9gg6meyHv8U3NyWfWTehd2Ds735VzZC1U0oqpbtWpU5xPKV+yXbfRe\nBi9Fi1jUIxaS5BZuKGNZMN9QAZxjiRqf2xeUgnA3wySemkfWWspOqGmJch+RbNt+\nnhutxx9z3SxPGWX9f5NAEC7S8O08ni4oPmkmM8V7AgMBAAGjYzBhMA8GA1UdEwEB\n/wQFMAMBAf8wHQYDVR0OBBYEFNq7LqqwDLiIJlF0XG0D08DYj3rWMB8GA1UdIwQY\nMBaAFNq7LqqwDLiIJlF0XG0D08DYj3rWMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG\n9w0BAQUFAAOCAgEAMXjmx7XfuJRAyXHEqDXsRh3ChfMoWIawC/yOsjmPRFWrZIRc\naanQmjg8+uUfNeVE44B5lGiku8SfPeE0zTBGi1QrlaXv9z+ZhP015s8xxtxqv6fX\nIwjhmF7DWgh2qaavdy+3YL1ERmrvl/9zlcGO6JP7/TG37FcREUWbMPEaiDnBTzyn\nANXH/KttgCJwpQzgXQQpAvvLoJHRfNbDflDVnVi+QTjruXU8FdmbyUqDWcDaU/0z\nuzYYm4UPFd3uLax2k7nZAY1IEKj79TiG8dsKxr2EoyNB3tZ3b4XUhRxQ4K5RirqN\nPnbiucon8l+f725ZDQbYKxek0nxru18UGkiPGkzns0ccjkxFKyDuSN/n3QmOGKja\nQI2SJhFTYXNd673nxE0pN2HrrDktZy4W1vUAg4WhzH92xH3kt0tm7wNFYGm2DFKW\nkoRepqO1pD4r2czYG0eq8kTaT/kD6PAUyz/zg97QwVTjt+gKN02LIFkDMBmhLMi9\nER/frslKxfMnZmaGrGiR/9nmUxwPi1xpZQomyB40w11Re9epnAahNt3ViZS82eQt\nDF4JbAiXfKM9fJP/P6EUp8+1Xevb2xzEdt+Iub1FBZUbrvxGakyvSOPOrg/Sfuvm\nbJxPgWp6ZKy7PtXny3YuxadIwVyQD8vIP/rmMuGNG2+k5o7Y+SlIis5z/iw=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFbDCCA1SgAwIBAgIBATANBgkqhkiG9w0BAQUFADBHMQswCQYDVQQGEwJVUzEW\nMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEgMB4GA1UEAxMXR2VvVHJ1c3QgVW5pdmVy\nc2FsIENBIDIwHhcNMDQwMzA0MDUwMDAwWhcNMjkwMzA0MDUwMDAwWjBHMQswCQYD\nVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEgMB4GA1UEAxMXR2VvVHJ1\nc3QgVW5pdmVyc2FsIENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC\nAQCzVFLByT7y2dyxUxpZKeexw0Uo5dfR7cXFS6GqdHtXr0om/Nj1XqduGdt0DE81\nWzILAePb63p3NeqqWuDW6KFXlPCQo3RWlEQwAx5cTiuFJnSCegx2oG9NzkEtoBUG\nFF+3Qs17j1hhNNwqCPkuwwGmIkQcTAeC5lvO0Ep8BNMZcyfwqph/Lq9O64ceJHdq\nXbboW0W63MOhBW9Wjo8QJqVJwy7XQYci4E+GymC16qFjwAGXEHm9ADwSbSsVsaxL\nse4YuU6W3Nx2/zu+z18DwPw76L5GG//aQMJS9/7jOvdqdzXQ2o3rXhhqMcceujwb\nKNZrVMaqW9eiLBsZzKIC9ptZvTdrhrVtgrrY6slWvKk2WP0+GfPtDCapkzj4T8Fd\nIgbQl+rhrcZV4IErKIM6+vR7IVEAvlI4zs1meaj0gVbi0IMJR1FbUGrP20gaXT73\ny/Zl92zxlfgCOzJWgjl6W70viRu/obTo/3+NjN8D8WBOWBFM66M/ECuDmgFz2ZRt\nhAAnZqzwcEAJQpKtT5MNYQlRJNiS1QuUYbKHsu3/mjX/hVTK7URDrBs8FmtISgoc\nQIgfksILAAX/8sgCSqSqqcyZlpwvWOB94b67B9xfBHJcMTTD7F8t4D1kkCLm0ey4\nLt1ZrtmhN79UNdxzMk+MBB4zsslG8dhcyFVQyWi9qLo2CQIDAQABo2MwYTAPBgNV\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR281Xh+qQ2+/CfXGJx7Tz0RzgQKzAfBgNV\nHSMEGDAWgBR281Xh+qQ2+/CfXGJx7Tz0RzgQKzAOBgNVHQ8BAf8EBAMCAYYwDQYJ\nKoZIhvcNAQEFBQADggIBAGbBxiPz2eAubl/oz66wsCVNK/g7WJtAJDday6sWSf+z\ndXkzoS9tcBc0kf5nfo/sm+VegqlVHy/c1FEHEv6sFj4sNcZj/NwQ6w2jqtB8zNHQ\nL1EuxBRa3ugZ4T7GzKQp5y6EqgYweHZUcyiYWTjgAA1i00J9IZ+uPTqM1fp3DRgr\nFg5fNuH8KrUwJM/gYwx7WBr+mbpCErGR9Hxo4sjoryzqyX6uuyo9DRXcNJW2GHSo\nag/HtPQTxORb7QrSpJdMKu0vbBKJPfEncKpqA1Ihn0CoZ1Dy81of398j9tx4TuaY\nT1U6U+Pv8vSfx3zYWK8pIpe44L2RLrB27FcRz+8pRPPphXpgY+RdM4kX2TGq2tbz\nGDVyz4crL2MjhF2EjD9XoIj8mZEoJmmZ1I+XRL6O1UixpCgp8RW04eWe3fiPpm8m\n1wk8OhwRDqZsN/etRIcsKMfYdIKz0G9KV7s1KSegi+ghp4dkNl3M2Basx7InQJJV\nOCiNUW7dFGdTbHFcJoRNdVq2fmBWqU2t+5sel/MN2dKXVHfaPRK34B7vCAas+YWH\n6aLcr34YEoP9VhdBLtUpgn2Z9DH2canPLAEnpQW5qrJITirvn5NSUZU8UnOOVkwX\nQMAJKOSLakhT2+zNVVXxxvjpoixMptEmX36vWkzaH6byHCx+rgIW0lbQL1dTR+iS\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIB4TCCAYegAwIBAgIRKjikHJYKBN5CsiilC+g0mAIwCgYIKoZIzj0EAwIwUDEk\nMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI0MRMwEQYDVQQKEwpH\nbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX\nDTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD\nQSAtIFI0MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEuMZ5049sJQ6fLjkZHAOkrprlOQcJ\nFspjsbmG+IpXwVfOQvpzofdlQv8ewQCybnMO/8ch5RikqtlxP6jUuc6MHaNCMEAw\nDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFSwe61F\nuOJAf/sKbvu+M8k8o4TVMAoGCCqGSM49BAMCA0gAMEUCIQDckqGgE6bPA7DmxCGX\nkPoUVy0D7O48027KqGx2vKLeuwIgJ6iFJzWbVsaj8kfSt24bAgAXqmemFZHe+pTs\newv4n4Q=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICHjCCAaSgAwIBAgIRYFlJ4CYuu1X5CneKcflK2GwwCgYIKoZIzj0EAwMwUDEk\nMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI1MRMwEQYDVQQKEwpH\nbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX\nDTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD\nQSAtIFI1MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAER0UOlvt9Xb/pOdEh+J8LttV7HpI6SFkc\n8GIxLcB6KP4ap1yztsyX50XUWPrRd21DosCHZTQKH3rd6zwzocWdTaRvQZU4f8ke\nhOvRnkmSh5SHDDqFSmafnVmTTZdhBoZKo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYD\nVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUPeYpSJvqB8ohREom3m7e0oPQn1kwCgYI\nKoZIzj0EAwMDaAAwZQIxAOVpEslu28YxuglB4Zf4+/2a4n0Sye18ZNPLBSWLVtmg\n515dTguDnFt2KaAJJiFqYgIwcdK1j1zqO+F4CYWodZI7yFz9SO8NdCKoCOJuxUnO\nxwy8p2Fp8fc74SrL+SvzZpA3\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG\nA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv\nb3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw\nMDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i\nYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT\naWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ\njc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp\nxy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp\n1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG\nsnUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ\nU26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8\n9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E\nBTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B\nAQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz\nyj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE\n38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP\nAbEVtQwdpf5pLGkkeB6zpxxxYu7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad\nDKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME\nHMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDujCCAqKgAwIBAgILBAAAAAABD4Ym5g0wDQYJKoZIhvcNAQEFBQAwTDEgMB4G\nA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjIxEzARBgNVBAoTCkdsb2JhbFNp\nZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMDYxMjE1MDgwMDAwWhcNMjExMjE1\nMDgwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMjETMBEG\nA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAKbPJA6+Lm8omUVCxKs+IVSbC9N/hHD6ErPL\nv4dfxn+G07IwXNb9rfF73OX4YJYJkhD10FPe+3t+c4isUoh7SqbKSaZeqKeMWhG8\neoLrvozps6yWJQeXSpkqBy+0Hne/ig+1AnwblrjFuTosvNYSuetZfeLQBoZfXklq\ntTleiDTsvHgMCJiEbKjNS7SgfQx5TfC4LcshytVsW33hoCmEofnTlEnLJGKRILzd\nC9XZzPnqJworc5HGnRusyMvo4KD0L5CLTfuwNhv2GXqF4G3yYROIXJ/gkwpRl4pa\nzq+r1feqCapgvdzZX99yqWATXgAByUr6P6TqBwMhAo6CygPCm48CAwEAAaOBnDCB\nmTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUm+IH\nV2ccHsBqBt5ZtJot39wZhi4wNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2NybC5n\nbG9iYWxzaWduLm5ldC9yb290LXIyLmNybDAfBgNVHSMEGDAWgBSb4gdXZxwewGoG\n3lm0mi3f3BmGLjANBgkqhkiG9w0BAQUFAAOCAQEAmYFThxxol4aR7OBKuEQLq4Gs\nJ0/WwbgcQ3izDJr86iw8bmEbTUsp9Z8FHSbBuOmDAGJFtqkIk7mpM0sYmsL4h4hO\n291xNBrBVNpGP+DTKqttVCL1OmLNIG+6KYnX3ZHu01yiPqFbQfXf5WRDLenVOavS\not+3i9DAgBkcRcAtjOj4LaR0VknFBbVPFd5uRHg5h6h+u/N5GJG79G+dwfCMNYxd\nAfvDbbnvRG15RjF+Cv6pgsH/76tuIMRQyV+dTZsXjAzlAcmgQWpzU/qlULRuJQ/7\nTBj0/VLZjmmx6BEP3ojY+x1J96relc8geMJgEtslQIxq/H5COEBkEveegeGTLg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDXzCCAkegAwIBAgILBAAAAAABIVhTCKIwDQYJKoZIhvcNAQELBQAwTDEgMB4G\nA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNp\nZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMDkwMzE4MTAwMDAwWhcNMjkwMzE4\nMTAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEG\nA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAMwldpB5BngiFvXAg7aEyiie/QV2EcWtiHL8\nRgJDx7KKnQRfJMsuS+FggkbhUqsMgUdwbN1k0ev1LKMPgj0MK66X17YUhhB5uzsT\ngHeMCOFJ0mpiLx9e+pZo34knlTifBtc+ycsmWQ1z3rDI6SYOgxXG71uL0gRgykmm\nKPZpO/bLyCiR5Z2KYVc3rHQU3HTgOu5yLy6c+9C7v/U9AOEGM+iCK65TpjoWc4zd\nQQ4gOsC0p6Hpsk+QLjJg6VfLuQSSaGjlOCZgdbKfd/+RFO+uIEn8rUAVSNECMWEZ\nXriX7613t2Saer9fwRPvm2L7DWzgVGkWqQPabumDk3F2xmmFghcCAwEAAaNCMEAw\nDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFI/wS3+o\nLkUkrk1Q+mOai97i3Ru8MA0GCSqGSIb3DQEBCwUAA4IBAQBLQNvAUKr+yAzv95ZU\nRUm7lgAJQayzE4aGKAczymvmdLm6AC2upArT9fHxD4q/c2dKg8dEe3jgr25sbwMp\njjM5RcOO5LlXbKr8EpbsU8Yt5CRsuZRj+9xTaGdWPoO4zzUhw8lo/s7awlOqzJCK\n6fBdRoyV3XpYKBovHd7NADdBj+1EbddTKJd+82cEHhXXipa0095MJ6RMG3NzdvQX\nmcIfeg7jLQitChws/zyrVQ4PkX4268NXSb7hLi18YIvDQVETI53O9zJrlAGomecs\nMx86OyXShkDOOyyGeMlhLxS67ttVb9+E7gUJTb0o2HLO02JQZR7rkpeDMdmztcpH\nWD9f\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIHSTCCBTGgAwIBAgIJAMnN0+nVfSPOMA0GCSqGSIb3DQEBBQUAMIGsMQswCQYD\nVQQGEwJFVTFDMEEGA1UEBxM6TWFkcmlkIChzZWUgY3VycmVudCBhZGRyZXNzIGF0\nIHd3dy5jYW1lcmZpcm1hLmNvbS9hZGRyZXNzKTESMBAGA1UEBRMJQTgyNzQzMjg3\nMRswGQYDVQQKExJBQyBDYW1lcmZpcm1hIFMuQS4xJzAlBgNVBAMTHkdsb2JhbCBD\naGFtYmVyc2lnbiBSb290IC0gMjAwODAeFw0wODA4MDExMjMxNDBaFw0zODA3MzEx\nMjMxNDBaMIGsMQswCQYDVQQGEwJFVTFDMEEGA1UEBxM6TWFkcmlkIChzZWUgY3Vy\ncmVudCBhZGRyZXNzIGF0IHd3dy5jYW1lcmZpcm1hLmNvbS9hZGRyZXNzKTESMBAG\nA1UEBRMJQTgyNzQzMjg3MRswGQYDVQQKExJBQyBDYW1lcmZpcm1hIFMuQS4xJzAl\nBgNVBAMTHkdsb2JhbCBDaGFtYmVyc2lnbiBSb290IC0gMjAwODCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBAMDfVtPkOpt2RbQT2//BthmLN0EYlVJH6xed\nKYiONWwGMi5HYvNJBL99RDaxccy9Wglz1dmFRP+RVyXfXjaOcNFccUMd2drvXNL7\nG706tcuto8xEpw2uIRU/uXpbknXYpBI4iRmKt4DS4jJvVpyR1ogQC7N0ZJJ0YPP2\nzxhPYLIj0Mc7zmFLmY/CDNBAspjcDahOo7kKrmCgrUVSY7pmvWjg+b4aqIG7HkF4\nddPB/gBVsIdU6CeQNR1MM62X/JcumIS/LMmjv9GYERTtY/jKmIhYF5ntRQOXfjyG\nHoiMvvKRhI9lNNgATH23MRdaKXoKGCQwoze1eqkBfSbW+Q6OWfH9GzO1KTsXO0G2\nId3UwD2ln58fQ1DJu7xsepeY7s2MH/ucUa6LcL0nn3HAa6x9kGbo1106DbDVwo3V\nyJ2dwW3Q0L9R5OP4wzg2rtandeavhENdk5IMagfeOx2YItaswTXbo6Al/3K1dh3e\nbeksZixShNBFks4c5eUzHdwHU1SjqoI7mjcv3N2gZOnm3b2u/GSFHTynyQbehP9r\n6GsaPMWis0L7iwk+XwhSx2LE1AVxv8Rk5Pihg+g+EpuoHtQ2TS9x9o0o9oOpE9Jh\nwZG7SMA0j0GMS0zbaRL/UJScIINZc+18ofLx/d33SdNDWKBWY8o9PeU1VlnpDsog\nzCtLkykPAgMBAAGjggFqMIIBZjASBgNVHRMBAf8ECDAGAQH/AgEMMB0GA1UdDgQW\nBBS5CcqcHtvTbDprru1U8VuTBjUuXjCB4QYDVR0jBIHZMIHWgBS5CcqcHtvTbDpr\nru1U8VuTBjUuXqGBsqSBrzCBrDELMAkGA1UEBhMCRVUxQzBBBgNVBAcTOk1hZHJp\nZCAoc2VlIGN1cnJlbnQgYWRkcmVzcyBhdCB3d3cuY2FtZXJmaXJtYS5jb20vYWRk\ncmVzcykxEjAQBgNVBAUTCUE4Mjc0MzI4NzEbMBkGA1UEChMSQUMgQ2FtZXJmaXJt\nYSBTLkEuMScwJQYDVQQDEx5HbG9iYWwgQ2hhbWJlcnNpZ24gUm9vdCAtIDIwMDiC\nCQDJzdPp1X0jzjAOBgNVHQ8BAf8EBAMCAQYwPQYDVR0gBDYwNDAyBgRVHSAAMCow\nKAYIKwYBBQUHAgEWHGh0dHA6Ly9wb2xpY3kuY2FtZXJmaXJtYS5jb20wDQYJKoZI\nhvcNAQEFBQADggIBAICIf3DekijZBZRG/5BXqfEv3xoNa/p8DhxJJHkn2EaqbylZ\nUohwEurdPfWbU1Rv4WCiqAm57OtZfMY18dwY6fFn5a+6ReAJ3spED8IXDneRRXoz\nX1+WLGiLwUePmJs9wOzL9dWCkoQ10b42OFZyMVtHLaoXpGNR6woBrX/sdZ7LoR/x\nfxKxueRkf2fWIyr0uDldmOghp+G9PUIadJpwr2hsUF1Jz//7Dl3mLEfXgTpZALVz\na2Mg9jFFCDkO9HB+QHBaP9BrQql0PSgvAm11cpUJjUhjxsYjV5KTXjXBjfkK9yyd\nYhz2rXzdpjEetrHHfoUm+qRqtdpjMNHvkzeyZi99Bffnt0uYlDXA2TopwZ2yUDMd\nSqlapskD7+3056huirRXhOukP9DuqqqHW2Pok+JrqNS4cnhrG+055F3Lm6qH1U9O\nAP7Zap88MQ8oAgF9mOinsKJknnn4SPIVqczmyETrP3iZ8ntxPjzxmKfFGBI/5rso\nM0LpRQp8bfKGeS/Fghl9CYl8slR2iK7ewfPM4W7bMdaTrpmg7yVqc5iJWzouE4ge\nv8CSlDQb4ye3ix5vQv/n6TebUB0tovkC7stYWDpxvGjjqsGvHCgfotwjZT+B6q6Z\n09gwzxMNTxXJhLynSC34MCN32EZLeW32jO06f2ARePTpm67VVMB0gNELQp/B\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEADCCAuigAwIBAgIBADANBgkqhkiG9w0BAQUFADBjMQswCQYDVQQGEwJVUzEh\nMB8GA1UEChMYVGhlIEdvIERhZGR5IEdyb3VwLCBJbmMuMTEwLwYDVQQLEyhHbyBE\nYWRkeSBDbGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA0MDYyOTE3\nMDYyMFoXDTM0MDYyOTE3MDYyMFowYzELMAkGA1UEBhMCVVMxITAfBgNVBAoTGFRo\nZSBHbyBEYWRkeSBHcm91cCwgSW5jLjExMC8GA1UECxMoR28gRGFkZHkgQ2xhc3Mg\nMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTCCASAwDQYJKoZIhvcNAQEBBQADggEN\nADCCAQgCggEBAN6d1+pXGEmhW+vXX0iG6r7d/+TvZxz0ZWizV3GgXne77ZtJ6XCA\nPVYYYwhv2vLM0D9/AlQiVBDYsoHUwHU9S3/Hd8M+eKsaA7Ugay9qK7HFiH7Eux6w\nwdhFJ2+qN1j3hybX2C32qRe3H3I2TqYXP2WYktsqbl2i/ojgC95/5Y0V4evLOtXi\nEqITLdiOr18SPaAIBQi2XKVlOARFmR6jYGB0xUGlcmIbYsUfb18aQr4CUWWoriMY\navx4A6lNf4DD+qta/KFApMoZFv6yyO9ecw3ud72a9nmYvLEHZ6IVDd2gWMZEewo+\nYihfukEHU1jPEX44dMX4/7VpkI+EdOqXG68CAQOjgcAwgb0wHQYDVR0OBBYEFNLE\nsNKR1EwRcbNhyz2h/t2oatTjMIGNBgNVHSMEgYUwgYKAFNLEsNKR1EwRcbNhyz2h\n/t2oatTjoWekZTBjMQswCQYDVQQGEwJVUzEhMB8GA1UEChMYVGhlIEdvIERhZGR5\nIEdyb3VwLCBJbmMuMTEwLwYDVQQLEyhHbyBEYWRkeSBDbGFzcyAyIENlcnRpZmlj\nYXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQAD\nggEBADJL87LKPpH8EsahB4yOd6AzBhRckB4Y9wimPQoZ+YeAEW5p5JYXMP80kWNy\nOO7MHAGjHZQopDH2esRU1/blMVgDoszOYtuURXO1v0XJJLXVggKtI3lpjbi2Tc7P\nTMozI+gciKqdi0FuFskg5YmezTvacPd+mSYgFFQlq25zheabIZ0KbIIOqPjCDPoQ\nHmyW74cNxA9hi63ugyuV+I6ShHI56yDqg+2DzZduCLzrTia2cyvk0/ZM/iZx4mER\ndEr/VxqHD3VILs9RaRegAhJhldXRQLIQTO7ErBBDpqWeCtWVYpoNz4iCxTIM5Cuf\nReYNnyicsbkqWletNw+vHX/bvZ8=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDxTCCAq2gAwIBAgIBADANBgkqhkiG9w0BAQsFADCBgzELMAkGA1UEBhMCVVMx\nEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAYBgNVBAoT\nEUdvRGFkZHkuY29tLCBJbmMuMTEwLwYDVQQDEyhHbyBEYWRkeSBSb290IENlcnRp\nZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5MDkwMTAwMDAwMFoXDTM3MTIzMTIz\nNTk1OVowgYMxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdBcml6b25hMRMwEQYDVQQH\nEwpTY290dHNkYWxlMRowGAYDVQQKExFHb0RhZGR5LmNvbSwgSW5jLjExMC8GA1UE\nAxMoR28gRGFkZHkgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjCCASIw\nDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL9xYgjx+lk09xvJGKP3gElY6SKD\nE6bFIEMBO4Tx5oVJnyfq9oQbTqC023CYxzIBsQU+B07u9PpPL1kwIuerGVZr4oAH\n/PMWdYA5UXvl+TW2dE6pjYIT5LY/qQOD+qK+ihVqf94Lw7YZFAXK6sOoBJQ7Rnwy\nDfMAZiLIjWltNowRGLfTshxgtDj6AozO091GB94KPutdfMh8+7ArU6SSYmlRJQVh\nGkSBjCypQ5Yj36w6gZoOKcUcqeldHraenjAKOc7xiID7S13MMuyFYkMlNAJWJwGR\ntDtwKj9useiciAF9n9T521NtYJ2/LOdYq7hfRvzOxBsDPAnrSTFcaUaz4EcCAwEA\nAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE\nFDqahQcQZyi27/a9BUFuIMGU2g/eMA0GCSqGSIb3DQEBCwUAA4IBAQCZ21151fmX\nWWcDYfF+OwYxdS2hII5PZYe096acvNjpL9DbWu7PdIxztDhC2gV7+AJ1uP2lsdeu\n9tfeE8tTEH6KRtGX+rcuKxGrkLAngPnon1rpN5+r5N9ss4UXnT3ZJE95kTXWXwTr\ngIOrmgIttRD02JDHBHNA7XIloKmf7J6raBKZV8aPEjoJpL1E/QYVN8Gb5DKj7Tjo\n2GTzLH4U/ALqn83/B2gX2yKQOC16jdFU8WnjXzPKej17CuPKf1855eJ1usV2GDPO\nLPAvTK33sefOT6jEm0pUBsV/fdUID+Ic/n4XuKxe9tQWskMJDE32p2u0mYRlynqI\n4uJEvlz36hz1\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEMTCCAxmgAwIBAgIBADANBgkqhkiG9w0BAQUFADCBlTELMAkGA1UEBhMCR1Ix\nRDBCBgNVBAoTO0hlbGxlbmljIEFjYWRlbWljIGFuZCBSZXNlYXJjaCBJbnN0aXR1\ndGlvbnMgQ2VydC4gQXV0aG9yaXR5MUAwPgYDVQQDEzdIZWxsZW5pYyBBY2FkZW1p\nYyBhbmQgUmVzZWFyY2ggSW5zdGl0dXRpb25zIFJvb3RDQSAyMDExMB4XDTExMTIw\nNjEzNDk1MloXDTMxMTIwMTEzNDk1MlowgZUxCzAJBgNVBAYTAkdSMUQwQgYDVQQK\nEztIZWxsZW5pYyBBY2FkZW1pYyBhbmQgUmVzZWFyY2ggSW5zdGl0dXRpb25zIENl\ncnQuIEF1dGhvcml0eTFAMD4GA1UEAxM3SGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl\nc2VhcmNoIEluc3RpdHV0aW9ucyBSb290Q0EgMjAxMTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAKlTAOMupvaO+mDYLZU++CwqVE7NuYRhlFhPjz2L5EPz\ndYmNUeTDN9KKiE15HrcS3UN4SoqS5tdI1Q+kOilENbgH9mgdVc04UfCMJDGFr4PJ\nfel3r+0ae50X+bOdOFAPplp5kYCvN66m0zH7tSYJnTxa71HFK9+WXesyHgLacEns\nbgzImjeN9/E2YEsmLIKe0HjzDQ9jpFEw4fkrJxIH2Oq9GGKYsFk3fb7u8yBRQlqD\n75O6aRXxYp2fmTmCobd0LovUxQt7L/DICto9eQqakxylKHJzkUOap9FNhYS5qXSP\nFEDH3N6sQWRstBmbAmNtJGSPRLIl6s5ddAxjMlyNh+UCAwEAAaOBiTCBhjAPBgNV\nHRMBAf8EBTADAQH/MAsGA1UdDwQEAwIBBjAdBgNVHQ4EFgQUppFC/RNhSiOeCKQp\n5dgTBCPuQSUwRwYDVR0eBEAwPqA8MAWCAy5ncjAFggMuZXUwBoIELmVkdTAGggQu\nb3JnMAWBAy5ncjAFgQMuZXUwBoEELmVkdTAGgQQub3JnMA0GCSqGSIb3DQEBBQUA\nA4IBAQAf73lB4XtuP7KMhjdCSk4cNx6NZrokgclPEg8hwAOXhiVtXdMiKahsog2p\n6z0GW5k6x8zDmjR/qw7IThzh+uTczQ2+vyT+bOdrwg3IBp5OjWEopmr95fZi6hg8\nTqBTnbI6nOulnJEWtk2C4AwFSKls9cz4y51JtPACpf1wA+2KIaWuE4ZJwzNzvoc7\ndIsXRSZMFpGD/md9zU1jZ/rzAxKWeAaNsWftjj++n08C9bMJL/NMh98qy5V8Acys\nNnq/onN694/BtZqhFLKPM58N7yLcZnuEvUUXBj08yrl3NI/K6s8/MT7jiOOASSXI\nl7WdmplNsDz4SgCbZN2fOUvRJ9e4\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDMDCCAhigAwIBAgICA+gwDQYJKoZIhvcNAQEFBQAwRzELMAkGA1UEBhMCSEsx\nFjAUBgNVBAoTDUhvbmdrb25nIFBvc3QxIDAeBgNVBAMTF0hvbmdrb25nIFBvc3Qg\nUm9vdCBDQSAxMB4XDTAzMDUxNTA1MTMxNFoXDTIzMDUxNTA0NTIyOVowRzELMAkG\nA1UEBhMCSEsxFjAUBgNVBAoTDUhvbmdrb25nIFBvc3QxIDAeBgNVBAMTF0hvbmdr\nb25nIFBvc3QgUm9vdCBDQSAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEArP84tulmAknjorThkPlAj3n54r15/gK97iSSHSL22oVyaf7XPwnU3ZG1ApzQ\njVrhVcNQhrkpJsLj2aDxaQMoIIBFIi1WpztUlVYiWR8o3x8gPW2iNr4joLFutbEn\nPzlTCeqrauh0ssJlXI6/fMN4hM2eFvz1Lk8gKgifd/PFHsSaUmYeSF7jEAaPIpjh\nZY4bXSNmO7ilMlHIhqqhqZ5/dpTCpmy3QfDVyAY45tQM4vM7TG1QjMSDJ8EThFk9\nnnV0ttgCXjqQesBCNnLsak3c78QA3xMYV18meMjWCnl3v/evt3a5pQuEF10Q6m/h\nq5URX208o1xNg1vysxmKgIsLhwIDAQABoyYwJDASBgNVHRMBAf8ECDAGAQH/AgED\nMA4GA1UdDwEB/wQEAwIBxjANBgkqhkiG9w0BAQUFAAOCAQEADkbVPK7ih9legYsC\nmEEIjEy82tvuJxuC52pF7BaLT4Wg87JwvVqWuspube5Gi27nKi6Wsxkz67SfqLI3\n7piol7Yutmcn1KZJ/RyTZXaeQi/cImyaT/JaFTmxcdcrUehtHJjA2Sr0oYJ71clB\noiMBdDhViw+5LmeiIAQ32pwL0xch4I+XeTRvhEgCIDMb5jREn5Fw9IBehEPCKdJs\nEhTkYY2sEJCehFC78JZvRZ+K88psT/oROhUVRsPNH4NbLUES7VBnQRM9IauUiqpO\nfMGx+6fWtScvl6tu4B3i0RwsH0Ti/L6RoZz71ilTc4afU9hDDl3WY4JxHYB0yvbi\nAmvZWg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEAjCCAuqgAwIBAgIFORFFEJQwDQYJKoZIhvcNAQEFBQAwgYUxCzAJBgNVBAYT\nAkZSMQ8wDQYDVQQIEwZGcmFuY2UxDjAMBgNVBAcTBVBhcmlzMRAwDgYDVQQKEwdQ\nTS9TR0ROMQ4wDAYDVQQLEwVEQ1NTSTEOMAwGA1UEAxMFSUdDL0ExIzAhBgkqhkiG\n9w0BCQEWFGlnY2FAc2dkbi5wbS5nb3V2LmZyMB4XDTAyMTIxMzE0MjkyM1oXDTIw\nMTAxNzE0MjkyMlowgYUxCzAJBgNVBAYTAkZSMQ8wDQYDVQQIEwZGcmFuY2UxDjAM\nBgNVBAcTBVBhcmlzMRAwDgYDVQQKEwdQTS9TR0ROMQ4wDAYDVQQLEwVEQ1NTSTEO\nMAwGA1UEAxMFSUdDL0ExIzAhBgkqhkiG9w0BCQEWFGlnY2FAc2dkbi5wbS5nb3V2\nLmZyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsh/R0GLFMzvABIaI\ns9z4iPf930Pfeo2aSVz2TqrMHLmh6yeJ8kbpO0px1R2OLc/mratjUMdUC24SyZA2\nxtgv2pGqaMVy/hcKshd+ebUyiHDKcMCWSo7kVc0dJ5S/znIq7Fz5cyD+vfcuiWe4\nu0dzEvfRNWk68gq5rv9GQkaiv6GFGvm/5P9JhfejcIYyHF2fYPepraX/z9E0+X1b\nF8bc1g4oa8Ld8fUzaJ1O/Id8NhLWo4DoQw1VYZTqZDdH6nfK0LJYBcNdfrGoRpAx\nVs5wKpayMLh35nnAvSk7/ZR3TL0gzUEl4C7HG7vupARB0l2tEmqKm0f7yd1GQOGd\nPDPQtQIDAQABo3cwdTAPBgNVHRMBAf8EBTADAQH/MAsGA1UdDwQEAwIBRjAVBgNV\nHSAEDjAMMAoGCCqBegF5AQEBMB0GA1UdDgQWBBSjBS8YYFDCiQrdKyFP/45OqDAx\nNjAfBgNVHSMEGDAWgBSjBS8YYFDCiQrdKyFP/45OqDAxNjANBgkqhkiG9w0BAQUF\nAAOCAQEABdwm2Pp3FURo/C9mOnTgXeQp/wYHE4RKq89toB9RlPhJy3Q2FLwV3duJ\nL92PoF189RLrn544pEfMs5bZvpwlqwN+Mw+VgQ39FuCIvjfwbF3QMZsyK10XZZOY\nYLxuj7GoPB7ZHPOpJkL5ZB3C55L29B5aqhlSXa/oovdgoPaN8In1buAKBQGVyYsg\nCrpa/JosPL3Dt8ldeCUFP1YUmwza+zpI/pdpXsoQhvdOlgQITeywvl3cO45Pwf2a\nNjSaTFR+FwNIlQgRHAdvhQh+XU3Endv7rs6y0bO4g2wdsrN58dhwmX7wEwLOXt1R\n0982gaEbeC9xs/FZTEYYKKuF0mBWWg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF8TCCA9mgAwIBAgIQALC3WhZIX7/hy/WL1xnmfTANBgkqhkiG9w0BAQsFADA4\nMQswCQYDVQQGEwJFUzEUMBIGA1UECgwLSVpFTlBFIFMuQS4xEzARBgNVBAMMCkl6\nZW5wZS5jb20wHhcNMDcxMjEzMTMwODI4WhcNMzcxMjEzMDgyNzI1WjA4MQswCQYD\nVQQGEwJFUzEUMBIGA1UECgwLSVpFTlBFIFMuQS4xEzARBgNVBAMMCkl6ZW5wZS5j\nb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDJ03rKDx6sp4boFmVq\nscIbRTJxldn+EFvMr+eleQGPicPK8lVx93e+d5TzcqQsRNiekpsUOqHnJJAKClaO\nxdgmlOHZSOEtPtoKct2jmRXagaKH9HtuJneJWK3W6wyyQXpzbm3benhB6QiIEn6H\nLmYRY2xU+zydcsC8Lv/Ct90NduM61/e0aL6i9eOBbsFGb12N4E3GVFWJGjMxCrFX\nuaOKmMPsOzTFlUFpfnXCPCDFYbpRR6AgkJOhkEvzTnyFRVSa0QUmQbC1TR0zvsQD\nyCV8wXDbO/QJLVQnSKwv4cSsPsjLkkxTOTcj7NMB+eAJRE1NZMDhDVqHIrytG6P+\nJrUV86f8hBnp7KGItERphIPzidF0BqnMC9bC3ieFUCbKF7jJeodWLBoBHmy+E60Q\nrLUk9TiRodZL2vG70t5HtfG8gfZZa88ZU+mNFctKy6lvROUbQc/hhqfK0GqfvEyN\nBjNaooXlkDWgYlwWTvDjovoDGrQscbNYLN57C9saD+veIR8GdwYDsMnvmfzAuU8L\nhij+0rnq49qlw0dpEuDb8PYZi+17cNcC1u2HGCgsBCRMd+RIihrGO5rUD8r6ddIB\nQFqNeb+Lz0vPqhbBleStTIo+F5HUsWLlguWABKQDfo2/2n+iD5dPDNMN+9fR5XJ+\nHMh3/1uaD7euBUbl8agW7EekFwIDAQABo4H2MIHzMIGwBgNVHREEgagwgaWBD2lu\nZm9AaXplbnBlLmNvbaSBkTCBjjFHMEUGA1UECgw+SVpFTlBFIFMuQS4gLSBDSUYg\nQTAxMzM3MjYwLVJNZXJjLlZpdG9yaWEtR2FzdGVpeiBUMTA1NSBGNjIgUzgxQzBB\nBgNVBAkMOkF2ZGEgZGVsIE1lZGl0ZXJyYW5lbyBFdG9yYmlkZWEgMTQgLSAwMTAx\nMCBWaXRvcmlhLUdhc3RlaXowDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC\nAQYwHQYDVR0OBBYEFB0cZQ6o8iV7tJHP5LGx5r1VdGwFMA0GCSqGSIb3DQEBCwUA\nA4ICAQB4pgwWSp9MiDrAyw6lFn2fuUhfGI8NYjb2zRlrrKvV9pF9rnHzP7MOeIWb\nlaQnIUdCSnxIOvVFfLMMjlF4rJUT3sb9fbgakEyrkgPH7UIBzg/YsfqikuFgba56\nawmqxinuaElnMIAkejEWOVt+8Rwu3WwJrfIxwYJOubv5vr8qhT/AQKM6WfxZSzwo\nJNu0FXWuDYi6LnPAvViH5ULy617uHjAimcs30cQhbIHsvm0m5hzkQiCeR7Csg1lw\nLDXWrzY0tM07+DKo7+N4ifuNRSzanLh+QBxh5z6ikixL8s36mLYp//Pye6kfLqCT\nVyvehQP5aTfLnnhqBbTFMXiJ7HqnheG5ezzevh55hM6fcA5ZwjUukCox2eRFekGk\nLhObNA5me0mrZJfQRsN5nXJQY6aYWwa9SG3YOYNw6DXwBdGqvOPbyALqfP2C2sJb\nUjWumDqtujWTI6cfSN01RpiyEGjkpTHCClguGYEQyVB1/OpaFs4R1+7vUIgtYf8/\nQnMFlEPVjjxOAToZpR9GTnfQXeWBIiGH/pR9hNiTrdZoQ0iy2+tzJOeRf1SktoA+\nnaM8THLCV8Sg1Mw4J87VBp6iSNnpn86CcDaTmjvfliHjWbcM2pE38P1ZWrOZyGls\nQyYBNWNgVYkDOnXYukrZVP/u3oDYLdE41V4tC5h9Pmzb/CaIxw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIE5jCCA86gAwIBAgIEO45L/DANBgkqhkiG9w0BAQUFADBdMRgwFgYJKoZIhvcN\nAQkBFglwa2lAc2suZWUxCzAJBgNVBAYTAkVFMSIwIAYDVQQKExlBUyBTZXJ0aWZp\ndHNlZXJpbWlza2Vza3VzMRAwDgYDVQQDEwdKdXVyLVNLMB4XDTAxMDgzMDE0MjMw\nMVoXDTE2MDgyNjE0MjMwMVowXTEYMBYGCSqGSIb3DQEJARYJcGtpQHNrLmVlMQsw\nCQYDVQQGEwJFRTEiMCAGA1UEChMZQVMgU2VydGlmaXRzZWVyaW1pc2tlc2t1czEQ\nMA4GA1UEAxMHSnV1ci1TSzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB\nAIFxNj4zB9bjMI0TfncyRsvPGbJgMUaXhvSYRqTCZUXP00B841oiqBB4M8yIsdOB\nSvZiF3tfTQou0M+LI+5PAk676w7KvRhj6IAcjeEcjT3g/1tf6mTll+g/mX8MCgkz\nABpTpyHhOEvWgxutr2TC+Rx6jGZITWYfGAriPrsfB2WThbkasLnE+w0R9vXW+RvH\nLCu3GFH+4Hv2qEivbDtPL+/40UceJlfwUR0zlv/vWT3aTdEVNMfqPxZIe5EcgEMP\nPbgFPtGzlc3Yyg/CQ2fbt5PgIoIuvvVoKIO5wTtpeyDaTpxt4brNj3pssAki14sL\n2xzVWiZbDcDq5WDQn/413z8CAwEAAaOCAawwggGoMA8GA1UdEwEB/wQFMAMBAf8w\nggEWBgNVHSAEggENMIIBCTCCAQUGCisGAQQBzh8BAQEwgfYwgdAGCCsGAQUFBwIC\nMIHDHoHAAFMAZQBlACAAcwBlAHIAdABpAGYAaQBrAGEAYQB0ACAAbwBuACAAdgDk\nAGwAagBhAHMAdABhAHQAdQBkACAAQQBTAC0AaQBzACAAUwBlAHIAdABpAGYAaQB0\nAHMAZQBlAHIAaQBtAGkAcwBrAGUAcwBrAHUAcwAgAGEAbABhAG0ALQBTAEsAIABz\nAGUAcgB0AGkAZgBpAGsAYQBhAHQAaQBkAGUAIABrAGkAbgBuAGkAdABhAG0AaQBz\nAGUAawBzMCEGCCsGAQUFBwIBFhVodHRwOi8vd3d3LnNrLmVlL2Nwcy8wKwYDVR0f\nBCQwIjAgoB6gHIYaaHR0cDovL3d3dy5zay5lZS9qdXVyL2NybC8wHQYDVR0OBBYE\nFASqekej5ImvGs8KQKcYP2/v6X2+MB8GA1UdIwQYMBaAFASqekej5ImvGs8KQKcY\nP2/v6X2+MA4GA1UdDwEB/wQEAwIB5jANBgkqhkiG9w0BAQUFAAOCAQEAe8EYlFOi\nCfP+JmeaUOTDBS8rNXiRTHyoERF5TElZrMj3hWVcRrs7EKACr81Ptcw2Kuxd/u+g\nkcm2k298gFTsxwhwDY77guwqYHhpNjbRxZyLabVAyJRld/JXIWY7zoVAtjNjGr95\nHvxcHdMdkxuLDF2FvZkwMhgJkVLpfKG6/2SSmuz+Ne6ML678IIbsSt4beDI3poHS\nna9aEhbKmVv8b20OxaAehsmR0FyYgl9jDIpaq9iVpszLita/ZEuOyoqysOkhMp6q\nqIWYNIE5ITuoOlIyPfZrN4YGWhWY3PARZv40ILcD9EEQfTmEeZZyY7aWAuVrua0Z\nTbvGRNs2yyqcjg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIHqDCCBpCgAwIBAgIRAMy4579OKRr9otxmpRwsDxEwDQYJKoZIhvcNAQEFBQAw\ncjELMAkGA1UEBhMCSFUxETAPBgNVBAcTCEJ1ZGFwZXN0MRYwFAYDVQQKEw1NaWNy\nb3NlYyBMdGQuMRQwEgYDVQQLEwtlLVN6aWdubyBDQTEiMCAGA1UEAxMZTWljcm9z\nZWMgZS1Temlnbm8gUm9vdCBDQTAeFw0wNTA0MDYxMjI4NDRaFw0xNzA0MDYxMjI4\nNDRaMHIxCzAJBgNVBAYTAkhVMREwDwYDVQQHEwhCdWRhcGVzdDEWMBQGA1UEChMN\nTWljcm9zZWMgTHRkLjEUMBIGA1UECxMLZS1Temlnbm8gQ0ExIjAgBgNVBAMTGU1p\nY3Jvc2VjIGUtU3ppZ25vIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\nggEKAoIBAQDtyADVgXvNOABHzNuEwSFpLHSQDCHZU4ftPkNEU6+r+ICbPHiN1I2u\nuO/TEdyB5s87lozWbxXGd36hL+BfkrYn13aaHUM86tnsL+4582pnS4uCzyL4ZVX+\nLMsvfUh6PXX5qqAnu3jCBspRwn5mS6/NoqdNAoI/gqyFxuEPkEeZlApxcpMqyabA\nvjxWTHOSJ/FrtfX9/DAFYJLG65Z+AZHCabEeHXtTRbjcQR/Ji3HWVBTji1R4P770\nYjtb9aPs1ZJ04nQw7wHb4dSrmZsqa/i9phyGI0Jf7Enemotb9HI6QMVJPqW+jqpx\n62z69Rrkav17fVVA71hu5tnVvCSrwe+3AgMBAAGjggQ3MIIEMzBnBggrBgEFBQcB\nAQRbMFkwKAYIKwYBBQUHMAGGHGh0dHBzOi8vcmNhLmUtc3ppZ25vLmh1L29jc3Aw\nLQYIKwYBBQUHMAKGIWh0dHA6Ly93d3cuZS1zemlnbm8uaHUvUm9vdENBLmNydDAP\nBgNVHRMBAf8EBTADAQH/MIIBcwYDVR0gBIIBajCCAWYwggFiBgwrBgEEAYGoGAIB\nAQEwggFQMCgGCCsGAQUFBwIBFhxodHRwOi8vd3d3LmUtc3ppZ25vLmh1L1NaU1ov\nMIIBIgYIKwYBBQUHAgIwggEUHoIBEABBACAAdABhAG4A+gBzAO0AdAB2AOEAbgB5\nACAA6QByAHQAZQBsAG0AZQB6AOkAcwDpAGgAZQB6ACAA6QBzACAAZQBsAGYAbwBn\nAGEAZADhAHMA4QBoAG8AegAgAGEAIABTAHoAbwBsAGcA4QBsAHQAYQB0APMAIABT\nAHoAbwBsAGcA4QBsAHQAYQB0AOEAcwBpACAAUwB6AGEAYgDhAGwAeQB6AGEAdABh\nACAAcwB6AGUAcgBpAG4AdAAgAGsAZQBsAGwAIABlAGwAagDhAHIAbgBpADoAIABo\nAHQAdABwADoALwAvAHcAdwB3AC4AZQAtAHMAegBpAGcAbgBvAC4AaAB1AC8AUwBa\nAFMAWgAvMIHIBgNVHR8EgcAwgb0wgbqggbeggbSGIWh0dHA6Ly93d3cuZS1zemln\nbm8uaHUvUm9vdENBLmNybIaBjmxkYXA6Ly9sZGFwLmUtc3ppZ25vLmh1L0NOPU1p\nY3Jvc2VjJTIwZS1Temlnbm8lMjBSb290JTIwQ0EsT1U9ZS1Temlnbm8lMjBDQSxP\nPU1pY3Jvc2VjJTIwTHRkLixMPUJ1ZGFwZXN0LEM9SFU/Y2VydGlmaWNhdGVSZXZv\nY2F0aW9uTGlzdDtiaW5hcnkwDgYDVR0PAQH/BAQDAgEGMIGWBgNVHREEgY4wgYuB\nEGluZm9AZS1zemlnbm8uaHWkdzB1MSMwIQYDVQQDDBpNaWNyb3NlYyBlLVN6aWdu\nw7MgUm9vdCBDQTEWMBQGA1UECwwNZS1TemlnbsOzIEhTWjEWMBQGA1UEChMNTWlj\ncm9zZWMgS2Z0LjERMA8GA1UEBxMIQnVkYXBlc3QxCzAJBgNVBAYTAkhVMIGsBgNV\nHSMEgaQwgaGAFMegSXUWYYTbMUuE0vE3QJDvTtz3oXakdDByMQswCQYDVQQGEwJI\nVTERMA8GA1UEBxMIQnVkYXBlc3QxFjAUBgNVBAoTDU1pY3Jvc2VjIEx0ZC4xFDAS\nBgNVBAsTC2UtU3ppZ25vIENBMSIwIAYDVQQDExlNaWNyb3NlYyBlLVN6aWdubyBS\nb290IENBghEAzLjnv04pGv2i3GalHCwPETAdBgNVHQ4EFgQUx6BJdRZhhNsxS4TS\n8TdAkO9O3PcwDQYJKoZIhvcNAQEFBQADggEBANMTnGZjWS7KXHAM/IO8VbH0jgds\nZifOwTsgqRy7RlRw7lrMoHfqaEQn6/Ip3Xep1fvj1KcExJW4C+FEaGAHQzAxQmHl\n7tnlJNUb3+FKG6qfx1/4ehHqE5MAyopYse7tDk2016g2JnzgOsHVV4Lxdbb9iV/a\n86g4nzUGCM4ilb7N1fy+W955a9x6qWVmvrElWl/tftOsRm1M9DKHtCAE4Gx4sHfR\nhUZLphK3dehKyVZs15KrnfVJONJPU+NVkBHbmJbGSfI+9J8b4PeI3CVimUTYc78/\nMPMMNz7UwiiAc7EBt51alhQBS6kRnSlqLtBdgcDPsiBDxwPgN05dCtxZICU=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECjCCAvKgAwIBAgIJAMJ+QwRORz8ZMA0GCSqGSIb3DQEBCwUAMIGCMQswCQYD\nVQQGEwJIVTERMA8GA1UEBwwIQnVkYXBlc3QxFjAUBgNVBAoMDU1pY3Jvc2VjIEx0\nZC4xJzAlBgNVBAMMHk1pY3Jvc2VjIGUtU3ppZ25vIFJvb3QgQ0EgMjAwOTEfMB0G\nCSqGSIb3DQEJARYQaW5mb0BlLXN6aWduby5odTAeFw0wOTA2MTYxMTMwMThaFw0y\nOTEyMzAxMTMwMThaMIGCMQswCQYDVQQGEwJIVTERMA8GA1UEBwwIQnVkYXBlc3Qx\nFjAUBgNVBAoMDU1pY3Jvc2VjIEx0ZC4xJzAlBgNVBAMMHk1pY3Jvc2VjIGUtU3pp\nZ25vIFJvb3QgQ0EgMjAwOTEfMB0GCSqGSIb3DQEJARYQaW5mb0BlLXN6aWduby5o\ndTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOn4j/NjrdqG2KfgQvvP\nkd6mJviZpWNwrZuuyjNAfW2WbqEORO7hE52UQlKavXWFdCyoDh2Tthi3jCyoz/tc\ncbna7P7ofo/kLx2yqHWH2Leh5TvPmUpG0IMZfcChEhyVbUr02MelTTMuhTlAdX4U\nfIASmFDHQWe4oIBhVKZsTh/gnQ4H6cm6M+f+wFUoLAKApxn1ntxVUwOXewdI/5n7\nN4okxFnMUBBjjqqpGrCEGob5X7uxUG6k0QrM1XF+H6cbfPVTbiJfyyvm1HxdrtbC\nxkzlBQHZ7Vf8wSN5/PrIJIOV87VqUQHQd9bpEqH5GoP7ghu5sJf0dgYzQ0mg/wu1\n+rUCAwEAAaOBgDB+MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G\nA1UdDgQWBBTLD8bfQkPMPcu1SCOhGnqmKrs0aDAfBgNVHSMEGDAWgBTLD8bfQkPM\nPcu1SCOhGnqmKrs0aDAbBgNVHREEFDASgRBpbmZvQGUtc3ppZ25vLmh1MA0GCSqG\nSIb3DQEBCwUAA4IBAQDJ0Q5eLtXMs3w+y/w9/w0olZMEyL/azXm4Q5DwpL7v8u8h\nmLzU1F0G9u5C7DBsoKqpyvGvivo/C3NqPuouQH4frlRheesuCDfXI/OMn74dseGk\nddug4lQUsbocKaQY9hK6ohQU4zE1yED/t+AFdlfBHFny+L/k7SViXITwfn4fs775\ntyERzAMBVnCnEJIeGzSBHq2cGsMEPO0CYdYeBvNfOofyK/FFh+U9rNHHV4S9a67c\n2Pm2G2JwCz02yULyMtd6YebS2z3PyKnJm9zbWETXbzivf3jTo60adbocwTZ8jx5t\nHMN1Rq41Bab2XD0h7lbwyYIiLXpUq3DDfSJlgnCW\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEFTCCAv2gAwIBAgIGSUEs5AAQMA0GCSqGSIb3DQEBCwUAMIGnMQswCQYDVQQG\nEwJIVTERMA8GA1UEBwwIQnVkYXBlc3QxFTATBgNVBAoMDE5ldExvY2sgS2Z0LjE3\nMDUGA1UECwwuVGFuw7pzw610dsOhbnlraWFkw7NrIChDZXJ0aWZpY2F0aW9uIFNl\ncnZpY2VzKTE1MDMGA1UEAwwsTmV0TG9jayBBcmFueSAoQ2xhc3MgR29sZCkgRsWR\ndGFuw7pzw610dsOhbnkwHhcNMDgxMjExMTUwODIxWhcNMjgxMjA2MTUwODIxWjCB\npzELMAkGA1UEBhMCSFUxETAPBgNVBAcMCEJ1ZGFwZXN0MRUwEwYDVQQKDAxOZXRM\nb2NrIEtmdC4xNzA1BgNVBAsMLlRhbsO6c8OtdHbDoW55a2lhZMOzayAoQ2VydGlm\naWNhdGlvbiBTZXJ2aWNlcykxNTAzBgNVBAMMLE5ldExvY2sgQXJhbnkgKENsYXNz\nIEdvbGQpIEbFkXRhbsO6c8OtdHbDoW55MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A\nMIIBCgKCAQEAxCRec75LbRTDofTjl5Bu0jBFHjzuZ9lk4BqKf8owyoPjIMHj9DrT\nlF8afFttvzBPhCf2nx9JvMaZCpDyD/V/Q4Q3Y1GLeqVw/HpYzY6b7cNGbIRwXdrz\nAZAj/E4wqX7hJ2Pn7WQ8oLjJM2P+FpD/sLj916jAwJRDC7bVWaaeVtAkH3B5r9s5\nVA1lddkVQZQBr17s9o3x/61k/iCa11zr/qYfCGSji3ZVrR47KGAuhyXoqq8fxmRG\nILdwfzzeSNuWU7c5d+Qa4scWhHaXWy+7GRWF+GmF9ZmnqfI0p6m2pgP8b4Y9VHx2\nBJtr+UBdADTHLpl1neWIA6pN+APSQnbAGwIDAKiLo0UwQzASBgNVHRMBAf8ECDAG\nAQH/AgEEMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUzPpnk/C2uNClwB7zU/2M\nU9+D15YwDQYJKoZIhvcNAQELBQADggEBAKt/7hwWqZw8UQCgwBEIBaeZ5m8BiFRh\nbvG5GK1Krf6BQCOUL/t1fC8oS2IkgYIL9WHxHG64YTjrgfpioTtaYtOUZcTh5m2C\n+C8lcLIhJsFyUR+MLMOEkMNaj7rP9KdlpeuY0fsFskZ1FSNqb4VjMIDw1Z4fKRzC\nbLBQWV2QWzuoDTDPv31/zvGdg73JRm4gpvlhUbohL3u+pRVjodSVh/GeufOJ8z2F\nuLjbvrW5KfnaNwUASZQDhETnv0Mxz3WLJdH0pmT1kvarBes96aULNmLazAZfNou2\nXjG4Kvte9nHfRCaexOYNkbQudZWAUWpLMKawYqGT8ZvYzsRjdT9ZR7E=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIGfTCCBWWgAwIBAgICAQMwDQYJKoZIhvcNAQEEBQAwga8xCzAJBgNVBAYTAkhV\nMRAwDgYDVQQIEwdIdW5nYXJ5MREwDwYDVQQHEwhCdWRhcGVzdDEnMCUGA1UEChMe\nTmV0TG9jayBIYWxvemF0Yml6dG9uc2FnaSBLZnQuMRowGAYDVQQLExFUYW51c2l0\ndmFueWtpYWRvazE2MDQGA1UEAxMtTmV0TG9jayBLb3pqZWd5em9pIChDbGFzcyBB\nKSBUYW51c2l0dmFueWtpYWRvMB4XDTk5MDIyNDIzMTQ0N1oXDTE5MDIxOTIzMTQ0\nN1owga8xCzAJBgNVBAYTAkhVMRAwDgYDVQQIEwdIdW5nYXJ5MREwDwYDVQQHEwhC\ndWRhcGVzdDEnMCUGA1UEChMeTmV0TG9jayBIYWxvemF0Yml6dG9uc2FnaSBLZnQu\nMRowGAYDVQQLExFUYW51c2l0dmFueWtpYWRvazE2MDQGA1UEAxMtTmV0TG9jayBL\nb3pqZWd5em9pIChDbGFzcyBBKSBUYW51c2l0dmFueWtpYWRvMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvHSMD7tM9DceqQWC2ObhbHDqeLVu0ThEDaiD\nzl3S1tWBxdRL51uUcCbbO51qTGL3cfNk1mE7PetzozfZz+qMkjvN9wfcZnSX9EUi\n3fRc4L9t875lM+QVOr/bmJBVOMTtplVjC7B4BPTjbsE/jvxReB+SnoPC/tmwqcm8\nWgD/qaiYdPv2LD4VOQ22BFWoDpggQrOxJa1+mm9dU7GrDPzr4PN6s6iz/0b2Y6LY\nOph7tqyF/7AlT3Rj5xMHpQqPBffAZG9+pyeAlt7ULoZgx2srXnN7F+eRP2QM2Esi\nNCubMvJIH5+hCoR64sKtlz2O1cH5VqNQ6ca0+pii7pXmKgOM3wIDAQABo4ICnzCC\nApswDgYDVR0PAQH/BAQDAgAGMBIGA1UdEwEB/wQIMAYBAf8CAQQwEQYJYIZIAYb4\nQgEBBAQDAgAHMIICYAYJYIZIAYb4QgENBIICURaCAk1GSUdZRUxFTSEgRXplbiB0\nYW51c2l0dmFueSBhIE5ldExvY2sgS2Z0LiBBbHRhbGFub3MgU3pvbGdhbHRhdGFz\naSBGZWx0ZXRlbGVpYmVuIGxlaXJ0IGVsamFyYXNvayBhbGFwamFuIGtlc3p1bHQu\nIEEgaGl0ZWxlc2l0ZXMgZm9seWFtYXRhdCBhIE5ldExvY2sgS2Z0LiB0ZXJtZWtm\nZWxlbG9zc2VnLWJpenRvc2l0YXNhIHZlZGkuIEEgZGlnaXRhbGlzIGFsYWlyYXMg\nZWxmb2dhZGFzYW5hayBmZWx0ZXRlbGUgYXogZWxvaXJ0IGVsbGVub3J6ZXNpIGVs\namFyYXMgbWVndGV0ZWxlLiBBeiBlbGphcmFzIGxlaXJhc2EgbWVndGFsYWxoYXRv\nIGEgTmV0TG9jayBLZnQuIEludGVybmV0IGhvbmxhcGphbiBhIGh0dHBzOi8vd3d3\nLm5ldGxvY2submV0L2RvY3MgY2ltZW4gdmFneSBrZXJoZXRvIGF6IGVsbGVub3J6\nZXNAbmV0bG9jay5uZXQgZS1tYWlsIGNpbWVuLiBJTVBPUlRBTlQhIFRoZSBpc3N1\nYW5jZSBhbmQgdGhlIHVzZSBvZiB0aGlzIGNlcnRpZmljYXRlIGlzIHN1YmplY3Qg\ndG8gdGhlIE5ldExvY2sgQ1BTIGF2YWlsYWJsZSBhdCBodHRwczovL3d3dy5uZXRs\nb2NrLm5ldC9kb2NzIG9yIGJ5IGUtbWFpbCBhdCBjcHNAbmV0bG9jay5uZXQuMA0G\nCSqGSIb3DQEBBAUAA4IBAQBIJEb3ulZv+sgoA0BO5TE5ayZrU3/b39/zcT0mwBQO\nxmd7I6gMc90Bu8bKbjc5VdXHjFYgDigKDtIqpLBJUsY4B/6+CgmM0ZjPytoUMaFP\n0jn8DxEsQ8Pdq5PHVT5HfBgaANzze9jyf1JsIPQLX2lS9O74silg6+NJMSEN1rUQ\nQeJBCWziGppWS3cC9qCbmieH6FUpccKQn0V4GuEVZD3QDtigdp+uxdAu6tYPVuxk\nf1qbFFgBJ34TUMdrKuZoPL9coAob4Q566eKAw+np9v1sEZ7Q5SgnK1QyQhSCdeZK\n8CtmdWOMovsEPoMOmzbwGOQmIMOM8CgHrTwXZoi1/baI\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID5jCCAs6gAwIBAgIQV8szb8JcFuZHFhfjkDFo4DANBgkqhkiG9w0BAQUFADBi\nMQswCQYDVQQGEwJVUzEhMB8GA1UEChMYTmV0d29yayBTb2x1dGlvbnMgTC5MLkMu\nMTAwLgYDVQQDEydOZXR3b3JrIFNvbHV0aW9ucyBDZXJ0aWZpY2F0ZSBBdXRob3Jp\ndHkwHhcNMDYxMjAxMDAwMDAwWhcNMjkxMjMxMjM1OTU5WjBiMQswCQYDVQQGEwJV\nUzEhMB8GA1UEChMYTmV0d29yayBTb2x1dGlvbnMgTC5MLkMuMTAwLgYDVQQDEydO\nZXR3b3JrIFNvbHV0aW9ucyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDkvH6SMG3G2I4rC7xGzuAnlt7e+foS0zwz\nc7MEL7xxjOWftiJgPl9dzgn/ggwbmlFQGiaJ3dVhXRncEg8tCqJDXRfQNJIg6nPP\nOCwGJgl6cvf6UDL4wpPTaaIjzkGxzOTVHzbRijr4jGPiFFlp7Q3Tf2vouAPlT2rl\nmGNpSAW+Lv8ztumXWWn4Zxmuk2GWRBXTcrA/vGp97Eh/jcOrqnErU2lBUzS1sLnF\nBgrEsEX1QV1uiUV7PTsmjHTC5dLRfbIR1PtYMiKagMnc/Qzpf14Dl847ABSHJ3A4\nqY5usyd2mFHgBeMhqxrVhSI8KbWaFsWAqPS7azCPL0YCorEMIuDTAgMBAAGjgZcw\ngZQwHQYDVR0OBBYEFCEwyfsA106Y2oeqKtCnLrFAMadMMA4GA1UdDwEB/wQEAwIB\nBjAPBgNVHRMBAf8EBTADAQH/MFIGA1UdHwRLMEkwR6BFoEOGQWh0dHA6Ly9jcmwu\nbmV0c29sc3NsLmNvbS9OZXR3b3JrU29sdXRpb25zQ2VydGlmaWNhdGVBdXRob3Jp\ndHkuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQC7rkvnt1frf6ott3NHhWrB5KUd5Oc8\n6fRZZXe1eltajSU24HqXLjjAV2CDmAaDn7l2em5Q4LqILPxFzBiwmZVRDuwduIj/\nh1AcgsLj4DKAv6ALR8jDMe+ZZzKATxcheQxpXN5eNK4CtSbqUN9/GGUsyfJj4akH\n/nxxH2szJGoeBfcFaMBqEssuXmHLrijTfsK0ZpEmXzwuJF/LWA/rKOyvEZbz3Htv\nwKeI8lN3s2Berq4o2jUsbzRF0ybh3uxbTydrFny9RAQYgrOJeRcQcT16ohZO9QHN\npGxlaKFJdlxDydi8NmdspZS11My5vWo1ViHe2MPr+8ukYEywVaCge1ey\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID8TCCAtmgAwIBAgIQQT1yx/RrH4FDffHSKFTfmjANBgkqhkiG9w0BAQUFADCB\nijELMAkGA1UEBhMCQ0gxEDAOBgNVBAoTB1dJU2VLZXkxGzAZBgNVBAsTEkNvcHly\naWdodCAoYykgMjAwNTEiMCAGA1UECxMZT0lTVEUgRm91bmRhdGlvbiBFbmRvcnNl\nZDEoMCYGA1UEAxMfT0lTVEUgV0lTZUtleSBHbG9iYWwgUm9vdCBHQSBDQTAeFw0w\nNTEyMTExNjAzNDRaFw0zNzEyMTExNjA5NTFaMIGKMQswCQYDVQQGEwJDSDEQMA4G\nA1UEChMHV0lTZUtleTEbMBkGA1UECxMSQ29weXJpZ2h0IChjKSAyMDA1MSIwIAYD\nVQQLExlPSVNURSBGb3VuZGF0aW9uIEVuZG9yc2VkMSgwJgYDVQQDEx9PSVNURSBX\nSVNlS2V5IEdsb2JhbCBSb290IEdBIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A\nMIIBCgKCAQEAy0+zAJs9Nt350UlqaxBJH+zYK7LG+DKBKUOVTJoZIyEVRd7jyBxR\nVVuuk+g3/ytr6dTqvirdqFEr12bDYVxgAsj1znJ7O7jyTmUIms2kahnBAbtzptf2\nw93NvKSLtZlhuAGio9RN1AU9ka34tAhxZK9w8RxrfvbDd50kc3vkDIzh2TbhmYsF\nmQvtRTEJysIA2/dyoJaqlYfQjse2YXMNdmaM3Bu0Y6Kff5MTMPGhJ9vZ/yxViJGg\n4E8HsChWjBgbl0SOid3gF27nKu+POQoxhILYQBRJLnpB5Kf+42TMwVlxSywhp1t9\n4B3RLoGbw9ho972WG6xwsRYUC9tguSYBBQIDAQABo1EwTzALBgNVHQ8EBAMCAYYw\nDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUswN+rja8sHnR3JQmthG+IbJphpQw\nEAYJKwYBBAGCNxUBBAMCAQAwDQYJKoZIhvcNAQEFBQADggEBAEuh/wuHbrP5wUOx\nSPMowB0uyQlB+pQAHKSkq0lPjz0e701vvbyk9vImMMkQyh2I+3QZH4VFvbBsUfk2\nftv1TDI6QU9bR8/oCy22xBmddMVHxjtqD6wU2zz0c5ypBd8A3HR4+vg1YFkCExh8\nvPtNsCBtQ7tgMHpnM1zFmdH4LTlSc/uMqpclXHLZCB6rTjzjgTGfA6b7wP4piFXa\nhNVQA7bihKOmNqoROgHhGEvWRGizPflTdISzRpFGlgC3gCy24eMQ4tui5yiPAZZi\nFj4A4xylNoEYokxSdsARo27mHbrjWr42U8U+dY+GaSlYU7Wcu2+fXMUY7N0v4ZjJ\n/L7fCg0=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIJhjCCB26gAwIBAgIBCzANBgkqhkiG9w0BAQsFADCCAR4xPjA8BgNVBAMTNUF1\ndG9yaWRhZCBkZSBDZXJ0aWZpY2FjaW9uIFJhaXogZGVsIEVzdGFkbyBWZW5lem9s\nYW5vMQswCQYDVQQGEwJWRTEQMA4GA1UEBxMHQ2FyYWNhczEZMBcGA1UECBMQRGlz\ndHJpdG8gQ2FwaXRhbDE2MDQGA1UEChMtU2lzdGVtYSBOYWNpb25hbCBkZSBDZXJ0\naWZpY2FjaW9uIEVsZWN0cm9uaWNhMUMwQQYDVQQLEzpTdXBlcmludGVuZGVuY2lh\nIGRlIFNlcnZpY2lvcyBkZSBDZXJ0aWZpY2FjaW9uIEVsZWN0cm9uaWNhMSUwIwYJ\nKoZIhvcNAQkBFhZhY3JhaXpAc3VzY2VydGUuZ29iLnZlMB4XDTEwMTIyODE2NTEw\nMFoXDTIwMTIyNTIzNTk1OVowgdExJjAkBgkqhkiG9w0BCQEWF2NvbnRhY3RvQHBy\nb2NlcnQubmV0LnZlMQ8wDQYDVQQHEwZDaGFjYW8xEDAOBgNVBAgTB01pcmFuZGEx\nKjAoBgNVBAsTIVByb3ZlZWRvciBkZSBDZXJ0aWZpY2Fkb3MgUFJPQ0VSVDE2MDQG\nA1UEChMtU2lzdGVtYSBOYWNpb25hbCBkZSBDZXJ0aWZpY2FjaW9uIEVsZWN0cm9u\naWNhMQswCQYDVQQGEwJWRTETMBEGA1UEAxMKUFNDUHJvY2VydDCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBANW39KOUM6FGqVVhSQ2oh3NekS1wwQYalNo9\n7BVCwfWMrmoX8Yqt/ICV6oNEolt6Vc5Pp6XVurgfoCfAUFM+jbnADrgV3NZs+J74\nBCXfgI8Qhd19L3uA3VcAZCP4bsm+lU/hdezgfl6VzbHvvnpC2Mks0+saGiKLt38G\nieU89RLAu9MLmV+QfI4tL3czkkohRqipCKzx9hEC2ZUWno0vluYC3XXCFCpa1sl9\nJcLB/KpnheLsvtF8PPqv1W7/U0HU9TI4seJfxPmOEO8GqQKJ/+MMbpfg353bIdD0\nPghpbNjU5Db4g7ayNo+c7zo3Fn2/omnXO1ty0K+qP1xmk6wKImG20qCZyFSTXai2\n0b1dCl53lKItwIKOvMoDKjSuc/HUtQy9vmebVOvh+qBa7Dh+PsHMosdEMXXqP+UH\n0quhJZb25uSgXTcYOWEAM11G1ADEtMo88aKjPvM6/2kwLkDd9p+cJsmWN63nOaK/\n6mnbVSKVUyqUtd+tFjiBdWbjxywbk5yqjKPK2Ww8F22c3HxT4CAnQzb5EuE8XL1m\nv6JpIzi4mWCZDlZTOpx+FIywBm/xhnaQr/2v/pDGj59/i5IjnOcVdo/Vi5QTcmn7\nK2FjiO/mpF7moxdqWEfLcU8UC17IAggmosvpr2uKGcfLFFb14dq12fy/czja+eev\nbqQ34gcnAgMBAAGjggMXMIIDEzASBgNVHRMBAf8ECDAGAQH/AgEBMDcGA1UdEgQw\nMC6CD3N1c2NlcnRlLmdvYi52ZaAbBgVghl4CAqASDBBSSUYtRy0yMDAwNDAzNi0w\nMB0GA1UdDgQWBBRBDxk4qpl/Qguk1yeYVKIXTC1RVDCCAVAGA1UdIwSCAUcwggFD\ngBStuyIdxuDSAaj9dlBSk+2YwU2u06GCASakggEiMIIBHjE+MDwGA1UEAxM1QXV0\nb3JpZGFkIGRlIENlcnRpZmljYWNpb24gUmFpeiBkZWwgRXN0YWRvIFZlbmV6b2xh\nbm8xCzAJBgNVBAYTAlZFMRAwDgYDVQQHEwdDYXJhY2FzMRkwFwYDVQQIExBEaXN0\ncml0byBDYXBpdGFsMTYwNAYDVQQKEy1TaXN0ZW1hIE5hY2lvbmFsIGRlIENlcnRp\nZmljYWNpb24gRWxlY3Ryb25pY2ExQzBBBgNVBAsTOlN1cGVyaW50ZW5kZW5jaWEg\nZGUgU2VydmljaW9zIGRlIENlcnRpZmljYWNpb24gRWxlY3Ryb25pY2ExJTAjBgkq\nhkiG9w0BCQEWFmFjcmFpekBzdXNjZXJ0ZS5nb2IudmWCAQowDgYDVR0PAQH/BAQD\nAgEGME0GA1UdEQRGMESCDnByb2NlcnQubmV0LnZloBUGBWCGXgIBoAwMClBTQy0w\nMDAwMDKgGwYFYIZeAgKgEgwQUklGLUotMzE2MzUzNzMtNzB2BgNVHR8EbzBtMEag\nRKBChkBodHRwOi8vd3d3LnN1c2NlcnRlLmdvYi52ZS9sY3IvQ0VSVElGSUNBRE8t\nUkFJWi1TSEEzODRDUkxERVIuY3JsMCOgIaAfhh1sZGFwOi8vYWNyYWl6LnN1c2Nl\ncnRlLmdvYi52ZTA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9v\nY3NwLnN1c2NlcnRlLmdvYi52ZTBBBgNVHSAEOjA4MDYGBmCGXgMBAjAsMCoGCCsG\nAQUFBwIBFh5odHRwOi8vd3d3LnN1c2NlcnRlLmdvYi52ZS9kcGMwDQYJKoZIhvcN\nAQELBQADggIBACtZ6yKZu4SqT96QxtGGcSOeSwORR3C7wJJg7ODU523G0+1ng3dS\n1fLld6c2suNUvtm7CpsR72H0xpkzmfWvADmNg7+mvTV+LFwxNG9s2/NkAZiqlCxB\n3RWGymspThbASfzXg0gTB1GEMVKIu4YXx2sviiCtxQuPcD4quxtxj7mkoP3Yldmv\nWb8lK5jpY5MvYB7Eqvh39YtsL+1+LrVPQA3uvFd359m21D+VJzog1eWuq2w1n8Gh\nHVnchIHuTQfiSLaeS5UtQbHh6N5+LwUeaO6/u5BlOsju6rEYNxxik6SgMexxbJHm\npHmJWhSnFFAFTKQAVzAswbVhltw+HoSvOULP5dAssSS830DD7X9jSr3hTxJkhpXz\nsOfIt+FTvZLm8wyWuevo5pLtp4EJFAv8lXrPj9Y0TzYS3F7RNHXGRoAvlQSMx4bE\nqCaJqD8Zm4G7UaRKhqsLEQ+xrmNTbSjq3TNWOByyrYDT13K9mmyZY+gAu0F2Bbdb\nmRiKw7gSXFbPVgx96OLP7bx0R/vu0xdOIk9W/1DzLuY5poLWccret9W6aAjtmcz9\nopLLabid+Qqkpj5PkygqYWwHJgD/ll9ohri4zspV4KuxPX+Y1zMOWj3YeMLEYC/H\nYvBhkdI4sPaeVdtAgAUSM84dkpvRabP/v/GSCmE1P93+hvS84Bpxs2Km\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF0DCCBLigAwIBAgIEOrZQizANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJC\nTTEZMBcGA1UEChMQUXVvVmFkaXMgTGltaXRlZDElMCMGA1UECxMcUm9vdCBDZXJ0\naWZpY2F0aW9uIEF1dGhvcml0eTEuMCwGA1UEAxMlUXVvVmFkaXMgUm9vdCBDZXJ0\naWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wMTAzMTkxODMzMzNaFw0yMTAzMTcxODMz\nMzNaMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMSUw\nIwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYDVQQDEyVR\ndW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2G1lVO6V/z68mcLOhrfEYBklbTRvM16z/Yp\nli4kVEAkOPcahdxYTMukJ0KX0J+DisPkBgNbAKVRHnAEdOLB1Dqr1607BxgFjv2D\nrOpm2RgbaIr1VxqYuvXtdj182d6UajtLF8HVj71lODqV0D1VNk7feVcxKh7YWWVJ\nWCCYfqtffp/p1k3sg3Spx2zY7ilKhSoGFPlU5tPaZQeLYzcS19Dsw3sgQUSj7cug\nF+FxZc4dZjH3dgEZyH0DWLaVSR2mEiboxgx24ONmy+pdpibu5cxfvWenAScOospU\nxbF6lR1xHkopigPcakXBpBlebzbNw6Kwt/5cOOJSvPhEQ+aQuwIDAQABo4ICUjCC\nAk4wPQYIKwYBBQUHAQEEMTAvMC0GCCsGAQUFBzABhiFodHRwczovL29jc3AucXVv\ndmFkaXNvZmZzaG9yZS5jb20wDwYDVR0TAQH/BAUwAwEB/zCCARoGA1UdIASCAREw\nggENMIIBCQYJKwYBBAG+WAABMIH7MIHUBggrBgEFBQcCAjCBxxqBxFJlbGlhbmNl\nIG9uIHRoZSBRdW9WYWRpcyBSb290IENlcnRpZmljYXRlIGJ5IGFueSBwYXJ0eSBh\nc3N1bWVzIGFjY2VwdGFuY2Ugb2YgdGhlIHRoZW4gYXBwbGljYWJsZSBzdGFuZGFy\nZCB0ZXJtcyBhbmQgY29uZGl0aW9ucyBvZiB1c2UsIGNlcnRpZmljYXRpb24gcHJh\nY3RpY2VzLCBhbmQgdGhlIFF1b1ZhZGlzIENlcnRpZmljYXRlIFBvbGljeS4wIgYI\nKwYBBQUHAgEWFmh0dHA6Ly93d3cucXVvdmFkaXMuYm0wHQYDVR0OBBYEFItLbe3T\nKbkGGew5Oanwl4Rqy+/fMIGuBgNVHSMEgaYwgaOAFItLbe3TKbkGGew5Oanwl4Rq\ny+/foYGEpIGBMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1p\ndGVkMSUwIwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYD\nVQQDEyVRdW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggQ6tlCL\nMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOCAQEAitQUtf70mpKnGdSk\nfnIYj9lofFIk3WdvOXrEql494liwTXCYhGHoG+NpGA7O+0dQoE7/8CQfvbLO9Sf8\n7C9TqnN7Az10buYWnuulLsS/VidQK2K6vkscPFVcQR0kvoIgR13VRH56FmjffU1R\ncHhXHTMe/QKZnAzNCgVPx7uOpHX6Sm2xgI4JVrmcGmD+XcHXetwReNDWXcG31a0y\nmQM6isxUJTkxgXsTIlG6Rmyhu576BGxJJnSP0nPrzDCi5upZIof4l/UO/erMkqQW\nxFIY6iHOsfHmhIHluqmGKPJDWl0Snawe2ajlCmqnf6CHKc/yiU3U7MXi5nrQNiOK\nSnQ2+Q==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFYDCCA0igAwIBAgIUeFhfLq0sGUvjNwc1NBMotZbUZZMwDQYJKoZIhvcNAQEL\nBQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc\nBgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMSBHMzAeFw0xMjAxMTIxNzI3NDRaFw00\nMjAxMTIxNzI3NDRaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM\naW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDEgRzMwggIiMA0GCSqG\nSIb3DQEBAQUAA4ICDwAwggIKAoICAQCgvlAQjunybEC0BJyFuTHK3C3kEakEPBtV\nwedYMB0ktMPvhd6MLOHBPd+C5k+tR4ds7FtJwUrVu4/sh6x/gpqG7D0DmVIB0jWe\nrNrwU8lmPNSsAgHaJNM7qAJGr6Qc4/hzWHa39g6QDbXwz8z6+cZM5cOGMAqNF341\n68Xfuw6cwI2H44g4hWf6Pser4BOcBRiYz5P1sZK0/CPTz9XEJ0ngnjybCKOLXSoh\n4Pw5qlPafX7PGglTvF0FBM+hSo+LdoINofjSxxR3W5A2B4GbPgb6Ul5jxaYA/qXp\nUhtStZI5cgMJYr2wYBZupt0lwgNm3fME0UDiTouG9G/lg6AnhF4EwfWQvTA9xO+o\nabw4m6SkltFi2mnAAZauy8RRNOoMqv8hjlmPSlzkYZqn0ukqeI1RPToV7qJZjqlc\n3sX5kCLliEVx3ZGZbHqfPT2YfF72vhZooF6uCyP8Wg+qInYtyaEQHeTTRCOQiJ/G\nKubX9ZqzWB4vMIkIG1SitZgj7Ah3HJVdYdHLiZxfokqRmu8hqkkWCKi9YSgxyXSt\nhfbZxbGL0eUQMk1fiyA6PEkfM4VZDdvLCXVDaXP7a3F98N/ETH3Goy7IlXnLc6KO\nTk0k+17kBL5yG6YnLUlamXrXXAkgt3+UuU/xDRxeiEIbEbfnkduebPRq34wGmAOt\nzCjvpUfzUwIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nBjAdBgNVHQ4EFgQUo5fW816iEOGrRZ88F2Q87gFwnMwwDQYJKoZIhvcNAQELBQAD\nggIBABj6W3X8PnrHX3fHyt/PX8MSxEBd1DKquGrX1RUVRpgjpeaQWxiZTOOtQqOC\nMTaIzen7xASWSIsBx40Bz1szBpZGZnQdT+3Btrm0DWHMY37XLneMlhwqI2hrhVd2\ncDMT/uFPpiN3GPoajOi9ZcnPP/TJF9zrx7zABC4tRi9pZsMbj/7sPtPKlL92CiUN\nqXsCHKnQO18LwIE6PWThv6ctTr1NxNgpxiIY0MWscgKCP6o6ojoilzHdCGPDdRS5\nYCgtW2jgFqlmgiNR9etT2DGbe+m3nUvriBbP+V04ikkwj+3x6xn0dxoxGE1nVGwv\nb2X52z3sIexe9PSLymBlVNFxZPT5pqOBMzYzcfCkeF9OrYMh3jRJjehZrJ3ydlo2\n8hP0r+AJx2EqbPfgna67hkooby7utHnNkDPDs3b69fBsnQGQ+p6Q9pxyz0fawx/k\nNSBT8lTR32GDpgLiJTjehTItXnOQUl1CxM49S+H5GYQd1aJQzEH7QRTDvdbJWqNj\nZgKAvQU6O0ec7AAmTPWIUb+oI38YB7AL7YsmoWTTYUrrXJ/es69nA7Mf3W1daWhp\nq1467HxpvMc7hU6eFbm0FU/DlXpY18ls6Wy58yljXrQs8C097Vpl4KlbQMJImYFt\nnh8GKjwStIsPm6Ik8KaN1nrgS7ZklmOVhMJKzRwuJIczYOXD\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFtzCCA5+gAwIBAgICBQkwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x\nGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv\nb3QgQ0EgMjAeFw0wNjExMjQxODI3MDBaFw0zMTExMjQxODIzMzNaMEUxCzAJBgNV\nBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W\nYWRpcyBSb290IENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCa\nGMpLlA0ALa8DKYrwD4HIrkwZhR0In6spRIXzL4GtMh6QRr+jhiYaHv5+HBg6XJxg\nFyo6dIMzMH1hVBHL7avg5tKifvVrbxi3Cgst/ek+7wrGsxDp3MJGF/hd/aTa/55J\nWpzmM+Yklvc/ulsrHHo1wtZn/qtmUIttKGAr79dgw8eTvI02kfN/+NsRE8Scd3bB\nrrcCaoF6qUWD4gXmuVbBlDePSHFjIuwXZQeVikvfj8ZaCuWw419eaxGrDPmF60Tp\n+ARz8un+XJiM9XOva7R+zdRcAitMOeGylZUtQofX1bOQQ7dsE/He3fbE+Ik/0XX1\nksOR1YqI0JDs3G3eicJlcZaLDQP9nL9bFqyS2+r+eXyt66/3FsvbzSUr5R/7mp/i\nUcw6UwxI5g69ybR2BlLmEROFcmMDBOAENisgGQLodKcftslWZvB1JdxnwQ5hYIiz\nPtGo/KPaHbDRsSNU30R2be1B2MGyIrZTHN81Hdyhdyox5C315eXbyOD/5YDXC2Og\n/zOhD7osFRXql7PSorW+8oyWHhqPHWykYTe5hnMz15eWniN9gqRMgeKh0bpnX5UH\noycR7hYQe7xFSkyyBNKr79X9DFHOUGoIMfmR2gyPZFwDwzqLID9ujWc9Otb+fVuI\nyV77zGHcizN300QyNQliBJIWENieJ0f7OyHj+OsdWwIDAQABo4GwMIGtMA8GA1Ud\nEwEB/wQFMAMBAf8wCwYDVR0PBAQDAgEGMB0GA1UdDgQWBBQahGK8SEwzJQTU7tD2\nA8QZRtGUazBuBgNVHSMEZzBlgBQahGK8SEwzJQTU7tD2A8QZRtGUa6FJpEcwRTEL\nMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMT\nElF1b1ZhZGlzIFJvb3QgQ0EgMoICBQkwDQYJKoZIhvcNAQEFBQADggIBAD4KFk2f\nBluornFdLwUvZ+YTRYPENvbzwCYMDbVHZF34tHLJRqUDGCdViXh9duqWNIAXINzn\ng/iN/Ae42l9NLmeyhP3ZRPx3UIHmfLTJDQtyU/h2BwdBR5YM++CCJpNVjP4iH2Bl\nfF/nJrP3MpCYUNQ3cVX2kiF495V5+vgtJodmVjB3pjd4M1IQWK4/YY7yarHvGH5K\nWWPKjaJW1acvvFYfzznB4vsKqBUsfU16Y8Zsl0Q80m/DShcK+JDSV6IZUaUtl0Ha\nB0+pUNqQjZRG4T7wlP0QADj1O+hA4bRuVhogzG9Yje0uRY/W6ZM/57Es3zrWIozc\nhLsib9D45MY56QSIPMO661V6bYCZJPVsAfv4l7CUW+v90m/xd2gNNWQjrLhVoQPR\nTUIZ3Ph1WVaj+ahJefivDrkRoHy3au000LYmYjgahwz46P0u05B/B5EqHdZ+XIWD\nmbA4CD/pXvk1B+TJYm5Xf6dQlfe6yJvmjqIBxdZmv3lh8zwc4bmCXF2gw+nYSL0Z\nohEUGW6yhhtoPkg3Goi3XZZenMfvJ2II4pEZXNLxId26F0KCl3GBUzGpn/Z9Yr9y\n4aOTHcyKJloJONDO1w2AFrR4pTqHTI2KpdVGl/IsELm8VCLAAVBpQ570su9t+Oza\n8eOx79+Rj1QqCyXBJhnEUhAFZdWCEOrCMc0u\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFYDCCA0igAwIBAgIURFc0JFuBiZs18s64KztbpybwdSgwDQYJKoZIhvcNAQEL\nBQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc\nBgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMiBHMzAeFw0xMjAxMTIxODU5MzJaFw00\nMjAxMTIxODU5MzJaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM\naW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDIgRzMwggIiMA0GCSqG\nSIb3DQEBAQUAA4ICDwAwggIKAoICAQChriWyARjcV4g/Ruv5r+LrI3HimtFhZiFf\nqq8nUeVuGxbULX1QsFN3vXg6YOJkApt8hpvWGo6t/x8Vf9WVHhLL5hSEBMHfNrMW\nn4rjyduYNM7YMxcoRvynyfDStNVNCXJJ+fKH46nafaF9a7I6JaltUkSs+L5u+9ym\nc5GQYaYDFCDy54ejiK2toIz/pgslUiXnFgHVy7g1gQyjO/Dh4fxaXc6AcW34Sas+\nO7q414AB+6XrW7PFXmAqMaCvN+ggOp+oMiwMzAkd056OXbxMmO7FGmh77FOm6RQ1\no9/NgJ8MSPsc9PG/Srj61YxxSscfrf5BmrODXfKEVu+lV0POKa2Mq1W/xPtbAd0j\nIaFYAI7D0GoT7RPjEiuA3GfmlbLNHiJuKvhB1PLKFAeNilUSxmn1uIZoL1NesNKq\nIcGY5jDjZ1XHm26sGahVpkUG0CM62+tlXSoREfA7T8pt9DTEceT/AFr2XK4jYIVz\n8eQQsSWu1ZK7E8EM4DnatDlXtas1qnIhO4M15zHfeiFuuDIIfR0ykRVKYnLP43eh\nvNURG3YBZwjgQQvD6xVu+KQZ2aKrr+InUlYrAoosFCT5v0ICvybIxo/gbjh9Uy3l\n7ZizlWNof/k19N+IxWA1ksB8aRxhlRbQ694Lrz4EEEVlWFA4r0jyWbYW8jwNkALG\ncC4BrTwV1wIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nBjAdBgNVHQ4EFgQU7edvdlq/YOxJW8ald7tyFnGbxD0wDQYJKoZIhvcNAQELBQAD\nggIBAJHfgD9DCX5xwvfrs4iP4VGyvD11+ShdyLyZm3tdquXK4Qr36LLTn91nMX66\nAarHakE7kNQIXLJgapDwyM4DYvmL7ftuKtwGTTwpD4kWilhMSA/ohGHqPHKmd+RC\nroijQ1h5fq7KpVMNqT1wvSAZYaRsOPxDMuHBR//47PERIjKWnML2W2mWeyAMQ0Ga\nW/ZZGYjeVYg3UQt4XAoeo0L9x52ID8DyeAIkVJOviYeIyUqAHerQbj5hLja7NQ4n\nlv1mNDthcnPxFlxHBlRJAHpYErAK74X9sbgzdWqTHBLmYF5vHX/JHyPLhGGfHoJE\n+V+tYlUkmlKY7VHnoX6XOuYvHxHaU4AshZ6rNRDbIl9qxV6XU/IyAgkwo1jwDQHV\ncsaxfGl7w/U2Rcxhbl5MlMVerugOXou/983g7aEOGzPuVBj+D77vfoRrQ+NwmNtd\ndbINWQeFFSM51vHfqSYP1kjHs6Yi9TM3WpVHn3u6GBVv/9YUZINJ0gpnIdsPNWNg\nKCLjsZWDzYWm3S8P52dSbrsvhXz1SnPnxT7AvSESBT/8twNJAlvIJebiVDj1eYeM\nHVOyToV7BjjHLPj4sHKNJeV3UvQDHEimUF+IIDBu8oJDqz2XhOdT+yHBTw8imoa4\nWSr2Rz0ZiC3oheGe7IUIarFsNMkd7EgrO3jtZsSOeWmD3n+M\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIGnTCCBIWgAwIBAgICBcYwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x\nGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv\nb3QgQ0EgMzAeFw0wNjExMjQxOTExMjNaFw0zMTExMjQxOTA2NDRaMEUxCzAJBgNV\nBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W\nYWRpcyBSb290IENBIDMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDM\nV0IWVJzmmNPTTe7+7cefQzlKZbPoFog02w1ZkXTPkrgEQK0CSzGrvI2RaNggDhoB\n4hp7Thdd4oq3P5kazethq8Jlph+3t723j/z9cI8LoGe+AaJZz3HmDyl2/7FWeUUr\nH556VOijKTVopAFPD6QuN+8bv+OPEKhyq1hX51SGyMnzW9os2l2ObjyjPtr7guXd\n8lyyBTNvijbO0BNO/79KDDRMpsMhvVAEVeuxu537RR5kFd5VAYwCdrXLoT9Cabwv\nvWhDFlaJKjdhkf2mrk7AyxRllDdLkgbvBNDInIjbC3uBr7E9KsRlOni27tyAsdLT\nmZw67mtaa7ONt9XOnMK+pUsvFrGeaDsGb659n/je7Mwpp5ijJUMv7/FfJuGITfhe\nbtfZFG4ZM2mnO4SJk8RTVROhUXhA+LjJou57ulJCg54U7QVSWllWp5f8nT8KKdjc\nT5EOE7zelaTfi5m+rJsziO+1ga8bxiJTyPbH7pcUsMV8eFLI8M5ud2CEpukqdiDt\nWAEXMJPpGovgc2PZapKUSU60rUqFxKMiMPwJ7Wgic6aIDFUhWMXhOp8q3crhkODZ\nc6tsgLjoC2SToJyMGf+z0gzskSaHirOi4XCPLArlzW1oUevaPwV/izLmE1xr/l9A\n4iLItLRkT9a6fUg+qGkM17uGcclzuD87nSVL2v9A6wIDAQABo4IBlTCCAZEwDwYD\nVR0TAQH/BAUwAwEB/zCB4QYDVR0gBIHZMIHWMIHTBgkrBgEEAb5YAAMwgcUwgZMG\nCCsGAQUFBwICMIGGGoGDQW55IHVzZSBvZiB0aGlzIENlcnRpZmljYXRlIGNvbnN0\naXR1dGVzIGFjY2VwdGFuY2Ugb2YgdGhlIFF1b1ZhZGlzIFJvb3QgQ0EgMyBDZXJ0\naWZpY2F0ZSBQb2xpY3kgLyBDZXJ0aWZpY2F0aW9uIFByYWN0aWNlIFN0YXRlbWVu\ndC4wLQYIKwYBBQUHAgEWIWh0dHA6Ly93d3cucXVvdmFkaXNnbG9iYWwuY29tL2Nw\nczALBgNVHQ8EBAMCAQYwHQYDVR0OBBYEFPLAE+CCQz777i9nMpY1XNu4ywLQMG4G\nA1UdIwRnMGWAFPLAE+CCQz777i9nMpY1XNu4ywLQoUmkRzBFMQswCQYDVQQGEwJC\nTTEZMBcGA1UEChMQUXVvVmFkaXMgTGltaXRlZDEbMBkGA1UEAxMSUXVvVmFkaXMg\nUm9vdCBDQSAzggIFxjANBgkqhkiG9w0BAQUFAAOCAgEAT62gLEz6wPJv92ZVqyM0\n7ucp2sNbtrCD2dDQ4iH782CnO11gUyeim/YIIirnv6By5ZwkajGxkHon24QRiSem\nd1o417+shvzuXYO8BsbRd2sPbSQvS3pspweWyuOEn62Iix2rFo1bZhfZFvSLgNLd\n+LJ2w/w4E6oM3kJpK27zPOuAJ9v1pkQNn1pVWQvVDVJIxa6f8i+AxeoyUDUSly7B\n4f/xI4hROJ/yZlZ25w9Rl6VSDE1JUZU2Pb+iSwwQHYaZTKrzchGT5Or2m9qoXadN\nt54CrnMAyNojA+j56hl0YgCUyyIgvpSnWbWCar6ZeXqp8kokUvd0/bpO5qgdAm6x\nDYBEwa7TIzdfu4V8K5Iu6H6li92Z4b8nby1dqnuH/grdS/yO9SbkbnBCbjPsMZ57\nk8HkyWkaPcBrTiJt7qtYTcbQQcEr6k8Sh17rRdhs9ZgC06DYVYoGmRmioHfRMJ6s\nzHXug/WwYjnPbFfiTNKRCw51KBuav/0aQ/HKd/s7j2G4aSgWQgRecCocIdiP4b0j\nWy10QJLZYxkNc91pvGJHvOB0K7Lrfb5BG7XARsWhIstfTsEokt4YutUqKLsRixeT\nmJlglFwjz1onl14LBQaTNx47aTbrqZ5hHY8y2o4M1nQ+ewkk2gF3R8Q7zTSMmfXK\n4SVhM7JZG+Ju1zdXtg2pEto=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFYDCCA0igAwIBAgIULvWbAiin23r/1aOp7r0DoM8Sah0wDQYJKoZIhvcNAQEL\nBQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc\nBgNVBAMTFVF1b1ZhZGlzIFJvb3QgQ0EgMyBHMzAeFw0xMjAxMTIyMDI2MzJaFw00\nMjAxMTIyMDI2MzJaMEgxCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBM\naW1pdGVkMR4wHAYDVQQDExVRdW9WYWRpcyBSb290IENBIDMgRzMwggIiMA0GCSqG\nSIb3DQEBAQUAA4ICDwAwggIKAoICAQCzyw4QZ47qFJenMioKVjZ/aEzHs286IxSR\n/xl/pcqs7rN2nXrpixurazHb+gtTTK/FpRp5PIpM/6zfJd5O2YIyC0TeytuMrKNu\nFoM7pmRLMon7FhY4futD4tN0SsJiCnMK3UmzV9KwCoWdcTzeo8vAMvMBOSBDGzXR\nU7Ox7sWTaYI+FrUoRqHe6okJ7UO4BUaKhvVZR74bbwEhELn9qdIoyhA5CcoTNs+c\nra1AdHkrAj80//ogaX3T7mH1urPnMNA3I4ZyYUUpSFlob3emLoG+B01vr87ERROR\nFHAGjx+f+IdpsQ7vw4kZ6+ocYfx6bIrc1gMLnia6Et3UVDmrJqMz6nWB2i3ND0/k\nA9HvFZcba5DFApCTZgIhsUfei5pKgLlVj7WiL8DWM2fafsSntARE60f75li59wzw\neyuxwHApw0BiLTtIadwjPEjrewl5qW3aqDCYz4ByA4imW0aucnl8CAMhZa634Ryl\nsSqiMd5mBPfAdOhx3v89WcyWJhKLhZVXGqtrdQtEPREoPHtht+KPZ0/l7DxMYIBp\nVzgeAVuNVejH38DMdyM0SXV89pgR6y3e7UEuFAUCf+D+IOs15xGsIs5XPd7JMG0Q\nA4XN8f+MFrXBsj6IbGB/kE+V9/YtrQE5BwT6dYB9v0lQ7e/JxHwc64B+27bQ3RP+\nydOc17KXqQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nBjAdBgNVHQ4EFgQUxhfQvKjqAkPyGwaZXSuQILnXnOQwDQYJKoZIhvcNAQELBQAD\nggIBADRh2Va1EodVTd2jNTFGu6QHcrxfYWLopfsLN7E8trP6KZ1/AvWkyaiTt3px\nKGmPc+FSkNrVvjrlt3ZqVoAh313m6Tqe5T72omnHKgqwGEfcIHB9UqM+WXzBusnI\nFUBhynLWcKzSt/Ac5IYp8M7vaGPQtSCKFWGafoaYtMnCdvvMujAWzKNhxnQT5Wvv\noxXqA/4Ti2Tk08HS6IT7SdEQTXlm66r99I0xHnAUrdzeZxNMgRVhvLfZkXdxGYFg\nu/BYpbWcC/ePIlUnwEsBbTuZDdQdm2NnL9DuDcpmvJRPpq3t/O5jrFc/ZSXPsoaP\n0Aj/uHYUbt7lJ+yreLVTubY/6CD50qi+YUbKh4yE8/nxoGibIh6BJpsQBJFxwAYf\n3KDTuVan45gtf4Od34wrnDKOMpTwATwiKp9Dwi7DmDkHOHv8XgBCH/MyJnmDhPbl\n8MFREsALHgQjDFSlTC9JxUrRtm5gDWv8a4uFJGS3iQ6rJUdbPM9+Sb3H6QrG2vd+\nDhcI00iX0HGS8A85PjRqHH3Y8iKuu2n0M7SmSFXRDw4m6Oy2Cy2nhTXN/VnIn9HN\nPlopNLk9hM6xZdRZkZFWdSHBd575euFgndOtBBj0fOtek49TSiIp+EgrPk2GrFt/\nywaZWWDYWGWVjUTR939+J399roD1B0y2PpxxVJkES/1Y+Zj0\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDYTCCAkmgAwIBAgIQCgEBAQAAAnwAAAAKAAAAAjANBgkqhkiG9w0BAQUFADA6\nMRkwFwYDVQQKExBSU0EgU2VjdXJpdHkgSW5jMR0wGwYDVQQLExRSU0EgU2VjdXJp\ndHkgMjA0OCBWMzAeFw0wMTAyMjIyMDM5MjNaFw0yNjAyMjIyMDM5MjNaMDoxGTAX\nBgNVBAoTEFJTQSBTZWN1cml0eSBJbmMxHTAbBgNVBAsTFFJTQSBTZWN1cml0eSAy\nMDQ4IFYzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt49VcdKA3Xtp\neafwGFAyPGJn9gqVB93mG/Oe2dJBVGutn3y+Gc37RqtBaB4Y6lXIL5F4iSj7Jylg\n/9+PjDvJSZu1pJTOAeo+tWN7fyb9Gd3AIb2E0S1PRsNO3Ng3OTsor8udGuorryGl\nwSMiuLgbWhOHV4PR8CDn6E8jQrAApX2J6elhc5SYcSa8LWrg903w8bYqODGBDSnh\nAMFRD0xS+ARaqn1y07iHKrtjEAMqs6FPDVpeRrc9DvV07Jmf+T0kgYim3WBU6JU2\nPcYJk5qjEoAAVZkZR73QpXzDuvsf9/UP+Ky5tfQ3mBMY3oVbtwyCO4dvlTlYMNpu\nAWgXIszACwIDAQABo2MwYTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIB\nBjAfBgNVHSMEGDAWgBQHw1EwpKrpRa41JPr/JCwz0LGdjDAdBgNVHQ4EFgQUB8NR\nMKSq6UWuNST6/yQsM9CxnYwwDQYJKoZIhvcNAQEFBQADggEBAF8+hnZuuDU8TjYc\nHnmYv/3VEhF5Ug7uMYm83X/50cYVIeiKAVQNOvtUudZj1LGqlk2iQk3UUx+LEN5/\nZb5gEydxiKRz44Rj0aRV4VCT5hsOedBnvEbIvz8XDZXmxpBp3ue0L96VfdASPz0+\nf00/FGj1EVDVwfSQpQgdMWD/YIwjVAqv/qFuxdF6Kmh4zx6CCiC0H63lhbJqaHVO\nrSU3lIW+vaHU6rcMSzyd6BIA8F+sDeGscGNz9395nzIlQnQFgCi/vcEkllgVsRch\n6YlL2weIZ/QVrXA+L02FO8K32/6YaCOJ4XQP3vTFhGMpG8zLB8kApKnXwiJPZ9d3\n7CAFYd4=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIGizCCBXOgAwIBAgIEO0XlaDANBgkqhkiG9w0BAQUFADBoMQswCQYDVQQGEwJF\nUzEfMB0GA1UEChMWR2VuZXJhbGl0YXQgVmFsZW5jaWFuYTEPMA0GA1UECxMGUEtJ\nR1ZBMScwJQYDVQQDEx5Sb290IENBIEdlbmVyYWxpdGF0IFZhbGVuY2lhbmEwHhcN\nMDEwNzA2MTYyMjQ3WhcNMjEwNzAxMTUyMjQ3WjBoMQswCQYDVQQGEwJFUzEfMB0G\nA1UEChMWR2VuZXJhbGl0YXQgVmFsZW5jaWFuYTEPMA0GA1UECxMGUEtJR1ZBMScw\nJQYDVQQDEx5Sb290IENBIEdlbmVyYWxpdGF0IFZhbGVuY2lhbmEwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDGKqtXETcvIorKA3Qdyu0togu8M1JAJke+\nWmmmO3I2F0zo37i7L3bhQEZ0ZQKQUgi0/6iMweDHiVYQOTPvaLRfX9ptI6GJXiKj\nSgbwJ/BXufjpTjJ3Cj9BZPPrZe52/lSqfR0grvPXdMIKX/UIKFIIzFVd0g/bmoGl\nu6GzwZTNVOAydTGRGmKy3nXiz0+J2ZGQD0EbtFpKd71ng+CT516nDOeB0/RSrFOy\nA8dEJvt55cs0YFAQexvba9dHq198aMpunUEDEO5rmXteJajCq+TA81yc477OMUxk\nHl6AovWDfgzWyoxVjr7gvkkHD6MkQXpYHYTqWBLI4bft75PelAgxAgMBAAGjggM7\nMIIDNzAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAGGFmh0dHA6Ly9vY3NwLnBr\naS5ndmEuZXMwEgYDVR0TAQH/BAgwBgEB/wIBAjCCAjQGA1UdIASCAiswggInMIIC\nIwYKKwYBBAG/VQIBADCCAhMwggHoBggrBgEFBQcCAjCCAdoeggHWAEEAdQB0AG8A\ncgBpAGQAYQBkACAAZABlACAAQwBlAHIAdABpAGYAaQBjAGEAYwBpAPMAbgAgAFIA\nYQDtAHoAIABkAGUAIABsAGEAIABHAGUAbgBlAHIAYQBsAGkAdABhAHQAIABWAGEA\nbABlAG4AYwBpAGEAbgBhAC4ADQAKAEwAYQAgAEQAZQBjAGwAYQByAGEAYwBpAPMA\nbgAgAGQAZQAgAFAAcgDhAGMAdABpAGMAYQBzACAAZABlACAAQwBlAHIAdABpAGYA\naQBjAGEAYwBpAPMAbgAgAHEAdQBlACAAcgBpAGcAZQAgAGUAbAAgAGYAdQBuAGMA\naQBvAG4AYQBtAGkAZQBuAHQAbwAgAGQAZQAgAGwAYQAgAHAAcgBlAHMAZQBuAHQA\nZQAgAEEAdQB0AG8AcgBpAGQAYQBkACAAZABlACAAQwBlAHIAdABpAGYAaQBjAGEA\nYwBpAPMAbgAgAHMAZQAgAGUAbgBjAHUAZQBuAHQAcgBhACAAZQBuACAAbABhACAA\nZABpAHIAZQBjAGMAaQDzAG4AIAB3AGUAYgAgAGgAdAB0AHAAOgAvAC8AdwB3AHcA\nLgBwAGsAaQAuAGcAdgBhAC4AZQBzAC8AYwBwAHMwJQYIKwYBBQUHAgEWGWh0dHA6\nLy93d3cucGtpLmd2YS5lcy9jcHMwHQYDVR0OBBYEFHs100DSHHgZZu90ECjcPk+y\neAT8MIGVBgNVHSMEgY0wgYqAFHs100DSHHgZZu90ECjcPk+yeAT8oWykajBoMQsw\nCQYDVQQGEwJFUzEfMB0GA1UEChMWR2VuZXJhbGl0YXQgVmFsZW5jaWFuYTEPMA0G\nA1UECxMGUEtJR1ZBMScwJQYDVQQDEx5Sb290IENBIEdlbmVyYWxpdGF0IFZhbGVu\nY2lhbmGCBDtF5WgwDQYJKoZIhvcNAQEFBQADggEBACRhTvW1yEICKrNcda3Fbcrn\nlD+laJWIwVTAEGmiEi8YPyVQqHxK6sYJ2fR1xkDar1CdPaUWu20xxsdzCkj+IHLt\nb8zog2EWRpABlUt9jppSCS/2bxzkoXHPjCpaF3ODR00PNvsETUlR4hTJZGH71BTg\n9J63NI8KJr2XXPR5OkowGcytT6CYirQxlyric21+eLj4iIlPsSKRZEv1UN4D2+XF\nducTZnV+ZfsBn5OHiJ35Rld8TWCvmHMTI6QgkYH60GFmuH3Rr9ZvHmw96RH9qfmC\nIoaZM3Fa6hlXPZHNqcCjbgcTpsnt+GijnsNacgmHKNHEc8RzGF9QdRYxn7fofMM=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDbTCCAlWgAwIBAgIBATANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJKUDEr\nMCkGA1UEChMiSmFwYW4gQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcywgSW5jLjEcMBoG\nA1UEAxMTU2VjdXJlU2lnbiBSb290Q0ExMTAeFw0wOTA0MDgwNDU2NDdaFw0yOTA0\nMDgwNDU2NDdaMFgxCzAJBgNVBAYTAkpQMSswKQYDVQQKEyJKYXBhbiBDZXJ0aWZp\nY2F0aW9uIFNlcnZpY2VzLCBJbmMuMRwwGgYDVQQDExNTZWN1cmVTaWduIFJvb3RD\nQTExMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA/XeqpRyQBTvLTJsz\ni1oURaTnkBbR31fSIRCkF/3frNYfp+TbfPfs37gD2pRY/V1yfIw/XwFndBWW4wI8\nh9uuywGOwvNmxoVF9ALGOrVisq/6nL+k5tSAMJjzDbaTj6nU2DbysPyKyiyhFTOV\nMdrAG/LuYpmGYz+/3ZMqg6h2uRMft85OQoWPIucuGvKVCbIFtUROd6EgvanyTgp9\nUK31BQ1FT0Zx/Sg+U/sE2C3XZR1KG/rPO7AxmjVuyIsG0wCR8pQIZUyxNAYAeoni\n8McDWc/V1uinMrPmmECGxc0nEovMe863ETxiYAcjPitAbpSACW22s293bzUIUPsC\nh8U+iQIDAQABo0IwQDAdBgNVHQ4EFgQUW/hNT7KlhtQ60vFjmqC+CfZXt94wDgYD\nVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB\nAKChOBZmLqdWHyGcBvod7bkixTgm2E5P7KN/ed5GIaGHd48HCJqypMWvDzKYC3xm\nKbabfSVSSUOrTC4rbnpwrxYO4wJs+0LmGJ1F2FXI6Dvd5+H0LgscNFxsWEr7jIhQ\nX5Ucv+2rIrVls4W6ng+4reV6G4pQOh29Dbx7VFALuUKvVaAYga1lme++5Jy/xIWr\nQbJUb9wlze144o4MjQlJ3WN7WmmWAiGovVJZ6X01y8hSyn+B/tlr0/cR7SXf+Of5\npPpyl4RTDaXQMhhRdlkUbA/r7F+AjHVDg8OFmP9Mni0N5HeDk061lgeLKBObjBmN\nQSdJQO7e5iNEOdyhIta6A/I=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDuDCCAqCgAwIBAgIQDPCOXAgWpa1Cf/DrJxhZ0DANBgkqhkiG9w0BAQUFADBI\nMQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3QgQ29ycG9yYXRpb24x\nFzAVBgNVBAMTDlNlY3VyZVRydXN0IENBMB4XDTA2MTEwNzE5MzExOFoXDTI5MTIz\nMTE5NDA1NVowSDELMAkGA1UEBhMCVVMxIDAeBgNVBAoTF1NlY3VyZVRydXN0IENv\ncnBvcmF0aW9uMRcwFQYDVQQDEw5TZWN1cmVUcnVzdCBDQTCCASIwDQYJKoZIhvcN\nAQEBBQADggEPADCCAQoCggEBAKukgeWVzfX2FI7CT8rU4niVWJxB4Q2ZQCQXOZEz\nZum+4YOvYlyJ0fwkW2Gz4BERQRwdbvC4u/jep4G6pkjGnx29vo6pQT64lO0pGtSO\n0gMdA+9tDWccV9cGrcrI9f4Or2YlSASWC12juhbDCE/RRvgUXPLIXgGZbf2IzIao\nwW8xQmxSPmjL8xk037uHGFaAJsTQ3MBv396gwpEWoGQRS0S8Hvbn+mPeZqx2pHGj\n7DaUaHp3pLHnDi+BeuK1cobvomuL8A/b01k/unK8RCSc43Oz969XL0Imnal0ugBS\n8kvNU3xHCzaFDmapCJcWNFfBZveA4+1wVMeT4C4oFVmHursCAwEAAaOBnTCBmjAT\nBgkrBgEEAYI3FAIEBh4EAEMAQTALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB\n/zAdBgNVHQ4EFgQUQjK2FvoE/f5dS3rD/fdMQB1aQ68wNAYDVR0fBC0wKzApoCeg\nJYYjaHR0cDovL2NybC5zZWN1cmV0cnVzdC5jb20vU1RDQS5jcmwwEAYJKwYBBAGC\nNxUBBAMCAQAwDQYJKoZIhvcNAQEFBQADggEBADDtT0rhWDpSclu1pqNlGKa7UTt3\n6Z3q059c4EVlew3KW+JwULKUBRSuSceNQQcSc5R+DCMh/bwQf2AQWnL1mA6s7Ll/\n3XpvXdMc9P+IBWlCqQVxyLesJugutIxq/3HcuLHfmbx8IVQr5Fiiu1cprp6poxkm\nD5kuCLDv/WnPmRoJjeOnnyvJNjR7JLN4TJUXpAYmHrZkUjZfYGfZnMUFdAvnZyPS\nCPyI6a6Lf+Ew9Dd+/cYy2i2eRDAwbO4H3tI0/NL/QPZL9GZGBlSm8jIKYyYwa5vR\n3ItHuuG51WLQoqD0ZwV4KWMabwTW+MZMo5qxN7SN5ShLHZ4swrhovO0C7jE=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDvDCCAqSgAwIBAgIQB1YipOjUiolN9BPI8PjqpTANBgkqhkiG9w0BAQUFADBK\nMQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3QgQ29ycG9yYXRpb24x\nGTAXBgNVBAMTEFNlY3VyZSBHbG9iYWwgQ0EwHhcNMDYxMTA3MTk0MjI4WhcNMjkx\nMjMxMTk1MjA2WjBKMQswCQYDVQQGEwJVUzEgMB4GA1UEChMXU2VjdXJlVHJ1c3Qg\nQ29ycG9yYXRpb24xGTAXBgNVBAMTEFNlY3VyZSBHbG9iYWwgQ0EwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvNS7YrGxVaQZx5RNoJLNP2MwhR/jxYDiJ\niQPpvepeRlMJ3Fz1Wuj3RSoC6zFh1ykzTM7HfAo3fg+6MpjhHZevj8fcyTiW89sa\n/FHtaMbQbqR8JNGuQsiWUGMu4P51/pinX0kuleM5M2SOHqRfkNJnPLLZ/kG5VacJ\njnIFHovdRIWCQtBJwB1g8NEXLJXr9qXBkqPFwqcIYA1gBBCWeZ4WNOaptvolRTnI\nHmX5k/Wq8VLcmZg9pYYaDDUz+kulBAYVHDGA76oYa8J719rO+TMg1fW9ajMtgQT7\nsFzUnKPiXB3jqUJ1XnvUd+85VLrJChgbEplJL4hL/VBi0XPnj3pDAgMBAAGjgZ0w\ngZowEwYJKwYBBAGCNxQCBAYeBABDAEEwCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQF\nMAMBAf8wHQYDVR0OBBYEFK9EBMJBfkiD2045AuzshHrmzsmkMDQGA1UdHwQtMCsw\nKaAnoCWGI2h0dHA6Ly9jcmwuc2VjdXJldHJ1c3QuY29tL1NHQ0EuY3JsMBAGCSsG\nAQQBgjcVAQQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQBjGghAfaReUw132HquHw0L\nURYD7xh8yOOvaliTFGCRsoTciE6+OYo68+aCiV0BN7OrJKQVDpI1WkpEXk5X+nXO\nH0jOZvQ8QCaSmGwb7iRGDBezUqXbpZGRzzfTb+cnCDpOGR86p1hcF895P4vkp9Mm\nI50mD1hp/Ed+stCNi5O/KU9DaXR2Z0vPB4zmAve14bRDtUstFJ/53CYNv6ZHdAbY\niNE6KTCEztI5gGIbqMdXSbxqVVFnFUq+NQfk1XWYN3kwFNspnWzFacxHVaIw98xc\nf8LDmBxrThaA63p4ZUWiABqvDA1VZDRIuJK58bRQKfJPIx/abKwfROHdI3hRW8cW\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDfTCCAmWgAwIBAgIBADANBgkqhkiG9w0BAQUFADBgMQswCQYDVQQGEwJKUDEl\nMCMGA1UEChMcU0VDT00gVHJ1c3QgU3lzdGVtcyBDTy4sTFRELjEqMCgGA1UECxMh\nU2VjdXJpdHkgQ29tbXVuaWNhdGlvbiBFViBSb290Q0ExMB4XDTA3MDYwNjAyMTIz\nMloXDTM3MDYwNjAyMTIzMlowYDELMAkGA1UEBhMCSlAxJTAjBgNVBAoTHFNFQ09N\nIFRydXN0IFN5c3RlbXMgQ08uLExURC4xKjAoBgNVBAsTIVNlY3VyaXR5IENvbW11\nbmljYXRpb24gRVYgUm9vdENBMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBALx/7FebJOD+nLpCeamIivqA4PUHKUPqjgo0No0c+qe1OXj/l3X3L+SqawSE\nRMqm4miO/VVQYg+kcQ7OBzgtQoVQrTyWb4vVog7P3kmJPdZkLjjlHmy1V4qe70gO\nzXppFodEtZDkBp2uoQSXWHnvIEqCa4wiv+wfD+mEce3xDuS4GBPMVjZd0ZoeUWs5\nbmB2iDQL87PRsJ3KYeJkHcFGB7hj3R4zZbOOCVVSPbW9/wfrrWFVGCypaZhKqkDF\nMxRldAD5kd6vA0jFQFTcD4SQaCDFkpbcLuUCRarAX1T4bepJz11sS6/vmsJWXMY1\nVkJqMF/Cq/biPT+zyRGPMUzXn0kCAwEAAaNCMEAwHQYDVR0OBBYEFDVK9U2vP9eC\nOKyrcWUXdYydVZPmMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MA0G\nCSqGSIb3DQEBBQUAA4IBAQCoh+ns+EBnXcPBZsdAS5f8hxOQWsTvoMpfi7ent/HW\ntWS3irO4G8za+6xmiEHO6Pzk2x6Ipu0nUBsCMCRGef4Eh3CXQHPRwMFXGZpppSeZ\nq51ihPZRwSzJIxXYKLerJRO1RuGGAv8mjMSIkh1W/hln8lXkgKNrnKt34VFxDSDb\nEJrbvXZ5B3eZKK2aXtqxT0QsNY6llsf9g/BYxnnWmHyojf6GPgcWkuF75x3sM3Z+\nQi5KhfmRiWiEA4Glm5q+4zfFVKtWOxgtQaQM+ELbmaDgcm+7XeEWT1MKZPlO9L9O\nVL14bIjqv5wTJMJwaaJ/D8g8rQjJsJhAoyrniIPtd490\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDdzCCAl+gAwIBAgIBADANBgkqhkiG9w0BAQsFADBdMQswCQYDVQQGEwJKUDEl\nMCMGA1UEChMcU0VDT00gVHJ1c3QgU3lzdGVtcyBDTy4sTFRELjEnMCUGA1UECxMe\nU2VjdXJpdHkgQ29tbXVuaWNhdGlvbiBSb290Q0EyMB4XDTA5MDUyOTA1MDAzOVoX\nDTI5MDUyOTA1MDAzOVowXTELMAkGA1UEBhMCSlAxJTAjBgNVBAoTHFNFQ09NIFRy\ndXN0IFN5c3RlbXMgQ08uLExURC4xJzAlBgNVBAsTHlNlY3VyaXR5IENvbW11bmlj\nYXRpb24gUm9vdENBMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANAV\nOVKxUrO6xVmCxF1SrjpDZYBLx/KWvNs2l9amZIyoXvDjChz335c9S672XewhtUGr\nzbl+dp+++T42NKA7wfYxEUV0kz1XgMX5iZnK5atq1LXaQZAQwdbWQonCv/Q4EpVM\nVAX3NuRFg3sUZdbcDE3R3n4MqzvEFb46VqZab3ZpUql6ucjrappdUtAtCms1FgkQ\nhNBqyjoGADdH5H5XTz+L62e4iKrFvlNVspHEfbmwhRkGeC7bYRr6hfVKkaHnFtWO\nojnflLhwHyg/i/xAXmODPIMqGplrz95Zajv8bxbXH/1KEOtOghY6rCcMU/Gt1SSw\nawNQwS08Ft1ENCcadfsCAwEAAaNCMEAwHQYDVR0OBBYEFAqFqXdlBZh8QIH4D5cs\nOPEK7DzPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3\nDQEBCwUAA4IBAQBMOqNErLlFsceTfsgLCkLfZOoc7llsCLqJX2rKSpWeeo8HxdpF\ncoJxDjrSzG+ntKEju/Ykn8sX/oymzsLS28yN/HH8AynBbF0zX2S2ZTuJbxh2ePXc\nokgfGT+Ok+vx+hfuzU7jBBJV1uXk3fs+BXziHV7Gp7yXT2g69ekuCkO2r1dcYmh8\nt/2jioSgrGK+KwmHNPBqAbubKVY8/gA3zyNs8U6qtnRGEmyR7jTV7JqR50S+kDFy\n1UkC9gLl9B/rfNmWVan/7Ir5mUf/NVoCqgTLiluHcSmRvaS0eg29mvVXIwAHIRc/\nSjnRBUkLp7Y3gaVdjKozXoEofKd9J+sAro03\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDWjCCAkKgAwIBAgIBADANBgkqhkiG9w0BAQUFADBQMQswCQYDVQQGEwJKUDEY\nMBYGA1UEChMPU0VDT00gVHJ1c3QubmV0MScwJQYDVQQLEx5TZWN1cml0eSBDb21t\ndW5pY2F0aW9uIFJvb3RDQTEwHhcNMDMwOTMwMDQyMDQ5WhcNMjMwOTMwMDQyMDQ5\nWjBQMQswCQYDVQQGEwJKUDEYMBYGA1UEChMPU0VDT00gVHJ1c3QubmV0MScwJQYD\nVQQLEx5TZWN1cml0eSBDb21tdW5pY2F0aW9uIFJvb3RDQTEwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQCzs/5/022x7xZ8V6UMbXaKL0u/ZPtM7orw8yl8\n9f/uKuDp6bpbZCKamm8sOiZpUQWZJtzVHGpxxpp9Hp3dfGzGjGdnSj74cbAZJ6kJ\nDKaVv0uMDPpVmDvY6CKhS3E4eayXkmmziX7qIWgGmBSWh9JhNrxtJ1aeV+7AwFb9\nMs+k2Y7CI9eNqPPYJayX5HA49LY6tJ07lyZDo6G8SVlyTCMwhwFY9k6+HGhWZq/N\nQV3Is00qVUarH9oe4kA92819uZKAnDfdDJZkndwi92SL32HeFZRSFaB9UslLqCHJ\nxrHty8OVYNEP8Ktw+N/LTX7s1vqr2b1/VPKl6Xn62dZ2JChzAgMBAAGjPzA9MB0G\nA1UdDgQWBBSgc0mZaNyFW2XjmygvV5+9M7wHSDALBgNVHQ8EBAMCAQYwDwYDVR0T\nAQH/BAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAaECpqLvkT115swW1F7NgE+vG\nkl3g0dNq/vu+m22/xwVtWSDEHPC32oRYAmP6SBbvT6UL90qY8j+eG61Ha2POCEfr\nUj94nK9NrvjVT8+amCoQQTlSxN3Zmw7vkwGusi7KaEIkQmywszo+zenaSMQVy+n5\nBw+SUEmK3TGXX8npN6o7WWWXlDLJs58+OmJYxUmtYg5xpTKqL8aJdkNAExNnPaJU\nJRDL8Try2frbSVa7pv6nQTXD4IhhyYjH3zYQIphZ6rBK+1YWc26sTfcioU+tHXot\nRSflMMFe8toTyyVCUZVHA4xsIcx0Qu1T/zOLjw9XARYvz6buyXAiFL39vmwLAw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDIDCCAgigAwIBAgIBHTANBgkqhkiG9w0BAQUFADA5MQswCQYDVQQGEwJGSTEP\nMA0GA1UEChMGU29uZXJhMRkwFwYDVQQDExBTb25lcmEgQ2xhc3MyIENBMB4XDTAx\nMDQwNjA3Mjk0MFoXDTIxMDQwNjA3Mjk0MFowOTELMAkGA1UEBhMCRkkxDzANBgNV\nBAoTBlNvbmVyYTEZMBcGA1UEAxMQU29uZXJhIENsYXNzMiBDQTCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAJAXSjWdyvANlsdE+hY3/Ei9vX+ALTU74W+o\nZ6m/AxxNjG8yR9VBaKQTBME1DJqEQ/xcHf+Js+gXGM2RX/uJ4+q/Tl18GybTdXnt\n5oTjV+WtKcT0OijnpXuENmmz/V52vaMtmdOQTiMofRhj8VQ7Jp12W5dCsv+u8E7s\n3TmVToMGf+dJQMjFAbJUWmYdPfz56TwKnoG4cPABi+QjVHzIrviQHgCWctRUz2Ej\nvOr7nQKV0ba5cTppCD8PtOFCx4j1P5iop7oc4HFx71hXgVB6XGt0Rg6DA5jDjqhu\n8nYybieDwnPz3BjotJPqdURrBGAgcVeHnfO+oJAjPYok4doh28MCAwEAAaMzMDEw\nDwYDVR0TAQH/BAUwAwEB/zARBgNVHQ4ECgQISqCqWITTXjwwCwYDVR0PBAQDAgEG\nMA0GCSqGSIb3DQEBBQUAA4IBAQBazof5FnIVV0sd2ZvnoiYw7JNn39Yt0jSv9zil\nzqsWuasvfDXLrNAPtEwr/IDva4yRXzZ299uzGxnq9LIR/WFxRL8oszodv7ND6J+/\n3DEIcbCdjdY0RzKQxmUk96BKfARzjzlvF4xytb1LyHr4e4PDKE6cCepnP7JnBBvD\nFNr450kkkdAdavphOe9r5yF1BgfYErQhIHBCcYHaPJo2vqZbDWpsmh+Re/n570K6\nTk6ezAyNlNzZRZxe7EJQY670XcSxEtzKO6gunRRaBXW37Ndj4ro1tgQIkejanZz2\nZrUYrAqmVCY0M9IbwdR/GjqOC6oybtv8TyWf2TLHllpwrN9M\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDujCCAqKgAwIBAgIEAJiWijANBgkqhkiG9w0BAQUFADBVMQswCQYDVQQGEwJO\nTDEeMBwGA1UEChMVU3RhYXQgZGVyIE5lZGVybGFuZGVuMSYwJAYDVQQDEx1TdGFh\ndCBkZXIgTmVkZXJsYW5kZW4gUm9vdCBDQTAeFw0wMjEyMTcwOTIzNDlaFw0xNTEy\nMTYwOTE1MzhaMFUxCzAJBgNVBAYTAk5MMR4wHAYDVQQKExVTdGFhdCBkZXIgTmVk\nZXJsYW5kZW4xJjAkBgNVBAMTHVN0YWF0IGRlciBOZWRlcmxhbmRlbiBSb290IENB\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmNK1URF6gaYUmHFtvszn\nExvWJw56s2oYHLZhWtVhCb/ekBPHZ+7d89rFDBKeNVU+LCeIQGv33N0iYfXCxw71\n9tV2U02PjLwYdjeFnejKScfST5gTCaI+Ioicf9byEGW07l8Y1Rfj+MX94p2i71MO\nhXeiD+EwR+4A5zN9RGcaC1Hoi6CeUJhoNFIfLm0B8mBF8jHrqTFoKbt6QZ7GGX+U\ntFE5A3+y3qcym7RHjm+0Sq7lr7HcsBthvJly3uSJt3omXdozSVtSnA71iq3DuD3o\nBmrC1SoLbHuEvVYFy4ZlkuxEK7COudxwC0barbxjiDn622r+I/q85Ej0ZytqERAh\nSQIDAQABo4GRMIGOMAwGA1UdEwQFMAMBAf8wTwYDVR0gBEgwRjBEBgRVHSAAMDww\nOgYIKwYBBQUHAgEWLmh0dHA6Ly93d3cucGtpb3ZlcmhlaWQubmwvcG9saWNpZXMv\ncm9vdC1wb2xpY3kwDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBSofeu8Y6R0E3QA\n7Jbg0zTBLL9s+DANBgkqhkiG9w0BAQUFAAOCAQEABYSHVXQ2YcG70dTGFagTtJ+k\n/rvuFbQvBgwp8qiSpGEN/KtcCFtREytNwiphyPgJWPwtArI5fZlmgb9uXJVFIGzm\neafR2Bwp/MIgJ1HI8XxdNGdphREwxgDS1/PTfLbwMVcoEoJz6TMvplW0C5GUR5z6\nu3pCMuiufi3IvKwUv9kP2Vv8wfl6leF9fpb8cbDCTMjfRTTJzg3ynGQI0DvDKcWy\n7ZAEwbEpkcUwb8GpcjPM/l0WFywRaed+/sWDCN+83CI6LiBpIzlWYGeQiy52OfsR\niJf2fL1LuCAWZwWN4jvBcj+UlTfHXbme2JOhF4//DGYVwSR8MnwDHTuhWEUykw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFyjCCA7KgAwIBAgIEAJiWjDANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJO\nTDEeMBwGA1UECgwVU3RhYXQgZGVyIE5lZGVybGFuZGVuMSswKQYDVQQDDCJTdGFh\ndCBkZXIgTmVkZXJsYW5kZW4gUm9vdCBDQSAtIEcyMB4XDTA4MDMyNjExMTgxN1oX\nDTIwMDMyNTExMDMxMFowWjELMAkGA1UEBhMCTkwxHjAcBgNVBAoMFVN0YWF0IGRl\nciBOZWRlcmxhbmRlbjErMCkGA1UEAwwiU3RhYXQgZGVyIE5lZGVybGFuZGVuIFJv\nb3QgQ0EgLSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMVZ5291\nqj5LnLW4rJ4L5PnZyqtdj7U5EILXr1HgO+EASGrP2uEGQxGZqhQlEq0i6ABtQ8Sp\nuOUfiUtnvWFI7/3S4GCI5bkYYCjDdyutsDeqN95kWSpGV+RLufg3fNU254DBtvPU\nZ5uW6M7XxgpT0GtJlvOjCwV3SPcl5XCsMBQgJeN/dVrlSPhOewMHBPqCYYdu8DvE\npMfQ9XQ+pV0aCPKbJdL2rAQmPlU6Yiile7Iwr/g3wtG61jj99O9JMDeZJiFIhQGp\n5Rbn3JBV3w/oOM2ZNyFPXfUib2rFEhZgF1XyZWampzCROME4HYYEhLoaJXhena/M\nUGDWE4dS7WMfbWV9whUYdMrhfmQpjHLYFhN9C0lK8SgbIHRrxT3dsKpICT0ugpTN\nGmXZK4iambwYfp/ufWZ8Pr2UuIHOzZgweMFvZ9C+X+Bo7d7iscksWXiSqt8rYGPy\n5V6548r6f1CGPqI0GAwJaCgRHOThuVw+R7oyPxjMW4T182t0xHJ04eOLoEq9jWYv\n6q012iDTiIJh8BIitrzQ1aTsr1SIJSQ8p22xcik/Plemf1WvbibG/ufMQFxRRIEK\neN5KzlW/HdXZt1bv8Hb/C3m1r737qWmRRpdogBQ2HbN/uymYNqUg+oJgYjOk7Na6\nB6duxc8UpufWkjTYgfX8HV2qXB72o007uPc5AgMBAAGjgZcwgZQwDwYDVR0TAQH/\nBAUwAwEB/zBSBgNVHSAESzBJMEcGBFUdIAAwPzA9BggrBgEFBQcCARYxaHR0cDov\nL3d3dy5wa2lvdmVyaGVpZC5ubC9wb2xpY2llcy9yb290LXBvbGljeS1HMjAOBgNV\nHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJFoMocVHYnitfGsNig0jQt8YojrMA0GCSqG\nSIb3DQEBCwUAA4ICAQCoQUpnKpKBglBu4dfYszk78wIVCVBR7y29JHuIhjv5tLyS\nCZa59sCrI2AGeYwRTlHSeYAz+51IvuxBQ4EffkdAHOV6CMqqi3WtFMTC6GY8ggen\n5ieCWxjmD27ZUD6KQhgpxrRW/FYQoAUXvQwjf/ST7ZwaUb7dRUG/kSS0H4zpX897\nIZmflZ85OkYcbPnNe5yQzSipx6lVu6xiNGI1E0sUOlWDuYaNkqbG9AclVMwWVxJK\ngnjIFNkXgiYtXSAfea7+1HAWFpWD2DU5/1JddRwWxRNVz0fMdWVSSt7wsKfkCpYL\n+63C4iWEst3kvX5ZbJvw8NjnyvLplzh+ib7M+zkXYT9y2zqR2GUBGR2tUKRXCnxL\nvJxxcypFURmFzI79R6d0lR2o0a9OF7FpJsKqeFdbxU2n5Z4FF5TKsl+gSRiNNOkm\nbEgeqmiSBeGCc1qb3AdbCG19ndeNIdn8FCCqwkXfP+cAslHkwvgFuXkajDTznlvk\nN1trSt8sV4pAWja63XVECDdCcAz+3F4hoKOKwJCcaNpQ5kUQR3i2TtJlycM33+FC\nY7BXN0Ute4qcvwXqZVUz9zkQxSgqIXobisQk+T8VyJoVIPVVYpbtbZNQvOSqeK3Z\nywplh6ZmwcSBo3c6WB4L7oOLnR7SUqTMHW+wmG2UMbX4cQrcufx9MmDm66+KAQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEDzCCAvegAwIBAgIBADANBgkqhkiG9w0BAQUFADBoMQswCQYDVQQGEwJVUzEl\nMCMGA1UEChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMp\nU3RhcmZpZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQw\nNjI5MTczOTE2WhcNMzQwNjI5MTczOTE2WjBoMQswCQYDVQQGEwJVUzElMCMGA1UE\nChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjEyMDAGA1UECxMpU3RhcmZp\nZWxkIENsYXNzIDIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggEgMA0GCSqGSIb3\nDQEBAQUAA4IBDQAwggEIAoIBAQC3Msj+6XGmBIWtDBFk385N78gDGIc/oav7PKaf\n8MOh2tTYbitTkPskpD6E8J7oX+zlJ0T1KKY/e97gKvDIr1MvnsoFAZMej2YcOadN\n+lq2cwQlZut3f+dZxkqZJRRU6ybH838Z1TBwj6+wRir/resp7defqgSHo9T5iaU0\nX9tDkYI22WY8sbi5gv2cOj4QyDvvBmVmepsZGD3/cVE8MC5fvj13c7JdBmzDI1aa\nK4UmkhynArPkPw2vCHmCuDY96pzTNbO8acr1zJ3o/WSNF4Azbl5KXZnJHoe0nRrA\n1W4TNSNe35tfPe/W93bC6j67eA0cQmdrBNj41tpvi/JEoAGrAgEDo4HFMIHCMB0G\nA1UdDgQWBBS/X7fRzt0fhvRbVazc1xDCDqmI5zCBkgYDVR0jBIGKMIGHgBS/X7fR\nzt0fhvRbVazc1xDCDqmI56FspGowaDELMAkGA1UEBhMCVVMxJTAjBgNVBAoTHFN0\nYXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAsTKVN0YXJmaWVsZCBD\nbGFzcyAyIENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggEAMAwGA1UdEwQFMAMBAf8w\nDQYJKoZIhvcNAQEFBQADggEBAAWdP4id0ckaVaGsafPzWdqbAYcaT1epoXkJKtv3\nL7IezMdeatiDh6GX70k1PncGQVhiv45YuApnP+yz3SFmH8lU+nLMPUxA2IGvd56D\neruix/U0F47ZEUD0/CwqTRV/p2JdLiXTAAsgGh1o+Re49L2L7ShZ3U0WixeDyLJl\nxy16paq8U4Zt3VekyvggQQto8PT7dL5WXXp59fkdheMtlb71cZBDzI0fmgAKhynp\nVSJYACPq4xJDKVtHCN2MQWplBqjlIapBtJUhlbl90TSrE9atvNziPTnNvT51cKEY\nWQPJIrSPnNVeKtelttQKbfi3QBFGmh95DmK/D5fs4C8fF5Q=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID3TCCAsWgAwIBAgIBADANBgkqhkiG9w0BAQsFADCBjzELMAkGA1UEBhMCVVMx\nEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT\nHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xMjAwBgNVBAMTKVN0YXJmaWVs\nZCBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5MDkwMTAwMDAw\nMFoXDTM3MTIzMTIzNTk1OVowgY8xCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdBcml6\nb25hMRMwEQYDVQQHEwpTY290dHNkYWxlMSUwIwYDVQQKExxTdGFyZmllbGQgVGVj\naG5vbG9naWVzLCBJbmMuMTIwMAYDVQQDEylTdGFyZmllbGQgUm9vdCBDZXJ0aWZp\nY2F0ZSBBdXRob3JpdHkgLSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAL3twQP89o/8ArFvW59I2Z154qK3A2FWGMNHttfKPTUuiUP3oWmb3ooa/RMg\nnLRJdzIpVv257IzdIvpy3Cdhl+72WoTsbhm5iSzchFvVdPtrX8WJpRBSiUZV9Lh1\nHOZ/5FSuS/hVclcCGfgXcVnrHigHdMWdSL5stPSksPNkN3mSwOxGXn/hbVNMYq/N\nHwtjuzqd+/x5AJhhdM8mgkBj87JyahkNmcrUDnXMN/uLicFZ8WJ/X7NfZTD4p7dN\ndloedl40wOiWVpmKs/B/pM293DIxfJHP4F8R+GuqSVzRmZTRouNjWwl2tVZi4Ut0\nHZbUJtQIBFnQmA4O5t78w+wfkPECAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAO\nBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFHwMMh+n2TB/xH1oo2Kooc6rB1snMA0G\nCSqGSIb3DQEBCwUAA4IBAQARWfolTwNvlJk7mh+ChTnUdgWUXuEok21iXQnCoKjU\nsHU48TRqneSfioYmUeYs0cYtbpUgSpIB7LiKZ3sx4mcujJUDJi5DnUox9g61DLu3\n4jd/IroAow57UvtruzvE03lRTs2Q9GcHGcg8RnoNAX3FWOdt5oUwF5okxBDgBPfg\n8n/Uqgr/Qh037ZTlZFkSIHc40zI+OIF1lnP6aI+xy84fxez6nH7PfrHxBy22/L/K\npL/QlwVKvOoYKAKQvVR4CSFx09F9HdkWsKlhPdAKACL8x3vLCWRFCztAgfd9fDL1\nmMpYjn0q7pBZc2T5NnReJaH1ZgUufzkVqSr7UIuOhWn0\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID7zCCAtegAwIBAgIBADANBgkqhkiG9w0BAQsFADCBmDELMAkGA1UEBhMCVVMx\nEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT\nHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4xOzA5BgNVBAMTMlN0YXJmaWVs\nZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTA5\nMDkwMTAwMDAwMFoXDTM3MTIzMTIzNTk1OVowgZgxCzAJBgNVBAYTAlVTMRAwDgYD\nVQQIEwdBcml6b25hMRMwEQYDVQQHEwpTY290dHNkYWxlMSUwIwYDVQQKExxTdGFy\nZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTswOQYDVQQDEzJTdGFyZmllbGQgU2Vy\ndmljZXMgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBANUMOsQq+U7i9b4Zl1+OiFOxHz/Lz58gE20p\nOsgPfTz3a3Y4Y9k2YKibXlwAgLIvWX/2h/klQ4bnaRtSmpDhcePYLQ1Ob/bISdm2\n8xpWriu2dBTrz/sm4xq6HZYuajtYlIlHVv8loJNwU4PahHQUw2eeBGg6345AWh1K\nTs9DkTvnVtYAcMtS7nt9rjrnvDH5RfbCYM8TWQIrgMw0R9+53pBlbQLPLJGmpufe\nhRhJfGZOozptqbXuNC66DQO4M99H67FrjSXZm86B0UVGMpZwh94CDklDhbZsc7tk\n6mFBrMnUVN+HL8cisibMn1lUaJ/8viovxFUcdUBgF4UCVTmLfwUCAwEAAaNCMEAw\nDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJxfAN+q\nAdcwKziIorhtSpzyEZGDMA0GCSqGSIb3DQEBCwUAA4IBAQBLNqaEd2ndOxmfZyMI\nbw5hyf2E3F/YNoHN2BtBLZ9g3ccaaNnRbobhiCPPE95Dz+I0swSdHynVv/heyNXB\nve6SbzJ08pGCL72CQnqtKrcgfU28elUSwhXqvfdqlS5sdJ/PHLTyxQGjhdByPq1z\nqwubdQxtRbeOlKyWN7Wg0I8VRw7j6IPdj/3vQQF3zCepYoUz8jcI73HPdwbeyBkd\niEDPfUYd/x7H4c7/I9vG+o1VTqkC50cRRj70/b17KSa7qWFiNyi2LSr2EIZkyXCn\n0q23KXB56jzaYyWf/Wi3MOxw+3WKt21gZ7IeyLnp2KhvAotnDU0mV3HaIPzBSlCN\nsSi6\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIHyTCCBbGgAwIBAgIBATANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEW\nMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwg\nQ2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3RhcnRDb20gQ2VydGlmaWNh\ndGlvbiBBdXRob3JpdHkwHhcNMDYwOTE3MTk0NjM2WhcNMzYwOTE3MTk0NjM2WjB9\nMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi\nU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3Rh\ncnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUA\nA4ICDwAwggIKAoICAQDBiNsJvGxGfHiflXu1M5DycmLWwTYgIiRezul38kMKogZk\npMyONvg45iPwbm2xPN1yo4UcodM9tDMr0y+v/uqwQVlntsQGfQqedIXWeUyAN3rf\nOQVSWff0G0ZDpNKFhdLDcfN1YjS6LIp/Ho/u7TTQEceWzVI9ujPW3U3eCztKS5/C\nJi/6tRYccjV3yjxd5srhJosaNnZcAdt0FCX+7bWgiA/deMotHweXMAEtcnn6RtYT\nKqi5pquDSR3l8u/d5AGOGAqPY1MWhWKpDhk6zLVmpsJrdAfkK+F2PrRt2PZE4XNi\nHzvEvqBTViVsUQn3qqvKv3b9bZvzndu/PWa8DFaqr5hIlTpL36dYUNk4dalb6kMM\nAv+Z6+hsTXBbKWWc3apdzK8BMewM69KN6Oqce+Zu9ydmDBpI125C4z/eIT574Q1w\n+2OqqGwaVLRcJXrJosmLFqa7LH4XXgVNWG4SHQHuEhANxjJ/GP/89PrNbpHoNkm+\nGkhpi8KWTRoSsmkXwQqQ1vp5Iki/untp+HDH+no32NgN0nZPV/+Qt+OR0t3vwmC3\nZzrd/qqc8NSLf3Iizsafl7b4r4qgEKjZ+xjGtrVcUjyJthkqcwEKDwOzEmDyei+B\n26Nu/yYwl/WL3YlXtq09s68rxbd2AvCl1iuahhQqcvbjM4xdCUsT37uMdBNSSwID\nAQABo4ICUjCCAk4wDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAa4wHQYDVR0OBBYE\nFE4L7xqkQFulF2mHMMo0aEPQQa7yMGQGA1UdHwRdMFswLKAqoCiGJmh0dHA6Ly9j\nZXJ0LnN0YXJ0Y29tLm9yZy9zZnNjYS1jcmwuY3JsMCugKaAnhiVodHRwOi8vY3Js\nLnN0YXJ0Y29tLm9yZy9zZnNjYS1jcmwuY3JsMIIBXQYDVR0gBIIBVDCCAVAwggFM\nBgsrBgEEAYG1NwEBATCCATswLwYIKwYBBQUHAgEWI2h0dHA6Ly9jZXJ0LnN0YXJ0\nY29tLm9yZy9wb2xpY3kucGRmMDUGCCsGAQUFBwIBFilodHRwOi8vY2VydC5zdGFy\ndGNvbS5vcmcvaW50ZXJtZWRpYXRlLnBkZjCB0AYIKwYBBQUHAgIwgcMwJxYgU3Rh\ncnQgQ29tbWVyY2lhbCAoU3RhcnRDb20pIEx0ZC4wAwIBARqBl0xpbWl0ZWQgTGlh\nYmlsaXR5LCByZWFkIHRoZSBzZWN0aW9uICpMZWdhbCBMaW1pdGF0aW9ucyogb2Yg\ndGhlIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFBvbGljeSBhdmFp\nbGFibGUgYXQgaHR0cDovL2NlcnQuc3RhcnRjb20ub3JnL3BvbGljeS5wZGYwEQYJ\nYIZIAYb4QgEBBAQDAgAHMDgGCWCGSAGG+EIBDQQrFilTdGFydENvbSBGcmVlIFNT\nTCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTANBgkqhkiG9w0BAQUFAAOCAgEAFmyZ\n9GYMNPXQhV59CuzaEE44HF7fpiUFS5Eyweg78T3dRAlbB0mKKctmArexmvclmAk8\njhvh3TaHK0u7aNM5Zj2gJsfyOZEdUauCe37Vzlrk4gNXcGmXCPleWKYK34wGmkUW\nFjgKXlf2Ysd6AgXmvB618p70qSmD+LIU424oh0TDkBreOKk8rENNZEXO3SipXPJz\newT4F+irsfMuXGRuczE6Eri8sxHkfY+BUZo7jYn0TZNmezwD7dOaHZrzZVD1oNB1\nny+v8OqCQ5j4aZyJecRDjkZy42Q2Eq/3JR44iZB3fsNrarnDy0RLrHiQi+fHLB5L\nEUTINFInzQpdn4XBidUaePKVEFMy3YCEZnXZtWgo+2EuvoSoOMCZEoalHmdkrQYu\nL6lwhceWD3yJZfWOQ1QOq92lgDmUYMA0yZZwLKMS9R9Ie70cfmu3nZD0Ijuu+Pwq\nyvqCUqDvr0tVk+vBtfAii6w0TiYiBKGHLHVKt+V9E9e4DGTANtLJL4YSjCMJwRuC\nO3NJo2pXh5Tl1njFmUNj403gdy3hZZlyaQQaRwnmDwFWJPsfvw55qVguucQJAX6V\num0ABj6y6koQOdjQK/W/7HW/lwLFCRsI3FU34oH7N4RDYiDK51ZLZer+bMEkkySh\nNOsF/5oirpt9P/FlUQqmMGqz9IgcgA38corog14=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIHhzCCBW+gAwIBAgIBLTANBgkqhkiG9w0BAQsFADB9MQswCQYDVQQGEwJJTDEW\nMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwg\nQ2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3RhcnRDb20gQ2VydGlmaWNh\ndGlvbiBBdXRob3JpdHkwHhcNMDYwOTE3MTk0NjM3WhcNMzYwOTE3MTk0NjM2WjB9\nMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi\nU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3Rh\ncnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUA\nA4ICDwAwggIKAoICAQDBiNsJvGxGfHiflXu1M5DycmLWwTYgIiRezul38kMKogZk\npMyONvg45iPwbm2xPN1yo4UcodM9tDMr0y+v/uqwQVlntsQGfQqedIXWeUyAN3rf\nOQVSWff0G0ZDpNKFhdLDcfN1YjS6LIp/Ho/u7TTQEceWzVI9ujPW3U3eCztKS5/C\nJi/6tRYccjV3yjxd5srhJosaNnZcAdt0FCX+7bWgiA/deMotHweXMAEtcnn6RtYT\nKqi5pquDSR3l8u/d5AGOGAqPY1MWhWKpDhk6zLVmpsJrdAfkK+F2PrRt2PZE4XNi\nHzvEvqBTViVsUQn3qqvKv3b9bZvzndu/PWa8DFaqr5hIlTpL36dYUNk4dalb6kMM\nAv+Z6+hsTXBbKWWc3apdzK8BMewM69KN6Oqce+Zu9ydmDBpI125C4z/eIT574Q1w\n+2OqqGwaVLRcJXrJosmLFqa7LH4XXgVNWG4SHQHuEhANxjJ/GP/89PrNbpHoNkm+\nGkhpi8KWTRoSsmkXwQqQ1vp5Iki/untp+HDH+no32NgN0nZPV/+Qt+OR0t3vwmC3\nZzrd/qqc8NSLf3Iizsafl7b4r4qgEKjZ+xjGtrVcUjyJthkqcwEKDwOzEmDyei+B\n26Nu/yYwl/WL3YlXtq09s68rxbd2AvCl1iuahhQqcvbjM4xdCUsT37uMdBNSSwID\nAQABo4ICEDCCAgwwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD\nVR0OBBYEFE4L7xqkQFulF2mHMMo0aEPQQa7yMB8GA1UdIwQYMBaAFE4L7xqkQFul\nF2mHMMo0aEPQQa7yMIIBWgYDVR0gBIIBUTCCAU0wggFJBgsrBgEEAYG1NwEBATCC\nATgwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5w\nZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL2ludGVybWVk\naWF0ZS5wZGYwgc8GCCsGAQUFBwICMIHCMCcWIFN0YXJ0IENvbW1lcmNpYWwgKFN0\nYXJ0Q29tKSBMdGQuMAMCAQEagZZMaW1pdGVkIExpYWJpbGl0eSwgcmVhZCB0aGUg\nc2VjdGlvbiAqTGVnYWwgTGltaXRhdGlvbnMqIG9mIHRoZSBTdGFydENvbSBDZXJ0\naWZpY2F0aW9uIEF1dGhvcml0eSBQb2xpY3kgYXZhaWxhYmxlIGF0IGh0dHA6Ly93\nd3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwEQYJYIZIAYb4QgEBBAQDAgAHMDgG\nCWCGSAGG+EIBDQQrFilTdGFydENvbSBGcmVlIFNTTCBDZXJ0aWZpY2F0aW9uIEF1\ndGhvcml0eTANBgkqhkiG9w0BAQsFAAOCAgEAjo/n3JR5fPGFf59Jb2vKXfuM/gTF\nwWLRfUKKvFO3lANmMD+x5wqnUCBVJX92ehQN6wQOQOY+2IirByeDqXWmN3PH/UvS\nTa0XQMhGvjt/UfzDtgUx3M2FIk5xt/JxXrAaxrqTi3iSSoX4eA+D/i+tLPfkpLst\n0OcNOrg+zvZ49q5HJMqjNTbOx8aHmNrs++myziebiMMEofYLWWivydsQD032ZGNc\npRJvkrKTlMeIFw6Ttn5ii5B/q06f/ON1FE8qMt9bDeD1e5MNq6HPh+GlBEXoPBKl\nCcWw0bdT82AUuoVpaiF8H3VhFyAXe2w7QSlc4axa0c2Mm+tgHRns9+Ww2vl5GKVF\nP0lDV9LdJNUso/2RjSe15esUBppMeyG7Oq0wBhjA2MFrLH9ZXF2RsXAiV+uKa0hK\n1Q8p7MZAwC+ITGgBF3f0JBlPvfrhsiAhS90a2Cl9qrjeVOwhVYBsHvUwyKMQ5bLm\nKhQxw4UtjJixhlpPiVktucf3HMiKf8CdBUrmQk9io20ppB+Fq9vlgcitKj1MXVuE\nJnHEhV5xJMqlG2zYYdMa4FTbzrqpMrUi9nNBCV24F10OD5mQ1kfabwo6YigUZ4LZ\n8dCAWZvLMdibD4x3TrVoivJs9iQOLWxwxXPR3hTQcY+203sC9uO41Alua551hDnm\nfyWl8kgAwKQB2j8=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFYzCCA0ugAwIBAgIBOzANBgkqhkiG9w0BAQsFADBTMQswCQYDVQQGEwJJTDEW\nMBQGA1UEChMNU3RhcnRDb20gTHRkLjEsMCoGA1UEAxMjU3RhcnRDb20gQ2VydGlm\naWNhdGlvbiBBdXRob3JpdHkgRzIwHhcNMTAwMTAxMDEwMDAxWhcNMzkxMjMxMjM1\nOTAxWjBTMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEsMCoG\nA1UEAxMjU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgRzIwggIiMA0G\nCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC2iTZbB7cgNr2Cu+EWIAOVeq8Oo1XJ\nJZlKxdBWQYeQTSFgpBSHO839sj60ZwNq7eEPS8CRhXBF4EKe3ikj1AENoBB5uNsD\nvfOpL9HG4A/LnooUCri99lZi8cVytjIl2bLzvWXFDSxu1ZJvGIsAQRSCb0AgJnoo\nD/Uefyf3lLE3PbfHkffiAez9lInhzG7TNtYKGXmu1zSCZf98Qru23QumNK9LYP5/\nQ0kGi4xDuFby2X8hQxfqp0iVAXV16iulQ5XqFYSdCI0mblWbq9zSOdIxHWDirMxW\nRST1HFSr7obdljKF+ExP6JV2tgXdNiNnvP8V4so75qbsO+wmETRIjfaAKxojAuuK\nHDp2KntWFhxyKrOq42ClAJ8Em+JvHhRYW6Vsi1g8w7pOOlz34ZYrPu8HvKTlXcxN\nnw3h3Kq74W4a7I/htkxNeXJdFzULHdfBR9qWJODQcqhaX2YtENwvKhOuJv4KHBnM\n0D4LnMgJLvlblnpHnOl68wVQdJVznjAJ85eCXuaPOQgeWeU1FEIT/wCc976qUM/i\nUUjXuG+v+E5+M5iSFGI6dWPPe/regjupuznixL0sAA7IF6wT700ljtizkC+p2il9\nHa90OrInwMEePnWjFqmveiJdnxMaz6eg6+OGCtP95paV1yPIN93EfKo2rJgaErHg\nTuixO/XWb/Ew1wIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE\nAwIBBjAdBgNVHQ4EFgQUS8W0QGutHLOlHGVuRjaJhwUMDrYwDQYJKoZIhvcNAQEL\nBQADggIBAHNXPyzVlTJ+N9uWkusZXn5T50HsEbZH77Xe7XRcxfGOSeD8bpkTzZ+K\n2s06Ctg6Wgk/XzTQLwPSZh0avZyQN8gMjgdalEVGKua+etqhqaRpEpKwfTbURIfX\nUfEpY9Z1zRbkJ4kd+MIySP3bmdCPX1R0zKxnNBFi2QwKN4fRoxdIjtIXHfbX/dtl\n6/2o1PXWT6RbdejF0mCy2wl+JYt7ulKSnj7oxXehPOBKc2thz4bcQ///If4jXSRK\n9dNtD2IEBVeC2m6kMyV5Sy5UGYvMLD0w6dEG/+gyRr61M3Z3qAFdlsHB1b6uJcDJ\nHgoJIIihDsnzb02CVAAgp9KP5DlUFy6NHrgbuxu9mk47EDTcnIhT76IxW1hPkWLI\nwpqazRVdOKnWvvgTtZ8SafJQYqz7Fzf07rh1Z2AQ+4NQ+US1dZxAF7L+/XldblhY\nXzD8AK6vM8EOTmy6p6ahfzLbOOCxchcKK5HsamMm7YnUeMx0HgX4a/6ManY5Ka5l\nIxKVCCIcl85bBu4M4ru8H0ST9tg4RQUh7eStqxK2A6RCLi3ECToDZ2mEmuFZkIoo\nhdVddLHRDiBYmxOlsGOm7XtH/UVVMKTumtTm4ofvmMkyghEpIrwACjFeLQ/Ajulr\nso8uBtjRkcfGEvRM/TAXw8HaOFvjqermobp573PYtlNXLfbQ4ddI\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFujCCA6KgAwIBAgIJALtAHEP1Xk+wMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV\nBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxHzAdBgNVBAMTFlN3aXNzU2ln\nbiBHb2xkIENBIC0gRzIwHhcNMDYxMDI1MDgzMDM1WhcNMzYxMDI1MDgzMDM1WjBF\nMQswCQYDVQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMR8wHQYDVQQDExZT\nd2lzc1NpZ24gR29sZCBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC\nCgKCAgEAr+TufoskDhJuqVAtFkQ7kpJcyrhdhJJCEyq8ZVeCQD5XJM1QiyUqt2/8\n76LQwB8CJEoTlo8jE+YoWACjR8cGp4QjK7u9lit/VcyLwVcfDmJlD909Vopz2q5+\nbbqBHH5CjCA12UNNhPqE21Is8w4ndwtrvxEvcnifLtg+5hg3Wipy+dpikJKVyh+c\n6bM8K8vzARO/Ws/BtQpgvd21mWRTuKCWs2/iJneRjOBiEAKfNA+k1ZIzUd6+jbqE\nemA8atufK+ze3gE/bk3lUIbLtK/tREDFylqM2tIrfKjuvqblCqoOpd8FUrdVxyJd\nMmqXl2MT28nbeTZ7hTpKxVKJ+STnnXepgv9VHKVxaSvRAiTysybUa9oEVeXBCsdt\nMDeQKuSeFDNeFhdVxVu1yzSJkvGdJo+hB9TGsnhQ2wwMC3wLjEHXuendjIj3o02y\nMszYF9rNt85mndT9Xv+9lz4pded+p2JYryU0pUHHPbwNUMoDAw8IWh+Vc3hiv69y\nFGkOpeUDDniOJihC8AcLYiAQZzlG+qkDzAQ4embvIIO1jEpWjpEA/I5cgt6IoMPi\naG59je883WX0XaxR7ySArqpWl2/5rX3aYT+YdzylkbYcjCbaZaIJbcHiVOO5ykxM\ngI93e2CaHt+28kgeDrpOVG2Y4OGiGqJ3UM/EY5LsRxmd6+ZrzsECAwEAAaOBrDCB\nqTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUWyV7\nlqRlUX64OfPAeGZe6Drn8O4wHwYDVR0jBBgwFoAUWyV7lqRlUX64OfPAeGZe6Drn\n8O4wRgYDVR0gBD8wPTA7BglghXQBWQECAQEwLjAsBggrBgEFBQcCARYgaHR0cDov\nL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZIhvcNAQEFBQADggIBACe6\n45R88a7A3hfm5djV9VSwg/S7zV4Fe0+fdWavPOhWfvxyeDgD2StiGwC5+OlgzczO\nUYrHUDFu4Up+GC9pWbY9ZIEr44OE5iKHjn3g7gKZYbge9LgriBIWhMIxkziWMaa5\nO1M/wySTVltpkuzFwbs4AOPsF6m43Md8AYOfMke6UiI0HTJ6CVanfCU2qT1L2sCC\nbwq7EsiHSycR+R4tx5M/nttfJmtS2S6K8RTGRI0Vqbe/vd6mGu6uLftIdxf+u+yv\nGPUqUfA5hJeVbG4bwyvEdGB5JbAKJ9/fXtI5z0V9QkvfsywexcZdylU6oJxpmo/a\n77KwPJ+HbBIrZXAVUjEaJM9vMSNQH4xPjyPDdEFjHFWoFN0+4FFQz/EbMFYOkrCC\nhdiDyyJkvC24JdVUorgG6q2SpCSgwYa1ShNqR88uC1aVVMvOmttqtKay20EIhid3\n92qgQmwLOM7XdVAyksLfKzAiSNDVQTglXaTpXZ/GlHXQRf0wl0OPkKsKx4ZzYEpp\nLd6leNcG2mqeSz53OiATIgHQv2ieY2BrNU0LbbqhPcCT4H8js1WtciVORvnSFu+w\nZMEBnunKoGqYDs/YYPIvSbjkQuE4NRb0yG5P94FW6LqjviOvrv1vA+ACOzB2+htt\nQc8Bsem4yWb02ybzOqR08kkkW8mw0FfB+j564ZfJ\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFvTCCA6WgAwIBAgIITxvUL1S7L0swDQYJKoZIhvcNAQEFBQAwRzELMAkGA1UE\nBhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEhMB8GA1UEAxMYU3dpc3NTaWdu\nIFNpbHZlciBDQSAtIEcyMB4XDTA2MTAyNTA4MzI0NloXDTM2MTAyNTA4MzI0Nlow\nRzELMAkGA1UEBhMCQ0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEhMB8GA1UEAxMY\nU3dpc3NTaWduIFNpbHZlciBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A\nMIICCgKCAgEAxPGHf9N4Mfc4yfjDmUO8x/e8N+dOcbpLj6VzHVxumK4DV644N0Mv\nFz0fyM5oEMF4rhkDKxD6LHmD9ui5aLlV8gREpzn5/ASLHvGiTSf5YXu6t+WiE7br\nYT7QbNHm+/pe7R20nqA1W6GSy/BJkv6FCgU+5tkL4k+73JU3/JHpMjUi0R86TieF\nnbAVlDLaYQ1HTWBCrpJH6INaUFjpiou5XaHc3ZlKHzZnu0jkg7Y360g6rw9njxcH\n6ATK72oxh9TAtvmUcXtnZLi2kUpCe2UuMGoM9ZDulebyzYLs2aFK7PayS+VFheZt\neJMELpyCbTapxDFkH4aDCyr0NQp4yVXPQbBH6TCfmb5hqAaEuSh6XzjZG6k4sIN/\nc8HDO0gqgg8hm7jMqDXDhBuDsz6+pJVpATqJAHgE2cn0mRmrVn5bi4Y5FZGkECwJ\nMoBgs5PAKrYYC51+jUnyEEp/+dVGLxmSo5mnJqy7jDzmDrxHB9xzUfFwZC8I+bRH\nHTBsROopN4WSaGa8gzj+ezku01DwH/teYLappvonQfGbGHLy9YR0SslnxFSuSGTf\njNFusB3hB48IHpmccelM2KX3RxIfdNFRnobzwqIjQAtz20um53MGjMGg6cFZrEb6\n5i/4z3GcRm25xBWNOHkDRUjvxF3XCO6HOSKGsg0PWEP3calILv3q1h8CAwEAAaOB\nrDCBqTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU\nF6DNweRBtjpbO8tFnb0cwpj6hlgwHwYDVR0jBBgwFoAUF6DNweRBtjpbO8tFnb0c\nwpj6hlgwRgYDVR0gBD8wPTA7BglghXQBWQEDAQEwLjAsBggrBgEFBQcCARYgaHR0\ncDovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS8wDQYJKoZIhvcNAQEFBQADggIB\nAHPGgeAn0i0P4JUw4ppBf1AsX19iYamGamkYDHRJ1l2E6kFSGG9YrVBWIGrGvShp\nWJHckRE1qTodvBqlYJ7YH39FkWnZfrt4csEGDyrOj4VwYaygzQu4OSlWhDJOhrs9\nxCrZ1x9y7v5RoSJBsXECYxqCsGKrXlcSH9/L3XWgwF15kIwb4FDm3jH+mHtwX6WQ\n2K34ArZv02DdQEsixT2tOnqfGhpHkXkzuoLcMmkDlm4fS/Bx/uNncqCxv1yL5PqZ\nIseEuRuNI5c/7SXgz2W79WEE790eslpBIlqhn10s6FvJbakMDHiqYMZWjwFaDGi8\naRl5xB9+lwW/xekkUV7U1UtT7dkjWjYDZaPBA61BMPNGG4WQr2W11bHkFlt4dR2X\nem1ZqSqPe97Dh4kQmUlzeMg9vVE1dCrV8X5pGyq7O70luJpaPXJhkGaH7gzWTdQR\ndAtq/gsD/KNVV4n+SsuuWxcFyPKNIzFTONItaj+CuY0IavdeQXRuwxF+B6wpYJE/\nOMpXEA29MC/HpeZBoNquBYeaoKRlbEwJDIm6uNO5wJOKMPqN5ZprFQFOZ6raYlY+\nhAhm0sQ2fac+EPyI4NSA5QC9qvNOBqN6avlicuMJT+ubDgEj8Z+7fNzcbBGXJbLy\ntGMU0gYqZ4yD9c7qB9iaah7s5Aq7KkzrCWA5zspi2C5u\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF2TCCA8GgAwIBAgIQXAuFXAvnWUHfV8w/f52oNjANBgkqhkiG9w0BAQUFADBk\nMQswCQYDVQQGEwJjaDERMA8GA1UEChMIU3dpc3Njb20xJTAjBgNVBAsTHERpZ2l0\nYWwgQ2VydGlmaWNhdGUgU2VydmljZXMxGzAZBgNVBAMTElN3aXNzY29tIFJvb3Qg\nQ0EgMTAeFw0wNTA4MTgxMjA2MjBaFw0yNTA4MTgyMjA2MjBaMGQxCzAJBgNVBAYT\nAmNoMREwDwYDVQQKEwhTd2lzc2NvbTElMCMGA1UECxMcRGlnaXRhbCBDZXJ0aWZp\nY2F0ZSBTZXJ2aWNlczEbMBkGA1UEAxMSU3dpc3Njb20gUm9vdCBDQSAxMIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0LmwqAzZuz8h+BvVM5OAFmUgdbI9\nm2BtRsiMMW8Xw/qabFbtPMWRV8PNq5ZJkCoZSx6jbVfd8StiKHVFXqrWW/oLJdih\nFvkcxC7mlSpnzNApbjyFNDhhSbEAn9Y6cV9Nbc5fuankiX9qUvrKm/LcqfmdmUc/\nTilftKaNXXsLmREDA/7n29uj/x2lzZAeAR81sH8A25Bvxn570e56eqeqDFdvpG3F\nEzuwpdntMhy0XmeLVNxzh+XTF3xmUHJd1BpYwdnP2IkCb6dJtDZd0KTeByy2dbco\nkdaXvij1mB7qWybJvbCXc9qukSbraMH5ORXWZ0sKbU/Lz7DkQnGMU3nn7uHbHaBu\nHYwadzVcFh4rUx80i9Fs/PJnB3r1re3WmquhsUvhzDdf/X/NTa64H5xD+SpYVUNF\nvJbNcA78yeNmuk6NO4HLFWR7uZToXTNShXEuT46iBhFRyePLoW4xCGQMwtI89Tbo\n19AOeCMgkckkKmUpWyL3Ic6DXqTz3kvTaI9GdVyDCW4pa8RwjPWd1yAv/0bSKzjC\nL3UcPX7ape8eYIVpQtPM+GP+HkM5haa2Y0EQs3MevNP6yn0WR+Kn1dCjigoIlmJW\nbjTb2QK5MHXjBNLnj8KwEUAKrNVxAmKLMb7dxiNYMUJDLXT5xp6mig/p/r+D5kNX\nJLrvRjSq1xIBOO0CAwEAAaOBhjCBgzAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0hBBYw\nFDASBgdghXQBUwABBgdghXQBUwABMBIGA1UdEwEB/wQIMAYBAf8CAQcwHwYDVR0j\nBBgwFoAUAyUv3m+CATpcLNwroWm1Z9SM0/0wHQYDVR0OBBYEFAMlL95vggE6XCzc\nK6FptWfUjNP9MA0GCSqGSIb3DQEBBQUAA4ICAQA1EMvspgQNDQ/NwNurqPKIlwzf\nky9NfEBWMXrrpA9gzXrzvsMnjgM+pN0S734edAY8PzHyHHuRMSG08NBsl9Tpl7Ik\nVh5WwzW9iAUPWxAaZOHHgjD5Mq2eUCzneAXQMbFamIp1TpBcahQq4FJHgmDmHtqB\nsfsUC1rxn9KVuj7QG9YVHaO+htXbD8BJZLsuUBlL0iT43R4HVtA4oJVwIHaM190e\n3p9xxCPvgxNcoyQVTSlAPGrEqdi3pkSlDfTgnXceQHAm/NrZNuR55LU/vJtlvrsR\nls/bxig5OgjOR1tTWsWZ/l2p3e9M1MalrQLmjAcSHm8D0W+go/MpvRLHUKKwf4ip\nmXeascClOS5cfGniLLDqN2qk4Vrh9VDlg++luyqI54zb/W1elxmofmZ1a3Hqv7HH\nb6D0jqTsNFFbjCYDcKF31QESVwA12yPeDooomf2xEG9L/zgtYE4snOtnta1J7ksf\nrK/7DZBaZmBwXarNeNQk7shBoJMBkpxqnvy5JMWzFYJ+vq6VK+uxwNrjAWALXmms\nhFZhvnEX/h0TD/7Gh0Xp/jKgGg0TpJRVcaUWi7rKibCyx/yP2FS1k2Kdzs9Z+z0Y\nzirLNRWCXf9UIltxUvu3yf5gmwBBZPCqKuy2QkPOiWaByIufOVQDJdMWNY6E0F/6\nMBr1mmz0DlP5OlvRHA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF2TCCA8GgAwIBAgIQHp4o6Ejy5e/DfEoeWhhntjANBgkqhkiG9w0BAQsFADBk\nMQswCQYDVQQGEwJjaDERMA8GA1UEChMIU3dpc3Njb20xJTAjBgNVBAsTHERpZ2l0\nYWwgQ2VydGlmaWNhdGUgU2VydmljZXMxGzAZBgNVBAMTElN3aXNzY29tIFJvb3Qg\nQ0EgMjAeFw0xMTA2MjQwODM4MTRaFw0zMTA2MjUwNzM4MTRaMGQxCzAJBgNVBAYT\nAmNoMREwDwYDVQQKEwhTd2lzc2NvbTElMCMGA1UECxMcRGlnaXRhbCBDZXJ0aWZp\nY2F0ZSBTZXJ2aWNlczEbMBkGA1UEAxMSU3dpc3Njb20gUm9vdCBDQSAyMIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAlUJOhJ1R5tMJ6HJaI2nbeHCOFvEr\njw0DzpPMLgAIe6szjPTpQOYXTKueuEcUMncy3SgM3hhLX3af+Dk7/E6J2HzFZ++r\n0rk0X2s682Q2zsKwzxNoysjL67XiPS4h3+os1OD5cJZM/2pYmLcX5BtS5X4HAB1f\n2uY+lQS3aYg5oUFgJWFLlTloYhyxCwWJwDaCFCE/rtuh/bxvHGCGtlOUSbkrRsVP\nACu/obvLP+DHVxxX6NZp+MEkUp2IVd3Chy50I9AU/SpHWrumnf2U5NGKpV+GY3aF\ny6//SSj8gO1MedK75MDvAe5QQQg1I3ArqRa0jG6F6bYRzzHdUyYb3y1aSgJA/MTA\ntukxGggo5WDDH8SQjhBiYEQN7Aq+VRhxLKX0srwVYv8c474d2h5Xszx+zYIdkeNL\n6yxSNLCK/RJOlrDrcH+eOfdmQrGrrFLadkBXeyq96G4DsguAhYidDMfCd7Camlf0\nuPoTXGiTOmekl9AbmbeGMktg2M7v0Ax/lZ9vh0+Hio5fCHyqW/xavqGRn1V9TrAL\nacywlKinh/LTSlDcX3KwFnUey7QYYpqwpzmqm59m2I2mbJYV4+by+PGDYmy7Velh\nk6M99bFXi08jsJvllGov34zflVEpYKELKeRcVVi3qPyZ7iVNTA6z00yPhOgpD/0Q\nVAKFyPnlw4vP5w8CAwEAAaOBhjCBgzAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0hBBYw\nFDASBgdghXQBUwIBBgdghXQBUwIBMBIGA1UdEwEB/wQIMAYBAf8CAQcwHQYDVR0O\nBBYEFE0mICKJS9PVpAqhb97iEoHF8TwuMB8GA1UdIwQYMBaAFE0mICKJS9PVpAqh\nb97iEoHF8TwuMA0GCSqGSIb3DQEBCwUAA4ICAQAyCrKkG8t9voJXiblqf/P0wS4R\nfbgZPnm3qKhyN2abGu2sEzsOv2LwnN+ee6FTSA5BesogpxcbtnjsQJHzQq0Qw1zv\n/2BZf82Fo4s9SBwlAjxnffUy6S8w5X2lejjQ82YqZh6NM4OKb3xuqFp1mrjX2lhI\nREeoTPpMSQpKwhI3qEAMw8jh0FcNlzKVxzqfl9NX+Ave5XLzo9v/tdhZsnPdTSpx\nsrpJ9csc1fV5yJmz/MFMdOO0vSk3FQQoHt5FRnDsr7p4DooqzgB53MBfGWcsa0vv\naGgLQ+OswWIJ76bdZWGgr4RVSJFSHMYlkSrQwSIjYVmvRRGFHQEkNI/Ps/8XciAT\nwoCqISxxOQ7Qj1zB09GOInJGTB2Wrk9xseEFKZZZ9LuedT3PDTcNYtsmjGOpI99n\nBjx8Oto0QuFmtEYE3saWmA9LSHokMnWRn6z3aOkquVVlzl1h0ydw2Df+n7mvoC5W\nt6NlUe07qxS/TFED6F+KBZvuim6c779o+sjaC+NCydAXFJy3SuCvkychVSa1ZC+N\n8f+mQAWFBVzKBxlcCxMoTFh/wqXvRdpg065lYZ1Tg3TCrvJcwhbtkj6EPnNgiLx2\n9CzP0H1907he0ZESEOnN3col49XtmS++dYFLJPlFRpTJKSFTnCZFqhMX5OfNeOI5\nwSsSnqaeG8XmDtkx2Q==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF4DCCA8igAwIBAgIRAPL6ZOJ0Y9ON/RAdBB92ylgwDQYJKoZIhvcNAQELBQAw\nZzELMAkGA1UEBhMCY2gxETAPBgNVBAoTCFN3aXNzY29tMSUwIwYDVQQLExxEaWdp\ndGFsIENlcnRpZmljYXRlIFNlcnZpY2VzMR4wHAYDVQQDExVTd2lzc2NvbSBSb290\nIEVWIENBIDIwHhcNMTEwNjI0MDk0NTA4WhcNMzEwNjI1MDg0NTA4WjBnMQswCQYD\nVQQGEwJjaDERMA8GA1UEChMIU3dpc3Njb20xJTAjBgNVBAsTHERpZ2l0YWwgQ2Vy\ndGlmaWNhdGUgU2VydmljZXMxHjAcBgNVBAMTFVN3aXNzY29tIFJvb3QgRVYgQ0Eg\nMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMT3HS9X6lds93BdY7Bx\nUglgRCgzo3pOCvrY6myLURYaVa5UJsTMRQdBTxB5f3HSek4/OE6zAMaVylvNwSqD\n1ycfMQ4jFrclyxy0uYAyXhqdk/HoPGAsp15XGVhRXrwsVgu42O+LgrQ8uMIkqBPH\noCE2G3pXKSinLr9xJZDzRINpUKTk4RtiGZQJo/PDvO/0vezbE53PnUgJUmfANykR\nHvvSEaeFGHR55E+FFOtSN+KxRdjMDUN/rhPSays/p8LiqG12W0OfvrSdsyaGOx9/\n5fLoZigWJdBLlzin5M8J0TbDC77aO0RYjb7xnglrPvMyxyuHxuxenPaHZa0zKcQv\nidm5y8kDnftslFGXEBuGCxobP/YCfnvUxVFkKJ3106yDgYjTdLRZncHrYTNaRdHL\nOdAGalNgHa/2+2m8atwBz735j9m9W8E6X47aD0upm50qKGsaCnw8qyIL5XctcfaC\nNYGu+HuB5ur+rPQam3Rc6I8k9l2dRsQs0h4rIWqDJ2dVSqTjyDKXZpBy2uPUZC5f\n46Fq9mDU5zXNysRojddxyNMkM3OxbPlq4SjbX8Y96L5V5jcb7STZDxmPX2MYWFCB\nUWVv8p9+agTnNCRxunZLWB4ZvRVgRaoMEkABnRDixzgHcgplwLa7JSnaFp6LNYth\n7eVxV4O1PHGf40+/fh6Bn0GXAgMBAAGjgYYwgYMwDgYDVR0PAQH/BAQDAgGGMB0G\nA1UdIQQWMBQwEgYHYIV0AVMCAgYHYIV0AVMCAjASBgNVHRMBAf8ECDAGAQH/AgED\nMB0GA1UdDgQWBBRF2aWBbj2ITY1x0kbBbkUe88SAnTAfBgNVHSMEGDAWgBRF2aWB\nbj2ITY1x0kbBbkUe88SAnTANBgkqhkiG9w0BAQsFAAOCAgEAlDpzBp9SSzBc1P6x\nXCX5145v9Ydkn+0UjrgEjihLj6p7jjm02Vj2e6E1CqGdivdj5eu9OYLU43otb98T\nPLr+flaYC/NUn81ETm484T4VvwYmneTwkLbUwp4wLh/vx3rEUMfqe9pQy3omywC0\nWqu1kx+AiYQElY2NfwmTv9SoqORjbdlk5LgpWgi/UOGED1V7XwgiG/W9mR4U9s70\nWBCCswo9GcG/W6uqmdjyMb3lOGbcWAXH7WMaLgqXfIeTK7KK4/HsGOV1timH59yL\nGn602MnTihdsfSlEvoqq9X46Lmgxk7lq2prg2+kupYTNHAq4Sgj5nPFhJpiTt3tm\n7JFe3VE/23MPrQRYCd0EApUKPtN236YQHoA96M2kZNEzx5LH4k5E4wnJTsJdhw4S\nnr8PyQUQ3nqjsTzyP6WqJ3mtMX0f/fwZacXduT98zca0wjAefm6S139hdlqP65VN\nvBFuIXxZN5nQBrz5Bm0yFqXZaajh3DyAHmBR3NdUIR7KYndP+tiPsys6DXhyyWhB\nWkdKwqPrGtcKqzwyVcgKEZzfdNbwQBUdyLmPtTbFr/giuMod89a2GQ+fYWVq6nTI\nfI/DT11lgh/ZDYnadXL77/FHZxOzyNEZiCcmmpl5fx7kLD977vHeTYuWl8PVP3wb\nI+2ksx0WckNLIOFZfsLorSa/ovc=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDwzCCAqugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgjELMAkGA1UEBhMCREUx\nKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAd\nBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50ZXIxJTAjBgNVBAMMHFQtVGVsZVNl\nYyBHbG9iYWxSb290IENsYXNzIDIwHhcNMDgxMDAxMTA0MDE0WhcNMzMxMDAxMjM1\nOTU5WjCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnBy\naXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50\nZXIxJTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDIwggEiMA0G\nCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqX9obX+hzkeXaXPSi5kfl82hVYAUd\nAqSzm1nzHoqvNK38DcLZSBnuaY/JIPwhqgcZ7bBcrGXHX+0CfHt8LRvWurmAwhiC\nFoT6ZrAIxlQjgeTNuUk/9k9uN0goOA/FvudocP05l03Sx5iRUKrERLMjfTlH6VJi\n1hKTXrcxlkIF+3anHqP1wvzpesVsqXFP6st4vGCvx9702cu+fjOlbpSD8DT6Iavq\njnKgP6TeMFvvhk1qlVtDRKgQFRzlAVfFmPHmBiiRqiDFt1MmUUOyCxGVWOHAD3bZ\nwI18gfNycJ5v/hqO2V81xrJvNHy+SE/iWjnX2J14np+GPgNeGYtEotXHAgMBAAGj\nQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBS/\nWSA2AHmgoCJrjNXyYdK4LMuCSjANBgkqhkiG9w0BAQsFAAOCAQEAMQOiYQsfdOhy\nNsZt+U2e+iKo4YFWz827n+qrkRk4r6p8FU3ztqONpfSO9kSpp+ghla0+AGIWiPAC\nuvxhI+YzmzB6azZie60EI4RYZeLbK4rnJVM3YlNfvNoBYimipidx5joifsFvHZVw\nIEoHNN/q/xWA5brXethbdXwFeilHfkCoMRN3zUA7tFFHei4R40cR3p1m0IvVVGb6\ng1XqfMIpiRvpb7PO4gWEyS8+eIVibslfwXhjdFjASBgMmTnrpMwatXlajRWc2BQN\n9noHV8cigwUtPJslJj0Ys6lDfMjIq2SPDqO/nBudMNva0Bkuqjzx+zOAduTNrRlP\nBSeOE6Fuwg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDwzCCAqugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgjELMAkGA1UEBhMCREUx\nKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAd\nBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50ZXIxJTAjBgNVBAMMHFQtVGVsZVNl\nYyBHbG9iYWxSb290IENsYXNzIDMwHhcNMDgxMDAxMTAyOTU2WhcNMzMxMDAxMjM1\nOTU5WjCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnBy\naXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50\nZXIxJTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDMwggEiMA0G\nCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC9dZPwYiJvJK7genasfb3ZJNW4t/zN\n8ELg63iIVl6bmlQdTQyK9tPPcPRStdiTBONGhnFBSivwKixVA9ZIw+A5OO3yXDw/\nRLyTPWGrTs0NvvAgJ1gORH8EGoel15YUNpDQSXuhdfsaa3Ox+M6pCSzyU9XDFES4\nhqX2iys52qMzVNn6chr3IhUciJFrf2blw2qAsCTz34ZFiP0Zf3WHHx+xGwpzJFu5\nZeAsVMhg02YXP+HMVDNzkQI6pn97djmiH5a2OK61yJN0HZ65tOVgnS9W0eDrXltM\nEnAMbEQgqxHY9Bn20pxSN+f6tsIxO0rUFJmtxxr1XV/6B7h8DR/Wgx6zAgMBAAGj\nQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBS1\nA/d2O2GCahKqGFPrAyGUv/7OyjANBgkqhkiG9w0BAQsFAAOCAQEAVj3vlNW92nOy\nWL6ukK2YJ5f+AbGwUgC4TeQbIXQbfsDuXmkqJa9c1h3a0nnJ85cp4IaH3gRZD/FZ\n1GSFS5mvJQQeyUapl96Cshtwn5z2r3Ex3XsFpSzTucpH9sry9uetuUg/vBa3wW30\n6gmv7PO15wWeph6KU1HWk4HMdJP2udqmJQV0eVp+QD6CSyYRMG7hP0HHRwA11fXT\n91Q+gT3aSWqas+8QPebrb9HIIkfLzM8BMZLZGOMivgkeGj5asuRrDFR6fUNOuIml\ne9eiPZaGzPImNC1qkp2aGtAw4l1OBLBfiyB+d8E9lYLRRpo7PHi4b6HQDWSieB4p\nTpPDpFQUWw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEqjCCA5KgAwIBAgIOLmoAAQACH9dSISwRXDswDQYJKoZIhvcNAQEFBQAwdjEL\nMAkGA1UEBhMCREUxHDAaBgNVBAoTE1RDIFRydXN0Q2VudGVyIEdtYkgxIjAgBgNV\nBAsTGVRDIFRydXN0Q2VudGVyIENsYXNzIDIgQ0ExJTAjBgNVBAMTHFRDIFRydXN0\nQ2VudGVyIENsYXNzIDIgQ0EgSUkwHhcNMDYwMTEyMTQzODQzWhcNMjUxMjMxMjI1\nOTU5WjB2MQswCQYDVQQGEwJERTEcMBoGA1UEChMTVEMgVHJ1c3RDZW50ZXIgR21i\nSDEiMCAGA1UECxMZVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMiBDQTElMCMGA1UEAxMc\nVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMiBDQSBJSTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAKuAh5uO8MN8h9foJIIRszzdQ2Lu+MNF2ujhoF/RKrLqk2jf\ntMjWQ+nEdVl//OEd+DFwIxuInie5e/060smp6RQvkL4DUsFJzfb95AhmC1eKokKg\nuNV/aVyQMrKXDcpK3EY+AlWJU+MaWss2xgdW94zPEfRMuzBwBJWl9jmM/XOBCH2J\nXjIeIqkiRUuwZi4wzJ9l/fzLganx4Duvo4bRierERXlQXa7pIXSSTYtZgo+U4+lK\n8edJsBTj9WLL1XK9H7nSn6DNqPoByNkN39r8R52zyFTfSUrxIan+GE7uSNQZu+99\n5OKdy1u2bv/jzVrndIIFuoAlOMvkaZ6vQaoahPUCAwEAAaOCATQwggEwMA8GA1Ud\nEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBTjq1RMgKHbVkO3\nkUrL84J6E1wIqzCB7QYDVR0fBIHlMIHiMIHfoIHcoIHZhjVodHRwOi8vd3d3LnRy\ndXN0Y2VudGVyLmRlL2NybC92Mi90Y19jbGFzc18yX2NhX0lJLmNybIaBn2xkYXA6\nLy93d3cudHJ1c3RjZW50ZXIuZGUvQ049VEMlMjBUcnVzdENlbnRlciUyMENsYXNz\nJTIwMiUyMENBJTIwSUksTz1UQyUyMFRydXN0Q2VudGVyJTIwR21iSCxPVT1yb290\nY2VydHMsREM9dHJ1c3RjZW50ZXIsREM9ZGU/Y2VydGlmaWNhdGVSZXZvY2F0aW9u\nTGlzdD9iYXNlPzANBgkqhkiG9w0BAQUFAAOCAQEAjNfffu4bgBCzg/XbEeprS6iS\nGNn3Bzn1LL4GdXpoUxUc6krtXvwjshOg0wn/9vYua0Fxec3ibf2uWWuFHbhOIprt\nZjluS5TmVfwLG4t3wVMTZonZKNaL80VKY7f9ewthXbhtvsPcW3nS7Yblok2+XnR8\nau0WOB9/WIFaGusyiC2y8zl3gK9etmF1KdsjTYjKUCjLhdLTEKJZbtOTVAB6okaV\nhgWcqRmY5TFyDADiZ9lA4CQze28suVyrZZ0srHbqNZn1l7kPJOzHdiEoZa5X6AeI\ndUpWoNIFOqTmjZKILPPy4cHGYdtBxceb9w4aUUXCYWvcZCcXjFq32nQozZfkvQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEqjCCA5KgAwIBAgIOSkcAAQAC5aBd1j8AUb8wDQYJKoZIhvcNAQEFBQAwdjEL\nMAkGA1UEBhMCREUxHDAaBgNVBAoTE1RDIFRydXN0Q2VudGVyIEdtYkgxIjAgBgNV\nBAsTGVRDIFRydXN0Q2VudGVyIENsYXNzIDMgQ0ExJTAjBgNVBAMTHFRDIFRydXN0\nQ2VudGVyIENsYXNzIDMgQ0EgSUkwHhcNMDYwMTEyMTQ0MTU3WhcNMjUxMjMxMjI1\nOTU5WjB2MQswCQYDVQQGEwJERTEcMBoGA1UEChMTVEMgVHJ1c3RDZW50ZXIgR21i\nSDEiMCAGA1UECxMZVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMyBDQTElMCMGA1UEAxMc\nVEMgVHJ1c3RDZW50ZXIgQ2xhc3MgMyBDQSBJSTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBALTgu1G7OVyLBMVMeRwjhjEQY0NVJz/GRcekPewJDRoeIMJW\nHt4bNwcwIi9v8Qbxq63WyKthoy9DxLCyLfzDlml7forkzMA5EpBCYMnMNWju2l+Q\nVl/NHE1bWEnrDgFPZPosPIlY2C8u4rBo6SI7dYnWRBpl8huXJh0obazovVkdKyT2\n1oQDZogkAHhg8fir/gKya/si+zXmFtGt9i4S5Po1auUZuV3bOx4a+9P/FRQI2Alq\nukWdFHlgfa9Aigdzs5OW03Q0jTo3Kd5c7PXuLjHCINy+8U9/I1LZW+Jk2ZyqBwi1\nRb3R0DHBq1SfqdLDYmAD8bs5SpJKPQq5ncWg/jcCAwEAAaOCATQwggEwMA8GA1Ud\nEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBTUovyfs8PYA9NX\nXAek0CSnwPIA1DCB7QYDVR0fBIHlMIHiMIHfoIHcoIHZhjVodHRwOi8vd3d3LnRy\ndXN0Y2VudGVyLmRlL2NybC92Mi90Y19jbGFzc18zX2NhX0lJLmNybIaBn2xkYXA6\nLy93d3cudHJ1c3RjZW50ZXIuZGUvQ049VEMlMjBUcnVzdENlbnRlciUyMENsYXNz\nJTIwMyUyMENBJTIwSUksTz1UQyUyMFRydXN0Q2VudGVyJTIwR21iSCxPVT1yb290\nY2VydHMsREM9dHJ1c3RjZW50ZXIsREM9ZGU/Y2VydGlmaWNhdGVSZXZvY2F0aW9u\nTGlzdD9iYXNlPzANBgkqhkiG9w0BAQUFAAOCAQEANmDkcPcGIEPZIxpC8vijsrlN\nirTzwppVMXzEO2eatN9NDoqTSheLG43KieHPOh6sHfGcMrSOWXaiQYUlN6AT0PV8\nTtXqluJucsG7Kv5sbviRmEb8yRtXW+rIGjs/sFGYPAfaLFkB2otE6OF0/ado3VS6\ng0bsyEa1+K+XwDsJHI/OcpY9M1ZwvJbL2NV9IJqDnxrcOfHFcqMRA/07QlIp2+gB\n95tejNaNhk4Z+rwcvsUhpYeeeC422wlxo3I0+GzjBgnyXlal092Y+tTmBvTwtiBj\nS+opvaqCZh77gaqnN60TGOaSw4HBM7uIHqHn4rS9MWwOUT1v+5ZWgOI2F9Hc5A==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID3TCCAsWgAwIBAgIOHaIAAQAC7LdggHiNtgYwDQYJKoZIhvcNAQEFBQAweTEL\nMAkGA1UEBhMCREUxHDAaBgNVBAoTE1RDIFRydXN0Q2VudGVyIEdtYkgxJDAiBgNV\nBAsTG1RDIFRydXN0Q2VudGVyIFVuaXZlcnNhbCBDQTEmMCQGA1UEAxMdVEMgVHJ1\nc3RDZW50ZXIgVW5pdmVyc2FsIENBIEkwHhcNMDYwMzIyMTU1NDI4WhcNMjUxMjMx\nMjI1OTU5WjB5MQswCQYDVQQGEwJERTEcMBoGA1UEChMTVEMgVHJ1c3RDZW50ZXIg\nR21iSDEkMCIGA1UECxMbVEMgVHJ1c3RDZW50ZXIgVW5pdmVyc2FsIENBMSYwJAYD\nVQQDEx1UQyBUcnVzdENlbnRlciBVbml2ZXJzYWwgQ0EgSTCCASIwDQYJKoZIhvcN\nAQEBBQADggEPADCCAQoCggEBAKR3I5ZEr5D0MacQ9CaHnPM42Q9e3s9B6DGtxnSR\nJJZ4Hgmgm5qVSkr1YnwCqMqs+1oEdjneX/H5s7/zA1hV0qq34wQi0fiU2iIIAI3T\nfCZdzHd55yx4Oagmcw6iXSVphU9VDprvxrlE4Vc93x9UIuVvZaozhDrzznq+VZeu\njRIPFDPiUHDDSYcTvFHe15gSWu86gzOSBnWLknwSaHtwag+1m7Z3W0hZneTvWq3z\nwZ7U10VOylY0Ibw+F1tvdwxIAUMpsN0/lm7mlaoMwCC2/T42J5zjXM9OgdwZu5GQ\nfezmlwQek8wiSdeXhrYTCjxDI3d+8NzmzSQfO4ObNDqDNOMCAwEAAaNjMGEwHwYD\nVR0jBBgwFoAUkqR1LKSevoFE63n8isWVpesQdXMwDwYDVR0TAQH/BAUwAwEB/zAO\nBgNVHQ8BAf8EBAMCAYYwHQYDVR0OBBYEFJKkdSyknr6BROt5/IrFlaXrEHVzMA0G\nCSqGSIb3DQEBBQUAA4IBAQAo0uCG1eb4e/CX3CJrO5UUVg8RMKWaTzqwOuAGy2X1\n7caXJ/4l8lfmXpWMPmRgFVp/Lw0BxbFg/UU1z/CyvwbZ71q+s2IhtNerNXxTPqYn\n8aEt2hojnczd7Dwtnic0XQ/CNnm8yUpiLe1r2X1BQ3y2qsrtYbE3ghUJGooWMNjs\nydZHcnhLEEYUjl8Or+zHL6sQ17bxbuyGssLoDZJz3KL0Dzq/YSMQiZxIQG5wALPT\nujdEWBF6AmqI8Dc08BnprNRlc/ZpjGSUOnmFKbAWKwyCPwacx/0QK54PLLae4xW/\n2TYcuiUaUj0a7CIMHOCkoj3w6DnPgcB77V0fb8XQC9eY\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIID+zCCAuOgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBtzE/MD0GA1UEAww2VMOc\nUktUUlVTVCBFbGVrdHJvbmlrIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sx\nc8SxMQswCQYDVQQGDAJUUjEPMA0GA1UEBwwGQU5LQVJBMVYwVAYDVQQKDE0oYykg\nMjAwNSBUw5xSS1RSVVNUIEJpbGdpIMSwbGV0acWfaW0gdmUgQmlsacWfaW0gR8O8\ndmVubGnEn2kgSGl6bWV0bGVyaSBBLsWeLjAeFw0wNTA1MTMxMDI3MTdaFw0xNTAz\nMjIxMDI3MTdaMIG3MT8wPQYDVQQDDDZUw5xSS1RSVVNUIEVsZWt0cm9uaWsgU2Vy\ndGlmaWthIEhpem1ldCBTYcSfbGF5xLFjxLFzxLExCzAJBgNVBAYMAlRSMQ8wDQYD\nVQQHDAZBTktBUkExVjBUBgNVBAoMTShjKSAyMDA1IFTDnFJLVFJVU1QgQmlsZ2kg\nxLBsZXRpxZ9pbSB2ZSBCaWxpxZ9pbSBHw7x2ZW5sacSfaSBIaXptZXRsZXJpIEEu\nxZ4uMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAylIF1mMD2Bxf3dJ7\nXfIMYGFbazt0K3gNfUW9InTojAPBxhEqPZW8qZSwu5GXyGl8hMW0kWxsE2qkVa2k\nheiVfrMArwDCBRj1cJ02i67L5BuBf5OI+2pVu32Fks66WJ/bMsW9Xe8iSi9BB35J\nYbOG7E6mQW6EvAPs9TscyB/C7qju6hJKjRTP8wrgUDn5CDX4EVmt5yLqS8oUBt5C\nurKZ8y1UiBAG6uEaPj1nH/vO+3yC6BFdSsG5FOpU2WabfIl9BJpiyelSPJ6c79L1\nJuTm5Rh8i27fbMx4W09ysstcP4wFjdFMjK2Sx+F4f2VsSQZQLJ4ywtdKxnWKWU51\nb0dewQIDAQABoxAwDjAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4IBAQAV\n9VX/N5aAWSGk/KEVTCD21F/aAyT8z5Aa9CEKmu46sWrv7/hg0Uw2ZkUd82YCdAR7\nkjCo3gp2D++Vbr3JN+YaDayJSFvMgzbC9UZcWYJWtNX+I7TYVBxEq8Sn5RTOPEFh\nfEPmzcSBCYsk+1Ql1haolgxnB2+zUEfjHCQo3SqYpGH+2+oSN7wBGjSFvW5P55Fy\nB0SFHljKVETd96y5y4khctuPwGkplyqjrhgjlxxBKot8KsF8kOipKMDTkcatKIdA\naLX/7KfS0zgYnNN9aV3wxqUeJBujR/xpB2jn5Jq07Q+hh4cCzofSSE7hvP/L8XKS\nRGQDJereW26fyfJOrN3H\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEPTCCAyWgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBvzE/MD0GA1UEAww2VMOc\nUktUUlVTVCBFbGVrdHJvbmlrIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sx\nc8SxMQswCQYDVQQGEwJUUjEPMA0GA1UEBwwGQW5rYXJhMV4wXAYDVQQKDFVUw5xS\nS1RSVVNUIEJpbGdpIMSwbGV0acWfaW0gdmUgQmlsacWfaW0gR8O8dmVubGnEn2kg\nSGl6bWV0bGVyaSBBLsWeLiAoYykgQXJhbMSxayAyMDA3MB4XDTA3MTIyNTE4Mzcx\nOVoXDTE3MTIyMjE4MzcxOVowgb8xPzA9BgNVBAMMNlTDnFJLVFJVU1QgRWxla3Ry\nb25payBTZXJ0aWZpa2EgSGl6bWV0IFNhxJ9sYXnEsWPEsXPEsTELMAkGA1UEBhMC\nVFIxDzANBgNVBAcMBkFua2FyYTFeMFwGA1UECgxVVMOcUktUUlVTVCBCaWxnaSDE\nsGxldGnFn2ltIHZlIEJpbGnFn2ltIEfDvHZlbmxpxJ9pIEhpem1ldGxlcmkgQS7F\nni4gKGMpIEFyYWzEsWsgMjAwNzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAKu3PgqMyKVYFeaK7yc9SrToJdPNM8Ig3BnuiD9NYvDdE3ePYakqtdTyuTFY\nKTsvP2qcb3N2Je40IIDu6rfwxArNK4aUyeNgsURSsloptJGXg9i3phQvKUmi8wUG\n+7RP2qFsmmaf8EMJyupyj+sA1zU511YXRxcw9L6/P8JorzZAwan0qafoEGsIiveG\nHtyaKhUG9qPw9ODHFNRRf8+0222vR5YXm3dx2KdxnSQM9pQ/hTEST7ruToK4uT6P\nIzdezKKqdfcYbwnTrqdUKDT74eA7YH2gvnmJhsifLfkKS8RQouf9eRbHegsYz85M\n733WB2+Y8a+xwXrXgTW4qhe04MsCAwEAAaNCMEAwHQYDVR0OBBYEFCnFkKslrxHk\nYb+j/4hhkeYO/pyBMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MA0G\nCSqGSIb3DQEBBQUAA4IBAQAQDdr4Ouwo0RSVgrESLFF6QSU2TJ/sPx+EnWVUXKgW\nAkD6bho3hO9ynYYKVZ1WKKxmLNA6VpM0ByWtCLCPyA8JWcqdmBzlVPi5RX9ql2+I\naE1KBiY3iAIOtsbWcpnOa3faYjGkVh+uX4132l32iPwa2Z61gfAyuOOI0JzzaqC5\nmxRZNTZPz/OOXl0XrRWV2N2y1RVuAE6zS89mlOTgzbUF2mNXi+WzqtvALhyQRNsa\nXRik7r4EW5nVcV9VZWRi1aKbBFmGyGJ353yCRWo9F7/snXUMrqNvWtMvmDb08PUZ\nqxFdyKbjKlhqQgnDvZImZjINXQhVdP+MmNAKpoRq0Tl9\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEPDCCAySgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBvjE/MD0GA1UEAww2VMOc\nUktUUlVTVCBFbGVrdHJvbmlrIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sx\nc8SxMQswCQYDVQQGEwJUUjEPMA0GA1UEBwwGQW5rYXJhMV0wWwYDVQQKDFRUw5xS\nS1RSVVNUIEJpbGdpIMSwbGV0acWfaW0gdmUgQmlsacWfaW0gR8O8dmVubGnEn2kg\nSGl6bWV0bGVyaSBBLsWeLiAoYykgS2FzxLFtIDIwMDUwHhcNMDUxMTA3MTAwNzU3\nWhcNMTUwOTE2MTAwNzU3WjCBvjE/MD0GA1UEAww2VMOcUktUUlVTVCBFbGVrdHJv\nbmlrIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sxc8SxMQswCQYDVQQGEwJU\nUjEPMA0GA1UEBwwGQW5rYXJhMV0wWwYDVQQKDFRUw5xSS1RSVVNUIEJpbGdpIMSw\nbGV0acWfaW0gdmUgQmlsacWfaW0gR8O8dmVubGnEn2kgSGl6bWV0bGVyaSBBLsWe\nLiAoYykgS2FzxLFtIDIwMDUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQCpNn7DkUNMwxmYCMjHWHtPFoylzkkBH3MOrHUTpvqeLCDe2JAOCtFp0if7qnef\nJ1Il4std2NiDUBd9irWCPwSOtNXwSadktx4uXyCcUHVPr+G1QRT0mJKIx+XlZEdh\nR3n9wFHxwZnn3M5q+6+1ATDcRhzviuyV79z/rxAc653YsKpqhRgNF8k+v/Gb0AmJ\nQv2gQrSdiVFVKc8bcLyEVK3BEx+Y9C52YItdP5qtygy/p1Zbj3e41Z55SZI/4PGX\nJHpsmxcPbe9TmJEr5A++WXkHeLuXlfSfadRYhwqp48y2WBmfJiGxxFmNskF1wK1p\nzpwACPI2/z7woQ8arBT9pmAPAgMBAAGjQzBBMB0GA1UdDgQWBBTZN7NOBf3Zz58S\nFq62iS/rJTqIHDAPBgNVHQ8BAf8EBQMDBwYAMA8GA1UdEwEB/wQFMAMBAf8wDQYJ\nKoZIhvcNAQEFBQADggEBAHJglrfJ3NgpXiOFX7KzLXb7iNcX/nttRbj2hWyfIvwq\nECLsqrkw9qtY1jkQMZkpAL2JZkH7dN6RwRgLn7Vhy506vvWolKMiVW4XSf/SKfE4\nJl3vpao6+XF75tpYHdN0wgH6PmlYX63LaL4ULptswLbcoCb6dxriJNoaN+BnrdFz\ngw2lGh1uEpJ+hGIAF728JRhX8tepb1mIvDS3LoV4nZbcFMMsilKbloxSZj2GFotH\nuFEJjOp9zYhys2AzsfAKRO8P9Qk3iCQOLGsgOqL6EfJANZxEaGM7rDNvY7wsu/LS\ny3Z9fYjYHcgFHW68lKlmjHdxx/qR+i9Rnuk5UrbnBEI=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFQTCCAymgAwIBAgICDL4wDQYJKoZIhvcNAQELBQAwUTELMAkGA1UEBhMCVFcx\nEjAQBgNVBAoTCVRBSVdBTi1DQTEQMA4GA1UECxMHUm9vdCBDQTEcMBoGA1UEAxMT\nVFdDQSBHbG9iYWwgUm9vdCBDQTAeFw0xMjA2MjcwNjI4MzNaFw0zMDEyMzExNTU5\nNTlaMFExCzAJBgNVBAYTAlRXMRIwEAYDVQQKEwlUQUlXQU4tQ0ExEDAOBgNVBAsT\nB1Jvb3QgQ0ExHDAaBgNVBAMTE1RXQ0EgR2xvYmFsIFJvb3QgQ0EwggIiMA0GCSqG\nSIb3DQEBAQUAA4ICDwAwggIKAoICAQCwBdvI64zEbooh745NnHEKH1Jw7W2CnJfF\n10xORUnLQEK1EjRsGcJ0pDFfhQKX7EMzClPSnIyOt7h52yvVavKOZsTuKwEHktSz\n0ALfUPZVr2YOy+BHYC8rMjk1Ujoog/h7FsYYuGLWRyWRzvAZEk2tY/XTP3VfKfCh\nMBwqoJimFb3u/Rk28OKRQ4/6ytYQJ0lM793B8YVwm8rqqFpD/G2Gb3PpN0Wp8DbH\nzIh1HrtsBv+baz4X7GGqcXzGHaL3SekVtTzWoWH1EfcFbx39Eb7QMAfCKbAJTibc\n46KokWofwpFFiFzlmLhxpRUZyXx1EcxwdE8tmx2RRP1WKKD+u4ZqyPpcC1jcxkt2\nyKsi2XMPpfRaAok/T54igu6idFMqPVMnaR1sjjIsZAAmY2E2TqNGtz99sy2sbZCi\nlaLOz9qC5wc0GZbpuCGqKX6mOL6OKUohZnkfs8O1CWfe1tQHRvMq2uYiN2DLgbYP\noA/pyJV/v1WRBXrPPRXAb94JlAGD1zQbzECl8LibZ9WYkTunhHiVJqRaCPgrdLQA\nBDzfuBSO6N+pjWxnkjMdwLfS7JLIvgm/LCkFbwJrnu+8vyq8W8BQj0FwcYeyTbcE\nqYSjMq+u7msXi7Kx/mzhkIyIqJdIzshNy/MGz19qCkKxHh53L46g5pIOBvwFItIm\n4TFRfTLcDwIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB\n/zANBgkqhkiG9w0BAQsFAAOCAgEAXzSBdu+WHdXltdkCY4QWwa6gcFGn90xHNcgL\n1yg9iXHZqjNB6hQbbCEAwGxCGX6faVsgQt+i0trEfJdLjbDorMjupWkEmQqSpqsn\nLhpNgb+E1HAerUf+/UqdM+DyucRFCCEK2mlpc3INvjT+lIutwx4116KD7+U4x6WF\nH6vPNOw/KP4M8VeGTslV9xzU2KV9Bnpv1d8Q34FOIWWxtuEXeZVFBs5fzNxGiWNo\nRI2T9GRwoD2dKAXDOXC4Ynsg/eTb6QihuJ49CcdP+yz4k3ZB3lLg4VfSnQO8d57+\nnile98FRYB/e2guyLXW3Q0iT5/Z5xoRdgFlglPx4mI88k1HtQJAH32RjJMtOcQWh\n15QaiDLxInQirqWm2BJpTGCjAu4r7NRjkgtevi92a6O2JryPA9gK8kxkRr05YuWW\n6zRjESjMlfGt7+/cgFhI6Uu46mWs6fyAtbXIRfmswZ/ZuepiiI7E8UuDEq3mi4TW\nnsLrgxifarsbJGAzcMzs9zLzXNl5fe+epP7JI8Mk7hWSsT2RTyaGvWZzJBPqpK5j\nwa19hAM8EHiGG3njxPPyBJUgriOCxLM6AGK/5jYk4Ve6xx6QddVfP5VhK8E7zeWz\naGHQRiapIVJpLesux+t3zqY6tQMzT3bR51xUAV3LePTJDL/PEo4XLSNolOer/qmy\nKwbQBM0=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDezCCAmOgAwIBAgIBATANBgkqhkiG9w0BAQUFADBfMQswCQYDVQQGEwJUVzES\nMBAGA1UECgwJVEFJV0FOLUNBMRAwDgYDVQQLDAdSb290IENBMSowKAYDVQQDDCFU\nV0NBIFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDgwODI4MDcyNDMz\nWhcNMzAxMjMxMTU1OTU5WjBfMQswCQYDVQQGEwJUVzESMBAGA1UECgwJVEFJV0FO\nLUNBMRAwDgYDVQQLDAdSb290IENBMSowKAYDVQQDDCFUV0NBIFJvb3QgQ2VydGlm\naWNhdGlvbiBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQCwfnK4pAOU5qfeCTiRShFAh6d8WWQUe7UREN3+v9XAu1bihSX0NXIP+FPQQeFE\nAcK0HMMxQhZHhTMidrIKbw/lJVBPhYa+v5guEGcevhEFhgWQxFnQfHgQsIBct+HH\nK3XLfJ+utdGdIzdjp9xCoi2SBBtQwXu4PhvJVgSLL1KbralW6cH/ralYhzC2gfeX\nRfwZVzsrb+RH9JlF/h3x+JejiB03HFyP4HYlmlD4oFT/RJB2I9IyxsOrBr/8+7/z\nrX2SYgJbKdM1o5OaQ2RgXbL6Mv87BK9NQGr5x+PvI/1ry+UPizgN7gr8/g+YnzAx\n3WxSZfmLgb4i4RxYA7qRG4kHAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRqOFsmjd6LWvJPelSDGRjjCDWmujANBgkq\nhkiG9w0BAQUFAAOCAQEAPNV3PdrfibqHDAhUaiBQkr6wQT25JmSDCi/oQMCXKCeC\nMErJk/9q56YAf4lCmtYR5VPOL8zy2gXE/uJQxDqGfczafhAJO5I1KlOy/usrBdls\nXebQ79NqZp4VKIV66IIArB6nCWlWQtNoURi+VJq/REG6Sb4gumlc7rh3zc5sH62D\nlhh9DrUUOYTxKOkto557HnpyWoOzeW/vtPzQCqVYT0bf+215WfKEIlKuD8z7fDvn\naspHYcN6+NOSBB+4IIThNlQWx0DeO4pz3N/GCUzf7Nr/1FNCocnyYh0igzyXxfkZ\nYiesZSLX0zzG5Y6yU8xJzrww/nsOM5D77dIUkR8Hrw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFcjCCA1qgAwIBAgIQH51ZWtcvwgZEpYAIaeNe9jANBgkqhkiG9w0BAQUFADA/\nMQswCQYDVQQGEwJUVzEwMC4GA1UECgwnR292ZXJubWVudCBSb290IENlcnRpZmlj\nYXRpb24gQXV0aG9yaXR5MB4XDTAyMTIwNTEzMjMzM1oXDTMyMTIwNTEzMjMzM1ow\nPzELMAkGA1UEBhMCVFcxMDAuBgNVBAoMJ0dvdmVybm1lbnQgUm9vdCBDZXJ0aWZp\nY2F0aW9uIEF1dGhvcml0eTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIB\nAJoluOzMonWoe/fOW1mKydGGEghU7Jzy50b2iPN86aXfTEc2pBsBHH8eV4qNw8XR\nIePaJD9IK/ufLqGU5ywck9G/GwGHU5nOp/UKIXZ3/6m3xnOUT0b3EEk3+qhZSV1q\ngQdW8or5BtD3cCJNtLdBuTK4sfCxw5w/cP1T3YGq2GN49thTbqGsaoQkclSGxtKy\nyhwOeYHWtXBiCAEuTk8O1RGvqa/lmr/czIdtJuTJV6L7lvnM4T9TjGxMfptTCAts\nF/tnyMKtsc2AtJfcdgEWFelq16TheEfOhtX7MfP6Mb40qij7cEwdScevLJ1tZqa2\njWR+tSBqnTuBto9AAGdLiYa4zGX+FVPpBMHWXx1E1wovJ5pGfaENda1UhhXcSTvx\nls4Pm6Dso3pdvtUqdULle96ltqqvKKyskKw4t9VoNSZ63Pc78/1Fm9G7Q3hub/FC\nVGqY8A2tl+lSXunVanLeavcbYBT0peS2cWeqH+riTcFCQP5nRhc4L0c/cZyu5SHK\nYS1tB6iEfC3uUSXxY5Ce/eFXiGvviiNtsea9P63RPZYLhY3Naye7twWb7LuRqQoH\nEgKXTiCQ8P8NHuJBO9NAOueNXdpm5AKwB1KYXA6OM5zCppX7VRluTI6uSw+9wThN\nXo+EHWbNxWCWtFJaBYmOlXqYwZE8lSOyDvR5tMl8wUohAgMBAAGjajBoMB0GA1Ud\nDgQWBBTMzO/MKWCkO7GStjz6MmKPrCUVOzAMBgNVHRMEBTADAQH/MDkGBGcqBwAE\nMTAvMC0CAQAwCQYFKw4DAhoFADAHBgVnKgMAAAQUA5vwIhP/lSg209yewDL7MTqK\nUWUwDQYJKoZIhvcNAQEFBQADggIBAECASvomyc5eMN1PhnR2WPWus4MzeKR6dBcZ\nTulStbngCnRiqmjKeKBMmo4sIy7VahIkv9Ro04rQ2JyftB8M3jh+Vzj8jeJPXgyf\nqzvS/3WXy6TjZwj/5cAWtUgBfen5Cv8b5Wppv3ghqMKnI6mGq3ZW6A4M9hPdKmaK\nZEk9GhiHkASfQlK3T8v+R0F2Ne//AHY2RTKbxkaFXeIksB7jSJaYV0eUVXoPQbFE\nJPPB/hprv4j9wabak2BegUqZIJxIZhm1AHlUD7gsL0u8qV1bYH+Mh6XgUmMqvtg7\nhUAV/h62ZT/FS9p+tXo1KaMuephgIqP0fSdOLeq0dDzpD6QzDxARvBMB1uUO07+1\nEqLhRSPAzAhuYbeJq4PjJB7mXQfnHyA+z2fI56wwbSdLaG5LKlwCCDTb+HbkZ6Mm\nnD+iMsJKxYEYMRBWqoTvLQr/uB930r+lWKBi5NdLkXWNiYCYfm3LU05er/ayl4WX\nudpVBrkk7tfGOB5jGxI7leFYrPLfhNVfmS8NVVvmONsuP3LpSIXLuykTjx44Vbnz\nssQwmSNOXfJIoRIM3BKQCZBUkQM8R+XVyWXgt0t97EfTsws+rZ7QdAAO671RrcDe\nLMDDav7v3Aun+kbfYNucpllQdSNpc5Oy+fwC00fmcc4QAu4njIT/rEUNE1yDMuAl\npYYsfPQS\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFODCCAyCgAwIBAgIRAJW+FqD3LkbxezmCcvqLzZYwDQYJKoZIhvcNAQEFBQAw\nNzEUMBIGA1UECgwLVGVsaWFTb25lcmExHzAdBgNVBAMMFlRlbGlhU29uZXJhIFJv\nb3QgQ0EgdjEwHhcNMDcxMDE4MTIwMDUwWhcNMzIxMDE4MTIwMDUwWjA3MRQwEgYD\nVQQKDAtUZWxpYVNvbmVyYTEfMB0GA1UEAwwWVGVsaWFTb25lcmEgUm9vdCBDQSB2\nMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMK+6yfwIaPzaSZVfp3F\nVRaRXP3vIb9TgHot0pGMYzHw7CTww6XScnwQbfQ3t+XmfHnqjLWCi65ItqwA3GV1\n7CpNX8GH9SBlK4GoRz6JI5UwFpB/6FcHSOcZrr9FZ7E3GwYq/t75rH2D+1665I+X\nZ75Ljo1kB1c4VWk0Nj0TSO9P4tNmHqTPGrdeNjPUtAa9GAH9d4RQAEX1jF3oI7x+\n/jXh7VB7qTCNGdMJjmhnXb88lxhTuylixcpecsHHltTbLaC0H2kD7OriUPEMPPCs\n81Mt8Bz17Ww5OXOAFshSsCPN4D7c3TxHoLs1iuKYaIu+5b9y7tL6pe0S7fyYGKkm\ndtwoSxAgHNN/Fnct7W+A90m7UwW7XWjH1Mh1Fj+JWov3F0fUTPHSiXk+TT2YqGHe\nOh7S+F4D4MHJHIzTjU3TlTazN19jY5szFPAtJmtTfImMMsJu7D0hADnJoWjiUIMu\nsDor8zagrC/kb2HCUQk5PotTubtn2txTuXZZNp1D5SDgPTJghSJRt8czu90VL6R4\npgd7gUY2BIbdeTXHlSw7sKMXNeVzH7RcWe/a6hBle3rQf5+ztCo3O3CLm1u5K7fs\nslESl1MpWtTwEhDcTwK7EpIvYtQ/aUN8Ddb8WHUBiJ1YFkveupD/RwGJBmr2X7KQ\narMCpgKIv7NHfirZ1fpoeDVNAgMBAAGjPzA9MA8GA1UdEwEB/wQFMAMBAf8wCwYD\nVR0PBAQDAgEGMB0GA1UdDgQWBBTwj1k4ALP1j5qWDNXr+nuqF+gTEjANBgkqhkiG\n9w0BAQUFAAOCAgEAvuRcYk4k9AwI//DTDGjkk0kiP0Qnb7tt3oNmzqjMDfz1mgbl\ndxSR651Be5kqhOX//CHBXfDkH1e3damhXwIm/9fH907eT/j3HEbAek9ALCI18Bmx\n0GtnLLCo4MBANzX2hFxc469CeP6nyQ1Q6g2EdvZR74NTxnr/DlZJLo961gzmJ1Tj\nTQpgcmLNkQfWpb/ImWvtxBnmq0wROMVvMeJuScg/doAmAyYp4Db29iBT4xdwNBed\nY2gea+zDTYa4EzAvXUYNR0PVG6pZDrlcjQZIrXSHX8f8MVRBE+LHIQ6e4B4N4cB7\nQ4WQxYpYxmUKeFfyxiMPAdkgS94P+5KFdSpcc41teyWRyu5FrgZLAMzTsVlQ2jqI\nOylDRl6XK1TOU2+NSueW+r9xDkKLfP0ooNBIytrEgUy7onOTJsjrDNYmiLbAJM+7\nvVvrdX3pCI6GMyx5dwlppYn8s3CQh3aP0yK7Qs69cwsgJirQmz1wHiRszYd2qReW\nt88NkvuOGKmYSdGe/mBEciG5Ge3C9THxOUiIkCR1VBatzvT4aRRkOfujuLpwQMcn\nHL/EVlP6Y2XQ8xwOFvVrhlhNGNTkDY6lnVuR3HYkUD/GKvvZt5y11ubQ2egZixVx\nSK236thZiNSQvxaz2emsWWFUyBy6ysHK4bkgTI86k4mloMy/0/Z1pHWWbVY=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDZzCCAk+gAwIBAgIQGx+ttiD5JNM2a/fH8YygWTANBgkqhkiG9w0BAQUFADBF\nMQswCQYDVQQGEwJHQjEYMBYGA1UEChMPVHJ1c3RpcyBMaW1pdGVkMRwwGgYDVQQL\nExNUcnVzdGlzIEZQUyBSb290IENBMB4XDTAzMTIyMzEyMTQwNloXDTI0MDEyMTEx\nMzY1NFowRTELMAkGA1UEBhMCR0IxGDAWBgNVBAoTD1RydXN0aXMgTGltaXRlZDEc\nMBoGA1UECxMTVHJ1c3RpcyBGUFMgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAMVQe547NdDfxIzNjpvto8A2mfRC6qc+gIMPpqdZh8mQRUN+\nAOqGeSoDvT03mYlmt+WKVoaTnGhLaASMk5MCPjDSNzoiYYkchU59j9WvezX2fihH\niTHcDnlkH5nSW7r+f2C/revnPDgpai/lkQtV/+xvWNUtyd5MZnGPDNcE2gfmHhjj\nvSkCqPoc4Vu5g6hBSLwacY3nYuUtsuvffM/bq1rKMfFMIvMFE/eC+XN5DL7XSxzA\n0RU8k0Fk0ea+IxciAIleH2ulrG6nS4zto3Lmr2NNL4XSFDWaLk6M6jKYKIahkQlB\nOrTh4/L68MkKokHdqeMDx4gVOxzUGpTXn2RZEm0CAwEAAaNTMFEwDwYDVR0TAQH/\nBAUwAwEB/zAfBgNVHSMEGDAWgBS6+nEleYtXQSUhhgtx67JkDoshZzAdBgNVHQ4E\nFgQUuvpxJXmLV0ElIYYLceuyZA6LIWcwDQYJKoZIhvcNAQEFBQADggEBAH5Y//01\nGX2cGE+esCu8jowU/yyg2kdbw++BLa8F6nRIW/M+TgfHbcWzk88iNVy2P3UnXwmW\nzaD+vkAMXBJV+JOCyinpXj9WV4s4NvdFGkwozZ5BuO1WTISkQMi4sKUraXAEasP4\n1BIy+Q7DsdwyhEQsb8tGD+pmQQ9P8Vilpg0ND2HepZ5dfWWhPBfnqFVO76DH7cZE\nf1T1o+CP8HxVIo8ptoGj4W1OLBuAZ+ytIJ8MYmHVl/9D7S3B2l0pKoU/rGXuhg8F\njZBf3+6f9L/uHfuY5H+QK4R4EA5sSVPvFVtlRkpdr7r7OnIdzfYliB6XzCGcKQEN\nZetX2fNXlrtIzYE=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFFzCCA/+gAwIBAgIBETANBgkqhkiG9w0BAQUFADCCASsxCzAJBgNVBAYTAlRS\nMRgwFgYDVQQHDA9HZWJ6ZSAtIEtvY2FlbGkxRzBFBgNVBAoMPlTDvHJraXllIEJp\nbGltc2VsIHZlIFRla25vbG9qaWsgQXJhxZ90xLFybWEgS3VydW11IC0gVMOcQsSw\nVEFLMUgwRgYDVQQLDD9VbHVzYWwgRWxla3Ryb25payB2ZSBLcmlwdG9sb2ppIEFy\nYcWfdMSxcm1hIEVuc3RpdMO8c8O8IC0gVUVLQUUxIzAhBgNVBAsMGkthbXUgU2Vy\ndGlmaWthc3lvbiBNZXJrZXppMUowSAYDVQQDDEFUw5xCxLBUQUsgVUVLQUUgS8O2\nayBTZXJ0aWZpa2EgSGl6bWV0IFNhxJ9sYXnEsWPEsXPEsSAtIFPDvHLDvG0gMzAe\nFw0wNzA4MjQxMTM3MDdaFw0xNzA4MjExMTM3MDdaMIIBKzELMAkGA1UEBhMCVFIx\nGDAWBgNVBAcMD0dlYnplIC0gS29jYWVsaTFHMEUGA1UECgw+VMO8cmtpeWUgQmls\naW1zZWwgdmUgVGVrbm9sb2ppayBBcmHFn3TEsXJtYSBLdXJ1bXUgLSBUw5xCxLBU\nQUsxSDBGBgNVBAsMP1VsdXNhbCBFbGVrdHJvbmlrIHZlIEtyaXB0b2xvamkgQXJh\nxZ90xLFybWEgRW5zdGl0w7xzw7wgLSBVRUtBRTEjMCEGA1UECwwaS2FtdSBTZXJ0\naWZpa2FzeW9uIE1lcmtlemkxSjBIBgNVBAMMQVTDnELEsFRBSyBVRUtBRSBLw7Zr\nIFNlcnRpZmlrYSBIaXptZXQgU2HEn2xhecSxY8Sxc8SxIC0gU8O8csO8bSAzMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAim1L/xCIOsP2fpTo6iBkcK4h\ngb46ezzb8R1Sf1n68yJMlaCQvEhOEav7t7WNeoMojCZG2E6VQIdhn8WebYGHV2yK\nO7Rm6sxA/OOqbLLLAdsyv9Lrhc+hDVXDWzhXcLh1xnnRFDDtG1hba+818qEhTsXO\nfJlfbLm4IpNQp81McGq+agV/E5wrHur+R84EpW+sky58K5+eeROR6Oqeyjh1jmKw\nlZMq5d/pXpduIF9fhHpEORlAHLpVK/swsoHvhOPc7Jg4OQOFCKlUAwUp8MmPi+oL\nhmUZEdPpCSPeaJMDyTYcIW7OjGbxmTDY17PDHfiBLqi9ggtm/oLL4eAagsNAgQID\nAQABo0IwQDAdBgNVHQ4EFgQUvYiHyY/2pAoLquvF/pEjnatKijIwDgYDVR0PAQH/\nBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEBAB18+kmP\nNOm3JpIWmgV050vQbTlswyb2zrgxvMTfvCr4N5EY3ATIZJkrGG2AA1nJrvhY0D7t\nwyOfaTyGOBye79oneNGEN3GKPEs5z35FBtYt2IpNeBLWrcLTy9LQQfMmNkqblWwM\n7uXRQydmwYj3erMgbOqwaSvHIOgMA8RBBZniP+Rr+KCGgceExh/VS4ESshYhLBOh\ngLJeDEoTniDYYkCrkOpkSi+sDQESeUWoL4cZaMjihccwsnX5OD+ywJO0a+IDRM5n\noN+J1q2MdqMTw5RhK2vZbMEHCiIHhWyFJEapvj+LeISCfiQMnf2BN+MlqO02TpUs\nyZyQ2uypQjyttgI=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICjzCCAhWgAwIBAgIQXIuZxVqUxdJxVt7NiYDMJjAKBggqhkjOPQQDAzCBiDEL\nMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNl\neSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMT\nJVVTRVJUcnVzdCBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAwMjAx\nMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNVBAgT\nCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVUaGUg\nVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBFQ0MgQ2VydGlm\naWNhdGlvbiBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQarFRaqflo\nI+d61SRvU8Za2EurxtW20eZzca7dnNYMYf3boIkDuAUU7FfO7l0/4iGzzvfUinng\no4N+LZfQYcTxmdwlkWOrfzCjtHDix6EznPO/LlxTsV+zfTJ/ijTjeXmjQjBAMB0G\nA1UdDgQWBBQ64QmG1M8ZwpZ2dEl23OA1xmNjmjAOBgNVHQ8BAf8EBAMCAQYwDwYD\nVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNoADBlAjA2Z6EWCNzklwBBHU6+4WMB\nzzuqQhFkoJ2UOQIReVx7Hfpkue4WQrO/isIJxOzksU0CMQDpKmFHjFJKS04YcPbW\nRNZu9YO6bVi9JNlWSOrvxKJGgYhqOkbRqZtNyWHa0V1Xahg=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB\niDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\ncnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\nBAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw\nMjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV\nBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU\naGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy\ndGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK\nAoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B\n3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY\ntJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/\nFp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2\nVN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT\n79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6\nc0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT\nYo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l\nc6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee\nUB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE\nHg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd\nBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF\nUp/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO\nVWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3\nATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs\n8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR\niQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze\nSf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ\nXHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/\nqS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB\nVXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB\nL6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG\njjxDah2nGN59PRbxYvnKkKj9\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEXjCCA0agAwIBAgIQRL4Mi1AAIbQR0ypoBqmtaTANBgkqhkiG9w0BAQUFADCB\nkzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug\nQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho\ndHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xGzAZBgNVBAMTElVUTiAtIERBVEFDb3Jw\nIFNHQzAeFw05OTA2MjQxODU3MjFaFw0xOTA2MjQxOTA2MzBaMIGTMQswCQYDVQQG\nEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYD\nVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cu\ndXNlcnRydXN0LmNvbTEbMBkGA1UEAxMSVVROIC0gREFUQUNvcnAgU0dDMIIBIjAN\nBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3+5YEKIrblXEjr8uRgnn4AgPLit6\nE5Qbvfa2gI5lBZMAHryv4g+OGQ0SR+ysraP6LnD43m77VkIVni5c7yPeIbkFdicZ\nD0/Ww5y0vpQZY/KmEQrrU0icvvIpOxboGqBMpsn0GFlowHDyUwDAXlCCpVZvNvlK\n4ESGoE1O1kduSUrLZ9emxAW5jh70/P/N5zbgnAVssjMiFdC04MwXwLLA9P4yPykq\nlXvY8qdOD1R8oQ2AswkDwf9c3V6aPryuvEeKaq5xyh+xKrhfQgUL7EYw0XILyulW\nbfXv33i+Ybqypa4ETLyorGkVl73v67SMvzX41MPRKA5cOp9wGDMgd8SirwIDAQAB\no4GrMIGoMAsGA1UdDwQEAwIBxjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRT\nMtGzz3/64PGgXYVOktKeRR20TzA9BgNVHR8ENjA0MDKgMKAuhixodHRwOi8vY3Js\nLnVzZXJ0cnVzdC5jb20vVVROLURBVEFDb3JwU0dDLmNybDAqBgNVHSUEIzAhBggr\nBgEFBQcDAQYKKwYBBAGCNwoDAwYJYIZIAYb4QgQBMA0GCSqGSIb3DQEBBQUAA4IB\nAQAnNZcAiosovcYzMB4p/OL31ZjUQLtgyr+rFywJNn9Q+kHcrpY6CiM+iVnJowft\nGzet/Hy+UUla3joKVAgWRcKZsYfNjGjgaQPpxE6YsjuMFrMOoAyYUJuTqXAJyCyj\nj98C5OBxOvG0I3KgqgHf35g+FFCgMSa9KOlaMCZ1+XtgHI3zzVAmbQQnmt/VDUVH\nKWss5nbZqSl9Mt3JNjy9rjXxEZ4du5A/EkdOjtd+D2JzHVImOBwYSf0wdJrE5SIv\n2MCN7ZF6TACPcn9d2t0bi0Vr591pl6jFVkwPDPafepE39peC4N1xaf92P2BNPM/3\nmfnGV/TJVTl4uix5yaaIK/QI\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEdDCCA1ygAwIBAgIQRL4Mi1AAJLQR0zYq/mUK/TANBgkqhkiG9w0BAQUFADCB\nlzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug\nQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho\ndHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt\nSGFyZHdhcmUwHhcNOTkwNzA5MTgxMDQyWhcNMTkwNzA5MTgxOTIyWjCBlzELMAkG\nA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEe\nMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v\nd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3QtSGFyZHdh\ncmUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx98M4P7Sof885glFn\n0G2f0v9Y8+efK+wNiVSZuTiZFvfgIXlIwrthdBKWHTxqctU8EGc6Oe0rE81m65UJ\nM6Rsl7HoxuzBdXmcRl6Nq9Bq/bkqVRcQVLMZ8Jr28bFdtqdt++BxF2uiiPsA3/4a\nMXcMmgF6sTLjKwEHOG7DpV4jvEWbe1DByTCP2+UretNb+zNAHqDVmBe8i4fDidNd\noI6yqqr2jmmIBsX6iSHzCJ1pLgkzmykNRg+MzEk0sGlRvfkGzWitZky8PqxhvQqI\nDsjfPe58BEydCl5rkdbux+0ojatNh4lz0G6k0B4WixThdkQDf2Os5M1JnMWS9Ksy\noUhbAgMBAAGjgbkwgbYwCwYDVR0PBAQDAgHGMA8GA1UdEwEB/wQFMAMBAf8wHQYD\nVR0OBBYEFKFyXyYbKJhDlV0HN9WFlp1L0sNFMEQGA1UdHwQ9MDswOaA3oDWGM2h0\ndHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUhhcmR3YXJlLmNy\nbDAxBgNVHSUEKjAoBggrBgEFBQcDAQYIKwYBBQUHAwUGCCsGAQUFBwMGBggrBgEF\nBQcDBzANBgkqhkiG9w0BAQUFAAOCAQEARxkP3nTGmZev/K0oXnWO6y1n7k57K9cM\n//bey1WiCuFMVGWTYGufEpytXoMs61quwOQt9ABjHbjAbPLPSbtNk28Gpgoiskli\nCE7/yMgUsogWXecB5BKV5UU0s4tpvc+0hY91UZ59Ojg6FEgSxvunOxqNDYJAB+gE\nCJChicsZUN/KHAG8HQQZexB2lzvukJDKxA4fFm517zP4029bHpbj4HR3dHuKom4t\n3XbWOTCC8KucUvIqx69JXn7HaOWCgchqJ/kniCrVWFCVH/A7HFe7fRQ5YiuayZSS\nKqMiDP+JJn1fIytH1xUdqWqeUQ0qUZ6B+dQ7XnASfxAynB67nfhmqA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDhDCCAwqgAwIBAgIQL4D+I4wOIg9IZxIokYesszAKBggqhkjOPQQDAzCByjEL\nMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW\nZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNyBWZXJpU2ln\nbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJp\nU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9y\naXR5IC0gRzQwHhcNMDcxMTA1MDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCByjELMAkG\nA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJp\nU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNyBWZXJpU2lnbiwg\nSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJpU2ln\nbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5\nIC0gRzQwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASnVnp8Utpkmw4tXNherJI9/gHm\nGUo9FANL+mAnINmDiWn6VMaaGF5VKmTeBvaNSjutEDxlPZCIBIngMGGzrl0Bp3ve\nfLK+ymVhAIau2o970ImtTR1ZmkGxvEeA3J5iw/mjgbIwga8wDwYDVR0TAQH/BAUw\nAwEB/zAOBgNVHQ8BAf8EBAMCAQYwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJ\naW1hZ2UvZ2lmMCEwHzAHBgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYj\naHR0cDovL2xvZ28udmVyaXNpZ24uY29tL3ZzbG9nby5naWYwHQYDVR0OBBYEFLMW\nkf3upm7ktS5Jj4d4gYDs5bG1MAoGCCqGSM49BAMDA2gAMGUCMGYhDBgmYFo4e1ZC\n4Kf8NoRRkSAsdk1DPcQdhCPQrNZ8NQbOzWm9kA3bbEhCHQ6qQgIxAJw9SDkjOVga\nFRJZap7v1VmyHVIsmXHNxynfGyphe3HR3vPA5Q06Sqotp9iGKt0uEA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIE0zCCA7ugAwIBAgIQGNrRniZ96LtKIVjNzGs7SjANBgkqhkiG9w0BAQUFADCB\nyjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL\nExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp\nU2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW\nZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0\naG9yaXR5IC0gRzUwHhcNMDYxMTA4MDAwMDAwWhcNMzYwNzE2MjM1OTU5WjCByjEL\nMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW\nZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2ln\nbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJp\nU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9y\naXR5IC0gRzUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1\nnmAMqudLO07cfLw8RRy7K+D+KQL5VwijZIUVJ/XxrcgxiV0i6CqqpkKzj/i5Vbex\nt0uz/o9+B1fs70PbZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6/WhkcIz\nSdhDY2pSS9KP6HBRTdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQG\nBO+QueQA5N06tRn/Arr0PO7gi+s3i+z016zy9vA9r911kTMZHRxAy3QkGSGT2RT+\nrCpSx4/VBEnkjWNHiDxpg8v+R70rfk/Fla4OndTRQ8Bnc+MUCH7lP59zuDMKz10/\nNIeWiu5T6CUVAgMBAAGjgbIwga8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E\nBAMCAQYwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAH\nBgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVy\naXNpZ24uY29tL3ZzbG9nby5naWYwHQYDVR0OBBYEFH/TZafC3ey78DAJ80M5+gKv\nMzEzMA0GCSqGSIb3DQEBBQUAA4IBAQCTJEowX2LP2BqYLz3q3JktvXf2pXkiOOzE\np6B4Eq1iDkVwZMXnl2YtmAl+X6/WzChl8gGqCBpH3vn5fJJaCGkgDdk+bW48DW7Y\n5gaRQBi5+MHt39tBquCWIMnNZBU4gcmU7qKEKQsTb47bDN0lAtukixlE0kF6BWlK\nWE9gyn6CagsCqiUXObXbf+eEZSqVir2G3l6BFoMtEMze/aiCKm0oHw0LxOXnGiYZ\n4fQRbxC1lfznQgUy286dUV4otp6F01vvpX1FQHKOtw5rDgb7MzVIcbidJ4vEZV8N\nhnacRHr2lVz2XTIIM6RUthg/aFzyQkqFOFSDX9HoLPKsEdao7WNq\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEuTCCA6GgAwIBAgIQQBrEZCGzEyEDDrvkEhrFHTANBgkqhkiG9w0BAQsFADCB\nvTELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL\nExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwOCBWZXJp\nU2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MTgwNgYDVQQDEy9W\nZXJpU2lnbiBVbml2ZXJzYWwgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAe\nFw0wODA0MDIwMDAwMDBaFw0zNzEyMDEyMzU5NTlaMIG9MQswCQYDVQQGEwJVUzEX\nMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0\nIE5ldHdvcmsxOjA4BgNVBAsTMShjKSAyMDA4IFZlcmlTaWduLCBJbmMuIC0gRm9y\nIGF1dGhvcml6ZWQgdXNlIG9ubHkxODA2BgNVBAMTL1ZlcmlTaWduIFVuaXZlcnNh\nbCBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEF\nAAOCAQ8AMIIBCgKCAQEAx2E3XrEBNNti1xWb/1hajCMj1mCOkdeQmIN65lgZOIzF\n9uVkhbSicfvtvbnazU0AtMgtc6XHaXGVHzk8skQHnOgO+k1KxCHfKWGPMiJhgsWH\nH26MfF8WIFFE0XBPV+rjHOPMee5Y2A7Cs0WTwCznmhcrewA3ekEzeOEz4vMQGn+H\nLL729fdC4uW/h2KJXwBL38Xd5HVEMkE6HnFuacsLdUYI0crSK5XQz/u5QGtkjFdN\n/BMReYTtXlT2NJ8IAfMQJQYXStrxHXpma5hgZqTZ79IugvHw7wnqRMkVauIDbjPT\nrJ9VAMf2CGqUuV/c4DPxhGD5WycRtPwW8rtWaoAljQIDAQABo4GyMIGvMA8GA1Ud\nEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMG0GCCsGAQUFBwEMBGEwX6FdoFsw\nWTBXMFUWCWltYWdlL2dpZjAhMB8wBwYFKw4DAhoEFI/l0xqGrI2Oa8PPgGrUSBgs\nexkuMCUWI2h0dHA6Ly9sb2dvLnZlcmlzaWduLmNvbS92c2xvZ28uZ2lmMB0GA1Ud\nDgQWBBS2d/ppSEefUxLVwuoHMnYH0ZcHGTANBgkqhkiG9w0BAQsFAAOCAQEASvj4\nsAPmLGd75JR3Y8xuTPl9Dg3cyLk1uXBPY/ok+myDjEedO2Pzmvl2MpWRsXe8rJq+\nseQxIcaBlVZaDrHC1LGmWazxY8u4TB1ZkErvkBYoH1quEPuBUDgMbMzxPcP1Y+Oz\n4yHJJDnp/RVmRvQbEdBNc6N9Rvk97ahfYtTxP/jgdFcrGJ2BtMQo2pSXpXDrrB2+\nBxHw1dvd5Yzw1TKwg+ZX4o+/vqGqvz0dtdQ46tewXDpPaj+PwGZsY6rp2aQW9IHR\nlRQOfc2VNNnSj3BzgXucfr2YYdhFh5iQxeuGMMY1v/D/w1WIg0vvBZIGcfK4mJO3\n7M2CYfE45k+XmCpajQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEGjCCAwICEQCbfgZJoz5iudXukEhxKe9XMA0GCSqGSIb3DQEBBQUAMIHKMQsw\nCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZl\ncmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWdu\nLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlT\naWduIENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3Jp\ndHkgLSBHMzAeFw05OTEwMDEwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMIHKMQswCQYD\nVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlT\naWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWduLCBJ\nbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlTaWdu\nIENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg\nLSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMu6nFL8eB8aHm8b\nN3O9+MlrlBIwT/A2R/XQkQr1F8ilYcEWQE37imGQ5XYgwREGfassbqb1EUGO+i2t\nKmFZpGcmTNDovFJbcCAEWNF6yaRpvIMXZK0Fi7zQWM6NjPXr8EJJC52XJ2cybuGu\nkxUccLwgTS8Y3pKI6GyFVxEa6X7jJhFUokWWVYPKMIno3Nij7SqAP395ZVc+FSBm\nCC+Vk7+qRy+oRpfwEuL+wgorUeZ25rdGt+INpsyow0xZVYnm6FNcHOqd8GIWC6fJ\nXwzw3sJ2zq/3avL6QaaiMxTJ5Xpj055iN9WFZZ4O5lMkdBteHRJTW8cs54NJOxWu\nimi5V5cCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAERSWwauSCPc/L8my/uRan2Te\n2yFPhpk0djZX3dAVL8WtfxUfN2JzPtTnX84XA9s1+ivbrmAJXx5fj267Cz3qWhMe\nDGBvtcC1IyIuBwvLqXTLR7sdwdela8wv0kL9Sd2nic9TutoAWii/gt/4uhMdUIaC\n/Y4wjylGsB49Ndo4YhYYSq3mtlFs3q9i6wHQHiT+eo8SGhJouPtmmRQURVyu565p\nF4ErWjfJXir0xuKhXFSbplQAz/DxwceYMBo7Nhbbo27q/a2ywtrvAkcTisDxszGt\nTxzhT5yvDwyd93gN2PQ1VoDat20Xj50egWTh/sVFuq1ruQp6Tk9LhO5L8X3dEQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEGjCCAwICEQDsoKeLbnVqAc/EfMwvlF7XMA0GCSqGSIb3DQEBBQUAMIHKMQsw\nCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZl\ncmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWdu\nLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlT\naWduIENsYXNzIDQgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3Jp\ndHkgLSBHMzAeFw05OTEwMDEwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMIHKMQswCQYD\nVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlT\naWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWduLCBJ\nbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlTaWdu\nIENsYXNzIDQgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg\nLSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK3LpRFpxlmr8Y+1\nGQ9Wzsy1HyDkniYlS+BzZYlZ3tCD5PUPtbut8XzoIfzk6AzufEUiGXaStBO3IFsJ\n+mGuqPKljYXCKtbeZjbSmwL0qJJgfJxptI8kHtCGUvYynEFYHiK9zUVilQhu0Gbd\nU6LM8BDcVHOLBKFGMzNcF0C5nk3T875Vg+ixiY5afJqWIpA7iCXy0lOIAgwLePLm\nNxdLMEYH5IBtptiWLugs+BGzOA1mppvqySNb247i8xOOGlktqgLw7KSHZtzBP/XY\nufTsgsbSPZUd5cBPhMnZo0QoBmrXRazwa2rvTl/4EYIeOGM0ZlDUPpNz+jDDZq3/\nky2X7wMCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAj/ola09b5KROJ1WrIhVZPMq1\nCtRK26vdoV9TxaBXOcLORyu+OshWv8LZJxA6sQU8wHcxuzrTBXttmhwwjIDLk5Mq\ng6sFUYICABFna/OIYUdfA5PVWw3g8dShMjWFsjrbsIKr0csKvE+MW8VLADsfKoKm\nfjaF3H48ZwC15DtS4KjrXRX5xm3wrR0OhbepmnMUWluPQSjA1egtTaRezarZ7c7c\n2NU8Qh0XwRJdRTjDOPP8hS6DRkiy1yBfkjaP53kPmF6Z6PDQpLv1U70qzlmwr25/\nbLvSHgCwIe34QWKCudiyxLtGUPMxxY8BqHTr9Xgn2uf3ZkPznoM+IKrDNWCRzg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDojCCAoqgAwIBAgIQE4Y1TR0/BvLB+WUF1ZAcYjANBgkqhkiG9w0BAQUFADBr\nMQswCQYDVQQGEwJVUzENMAsGA1UEChMEVklTQTEvMC0GA1UECxMmVmlzYSBJbnRl\ncm5hdGlvbmFsIFNlcnZpY2UgQXNzb2NpYXRpb24xHDAaBgNVBAMTE1Zpc2EgZUNv\nbW1lcmNlIFJvb3QwHhcNMDIwNjI2MDIxODM2WhcNMjIwNjI0MDAxNjEyWjBrMQsw\nCQYDVQQGEwJVUzENMAsGA1UEChMEVklTQTEvMC0GA1UECxMmVmlzYSBJbnRlcm5h\ndGlvbmFsIFNlcnZpY2UgQXNzb2NpYXRpb24xHDAaBgNVBAMTE1Zpc2EgZUNvbW1l\ncmNlIFJvb3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvV95WHm6h\n2mCxlCfLF9sHP4CFT8icttD0b0/Pmdjh28JIXDqsOTPHH2qLJj0rNfVIsZHBAk4E\nlpF7sDPwsRROEW+1QK8bRaVK7362rPKgH1g/EkZgPI2h4H3PVz4zHvtH8aoVlwdV\nZqW1LS7YgFmypw23RuwhY/81q6UCzyr0TP579ZRdhE2o8mCP2w4lPJ9zcc+U30rq\n299yOIzzlr3xF7zSujtFWsan9sYXiwGd/BmoKoMWuDpI/k4+oKsGGelT84ATB+0t\nvz8KPFUgOSwsAGl0lUq8ILKpeeUYiZGo3BxN77t+Nwtd/jmliFKMAGzsGHxBvfaL\ndXe6YJ2E5/4tAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQD\nAgEGMB0GA1UdDgQWBBQVOIMPPyw/cDMezUb+B4wg4NfDtzANBgkqhkiG9w0BAQUF\nAAOCAQEAX/FBfXxcCLkr4NWSR/pnXKUTwwMhmytMiUbPWU3J/qVAtmPN3XEolWcR\nzCSs00Rsca4BIGsDoo8Ytyk6feUWYFN4PMCvFYP3j1IzJL1kk5fui/fbGKhtcbP3\nLBfQdCVp9/5rPJS+TUtBjE7ic9DjkCJzQ83z7+pzzkWKsKZJ/0x9nXGIxHYdkFsd\n7v3M9+79YKWxehZx0RbQfBI8bGmX265fOZpwLwU8GUYEmSA20GBuYQa7FkKMcPcw\n++DbZqMAAb3mLNqRX6BGi01qnD093QVG/na/oAo85ADmJ7f/hC3euiInlhBx6yLt\n398znM/jra6O1I7mT1GvFpLgXPYHDw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEvTCCA6WgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBhTELMAkGA1UEBhMCVVMx\nIDAeBgNVBAoMF1dlbGxzIEZhcmdvIFdlbGxzU2VjdXJlMRwwGgYDVQQLDBNXZWxs\ncyBGYXJnbyBCYW5rIE5BMTYwNAYDVQQDDC1XZWxsc1NlY3VyZSBQdWJsaWMgUm9v\ndCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDcxMjEzMTcwNzU0WhcNMjIxMjE0\nMDAwNzU0WjCBhTELMAkGA1UEBhMCVVMxIDAeBgNVBAoMF1dlbGxzIEZhcmdvIFdl\nbGxzU2VjdXJlMRwwGgYDVQQLDBNXZWxscyBGYXJnbyBCYW5rIE5BMTYwNAYDVQQD\nDC1XZWxsc1NlY3VyZSBQdWJsaWMgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDub7S9eeKPCCGeOARBJe+r\nWxxTkqxtnt3CxC5FlAM1iGd0V+PfjLindo8796jE2yljDpFoNoqXjopxaAkH5OjU\nDk/41itMpBb570OYj7OeUt9tkTmPOL13i0Nj67eT/DBMHAGTthP796EfvyXhdDcs\nHqRePGj4S78NuR4uNuip5Kf4D8uCdXw1LSLWwr8L87T8bJVhHlfXBIEyg1J55oNj\nz7fLY4sR4r1e6/aN7ZVyKLSsEmLpSjPmgzKuBXWVvYSV2ypcm44uDLiBK0HmOFaf\nSZtsdvqKXfcBeYF8wYNABf5x/Qw/zE5gCQ5lRxAvAcAFP4/4s0HvWkJ+We/Slwxl\nAgMBAAGjggE0MIIBMDAPBgNVHRMBAf8EBTADAQH/MDkGA1UdHwQyMDAwLqAsoCqG\nKGh0dHA6Ly9jcmwucGtpLndlbGxzZmFyZ28uY29tL3dzcHJjYS5jcmwwDgYDVR0P\nAQH/BAQDAgHGMB0GA1UdDgQWBBQmlRkQ2eihl5H/3BnZtQQ+0nMKajCBsgYDVR0j\nBIGqMIGngBQmlRkQ2eihl5H/3BnZtQQ+0nMKaqGBi6SBiDCBhTELMAkGA1UEBhMC\nVVMxIDAeBgNVBAoMF1dlbGxzIEZhcmdvIFdlbGxzU2VjdXJlMRwwGgYDVQQLDBNX\nZWxscyBGYXJnbyBCYW5rIE5BMTYwNAYDVQQDDC1XZWxsc1NlY3VyZSBQdWJsaWMg\nUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3JpdHmCAQEwDQYJKoZIhvcNAQEFBQADggEB\nALkVsUSRzCPIK0134/iaeycNzXK7mQDKfGYZUMbVmO2rvwNa5U3lHshPcZeG1eMd\n/ZDJPHV3V3p9+N701NX3leZ0bh08rnyd2wIDBSxxSyU+B+NemvVmFymIGjifz6pB\nA4SXa5M4esowRBskRDPQ5NHcKDj0E0M1NSljqHyita04pO2t/caaH/+Xc/77szWn\nk4bGdpEA5qxRFsQnMlzbc9qlk1eOPm01JghZ1edE13YgY+esE2fDbbFwRnzVlhE9\niW9dqKHrjQrawx0zbKPqZxmamX9LPYNRKh3KL4YMon4QLSvUFpULB6ouFJJJtylv\n2G0xffX8oRAHh84vWdw+WNs=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFdjCCA16gAwIBAgIQXmjWEXGUY1BWAGjzPsnFkTANBgkqhkiG9w0BAQUFADBV\nMQswCQYDVQQGEwJDTjEaMBgGA1UEChMRV29TaWduIENBIExpbWl0ZWQxKjAoBgNV\nBAMTIUNlcnRpZmljYXRpb24gQXV0aG9yaXR5IG9mIFdvU2lnbjAeFw0wOTA4MDgw\nMTAwMDFaFw0zOTA4MDgwMTAwMDFaMFUxCzAJBgNVBAYTAkNOMRowGAYDVQQKExFX\nb1NpZ24gQ0EgTGltaXRlZDEqMCgGA1UEAxMhQ2VydGlmaWNhdGlvbiBBdXRob3Jp\ndHkgb2YgV29TaWduMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvcqN\nrLiRFVaXe2tcesLea9mhsMMQI/qnobLMMfo+2aYpbxY94Gv4uEBf2zmoAHqLoE1U\nfcIiePyOCbiohdfMlZdLdNiefvAA5A6JrkkoRBoQmTIPJYhTpA2zDxIIFgsDcScc\nf+Hb0v1naMQFXQoOXXDX2JegvFNBmpGN9J42Znp+VsGQX+axaCA2pIwkLCxHC1l2\nZjC1vt7tj/id07sBMOby8w7gLJKA84X5KIq0VC6a7fd2/BVoFutKbOsuEo/Uz/4M\nx1wdC34FMr5esAkqQtXJTpCzWQ27en7N1QhatH/YHGkR+ScPewavVIMYe+HdVHpR\naG53/Ma/UkpmRqGyZxq7o093oL5d//xWC0Nyd5DKnvnyOfUNqfTq1+ezEC8wQjch\nzDBwyYaYD8xYTYO7feUapTeNtqwylwA6Y3EkHp43xP901DfA4v6IRmAR3Qg/UDar\nuHqklWJqbrDKaiFaafPz+x1wOZXzp26mgYmhiMU7ccqjUu6Du/2gd/Tkb+dC221K\nmYo0SLwX3OSACCK28jHAPwQ+658geda4BmRkAjHXqc1S+4RFaQkAKtxVi8QGRkvA\nSh0JWzko/amrzgD5LkhLJuYwTKVYyrREgk/nkR4zw7CT/xH8gdLKH3Ep3XZPkiWv\nHYG3Dy+MwwbMLyejSuQOmbp8HkUff6oZRZb9/D0CAwEAAaNCMEAwDgYDVR0PAQH/\nBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOFmzw7R8bNLtwYgFP6H\nEtX2/vs+MA0GCSqGSIb3DQEBBQUAA4ICAQCoy3JAsnbBfnv8rWTjMnvMPLZdRtP1\nLOJwXcgu2AZ9mNELIaCJWSQBnfmvCX0KI4I01fx8cpm5o9dU9OpScA7F9dY74ToJ\nMuYhOZO9sxXqT2r09Ys/L3yNWC7F4TmgPsc9SnOeQHrAK2GpZ8nzJLmzbVUsWh2e\nJXLOC62qx1ViC777Y7NhRCOjy+EaDveaBk3e1CNOIZZbOVtXHS9dCF4Jef98l7VN\ng64N1uajeeAz0JmWAjCnPv/So0M/BVoG6kQC2nz4SNAzqfkHx5Xh9T71XXG68pWp\ndIhhWeO/yloTunK0jF02h+mmxTwTv97QRCbut+wucPrXnbes5cVAWubXbHssw1ab\nR80LzvobtCHXt2a49CUwi1wNuepnsvRtrtWhnk/Yn+knArAdBtaP4/tIEp9/EaEQ\nPkxROpaw0RPxx9gmrjrKkcRpnd8BKWRRb2jaFOwIQZeQjdCygPLPwj2/kWjFgGce\nxGATVdVhmVd8upUPYUk6ynW8yQqTP2cOEvIo4jEbwFcW3wh8GcF+Dx+FHgo2fFt+\nJ7x6v+Db9NpSvd4MVHAxkUOVyLzwPt0JfjBkUO1/AaQzZ01oT74V77D2AhGiGxMl\nOtzCWfHjXEa7ZywCRuoeSKbmW9m1vFGikpbbqsY3Iqb+zCB0oy2pLmvLwIIRIbWT\nee5Ehr7XHuQe+w==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFWDCCA0CgAwIBAgIQUHBrzdgT/BtOOzNy0hFIjTANBgkqhkiG9w0BAQsFADBG\nMQswCQYDVQQGEwJDTjEaMBgGA1UEChMRV29TaWduIENBIExpbWl0ZWQxGzAZBgNV\nBAMMEkNBIOayg+mAmuagueivgeS5pjAeFw0wOTA4MDgwMTAwMDFaFw0zOTA4MDgw\nMTAwMDFaMEYxCzAJBgNVBAYTAkNOMRowGAYDVQQKExFXb1NpZ24gQ0EgTGltaXRl\nZDEbMBkGA1UEAwwSQ0Eg5rKD6YCa5qC56K+B5LmmMIICIjANBgkqhkiG9w0BAQEF\nAAOCAg8AMIICCgKCAgEA0EkhHiX8h8EqwqzbdoYGTufQdDTc7WU1/FDWiD+k8H/r\nD195L4mx/bxjWDeTmzj4t1up+thxx7S8gJeNbEvxUNUqKaqoGXqW5pWOdO2XCld1\n9AXbbQs5uQF/qvbW2mzmBeCkTVL829B0txGMe41P/4eDrv8FAxNXUDf+jJZSEExf\nv5RxadmWPgxDT74wwJ85dE8GRV2j1lY5aAfMh09Qd5Nx2UQIsYo06Yms25tO4dnk\nUkWMLhQfkWsZHWgpLFbE4h4TV2TwYeO5Ed+w4VegG63XX9Gv2ystP9Bojg/qnw+L\nNVgbExz03jWhCl3W6t8Sb8D7aQdGctyB9gQjF+BNdeFyb7Ao65vh4YOhn0pdr8yb\n+gIgthhid5E7o9Vlrdx8kHccREGkSovrlXLp9glk3Kgtn3R46MGiCWOc76DbT52V\nqyBPt7D3h1ymoOQ3OMdc4zUPLK2jgKLsLl3Az+2LBcLmc272idX10kaO6m1jGx6K\nyX2m+Jzr5dVjhU1zZmkR/sgO9MHHZklTfuQZa/HpelmjbX7FF+Ynxu8b22/8DU0G\nAbQOXDBGVWCvOGU6yke6rCzMRh+yRpY/8+0mBe53oWprfi1tWFxK1I5nuPHa1UaK\nJ/kR8slC/k7e3x9cxKSGhxYzoacXGKUN5AXlK8IrC6KVkLn9YDxOiT7nnO4fuwEC\nAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0O\nBBYEFOBNv9ybQV0T6GTwp+kVpOGBwboxMA0GCSqGSIb3DQEBCwUAA4ICAQBqinA4\nWbbaixjIvirTthnVZil6Xc1bL3McJk6jfW+rtylNpumlEYOnOXOvEESS5iVdT2H6\nyAa+Tkvv/vMx/sZ8cApBWNromUuWyXi8mHwCKe0JgOYKOoICKuLJL8hWGSbueBwj\n/feTZU7n85iYr83d2Z5AiDEoOqsuC7CsDCT6eiaY8xJhEPRdF/d+4niXVOKM6Cm6\njBAyvd0zaziGfjk9DgNyp115j0WKWa5bIW4xRtVZjc8VX90xJc/bYNaBRHIpAlf2\nltTW/+op2znFuCyKGo3Oy+dCMYYFaA6eFN0AkLppRQjbbpCBhqcqBT/mhDn4t/lX\nX0ykeVoQDF7Va/81XwVRHmyjdanPUIPTfPRm94KNPQx96N97qA4bLJyuQHCH2u2n\nFoJavjVsIE4iYdm8UXrNemHcSxH5/mc0zy4EZmFcV5cjjPOGG0jfKq+nwf/Yjj4D\nu9gqsPoUJbJRa4ZDhS4HIxaAjUz7tGM7zMN07RujHv41D198HRaG9Q7DlfEvr10l\nO1Hm13ZBONFLAzkopR6RctR9q5czxNM+4Gm2KHmgCY0c0f9BckgG/Jou5yD5m6Le\nie2uPAmvylezkolwQOQvT8Jwg0DXJCxr5wkf09XHwQj02w47HAcLQxGEIYbpgNR1\n2KvxAmLBsX5VYc8T1yaw15zLKYs4SgsOkI26oQ==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEMDCCAxigAwIBAgIQUJRs7Bjq1ZxN1ZfvdY+grTANBgkqhkiG9w0BAQUFADCB\ngjELMAkGA1UEBhMCVVMxHjAcBgNVBAsTFXd3dy54cmFtcHNlY3VyaXR5LmNvbTEk\nMCIGA1UEChMbWFJhbXAgU2VjdXJpdHkgU2VydmljZXMgSW5jMS0wKwYDVQQDEyRY\nUmFtcCBHbG9iYWwgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDQxMTAxMTcx\nNDA0WhcNMzUwMTAxMDUzNzE5WjCBgjELMAkGA1UEBhMCVVMxHjAcBgNVBAsTFXd3\ndy54cmFtcHNlY3VyaXR5LmNvbTEkMCIGA1UEChMbWFJhbXAgU2VjdXJpdHkgU2Vy\ndmljZXMgSW5jMS0wKwYDVQQDEyRYUmFtcCBHbG9iYWwgQ2VydGlmaWNhdGlvbiBB\ndXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYJB69FbS6\n38eMpSe2OAtp87ZOqCwuIR1cRN8hXX4jdP5efrRKt6atH67gBhbim1vZZ3RrXYCP\nKZ2GG9mcDZhtdhAoWORlsH9KmHmf4MMxfoArtYzAQDsRhtDLooY2YKTVMIJt2W7Q\nDxIEM5dfT2Fa8OT5kavnHTu86M/0ay00fOJIYRyO82FEzG+gSqmUsE3a56k0enI4\nqEHMPJQRfevIpoy3hsvKMzvZPTeL+3o+hiznc9cKV6xkmxnr9A8ECIqsAxcZZPRa\nJSKNNCyy9mgdEm3Tih4U2sSPpuIjhdV6Db1q4Ons7Be7QhtnqiXtRYMh/MHJfNVi\nPvryxS3T/dRlAgMBAAGjgZ8wgZwwEwYJKwYBBAGCNxQCBAYeBABDAEEwCwYDVR0P\nBAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMZPoj0GY4QJnM5i5ASs\njVy16bYbMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwueHJhbXBzZWN1cml0\neS5jb20vWEdDQS5jcmwwEAYJKwYBBAGCNxUBBAMCAQEwDQYJKoZIhvcNAQEFBQAD\nggEBAJEVOQMBG2f7Shz5CmBbodpNl2L5JFMn14JkTpAuw0kbK5rc/Kh4ZzXxHfAR\nvbdI4xD2Dd8/0sm2qlWkSLoC295ZLhVbO50WfUfXN+pfTXYSNrsf16GBBEYgoyxt\nqZ4Bfj8pzgCT3/3JknOJiWSe5yvkHJEs0rnOfc5vMZnT5r7SHpDwCRR5XCOrTdLa\nIR9NmXmd4c8nnxCbHIgNsIpkQTG4DmyQJKSbXHGPurt+HBvbaoAPIbzp26a3QPSy\ni6mx5O+aGtA9aZnuqCij4Tyz8LIRnM98QObd50N9otg6tamN8jSZxNQQ4Qb9CYQQ\nO+7ETPTsJ3xCwnR8gooJybQDJbw=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDODCCAiCgAwIBAgIGIAYFFnACMA0GCSqGSIb3DQEBBQUAMDsxCzAJBgNVBAYT\nAlJPMREwDwYDVQQKEwhjZXJ0U0lHTjEZMBcGA1UECxMQY2VydFNJR04gUk9PVCBD\nQTAeFw0wNjA3MDQxNzIwMDRaFw0zMTA3MDQxNzIwMDRaMDsxCzAJBgNVBAYTAlJP\nMREwDwYDVQQKEwhjZXJ0U0lHTjEZMBcGA1UECxMQY2VydFNJR04gUk9PVCBDQTCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALczuX7IJUqOtdu0KBuqV5Do\n0SLTZLrTk+jUrIZhQGpgV2hUhE28alQCBf/fm5oqrl0Hj0rDKH/v+yv6efHHrfAQ\nUySQi2bJqIirr1qjAOm+ukbuW3N7LBeCgV5iLKECZbO9xSsAfsT8AzNXDe3i+s5d\nRdY4zTW2ssHQnIFKquSyAVwdj1+ZxLGt24gh65AIgoDzMKND5pCCrlUoSe1b16kQ\nOA7+j0xbm0bqQfWwCHTD0IgztnzXdN/chNFDDnU5oSVAKOp4yw4sLjmdjItuFhwv\nJoIQ4uNllAoEwF73XVv4EOLQunpL+943AAAaWyjj0pxzPjKHmKHJUS/X3qwzs08C\nAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYwHQYDVR0O\nBBYEFOCMm9slSbPxfIbWskKHC9BroNnkMA0GCSqGSIb3DQEBBQUAA4IBAQA+0hyJ\nLjX8+HXd5n9liPRyTMks1zJO890ZeUe9jjtbkw9QSSQTaxQGcu8J06Gh40CEyecY\nMnQ8SG4Pn0vU9x7Tk4ZkVJdjclDVVc/6IJMCopvDI5NOFlV2oHB5bc0hH88vLbwZ\n44gx+FkagQnIl6Z0x2DEW8xXjrJ1/RsCCdtZb3KTafcxQdaIOL+Hsr0Wefmq5L6I\nJd1hJyMctTEHBDa0GpC9oHRxUIltvBTjD4au8as+x6AJzKNI0eDbZOeStc+vckNw\ni/nDhDwTqn6Sm1dTk/pwwpEOMfmbZ13pljheX7NzTogVZ96edhBiIL5VaZVDADlN\n9u6wWk5JRFRYX0KD\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFsDCCA5igAwIBAgIQFci9ZUdcr7iXAF7kBtK8nTANBgkqhkiG9w0BAQUFADBe\nMQswCQYDVQQGEwJUVzEjMCEGA1UECgwaQ2h1bmdod2EgVGVsZWNvbSBDby4sIEx0\nZC4xKjAoBgNVBAsMIWVQS0kgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAe\nFw0wNDEyMjAwMjMxMjdaFw0zNDEyMjAwMjMxMjdaMF4xCzAJBgNVBAYTAlRXMSMw\nIQYDVQQKDBpDaHVuZ2h3YSBUZWxlY29tIENvLiwgTHRkLjEqMCgGA1UECwwhZVBL\nSSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEF\nAAOCAg8AMIICCgKCAgEA4SUP7o3biDN1Z82tH306Tm2d0y8U82N0ywEhajfqhFAH\nSyZbCUNsIZ5qyNUD9WBpj8zwIuQf5/dqIjG3LBXy4P4AakP/h2XGtRrBp0xtInAh\nijHyl3SJCRImHJ7K2RKilTza6We/CKBk49ZCt0Xvl/T29de1ShUCWH2YWEtgvM3X\nDZoTM1PRYfl61dd4s5oz9wCGzh1NlDivqOx4UXCKXBCDUSH3ET00hl7lSM2XgYI1\nTBnsZfZrxQWh7kcT1rMhJ5QQCtkkO7q+RBNGMD+XPNjX12ruOzjjK9SXDrkb5wdJ\nfzcq+Xd4z1TtW0ado4AOkUPB1ltfFLqfpo0kR0BZv3I4sjZsN/+Z0V0OWQqraffA\nsgRFelQArr5T9rXn4fg8ozHSqf4hUmTFpmfwdQcGlBSBVcYn5AGPF8Fqcde+S/uU\nWH1+ETOxQvdibBjWzwloPn9s9h6PYq2lY9sJpx8iQkEeb5mKPtf5P0B6ebClAZLS\nnT0IFaUQAS2zMnaolQ2zepr7BxB4EW/hj8e6DyUadCrlHJhBmd8hh+iVBmoKs2pH\ndmX2Os+PYhcZewoozRrSgx4hxyy/vv9haLdnG7t4TY3OZ+XkwY63I2binZB1NJip\nNiuKmpS5nezMirH4JYlcWrYvjB9teSSnUmjDhDXiZo1jDiVN1Rmy5nk3pyKdVDEC\nAwEAAaNqMGgwHQYDVR0OBBYEFB4M97Zn8uGSJglFwFU5Lnc/QkqiMAwGA1UdEwQF\nMAMBAf8wOQYEZyoHAAQxMC8wLQIBADAJBgUrDgMCGgUAMAcGBWcqAwAABBRFsMLH\nClZ87lt4DJX5GFPBphzYEDANBgkqhkiG9w0BAQUFAAOCAgEACbODU1kBPpVJufGB\nuvl2ICO1J2B01GqZNF5sAFPZn/KmsSQHRGoqxqWOeBLoR9lYGxMqXnmbnwoqZ6Yl\nPwZpVnPDimZI+ymBV3QGypzqKOg4ZyYr8dW1P2WT+DZdjo2NQCCHGervJ8A9tDkP\nJXtoUHRVnAxZfVo9QZQlUgjgRywVMRnVvwdVxrsStZf0X4OFunHB2WyBEXYKCrC/\ngpf36j36+uwtqSiUO1bd0lEursC9CBWMd1I0ltabrNMdjmEPNXubrjlpC2JgQCA2\nj6/7Nu4tCEoduL+bXPjqpRugc6bY+G7gMwRfaKonh+3ZwZCc7b3jajWvY9+rGNm6\n5ulK6lCKD2GTHuItGeIwlDWSXQ62B68ZgI9HkFFLLk3dheLSClIKF5r8GrBQAuUB\no2M3IUxExJtRmREOc5wGj1QupyheRDmHVi03vYVElOEMSyycw5KFNGHLD7ibSkNS\n/jQ6fbjpKdx2qcgw+BRxgMYeNkh0IkFch4LoGHGLQYlE535YW6i4jRPpp2zDR+2z\nGp1iro2C6pSe3VkQw63d4k3jMdXH7OjysP6SHhYKGvzZ8/gntsm+HbRsZJB/9OTE\nW9c3rkIO3aQab3yIVMUWbuF6aC74Or8NpDyJO3inTmODBCEIZ43ygknQW/2xzQ+D\nhNQ+IIX3Sj0rnP0qCglN6oH4EZw=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEIDCCAwigAwIBAgIQNE7VVyDV7exJ9C/ON9srbTANBgkqhkiG9w0BAQUFADCB\nqTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5jLjEoMCYGA1UECxMf\nQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjE4MDYGA1UECxMvKGMpIDIw\nMDYgdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxHzAdBgNV\nBAMTFnRoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EwHhcNMDYxMTE3MDAwMDAwWhcNMzYw\nNzE2MjM1OTU5WjCBqTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5j\nLjEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjE4MDYG\nA1UECxMvKGMpIDIwMDYgdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNl\nIG9ubHkxHzAdBgNVBAMTFnRoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsoPD7gFnUnMekz52hWXMJEEUMDSxuaPFs\nW0hoSVk3/AszGcJ3f8wQLZU0HObrTQmnHNK4yZc2AreJ1CRfBsDMRJSUjQJib+ta\n3RGNKJpchJAQeg29dGYvajig4tVUROsdB58Hum/u6f1OCyn1PoSgAfGcq/gcfomk\n6KHYcWUNo1F77rzSImANuVud37r8UVsLr5iy6S7pBOhih94ryNdOwUxkHt3Ph1i6\nSk/KaAcdHJ1KxtUvkcx8cXIcxcBn6zL9yZJclNqFwJu/U30rCfSMnZEfl2pSy94J\nNqR32HuHUETVPm4pafs5SSYeCaWAe0At6+gnhcn+Yf1+5nyXHdWdAgMBAAGjQjBA\nMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBR7W0XP\nr87Lev0xkhpqtvNG61dIUDANBgkqhkiG9w0BAQUFAAOCAQEAeRHAS7ORtvzw6WfU\nDW5FvlXok9LOAz/t2iWwHVfLHjp2oEzsUHboZHIMpKnxuIvW1oeEuzLlQRHAd9mz\nYJ3rG9XRbkREqaYB7FViHXe4XI5ISXycO1cRrK1zN44veFyQaEfZYGDm/Ac9IiAX\nxPcW6cTYcvnIc3zfFi8VqT79aie2oetaupgf1eNNZAqdE8hhuvU5HIe6uL17In/2\n/qxAeeWsEG89jxt5dovEN7MhGITlNgDrYyCZuen+MwS7QcjBAvlEYyCegc5C09Y/\nLHbTY5xZ3Y+m4Q6gLkH3LpVHz7z9M/P2C2F+fpErgUfCJzDupxBdN49cOSvkBPB7\njVaMaA==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIICiDCCAg2gAwIBAgIQNfwmXNmET8k9Jj1Xm67XVjAKBggqhkjOPQQDAzCBhDEL\nMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5jLjE4MDYGA1UECxMvKGMp\nIDIwMDcgdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxJDAi\nBgNVBAMTG3RoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EgLSBHMjAeFw0wNzExMDUwMDAw\nMDBaFw0zODAxMTgyMzU5NTlaMIGEMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMdGhh\nd3RlLCBJbmMuMTgwNgYDVQQLEy8oYykgMjAwNyB0aGF3dGUsIEluYy4gLSBGb3Ig\nYXV0aG9yaXplZCB1c2Ugb25seTEkMCIGA1UEAxMbdGhhd3RlIFByaW1hcnkgUm9v\ndCBDQSAtIEcyMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEotWcgnuVnfFSeIf+iha/\nBebfowJPDQfGAFG6DAJSLSKkQjnE/o/qycG+1E3/n3qe4rF8mq2nhglzh9HnmuN6\npapu+7qzcMBniKI11KOasf2twu8x+qi58/sIxpHR+ymVo0IwQDAPBgNVHRMBAf8E\nBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUmtgAMADna3+FGO6Lts6K\nDPgR4bswCgYIKoZIzj0EAwMDaQAwZgIxAN344FdHW6fmCsO99YCKlzUNG4k8VIZ3\nKMqh9HneteY4sPBlcIx/AlTCv//YoT7ZzwIxAMSNlPzcU9LcnXgWHxUzI1NS41ox\nXZ3Krr0TKUQNJ1uo52icEvdYPy5yAlejj6EULg==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEKjCCAxKgAwIBAgIQYAGXt0an6rS0mtZLL/eQ+zANBgkqhkiG9w0BAQsFADCB\nrjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5jLjEoMCYGA1UECxMf\nQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjE4MDYGA1UECxMvKGMpIDIw\nMDggdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxJDAiBgNV\nBAMTG3RoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EgLSBHMzAeFw0wODA0MDIwMDAwMDBa\nFw0zNzEyMDEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMdGhhd3Rl\nLCBJbmMuMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9u\nMTgwNgYDVQQLEy8oYykgMjAwOCB0aGF3dGUsIEluYy4gLSBGb3IgYXV0aG9yaXpl\nZCB1c2Ugb25seTEkMCIGA1UEAxMbdGhhd3RlIFByaW1hcnkgUm9vdCBDQSAtIEcz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsr8nLPvb2FvdeHsbnndm\ngcs+vHyu86YnmjSjaDFxODNi5PNxZnmxqWWjpYvVj2AtP0LMqmsywCPLLEHd5N/8\nYZzic7IilRFDGF/Eth9XbAoFWCLINkw6fKXRz4aviKdEAhN0cXMKQlkC+BsUa0Lf\nb1+6a4KinVvnSr0eAXLbS3ToO39/fR8EtCab4LRarEc9VbjXsCZSKAExQGbY2SS9\n9irY7CFJXJv2eul/VTV+lmuNk5Mny5K76qxAwJ/C+IDPXfRa3M50hqY+bAtTyr2S\nzhkGcuYMXDhpxwTWvGzOW/b3aJzcJRVIiKHpqfiYnODz1TEoYRFsZ5aNOZnLwkUk\nOQIDAQABo0IwQDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNV\nHQ4EFgQUrWyqlGCc7eT/+j4KdCtjA/e2Wb8wDQYJKoZIhvcNAQELBQADggEBABpA\n2JVlrAmSicY59BDlqQ5mU1143vokkbvnRFHfxhY0Cu9qRFHqKweKA3rD6z8KLFIW\noCtDuSWQP3CpMyVtRRooOyfPqsMpQhvfO0zAMzRbQYi/aytlryjvsvXDqmbOe1bu\nt8jLZ8HJnBoYuMTDSQPxYA5QzUbF83d597YV4Djbxy8ooAw/dyZ02SUS2jHaGh7c\nKUGRIjxpp7sC8rZcJwOJ9Abqm+RyguOhCcHpABnTPtRwa7pxpqpYrvS76Wy274fM\nm7v/OeZWYdMKp8RcTGB7BXcmer/YB1IsYvdwY9k5vG8cwnncdimvzsUsZAReiDZu\nMdRAGmI0Nj81Aa6sY6A=\n-----END CERTIFICATE-----\n"
  },
  {
    "path": "tests/unit/tls_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2015 Cloudius Systems, Ltd.\n */\n\n#include <ranges>\n#include <iostream>\n\n#include <seastar/core/do_with.hh>\n#include <seastar/core/sstring.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/loop.hh>\n#include <seastar/core/memory.hh>\n#include <seastar/core/sharded.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/core/gate.hh>\n#include <seastar/core/temporary_buffer.hh>\n#include <seastar/core/iostream.hh>\n#include <seastar/core/when_all.hh>\n#include <seastar/core/with_timeout.hh>\n#include <seastar/util/std-compat.hh>\n#include <seastar/util/process.hh>\n#include <seastar/net/tls.hh>\n#include <seastar/net/dns.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/util/defer.hh>\n\n#include <boost/dll.hpp>\n\n#include \"loopback_socket.hh\"\n#include \"tmpdir.hh\"\n\n#include <gnutls/gnutls.h>\n\n#if 0\n\nstatic void enable_gnutls_logging() {\n    gnutls_global_set_log_level(99);\n        gnutls_global_set_log_function([](int lv, const char * msg) {\n           std::cerr << \"GNUTLS (\" << lv << \") \" << msg << std::endl;\n        });\n}\n#endif\n\nstatic const auto cert_location = boost::dll::program_location().parent_path();\n\nstatic std::string certfile(const std::string& file) {\n    return (cert_location / file).string();\n}\n\nusing enable_if_with_networking = boost::unit_test::enable_if<SEASTAR_TESTING_WITH_NETWORKING>;\nusing enable_if_without_networking = boost::unit_test::enable_if<!SEASTAR_TESTING_WITH_NETWORKING>;\n\nusing namespace seastar;\n\nstatic future<> connect_to_ssl_addr(::shared_ptr<tls::certificate_credentials> certs, socket_address addr, const sstring& name = {}) {\n    return repeat_until_value([=]() mutable {\n        return tls::connect(certs, addr, tls::tls_options{.server_name = name}).then([](connected_socket s) {\n            return do_with(std::move(s), [](connected_socket& s) {\n                return do_with(s.output(), [&s](auto& os) {\n                    static const sstring msg(\"GET / HTTP/1.0\\r\\n\\r\\n\");\n                    auto f = os.write(msg);\n                    return f.then([&s, &os]() mutable {\n                        auto f = os.flush();\n                        return f.then([&s]() mutable {\n                            return do_with(s.input(), sstring{}, [](auto& in, sstring& buffer) {\n                                return do_until(std::bind(&input_stream<char>::eof, std::cref(in)), [&buffer, &in] {\n                                    auto f = in.read();\n                                    return f.then([&](temporary_buffer<char> buf) {\n                                        buffer.append(buf.get(), buf.size());\n                                    });\n                                }).then([&buffer]() -> future<std::optional<bool>> {\n                                    if (buffer.empty()) {\n                                        // # 1127 google servers have a (pretty short) timeout between connect and expected first\n                                        // write. If we are delayed inbetween connect and write above (cert verification, scheduling\n                                        // solar spots or just time sharing on AWS) we could get a short read here. Just retry.\n                                        // If we get an actual error, it is either on protocol level (exception) or HTTP error.\n                                        return make_ready_future<std::optional<bool>>(std::nullopt);\n                                    }\n                                    BOOST_CHECK(buffer.size() > 8);\n                                    BOOST_CHECK_EQUAL(buffer.substr(0, 5), sstring(\"HTTP/\"));\n                                    return make_ready_future<std::optional<bool>>(true);\n                                });\n                            });\n                        });\n                    }).finally([&os] {\n                        return os.close();\n                    });\n                });\n            });\n        });\n\n    }).discard_result();\n}\n\nstatic const auto google_name = \"www.google.com\";\n\n// broken out from below. to allow pre-lookup\nstatic future<socket_address> google_address() {\n    static socket_address google;\n\n    if (google.is_unspecified()) {\n        return net::dns::resolve_name(google_name, net::inet_address::family::INET).then([](net::inet_address addr) {\n            google = socket_address(addr, 443);\n            return google_address();\n        });\n    }\n    return make_ready_future<socket_address>(google);\n}\n\nstatic future<> connect_to_ssl_google(::shared_ptr<tls::certificate_credentials> certs) {\n    return google_address().then([certs](socket_address addr) {\n        return connect_to_ssl_addr(std::move(certs), addr, google_name);\n    });\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_with_google,\n                  *enable_if_with_networking()) {\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    return certs->set_x509_trust_file(certfile(\"tls-ca-bundle.pem\"), tls::x509_crt_format::PEM).then([certs]() {\n        return connect_to_ssl_google(certs);\n    });\n}\n\nSEASTAR_TEST_CASE(test_x509_client_with_system_trust,\n                  *enable_if_with_networking()) {\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    return certs->set_system_trust().then([certs]() {\n        return connect_to_ssl_google(certs);\n    });\n}\n\nSEASTAR_TEST_CASE(test_x509_client_with_builder_system_trust,\n                  *enable_if_with_networking()) {\n    tls::credentials_builder b;\n    (void)b.set_system_trust();\n    return connect_to_ssl_google(b.build_certificate_credentials());\n}\n\nSEASTAR_TEST_CASE(test_x509_client_with_builder_system_trust_multiple,\n                  *enable_if_with_networking()) {\n    // avoid getting parallel connects stuck on dns lookup (if running single case).\n    // pre-lookup www.google.com\n    return google_address().then([](socket_address) {\n        tls::credentials_builder b;\n        (void)b.set_system_trust();\n        auto creds = b.build_certificate_credentials();\n\n        return parallel_for_each(std::views::iota(0, 20), [creds](auto i) { return connect_to_ssl_google(creds); });\n    });\n}\n\nSEASTAR_TEST_CASE(test_x509_client_with_system_trust_and_priority_strings,\n                  *enable_if_with_networking()) {\n    static std::vector<sstring> prios( {\n        \"NORMAL:+ARCFOUR-128\", // means normal ciphers plus ARCFOUR-128.\n        \"SECURE128:-VERS-SSL3.0:+COMP-DEFLATE\", // means that only secure ciphers are enabled, SSL3.0 is disabled, and libz compression enabled.\n        \"SECURE256:+SECURE128\",\n        \"NORMAL:%COMPAT\",\n        \"NORMAL:-MD5\",\n        \"NONE:+VERS-TLS-ALL:+MAC-ALL:+RSA:+AES-128-CBC:+SIGN-ALL:+COMP-NULL\",\n        \"NORMAL:+ARCFOUR-128\",\n        \"SECURE128:-VERS-TLS1.0:+COMP-DEFLATE\",\n        \"SECURE128:+SECURE192:-VERS-TLS-ALL:+VERS-TLS1.2\"\n    });\n    return do_for_each(prios, [](const sstring & prio) {\n        tls::credentials_builder b;\n        (void)b.set_system_trust();\n        b.set_priority_string(prio);\n        return connect_to_ssl_google(b.build_certificate_credentials());\n    });\n}\n\nSEASTAR_TEST_CASE(test_x509_client_with_system_trust_and_priority_strings_fail,\n                  *enable_if_with_networking()) {\n    static std::vector<sstring> prios( { \"NONE\",\n        \"NONE:+CURVE-SECP256R1\"\n    });\n    return do_for_each(prios, [](const sstring & prio) {\n        tls::credentials_builder b;\n        (void)b.set_system_trust();\n        b.set_priority_string(prio);\n        try {\n            return connect_to_ssl_google(b.build_certificate_credentials()).then([] {\n                BOOST_FAIL(\"Expected exception\");\n            }).handle_exception([](auto ep) {\n                // ok.\n            });\n        } catch (...) {\n            // also ok\n        }\n        return make_ready_future<>();\n    });\n}\n\nSEASTAR_TEST_CASE(test_alpn_client_negotiate_h2_with_google,\n                  *enable_if_with_networking()) {\n    auto addr = co_await google_address();\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    co_await certs->set_system_trust();\n\n    tls::tls_options client_tls_options;\n    client_tls_options.server_name = google_name; // For SNI\n    client_tls_options.alpn_protocols = {\"h2\", \"http/1.1\"}; // Offer h2 first\n\n    auto c = co_await tls::connect(certs, addr, client_tls_options);\n    auto selected_alpn = co_await tls::get_selected_alpn_protocol(c);\n    BOOST_CHECK(selected_alpn.has_value());\n    BOOST_CHECK(*selected_alpn == \"h2\" || *selected_alpn == \"http/1.1\");\n    BOOST_TEST_MESSAGE(fmt::format(\"Google selected ALPN: {}\", *selected_alpn).c_str());\n    if (*selected_alpn != \"h2\") {\n        BOOST_TEST_MESSAGE(\"Warning: Google did not select 'h2'. Selected: \" + *selected_alpn);\n    }\n    co_return;\n}\n\nSEASTAR_TEST_CASE(test_alpn_client_server) {\n    auto certs = ::make_shared<tls::server_credentials>(::make_shared<tls::dh_params>());\n    co_await certs->set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM);\n    certs->set_alpn_protocols({\"h2\", \"http/1.1\"});\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    auto addr = ::make_ipv4_address({0x7f000001, 4712});\n    auto server = tls::listen(certs, addr, opts);\n\n    co_await when_all(\n        [server = std::move(server)]() mutable -> future<> {\n            auto s = co_await server.accept();\n            auto alpn = co_await tls::get_selected_alpn_protocol(s.connection);\n            BOOST_CHECK(alpn.has_value());\n            BOOST_CHECK_EQUAL(*alpn, \"h2\");\n            co_return;\n        },\n        [addr]() mutable -> future<> {\n            tls::credentials_builder b;\n            co_await b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM);\n            tls::tls_options client_opts{.alpn_protocols = {\"hx\", \"h2\"}};\n            auto c = co_await tls::connect(b.build_certificate_credentials(), addr, client_opts);\n            auto alpn = co_await tls::get_selected_alpn_protocol(c);\n            BOOST_CHECK(alpn.has_value());\n            BOOST_CHECK_EQUAL(*alpn, \"h2\");\n            co_return;\n        }\n    );\n    co_return;\n}\n\nclass https_server {\n    const sstring _cert;\n    const std::string _addr = \"127.0.0.1\";\n    experimental::process _process;\n    uint16_t _port;\n\n    static experimental::process spawn(const std::string& addr, const sstring& key, const sstring& cert) {\n        auto httpd = boost::dll::program_location().parent_path() / \"https-server.py\";\n        const std::vector<sstring> argv{\n          \"httpd\",\n          \"--server\", fmt::format(\"{}:{}\", addr, 0),\n          \"--key\", key,\n          \"--cert\", cert,\n        };\n        return experimental::spawn_process(httpd.string(), {.argv = argv}).get();\n    }\n\n    // https-server.py picks an available port and listens on it. when it is\n    // ready to serve, it prints out the listening port. without hardwiring to\n    // a fixed port, we are able to run multiple tests in parallel.\n    static uint16_t read_port(experimental::process& process) {\n        using consumption_result_type = typename input_stream<char>::consumption_result_type;\n        using stop_consuming_type = typename consumption_result_type::stop_consuming_type;\n        using tmp_buf = stop_consuming_type::tmp_buf;\n        struct consumer {\n            future<consumption_result_type> operator()(tmp_buf buf) {\n                if (auto newline = std::find(buf.begin(), buf.end(), '\\n'); newline != buf.end()) {\n                    size_t consumed = newline - buf.begin();\n                    line += std::string_view(buf.get(), consumed);\n                    buf.trim_front(consumed);\n                    return make_ready_future<consumption_result_type>(stop_consuming_type(std::move(buf)));\n                } else {\n                    line += std::string_view(buf.get(), buf.size());\n                    return make_ready_future<consumption_result_type>(stop_consuming_type({}));\n                }\n            }\n            std::string line;\n        };\n        auto reader = ::make_shared<consumer>();\n        process.cout().consume(*reader).get();\n        return std::stoul(reader->line);\n    }\n\npublic:\n    https_server(const std::string& ca = \"mtls_ca\")\n        : _cert(certfile(fmt::format(\"{}.crt\", ca)))\n        , _process(spawn(_addr, certfile(fmt::format(\"{}.key\", ca)), _cert))\n        , _port(read_port(_process))\n    {}\n    ~https_server() {\n        _process.terminate();\n        _process.wait().discard_result().get();\n    }\n    const sstring& cert() const {\n        return _cert;\n    }\n    socket_address addr() const {\n        return ipv4_addr(_addr, _port);\n    }\n    sstring name() const {\n        // should be identical to the one passed as the \"-addext\" option when\n        // generating the cert.\n        return \"127.0.0.1\";\n    }\n};\n\nSEASTAR_THREAD_TEST_CASE(test_simple_x509_client_with_local_server,\n                         *enable_if_without_networking()) {\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    https_server server;\n    certs->set_x509_trust_file(server.cert(), tls::x509_crt_format::PEM).get();\n    connect_to_ssl_addr(certs, server.addr(), server.name()).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_x509_client_with_builder) {\n    tls::credentials_builder b;\n    https_server server;\n    b.set_x509_trust_file(server.cert(), tls::x509_crt_format::PEM).get();\n    connect_to_ssl_addr(b.build_certificate_credentials(), server.addr()).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_x509_client_with_builder_multiple) {\n    tls::credentials_builder b;\n    https_server server;\n    b.set_x509_trust_file(server.cert(), tls::x509_crt_format::PEM).get();\n    auto creds = b.build_certificate_credentials();\n    auto addr = server.addr();\n    parallel_for_each(std::views::iota(0, 20), [creds, addr](auto i) {\n        return connect_to_ssl_addr(creds, addr);\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_x509_client_with_priority_strings) {\n    static std::vector<sstring> prios( {\n        \"NORMAL:+ARCFOUR-128\", // means normal ciphers plus ARCFOUR-128.\n        \"SECURE128:-VERS-SSL3.0:+COMP-DEFLATE\", // means that only secure ciphers are enabled, SSL3.0 is disabled, and libz compression enabled.\n        \"SECURE256:+SECURE128\",\n        \"NORMAL:%COMPAT\",\n        \"NORMAL:-MD5\",\n        \"NONE:+VERS-TLS-ALL:+MAC-ALL:+RSA:+AES-256-GCM:+SIGN-ALL:+COMP-NULL:+GROUP-EC-ALL\",\n        \"NORMAL:+ARCFOUR-128\",\n        \"SECURE128:-VERS-TLS1.0:+COMP-DEFLATE\",\n        \"SECURE128:+SECURE192:-VERS-TLS-ALL:+VERS-TLS1.2\"\n    });\n    tls::credentials_builder b;\n    https_server server;\n    b.set_x509_trust_file(server.cert(), tls::x509_crt_format::PEM).get();\n    auto addr = server.addr();\n    do_for_each(prios, [&b, addr](const sstring& prio) {\n        b.set_priority_string(prio);\n        return connect_to_ssl_addr(b.build_certificate_credentials(), addr);\n    }).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_x509_client_with_priority_strings_fail) {\n    static std::vector<sstring> prios( { \"NONE\",\n        \"NONE:+CURVE-SECP256R1\"\n    });\n    tls::credentials_builder b;\n    https_server server;\n    b.set_x509_trust_file(server.cert(), tls::x509_crt_format::PEM).get();\n    auto addr = server.addr();\n    do_for_each(prios, [&b, addr](const sstring& prio) {\n        b.set_priority_string(prio);\n        try {\n            return connect_to_ssl_addr(b.build_certificate_credentials(), addr).then([] {\n                BOOST_FAIL(\"Expected exception\");\n            }).handle_exception([](auto ep) {\n                // ok.\n            });\n        } catch (...) {\n            // also ok\n        }\n        return make_ready_future<>();\n    }).get();\n}\n\nSEASTAR_TEST_CASE(test_failed_connect) {\n    tls::credentials_builder b;\n    (void)b.set_system_trust();\n    return connect_to_ssl_addr(b.build_certificate_credentials(), ipv4_addr()).handle_exception([](auto) {});\n}\n\nSEASTAR_TEST_CASE(test_non_tls) {\n    ::listen_options opts;\n    opts.reuse_address = true;\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = server_socket(seastar::listen(addr, opts));\n\n    auto c = server.accept();\n\n    tls::credentials_builder b;\n    (void)b.set_system_trust();\n\n    auto f = connect_to_ssl_addr(b.build_certificate_credentials(), addr);\n\n\n    return c.then([f = std::move(f)](accept_result ar) mutable {\n        ::connected_socket s = std::move(ar.connection);\n        std::cerr << \"Established connection\" << std::endl;\n        auto sp = std::make_unique<::connected_socket>(std::move(s));\n        timer<> t([s = std::ref(*sp)] {\n            std::cerr << \"Killing server side\" << std::endl;\n            s.get() = ::connected_socket();\n        });\n        t.arm(timer<>::clock::now() + std::chrono::seconds(5));\n        return std::move(f).finally([t = std::move(t), sp = std::move(sp)] {});\n    }).handle_exception([server = std::move(server)](auto ep) {\n        std::cerr << \"Got expected exception\" << std::endl;\n    });\n}\n\nSEASTAR_TEST_CASE(test_abort_accept_before_handshake) {\n    auto certs = ::make_shared<tls::server_credentials>(::make_shared<tls::dh_params>());\n    return certs->set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).then([certs] {\n        ::listen_options opts;\n        opts.reuse_address = true;\n        auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n        auto server = server_socket(tls::listen(certs, addr, opts));\n        auto c = server.accept();\n        BOOST_CHECK(!c.available()); // should not be finished\n\n        server.abort_accept();\n\n        return c.then([](auto) { BOOST_FAIL(\"Should not reach\"); }).handle_exception([](auto) {\n            // ok\n        }).finally([server = std::move(server)] {});\n    });\n}\n\nSEASTAR_TEST_CASE(test_abort_accept_after_handshake) {\n    return async([] {\n        auto certs = ::make_shared<tls::server_credentials>(::make_shared<tls::dh_params>());\n        certs->set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n\n        ::listen_options opts;\n        opts.reuse_address = true;\n        auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n        auto server = tls::listen(certs, addr, opts);\n        auto sa = server.accept();\n\n        tls::credentials_builder b;\n        b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n\n        auto c = tls::connect(b.build_certificate_credentials(), addr).get();\n        auto s = sa.get();\n        server.abort_accept(); // should not affect the socket we got.\n        auto out = c.output();\n        auto in = s.connection.input();\n\n        out.write(\"apa\").get();\n        auto f = out.flush();\n        auto buf = in.read().get();\n        f.get();\n        BOOST_CHECK(sstring(buf.begin(), buf.end()) == \"apa\");\n\n        out.close().get();\n        in.close().get();\n    });\n}\n\nSEASTAR_TEST_CASE(test_abort_accept_on_server_before_handshake) {\n    return async([] {\n        ::listen_options opts;\n        opts.reuse_address = true;\n        auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n        auto server = server_socket(seastar::listen(addr, opts));\n        auto sa = server.accept();\n\n        tls::credentials_builder b;\n        b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n\n        auto creds = b.build_certificate_credentials();\n        auto f = tls::connect(creds, addr);\n\n        server.abort_accept();\n        try {\n            sa.get();\n        } catch (...) {\n        }\n        server = {};\n\n        try {\n            // the connect as such should succeed, but the handshare following it\n            // should not.\n            auto c = f.get();\n            auto out = c.output();\n            out.write(\"apa\").get();\n            out.flush().get();\n            out.close().get();\n\n            BOOST_FAIL(\"Expected exception\");\n        } catch (...) {\n            // ok\n        }\n    });\n}\n\n\nstruct streams {\n    ::connected_socket s;\n    input_stream<char> in;\n    output_stream<char> out;\n\n    // note: using custom output_stream, because we don't want polled flush\n    streams(::connected_socket cs) : s(std::move(cs)), in(s.input()), out(s.output().detach(), 8192)\n    {}\n};\n\nstatic const sstring message = \"hej lilla fisk du kan dansa fint\";\n\nclass echoserver {\n    ::server_socket _socket;\n    ::shared_ptr<tls::server_credentials> _certs;\n    seastar::gate _gate;\n    bool _stopped = false;\n    size_t _size;\n    std::exception_ptr _ex;\npublic:\n    echoserver(size_t message_size, bool use_dh_params = true)\n            : _certs(\n                    use_dh_params\n                        ? ::make_shared<tls::server_credentials>(::make_shared<tls::dh_params>())\n                        : ::make_shared<tls::server_credentials>()\n                    )\n            , _size(message_size)\n    {}\n\n    future<> listen(socket_address addr, sstring crtfile, sstring keyfile, tls::client_auth ca = tls::client_auth::NONE, sstring trust = {}) {\n        _certs->set_client_auth(ca);\n        auto f = _certs->set_x509_key_file(crtfile, keyfile, tls::x509_crt_format::PEM);\n        if (!trust.empty()) {\n            f = f.then([this, trust = std::move(trust)] {\n                return _certs->set_x509_trust_file(trust, tls::x509_crt_format::PEM);\n            });\n        }\n        return f.then([this, addr] {\n            ::listen_options opts;\n            opts.reuse_address = true;\n\n            _socket = tls::listen(_certs, addr, opts);\n\n            (void)try_with_gate(_gate, [this] {\n                return _socket.accept().then([this](accept_result ar) {\n                    ::connected_socket s = std::move(ar.connection);\n                    auto strms = ::make_lw_shared<streams>(std::move(s));\n                    return repeat([strms, this]() {\n                        return strms->in.read_exactly(_size).then([strms](temporary_buffer<char> buf) {\n                            if (buf.empty()) {\n                                return make_ready_future<stop_iteration>(stop_iteration::yes);\n                            }\n                            sstring tmp(buf.begin(), buf.end());\n                            return strms->out.write(tmp).then([strms]() {\n                                return strms->out.flush();\n                            }).then([] {\n                                return make_ready_future<stop_iteration>(stop_iteration::no);\n                            });\n                        });\n                    }).finally([strms]{\n                        return strms->out.close();\n                    }).finally([strms]{});\n                }).handle_exception([this](auto ep) {\n                    if (_stopped) {\n                        return make_ready_future<>();\n                    }\n                    _ex = ep;\n                    return make_ready_future<>();\n                });\n            }).handle_exception_type([] (const gate_closed_exception&) {/* ignore */});\n            return make_ready_future<>();\n        });\n    }\n\n    future<> stop() {\n        _stopped = true;\n        _socket.abort_accept();\n        return _gate.close().handle_exception([this] (std::exception_ptr ignored) {\n            if (_ex) {\n                std::rethrow_exception(_ex);\n            }\n        });\n    }\n};\n\nstatic future<> run_echo_test(sstring message,\n                int loops,\n                sstring trust,\n                sstring name,\n                sstring crt = certfile(\"test.crt\"),\n                sstring key = certfile(\"test.key\"),\n                tls::client_auth ca = tls::client_auth::NONE,\n                sstring client_crt = {},\n                sstring client_key = {},\n                bool do_read = true,\n                bool use_dh_params = true,\n                tls::dn_callback distinguished_name_callback = {}\n)\n{\n    static const auto port = 4711;\n\n    auto msg = ::make_shared<sstring>(std::move(message));\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    auto server = ::make_shared<seastar::sharded<echoserver>>();\n    auto addr = ::make_ipv4_address( {0x7f000001, port});\n\n    SEASTAR_ASSERT(do_read || loops == 1);\n\n    future<> f = make_ready_future();\n\n    if (!client_crt.empty() && !client_key.empty()) {\n        f = certs->set_x509_key_file(client_crt, client_key, tls::x509_crt_format::PEM);\n        if (distinguished_name_callback) {\n            certs->set_dn_verification_callback(std::move(distinguished_name_callback));\n        }\n    }\n\n    return f.then([=] {\n        return certs->set_x509_trust_file(trust, tls::x509_crt_format::PEM);\n    }).then([=] {\n        return server->start(msg->size(), use_dh_params).then([=]() {\n            sstring server_trust;\n            if (ca != tls::client_auth::NONE) {\n                server_trust = trust;\n            }\n            return server->invoke_on_all(&echoserver::listen, addr, crt, key, ca, server_trust);\n        }).then([=] {\n            return tls::connect(certs, addr, tls::tls_options{.server_name=name}).then([loops, msg, do_read](::connected_socket s) {\n                auto strms = ::make_lw_shared<streams>(std::move(s));\n                auto range = std::views::iota(0, loops);\n                return do_for_each(range, [strms, msg](auto) {\n                    auto f = strms->out.write(*msg);\n                    return f.then([strms, msg]() {\n                        return strms->out.flush().then([strms, msg] {\n                            return strms->in.read_exactly(msg->size()).then([msg](temporary_buffer<char> buf) {\n                                if (buf.empty()) {\n                                    throw std::runtime_error(\"Unexpected EOF\");\n                                }\n                                sstring tmp(buf.begin(), buf.end());\n                                BOOST_CHECK(*msg == tmp);\n                            });\n                        });\n                    });\n                }).then_wrapped([strms, do_read] (future<> f1) {\n                    // Always call close()\n                    return (do_read ? strms->out.close() : make_ready_future<>()).then_wrapped([strms, f1 = std::move(f1)] (future<> f2) mutable {\n                        // Verification errors will be reported by the call to output_stream::close(),\n                        // which waits for the flush to actually happen. They can also be reported by the\n                        // input_stream::read_exactly() call. We want to keep only one and avoid nested exception mess.\n                        if (f1.failed()) {\n                            (void)f2.handle_exception([] (std::exception_ptr ignored) { });\n                            return std::move(f1);\n                        }\n                        (void)f1.handle_exception([] (std::exception_ptr ignored) { });\n                        return f2;\n                    }).finally([strms] { });\n                });\n            });\n        }).finally([server] {\n            return server->stop().finally([server]{});\n        });\n    });\n}\n\n/*\n * Certificates:\n *\n * make -f tests/unit/mkcert.gmk domain=scylladb.org server=test\n *\n * ->   test.crt\n *      test.csr\n *      catest.pem\n *      catest.key\n *\n * catest == snakeoil root authority for these self-signed certs\n *\n */\nSEASTAR_TEST_CASE(test_simple_x509_client_server) {\n    // Make sure we load our own auth trust pem file, otherwise our certs\n    // will not validate\n    // Must match expected name with cert CA or give empty name to ignore\n    // server name\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\");\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_server_again) {\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\");\n}\n\n#if GNUTLS_VERSION_NUMBER >= 0x030600\n// Test #769 - do not set dh_params in server certs - let gnutls negotiate.\nSEASTAR_TEST_CASE(test_simple_server_default_dhparams) {\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\",\n        certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::NONE,\n        {}, {}, true, /* use_dh_params */ false\n    );\n}\n#endif\n\nSEASTAR_TEST_CASE(test_x509_client_server_cert_validation_fail) {\n    // Load a real trust authority here, which our certs are _not_ signed with.\n    return run_echo_test(message, 1, certfile(\"tls-ca-bundle.pem\"), {}).then([] {\n            BOOST_FAIL(\"Should have gotten validation error\");\n    }).handle_exception([](auto ep) {\n        try {\n            std::rethrow_exception(ep);\n        } catch (tls::verification_error& e) {\n            // Verify exception contains info on subject/issuer\n            BOOST_REQUIRE_NE(sstring(e.what()).find(\"Issuer\"), sstring::npos);\n            BOOST_REQUIRE_NE(sstring(e.what()).find(\"Subject\"), sstring::npos);\n            // ok.\n        } catch (...) {\n            BOOST_FAIL(\"Unexpected exception\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_x509_client_server_cert_validation_fail_name) {\n    // Use trust store with our signer, but wrong host name\n    return run_echo_test(message, 1, certfile(\"tls-ca-bundle.pem\"), \"nils.holgersson.gov\").then([] {\n            BOOST_FAIL(\"Should have gotten validation error\");\n    }).handle_exception([](auto ep) {\n        try {\n            std::rethrow_exception(ep);\n        } catch (tls::verification_error&) {\n            // ok.\n        } catch (...) {\n            BOOST_FAIL(\"Unexpected exception\");\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_large_message_x509_client_server) {\n    // Make sure we load our own auth trust pem file, otherwise our certs\n    // will not validate\n    // Must match expected name with cert CA or give empty name to ignore\n    // server name\n    sstring msg = uninitialized_string(512 * 1024);\n    for (size_t i = 0; i < msg.size(); ++i) {\n        msg[i] = '0' + char(i % 30);\n    }\n    return run_echo_test(std::move(msg), 20, certfile(\"catest.pem\"), \"test.scylladb.org\");\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_server_fail_client_auth) {\n    // Make sure we load our own auth trust pem file, otherwise our certs\n    // will not validate\n    // Must match expected name with cert CA or give empty name to ignore\n    // server name\n    // Server will require certificate auth. We supply none, so should fail connection\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\", certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::REQUIRE).then([] {\n        BOOST_FAIL(\"Expected exception\");\n    }).handle_exception([](auto ep) {\n        // ok.\n    });\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_server_client_auth) {\n    // Make sure we load our own auth trust pem file, otherwise our certs\n    // will not validate\n    // Must match expected name with cert CA or give empty name to ignore\n    // server name\n    // Server will require certificate auth. We supply one, so should succeed with connection\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\", certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::REQUIRE, certfile(\"test.crt\"), certfile(\"test.key\"));\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_server_client_auth_with_dn_callback) {\n    // In addition to the above test, the certificate's subject and issuer\n    // Distinguished Names (DNs) will be checked for the occurrence of a specific\n    // substring (in this case, the test.scylladb.org url)\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\", certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::REQUIRE, certfile(\"test.crt\"), certfile(\"test.key\"), true, true, [](tls::session_type t, sstring subject, sstring issuer) {\n        BOOST_REQUIRE(t == tls::session_type::CLIENT);\n        BOOST_REQUIRE(subject.find(\"test.scylladb.org\") != sstring::npos);\n        BOOST_REQUIRE(issuer.find(\"test.scylladb.org\") != sstring::npos);\n    });\n}\n\nSEASTAR_TEST_CASE(test_simple_x509_client_server_client_auth_dn_callback_fails) {\n    // Test throwing an exception from within the Distinguished Names callback\n    return run_echo_test(message, 20, certfile(\"catest.pem\"), \"test.scylladb.org\", certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::REQUIRE, certfile(\"test.crt\"), certfile(\"test.key\"), true, true, [](tls::session_type, sstring, sstring) {\n        throw tls::verification_error(\"to test throwing from within the callback\");\n    }).then([] {\n        BOOST_FAIL(\"Should have gotten a verification_error exception\");\n    }).handle_exception([](auto) {\n        // ok.\n    });\n}\n\nSEASTAR_TEST_CASE(test_many_large_message_x509_client_server) {\n    // Make sure we load our own auth trust pem file, otherwise our certs\n    // will not validate\n    // Must match expected name with cert CA or give empty name to ignore\n    // server name\n    sstring msg = uninitialized_string(4 * 1024 * 1024);\n    for (size_t i = 0; i < msg.size(); ++i) {\n        msg[i] = '0' + char(i % 30);\n    }\n    // Sending a huge-ish message a and immediately closing the session (see params)\n    // provokes case where tls::vec_push entered race and asserted on broken IO state\n    // machine.\n    auto range = std::views::iota(0, 20);\n    return do_for_each(range, [msg = std::move(msg)](auto) {\n        return run_echo_test(std::move(msg), 1, certfile(\"catest.pem\"), \"test.scylladb.org\", certfile(\"test.crt\"), certfile(\"test.key\"), tls::client_auth::NONE, {}, {}, false);\n    });\n}\n\nSEASTAR_THREAD_TEST_CASE(test_close_timout) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n    b.set_system_trust().get();\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    semaphore sem(0);\n\n    class my_loopback_connected_socket_impl : public loopback_connected_socket_impl {\n    public:\n        semaphore& _sem;\n        bool _close = false;\n\n        my_loopback_connected_socket_impl(semaphore& s, lw_shared_ptr<loopback_buffer> tx, lw_shared_ptr<loopback_buffer> rx)\n            : loopback_connected_socket_impl(tx, rx)\n            , _sem(s)\n        {}\n        ~my_loopback_connected_socket_impl() {\n            _sem.signal();\n        }\n        class my_sink_impl : public data_sink_impl {\n        public:\n            data_sink _sink;\n            my_loopback_connected_socket_impl& _impl;\n            promise<> _p;\n            my_sink_impl(data_sink sink, my_loopback_connected_socket_impl& impl)\n                : _sink(std::move(sink))\n                , _impl(impl)\n            {}\n            future<> flush() override {\n                return _sink.flush();\n            }\n#if SEASTAR_API_LEVEL >= 9\n            future<> put(std::span<temporary_buffer<char>> bufs) override {\n                std::vector<temporary_buffer<char>> stable_bufs(std::make_move_iterator(bufs.begin()), std::make_move_iterator(bufs.end()));\n                if (!std::exchange(_impl._close, false)) {\n                    return _sink.put(std::move(stable_bufs));\n                }\n\n                return _p.get_future().then([this, bufs = std::move(stable_bufs)] () mutable {\n                    return put(std::span(bufs));\n                });\n            }\n#else\n            using data_sink_impl::put;\n            future<> put(net::packet p) override {\n                if (std::exchange(_impl._close, false)) {\n                    return _p.get_future().then([this, p = std::move(p)]() mutable {\n                        return put(std::move(p));\n                    });\n                }\n                return _sink.put(std::move(p));\n            }\n#endif\n            future<> close() override {\n                _p.set_value();\n                return make_ready_future<>();\n            }\n        };\n        data_sink sink() override {\n            return data_sink(std::make_unique<my_sink_impl>(loopback_connected_socket_impl::sink(), *this));\n        }\n    };\n\n    auto constexpr iterations = 500;\n\n    for (int i = 0; i < iterations; ++i) {\n        auto b1 = ::make_lw_shared<loopback_buffer>(nullptr, loopback_buffer::type::SERVER_TX);\n        auto b2 = ::make_lw_shared<loopback_buffer>(nullptr, loopback_buffer::type::CLIENT_TX);\n        auto ssi = std::make_unique<my_loopback_connected_socket_impl>(sem, b1, b2);\n        auto csi = std::make_unique<my_loopback_connected_socket_impl>(sem, b2, b1);\n\n        auto& ssir = *ssi;\n        auto& csir = *csi;\n\n        auto ss = tls::wrap_server(serv, connected_socket(std::move(ssi))).get();\n        auto cs = tls::wrap_client(creds, connected_socket(std::move(csi))).get();\n\n        auto os = cs.output().detach();\n        auto is = ss.input();\n\n        auto f1 = os.put(temporary_buffer<char>(10));\n        auto f2 = is.read();\n        f1.get();\n        f2.get();\n\n        // block further writes\n        ssir._close = true;\n        csir._close = true;\n    }\n\n    sem.wait(2 * iterations).get();\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reload_certificates) {\n    tmpdir tmp;\n\n    namespace fs = std::filesystem;\n\n    // copy the wrong certs. We don't trust these\n    // blocking calls, but this is a test and seastar does not have a copy\n    // util and I am lazy...\n    fs::copy_file(certfile(\"other.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"other.key\"), tmp.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    std::unordered_set<sstring> changed;\n    promise<> p;\n\n    tls::credentials_builder b;\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n\n    auto certs = b.build_reloadable_server_credentials([&](const std::unordered_set<sstring>& files, std::exception_ptr ep) {\n        if (ep) {\n            return;\n        }\n        changed.insert(files.begin(), files.end());\n        if (changed.count(cert) && changed.count(key)) {\n            p.set_value();\n        }\n    }).get();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(certs, addr, opts);\n\n    tls::credentials_builder b2;\n    b2.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(b2.build_certificate_credentials(), addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n\n        try {\n            out.write(\"apa\").get();\n            auto f = out.flush();\n            auto f2 = in.read();\n\n            try {\n                f.get();\n                BOOST_FAIL(\"should not reach\");\n            } catch (tls::verification_error&) {\n                // ok\n            }\n            try {\n                out.close().get();\n            } catch (...) {\n            }\n\n            try {\n                f2.get();\n                BOOST_FAIL(\"should not reach\");\n            } catch (...) {\n                // ok\n            }\n            try {\n                in.close().get();\n            } catch (...) {\n            }\n        } catch (tls::verification_error&) {\n            // ok\n        }\n    }\n\n    // copy the right (trusted) certs over the old ones.\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test0.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test0.key\");\n\n    rename_file((tmp.path() / \"test0.crt\").native(), (tmp.path() / \"test.crt\").native()).get();\n    rename_file((tmp.path() / \"test0.key\").native(), (tmp.path() / \"test.key\").native()).get();\n\n    p.get_future().get();\n\n    // now it should work\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(b2.build_certificate_credentials(), addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n\n        out.write(\"apa\").get();\n        auto f = out.flush();\n        auto buf = in.read().get();\n        f.get();\n        out.close().get();\n        in.read().get(); // ignore - just want eof\n        in.close().get();\n\n        BOOST_CHECK_EQUAL(sstring(buf.begin(), buf.end()), \"apa\");\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reload_broken_certificates) {\n    tmpdir tmp;\n\n    namespace fs = std::filesystem;\n\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    std::unordered_set<sstring> changed;\n    promise<> p;\n\n    tls::credentials_builder b;\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n\n    queue<std::exception_ptr> q(10);\n\n    auto certs = b.build_reloadable_server_credentials([&](const std::unordered_set<sstring>& files, std::exception_ptr ep) {\n        if (ep) {\n            q.push(std::move(ep));\n            return;\n        }\n        changed.insert(files.begin(), files.end());\n        if (changed.count(cert) && changed.count(key)) {\n            p.set_value();\n        }\n    }).get();\n\n    // very intentionally use blocking calls. We want all our modifications to happen\n    // before any other continuation is allowed to process.\n\n    fs::remove(cert);\n    fs::remove(key);\n\n    std::ofstream(cert.c_str()) << \"lala land\" << std::endl;\n    std::ofstream(key.c_str()) << \"lala land\" << std::endl;\n\n    // should get one or two exceptions\n    q.pop_eventually().get();\n\n    fs::remove(cert);\n    fs::remove(key);\n\n    fs::copy_file(certfile(\"test.crt\"), cert);\n    fs::copy_file(certfile(\"test.key\"), key);\n\n    // now it should reload\n    p.get_future().get();\n}\n\nusing namespace std::chrono_literals;\n\n// the same as previous test, but we set a big tolerance for\n// reload errors, and verify that either our scheduling/fs is\n// super slow, or we got through the changes without failures.\nSEASTAR_THREAD_TEST_CASE(test_reload_tolerance) {\n    tmpdir tmp;\n\n    namespace fs = std::filesystem;\n\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    std::unordered_set<sstring> changed;\n    promise<> p;\n\n    tls::credentials_builder b;\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n\n    int nfails = 0;\n\n    // use 5s tolerance - this should ensure we don't generate any errors.\n    auto certs = b.build_reloadable_server_credentials([&](const std::unordered_set<sstring>& files, std::exception_ptr ep) {\n        if (ep) {\n            ++nfails;\n            return;\n        }\n        changed.insert(files.begin(), files.end());\n        if (changed.count(cert) && changed.count(key)) {\n            p.set_value();\n        }\n    }, std::chrono::milliseconds(5000)).get();\n\n    // very intentionally use blocking calls. We want all our modifications to happen\n    // before any other continuation is allowed to process.\n\n    auto start = std::chrono::system_clock::now();\n\n    fs::remove(cert);\n    fs::remove(key);\n\n    std::ofstream(cert.c_str()) << \"lala land\" << std::endl;\n    std::ofstream(key.c_str()) << \"lala land\" << std::endl;\n\n    fs::remove(cert);\n    fs::remove(key);\n\n    fs::copy_file(certfile(\"test.crt\"), cert);\n    fs::copy_file(certfile(\"test.key\"), key);\n\n    // now it should reload\n    p.get_future().get();\n\n    auto end = std::chrono::system_clock::now();\n\n    SEASTAR_ASSERT(nfails == 0 || (end - start) > 4s);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reload_by_move) {\n    tmpdir tmp;\n    tmpdir tmp2;\n\n    namespace fs = std::filesystem;\n\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test.key\");\n    fs::copy_file(certfile(\"test.crt\"), tmp2.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp2.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    auto cert2 = (tmp2.path() / \"test.crt\").native();\n    auto key2 = (tmp2.path() / \"test.key\").native();\n\n    std::unordered_set<sstring> changed;\n    promise<> p;\n\n    tls::credentials_builder b;\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n\n    int nfails = 0;\n\n    // use 5s tolerance - this should ensure we don't generate any errors.\n    auto certs = b.build_reloadable_server_credentials([&](const std::unordered_set<sstring>& files, std::exception_ptr ep) {\n        if (ep) {\n            ++nfails;\n            return;\n        }\n        changed.insert(files.begin(), files.end());\n        if (changed.count(cert) && changed.count(key)) {\n            p.set_value();\n        }\n    }, std::chrono::milliseconds(5000)).get();\n\n    // very intentionally use blocking calls. We want all our modifications to happen\n    // before any other continuation is allowed to process.\n\n    fs::remove(cert);\n    fs::remove(key);\n\n    // deletes should _not_ cause errors/reloads\n    try {\n        with_timeout(std::chrono::steady_clock::now() + 3s, p.get_future()).get();\n        BOOST_FAIL(\"should not reach\");\n    } catch (timed_out_error&) {\n        // ok\n    }\n\n    BOOST_REQUIRE_EQUAL(changed.size(), 0);\n\n    p = promise();\n\n    fs::rename(cert2, cert);\n    fs::rename(key2, key);\n\n    // now it should reload\n    p.get_future().get();\n\n    BOOST_REQUIRE_EQUAL(changed.size(), 2);\n    changed.clear();\n\n    // again, without delete\n\n    fs::copy_file(certfile(\"test.crt\"), tmp2.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp2.path() / \"test.key\");\n\n    p = promise();\n\n    fs::rename(cert2, cert);\n    fs::rename(key2, key);\n\n    // it should reload here as well.\n    p.get_future().get();\n\n    // could get two notifications. but not more.\n    for (int i = 0;; ++i) {\n        p = promise();\n        try {\n            with_timeout(std::chrono::steady_clock::now() + 3s, p.get_future()).get();\n            SEASTAR_ASSERT(i == 0);\n        } catch (timed_out_error&) {\n            // ok\n            break;\n        }\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_closed_write) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n    b.set_system_trust().get();\n    b.set_client_auth(tls::client_auth::REQUIRE);\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    auto check_same_message_two_writes = [](output_stream<char>& out) {\n        std::exception_ptr ep1, ep2;\n\n        try {\n            out.write(\"apa\").get();\n            out.flush().get();\n            BOOST_FAIL(\"should not reach\");\n        } catch (...) {\n            // ok\n            ep1 = std::current_exception();\n        }\n\n        try {\n            out.write(\"apa\").get();\n            out.flush().get();\n            BOOST_FAIL(\"should not reach\");\n        } catch (...) {\n            // ok\n            ep2 = std::current_exception();\n        }\n\n        try {\n            std::rethrow_exception(ep1);\n        } catch (std::exception& e1) {\n            try {\n                std::rethrow_exception(ep2);\n            } catch (std::exception& e2) {\n                BOOST_REQUIRE_EQUAL(std::string(e1.what()), std::string(e2.what()));\n                return;\n            }\n        }\n\n        BOOST_FAIL(\"should not reach\");\n    };\n\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n        // close on client end before writing\n        out.close().get();\n\n        check_same_message_two_writes(out);\n    }\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n\n        out.write(\"apa\").get();\n        auto f = out.flush();\n        in.read().get();\n        f.get();\n\n        // close on server end before writing\n        in.close().get();\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        // we won't get broken pipe until\n        // after a while (tm)\n        for (;;) {\n            try {\n                out.write(\"apa\").get();\n                out.flush().get();\n            } catch (...) {\n                break;\n            }\n        }\n\n        // now check we get the same message.\n        check_same_message_two_writes(out);\n    }\n\n}\n\n/*\n * Certificates:\n *\n * make -f tests/unit/mkmtls.gmk domain=scylladb.org server=test\n *\n * ->   mtls_ca.crt\n *      mtls_ca.key\n *      mtls_server.crt\n *      mtls_server.csr\n *      mtls_server.key\n *      mtls_client1.crt\n *      mtls_client1.csr\n *      mtls_client1.key\n *      mtls_client2.crt\n *      mtls_client2.csr\n *      mtls_client2.key\n *\n */\nSEASTAR_THREAD_TEST_CASE(test_dn_name_handling) {\n    // Connect to server using two different client certificates.\n    // Make sure that for every client the server can fetch DN string\n    // and the string is correct.\n    // The setup consist of several certificates:\n    // - CA\n    // - mtls_server.crt - server certificate\n    // - mtls_client1.crt - first client certificate\n    // - mtls_client2.crt - second client certificate\n    //\n    // The test runs server that uses mtls_server.crt.\n    // The server accepts two incoming connections, first one uses mtls_client1.crt\n    // and the second one uses mtls_client2.crt. Every client sends a short string\n    // that server receives and tries to find it in the DN string.\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n\n    auto client1_creds = [] {\n        tls::credentials_builder builder;\n        builder.set_x509_trust_file(certfile(\"mtls_ca.crt\"), tls::x509_crt_format::PEM).get();\n        builder.set_x509_key_file(certfile(\"mtls_client1.crt\"), certfile(\"mtls_client1.key\"), tls::x509_crt_format::PEM).get();\n        return builder.build_certificate_credentials();\n    }();\n\n    auto client2_creds = [] {\n        tls::credentials_builder builder;\n        builder.set_x509_trust_file(certfile(\"mtls_ca.crt\"), tls::x509_crt_format::PEM).get();\n        builder.set_x509_key_file(certfile(\"mtls_client2.crt\"), certfile(\"mtls_client2.key\"), tls::x509_crt_format::PEM).get();\n        return builder.build_certificate_credentials();\n    }();\n\n    auto server_creds = [] {\n        tls::credentials_builder builder;\n        builder.set_x509_trust_file(certfile(\"mtls_ca.crt\"), tls::x509_crt_format::PEM).get();\n        builder.set_x509_key_file(certfile(\"mtls_server.crt\"), certfile(\"mtls_server.key\"), tls::x509_crt_format::PEM).get();\n        builder.set_client_auth(tls::client_auth::REQUIRE);\n        return builder.build_server_credentials();\n    }();\n\n    auto fetch_dn = [server_creds, addr] (sstring id, shared_ptr<tls::certificate_credentials> client_cred) {\n        listen_options lo{};\n        lo.reuse_address = true;\n        auto server_sock = tls::listen(server_creds, addr, lo);\n\n        auto sa = server_sock.accept();\n        auto c = tls::connect(client_cred, addr).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        output_stream<char> out(c.output().detach(), 1024);\n        out.write(id).get();\n\n        auto fdn = tls::get_dn_information(s.connection);\n\n        auto fout = out.flush();\n        auto fin = in.read();\n\n        fout.get();\n\n        auto dn = fdn.get();\n        auto client_id = fin.get();\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n\n        auto it = dn->subject.find(sstring(client_id.get(), client_id.size()));\n        BOOST_REQUIRE(it != sstring::npos);\n    };\n\n    fetch_dn(\"client1.org\", client1_creds);\n    fetch_dn(\"client2.org\", client2_creds);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_alt_names) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_client_auth(tls::client_auth::REQUIRE);\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        output_stream<char> out(c.output().detach(), 1024);\n        out.write(\"nils\").get();\n\n        auto falt_names = tls::get_alt_name_information(s.connection);\n\n        auto fout = out.flush();\n        auto fin = in.read();\n\n        fout.get();\n\n        auto alt_names = falt_names.get();\n        fin.get();\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n\n        auto ensure_alt_name = [&](tls::subject_alt_name_type type, size_t min_count) {\n            for (auto& v : alt_names) {\n                if (type != v.type) {\n                    continue;\n                }\n                std::visit([&](auto& val) {\n                    BOOST_TEST_MESSAGE(fmt::format(\"Alt name type: {}: {}\", int(type), val).c_str());\n                }, v.value);\n                if (--min_count == 0) {\n                    return;\n                }\n            }\n            BOOST_FAIL(\"Missing \" + std::to_string(min_count) + \" alt name attributes of type \" + std::to_string(int(type)));\n        };\n\n        ensure_alt_name(tls::subject_alt_name_type::ipaddress, 1);\n        ensure_alt_name(tls::subject_alt_name_type::rfc822name, 2);\n        ensure_alt_name(tls::subject_alt_name_type::dnsname, 1);\n    }\n\n}\n\nSEASTAR_THREAD_TEST_CASE(test_peer_certificate_chain_handling) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_client_auth(tls::client_auth::REQUIRE);\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        output_stream<char> out(c.output().detach(), 1024);\n        out.write(\"nils\").get();\n\n        auto fscrts = tls::get_peer_certificate_chain(s.connection);\n        auto fccrts = tls::get_peer_certificate_chain(c);\n\n        auto fout = out.flush();\n        auto fin = in.read();\n\n        fout.get();\n\n        auto scrts = fscrts.get();\n        auto ccrts = fccrts.get();\n        fin.get();\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n\n        auto read_file = [](std::filesystem::path const& path) {\n            auto contents = tls::certificate_data(std::filesystem::file_size(path));\n            std::ifstream{path, std::ios_base::binary}.read(reinterpret_cast<char *>(contents.data()), contents.size());\n            return contents;\n        };\n\n        auto ders = {read_file(certfile(\"test.crt.der\"))};\n\n        BOOST_REQUIRE(std::ranges::equal(scrts, ders));\n        BOOST_REQUIRE(std::ranges::equal(ccrts, ders));\n    }\n}\n\nSEASTAR_THREAD_TEST_CASE(test_skip_wait_for_eof) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_client_auth(tls::client_auth::REQUIRE);\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address({0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    {\n        // Initiate a connection while specifying that it should not wait for eof on shutdown.\n        auto sa = server.accept();\n        auto c = engine().connect(addr).get();\n        auto c_tls = tls::wrap_client(creds, std::move(c),\n                                      tls::tls_options{.bye_timeout = std::chrono::seconds(0)}).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        auto out = c_tls.output();\n\n        // Write some data in the socket to handshake.\n        out.write(\"apa\").get();\n        auto f = out.flush();\n        auto buf = in.read().get();\n        f.get();\n        BOOST_CHECK(sstring(buf.begin(), buf.end()) == \"apa\");\n\n        // Prevent the server from reading from the connection.\n        // This ensures that it will miss the bye message and not\n        // reply with an eof.\n        server.abort_accept();\n\n        // Initiate closing of the TLS session\n        c_tls.shutdown_input();\n        c_tls.shutdown_output();\n\n        // Ensure that the session is closed promptly. When wait_for_eof_on_shutdown is not\n        // specified, the call to wait_input_shutdown will hang for 10 seconds waiting for\n        // and eof from the server.\n        try {\n            with_timeout(std::chrono::steady_clock::now() + 1s, c_tls.wait_input_shutdown()).get();\n        } catch (timed_out_error&) {\n            BOOST_FAIL(\"Timed out while waiting for input shutdown.\"\n                       \"This indicates the EOF wait was not skipped\");\n        }\n    }\n}\n\nstatic void do_test_tls13_session_tickets(bool reset_server) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_session_resume_mode(tls::session_resume_mode::TLS13_SESSION_TICKET);\n    b.set_priority_string(\"SECURE128:+SECURE192:-VERS-TLS-ALL:+VERS-TLS1.3\");\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    tls::session_data sess_data;\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        auto cin = c.input();\n        output_stream<char> out(c.output().detach(), 1024);\n        output_stream<char> sout(s.connection.output().detach(), 1024);\n\n        // write data in both directions. Required for session data to\n        // become available.\n        out.write(\"nils\").get();\n        auto fin = in.read();\n        auto fout = out.flush();\n\n        fout.get();\n        fin.get();\n\n        sout.write(\"banan\").get();\n        fin = cin.read();\n        fout = sout.flush();\n\n        fout.get();\n        fin.get();\n\n        BOOST_REQUIRE(!tls::check_session_is_resumed(c).get()); // no resume data\n\n        // get ticket data\n        sess_data = tls::get_session_resume_data(c).get();\n        BOOST_REQUIRE(!sess_data.empty());\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n    }\n\n    if (reset_server) {\n        server = {};\n        // rebuild creds\n        serv = b.build_server_credentials();\n        server = tls::listen(serv, addr, opts);\n    }\n\n    {\n        auto sa = server.accept();\n\n        // tell client to try resuming.\n        tls::tls_options tls_opts;\n        tls_opts.session_resume_data = sess_data;\n\n        auto c = tls::connect(creds, addr, tls_opts).get();\n        auto s = sa.get();\n\n        // This is ok. Will force a handshake.\n        auto f = tls::check_session_is_resumed(c);\n\n        // But we need to force some IO to make the\n        // handshake actually happen.\n        auto in = s.connection.input();\n        output_stream<char> out(c.output().detach(), 1024);\n\n        auto fin = in.read();\n\n        out.write(\"nils\").get();\n        auto fout = out.flush();\n\n        fout.get();\n        fin.get();\n\n        BOOST_REQUIRE(f.get()); // Should work\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n    }\n\n}\n\n/**\n * Test TLS13 session ticket support.\n*/\nSEASTAR_THREAD_TEST_CASE(test_tls13_session_tickets) {\n    do_test_tls13_session_tickets(false);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tls13_session_tickets_retain_session_key) {\n    do_test_tls13_session_tickets(true);\n}\n\nSEASTAR_THREAD_TEST_CASE(test_tls13_session_tickets_invalidated_by_reload) {\n    tls::credentials_builder b;\n    tmpdir tmp;\n\n    namespace fs = std::filesystem;\n\n    // copy the wrong certs. We don't trust these\n    // blocking calls, but this is a test and seastar does not have a copy\n    // util and I am lazy...\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    promise<> p;\n\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_session_resume_mode(tls::session_resume_mode::TLS13_SESSION_TICKET);\n    b.set_priority_string(\"SECURE128:+SECURE192:-VERS-TLS-ALL:+VERS-TLS1.3\");\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_reloadable_server_credentials([&p](const std::unordered_set<sstring>&, std::exception_ptr) {\n        p.set_value();\n    }).get();\n\n    auto reloaded = p.get_future();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    tls::session_data sess_data;\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n\n        auto in = s.connection.input();\n        auto cin = c.input();\n        output_stream<char> out(c.output().detach(), 1024);\n        output_stream<char> sout(s.connection.output().detach(), 1024);\n\n        // write data in both directions. Required for session data to\n        // become available.\n        out.write(\"nils\").get();\n        auto fin = in.read();\n        auto fout = out.flush();\n\n        fout.get();\n        fin.get();\n\n        sout.write(\"banan\").get();\n        fin = cin.read();\n        fout = sout.flush();\n\n        fout.get();\n        fin.get();\n\n        BOOST_REQUIRE(!tls::check_session_is_resumed(c).get()); // no resume data\n\n        // get ticket data\n        sess_data = tls::get_session_resume_data(c).get();\n        BOOST_REQUIRE(!sess_data.empty());\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n    }\n\n    BOOST_REQUIRE(!reloaded.available());\n\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test.crt\", fs::copy_options::overwrite_existing);\n    reloaded.get();\n\n    {\n        auto sa = server.accept();\n\n        // tell client to try resuming.\n        tls::tls_options tls_opts;\n        tls_opts.session_resume_data = sess_data;\n\n        auto c = tls::connect(creds, addr, tls_opts).get();\n        auto s = sa.get();\n\n        // This is ok. Will force a handshake.\n        auto f = tls::check_session_is_resumed(c);\n\n        // But we need to force some IO to make the\n        // handshake actually happen.\n        auto in = s.connection.input();\n        output_stream<char> out(c.output().detach(), 1024);\n\n        auto fin = in.read();\n\n        out.write(\"nils\").get();\n        auto fout = out.flush();\n\n        fout.get();\n        fin.get();\n\n        BOOST_REQUIRE(!f.get()); // Should NOT work. Keys should have been replaced\n\n        in.close().get();\n        out.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n    }\n\n}\n\nSEASTAR_THREAD_TEST_CASE(test_reload_certificates_with_only_shard0_notify) {\n    tmpdir tmp;\n\n    namespace fs = std::filesystem;\n\n    // copy the wrong certs. We don't trust these\n    // blocking calls, but this is a test and seastar does not have a copy\n    // util and I am lazy...\n    fs::copy_file(certfile(\"other.crt\"), tmp.path() / \"test.crt\");\n    fs::copy_file(certfile(\"other.key\"), tmp.path() / \"test.key\");\n\n    auto cert = (tmp.path() / \"test.crt\").native();\n    auto key = (tmp.path() / \"test.key\").native();\n    promise<> p;\n\n    tls::credentials_builder b;\n    b.set_x509_key_file(cert, key, tls::x509_crt_format::PEM).get();\n    b.set_dh_level();\n\n    auto certs = b.build_server_credentials();\n\n    auto shard_1_certs = smp::submit_to(1, [&]() -> future<shared_ptr<tls::server_credentials>> {\n        co_return co_await b.build_reloadable_server_credentials([&, changed = std::unordered_set<sstring>{}](const tls::credentials_builder& builder, const std::unordered_set<sstring>& files, std::exception_ptr ep) mutable -> future<> {\n            if (ep) {\n                co_return;\n            }\n            changed.insert(files.begin(), files.end());\n            if (changed.count(cert) && changed.count(key)) {\n                // shard one certs are not reloadable. We issue a reload of them from shard 0\n                // - to save inotify instances.\n                co_await smp::submit_to(0, [&] {\n                    builder.rebuild(*certs);\n                    p.set_value();\n                });\n            }\n        });\n    }).get();\n\n    auto def = defer([&]() noexcept {\n        try {\n            smp::submit_to(1, [&] {\n                shard_1_certs = nullptr;\n            }).get();\n        } catch (...) {}\n    });\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(certs, addr, opts);\n\n    tls::credentials_builder b2;\n    b2.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(b2.build_certificate_credentials(), addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n\n        try {\n            out.write(\"apa\").get();\n            auto f = out.flush();\n            auto f2 = in.read();\n\n            try {\n                f.get();\n                BOOST_FAIL(\"should not reach\");\n            } catch (tls::verification_error&) {\n                // ok\n            }\n            try {\n                out.close().get();\n            } catch (...) {\n            }\n\n            try {\n                f2.get();\n                BOOST_FAIL(\"should not reach\");\n            } catch (...) {\n                // ok\n            }\n            try {\n                in.close().get();\n            } catch (...) {\n            }\n        } catch (tls::verification_error&) {\n            // ok\n        }\n    }\n\n    // copy the right (trusted) certs over the old ones.\n    fs::copy_file(certfile(\"test.crt\"), tmp.path() / \"test0.crt\");\n    fs::copy_file(certfile(\"test.key\"), tmp.path() / \"test0.key\");\n\n    rename_file((tmp.path() / \"test0.crt\").native(), (tmp.path() / \"test.crt\").native()).get();\n    rename_file((tmp.path() / \"test0.key\").native(), (tmp.path() / \"test.key\").native()).get();\n\n    p.get_future().get();\n\n    // now it should work\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(b2.build_certificate_credentials(), addr).get();\n        auto s = sa.get();\n        auto in = s.connection.input();\n\n        output_stream<char> out(c.output().detach(), 4096);\n\n        out.write(\"apa\").get();\n        auto f = out.flush();\n        auto buf = in.read().get();\n        f.get();\n        out.close().get();\n        in.read().get(); // ignore - just want eof\n        in.close().get();\n\n        BOOST_CHECK_EQUAL(sstring(buf.begin(), buf.end()), \"apa\");\n    }\n}\n\nSEASTAR_TEST_CASE(test_tls_cipher_suite_and_protocol_version, *enable_if_with_networking()) {\n    auto certs = ::make_shared<tls::certificate_credentials>();\n    co_await certs->set_system_trust();\n\n    auto c = co_await tls::connect(certs, co_await google_address(), tls::tls_options{ .server_name = google_name });\n    BOOST_CHECK_EQUAL(co_await tls::get_cipher_suite(c), \"TLS_AES_256_GCM_SHA384\");\n    BOOST_CHECK_EQUAL(co_await tls::get_protocol_version(c), \"TLS1.3\");\n}\n\nSEASTAR_TEST_CASE(test_cipher_suite_and_protocol_version_for_non_tls_connection, *enable_if_with_networking()) {\n    auto c = co_await seastar::connect(co_await google_address());\n    BOOST_CHECK_THROW(co_await tls::get_cipher_suite(c), std::invalid_argument);\n    BOOST_CHECK_THROW(co_await tls::get_protocol_version(c), std::invalid_argument);\n}\n\n/**\n * Tests that re-handshaking during a connection lifetime\n * does not cause large buffer allocations inside gnutls.\n * #2859\n */\nSEASTAR_THREAD_TEST_CASE(test_send_recv_alloc_limits) {\n    tls::credentials_builder b;\n\n    b.set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n    b.set_client_auth(tls::client_auth::REQUIRE);\n    b.set_session_resume_mode(tls::session_resume_mode::TLS13_SESSION_TICKET);\n    b.set_priority_string(\"SECURE128:+SECURE192:-VERS-TLS-ALL:-VERS-TLS1.2:+VERS-TLS1.3\");\n\n    auto creds = b.build_certificate_credentials();\n    auto serv = b.build_server_credentials();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    opts.set_fixed_cpu(this_shard_id());\n\n    // cannot use 128k because auto-resize of recv buffers\n    // in posix stack will hit 128 by default, and we cannot\n    // control this on underlying socket side for tls.\n    static constexpr size_t size_limit = 256 * 1024;\n    // make data payload somewhat smaller than maximum\n    static constexpr size_t buf_size = size_limit - 512;\n\n    temporary_buffer<char> to_send(buf_size);\n    std::fill(to_send.get_write(), to_send.get_write() + to_send.size(), 'a');\n\n    auto stats_before = memory::stats();\n    memory::scoped_large_allocation_warning_threshold sct(size_limit);\n\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto server = tls::listen(serv, addr, opts);\n\n    {\n        auto sa = server.accept();\n        auto c = tls::connect(creds, addr).get();\n        auto s = sa.get();\n\n        auto cin = c.input();\n        auto cout = output_stream<char>(c.output().detach(), 1024);\n        auto sin = s.connection.input();\n        auto sout = output_stream<char>(s.connection.output().detach(), 1024);\n\n        constexpr size_t num_loops = (1 << 10);\n\n        auto write = [&](auto& out) {\n            return out.write(to_send.share()).then([&] {\n                return out.flush();\n            });\n        };\n        auto read = [&](auto& in) {\n            // ensure we don't try to read into a large buffer.\n            return in.read_exactly(buf_size / 4).then([&](auto&&) {\n                    return in.read_exactly(buf_size / 4);\n                }).then([&](auto&&) {\n                    return in.read_exactly(buf_size / 4);\n                }).then([&](auto&&) {\n                    return in.read_exactly(buf_size / 4);\n                })\n                ;\n        };\n        for (size_t i = 0; i < num_loops; ++i) {\n            auto fout = write(sout);\n            auto fin = read(cin);\n\n            auto h = (i > 0 && (i & 0xff) == 0)\n                ? BOOST_TEST_MESSAGE(\"Forcing re-handshake\"), tls::force_rehandshake(s.connection)\n                : make_ready_future<>()\n                ;\n\n            fin.get();\n            fout.get();\n\n            fout = write(cout);\n            fin = read(sin);\n\n            fin.get();\n            fout.get();\n\n            h.get();\n\n            if ((i & 0xff) == 0) {\n                BOOST_TEST_MESSAGE(fmt::format(\"Did {} loops\", i));\n            }\n        }\n\n        sout.close().get();\n        sin.close().get();\n        cout.close().get();\n        cin.close().get();\n\n        s.connection.shutdown_input();\n        s.connection.shutdown_output();\n\n        c.shutdown_input();\n        c.shutdown_output();\n\n        auto stats_after = memory::stats();\n\n        BOOST_CHECK_EQUAL(stats_after.large_allocations(), stats_before.large_allocations());\n    }\n}\n\nstatic std::pair<connected_socket, connected_socket> tls_socketpair() {\n    auto certs = ::make_shared<tls::server_credentials>(::make_shared<tls::dh_params>());\n    certs->set_x509_key_file(certfile(\"test.crt\"), certfile(\"test.key\"), tls::x509_crt_format::PEM).get();\n\n    ::listen_options opts;\n    opts.reuse_address = true;\n    auto addr = ::make_ipv4_address( {0x7f000001, 4712});\n    auto ss = tls::listen(certs, addr, opts);\n    tls::credentials_builder b;\n    b.set_x509_trust_file(certfile(\"catest.pem\"), tls::x509_crt_format::PEM).get();\n\n    auto cf = tls::connect(b.build_certificate_credentials(), addr);\n    auto ar = ss.accept().get();\n    auto cs = cf.get();\n\n    return std::make_pair(std::move(ar.connection), std::move(cs));\n}\n\nSEASTAR_THREAD_TEST_CASE(test_session_close_with_unread_data) {\n    auto p = tls_socketpair();\n\n    auto c = seastar::async([c = std::move(p.first)] () mutable {\n        auto in = c.input();\n        auto b = in.read_exactly(1).get();\n        in.close().get();\n        c.shutdown_output();\n    });\n\n    auto s = seastar::async([s = std::move(p.second)] () mutable {\n        auto out = s.output();\n        size_t bytes_sent = 0;\n        auto buf = temporary_buffer<char>(1024);\n        std::memset(buf.get_write(), '\\0', buf.size());\n        auto start = std::chrono::steady_clock::now();\n        while (true) {\n            try {\n                out.write(buf.get(), buf.size()).get();\n                out.flush().get();\n                bytes_sent += buf.size();\n            } catch (...) {\n                break;\n            }\n        }\n        auto delay = std::chrono::duration_cast<std::chrono::duration<double>>(std::chrono::steady_clock::now() - start);\n        BOOST_TEST_MESSAGE(fmt::format(\"Wrote {} bytes in {:.3f} seconds\\n\", bytes_sent, delay.count()));\n        out.close().handle_exception([] (auto x) {}).get();\n        s.shutdown_input();\n        BOOST_CHECK_LT(delay.count(), 1.0);\n    });\n\n    seastar::when_all(std::move(c), std::move(s)).discard_result().get();\n}\n"
  },
  {
    "path": "tests/unit/tmpdir.hh",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2020 ScyllaDB Ltd.\n */\n\n#pragma once\n\n#include <seastar/util/tmp_file.hh>\n\nnamespace seastar {\n\n/**\n * Temp dir helper for RAII usage when doing tests\n * in seastar threads. Will not work in \"normal\" mode.\n * Just use tmp_dir::do_with for that.\n */\nclass tmpdir {\n    seastar::tmp_dir _tmp;\npublic:\n    tmpdir(tmpdir&&) = default;\n    tmpdir(const tmpdir&) = delete;\n\n    tmpdir(const sstring& name = sstring(seastar::default_tmpdir()) + \"/testXXXX\") {\n        _tmp.create(std::filesystem::path(name)).get();\n    }\n    ~tmpdir() {\n        _tmp.remove().get();\n    }\n    auto path() const {\n        return _tmp.get_path();\n    }\n};\n\n}\n"
  },
  {
    "path": "tests/unit/tuple_utils_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2017 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <seastar/util/tuple_utils.hh>\n\n#include <boost/test/unit_test.hpp>\n\n#include <sstream>\n#include <type_traits>\n\nusing namespace seastar;\n\nBOOST_AUTO_TEST_CASE(map) {\n    const auto pairs = tuple_map(std::make_tuple(10, 5.5, true), [](auto&& e) { return std::make_tuple(e, e); });\n\n    BOOST_REQUIRE(pairs == std::make_tuple(std::make_tuple(10, 10),\n                                           std::make_tuple(5.5, 5.5),\n                                           std::make_tuple(true, true)));\n}\n\nBOOST_AUTO_TEST_CASE(for_each) {\n    std::ostringstream os;\n\n    tuple_for_each(std::make_tuple('a', 10, false, 5.4), [&os](auto&& e) {\n        os << e;\n    });\n\n    BOOST_REQUIRE_EQUAL(os.str(), \"a1005.4\");\n}\n\nnamespace {\n\ntemplate <typename T>\nstruct transform_type final {\n    using type = T;\n};\n\ntemplate <>\nstruct transform_type<bool> final { using type = int; };\n\ntemplate <>\nstruct transform_type<double> final { using type = char; };\n\n}\n\nBOOST_AUTO_TEST_CASE(map_types) {\n    using before_tuple = std::tuple<double, bool, const char*>;\n    using after_tuple = typename tuple_map_types<transform_type, before_tuple>::type;\n\n    BOOST_REQUIRE((std::is_same_v<after_tuple, std::tuple<char, int, const char*>>));\n}\n\nnamespace {\n\n//\n// Strip all `bool` fields.\n//\n\ntemplate <typename>\nstruct keep_type final {\n    static constexpr auto value = true;\n};\n\ntemplate <>\nstruct keep_type<bool> final {\n    static constexpr auto value = false;\n};\n\n}\n\nBOOST_AUTO_TEST_CASE(filter_by_type) {\n    using before_tuple = std::tuple<bool, int, bool, double, bool, char>;\n\n    const auto t = tuple_filter_by_type<keep_type>(before_tuple{true, 10, false, 5.5, true, 'a'});\n    using filtered_type = std::decay_t<decltype(t)>;\n\n    BOOST_REQUIRE((std::is_same_v<filtered_type, std::tuple<int, double, char>>));\n    BOOST_REQUIRE(t == std::make_tuple(10, 5.5, 'a'));\n}\n"
  },
  {
    "path": "tests/unit/uname_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright (C) 2019 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/internal/uname.hh>\n\nusing namespace seastar::internal;\n\nBOOST_AUTO_TEST_CASE(test_nowait_aio_fix) {\n    auto check = [] (const char* uname) {\n        return parse_uname(uname).whitelisted({\"5.1\", \"5.0.8\", \"4.19.35\", \"4.14.112\"});\n    };\n    BOOST_REQUIRE_EQUAL(check(\"5.1.0\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.1.1\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.1.1-44.distro\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.1.1-44.7.distro\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.0\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.7\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.7-55.el19\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.8\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.9\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.8-200.fedora\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.0.9-200.fedora\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.2.0\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.2.9\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.2.9-77.el153\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"6.0.0\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"3.9.0\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"4.19\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"4.19.34\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"4.19.35\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"4.19.36\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"4.20.36\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"4.14.111\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"4.14.112\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"4.14.113\"), true);\n}\n\n\nBOOST_AUTO_TEST_CASE(test_xfs_concurrency_fix) {\n    auto check = [] (const char* uname) {\n        return parse_uname(uname).whitelisted({\"3.15\", \"3.10.0-325.el7\"});\n    };\n    BOOST_REQUIRE_EQUAL(check(\"3.15.0\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"5.1.0\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"3.14.0\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.14\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-325.ubuntu\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-325\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-325.el7\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-326.el7\"), true);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-324.el7\"), false);\n    BOOST_REQUIRE_EQUAL(check(\"3.10.0-325.665.el7\"), true);\n}\n"
  },
  {
    "path": "tests/unit/unix_domain_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright (C) 2019 Red Hat, Inc.\n */\n\n#include <filesystem>\n\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/core/seastar.hh>\n#include <seastar/net/api.hh>\n#include <seastar/net/inet_address.hh>\n#include <seastar/net/socket_defs.hh>\n#include <seastar/core/print.hh>\n#include <seastar/core/reactor.hh>\n#include <seastar/core/thread.hh>\n#include <seastar/util/log.hh>\n#include <seastar/util/std-compat.hh>\n\nusing namespace seastar;\nusing std::string;\nusing namespace std::string_literals;\nusing namespace std::chrono_literals;\n\nstatic logger iplog(\"unix_domain\");\n\nclass ud_server_client {\npublic:\n    ud_server_client(string server_path, std::optional<string> client_path, int rounds) :\n        ud_server_client(server_path, client_path, rounds, 0) {};\n\n    ud_server_client(string server_path, std::optional<string> client_path, int rounds,\n                     int abort_run) :\n        server_addr{unix_domain_addr{server_path}}, client_path{client_path},\n                    rounds{rounds},\n                    rounds_left{rounds}, abort_after{abort_run} {}\n\n    future<> run();\n    ud_server_client(ud_server_client&&) = default;\n    ud_server_client(const ud_server_client&) = delete;\n\nprivate:\n    const string test_message{\"are you still the same?\"s};\n    future<> init_server();\n    void client_round();\n    const socket_address server_addr;\n\n    const std::optional<string> client_path;\n    server_socket server;\n    const int rounds;\n    int rounds_left;\n    server_socket* lstn_sock;\n    seastar::thread th;\n    int abort_after; // if set - force the listening socket down after that number of rounds\n    bool planned_abort{false}; // set when abort_accept() is called\n};\n\nfuture<> ud_server_client::init_server() {\n    return do_with(seastar::listen(server_addr), [this](server_socket& lstn) mutable {\n\n        lstn_sock = &lstn; // required when aborting (on some tests)\n\n        //  start the clients here, where we know the server is listening\n\n        th = seastar::thread([this]{\n            for (int i=0; i<rounds; ++i) {\n                if (abort_after) {\n                    if (--abort_after == 0) {\n                        planned_abort = true;\n                        lstn_sock->abort_accept();\n                        break;\n                    }\n                }\n                client_round();\n            }\n        });\n\n        return do_until([this](){return rounds_left<=0;}, [&lstn,this]() {\n            return lstn.accept().then([this](accept_result from_accept) {\n                connected_socket cn    = std::move(from_accept.connection);\n                socket_address cn_addr = std::move(from_accept.remote_address);\n                --rounds_left;\n                //  verify the client address\n                if (client_path) {\n                    socket_address tmmp{unix_domain_addr{*client_path}};\n                    BOOST_REQUIRE_EQUAL(cn_addr, socket_address{unix_domain_addr{*client_path}});\n                }\n\n                return do_with(cn.input(), cn.output(), [](auto& inp, auto& out) {\n\n                    return inp.read().then([&out](auto bb) {\n                        string ans = \"-\"s;\n                        if (bb && bb.size()) {\n                            ans = \"+\"s + string{bb.get(), bb.size()};\n                        }\n                        return out.write(ans).then([&out](){return out.flush();}).\n                        then([&out](){return out.close();});\n                    }).then([&inp]() { return inp.close(); }).\n                    then([]() { return make_ready_future<>(); });\n\n                }).then([]{ return make_ready_future<>();});\n            });\n        }).handle_exception([this](auto e) {\n            // OK to get here only if the test was a \"planned abort\" one\n            if (!planned_abort) {\n                std::rethrow_exception(e);\n            }\n        }).finally([this]{\n            return th.join();\n        });\n    });\n}\n\n/// Send a message to the server, and expect (almost) the same string back.\n/// If 'client_path' is set, the client binds to the named path.\n// Runs in a seastar::thread.\nvoid ud_server_client::client_round() {\n    auto cc = client_path ?\n        engine().net().connect(server_addr, socket_address{unix_domain_addr{*client_path}}).get() :\n        engine().net().connect(server_addr).get();\n\n    auto inp = cc.input();\n    auto out = cc.output();\n\n    out.write(test_message).get();\n    out.flush().get();\n    auto bb = inp.read().get();\n    BOOST_REQUIRE_EQUAL(std::string_view(bb.begin(), bb.size()), \"+\"s+test_message);\n    inp.close().get();\n    out.close().get();\n}\n\nfuture<> ud_server_client::run() {\n    return async([this] {\n        auto serverfut = init_server();\n        (void)serverfut.get();\n    });\n\n}\n\nvoid rm(std::string_view what) {\n    auto res = system(fmt::format(\"rm -f {}\", what).c_str());\n    BOOST_REQUIRE_EQUAL(res, 0);\n}\n\n//  testing the various address types, both on the server and on the\n//  client side\n\nSEASTAR_TEST_CASE(unixdomain_server) {\n    rm(\"/tmp/ry\");\n    ud_server_client uds(\"/tmp/ry\", std::nullopt, 3);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(unixdomain_abs) {\n    char sv_name[]{'\\0', '1', '1', '1'};\n    //ud_server_client uds(string{\"\\0111\",4}, string{\"\\0112\",4}, 1);\n    ud_server_client uds(string{sv_name,4}, std::nullopt, 4);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n    //return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(unixdomain_abs_bind) {\n    char sv_name[]{'\\0', '1', '1', '1'};\n    char cl_name[]{'\\0', '1', '1', '2'};\n    ud_server_client uds(string{sv_name,4}, string{cl_name,4}, 1);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n}\n\nSEASTAR_TEST_CASE(unixdomain_abs_bind_2) {\n    char sv_name[]{'\\0', '1', '\\0', '\\12', '1'};\n    char cl_name[]{'\\0', '1', '\\0', '\\12', '2'};\n    ud_server_client uds(string{sv_name,5}, string{cl_name,5}, 2);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n}\n\nSEASTAR_TEST_CASE(unixdomain_text) {\n    socket_address addr1{unix_domain_addr{\"abc\"}};\n    BOOST_REQUIRE_EQUAL(format(\"{}\", addr1), \"abc\");\n    socket_address addr2{unix_domain_addr{\"\"}};\n    BOOST_REQUIRE_EQUAL(format(\"{}\", addr2), \"{unnamed}\");\n    socket_address addr3{unix_domain_addr{std::string(\"\\0abc\", 5)}};\n    BOOST_REQUIRE_EQUAL(format(\"{}\", addr3), \"@abc_\");\n    return make_ready_future<>();\n}\n\nSEASTAR_TEST_CASE(unixdomain_bind) {\n    rm(\"111 112\");\n    ud_server_client uds(\"111\"s, \"112\"s, 1);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n}\n\nSEASTAR_TEST_CASE(unixdomain_short) {\n    rm(\"3\");\n    ud_server_client uds(\"3\"s, std::nullopt, 10);\n    return do_with(std::move(uds), [](auto& uds){\n        return uds.run();\n    });\n}\n\n//  test our ability to abort the accept()'ing on a socket.\n//  The test covers a specific bug in the handling of abort_accept()\nSEASTAR_TEST_CASE(unixdomain_abort) {\n    std::string sockname{\"7\"s}; // note: no portable & warnings-free option\n    std::ignore = ::unlink(sockname.c_str());\n    ud_server_client uds(sockname, std::nullopt, 10, 4);\n    return do_with(std::move(uds), [sockname](auto& uds){\n        return uds.run().finally([sockname](){\n            std::ignore = ::unlink(sockname.c_str());\n            return seastar::make_ready_future<>();\n        });\n    });\n}\n\n// From man 7 unix:\n//  If a bind(2) call specifies addrlen as sizeof(sa_family_t), or\n//  the SO_PASSCRED socket option was specified for a socket that was\n//  not explicitly bound to an address, then the socket is autobound\n//  to an abstract address.\nstatic socket_address autobind() {\n    socket_address addr;\n    addr.addr_length = offsetof(sockaddr_un, sun_path);\n    addr.u.sa.sa_family = AF_UNIX;\n    return addr;\n}\n\nSEASTAR_THREAD_TEST_CASE(unixdomain_datagram_autobind) {\n    auto chan1 = make_unbound_datagram_channel(AF_UNIX);\n    auto chan2 = make_bound_datagram_channel(autobind());\n\n    chan1.send(chan2.local_address(), \"hello\").get();\n    net::datagram dgram = chan2.receive().get();\n\n    auto bufs = dgram.get_buffers();\n    // POSIX impementation uses single buffer\n    BOOST_REQUIRE_EQUAL(bufs.size(), 1);\n    string received = internal::to_sstring<sstring>(bufs[0]);\n    BOOST_REQUIRE_EQUAL(received, \"hello\");\n}\n\nSEASTAR_THREAD_TEST_CASE(unixdomain_datagram_named_bound) {\n    // Create a temporary directory for the socket file using mkdtemp.\n    char tmpdir[] = \"/tmp/seastar-test-XXXXXX\";\n    char* tmpdir_ptr = mkdtemp(tmpdir);\n    if (tmpdir_ptr == nullptr) {\n        throw std::runtime_error(\"mkdtemp failed\");\n    }\n\n    // Create a socket file in the temporary directory.\n    std::string socket_path = format(\"{}/socket\", tmpdir_ptr);\n    auto named_receiver = make_bound_datagram_channel(socket_address{unix_domain_addr{socket_path}});\n    // Verify that a socket file was created.\n    BOOST_REQUIRE(std::filesystem::exists(socket_path));\n\n    // Send a message to the named socket using an unbound socket.\n    auto sender = make_unbound_datagram_channel(AF_UNIX);\n    sender.send(socket_address{unix_domain_addr{socket_path}}, \"hihi\").get();\n\n    net::udp_datagram dgram = named_receiver.receive().get();\n    auto bufs = dgram.get_buffers();\n    // POSIX impementation uses single buffer\n    BOOST_REQUIRE_EQUAL(bufs.size(), 1);\n    string received = internal::to_sstring<sstring>(bufs[0]);\n    BOOST_REQUIRE_EQUAL(received, \"hihi\");\n\n    // Try to be nice and remove the temporary directory.\n    std::filesystem::remove_all(tmpdir_ptr);\n}\n"
  },
  {
    "path": "tests/unit/unwind_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <pthread.h>\n#include <seastar/core/posix.hh>\n#include <seastar/util/backtrace.hh>\n\nusing namespace seastar;\n\nvoid foo() {\n    throw std::runtime_error(\"foo\");\n}\n\n// Exploits issue #1725\nBOOST_AUTO_TEST_CASE(test_signal_mask_is_preserved_on_unwinding) {\n    sigset_t mask;\n    sigset_t old;\n    sigfillset(&mask);\n    auto res = ::pthread_sigmask(SIG_SETMASK, &mask, &old);\n    throw_pthread_error(res);\n\n    // Check which signals we actually managed to block\n    res = ::pthread_sigmask(SIG_SETMASK, NULL, &mask);\n    throw_pthread_error(res);\n\n    try {\n        foo();\n    } catch (...) {\n        // ignore\n    }\n\n    // Check backtrace() if execinfo.h is present\n#ifndef SEASTAR_BACKTRACE_UNIMPLEMENTED\n    {\n        size_t count = 0;\n        backtrace([&count] (auto) { ++count; });\n        BOOST_REQUIRE(count > 0);\n    }\n#endif\n\n    {\n        sigset_t mask2;\n        auto res = ::pthread_sigmask(SIG_SETMASK, &old, &mask2);\n        throw_pthread_error(res);\n\n        for (int i = 1; i < NSIG; ++i) {\n            BOOST_REQUIRE(sigismember(&mask2, i) == sigismember(&mask, i));\n        }\n    }\n}\n"
  },
  {
    "path": "tests/unit/variant_utils_test.cc",
    "content": "#define BOOST_TEST_MODULE core\n\n#include <seastar/util/variant_utils.hh>\n\n#include <boost/test/unit_test.hpp>\n\n#include <variant>\n\nstruct noncopyable_type {\n    noncopyable_type() = default;\n    ~noncopyable_type() = default;\n    noncopyable_type(const noncopyable_type&) = delete;\n    noncopyable_type& operator=(const noncopyable_type&) = delete;\n    noncopyable_type(noncopyable_type&&) noexcept = default;\n    noncopyable_type& operator=(noncopyable_type&&) noexcept = default;\n};\n\nusing variant_t = std::variant<int, bool>;\nstatic noncopyable_type t{};\n\nstatic_assert(\n  std::is_same_v<\n    decltype(seastar::visit(\n      variant_t{true},\n      [](const int& i) -> noncopyable_type { return noncopyable_type{}; },\n      [](const bool& b) -> noncopyable_type { return noncopyable_type{}; })),\n    noncopyable_type>);\n\nstatic_assert(std::is_same_v<\n              decltype(seastar::visit(\n                variant_t{true},\n                [](const int& i) -> noncopyable_type& { return t; },\n                [](const bool& b) -> noncopyable_type& { return t; })),\n              noncopyable_type&>);\n\nstatic_assert(\n  std::is_same_v<\n    decltype(std::visit(\n      [](auto&& v) -> noncopyable_type& { return t; }, variant_t{true})),\n    decltype(seastar::visit(\n      variant_t{true},\n      [](const int& i) -> noncopyable_type& { return t; },\n      [](const bool& b) -> noncopyable_type& { return t; }))>);\n\nstatic_assert(\n  std::is_same_v<\n    decltype(std::visit(\n      [](auto&& v) -> noncopyable_type { return noncopyable_type{}; },\n      variant_t{true})),\n    decltype(seastar::visit(\n      variant_t{true},\n      [](const int& i) -> noncopyable_type { return noncopyable_type{}; },\n      [](const bool& b) -> noncopyable_type { return noncopyable_type{}; }))>);\n\nBOOST_AUTO_TEST_CASE(test_visit_can_return_reference) {\n    auto& std_visit_ref = std::visit(\n      [](auto&& v) -> noncopyable_type& { return t; }, variant_t{true});\n    auto& ss_visit_ref = seastar::visit(\n      variant_t{true},\n      [](const int& i) -> noncopyable_type& { return t; },\n      [](const bool& b) -> noncopyable_type& { return t; });\n    BOOST_REQUIRE_EQUAL(\n      std::addressof(std_visit_ref), std::addressof(ss_visit_ref));\n    BOOST_REQUIRE_EQUAL(std::addressof(t), std::addressof(std_visit_ref));\n}\n"
  },
  {
    "path": "tests/unit/weak_ptr_test.cc",
    "content": "/*\n * This file is open source software, licensed to you under the terms\n * of the Apache License, Version 2.0 (the \"License\").  See the NOTICE file\n * distributed with this work for additional information regarding copyright\n * ownership.  You may not use this file except in compliance with the License.\n *\n * You may obtain a copy of the License at\n *\n *   http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing,\n * software distributed under the License is distributed on an\n * \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n * KIND, either express or implied.  See the License for the\n * specific language governing permissions and limitations\n * under the License.\n */\n\n/*\n * Copyright 2016 ScyllaDB\n */\n\n#define BOOST_TEST_MODULE core\n\n#include <boost/test/unit_test.hpp>\n#include <seastar/core/weak_ptr.hh>\n\nusing namespace seastar;\n\nclass myclass : public weakly_referencable<myclass> {};\n\nstatic_assert(std::is_nothrow_default_constructible_v<myclass>);\n\nstatic_assert(std::is_nothrow_default_constructible_v<weak_ptr<myclass>>);\nstatic_assert(std::is_nothrow_move_constructible_v<weak_ptr<myclass>>);\n\nBOOST_AUTO_TEST_CASE(test_weak_ptr_is_empty_when_default_initialized) {\n    weak_ptr<myclass> wp;\n    BOOST_REQUIRE(!bool(wp));\n}\n\nBOOST_AUTO_TEST_CASE(test_weak_ptr_is_reset) {\n    auto owning_ptr = std::make_unique<myclass>();\n    weak_ptr<myclass> wp = owning_ptr->weak_from_this();\n    BOOST_REQUIRE(bool(wp));\n    BOOST_REQUIRE(&*wp == &*owning_ptr);\n    owning_ptr = {};\n    BOOST_REQUIRE(!bool(wp));\n}\n\nBOOST_AUTO_TEST_CASE(test_const_weak_ptr) {\n    auto owning_ptr = std::make_unique<myclass>();\n\n    weak_ptr<const myclass> cwptr = const_cast<const myclass&>(*owning_ptr).weak_from_this();\n    BOOST_REQUIRE(bool(cwptr));\n    owning_ptr.reset();\n    BOOST_REQUIRE(!bool(cwptr));\n}\n\nclass baseclass {};\nclass myiclass : public baseclass, public weakly_referencable<myiclass> {};\n\nBOOST_AUTO_TEST_CASE(test_base_class_weak_ptr) {\n    auto owning_ptr = std::make_unique<myiclass>();\n\n    auto get_checker = [] (weak_ptr<baseclass> p) {\n        return [p = std::move(p)] (bool v) {\n            BOOST_REQUIRE_EQUAL(bool(p), v);\n        };\n    };\n\n    auto checker = get_checker(owning_ptr->weak_from_this());\n    checker(true);\n    owning_ptr.reset();\n    checker(false);\n}\n\nBOOST_AUTO_TEST_CASE(test_weak_ptr_can_be_moved) {\n    auto owning_ptr = std::make_unique<myclass>();\n    weak_ptr<myclass> wp1 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp2 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp3 = owning_ptr->weak_from_this();\n\n    weak_ptr<myclass> wp3_moved;\n    wp3_moved = std::move(wp3);\n    weak_ptr<myclass> wp1_moved(std::move(wp1));\n    auto wp2_moved = std::move(wp2);\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n    BOOST_REQUIRE(!bool(wp3));\n    BOOST_REQUIRE(bool(wp1_moved));\n    BOOST_REQUIRE(bool(wp2_moved));\n    BOOST_REQUIRE(bool(wp3_moved));\n    BOOST_REQUIRE(wp1_moved.get() == owning_ptr.get());\n    BOOST_REQUIRE(wp2_moved.get() == owning_ptr.get());\n    BOOST_REQUIRE(wp3_moved.get() == owning_ptr.get());\n\n    owning_ptr = {};\n\n    BOOST_REQUIRE(!bool(wp1_moved));\n    BOOST_REQUIRE(!bool(wp2_moved));\n    BOOST_REQUIRE(!bool(wp3_moved));\n}\n\nBOOST_AUTO_TEST_CASE(test_weak_ptr_can_be_copied) {\n    auto owning_ptr = std::make_unique<myclass>();\n    weak_ptr<myclass> wp1 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp2 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp3 = owning_ptr->weak_from_this();\n\n    weak_ptr<myclass> wp1_copied(wp1);\n    auto wp2_copied = wp2;\n    weak_ptr<myclass> wp3_copied;\n    wp3_copied = wp3;\n    BOOST_REQUIRE(bool(wp1));\n    BOOST_REQUIRE(bool(wp2));\n    BOOST_REQUIRE(bool(wp3));\n    BOOST_REQUIRE(bool(wp1_copied));\n    BOOST_REQUIRE(bool(wp2_copied));\n    BOOST_REQUIRE(bool(wp3_copied));\n\n    BOOST_REQUIRE(wp1.get() == wp1_copied.get());\n    BOOST_REQUIRE(wp2.get() == wp2_copied.get());\n    BOOST_REQUIRE(wp3.get() == wp3_copied.get());\n\n    owning_ptr = {};\n\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n    BOOST_REQUIRE(!bool(wp3));\n    BOOST_REQUIRE(!bool(wp1_copied));\n    BOOST_REQUIRE(!bool(wp2_copied));\n    BOOST_REQUIRE(!bool(wp3_copied));\n}\n\nBOOST_AUTO_TEST_CASE(test_multipe_weak_ptrs) {\n    auto owning_ptr = std::make_unique<myclass>();\n\n    weak_ptr<myclass> wp1 = owning_ptr->weak_from_this();\n    BOOST_REQUIRE(bool(wp1));\n    BOOST_REQUIRE(&*wp1 == &*owning_ptr);\n\n    weak_ptr<myclass> wp2 = owning_ptr->weak_from_this();\n    BOOST_REQUIRE(bool(wp2));\n    BOOST_REQUIRE(&*wp2 == &*owning_ptr);\n\n    owning_ptr = {};\n\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n}\n\nBOOST_AUTO_TEST_CASE(test_multipe_weak_ptrs_going_away_first) {\n    auto owning_ptr = std::make_unique<myclass>();\n\n    weak_ptr<myclass> wp1 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp2 = owning_ptr->weak_from_this();\n    weak_ptr<myclass> wp3 = owning_ptr->weak_from_this();\n\n    BOOST_REQUIRE(bool(wp1));\n    BOOST_REQUIRE(bool(wp2));\n    BOOST_REQUIRE(bool(wp3));\n\n    wp2 = {};\n\n    owning_ptr = std::make_unique<myclass>();\n\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n    BOOST_REQUIRE(!bool(wp3));\n\n    wp1 = owning_ptr->weak_from_this();\n    wp2 = owning_ptr->weak_from_this();\n    wp3 = owning_ptr->weak_from_this();\n\n    BOOST_REQUIRE(bool(wp1));\n    BOOST_REQUIRE(bool(wp2));\n    BOOST_REQUIRE(bool(wp3));\n\n    wp3 = {};\n    owning_ptr = std::make_unique<myclass>();\n\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n    BOOST_REQUIRE(!bool(wp3));\n\n    wp1 = owning_ptr->weak_from_this();\n    wp2 = owning_ptr->weak_from_this();\n    wp3 = owning_ptr->weak_from_this();\n\n    wp1 = {};\n    wp3 = {};\n    owning_ptr = std::make_unique<myclass>();\n\n    BOOST_REQUIRE(!bool(wp1));\n    BOOST_REQUIRE(!bool(wp2));\n    BOOST_REQUIRE(!bool(wp3));\n}\n"
  },
  {
    "path": "tests/unit/websocket_test.cc",
    "content": "/*\n * Copyright 2021 ScyllaDB\n */\n\n#include <seastar/core/future.hh>\n#include <seastar/websocket/server.hh>\n#include <seastar/websocket/client.hh>\n#include <seastar/testing/test_case.hh>\n#include <seastar/testing/thread_test_case.hh>\n#include <seastar/http/response_parser.hh>\n#include <seastar/util/defer.hh>\n#include \"loopback_socket.hh\"\n\nusing namespace seastar;\nusing namespace seastar::experimental;\nusing namespace std::literals::string_view_literals;\n\nstd::string build_request(std::string_view key_base64, std::string_view subprotocol) {\n    std::string subprotocol_line;\n    if (!subprotocol.empty()) {\n        subprotocol_line = fmt::format(\"Sec-WebSocket-Protocol: {}\\r\\n\", subprotocol);\n    }\n\n    return fmt::format(\n        \"GET / HTTP/1.1\\r\\n\"\n        \"Upgrade: websocket\\r\\n\"\n        \"Connection: Upgrade\\r\\n\"\n        \"Sec-WebSocket-Key: {}\\r\\n\"\n        \"Sec-WebSocket-Version: 13\\r\\n\"\n        \"{}\"\n        \"\\r\\n\",\n        key_base64,\n        subprotocol_line);\n}\n\nfuture<> test_websocket_handshake_common(std::string subprotocol) {\n    return seastar::async([=] {\n        const std::string request = build_request(\"dGhlIHNhbXBsZSBub25jZQ==\", subprotocol);\n\n        loopback_connection_factory factory;\n        loopback_socket_impl lsi(factory);\n\n        auto acceptor = factory.get_server_socket().accept();\n        auto connector = lsi.connect(socket_address(), socket_address());\n        connected_socket sock = connector.get();\n        auto input = sock.input();\n        auto output = sock.output();\n\n        websocket::server dummy;\n        dummy.register_handler(subprotocol, [] (input_stream<char>& in,\n                        output_stream<char>& out) {\n                return repeat([&in, &out]() {\n                    return in.read().then([&out](temporary_buffer<char> f) {\n                        std::cerr << \"f.size(): \" << f.size() << \"\\n\";\n                        if (f.empty()) {\n                            return make_ready_future<stop_iteration>(stop_iteration::yes);\n                        } else {\n                            return out.write(std::move(f)).then([&out]() {\n                                return out.flush().then([] {\n                                    return make_ready_future<stop_iteration>(stop_iteration::no);\n                                });\n                            });\n                        }\n                    });\n                });\n            });\n        websocket::server_connection conn(dummy, acceptor.get().connection);\n        future<> serve = conn.process();\n        auto close = defer([&conn, &input, &output, &serve] () noexcept {\n            conn.close().get();\n            input.close().get();\n            output.close().get();\n            serve.get();\n         });\n\n        // Send the handshake\n        output.write(request).get();\n        output.flush().get();\n        // Check that the server correctly computed the response\n        // according to WebSocket handshake specification\n        http_response_parser parser;\n        parser.init();\n        input.consume(parser).get();\n        std::unique_ptr<http::reply> resp = parser.get_parsed_response();\n        SEASTAR_ASSERT(resp);\n        sstring websocket_accept = resp->_headers[\"Sec-WebSocket-Accept\"];\n        // Trim possible whitespace prefix\n        auto it = std::find_if(websocket_accept.begin(), websocket_accept.end(), ::isalnum);\n        if (it != websocket_accept.end()) {\n            websocket_accept.erase(websocket_accept.begin(), it);\n        }\n        BOOST_REQUIRE_EQUAL(websocket_accept, \"s3pPLMBiTxaQ9kYGzzhZRbK+xOo=\");\n        for (auto& header : resp->_headers) {\n            std::cout << header.first << ':' << header.second << std::endl;\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_websocket_handshake) {\n    return test_websocket_handshake_common(\"echo\");\n}\n\nSEASTAR_TEST_CASE(test_websocket_handshake_no_subprotocol) {\n    return test_websocket_handshake_common(\"\");\n}\n\nfuture<> test_websocket_handler_registration_common(std::string subprotocol) {\n    return seastar::async([=] {\n        loopback_connection_factory factory;\n        loopback_socket_impl lsi(factory);\n\n        auto acceptor = factory.get_server_socket().accept();\n        auto connector = lsi.connect(socket_address(), socket_address());\n        connected_socket sock = connector.get();\n        auto input = sock.input();\n        auto output = sock.output();\n\n        // Setup server\n        websocket::server ws;\n        ws.register_handler(subprotocol, [] (input_stream<char>& in,\n                        output_stream<char>& out) {\n            return repeat([&in, &out]() {\n                return in.read().then([&out](temporary_buffer<char> f) {\n                    std::cerr << \"f.size(): \" << f.size() << \"\\n\";\n                    if (f.empty()) {\n                        return make_ready_future<stop_iteration>(stop_iteration::yes);\n                    } else {\n                        return out.write(std::move(f)).then([&out]() {\n                            return out.flush().then([] {\n                                return make_ready_future<stop_iteration>(stop_iteration::no);\n                            });\n                        });\n                    }\n                });\n            });\n        });\n        websocket::server_connection conn(ws, acceptor.get().connection);\n        future<> serve = conn.process();\n\n        auto close = defer([&conn, &input, &output, &serve] () noexcept {\n            conn.close().get();\n            input.close().get();\n            output.close().get();\n            serve.get();\n         });\n\n        // handshake\n        const std::string request = build_request(\"dGhlIHNhbXBsZSBub25jZQ==\", subprotocol);\n        output.write(request).get();\n        output.flush().get();\n\n        unsigned reply_size = 156;\n        if (!subprotocol.empty()) {\n            reply_size += (\"\\r\\nSec-WebSocket-Protocol: \"sv).size() + subprotocol.size();\n        }\n        input.read_exactly(reply_size).get();\n\n        unsigned ws_frame_len = 10;\n\n        // Sending and receiving a websocket frame\n        const std::string ws_frame = std::string(\n            \"\\202\\204\"  // 1000 0002 1000 0100\n            \"TEST\"      // Masking Key\n            \"\\0\\0\\0\\0\", ws_frame_len); // Masked Message - TEST\n        const auto rs_frame = std::string(\n            \"\\202\\004\" // 1000 0002 0000 0100\n            \"TEST\", 6);    // Message - TEST\n\n        output.write(ws_frame).get();\n        output.flush().get();\n\n        auto response = input.read_exactly(6).get();\n        auto response_str = std::string(response.begin(), response.end());\n        BOOST_REQUIRE_EQUAL(rs_frame, response_str);\n    });\n}\n\nSEASTAR_TEST_CASE(test_websocket_handler_registration) {\n    return test_websocket_handler_registration_common(\"echo\");\n}\n\nSEASTAR_TEST_CASE(test_websocket_handler_registration_no_subprotocol) {\n    return test_websocket_handler_registration_common(\"\");\n}\n\n// Simple wrapper to help create a testable input_stream.\nclass test_source_impl : public data_source_impl {\n    std::vector<temporary_buffer<char>> _bufs{};\n    size_t _idx = 0;\npublic:\n    void push_back(std::string s) {\n        auto buf = temporary_buffer<char>::copy_of(s);\n        _bufs.emplace_back(std::move(buf));\n    }\n    virtual future<temporary_buffer<char>> get() override {\n        if (_idx < _bufs.size()) {\n            return make_ready_future<temporary_buffer<char>>(_bufs[_idx++].share());\n        }\n        return make_ready_future<temporary_buffer<char>>(temporary_buffer<char>{});\n    }\n};\n\nSEASTAR_TEST_CASE(test_websocket_parser_split) {\n    return seastar::async([] {\n        // Two websocket frames\n        std::string ws_frames = std::string(\n            \"\\x82\\x85\"    // FIN, opcode, mask, payload len (5)\n            \"\\0\\0\\0\\0\"    // Masking Key\n            \"TEST1\"       // payload\n            \"\\x82\\x85\"    // FIN, opcode, mask, payload len (5)\n            \"\\0\\0\\0\\0\"    // Masking Key\n            \"TEST2\"\n            \"\\x82\\x85\"    // FIN, opcode, mask, payload len (5)\n            \"\\0\\0\\0\\0\"    // Masking Key\n            \"TEST3\", 33); // payload\n\n        for (unsigned split_i = 0; split_i < ws_frames.size() - 1; ++split_i) {\n            websocket::websocket_parser parser;\n\n            auto source = std::make_unique<test_source_impl>();\n\n            if (split_i == 0) {\n                source->push_back(ws_frames);\n            } else {\n                source->push_back(ws_frames.substr(0, split_i));\n                source->push_back(ws_frames.substr(split_i));\n            }\n\n            input_stream<char> in{data_source{std::move(source)}};\n\n            std::vector<sstring> results;\n\n            while (true) {\n                in.consume(parser).get();\n                if (parser.eof()) {\n                    break;\n                }\n\n                SEASTAR_ASSERT(parser.is_valid());\n                results.push_back(seastar::to_sstring(parser.result()));\n            }\n\n            SEASTAR_ASSERT(!parser.is_valid());\n            BOOST_REQUIRE_EQUAL(0, parser.result().size());\n\n            std::vector<sstring> expected = {\n                \"TEST1\",\n                \"TEST2\",\n                \"TEST3\",\n            };\n\n            BOOST_REQUIRE_EQUAL(results, expected);\n        }\n    });\n}\n\nSEASTAR_TEST_CASE(test_websocket_client_server) {\n    loopback_connection_factory factory;\n    loopback_socket_impl lsi(factory);\n\n    // Setup server side\n    websocket::server ws;\n    ws.register_handler(\"echo\", [] (input_stream<char>& in,\n                    output_stream<char>& out) -> future<> {\n        while (true) {\n            auto f = co_await in.read();\n            if (f.empty()) {\n                break;\n            }\n\n            co_await out.write(std::move(f));\n            co_await out.flush();\n        }\n    });\n\n    auto acceptor = factory.get_server_socket().accept();\n    auto connector = lsi.connect(socket_address(), socket_address());\n\n    // Server side\n    connected_socket server_sock = (co_await std::move(acceptor)).connection;\n    websocket::server_connection server_conn(ws, std::move(server_sock));\n    auto serve = server_conn.process();\n\n    // Client side\n    connected_socket client_sock = co_await std::move(connector);\n\n    sstring received_data;\n    promise<> client_done;\n\n    websocket::client_connection client_conn(std::move(client_sock), \"/\", \"localhost\",\n        \"echo\",\n        [&received_data, &client_done] (input_stream<char>& in, output_stream<char>& out) -> future<> {\n            co_await out.write(\"hello\");\n            co_await out.flush();\n\n            auto buf = co_await in.read();\n            received_data = sstring(buf.get(), buf.size());\n\n            co_await out.close();\n            client_done.set_value();\n        });\n\n    co_await client_conn.handshake();\n    auto client_process = client_conn.process().handle_exception(\n        [] (std::exception_ptr) {});\n\n    co_await client_done.get_future();\n\n    BOOST_REQUIRE_EQUAL(received_data, \"hello\");\n\n    // Shut down cleanly: close client first (sends CLOSE frame), then server\n    co_await client_conn.close();\n    co_await std::move(client_process);\n    co_await server_conn.close();\n    co_await std::move(serve);\n}\n"
  }
]